Tag: machine-learning
-
Token, Tokenizer (ko)
Token이란 …one can talk about tokens, individual occurrences of something, and types, the different things present. …English in general is not a stationary ergodic process. But we can nevertheless model it with various stochastic approximations. Tokenizer란 Tokenizer의 종류 WordPiece Byte Pair Encoding (BPE) Unigram LM …The unigram language model makes…
Frodo's 