Returns the vocabulary of a model — transformer_vocab • pangoling

,

Returns the vocabulary of a model.

Usage

transformer_vocab(
  model = getOption("pangoling.causal.default"),
  add_special_tokens = NULL,
  config_tokenizer = NULL
)

Arguments

model: Name of a pre-trained model or folder.
add_special_tokens: Whether to include special tokens. It has the same default as the AutoTokenizer method in Python.
config_tokenizer: List with other arguments that control how the tokenizer from Hugging Face is accessed.

Value

A vector with the vocabulary of a model.

See also

Other token-related functions: ntokens(), tokenize_lst()

Examples

if (FALSE) { # interactive()
transformer_vocab(model = "gpt2")
}