TokenIndexer(self, token_min_padding_length:int=0) -> None
TokenIndexer determines how string tokens get represented as arrays of indices in a model.
This class both converts strings into numerical values, with the help of a
~allennlp.data.vocabulary.Vocabulary, and it produces actual arrays.
Tokens can be represented as single IDs (e.g., the word "cat" gets represented by the number 34), or as lists of character IDs (e.g., "cat" gets represented by the numbers [23, 10, 18]), or in some other way that you can come up with (e.g., if you have some structured input you want to represent in a special way in your data arrays, you can do that here).
- token_min_padding_length :
int, optional (default=
- The minimum padding length required for the :class:
TokenIndexer. For example,
- the minimum padding length of :class:
SingleIdTokenIndexeris the largest size of
- filter when using :class:
CnnEncoder. Note that if you set this for one TokenIndexer, you likely have to set it for all
TokenIndexerfor the same field, otherwise you'll get mismatched tensor sizes.
str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.
bool(x) -> bool
Returns True when the argument x is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
TokenIndexer.count_vocab_items(self, token:allennlp.data.tokenizers.token.Token, counter:Dict[str, Dict[str, int]])
Vocabulary needs to assign indices to whatever strings we see in the training
data (possibly doing some frequency filtering and using an OOV, or out of vocabulary,
token). This method takes a token and a dictionary of counts and increments counts for
whatever vocabulary items are present in the token. If this is a single token ID
representation, the vocabulary item is likely the token itself. If this is a token
characters representation, the vocabulary items are all of the characters in the token.
TokenIndexer.tokens_to_indices(self, tokens:List[allennlp.data.tokenizers.token.Token], vocabulary:allennlp.data.vocabulary.Vocabulary) -> Dict[str, List[Any]]
Takes a list of tokens and converts them to an
This could be just an ID for each token from the vocabulary.
Or it could split each token into characters and return one ID per character.
Or (for instance, in the case of byte-pair encoding) there might not be a clean
mapping from individual tokens to indices, and the
IndexedTokenList could be a complex
TokenIndexer.get_empty_token_list(self) -> Dict[str, List[Any]]
already indexed version of an empty token list. This is typically just an
empty list for whatever keys are used in the indexer.
TokenIndexer.get_padding_lengths(self, indexed_tokens:Dict[str, List[Any]]) -> Dict[str, int]
This method returns a padding dictionary for the given
indexed_tokens specifying all
lengths that need padding. If all you have is a list of single ID tokens, this is just the
length of the list, and that's what the default implementation will give you. If you have
something more complicated, like a list of character ids for token, you'll need to override
TokenIndexer.as_padded_tensor_dict(self, tokens:Dict[str, List[Any]], padding_lengths:Dict[str, int]) -> Dict[str, torch.Tensor]
This method pads a list of tokens given the input padding lengths (which could actually
truncate things, depending on settings) and returns that padded list of input tokens as a
Dict[str, torch.Tensor]. This is a dictionary because there should be one key per
argument that the
TokenEmbedder corresponding to this class expects in its
method (where the argument name in the
TokenEmbedder needs to make the key in this
The base class implements the case when all you want to do is create a padded
for every list in the
tokens dictionary. If your
TokenIndexer needs more complex
logic than that, you need to override this method.