class allennlp.data.dataset_readers.penn_tree_bank.PennTreeBankConstituencySpanDatasetReader(token_indexers: typing.Dict[str, allennlp.data.token_indexers.token_indexer.TokenIndexer] = None, use_pos_tags: bool = True, lazy: bool = False) → None[source]
Reads constituency parses from the WSJ part of the Penn Tree Bank from the LDC. This DatasetReader is designed for use with a span labelling model, so it enumerates all possible spans in the sentence and returns them, along with gold labels for the relevant spans present in a gold tree, if provided.
Parameters: token_indexers : Dict[str, TokenIndexer], optional (default={“tokens”: SingleIdTokenIndexer()}) We use this to define the input representation for the text. See TokenIndexer. Note that the output tags will always correspond to single token IDs based on how they are pre-tokenised in the data file. use_pos_tags : bool, optional, (default = True) Whether or not the instance should contain gold POS tags as a field. lazy : bool, optional, (default = False) Whether or not instances can be consumed lazily.
classmethod from_params(params: allennlp.common.params.Params) → allennlp.data.dataset_readers.penn_tree_bank.PennTreeBankConstituencySpanDatasetReader[source]
text_to_instance(tokens: typing.List[str], pos_tags: typing.List[str] = None, gold_tree: nltk.tree.Tree = None) → allennlp.data.instance.Instance[source]
Parameters: tokens : List[str], required. The tokens in a given sentence. pos_tags List[str], optional, (default = None). The POS tags for the words in the sentence. gold_tree : Tree, optional (default = None). The gold parse tree to create span labels from. An Instance containing the following fields: tokens : TextField The tokens in the sentence. pos_tags : SequenceLabelField The POS tags of the words in the sentence. Only returned if use_pos_tags is True spans : ListField[SpanField] A ListField containing all possible subspans of the sentence. span_labels : SequenceLabelField, optional. The constiutency tags for each of the possible spans, with respect to a gold parse tree. If a span is not contained within the tree, a span will have a NO-LABEL label. gold_tree : MetadataField(Tree) The gold NLTK parse tree for use in evaluation.