class_docstr: A ``Tokenizer`` splits strings of text into tokens.  Typically, this either splits text into
  word tokens or character tokens, and those are the two tokenizer subclasses we have implemented
  here, though you could imagine wanting to do other kinds of tokenization for structured or
  other inputs.
  
  As part of tokenization, concrete implementations of this API will also handle stemming,
  stopword filtering, adding start and end tokens, or other kinds of things you might want to do
  to your tokens.  See the parameters to, e.g., :class:`~.WordTokenizer`, or whichever tokenizer
  you want to use.
  
  If the base input to your model is words, you should use a :class:`~.WordTokenizer`, even if
  you also want to have a character-level encoder to get an additional vector for each word
  token.  Splitting word tokens into character arrays is handled separately, in the
  :class:`..token_representations.TokenRepresentation` class.
class_docstr_tok: ["A","``","Tokenizer","``","splits","strings","of","text","into","tokens",".","Typically",",","this","either","splits","text","into","word","tokens","or","character","tokens",",","and","those","are","the","two","tokenizer","subclasses","we","have","implemented","here",",","though","you","could","imagine","wanting","to","do","other","kinds","of","tokenization","for","structured","or","other","inputs",".","As","part","of","tokenization",",","concrete","implementations","of","this","API","will","also","handle","stemming",",","stopword","filtering",",","adding","start","and","end","tokens",",","or","other","kinds","of","things","you","might","want","to","do","to","your","tokens",".","See","the","parameters","to",",","e.g.",",",":","class",":","`","~.WordTokenizer","`",",","or","whichever","tokenizer","you","want","to","use",".","If","the","base","input","to","your","model","is","words",",","you","should","use","a",":","class",":","`","~.WordTokenizer","`",",","even","if","you","also","want","to","have","a","character-level","encoder","to","get","an","additional","vector","for","each","word","token",".","Splitting","word","tokens","into","character","arrays","is","handled","separately",",","in","the",":","class",":","`","..","token_representations.TokenRepresentation","`","class","."]
docstr: Actually implements splitting words into tokens.
  
  Returns
  -------
  tokens : ``List[Token]``
docstr_tok: ["Actually","implements","splitting","words","into","tokens",".","Returns","--","--","--","-","tokens",":","``","List","[","Token","]","``"]
code: def tokenize(self, text):
      """
      Actually implements splitting words into tokens.
  
      Returns
      -------
      tokens : ``List[Token]``
      """
      raise NotImplementedError
  
imports: from __future__ import absolute_import
  from allennlp.common import Registrable
  from allennlp.data.tokenizers.token import Token
label_desc: Use image node_id 2 for calling the Tokenizer obj's underlying member method code with example usage: obj.tokenize(text) without return types