class_docstr: Basic tokenizer for for tokenizing text.
  
  A basic tokenizer that tokenizes using some deterministic rules:
  - For most languages, this tokenizer will split on whitespace.
  - For Chinese, Japanese, and Korean characters, this tokenizer will split on
      Unicode characters.
  
  Example:
  >>> text_inputs =  [b'taste the rustisc indiefrost']
  >>> tokenizer = BasicTokenizer(
  ...     lower_case=False, normalization_form='NFC')
  >>> tokenizer.tokenize(text_inputs)
  <tf.RaggedTensor [[b'taste', b'the', b'rustisc', b'indiefrost']]>
  
  Attributes:
    lower_case: bool - If true, a preprocessing step is added to lowercase the
      text, which also applies NFD normalization and strip accents from
      characters.
    keep_whitespace: bool - If true, preserves whitespace characters instead of
      stripping them away.
    normalization_form: If set to a valid value and lower_case=False, the input
      text will be normalized to `normalization_form`. See normalize_utf8() op
      for a list of valid values.
    preserve_unused_token: If true, text in the regex format "\\[unused\\d+\\]"
      will be treated as a token and thus remain preserved as-is to be looked up
      in the vocabulary.
class_docstr_tok: ["Basic","tokenizer","for","for","tokenizing","text",".","A","basic","tokenizer","that","tokenizes","using","some","deterministic","rules",":","-","For","most","languages",",","this","tokenizer","will","split","on","whitespace",".","-","For","Chinese",",","Japanese",",","and","Korean","characters",",","this","tokenizer","will","split","on","Unicode","characters",".","Example",":",">",">",">","text_inputs","=","[","b'taste","the","rustisc","indiefrost","'","]",">",">",">","tokenizer","=","BasicTokenizer","(","...","lower_case=False",",","normalization_form='NFC","'",")",">",">",">","tokenizer.tokenize","(","text_inputs",")","<","tf.RaggedTensor","[","[","b'taste","'",",","b'the","'",",","b'rustisc","'",",","b'indiefrost","'","]","]",">","Attributes",":","lower_case",":","bool","-","If","true",",","a","preprocessing","step","is","added","to","lowercase","the","text",",","which","also","applies","NFD","normalization","and","strip","accents","from","characters",".","keep_whitespace",":","bool","-","If","true",",","preserves","whitespace","characters","instead","of","stripping","them","away",".","normalization_form",":","If","set","to","a","valid","value","and","lower_case=False",",","the","input","text","will","be","normalized","to","`","normalization_form","`",".","See","normalize_utf8","(",")","op","for","a","list","of","valid","values",".","preserve_unused_token",":","If","true",",","text","in","the","regex","format","``","\\\\","[","unused\\\\d+\\\\","]","''","will","be","treated","as","a","token","and","thus","remain","preserved","as-is","to","be","looked","up","in","the","vocabulary","."]
code: def tokenize(self, text_input):
      tokens, _, _ = self.tokenize_with_offsets(text_input)
      return tokens
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import copy
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.ops import array_ops
  from tensorflow.python.ops import string_ops
  from tensorflow_text.python.ops import regex_split_ops
  from tensorflow_text.python.ops.normalize_ops import case_fold_utf8
  from tensorflow_text.python.ops.normalize_ops import normalize_utf8
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow_text.python.ops.wordpiece_tokenizer import WordpieceTokenizer
label_desc: Use image node_id 2 for calling the BasicTokenizer obj's underlying member method code with example usage: obj.tokenize(text_input) and returns: tokens