class_docstr: Tokenizer used for BERT.
  
    This tokenizer applies an end-to-end, text string to wordpiece tokenization.
    It first applies basic tokenization, followed by wordpiece
    tokenization.
  
    See `WordpieceTokenizer` for details on the subword tokenization.
  
    For an example of use, see
    https://www.tensorflow.org/text/guide/bert_preprocessing_guide
  
  Attributes:
    vocab_lookup_table: A lookup table implementing the LookupInterface
      containing the vocabulary of subwords or a string which is the file path
      to the vocab.txt file.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword. Default is '##'.
    max_bytes_per_word: (optional) Max size of input token. Default is 100.
    max_chars_per_token: (optional) Max size of subwords, excluding suffix
      indicator. If known, providing this improves the efficiency of decoding
      long words.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` IDs, or `tf.string` subwords. The default is `tf.int64`.
    unknown_token: (optional) The value to use when an unknown token is found.
      Default is "[UNK]". If this is set to a string, and `token_out_type` is
      `tf.int64`, the `vocab_lookup_table` is used to convert the
      `unknown_token` to an integer. If this is set to `None`, out-of-vocabulary
      tokens are left as is.
    split_unknown_characters: (optional) Whether to split out single unknown
      characters as subtokens. If False (default), words containing unknown
      characters will be treated as single unknown tokens.
    lower_case: bool - If true, a preprocessing step is added to lowercase the
      text, apply NFD normalization, and strip accents characters.
    keep_whitespace: bool - If true, preserves whitespace characters instead of
      stripping them away.
    normalization_form: If set to a valid value and lower_case=False, the input
      text will be normalized to `normalization_form`. See normalize_utf8() op
      for a list of valid values.
    preserve_unused_token: If true, text in the regex format `\\[unused\\d+\\]`
      will be treated as a token and thus remain preserved as is to be looked up
      in the vocabulary.
    basic_tokenizer_class: If set, the class to use instead of BasicTokenizer
class_docstr_tok: ["Tokenizer","used","for","BERT",".","This","tokenizer","applies","an","end-to-end",",","text","string","to","wordpiece","tokenization",".","It","first","applies","basic","tokenization",",","followed","by","wordpiece","tokenization",".","See","`","WordpieceTokenizer","`","for","details","on","the","subword","tokenization",".","For","an","example","of","use",",","see","https",":","\/\/www.tensorflow.org\/text\/guide\/bert_preprocessing_guide","Attributes",":","vocab_lookup_table",":","A","lookup","table","implementing","the","LookupInterface","containing","the","vocabulary","of","subwords","or","a","string","which","is","the","file","path","to","the","vocab.txt","file",".","suffix_indicator",":","(","optional",")","The","characters","prepended","to","a","wordpiece","to","indicate","that","it","is","a","suffix","to","another","subword",".","Default","is","'","#","#","'",".","max_bytes_per_word",":","(","optional",")","Max","size","of","input","token",".","Default","is","100.","max_chars_per_token",":","(","optional",")","Max","size","of","subwords",",","excluding","suffix","indicator",".","If","known",",","providing","this","improves","the","efficiency","of","decoding","long","words",".","token_out_type",":","(","optional",")","The","type","of","the","token","to","return",".","This","can","be","`","tf.int64","`","IDs",",","or","`","tf.string","`","subwords",".","The","default","is","`","tf.int64","`",".","unknown_token",":","(","optional",")","The","value","to","use","when","an","unknown","token","is","found",".","Default","is","``","[","UNK","]","''",".","If","this","is","set","to","a","string",",","and","`","token_out_type","`","is","`","tf.int64","`",",","the","`","vocab_lookup_table","`","is","used","to","convert","the","`","unknown_token","`","to","an","integer",".","If","this","is","set","to","`","None","`",",","out-of-vocabulary","tokens","are","left","as","is",".","split_unknown_characters",":","(","optional",")","Whether","to","split","out","single","unknown","characters","as","subtokens",".","If","False","(","default",")",",","words","containing","unknown","characters","will","be","treated","as","single","unknown","tokens",".","lower_case",":","bool","-","If","true",",","a","preprocessing","step","is","added","to","lowercase","the","text",",","apply","NFD","normalization",",","and","strip","accents","characters",".","keep_whitespace",":","bool","-","If","true",",","preserves","whitespace","characters","instead","of","stripping","them","away",".","normalization_form",":","If","set","to","a","valid","value","and","lower_case=False",",","the","input","text","will","be","normalized","to","`","normalization_form","`",".","See","normalize_utf8","(",")","op","for","a","list","of","valid","values",".","preserve_unused_token",":","If","true",",","text","in","the","regex","format","`","\\\\","[","unused\\\\d+\\\\","]","`","will","be","treated","as","a","token","and","thus","remain","preserved","as","is","to","be","looked","up","in","the","vocabulary",".","basic_tokenizer_class",":","If","set",",","the","class","to","use","instead","of","BasicTokenizer"]
code: def __init__(
      self,
      vocab_lookup_table,
      suffix_indicator="##",
      max_bytes_per_word=100,
      max_chars_per_token=None,
      token_out_type=dtypes.int64,
      unknown_token="[UNK]",
      split_unknown_characters=False,
      lower_case=False,
      keep_whitespace=False,
      normalization_form=None,
      preserve_unused_token=False,
      basic_tokenizer_class=BasicTokenizer,
  ):
      super(BertTokenizer, self).__init__()
      _tf_text_bert_tokenizer_op_create_counter.get_cell().increase_by(
          1
      )
  
      self._basic_tokenizer = basic_tokenizer_class(
          lower_case,
          keep_whitespace,
          normalization_form,
          preserve_unused_token,
      )
      self._wordpiece_tokenizer = WordpieceTokenizer(
          vocab_lookup_table,
          suffix_indicator,
          max_bytes_per_word,
          max_chars_per_token,
          token_out_type,
          unknown_token,
          split_unknown_characters,
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import copy
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.ops import array_ops
  from tensorflow.python.ops import string_ops
  from tensorflow_text.python.ops import regex_split_ops
  from tensorflow_text.python.ops.normalize_ops import case_fold_utf8
  from tensorflow_text.python.ops.normalize_ops import normalize_utf8
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow_text.python.ops.wordpiece_tokenizer import WordpieceTokenizer
label_desc: Use image node_id 1 to create a new BertTokenizer object from inherited base classes: TokenizerWithOffsets, Detokenizer with example: obj = BertTokenizer(vocab_lookup_table, suffix_indicator, max_bytes_per_word, max_chars_per_token, token_out_type, unknown_token, split_unknown_characters, lower_case, keep_whitespace, normalization_form, preserve_unused_token, basic_tokenizer_class)