class_docstr: Tokenizer used for BERT.
  
    This tokenizer applies an end-to-end, text string to wordpiece tokenization.
    It first applies basic tokenization, followed by wordpiece
    tokenization.
  
    See `WordpieceTokenizer` for details on the subword tokenization.
  
    For an example of use, see
    https://www.tensorflow.org/text/guide/bert_preprocessing_guide
  
  Attributes:
    vocab_lookup_table: A lookup table implementing the LookupInterface
      containing the vocabulary of subwords or a string which is the file path
      to the vocab.txt file.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword. Default is '##'.
    max_bytes_per_word: (optional) Max size of input token. Default is 100.
    max_chars_per_token: (optional) Max size of subwords, excluding suffix
      indicator. If known, providing this improves the efficiency of decoding
      long words.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` IDs, or `tf.string` subwords. The default is `tf.int64`.
    unknown_token: (optional) The value to use when an unknown token is found.
      Default is "[UNK]". If this is set to a string, and `token_out_type` is
      `tf.int64`, the `vocab_lookup_table` is used to convert the
      `unknown_token` to an integer. If this is set to `None`, out-of-vocabulary
      tokens are left as is.
    split_unknown_characters: (optional) Whether to split out single unknown
      characters as subtokens. If False (default), words containing unknown
      characters will be treated as single unknown tokens.
    lower_case: bool - If true, a preprocessing step is added to lowercase the
      text, apply NFD normalization, and strip accents characters.
    keep_whitespace: bool - If true, preserves whitespace characters instead of
      stripping them away.
    normalization_form: If set to a valid value and lower_case=False, the input
      text will be normalized to `normalization_form`. See normalize_utf8() op
      for a list of valid values.
    preserve_unused_token: If true, text in the regex format `\\[unused\\d+\\]`
      will be treated as a token and thus remain preserved as is to be looked up
      in the vocabulary.
    basic_tokenizer_class: If set, the class to use instead of BasicTokenizer
class_docstr_tok: ["Tokenizer","used","for","BERT",".","This","tokenizer","applies","an","end-to-end",",","text","string","to","wordpiece","tokenization",".","It","first","applies","basic","tokenization",",","followed","by","wordpiece","tokenization",".","See","`","WordpieceTokenizer","`","for","details","on","the","subword","tokenization",".","For","an","example","of","use",",","see","https",":","\/\/www.tensorflow.org\/text\/guide\/bert_preprocessing_guide","Attributes",":","vocab_lookup_table",":","A","lookup","table","implementing","the","LookupInterface","containing","the","vocabulary","of","subwords","or","a","string","which","is","the","file","path","to","the","vocab.txt","file",".","suffix_indicator",":","(","optional",")","The","characters","prepended","to","a","wordpiece","to","indicate","that","it","is","a","suffix","to","another","subword",".","Default","is","'","#","#","'",".","max_bytes_per_word",":","(","optional",")","Max","size","of","input","token",".","Default","is","100.","max_chars_per_token",":","(","optional",")","Max","size","of","subwords",",","excluding","suffix","indicator",".","If","known",",","providing","this","improves","the","efficiency","of","decoding","long","words",".","token_out_type",":","(","optional",")","The","type","of","the","token","to","return",".","This","can","be","`","tf.int64","`","IDs",",","or","`","tf.string","`","subwords",".","The","default","is","`","tf.int64","`",".","unknown_token",":","(","optional",")","The","value","to","use","when","an","unknown","token","is","found",".","Default","is","``","[","UNK","]","''",".","If","this","is","set","to","a","string",",","and","`","token_out_type","`","is","`","tf.int64","`",",","the","`","vocab_lookup_table","`","is","used","to","convert","the","`","unknown_token","`","to","an","integer",".","If","this","is","set","to","`","None","`",",","out-of-vocabulary","tokens","are","left","as","is",".","split_unknown_characters",":","(","optional",")","Whether","to","split","out","single","unknown","characters","as","subtokens",".","If","False","(","default",")",",","words","containing","unknown","characters","will","be","treated","as","single","unknown","tokens",".","lower_case",":","bool","-","If","true",",","a","preprocessing","step","is","added","to","lowercase","the","text",",","apply","NFD","normalization",",","and","strip","accents","characters",".","keep_whitespace",":","bool","-","If","true",",","preserves","whitespace","characters","instead","of","stripping","them","away",".","normalization_form",":","If","set","to","a","valid","value","and","lower_case=False",",","the","input","text","will","be","normalized","to","`","normalization_form","`",".","See","normalize_utf8","(",")","op","for","a","list","of","valid","values",".","preserve_unused_token",":","If","true",",","text","in","the","regex","format","`","\\\\","[","unused\\\\d+\\\\","]","`","will","be","treated","as","a","token","and","thus","remain","preserved","as","is","to","be","looked","up","in","the","vocabulary",".","basic_tokenizer_class",":","If","set",",","the","class","to","use","instead","of","BasicTokenizer"]
docstr: Convert a `Tensor` or `RaggedTensor` of wordpiece IDs to string-words.
  
  See `WordpieceTokenizer.detokenize` for details.
  
  Note: `BertTokenizer.tokenize`/`BertTokenizer.detokenize` does not round
  trip losslessly. The result of `detokenize` will not, in general, have the
  same content or offsets as the input to `tokenize`. This is because the
  "basic tokenization" step, that splits the strings into words before
  applying the `WordpieceTokenizer`, includes irreversible
  steps like lower-casing and splitting on punctuation. `WordpieceTokenizer`
  on the other hand **is** reversible.
  
  Note: This method assumes wordpiece IDs are dense on the interval
  `[0, vocab_size)`.
  
  Example:
  >>> import pathlib
  >>> pathlib.Path('/tmp/tok_vocab.txt').write_text(
  ...    "they ##' ##re the great ##est".replace(' ', '\n'))
  >>> tokenizer = BertTokenizer(
  ...    vocab_lookup_table='/tmp/tok_vocab.txt')
  >>> text_inputs = tf.constant(['greatest'.encode('utf-8')])
  >>> tokenizer.detokenize([[4, 5]])
  <tf.RaggedTensor [[b'greatest']]>
  
  Args:
    token_ids: A `RaggedTensor` or `Tensor` with an int dtype.
  
  Returns:
    A `RaggedTensor` with dtype `string` and the same rank as the input
    `token_ids`.
docstr_tok: ["Convert","a","`","Tensor","`","or","`","RaggedTensor","`","of","wordpiece","IDs","to","string-words",".","See","`","WordpieceTokenizer.detokenize","`","for","details",".","Note",":","`","BertTokenizer.tokenize","`","\/","`","BertTokenizer.detokenize","`","does","not","round","trip","losslessly",".","The","result","of","`","detokenize","`","will","not",",","in","general",",","have","the","same","content","or","offsets","as","the","input","to","`","tokenize","`",".","This","is","because","the","''","basic","tokenization","''","step",",","that","splits","the","strings","into","words","before","applying","the","`","WordpieceTokenizer","`",",","includes","irreversible","steps","like","lower-casing","and","splitting","on","punctuation",".","`","WordpieceTokenizer","`","on","the","other","hand","*","*","is","*","*","reversible",".","Note",":","This","method","assumes","wordpiece","IDs","are","dense","on","the","interval","`","[","0",",","vocab_size",")","`",".","Example",":",">",">",">","import","pathlib",">",">",">","pathlib.Path","(","'\/tmp\/tok_vocab.txt","'",")",".write_text","(","...","``","they","#","#","'","#","#","re","the","great","#","#","est","''",".replace","(","'","'",",","'\\n","'",")",")",">",">",">","tokenizer","=","BertTokenizer","(","...","vocab_lookup_table='\/tmp\/tok_vocab.txt","'",")",">",">",">","text_inputs","=","tf.constant","(","[","'greatest'.encode","(","'utf-8","'",")","]",")",">",">",">","tokenizer.detokenize","(","[","[","4",",","5","]","]",")","<","tf.RaggedTensor","[","[","b'greatest","'","]","]",">","Args",":","token_ids",":","A","`","RaggedTensor","`","or","`","Tensor","`","with","an","int","dtype",".","Returns",":","A","`","RaggedTensor","`","with","dtype","`","string","`","and","the","same","rank","as","the","input","`","token_ids","`","."]
code: def detokenize(self, token_ids):
      r"""Convert a `Tensor` or `RaggedTensor` of wordpiece IDs to string-words.
  
      See `WordpieceTokenizer.detokenize` for details.
  
      Note: `BertTokenizer.tokenize`/`BertTokenizer.detokenize` does not round
      trip losslessly. The result of `detokenize` will not, in general, have the
      same content or offsets as the input to `tokenize`. This is because the
      "basic tokenization" step, that splits the strings into words before
      applying the `WordpieceTokenizer`, includes irreversible
      steps like lower-casing and splitting on punctuation. `WordpieceTokenizer`
      on the other hand **is** reversible.
  
      Note: This method assumes wordpiece IDs are dense on the interval
      `[0, vocab_size)`.
  
      Example:
      >>> import pathlib
      >>> pathlib.Path('/tmp/tok_vocab.txt').write_text(
      ...    "they ##' ##re the great ##est".replace(' ', '\n'))
      >>> tokenizer = BertTokenizer(
      ...    vocab_lookup_table='/tmp/tok_vocab.txt')
      >>> text_inputs = tf.constant(['greatest'.encode('utf-8')])
      >>> tokenizer.detokenize([[4, 5]])
      <tf.RaggedTensor [[b'greatest']]>
  
      Args:
        token_ids: A `RaggedTensor` or `Tensor` with an int dtype.
  
      Returns:
        A `RaggedTensor` with dtype `string` and the same rank as the input
        `token_ids`.
      """
      return self._wordpiece_tokenizer.detokenize(token_ids)
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import copy
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.ops import array_ops
  from tensorflow.python.ops import string_ops
  from tensorflow_text.python.ops import regex_split_ops
  from tensorflow_text.python.ops.normalize_ops import case_fold_utf8
  from tensorflow_text.python.ops.normalize_ops import normalize_utf8
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow_text.python.ops.wordpiece_tokenizer import WordpieceTokenizer
label_desc: Use image node_id 4 for calling the BertTokenizer obj's underlying member method code with example usage: obj.detokenize(token_ids) and returns: self