class_docstr: This tokenizer is based on GPTNeoXJapaneseTokenizer and has the following modifications
  - Decoding byte0~byte255 tokens correctly
  - Added bagofword token handling
  - Return token_type_ids for Prefix-LM model
  The bagofword token represents a repetition of the previous token and is converted to 3 consecutive tokens when
  decoding In addition, the original Japanese special Sub-Word-Encoding has been released in this repository
  (https://github.com/tanreinama/Japanese-BPEEncoder_V2). The token_type_ids is a mask indicating the prefix input
  position of the Prefix-LM model. To specify a prefix position, specify a prefix input for prefix_text, or specify a
  sentence of the prefix part and the part after it as a text pair of batch input.
  
  Example:
  
  ```python
  >>> from transformers import GPTSanJapaneseTokenizer
  
  >>> tokenizer = GPTSanJapaneseTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
  >>> # You can confirm both 慶応 and 慶應 are encoded to 17750
  >>> tokenizer("吾輩は猫である🐯。実は慶応(慶應)大学出身")["input_ids"]
  [35993, 35998, 34347, 31459, 30647, 31448, 25, 30659, 35729, 35676, 32417, 30647, 17750, 35589, 17750, 35590, 321, 1281]
  
  >>> # Both 慶応 and 慶應 are decoded to 慶応
  >>> tokenizer.decode(tokenizer("吾輩は猫である🐯。実は慶応(慶應)大学出身")["input_ids"])
  '吾輩は猫である🐯。実は慶応(慶応)大学出身'
  ```
  
  Example for Prefix-LM:
  
  ```python
  >>> from transformers import GPTSanJapaneseTokenizer
  
  >>> tokenizer = GPTSanJapaneseTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
  >>> tokenizer("実は慶応(慶應)大学出身", prefix_text="吾輩は猫である🐯。")["input_ids"]
  [35993, 34347, 31459, 30647, 31448, 25, 30659, 35729, 35676, 35998, 32417, 30647, 17750, 35589, 17750, 35590, 321, 1281]
  
  >>> # Mask for Prefix-LM inputs
  >>> tokenizer("実は慶応(慶應)大学出身", prefix_text="吾輩は猫である🐯。")["token_type_ids"]
  [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
  ```
  
  Example for batch encode:
  
  ```python
  >>> from transformers import GPTSanJapaneseTokenizer
  
  >>> tokenizer = GPTSanJapaneseTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
  >>> tokenizer([["武田信玄", "は、"], ["織田信長", "の配下の、"]], padding=True)["input_ids"]
  [[35993, 8640, 25948, 35998, 30647, 35675, 35999, 35999], [35993, 10382, 9868, 35998, 30646, 9459, 30646, 35675]]
  
  >>> # Mask for Prefix-LM inputs
  >>> tokenizer([["武田信玄", "は、"], ["織田信長", "の配下の、"]], padding=True)["token_type_ids"]
  [[1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0]]
  
  >>> # Mask for padding
  >>> tokenizer([["武田信玄", "は、"], ["織田信長", "の配下の、"]], padding=True)["attention_mask"]
  [[1, 1, 1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1]]
  ```
  
  Args:
      vocab_file (`str`):
          File containing the vocabulary.
      emoji_file (`str`):
          File containing the emoji.
      unk_token (`str`, *optional*, defaults to `"<|nottoken|>"`):
          The token used for unknown charactor
      pad_token (`str`, *optional*, defaults to `"<|separator|>"`):
          The token used for padding
      bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`):
          The beginning of sequence token.
      eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
          The end of sequence token.
      sep_token (`str`, *optional*, defaults to `"<|segmenter|>"`):
          A special token to separate token to prefix part and general input part.
      do_clean_text (`bool`, *optional*, defaults to `False`):
          Whether or not to clean text for URL, EMAIL, TEL, Japanese DATE and Japanese PRICE.
class_docstr_tok: ["This","tokenizer","is","based","on","GPTNeoXJapaneseTokenizer","and","has","the","following","modifications","-","Decoding","byte0~byte255","tokens","correctly","-","Added","bagofword","token","handling","-","Return","token_type_ids","for","Prefix-LM","model","The","bagofword","token","represents","a","repetition","of","the","previous","token","and","is","converted","to","3","consecutive","tokens","when","decoding","In","addition",",","the","original","Japanese","special","Sub-Word-Encoding","has","been","released","in","this","repository","(","https",":","\/\/github.com\/tanreinama\/Japanese-BPEEncoder_V2",")",".","The","token_type_ids","is","a","mask","indicating","the","prefix","input","position","of","the","Prefix-LM","model",".","To","specify","a","prefix","position",",","specify","a","prefix","input","for","prefix_text",",","or","specify","a","sentence","of","the","prefix","part","and","the","part","after","it","as","a","text","pair","of","batch","input",".","Example",":","``","`","python",">",">",">","from","transformers","import","GPTSanJapaneseTokenizer",">",">",">","tokenizer","=","GPTSanJapaneseTokenizer.from_pretrained","(","``","Tanrei\/GPTSAN-japanese","''",")",">",">",">","#","You","can","confirm","both","\u6176\u5fdc","and","\u6176\u61c9","are","encoded","to","17750",">",">",">","tokenizer","(","``","\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\ud83d\udc2f\u3002\u5b9f\u306f\u6176\u5fdc","(","\u6176\u61c9",")","\u5927\u5b66\u51fa\u8eab","''",")","[","``","input_ids","''","]","[","35993",",","35998",",","34347",",","31459",",","30647",",","31448",",","25",",","30659",",","35729",",","35676",",","32417",",","30647",",","17750",",","35589",",","17750",",","35590",",","321",",","1281","]",">",">",">","#","Both","\u6176\u5fdc","and","\u6176\u61c9","are","decoded","to","\u6176\u5fdc",">",">",">","tokenizer.decode","(","tokenizer","(","``","\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\ud83d\udc2f\u3002\u5b9f\u306f\u6176\u5fdc","(","\u6176\u61c9",")","\u5927\u5b66\u51fa\u8eab","''",")","[","``","input_ids","''","]",")","'\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\ud83d\udc2f\u3002\u5b9f\u306f\u6176\u5fdc","(","\u6176\u5fdc",")","\u5927\u5b66\u51fa\u8eab'","``","`","Example","for","Prefix-LM",":","``","`","python",">",">",">","from","transformers","import","GPTSanJapaneseTokenizer",">",">",">","tokenizer","=","GPTSanJapaneseTokenizer.from_pretrained","(","``","Tanrei\/GPTSAN-japanese","''",")",">",">",">","tokenizer","(","``","\u5b9f\u306f\u6176\u5fdc","(","\u6176\u61c9",")","\u5927\u5b66\u51fa\u8eab","''",",","prefix_text=","''","\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\ud83d\udc2f\u3002","''",")","[","``","input_ids","''","]","[","35993",",","34347",",","31459",",","30647",",","31448",",","25",",","30659",",","35729",",","35676",",","35998",",","32417",",","30647",",","17750",",","35589",",","17750",",","35590",",","321",",","1281","]",">",">",">","#","Mask","for","Prefix-LM","inputs",">",">",">","tokenizer","(","``","\u5b9f\u306f\u6176\u5fdc","(","\u6176\u61c9",")","\u5927\u5b66\u51fa\u8eab","''",",","prefix_text=","''","\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\ud83d\udc2f\u3002","''",")","[","``","token_type_ids","''","]","[","1",",","1",",","1",",","1",",","1",",","1",",","1",",","1",",","1",",","0",",","0",",","0",",","0",",","0",",","0",",","0",",","0",",","0","]","``","`","Example","for","batch","encode",":","``","`","python",">",">",">","from","transformers","import","GPTSanJapaneseTokenizer",">",">",">","tokenizer","=","GPTSanJapaneseTokenizer.from_pretrained","(","``","Tanrei\/GPTSAN-japanese","''",")",">",">",">","tokenizer","(","[","[","``","\u6b66\u7530\u4fe1\u7384","''",",","``","\u306f\u3001","''","]",",","[","``","\u7e54\u7530\u4fe1\u9577","''",",","``","\u306e\u914d\u4e0b\u306e\u3001","''","]","]",",","padding=True",")","[","``","input_ids","''","]","[","[","35993",",","8640",",","25948",",","35998",",","30647",",","35675",",","35999",",","35999","]",",","[","35993",",","10382",",","9868",",","35998",",","30646",",","9459",",","30646",",","35675","]","]",">",">",">","#","Mask","for","Prefix-LM","inputs",">",">",">","tokenizer","(","[","[","``","\u6b66\u7530\u4fe1\u7384","''",",","``","\u306f\u3001","''","]",",","[","``","\u7e54\u7530\u4fe1\u9577","''",",","``","\u306e\u914d\u4e0b\u306e\u3001","''","]","]",",","padding=True",")","[","``","token_type_ids","''","]","[","[","1",",","1",",","1",",","0",",","0",",","0",",","0",",","0","]",",","[","1",",","1",",","1",",","0",",","0",",","0",",","0",",","0","]","]",">",">",">","#","Mask","for","padding",">",">",">","tokenizer","(","[","[","``","\u6b66\u7530\u4fe1\u7384","''",",","``","\u306f\u3001","''","]",",","[","``","\u7e54\u7530\u4fe1\u9577","''",",","``","\u306e\u914d\u4e0b\u306e\u3001","''","]","]",",","padding=True",")","[","``","attention_mask","''","]","[","[","1",",","1",",","1",",","1",",","1",",","1",",","0",",","0","]",",","[","1",",","1",",","1",",","1",",","1",",","1",",","1",",","1","]","]","``","`","Args",":","vocab_file","(","`","str","`",")",":","File","containing","the","vocabulary",".","emoji_file","(","`","str","`",")",":","File","containing","the","emoji",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|nottoken|",">","''","`",")",":","The","token","used","for","unknown","charactor","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|separator|",">","''","`",")",":","The","token","used","for","padding","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|startoftext|",">","''","`",")",":","The","beginning","of","sequence","token",".","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|endoftext|",">","''","`",")",":","The","end","of","sequence","token",".","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|segmenter|",">","''","`",")",":","A","special","token","to","separate","token","to","prefix","part","and","general","input","part",".","do_clean_text","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","clean","text","for","URL",",","EMAIL",",","TEL",",","Japanese","DATE","and","Japanese","PRICE","."]
code: def _tokenize(self, text):
      return self.subword_tokenizer.tokenize(
          text, clean=self.do_clean_text
      )
  
imports: import collections
  import json
  import os
  import re
  from typing import List, Optional, Tuple, Union
  import numpy
  from ...tokenization_utils import PreTrainedTokenizer
  from ...tokenization_utils_base import BatchEncoding, PreTokenizedInput, PreTokenizedInputPair, TextInput, TextInputPair, TruncationStrategy
  from ...utils import PaddingStrategy, logging
label_desc: Use image node_id 4 for calling the GPTSanJapaneseTokenizer obj's underlying member method code with example usage: obj._tokenize(text) and returns: self