class_docstr: Constructs a Ernie-M tokenizer. It uses the `sentencepiece` tools to cut the words to sub-words.
  
  Args:
      sentencepiece_model_file (`str`):
          The file path of sentencepiece model.
      vocab_file (`str`, *optional*):
          The file path of the vocabulary.
      do_lower_case (`str`, *optional*, defaults to `True`):
          Whether or not to lowercase the input when tokenizing.
      unk_token (`str`, *optional*, defaults to `"[UNK]"`):
          A special token representing the `unknown (out-of-vocabulary)` token. An unknown token is set to be
          `unk_token` inorder to be converted to an ID.
      sep_token (`str`, *optional*, defaults to `"[SEP]"`):
          A special token separating two different sentences in the same input.
      pad_token (`str`, *optional*, defaults to `"[PAD]"`):
          A special token used to make arrays of tokens the same size for batching purposes.
      cls_token (`str`, *optional*, defaults to `"[CLS]"`):
          A special token used for sequence classification. It is the last token of the sequence when built with
          special tokens.
      mask_token (`str`, *optional*, defaults to `"[MASK]"`):
          A special token representing a masked token. This is the token used in the masked language modeling task
          which the model tries to predict the original unmasked ones.
class_docstr_tok: ["Constructs","a","Ernie-M","tokenizer",".","It","uses","the","`","sentencepiece","`","tools","to","cut","the","words","to","sub-words",".","Args",":","sentencepiece_model_file","(","`","str","`",")",":","The","file","path","of","sentencepiece","model",".","vocab_file","(","`","str","`",",","*","optional","*",")",":","The","file","path","of","the","vocabulary",".","do_lower_case","(","`","str","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","lowercase","the","input","when","tokenizing",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","UNK","]","''","`",")",":","A","special","token","representing","the","`","unknown","(","out-of-vocabulary",")","`","token",".","An","unknown","token","is","set","to","be","`","unk_token","`","inorder","to","be","converted","to","an","ID",".","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","SEP","]","''","`",")",":","A","special","token","separating","two","different","sentences","in","the","same","input",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","PAD","]","''","`",")",":","A","special","token","used","to","make","arrays","of","tokens","the","same","size","for","batching","purposes",".","cls_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","CLS","]","''","`",")",":","A","special","token","used","for","sequence","classification",".","It","is","the","last","token","of","the","sequence","when","built","with","special","tokens",".","mask_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","MASK","]","''","`",")",":","A","special","token","representing","a","masked","token",".","This","is","the","token","used","in","the","masked","language","modeling","task","which","the","model","tries","to","predict","the","original","unmasked","ones","."]
docstr: Build offset map from a pair of offset map by concatenating and adding offsets of special tokens. An Ernie-M
  offset_mapping has the following format:
  
  - single sequence: `(0,0) X (0,0)`
  - pair of sequences: `(0,0) A (0,0) (0,0) B (0,0)`
  
  Args:
      offset_mapping_ids_0 (`List[tuple]`):
          List of char offsets to which the special tokens will be added.
      offset_mapping_ids_1 (`List[tuple]`, *optional*):
          Optional second list of wordpiece offsets for offset mapping pairs.
  Returns:
      `List[tuple]`: List of wordpiece offsets with the appropriate offsets of special tokens.
docstr_tok: ["Build","offset","map","from","a","pair","of","offset","map","by","concatenating","and","adding","offsets","of","special","tokens",".","An","Ernie-M","offset_mapping","has","the","following","format",":","-","single","sequence",":","`","(","0,0",")","X","(","0,0",")","`","-","pair","of","sequences",":","`","(","0,0",")","A","(","0,0",")","(","0,0",")","B","(","0,0",")","`","Args",":","offset_mapping_ids_0","(","`","List","[","tuple","]","`",")",":","List","of","char","offsets","to","which","the","special","tokens","will","be","added",".","offset_mapping_ids_1","(","`","List","[","tuple","]","`",",","*","optional","*",")",":","Optional","second","list","of","wordpiece","offsets","for","offset","mapping","pairs",".","Returns",":","`","List","[","tuple","]","`",":","List","of","wordpiece","offsets","with","the","appropriate","offsets","of","special","tokens","."]
code: def build_offset_mapping_with_special_tokens(
      self, offset_mapping_0, offset_mapping_1=None
  ):
      r"""
      Build offset map from a pair of offset map by concatenating and adding offsets of special tokens. An Ernie-M
      offset_mapping has the following format:
  
      - single sequence: `(0,0) X (0,0)`
      - pair of sequences: `(0,0) A (0,0) (0,0) B (0,0)`
  
      Args:
          offset_mapping_ids_0 (`List[tuple]`):
              List of char offsets to which the special tokens will be added.
          offset_mapping_ids_1 (`List[tuple]`, *optional*):
              Optional second list of wordpiece offsets for offset mapping pairs.
      Returns:
          `List[tuple]`: List of wordpiece offsets with the appropriate offsets of special tokens.
      """
      if offset_mapping_1 is None:
          return [(0, 0)] + offset_mapping_0 + [(0, 0)]
  
      return (
          [(0, 0)]
          + offset_mapping_0
          + [(0, 0), (0, 0)]
          + offset_mapping_1
          + [(0, 0)]
      )
  
imports: import io
  import os
  import unicodedata
  from typing import Any, Dict, List, Optional, Tuple
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 14 for calling the ErnieMTokenizer obj's underlying member method code with example usage: obj.build_offset_mapping_with_special_tokens(offset_mapping_0, offset_mapping_1) and returns: unknown, unknown