class_docstr: Adapted from [`XLMRobertaTokenizer`] and [`LukeTokenizer`]. Based on
  [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      entity_vocab_file (`str`):
          Path to the entity vocabulary file.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      cls_token (`str`, *optional*, defaults to `"<s>"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      mask_token (`str`, *optional*, defaults to `"<mask>"`):
          The token used for masking values. This is the token used when training this model with masked language
          modeling. This is the token which the model will try to predict.
      task (`str`, *optional*):
          Task for which you want to prepare sequences. One of `"entity_classification"`,
          `"entity_pair_classification"`, or `"entity_span_classification"`. If you specify this argument, the entity
          sequence is automatically created based on the given entity span(s).
      max_entity_length (`int`, *optional*, defaults to 32):
          The maximum length of `entity_ids`.
      max_mention_length (`int`, *optional*, defaults to 30):
          The maximum number of tokens inside an entity span.
      entity_token_1 (`str`, *optional*, defaults to `<ent>`):
          The special token used to represent an entity span in a word token sequence. This token is only used when
          `task` is set to `"entity_classification"` or `"entity_pair_classification"`.
      entity_token_2 (`str`, *optional*, defaults to `<ent2>`):
          The special token used to represent an entity span in a word token sequence. This token is only used when
          `task` is set to `"entity_pair_classification"`.
      additional_special_tokens (`List[str]`, *optional*, defaults to `["<s>NOTUSED", "</s>NOTUSED"]`):
          Additional special tokens used by the tokenizer.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Attributes:
      sp_model (`SentencePieceProcessor`):
          The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
class_docstr_tok: ["Adapted","from","[","`","XLMRobertaTokenizer","`","]","and","[","`","LukeTokenizer","`","]",".","Based","on","[","SentencePiece","]","(","https",":","\/\/github.com\/google\/sentencepiece",")",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","entity_vocab_file","(","`","str","`",")",":","Path","to","the","entity","vocabulary","file",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sequence","token","that","was","used","during","pretraining",".","Can","be","used","a","sequence","classifier","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","beginning","of","sequence",".","The","token","used","is","the","`","cls_token","`",".","<","\/Tip",">","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","end","of","sequence","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","end","of","sequence",".","The","token","used","is","the","`","sep_token","`",".","<","\/Tip",">","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","separator","token",",","which","is","used","when","building","a","sequence","from","multiple","sequences",",","e.g",".","two","sequences","for","sequence","classification","or","for","a","text","and","a","question","for","question","answering",".","It","is","also","used","as","the","last","token","of","a","sequence","built","with","special","tokens",".","cls_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","classifier","token","which","is","used","when","doing","sequence","classification","(","classification","of","the","whole","sequence","instead","of","per-token","classification",")",".","It","is","the","first","token","of","the","sequence","when","built","with","special","tokens",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","pad",">","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths",".","mask_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","mask",">","''","`",")",":","The","token","used","for","masking","values",".","This","is","the","token","used","when","training","this","model","with","masked","language","modeling",".","This","is","the","token","which","the","model","will","try","to","predict",".","task","(","`","str","`",",","*","optional","*",")",":","Task","for","which","you","want","to","prepare","sequences",".","One","of","`","``","entity_classification","''","`",",","`","``","entity_pair_classification","''","`",",","or","`","``","entity_span_classification","''","`",".","If","you","specify","this","argument",",","the","entity","sequence","is","automatically","created","based","on","the","given","entity","span","(","s",")",".","max_entity_length","(","`","int","`",",","*","optional","*",",","defaults","to","32",")",":","The","maximum","length","of","`","entity_ids","`",".","max_mention_length","(","`","int","`",",","*","optional","*",",","defaults","to","30",")",":","The","maximum","number","of","tokens","inside","an","entity","span",".","entity_token_1","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","ent",">","`",")",":","The","special","token","used","to","represent","an","entity","span","in","a","word","token","sequence",".","This","token","is","only","used","when","`","task","`","is","set","to","`","``","entity_classification","''","`","or","`","``","entity_pair_classification","''","`",".","entity_token_2","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","ent2",">","`",")",":","The","special","token","used","to","represent","an","entity","span","in","a","word","token","sequence",".","This","token","is","only","used","when","`","task","`","is","set","to","`","``","entity_pair_classification","''","`",".","additional_special_tokens","(","`","List","[","str","]","`",",","*","optional","*",",","defaults","to","`","[","``","<","s",">","NOTUSED","''",",","``","<","\/s",">","NOTUSED","''","]","`",")",":","Additional","special","tokens","used","by","the","tokenizer",".","sp_model_kwargs","(","`","dict","`",",","*","optional","*",")",":","Will","be","passed","to","the","`","SentencePieceProcessor.__init__","(",")","`","method",".","The","[","Python","wrapper","for","SentencePiece","]","(","https",":","\/\/github.com\/google\/sentencepiece\/tree\/master\/python",")","can","be","used",",","among","other","things",",","to","set",":","-","`","enable_sampling","`",":","Enable","subword","regularization",".","-","`","nbest_size","`",":","Sampling","parameters","for","unigram",".","Invalid","for","BPE-Dropout",".","-","`","nbest_size","=","{","0,1","}","`",":","No","sampling","is","performed",".","-","`","nbest_size",">","1","`",":","samples","from","the","nbest_size","results",".","-","`","nbest_size","<","0","`",":","assuming","that","nbest_size","is","infinite","and","samples","from","the","all","hypothesis","(","lattice",")","using","forward-filtering-and-backward-sampling","algorithm",".","-","`","alpha","`",":","Smoothing","parameter","for","unigram","sampling",",","and","dropout","probability","of","merge","operations","for","BPE-dropout",".","Attributes",":","sp_model","(","`","SentencePieceProcessor","`",")",":","The","*","SentencePiece","*","processor","that","is","used","for","every","conversion","(","string",",","tokens","and","IDs",")","."]
docstr: Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length
  in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with
  `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`) .. note:: If the `encoded_inputs` passed
  are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless
  you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the
  specific device of your tensors however.
  
  Args:
      encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`):
          Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of
          tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str,
          List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader
          collate function. Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or
          TensorFlow tensors), see the note above for the return type.
      padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
           Select a strategy to pad the returned sequences (according to the model's padding side and padding
           index) among:
  
          - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
            sequence if provided).
          - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
            acceptable input length for the model if that argument is not provided.
          - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
            lengths).
      max_length (`int`, *optional*):
          Maximum length of the returned list and optionally padding length (see above).
      max_entity_length (`int`, *optional*):
          The maximum length of the entity sequence.
      pad_to_multiple_of (`int`, *optional*):
          If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
          the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
      return_attention_mask (`bool`, *optional*):
          Whether to return the attention mask. If left to the default, will return the attention mask according
          to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention
          masks?](../glossary#attention-mask)
      return_tensors (`str` or [`~utils.TensorType`], *optional*):
          If set, will return tensors instead of list of python integers. Acceptable values are:
  
          - `'tf'`: Return TensorFlow `tf.constant` objects.
          - `'pt'`: Return PyTorch `torch.Tensor` objects.
          - `'np'`: Return Numpy `np.ndarray` objects.
      verbose (`bool`, *optional*, defaults to `True`):
          Whether or not to print more information and warnings.
docstr_tok: ["Pad","a","single","encoded","input","or","a","batch","of","encoded","inputs","up","to","predefined","length","or","to","the","max","sequence","length","in","the","batch",".","Padding","side","(","left\/right",")","padding","token","ids","are","defined","at","the","tokenizer","level","(","with","`","self.padding_side","`",",","`","self.pad_token_id","`","and","`","self.pad_token_type_id","`",")","..","note",":",":","If","the","`","encoded_inputs","`","passed","are","dictionary","of","numpy","arrays",",","PyTorch","tensors","or","TensorFlow","tensors",",","the","result","will","use","the","same","type","unless","you","provide","a","different","tensor","type","with","`","return_tensors","`",".","In","the","case","of","PyTorch","tensors",",","you","will","lose","the","specific","device","of","your","tensors","however",".","Args",":","encoded_inputs","(","[","`","BatchEncoding","`","]",",","list","of","[","`","BatchEncoding","`","]",",","`","Dict","[","str",",","List","[","int","]","]","`",",","`","Dict","[","str",",","List","[","List","[","int","]","]","`","or","`","List","[","Dict","[","str",",","List","[","int","]","]","]","`",")",":","Tokenized","inputs",".","Can","represent","one","input","(","[","`","BatchEncoding","`","]","or","`","Dict","[","str",",","List","[","int","]","]","`",")","or","a","batch","of","tokenized","inputs","(","list","of","[","`","BatchEncoding","`","]",",","*","Dict","[","str",",","List","[","List","[","int","]","]","]","*","or","*","List","[","Dict","[","str",",","List","[","int","]","]","]","*",")","so","you","can","use","this","method","during","preprocessing","as","well","as","in","a","PyTorch","Dataloader","collate","function",".","Instead","of","`","List","[","int","]","`","you","can","have","tensors","(","numpy","arrays",",","PyTorch","tensors","or","TensorFlow","tensors",")",",","see","the","note","above","for","the","return","type",".","padding","(","`","bool","`",",","`","str","`","or","[","`","~utils.PaddingStrategy","`","]",",","*","optional","*",",","defaults","to","`","True","`",")",":","Select","a","strategy","to","pad","the","returned","sequences","(","according","to","the","model","'s","padding","side","and","padding","index",")","among",":","-","`","True","`","or","`","'longest","'","`",":","Pad","to","the","longest","sequence","in","the","batch","(","or","no","padding","if","only","a","single","sequence","if","provided",")",".","-","`","'max_length","'","`",":","Pad","to","a","maximum","length","specified","with","the","argument","`","max_length","`","or","to","the","maximum","acceptable","input","length","for","the","model","if","that","argument","is","not","provided",".","-","`","False","`","or","`","'do_not_pad","'","`","(","default",")",":","No","padding","(","i.e.",",","can","output","a","batch","with","sequences","of","different","lengths",")",".","max_length","(","`","int","`",",","*","optional","*",")",":","Maximum","length","of","the","returned","list","and","optionally","padding","length","(","see","above",")",".","max_entity_length","(","`","int","`",",","*","optional","*",")",":","The","maximum","length","of","the","entity","sequence",".","pad_to_multiple_of","(","`","int","`",",","*","optional","*",")",":","If","set","will","pad","the","sequence","to","a","multiple","of","the","provided","value",".","This","is","especially","useful","to","enable","the","use","of","Tensor","Cores","on","NVIDIA","hardware","with","compute","capability",">","=","7.5","(","Volta",")",".","return_attention_mask","(","`","bool","`",",","*","optional","*",")",":","Whether","to","return","the","attention","mask",".","If","left","to","the","default",",","will","return","the","attention","mask","according","to","the","specific","tokenizer","'s","default",",","defined","by","the","`","return_outputs","`","attribute",".","[","What","are","attention","masks","?","]","(","..","\/glossary","#","attention-mask",")","return_tensors","(","`","str","`","or","[","`","~utils.TensorType","`","]",",","*","optional","*",")",":","If","set",",","will","return","tensors","instead","of","list","of","python","integers",".","Acceptable","values","are",":","-","`","'tf","'","`",":","Return","TensorFlow","`","tf.constant","`","objects",".","-","`","'pt","'","`",":","Return","PyTorch","`","torch.Tensor","`","objects",".","-","`","'np","'","`",":","Return","Numpy","`","np.ndarray","`","objects",".","verbose","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","print","more","information","and","warnings","."]
code: def pad(
      self,
      encoded_inputs: Union[
          BatchEncoding,
          List[BatchEncoding],
          Dict[str, EncodedInput],
          Dict[str, List[EncodedInput]],
          List[Dict[str, EncodedInput]],
      ],
      padding: Union[bool, str, PaddingStrategy] = True,
      max_length: Optional[int] = None,
      max_entity_length: Optional[int] = None,
      pad_to_multiple_of: Optional[int] = None,
      return_attention_mask: Optional[bool] = None,
      return_tensors: Optional[Union[str, TensorType]] = None,
      verbose: bool = True,
  ) -> BatchEncoding:
      """
      Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length
      in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with
      `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`) .. note:: If the `encoded_inputs` passed
      are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless
      you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the
      specific device of your tensors however.
  
      Args:
          encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`):
              Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of
              tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str,
              List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader
              collate function. Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or
              TensorFlow tensors), see the note above for the return type.
          padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
               Select a strategy to pad the returned sequences (according to the model's padding side and padding
               index) among:
  
              - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                sequence if provided).
              - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
                acceptable input length for the model if that argument is not provided.
              - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
                lengths).
          max_length (`int`, *optional*):
              Maximum length of the returned list and optionally padding length (see above).
          max_entity_length (`int`, *optional*):
              The maximum length of the entity sequence.
          pad_to_multiple_of (`int`, *optional*):
              If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
              the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
          return_attention_mask (`bool`, *optional*):
              Whether to return the attention mask. If left to the default, will return the attention mask according
              to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention
              masks?](../glossary#attention-mask)
          return_tensors (`str` or [`~utils.TensorType`], *optional*):
              If set, will return tensors instead of list of python integers. Acceptable values are:
  
              - `'tf'`: Return TensorFlow `tf.constant` objects.
              - `'pt'`: Return PyTorch `torch.Tensor` objects.
              - `'np'`: Return Numpy `np.ndarray` objects.
          verbose (`bool`, *optional*, defaults to `True`):
              Whether or not to print more information and warnings.
      """
      # If we have a list of dicts, let's convert it in a dict of lists
      # We do this to allow using this method as a collate_fn function in PyTorch Dataloader
      if isinstance(encoded_inputs, (list, tuple)) and isinstance(
          encoded_inputs[0], Mapping
      ):
          encoded_inputs = {
              key: [example[key] for example in encoded_inputs]
              for key in encoded_inputs[0].keys()
          }
  
      # The model's main input name, usually `input_ids`, has be passed for padding
      if self.model_input_names[0] not in encoded_inputs:
          raise ValueError(
              "You should supply an encoding or a list of encodings to this method "
              f"that includes {self.model_input_names[0]}, but you provided {list(encoded_inputs.keys())}"
          )
  
      required_input = encoded_inputs[self.model_input_names[0]]
  
      if not required_input:
          if return_attention_mask:
              encoded_inputs["attention_mask"] = []
          return encoded_inputs
  
      # If we have PyTorch/TF/NumPy tensors/arrays as inputs, we cast them as python objects
      # and rebuild them afterwards if no return_tensors is specified
      # Note that we lose the specific device the tensor may be on for PyTorch
  
      first_element = required_input[0]
      if isinstance(first_element, (list, tuple)):
          # first_element might be an empty list/tuple in some edge cases so we grab the first non empty element.
          index = 0
          while len(required_input[index]) == 0:
              index += 1
          if index < len(required_input):
              first_element = required_input[index][0]
      # At this state, if `first_element` is still a list/tuple, it's an empty one so there is nothing to do.
      if not isinstance(first_element, (int, list, tuple)):
          if is_tf_tensor(first_element):
              return_tensors = (
                  "tf" if return_tensors is None else return_tensors
              )
          elif is_torch_tensor(first_element):
              return_tensors = (
                  "pt" if return_tensors is None else return_tensors
              )
          elif isinstance(first_element, np.ndarray):
              return_tensors = (
                  "np" if return_tensors is None else return_tensors
              )
          else:
              raise ValueError(
                  f"type of {first_element} unknown: {type(first_element)}. "
                  "Should be one of a python, numpy, pytorch or tensorflow object."
              )
  
          for key, value in encoded_inputs.items():
              encoded_inputs[key] = to_py_obj(value)
  
      # Convert padding_strategy in PaddingStrategy
      (
          padding_strategy,
          _,
          max_length,
          _,
      ) = self._get_padding_truncation_strategies(
          padding=padding, max_length=max_length, verbose=verbose
      )
  
      if max_entity_length is None:
          max_entity_length = self.max_entity_length
  
      required_input = encoded_inputs[self.model_input_names[0]]
      if required_input and not isinstance(
          required_input[0], (list, tuple)
      ):
          encoded_inputs = self._pad(
              encoded_inputs,
              max_length=max_length,
              max_entity_length=max_entity_length,
              padding_strategy=padding_strategy,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
          return BatchEncoding(
              encoded_inputs, tensor_type=return_tensors
          )
  
      batch_size = len(required_input)
      if any(len(v) != batch_size for v in encoded_inputs.values()):
          raise ValueError(
              "Some items in the output dictionary have a different batch size than others."
          )
  
      if padding_strategy == PaddingStrategy.LONGEST:
          max_length = max(len(inputs) for inputs in required_input)
          max_entity_length = (
              max(
                  len(inputs) for inputs in encoded_inputs["entity_ids"]
              )
              if "entity_ids" in encoded_inputs
              else 0
          )
          padding_strategy = PaddingStrategy.MAX_LENGTH
  
      batch_outputs = {}
      for i in range(batch_size):
          inputs = dict((k, v[i]) for k, v in encoded_inputs.items())
          outputs = self._pad(
              inputs,
              max_length=max_length,
              max_entity_length=max_entity_length,
              padding_strategy=padding_strategy,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
  
          for key, value in outputs.items():
              if key not in batch_outputs:
                  batch_outputs[key] = []
              batch_outputs[key].append(value)
  
      return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  
imports: import itertools
  import json
  import os
  from collections.abc import Mapping
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple, Union
  import numpy
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...tokenization_utils_base import ENCODE_KWARGS_DOCSTRING, AddedToken, BatchEncoding, EncodedInput, PaddingStrategy, TensorType, TextInput, TextInputPair, TruncationStrategy, to_py_obj
  from ...utils import add_end_docstrings, is_tf_tensor, is_torch_tensor, logging
label_desc: Use image node_id 11 for calling the MLukeTokenizer obj's underlying member method code with example usage: obj.pad(encoded_inputs, padding, max_length, max_entity_length, pad_to_multiple_of, return_attention_mask, return_tensors, verbose) and returns: BatchEncoding, encoded_inputs, BatchEncoding