class_docstr: Construct a LayoutLMv3 tokenizer. Based on [`RoBERTatokenizer`] (Byte Pair Encoding or BPE).
  [`LayoutLMv3Tokenizer`] can be used to turn words, word-level bounding boxes and optional word labels to
  token-level `input_ids`, `attention_mask`, `token_type_ids`, `bbox`, and optional `labels` (for token
  classification).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  [`LayoutLMv3Tokenizer`] runs end-to-end tokenization: punctuation splitting and wordpiece. It also turns the
  word-level bounding boxes into token-level bounding boxes.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      merges_file (`str`):
          Path to the merges file.
      errors (`str`, *optional*, defaults to `"replace"`):
          Paradigm to follow when decoding bytes to UTF-8. See
          [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      cls_token (`str`, *optional*, defaults to `"<s>"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      mask_token (`str`, *optional*, defaults to `"<mask>"`):
          The token used for masking values. This is the token used when training this model with masked language
          modeling. This is the token which the model will try to predict.
      add_prefix_space (`bool`, *optional*, defaults to `False`):
          Whether or not to add an initial space to the input. This allows to treat the leading word just as any
          other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
      cls_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [CLS] token.
      sep_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [SEP] token.
      pad_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [PAD] token.
      pad_token_label (`int`, *optional*, defaults to -100):
          The label to use for padding tokens. Defaults to -100, which is the `ignore_index` of PyTorch's
          CrossEntropyLoss.
      only_label_first_subword (`bool`, *optional*, defaults to `True`):
          Whether or not to only label the first subword, in case word labels are provided.
class_docstr_tok: ["Construct","a","LayoutLMv3","tokenizer",".","Based","on","[","`","RoBERTatokenizer","`","]","(","Byte","Pair","Encoding","or","BPE",")",".","[","`","LayoutLMv3Tokenizer","`","]","can","be","used","to","turn","words",",","word-level","bounding","boxes","and","optional","word","labels","to","token-level","`","input_ids","`",",","`","attention_mask","`",",","`","token_type_ids","`",",","`","bbox","`",",","and","optional","`","labels","`","(","for","token","classification",")",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","[","`","LayoutLMv3Tokenizer","`","]","runs","end-to-end","tokenization",":","punctuation","splitting","and","wordpiece",".","It","also","turns","the","word-level","bounding","boxes","into","token-level","bounding","boxes",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","merges_file","(","`","str","`",")",":","Path","to","the","merges","file",".","errors","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","replace","''","`",")",":","Paradigm","to","follow","when","decoding","bytes","to","UTF-8",".","See","[","bytes.decode","]","(","https",":","\/\/docs.python.org\/3\/library\/stdtypes.html","#","bytes.decode",")","for","more","information",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sequence","token","that","was","used","during","pretraining",".","Can","be","used","a","sequence","classifier","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","beginning","of","sequence",".","The","token","used","is","the","`","cls_token","`",".","<","\/Tip",">","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","end","of","sequence","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","end","of","sequence",".","The","token","used","is","the","`","sep_token","`",".","<","\/Tip",">","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","separator","token",",","which","is","used","when","building","a","sequence","from","multiple","sequences",",","e.g",".","two","sequences","for","sequence","classification","or","for","a","text","and","a","question","for","question","answering",".","It","is","also","used","as","the","last","token","of","a","sequence","built","with","special","tokens",".","cls_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","classifier","token","which","is","used","when","doing","sequence","classification","(","classification","of","the","whole","sequence","instead","of","per-token","classification",")",".","It","is","the","first","token","of","the","sequence","when","built","with","special","tokens",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","pad",">","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths",".","mask_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","mask",">","''","`",")",":","The","token","used","for","masking","values",".","This","is","the","token","used","when","training","this","model","with","masked","language","modeling",".","This","is","the","token","which","the","model","will","try","to","predict",".","add_prefix_space","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","an","initial","space","to","the","input",".","This","allows","to","treat","the","leading","word","just","as","any","other","word",".","(","RoBERTa","tokenizer","detect","beginning","of","words","by","the","preceding","space",")",".","cls_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","CLS","]","token",".","sep_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","SEP","]","token",".","pad_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","PAD","]","token",".","pad_token_label","(","`","int","`",",","*","optional","*",",","defaults","to","-100",")",":","The","label","to","use","for","padding","tokens",".","Defaults","to","-100",",","which","is","the","`","ignore_index","`","of","PyTorch's","CrossEntropyLoss",".","only_label_first_subword","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","only","label","the","first","subword",",","in","case","word","labels","are","provided","."]
docstr: Prepares a sequence or a pair of sequences so that it can be used by the model. It adds special tokens,
  truncates sequences if overflowing while taking into account the special tokens and manages a moving window
  (with user defined stride) for overflowing tokens. Please Note, for *text_pair* different than `None` and
  *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a
  combination of arguments will raise an error.
  
  Word-level `boxes` are turned into token-level `bbox`. If provided, word-level `word_labels` are turned into
  token-level `labels`. The word label is used for the first token of the word, while remaining tokens are
  labeled with -100, such that they will be ignored by the loss function.
  
  Args:
      text (`str`, `List[str]`, `List[List[str]]`):
          The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
      text_pair (`List[str]` or `List[int]`, *optional*):
          Optional second sequence to be encoded. This can be a list of strings (words of a single example) or a
          list of list of strings (words of a batch of examples).
docstr_tok: ["Prepares","a","sequence","or","a","pair","of","sequences","so","that","it","can","be","used","by","the","model",".","It","adds","special","tokens",",","truncates","sequences","if","overflowing","while","taking","into","account","the","special","tokens","and","manages","a","moving","window","(","with","user","defined","stride",")","for","overflowing","tokens",".","Please","Note",",","for","*","text_pair","*","different","than","`","None","`","and","*","truncation_strategy","=","longest_first","*","or","`","True","`",",","it","is","not","possible","to","return","overflowing","tokens",".","Such","a","combination","of","arguments","will","raise","an","error",".","Word-level","`","boxes","`","are","turned","into","token-level","`","bbox","`",".","If","provided",",","word-level","`","word_labels","`","are","turned","into","token-level","`","labels","`",".","The","word","label","is","used","for","the","first","token","of","the","word",",","while","remaining","tokens","are","labeled","with","-100",",","such","that","they","will","be","ignored","by","the","loss","function",".","Args",":","text","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",")",":","The","first","sequence","to","be","encoded",".","This","can","be","a","string",",","a","list","of","strings","or","a","list","of","list","of","strings",".","text_pair","(","`","List","[","str","]","`","or","`","List","[","int","]","`",",","*","optional","*",")",":","Optional","second","sequence","to","be","encoded",".","This","can","be","a","list","of","strings","(","words","of","a","single","example",")","or","a","list","of","list","of","strings","(","words","of","a","batch","of","examples",")","."]
code: def prepare_for_model(
      self,
      text: Union[TextInput, PreTokenizedInput],
      text_pair: Optional[PreTokenizedInput] = None,
      boxes: Optional[List[List[int]]] = None,
      word_labels: Optional[List[int]] = None,
      add_special_tokens: bool = True,
      padding: Union[bool, str, PaddingStrategy] = False,
      truncation: Union[bool, str, TruncationStrategy] = None,
      max_length: Optional[int] = None,
      stride: int = 0,
      pad_to_multiple_of: Optional[int] = None,
      return_tensors: Optional[Union[str, TensorType]] = None,
      return_token_type_ids: Optional[bool] = None,
      return_attention_mask: Optional[bool] = None,
      return_overflowing_tokens: bool = False,
      return_special_tokens_mask: bool = False,
      return_offsets_mapping: bool = False,
      return_length: bool = False,
      verbose: bool = True,
      prepend_batch_axis: bool = False,
      **kwargs,
  ) -> BatchEncoding:
      """
      Prepares a sequence or a pair of sequences so that it can be used by the model. It adds special tokens,
      truncates sequences if overflowing while taking into account the special tokens and manages a moving window
      (with user defined stride) for overflowing tokens. Please Note, for *text_pair* different than `None` and
      *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a
      combination of arguments will raise an error.
  
      Word-level `boxes` are turned into token-level `bbox`. If provided, word-level `word_labels` are turned into
      token-level `labels`. The word label is used for the first token of the word, while remaining tokens are
      labeled with -100, such that they will be ignored by the loss function.
  
      Args:
          text (`str`, `List[str]`, `List[List[str]]`):
              The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
          text_pair (`List[str]` or `List[int]`, *optional*):
              Optional second sequence to be encoded. This can be a list of strings (words of a single example) or a
              list of list of strings (words of a batch of examples).
      """
  
      # Backward compatibility for 'truncation_strategy', 'pad_to_max_length'
      (
          padding_strategy,
          truncation_strategy,
          max_length,
          kwargs,
      ) = self._get_padding_truncation_strategies(
          padding=padding,
          truncation=truncation,
          max_length=max_length,
          pad_to_multiple_of=pad_to_multiple_of,
          verbose=verbose,
          **kwargs,
      )
  
      tokens = []
      pair_tokens = []
      token_boxes = []
      pair_token_boxes = []
      labels = []
  
      if text_pair is None:
          if word_labels is None:
              # CASE 1: document image classification (training + inference) + CASE 2: token classification (inference)
              for word, box in zip(text, boxes):
                  if len(word) < 1:  # skip empty words
                      continue
                  word_tokens = self.tokenize(word)
                  tokens.extend(word_tokens)
                  token_boxes.extend([box] * len(word_tokens))
          else:
              # CASE 2: token classification (training)
              for word, box, label in zip(text, boxes, word_labels):
                  if len(word) < 1:  # skip empty words
                      continue
                  word_tokens = self.tokenize(word)
                  tokens.extend(word_tokens)
                  token_boxes.extend([box] * len(word_tokens))
                  if self.only_label_first_subword:
                      # Use the real label id for the first token of the word, and padding ids for the remaining tokens
                      labels.extend(
                          [label]
                          + [self.pad_token_label]
                          * (len(word_tokens) - 1)
                      )
                  else:
                      labels.extend([label] * len(word_tokens))
      else:
          # CASE 3: document visual question answering (inference)
          # text = question
          # text_pair = words
          tokens = self.tokenize(text)
          token_boxes = [self.pad_token_box for _ in range(len(tokens))]
  
          for word, box in zip(text_pair, boxes):
              if len(word) < 1:  # skip empty words
                  continue
              word_tokens = self.tokenize(word)
              pair_tokens.extend(word_tokens)
              pair_token_boxes.extend([box] * len(word_tokens))
  
      # Create ids + pair_ids
      ids = self.convert_tokens_to_ids(tokens)
      pair_ids = (
          self.convert_tokens_to_ids(pair_tokens)
          if pair_tokens
          else None
      )
  
      if (
          return_overflowing_tokens
          and truncation_strategy == TruncationStrategy.LONGEST_FIRST
          and pair_ids is not None
      ):
          raise ValueError(
              "Not possible to return overflowing tokens for pair of sequences with the "
              "`longest_first`. Please select another truncation strategy than `longest_first`, "
              "for instance `only_second` or `only_first`."
          )
  
      # Compute the total size of the returned encodings
      pair = bool(pair_ids is not None)
      len_ids = len(ids)
      len_pair_ids = len(pair_ids) if pair else 0
      total_len = (
          len_ids
          + len_pair_ids
          + (
              self.num_special_tokens_to_add(pair=pair)
              if add_special_tokens
              else 0
          )
      )
  
      # Truncation: Handle max sequence length
      overflowing_tokens = []
      overflowing_token_boxes = []
      overflowing_labels = []
      if (
          truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE
          and max_length
          and total_len > max_length
      ):
          (
              ids,
              token_boxes,
              pair_ids,
              pair_token_boxes,
              labels,
              overflowing_tokens,
              overflowing_token_boxes,
              overflowing_labels,
          ) = self.truncate_sequences(
              ids,
              token_boxes,
              pair_ids=pair_ids,
              pair_token_boxes=pair_token_boxes,
              labels=labels,
              num_tokens_to_remove=total_len - max_length,
              truncation_strategy=truncation_strategy,
              stride=stride,
          )
  
      if return_token_type_ids and not add_special_tokens:
          raise ValueError(
              "Asking to return token_type_ids while setting add_special_tokens to False "
              "results in an undefined behavior. Please set add_special_tokens to True or "
              "set return_token_type_ids to None."
          )
  
      # Load from model defaults
      if return_token_type_ids is None:
          return_token_type_ids = (
              "token_type_ids" in self.model_input_names
          )
      if return_attention_mask is None:
          return_attention_mask = (
              "attention_mask" in self.model_input_names
          )
  
      encoded_inputs = {}
  
      if return_overflowing_tokens:
          encoded_inputs["overflowing_tokens"] = overflowing_tokens
          encoded_inputs[
              "overflowing_token_boxes"
          ] = overflowing_token_boxes
          encoded_inputs["overflowing_labels"] = overflowing_labels
          encoded_inputs["num_truncated_tokens"] = (
              total_len - max_length
          )
  
      # Add special tokens
      if add_special_tokens:
          sequence = self.build_inputs_with_special_tokens(
              ids, pair_ids
          )
          token_type_ids = self.create_token_type_ids_from_sequences(
              ids, pair_ids
          )
          token_boxes = (
              [self.cls_token_box] + token_boxes + [self.sep_token_box]
          )
          if pair_token_boxes:
              pair_token_boxes = (
                  [self.sep_token_box]
                  + pair_token_boxes
                  + [self.sep_token_box]
              )
          token_boxes = (
              token_boxes + pair_token_boxes if pair else token_boxes
          )
          if labels:
              labels = (
                  [self.pad_token_label]
                  + labels
                  + [self.pad_token_label]
              )
      else:
          sequence = ids + pair_ids if pair else ids
          token_type_ids = [0] * len(ids) + (
              [0] * len(pair_ids) if pair else []
          )
          token_boxes = (
              token_boxes + pair_token_boxes if pair else token_boxes
          )
  
      # Build output dictionary
      encoded_inputs["input_ids"] = sequence
      encoded_inputs["bbox"] = token_boxes
      if return_token_type_ids:
          encoded_inputs["token_type_ids"] = token_type_ids
      if return_special_tokens_mask:
          if add_special_tokens:
              encoded_inputs[
                  "special_tokens_mask"
              ] = self.get_special_tokens_mask(ids, pair_ids)
          else:
              encoded_inputs["special_tokens_mask"] = [0] * len(
                  sequence
              )
  
      if labels:
          encoded_inputs["labels"] = labels
  
      # Check lengths
      self._eventual_warn_about_too_long_sequence(
          encoded_inputs["input_ids"], max_length, verbose
      )
  
      # Padding
      if (
          padding_strategy != PaddingStrategy.DO_NOT_PAD
          or return_attention_mask
      ):
          encoded_inputs = self.pad(
              encoded_inputs,
              max_length=max_length,
              padding=padding_strategy.value,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
  
      if return_length:
          encoded_inputs["length"] = len(encoded_inputs["input_ids"])
  
      batch_outputs = BatchEncoding(
          encoded_inputs,
          tensor_type=return_tensors,
          prepend_batch_axis=prepend_batch_axis,
      )
  
      return batch_outputs
  
imports: import json
  import os
  from functools import lru_cache
  from typing import Dict, List, Optional, Tuple, Union
  import regex
  from ...tokenization_utils import AddedToken, PreTrainedTokenizer
  from ...tokenization_utils_base import BatchEncoding, EncodedInput, PreTokenizedInput, TextInput, TextInputPair, TruncationStrategy
  from ...utils import PaddingStrategy, TensorType, add_end_docstrings, logging
label_desc: Use image node_id 21 for calling the LayoutLMv3Tokenizer obj's underlying member method code with example usage: obj.prepare_for_model(text, text_pair, boxes, word_labels, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis) and returns: batch_outputs