class_docstr: Helper class for performing BPE subword tokenization.
  
  For technical details, please refer to https://arxiv.org/abs/1508.07909.
  This class just wraps around the official subword-nmt repository.
  
  This API expects the user to call tokenize() (encode) onto the training data,
  then call finalize() to learn the encodings, and then iterate over the data
  in a second pass, calling tokenize() again to get processed output.
class_docstr_tok: ["Helper","class","for","performing","BPE","subword","tokenization",".","For","technical","details",",","please","refer","to","https",":","\/\/arxiv.org\/abs\/1508.07909",".","This","class","just","wraps","around","the","official","subword-nmt","repository",".","This","API","expects","the","user","to","call","tokenize","(",")","(","encode",")","onto","the","training","data",",","then","call","finalize","(",")","to","learn","the","encodings",",","and","then","iterate","over","the","data","in","a","second","pass",",","calling","tokenize","(",")","again","to","get","processed","output","."]
docstr: Initialize the BPE module.
  
  :param opt:
      options
  :param shared:
      shared dictionary
docstr_tok: ["Initialize","the","BPE","module",".",":","param","opt",":","options",":","param","shared",":","shared","dictionary"]
code: def __init__(self, opt: Opt, shared: TShared = None):
      """
      Initialize the BPE module.
  
      :param opt:
          options
      :param shared:
          shared dictionary
      """
      super().__init__(opt, shared)
      if not SUBWORD_BPE_INSTALLED:
          raise RuntimeError("Please run `pip install subword-nmt`")
      if not opt.get("dict_file"):
          raise RuntimeError("--dict-file is mandatory.")
  
      self.splitter = re.compile(r"\w+|[^\w\s]", re.UNICODE)
  
      self.codecs = f"{opt['dict_file']}.codecs"
      if PathManager.exists(self.codecs):
          self._load_from_codecs()
  
imports: from parlai.core.params import ParlaiParser
  from abc import ABC, abstractmethod
  from functools import lru_cache
  import json
  import os
  import random
  import re
  from typing import Dict, List, Optional, Set, Tuple
  from typing_extensions import final
  from parlai.core.build_data import download, make_dir
  from parlai.core.opt import Opt
  from parlai.utils.misc import warn_once
  from parlai.utils.typing import TShared
  from parlai.utils.io import PathManager
  import parlai.utils.logging
label_desc: Use image node_id 1 to create a new SubwordBPEHelper object from inherited base classes: BPEHelper with example: obj = SubwordBPEHelper(opt, shared)