class_docstr: Helper class for performing BPE subword tokenization.
  
  For technical details, please refer to https://arxiv.org/abs/1508.07909.
  This class just wraps around the official subword-nmt repository.
  
  This API expects the user to call tokenize() (encode) onto the training data,
  then call finalize() to learn the encodings, and then iterate over the data
  in a second pass, calling tokenize() again to get processed output.
class_docstr_tok: ["Helper","class","for","performing","BPE","subword","tokenization",".","For","technical","details",",","please","refer","to","https",":","\/\/arxiv.org\/abs\/1508.07909",".","This","class","just","wraps","around","the","official","subword-nmt","repository",".","This","API","expects","the","user","to","call","tokenize","(",")","(","encode",")","onto","the","training","data",",","then","call","finalize","(",")","to","learn","the","encodings",",","and","then","iterate","over","the","data","in","a","second","pass",",","calling","tokenize","(",")","again","to","get","processed","output","."]
docstr: Return whether tokens should be sorted for this particular helper.
  
  We want to sort with SubwordBPEHelper.
docstr_tok: ["Return","whether","tokens","should","be","sorted","for","this","particular","helper",".","We","want","to","sort","with","SubwordBPEHelper","."]
code: def should_sort(self) -> bool:
      """
      Return whether tokens should be sorted for this particular helper.
  
      We want to sort with SubwordBPEHelper.
      """
      return True
  
imports: from parlai.core.params import ParlaiParser
  from abc import ABC, abstractmethod
  from functools import lru_cache
  import json
  import os
  import random
  import re
  from typing import Dict, List, Optional, Set, Tuple
  from typing_extensions import final
  from parlai.core.build_data import download, make_dir
  from parlai.core.opt import Opt
  from parlai.utils.misc import warn_once
  from parlai.utils.typing import TShared
  from parlai.utils.io import PathManager
  import parlai.utils.logging
label_desc: Use image node_id 9 for calling the SubwordBPEHelper obj's underlying member method code with example usage: obj.should_sort() and returns: True