class_docstr: CategoricalEncoder subclass which offers specific methods for encoding text and handle
  special tokens for training of sequence to sequence models.
  In detail, aside special <unk> token already present in CategoricalEncoder
  for handling out-of-vocab tokens here special methods to handle
  <bos> beginning of sequence and <eos> tokens are defined.
  
  Note: update_from_iterable and update_from_didataset here have as default
  sequence_input=True because it is assumed that this encoder is used on
  iterables of strings: e.g.
  
  >>> from speechbrain.dataio.encoder import TextEncoder
  >>> dataset = [["encode", "this", "textencoder"], ["foo", "bar"]]
  >>> encoder = TextEncoder()
  >>> encoder.update_from_iterable(dataset)
  >>> encoder.expect_len(5)
  >>> encoder.encode_label("this")
  1
  >>> encoder.add_unk()
  5
  >>> encoder.expect_len(6)
  >>> encoder.encode_sequence(["this", "out-of-vocab"])
  [1, 5]
  >>>
  
  Two methods can be used to add <bos> and <eos> to the internal dicts:
  insert_bos_eos, add_bos_eos.
  
  >>> encoder.add_bos_eos()
  >>> encoder.expect_len(8)
  >>> encoder.lab2ind[encoder.eos_label]
  7
  >>>
  add_bos_eos adds the special tokens at the end of the dict indexes
  >>> encoder = TextEncoder()
  >>> encoder.update_from_iterable(dataset)
  >>> encoder.insert_bos_eos(bos_index=0, eos_index=1)
  >>> encoder.expect_len(7)
  >>> encoder.lab2ind[encoder.eos_label]
  1
  >>>
  insert_bos_eos allows to specify whose index will correspond to each of them.
  Note that you can also specify the same integer encoding for both.
  
  Four methods can be used to prepend <bos> and append <eos>.
  prepend_bos_label and append_eos_label add respectively the <bos> and <eos>
  string tokens to the input sequence
  
  >>> words = ["foo", "bar"]
  >>> encoder.prepend_bos_label(words)
  ['<bos>', 'foo', 'bar']
  >>> encoder.append_eos_label(words)
  ['foo', 'bar', '<eos>']
  
  prepend_bos_index and append_eos_index add respectively the <bos> and <eos>
  indexes to the input encoded sequence.
  
  >>> words = ["foo", "bar"]
  >>> encoded = encoder.encode_sequence(words)
  >>> encoder.prepend_bos_index(encoded)
  [0, 3, 4]
  >>> encoder.append_eos_index(encoded)
  [3, 4, 1]
class_docstr_tok: ["CategoricalEncoder","subclass","which","offers","specific","methods","for","encoding","text","and","handle","special","tokens","for","training","of","sequence","to","sequence","models",".","In","detail",",","aside","special","<","unk",">","token","already","present","in","CategoricalEncoder","for","handling","out-of-vocab","tokens","here","special","methods","to","handle","<","bos",">","beginning","of","sequence","and","<","eos",">","tokens","are","defined",".","Note",":","update_from_iterable","and","update_from_didataset","here","have","as","default","sequence_input=True","because","it","is","assumed","that","this","encoder","is","used","on","iterables","of","strings",":","e.g",".",">",">",">","from","speechbrain.dataio.encoder","import","TextEncoder",">",">",">","dataset","=","[","[","``","encode","''",",","``","this","''",",","``","textencoder","''","]",",","[","``","foo","''",",","``","bar","''","]","]",">",">",">","encoder","=","TextEncoder","(",")",">",">",">","encoder.update_from_iterable","(","dataset",")",">",">",">","encoder.expect_len","(","5",")",">",">",">","encoder.encode_label","(","``","this","''",")","1",">",">",">","encoder.add_unk","(",")","5",">",">",">","encoder.expect_len","(","6",")",">",">",">","encoder.encode_sequence","(","[","``","this","''",",","``","out-of-vocab","''","]",")","[","1",",","5","]",">",">",">","Two","methods","can","be","used","to","add","<","bos",">","and","<","eos",">","to","the","internal","dicts",":","insert_bos_eos",",","add_bos_eos",".",">",">",">","encoder.add_bos_eos","(",")",">",">",">","encoder.expect_len","(","8",")",">",">",">","encoder.lab2ind","[","encoder.eos_label","]","7",">",">",">","add_bos_eos","adds","the","special","tokens","at","the","end","of","the","dict","indexes",">",">",">","encoder","=","TextEncoder","(",")",">",">",">","encoder.update_from_iterable","(","dataset",")",">",">",">","encoder.insert_bos_eos","(","bos_index=0",",","eos_index=1",")",">",">",">","encoder.expect_len","(","7",")",">",">",">","encoder.lab2ind","[","encoder.eos_label","]","1",">",">",">","insert_bos_eos","allows","to","specify","whose","index","will","correspond","to","each","of","them",".","Note","that","you","can","also","specify","the","same","integer","encoding","for","both",".","Four","methods","can","be","used","to","prepend","<","bos",">","and","append","<","eos",">",".","prepend_bos_label","and","append_eos_label","add","respectively","the","<","bos",">","and","<","eos",">","string","tokens","to","the","input","sequence",">",">",">","words","=","[","``","foo","''",",","``","bar","''","]",">",">",">","encoder.prepend_bos_label","(","words",")","[","'","<","bos",">","'",",","'foo","'",",","'bar","'","]",">",">",">","encoder.append_eos_label","(","words",")","[","'foo","'",",","'bar","'",",","'","<","eos",">","'","]","prepend_bos_index","and","append_eos_index","add","respectively","the","<","bos",">","and","<","eos",">","indexes","to","the","input","encoded","sequence",".",">",">",">","words","=","[","``","foo","''",",","``","bar","''","]",">",">",">","encoded","=","encoder.encode_sequence","(","words",")",">",">",">","encoder.prepend_bos_index","(","encoded",")","[","0",",","3",",","4","]",">",">",">","encoder.append_eos_index","(","encoded",")","[","3",",","4",",","1","]"]
docstr: Insert sentence boundary markers in the label set.
  
  If the beginning-of-sentence and end-of-sentence markers
  are the same, will just use one sentence-boundary label.
  
  Arguments
  ---------
  bos_label : hashable
      Beginning-of-sentence label, any label
  eos_label : hashable
      End-of-sentence label, any label. If set to the same label as
      bos_label, will just use one sentence-boundary label.
  bos_index : int
      Where to insert bos_label. eos_index = bos_index + 1
  bos_index : optional, int
      Where to insert eos_label. Default: eos_index = bos_index + 1
docstr_tok: ["Insert","sentence","boundary","markers","in","the","label","set",".","If","the","beginning-of-sentence","and","end-of-sentence","markers","are","the","same",",","will","just","use","one","sentence-boundary","label",".","Arguments","--","--","--","--","-","bos_label",":","hashable","Beginning-of-sentence","label",",","any","label","eos_label",":","hashable","End-of-sentence","label",",","any","label",".","If","set","to","the","same","label","as","bos_label",",","will","just","use","one","sentence-boundary","label",".","bos_index",":","int","Where","to","insert","bos_label",".","eos_index","=","bos_index","+","1","bos_index",":","optional",",","int","Where","to","insert","eos_label",".","Default",":","eos_index","=","bos_index","+","1"]
code: def insert_bos_eos(
      self,
      bos_label=DEFAULT_BOS,
      eos_label=DEFAULT_EOS,
      bos_index=0,
      eos_index=None,
  ):
      """Insert sentence boundary markers in the label set.
  
      If the beginning-of-sentence and end-of-sentence markers
      are the same, will just use one sentence-boundary label.
  
      Arguments
      ---------
      bos_label : hashable
          Beginning-of-sentence label, any label
      eos_label : hashable
          End-of-sentence label, any label. If set to the same label as
          bos_label, will just use one sentence-boundary label.
      bos_index : int
          Where to insert bos_label. eos_index = bos_index + 1
      bos_index : optional, int
          Where to insert eos_label. Default: eos_index = bos_index + 1
      """
      if bos_label == eos_label:
          logger.debug(
              "BOS and EOS labels are the same so using just one sentence "
              "boundary label"
          )
          self.insert_label(bos_label, bos_index)
      else:
          self.insert_label(bos_label, bos_index)
          if eos_index is None:
              logger.debug(
                  "EOS label not specified, using BOS label + 1"
              )
              self.insert_label(eos_label, bos_index + 1)
          else:
              self.insert_label(eos_label, eos_index)
      self.bos_label = bos_label
      self.eos_label = eos_label
  
imports: import ast
  import torch
  import collections
  import itertools
  import logging
  import warnings
  import speechbrain
  from speechbrain.utils.checkpoints import mark_as_saver, mark_as_loader, register_checkpoint_hooks
label_desc: Use image node_id 6 for calling the TextEncoder obj's underlying member method code with example usage: obj.insert_bos_eos(bos_label, eos_label, bos_index, eos_index) without return types