class_docstr: Helper class used in transducer models with stateless decoders. This stateless
  simply outputs embedding or concatenated embeddings for the input label[s],
  depending on the configured context size.
  
  Args:
      context_size: history context size for the stateless decoder network. Could be any positive integer. We recommend setting this as 2.
      vocab_size: total vocabulary size.
      emb_dim: total embedding size of the stateless net output.
      blank_idx: index for the blank symbol for the transducer model.
      normalization_mode: normalization run on the output embeddings. Could be either 'layer' or None. We recommend using 'layer' to stabilize training.
      dropout: dropout rate on the embedding outputs.
class_docstr_tok: ["Helper","class","used","in","transducer","models","with","stateless","decoders",".","This","stateless","simply","outputs","embedding","or","concatenated","embeddings","for","the","input","label","[","s","]",",","depending","on","the","configured","context","size",".","Args",":","context_size",":","history","context","size","for","the","stateless","decoder","network",".","Could","be","any","positive","integer",".","We","recommend","setting","this","as","2.","vocab_size",":","total","vocabulary","size",".","emb_dim",":","total","embedding","size","of","the","stateless","net","output",".","blank_idx",":","index","for","the","blank","symbol","for","the","transducer","model",".","normalization_mode",":","normalization","run","on","the","output","embeddings",".","Could","be","either","'layer","'","or","None",".","We","recommend","using","'layer","'","to","stabilize","training",".","dropout",":","dropout","rate","on","the","embedding","outputs","."]
code: def __init__(
      self,
      context_size,
      vocab_size,
      emb_dim,
      blank_idx,
      normalization_mode,
      dropout,
  ):
      super().__init__()
      assert context_size > 0
      self.context_size = context_size
      self.vocab_size = vocab_size
      self.emb_dim = emb_dim
      self.dropout = torch.nn.Dropout(dropout)
      self.norm = torch.nn.Identity()
      if normalization_mode == "layer":
          self.norm = torch.nn.LayerNorm(
              emb_dim, elementwise_affine=False
          )
  
      embeds = []
      for i in range(self.context_size):
          # We use different embedding matrices for different context positions.
          # In this list, a smaller index means more recent history word.
          # We assign more dimensions for the most recent word in the history.
          # The detailed method is, we first allocate half the embedding-size
          # to the most recent history word, and then allocate the remaining
          # dimensions evenly among all history contexts. E.g. if total embedding
          # size is 200, and context_size is 2, then we allocate 150 dimensions
          # to the last word, and 50 dimensions to the second-to-last word.
          if i != 0:
              embed_size = emb_dim // 2 // self.context_size
          else:
              embed_size = emb_dim - (
                  emb_dim // 2 // self.context_size
              ) * (self.context_size - 1)
  
          embed = torch.nn.Embedding(
              vocab_size + 1, embed_size, padding_idx=blank_idx
          )
          embeds.append(embed)
  
      self.embeds = torch.nn.ModuleList(embeds)
      self.blank_idx = blank_idx
  
imports: from typing import Any, Dict, List, Optional, Tuple, Union
  import torch
label_desc: Use image node_id 1 to create a new StatelessNet object from inherited base classes: torch.nn with example: obj = StatelessNet(context_size, vocab_size, emb_dim, blank_idx, normalization_mode, dropout)