class_docstr: Base `TransformerLayer` for vision transformer.
  
  It can be built from `mmcv.ConfigDict` and support more flexible
  customization, for example, using any number of `FFN or LN ` and
  use different kinds of `attention` by specifying a list of `ConfigDict`
  named `attn_cfgs`. It is worth mentioning that it supports `prenorm`
  when you specifying `norm` as the first element of `operation_order`.
  More details about the `prenorm`: `On Layer Normalization in the
  Transformer Architecture <https://arxiv.org/abs/2002.04745>`_ .
  
  Args:
      attn_cfgs (list[`mmcv.ConfigDict`] | obj:`mmcv.ConfigDict` | None )):
          Configs for `self_attention` or `cross_attention` modules,
          The order of the configs in the list should be consistent with
          corresponding attentions in operation_order.
          If it is a dict, all of the attention modules in operation_order
          will be built with this config. Default: None.
      ffn_cfgs (list[`mmcv.ConfigDict`] | obj:`mmcv.ConfigDict` | None )):
          Configs for FFN, The order of the configs in the list should be
          consistent with corresponding ffn in operation_order.
          If it is a dict, all of the attention modules in operation_order
          will be built with this config.
      operation_order (tuple[str]): The execution order of operation
          in transformer. Such as ('self_attn', 'norm', 'ffn', 'norm').
          Support `prenorm` when you specifying first element as `norm`.
          Default：None.
      norm_cfg (dict): Config dict for normalization layer.
          Default: dict(type='LN').
      init_cfg (obj:`mmcv.ConfigDict`): The Config for initialization.
          Default: None.
      batch_first (bool): Key, Query and Value are shape
          of (batch, n, embed_dim)
          or (n, batch, embed_dim). Default to False.
class_docstr_tok: ["Base","`","TransformerLayer","`","for","vision","transformer",".","It","can","be","built","from","`","mmcv.ConfigDict","`","and","support","more","flexible","customization",",","for","example",",","using","any","number","of","`","FFN","or","LN","`","and","use","different","kinds","of","`","attention","`","by","specifying","a","list","of","`","ConfigDict","`","named","`","attn_cfgs","`",".","It","is","worth","mentioning","that","it","supports","`","prenorm","`","when","you","specifying","`","norm","`","as","the","first","element","of","`","operation_order","`",".","More","details","about","the","`","prenorm","`",":","`","On","Layer","Normalization","in","the","Transformer","Architecture","<","https",":","\/\/arxiv.org\/abs\/2002.04745",">","`","_",".","Args",":","attn_cfgs","(","list","[","`","mmcv.ConfigDict","`","]","|","obj",":","`","mmcv.ConfigDict","`","|","None",")",")",":","Configs","for","`","self_attention","`","or","`","cross_attention","`","modules",",","The","order","of","the","configs","in","the","list","should","be","consistent","with","corresponding","attentions","in","operation_order",".","If","it","is","a","dict",",","all","of","the","attention","modules","in","operation_order","will","be","built","with","this","config",".","Default",":","None",".","ffn_cfgs","(","list","[","`","mmcv.ConfigDict","`","]","|","obj",":","`","mmcv.ConfigDict","`","|","None",")",")",":","Configs","for","FFN",",","The","order","of","the","configs","in","the","list","should","be","consistent","with","corresponding","ffn","in","operation_order",".","If","it","is","a","dict",",","all","of","the","attention","modules","in","operation_order","will","be","built","with","this","config",".","operation_order","(","tuple","[","str","]",")",":","The","execution","order","of","operation","in","transformer",".","Such","as","(","'self_attn","'",",","'norm","'",",","'ffn","'",",","'norm","'",")",".","Support","`","prenorm","`","when","you","specifying","first","element","as","`","norm","`",".","Default\uff1aNone",".","norm_cfg","(","dict",")",":","Config","dict","for","normalization","layer",".","Default",":","dict","(","type='LN","'",")",".","init_cfg","(","obj",":","`","mmcv.ConfigDict","`",")",":","The","Config","for","initialization",".","Default",":","None",".","batch_first","(","bool",")",":","Key",",","Query","and","Value","are","shape","of","(","batch",",","n",",","embed_dim",")","or","(","n",",","batch",",","embed_dim",")",".","Default","to","False","."]
docstr: Forward function for `TransformerDecoderLayer`.
  
  **kwargs contains some specific arguments of attentions.
  
  Args:
      query (Tensor): The input query with shape
          [num_queries, bs, embed_dims] if
          self.batch_first is False, else
          [bs, num_queries embed_dims].
      key (Tensor): The key tensor with shape [num_keys, bs,
          embed_dims] if self.batch_first is False, else
          [bs, num_keys, embed_dims] .
      value (Tensor): The value tensor with same shape as `key`.
      query_pos (Tensor): The positional encoding for `query`.
          Default: None.
      key_pos (Tensor): The positional encoding for `key`.
          Default: None.
      attn_masks (List[Tensor] | None): 2D Tensor used in
          calculation of corresponding attention. The length of
          it should equal to the number of `attention` in
          `operation_order`. Default: None.
      query_key_padding_mask (Tensor): ByteTensor for `query`, with
          shape [bs, num_queries]. Only used in `self_attn` layer.
          Defaults to None.
      key_padding_mask (Tensor): ByteTensor for `query`, with
          shape [bs, num_keys]. Default: None.
  
  Returns:
      Tensor: forwarded results with shape [num_queries, bs, embed_dims].
docstr_tok: ["Forward","function","for","`","TransformerDecoderLayer","`",".","*","*","kwargs","contains","some","specific","arguments","of","attentions",".","Args",":","query","(","Tensor",")",":","The","input","query","with","shape","[","num_queries",",","bs",",","embed_dims","]","if","self.batch_first","is","False",",","else","[","bs",",","num_queries","embed_dims","]",".","key","(","Tensor",")",":","The","key","tensor","with","shape","[","num_keys",",","bs",",","embed_dims","]","if","self.batch_first","is","False",",","else","[","bs",",","num_keys",",","embed_dims","]",".","value","(","Tensor",")",":","The","value","tensor","with","same","shape","as","`","key","`",".","query_pos","(","Tensor",")",":","The","positional","encoding","for","`","query","`",".","Default",":","None",".","key_pos","(","Tensor",")",":","The","positional","encoding","for","`","key","`",".","Default",":","None",".","attn_masks","(","List","[","Tensor","]","|","None",")",":","2D","Tensor","used","in","calculation","of","corresponding","attention",".","The","length","of","it","should","equal","to","the","number","of","`","attention","`","in","`","operation_order","`",".","Default",":","None",".","query_key_padding_mask","(","Tensor",")",":","ByteTensor","for","`","query","`",",","with","shape","[","bs",",","num_queries","]",".","Only","used","in","`","self_attn","`","layer",".","Defaults","to","None",".","key_padding_mask","(","Tensor",")",":","ByteTensor","for","`","query","`",",","with","shape","[","bs",",","num_keys","]",".","Default",":","None",".","Returns",":","Tensor",":","forwarded","results","with","shape","[","num_queries",",","bs",",","embed_dims","]","."]
code: def forward(
      self,
      query,
      key=None,
      value=None,
      query_pos=None,
      key_pos=None,
      attn_masks=None,
      query_key_padding_mask=None,
      key_padding_mask=None,
      **kwargs,
  ):
      """Forward function for `TransformerDecoderLayer`.
  
      **kwargs contains some specific arguments of attentions.
  
      Args:
          query (Tensor): The input query with shape
              [num_queries, bs, embed_dims] if
              self.batch_first is False, else
              [bs, num_queries embed_dims].
          key (Tensor): The key tensor with shape [num_keys, bs,
              embed_dims] if self.batch_first is False, else
              [bs, num_keys, embed_dims] .
          value (Tensor): The value tensor with same shape as `key`.
          query_pos (Tensor): The positional encoding for `query`.
              Default: None.
          key_pos (Tensor): The positional encoding for `key`.
              Default: None.
          attn_masks (List[Tensor] | None): 2D Tensor used in
              calculation of corresponding attention. The length of
              it should equal to the number of `attention` in
              `operation_order`. Default: None.
          query_key_padding_mask (Tensor): ByteTensor for `query`, with
              shape [bs, num_queries]. Only used in `self_attn` layer.
              Defaults to None.
          key_padding_mask (Tensor): ByteTensor for `query`, with
              shape [bs, num_keys]. Default: None.
  
      Returns:
          Tensor: forwarded results with shape [num_queries, bs, embed_dims].
      """
  
      norm_index = 0
      attn_index = 0
      ffn_index = 0
      identity = query
      if attn_masks is None:
          attn_masks = [None for _ in range(self.num_attn)]
      elif isinstance(attn_masks, torch.Tensor):
          attn_masks = [
              copy.deepcopy(attn_masks) for _ in range(self.num_attn)
          ]
          warnings.warn(
              f"Use same attn_mask in all attentions in "
              f"{self.__class__.__name__} "
          )
      else:
          assert len(attn_masks) == self.num_attn, (
              f"The length of "
              f"attn_masks {len(attn_masks)} must be equal "
              f"to the number of attention in "
              f"operation_order {self.num_attn}"
          )
  
      for layer in self.operation_order:
          if layer == "self_attn":
              temp_key = temp_value = query
              query = self.attentions[attn_index](
                  query,
                  temp_key,
                  temp_value,
                  identity if self.pre_norm else None,
                  query_pos=query_pos,
                  key_pos=query_pos,
                  attn_mask=attn_masks[attn_index],
                  key_padding_mask=query_key_padding_mask,
                  **kwargs,
              )
              attn_index += 1
              identity = query
  
          elif layer == "norm":
              query = self.norms[norm_index](query)
              norm_index += 1
  
          elif layer == "cross_attn":
              query = self.attentions[attn_index](
                  query,
                  key,
                  value,
                  identity if self.pre_norm else None,
                  query_pos=query_pos,
                  key_pos=key_pos,
                  attn_mask=attn_masks[attn_index],
                  key_padding_mask=key_padding_mask,
                  **kwargs,
              )
              attn_index += 1
              identity = query
  
          elif layer == "ffn":
              query = self.ffns[ffn_index](
                  query, identity if self.pre_norm else None
              )
              ffn_index += 1
  
      return query
  
imports: import copy
  import warnings
  import torch
  import torch.nn
  from annotator.uniformer.mmcv import ConfigDict, deprecated_api_warning
  from annotator.uniformer.mmcv.cnn import Linear, build_activation_layer, build_norm_layer
  from annotator.uniformer.mmcv.runner.base_module import BaseModule, ModuleList, Sequential
  from annotator.uniformer.mmcv.utils import build_from_cfg
  from .drop import build_dropout
  from .registry import ATTENTION, FEEDFORWARD_NETWORK, POSITIONAL_ENCODING, TRANSFORMER_LAYER, TRANSFORMER_LAYER_SEQUENCE
label_desc: Use image node_id 2 for calling the BaseTransformerLayer obj's underlying member method code with example usage: obj.forward(query, key, value, query_pos, key_pos, attn_masks, query_key_padding_mask, key_padding_mask) and returns: query