class_docstr: This module computes self-attention (or cross-attention), similar to the architecture in BERT.
  Details in the paper:
  [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al, 2019]
  (https://api.semanticscholar.org/CorpusID:52967399)
  
  Additionally, it has the following functionality:
  
  * the attention scoring function can be specified.
  * it can be used in encoders as well as decoders.
  * `position_bias` can be used, which makes it suitable for
  [T5-style attention](https://api.semanticscholar.org/CorpusID:204838007) as well.
  
  # Parameters
  
  hidden_size: `int` (default = `512`)
      The size of the expected input tensor.
  attention_head_size: `int` (default = `64`)
      The size of a single attention head.
  num_attention_heads: `int` (default = `8`)
      The number of attention heads.
  scoring_func: `str` (default = `scaled_dot_product`)
      The name of the attention-calculating function to be used.
      Eg. `additive`, `linear`, etc. For a complete list, please check
      :mod:`allennlp.modules.matrix_attention.matrix_attention`.
  output_linear: `bool` (default = `False`)
      Whether to add an additional output linear layer at the end.
  dropout: `float` (default = `0.0`)
      The dropout probability.
  bias: `bool` (default = `True`)
      Whether to include bias weights in query, key, value (and output) linear layers.
  normalize_weights: `bool` (default = `False`)
      Whether to normalize the initial weights.
  is_decoder: `bool` (default = `False`)
      Whether this module is being used in a decoder stack or not.
  is_cross_attention: `bool` (default = `False`)
      Whether this module is being used for cross-attention in a decoder stack or not.
      If `is_cross_attention` is `True`, then `is_decoder` must also be `True`.
  relative_attention_num_buckets: `int`,  optional (default = `None`)
      The number of buckets to use in relative attention; if `None`, relative attention
      will not be applied.
class_docstr_tok: ["This","module","computes","self-attention","(","or","cross-attention",")",",","similar","to","the","architecture","in","BERT",".","Details","in","the","paper",":","[","BERT",":","Pre-training","of","Deep","Bidirectional","Transformers","for","Language","Understanding",",","Devlin","et","al",",","2019","]","(","https",":","\/\/api.semanticscholar.org\/CorpusID:52967399",")","Additionally",",","it","has","the","following","functionality",":","*","the","attention","scoring","function","can","be","specified",".","*","it","can","be","used","in","encoders","as","well","as","decoders",".","*","`","position_bias","`","can","be","used",",","which","makes","it","suitable","for","[","T5-style","attention","]","(","https",":","\/\/api.semanticscholar.org\/CorpusID:204838007",")","as","well",".","#","Parameters","hidden_size",":","`","int","`","(","default","=","`","512","`",")","The","size","of","the","expected","input","tensor",".","attention_head_size",":","`","int","`","(","default","=","`","64","`",")","The","size","of","a","single","attention","head",".","num_attention_heads",":","`","int","`","(","default","=","`","8","`",")","The","number","of","attention","heads",".","scoring_func",":","`","str","`","(","default","=","`","scaled_dot_product","`",")","The","name","of","the","attention-calculating","function","to","be","used",".","Eg",".","`","additive","`",",","`","linear","`",",","etc",".","For","a","complete","list",",","please","check",":","mod",":","`","allennlp.modules.matrix_attention.matrix_attention","`",".","output_linear",":","`","bool","`","(","default","=","`","False","`",")","Whether","to","add","an","additional","output","linear","layer","at","the","end",".","dropout",":","`","float","`","(","default","=","`","0.0","`",")","The","dropout","probability",".","bias",":","`","bool","`","(","default","=","`","True","`",")","Whether","to","include","bias","weights","in","query",",","key",",","value","(","and","output",")","linear","layers",".","normalize_weights",":","`","bool","`","(","default","=","`","False","`",")","Whether","to","normalize","the","initial","weights",".","is_decoder",":","`","bool","`","(","default","=","`","False","`",")","Whether","this","module","is","being","used","in","a","decoder","stack","or","not",".","is_cross_attention",":","`","bool","`","(","default","=","`","False","`",")","Whether","this","module","is","being","used","for","cross-attention","in","a","decoder","stack","or","not",".","If","`","is_cross_attention","`","is","`","True","`",",","then","`","is_decoder","`","must","also","be","`","True","`",".","relative_attention_num_buckets",":","`","int","`",",","optional","(","default","=","`","None","`",")","The","number","of","buckets","to","use","in","relative","attention",";","if","`","None","`",",","relative","attention","will","not","be","applied","."]
code: def _transpose_for_scores(self, x: torch.Tensor) -> torch.Tensor:
      new_x_shape = x.size()[:-1] + (
          self.num_attention_heads,
          self.attention_head_size,
      )
      x = x.view(*new_x_shape)
      return x.permute(0, 2, 1, 3)
  
imports: import math
  from typing import Optional, Tuple, TYPE_CHECKING
  from dataclasses import dataclass
  import torch
  import torch.nn.functional
  from allennlp.common import FromParams
  from allennlp.common.checks import ConfigurationError
  from allennlp.modules.matrix_attention.matrix_attention import MatrixAttention
  from allennlp.modules.transformer.transformer_module import TransformerModule
  from allennlp.modules.transformer.util import apply_mask, FloatT, IntT, BoolT
label_desc: Use image node_id 3 for calling the AttentionModule obj's underlying member method code with example usage: obj._transpose_for_scores(x) and returns: x