class_docstr: This module computes self-attention (or cross-attention), similar to the architecture in BERT.
  Details in the paper:
  [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al, 2019]
  (https://api.semanticscholar.org/CorpusID:52967399)
  
  Additionally, it has the following functionality:
  
  * the attention scoring function can be specified.
  * it can be used in encoders as well as decoders.
  * `position_bias` can be used, which makes it suitable for
  [T5-style attention](https://api.semanticscholar.org/CorpusID:204838007) as well.
  
  # Parameters
  
  hidden_size: `int` (default = `512`)
      The size of the expected input tensor.
  attention_head_size: `int` (default = `64`)
      The size of a single attention head.
  num_attention_heads: `int` (default = `8`)
      The number of attention heads.
  scoring_func: `str` (default = `scaled_dot_product`)
      The name of the attention-calculating function to be used.
      Eg. `additive`, `linear`, etc. For a complete list, please check
      :mod:`allennlp.modules.matrix_attention.matrix_attention`.
  output_linear: `bool` (default = `False`)
      Whether to add an additional output linear layer at the end.
  dropout: `float` (default = `0.0`)
      The dropout probability.
  bias: `bool` (default = `True`)
      Whether to include bias weights in query, key, value (and output) linear layers.
  normalize_weights: `bool` (default = `False`)
      Whether to normalize the initial weights.
  is_decoder: `bool` (default = `False`)
      Whether this module is being used in a decoder stack or not.
  is_cross_attention: `bool` (default = `False`)
      Whether this module is being used for cross-attention in a decoder stack or not.
      If `is_cross_attention` is `True`, then `is_decoder` must also be `True`.
  relative_attention_num_buckets: `int`,  optional (default = `None`)
      The number of buckets to use in relative attention; if `None`, relative attention
      will not be applied.
class_docstr_tok: ["This","module","computes","self-attention","(","or","cross-attention",")",",","similar","to","the","architecture","in","BERT",".","Details","in","the","paper",":","[","BERT",":","Pre-training","of","Deep","Bidirectional","Transformers","for","Language","Understanding",",","Devlin","et","al",",","2019","]","(","https",":","\/\/api.semanticscholar.org\/CorpusID:52967399",")","Additionally",",","it","has","the","following","functionality",":","*","the","attention","scoring","function","can","be","specified",".","*","it","can","be","used","in","encoders","as","well","as","decoders",".","*","`","position_bias","`","can","be","used",",","which","makes","it","suitable","for","[","T5-style","attention","]","(","https",":","\/\/api.semanticscholar.org\/CorpusID:204838007",")","as","well",".","#","Parameters","hidden_size",":","`","int","`","(","default","=","`","512","`",")","The","size","of","the","expected","input","tensor",".","attention_head_size",":","`","int","`","(","default","=","`","64","`",")","The","size","of","a","single","attention","head",".","num_attention_heads",":","`","int","`","(","default","=","`","8","`",")","The","number","of","attention","heads",".","scoring_func",":","`","str","`","(","default","=","`","scaled_dot_product","`",")","The","name","of","the","attention-calculating","function","to","be","used",".","Eg",".","`","additive","`",",","`","linear","`",",","etc",".","For","a","complete","list",",","please","check",":","mod",":","`","allennlp.modules.matrix_attention.matrix_attention","`",".","output_linear",":","`","bool","`","(","default","=","`","False","`",")","Whether","to","add","an","additional","output","linear","layer","at","the","end",".","dropout",":","`","float","`","(","default","=","`","0.0","`",")","The","dropout","probability",".","bias",":","`","bool","`","(","default","=","`","True","`",")","Whether","to","include","bias","weights","in","query",",","key",",","value","(","and","output",")","linear","layers",".","normalize_weights",":","`","bool","`","(","default","=","`","False","`",")","Whether","to","normalize","the","initial","weights",".","is_decoder",":","`","bool","`","(","default","=","`","False","`",")","Whether","this","module","is","being","used","in","a","decoder","stack","or","not",".","is_cross_attention",":","`","bool","`","(","default","=","`","False","`",")","Whether","this","module","is","being","used","for","cross-attention","in","a","decoder","stack","or","not",".","If","`","is_cross_attention","`","is","`","True","`",",","then","`","is_decoder","`","must","also","be","`","True","`",".","relative_attention_num_buckets",":","`","int","`",",","optional","(","default","=","`","None","`",")","The","number","of","buckets","to","use","in","relative","attention",";","if","`","None","`",",","relative","attention","will","not","be","applied","."]
docstr: Adapted from Mesh Tensorflow:
  https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593
  
  Translate relative position to a bucket number for relative attention. The relative position is defined as
  memory_position - query_position, i.e. the distance in tokens from the attending position to the
  attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller
  buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All
  relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the
  same bucket. This should allow for more graceful generalization to longer sequences than the model has been
  trained on.
  
  Args:
      relative_position: an int32 Tensor
      bidirectional: a boolean - whether the attention is bidirectional
      num_buckets: an integer
      max_distance: an integer
  
  Returns:
      a Tensor with the same shape as relative_position, containing int32 values in the range
      [0, num_buckets)
docstr_tok: ["Adapted","from","Mesh","Tensorflow",":","https",":","\/\/github.com\/tensorflow\/mesh\/blob\/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d\/mesh_tensorflow\/transformer\/transformer_layers.py","#","L593","Translate","relative","position","to","a","bucket","number","for","relative","attention",".","The","relative","position","is","defined","as","memory_position","-","query_position",",","i.e",".","the","distance","in","tokens","from","the","attending","position","to","the","attended-to","position",".","If","bidirectional=False",",","then","positive","relative","positions","are","invalid",".","We","use","smaller","buckets","for","small","absolute","relative_position","and","larger","buckets","for","larger","absolute","relative_positions",".","All","relative","positions",">","=max_distance","map","to","the","same","bucket",".","All","relative","positions","<","=-max_distance","map","to","the","same","bucket",".","This","should","allow","for","more","graceful","generalization","to","longer","sequences","than","the","model","has","been","trained","on",".","Args",":","relative_position",":","an","int32","Tensor","bidirectional",":","a","boolean","-","whether","the","attention","is","bidirectional","num_buckets",":","an","integer","max_distance",":","an","integer","Returns",":","a","Tensor","with","the","same","shape","as","relative_position",",","containing","int32","values","in","the","range","[","0",",","num_buckets",")"]
code: def _relative_position_bucket(
      relative_position: IntT,
      bidirectional: bool = True,
      num_buckets: int = 32,
      max_distance: int = 128,
  ) -> IntT:
      """
      Adapted from Mesh Tensorflow:
      https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593
  
      Translate relative position to a bucket number for relative attention. The relative position is defined as
      memory_position - query_position, i.e. the distance in tokens from the attending position to the
      attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller
      buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All
      relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the
      same bucket. This should allow for more graceful generalization to longer sequences than the model has been
      trained on.
  
      Args:
          relative_position: an int32 Tensor
          bidirectional: a boolean - whether the attention is bidirectional
          num_buckets: an integer
          max_distance: an integer
  
      Returns:
          a Tensor with the same shape as relative_position, containing int32 values in the range
          [0, num_buckets)
      """
      relative_buckets = relative_position.new_zeros(
          relative_position.shape
      )
      if bidirectional:
          num_buckets //= 2
          relative_buckets += (relative_position > 0).to(
              torch.long
          ) * num_buckets
          relative_position = torch.abs(relative_position)
      else:
          relative_position = -torch.min(
              relative_position, torch.zeros_like(relative_position)
          )
      # now relative_position is in the range [0, inf)
  
      # half of the buckets are for exact increments in positions
      max_exact = num_buckets // 2
      is_small = relative_position < max_exact
  
      # The other half of the buckets are for logarithmically bigger bins in positions up to max_distance
      relative_postion_if_large = max_exact + (
          torch.log(relative_position.float() / max_exact)
          / math.log(max_distance / max_exact)
          * (num_buckets - max_exact)
      ).to(torch.long)
      relative_postion_if_large = torch.min(
          relative_postion_if_large,
          torch.full_like(relative_postion_if_large, num_buckets - 1),
      )
  
      relative_buckets += torch.where(
          is_small, relative_position, relative_postion_if_large
      )
      return relative_buckets
  
imports: import math
  from typing import Optional, Tuple, TYPE_CHECKING
  from dataclasses import dataclass
  import torch
  import torch.nn.functional
  from allennlp.common import FromParams
  from allennlp.common.checks import ConfigurationError
  from allennlp.modules.matrix_attention.matrix_attention import MatrixAttention
  from allennlp.modules.transformer.transformer_module import TransformerModule
  from allennlp.modules.transformer.util import apply_mask, FloatT, IntT, BoolT
label_desc: Use image node_id 11 for calling the AttentionModule obj's underlying member method code with example usage: obj._relative_position_bucket(relative_position, bidirectional, num_buckets, max_distance) and returns: relative_buckets