class_docstr: Graphormer Layer with Dense Multi-Head Attention, as introduced
  in `Do Transformers Really Perform Bad for Graph Representation?
  <https://arxiv.org/pdf/2106.05234>`__
  
  Parameters
  ----------
  feat_size : int
      Feature size.
  hidden_size : int
      Hidden size of feedforward layers.
  num_heads : int
      Number of attention heads, by which :attr:`feat_size` is divisible.
  attn_bias_type : str, optional
      The type of attention bias used for modifying attention. Selected from
      'add' or 'mul'. Default: 'add'.
  
      * 'add' is for additive attention bias.
      * 'mul' is for multiplicative attention bias.
  norm_first : bool, optional
      If True, it performs layer normalization before attention and
      feedforward operations. Otherwise, it applies layer normalization
      afterwards. Default: False.
  dropout : float, optional
      Dropout probability. Default: 0.1.
  attn_dropout : float, optional
      Attention dropout probability. Default: 0.1.
  activation : callable activation layer, optional
      Activation function. Default: nn.ReLU().
  
  Examples
  --------
  >>> import torch as th
  >>> from dgl.nn import GraphormerLayer
  
  >>> batch_size = 16
  >>> num_nodes = 100
  >>> feat_size = 512
  >>> num_heads = 8
  >>> nfeat = th.rand(batch_size, num_nodes, feat_size)
  >>> bias = th.rand(batch_size, num_nodes, num_nodes, num_heads)
  >>> net = GraphormerLayer(
          feat_size=feat_size,
          hidden_size=2048,
          num_heads=num_heads
      )
  >>> out = net(nfeat, bias)
class_docstr_tok: ["Graphormer","Layer","with","Dense","Multi-Head","Attention",",","as","introduced","in","`","Do","Transformers","Really","Perform","Bad","for","Graph","Representation","?","<","https",":","\/\/arxiv.org\/pdf\/2106.05234",">","`","__","Parameters","--","--","--","--","--","feat_size",":","int","Feature","size",".","hidden_size",":","int","Hidden","size","of","feedforward","layers",".","num_heads",":","int","Number","of","attention","heads",",","by","which",":","attr",":","`","feat_size","`","is","divisible",".","attn_bias_type",":","str",",","optional","The","type","of","attention","bias","used","for","modifying","attention",".","Selected","from","'add","'","or","'mul","'",".","Default",":","'add","'",".","*","'add","'","is","for","additive","attention","bias",".","*","'mul","'","is","for","multiplicative","attention","bias",".","norm_first",":","bool",",","optional","If","True",",","it","performs","layer","normalization","before","attention","and","feedforward","operations",".","Otherwise",",","it","applies","layer","normalization","afterwards",".","Default",":","False",".","dropout",":","float",",","optional","Dropout","probability",".","Default",":","0.1.","attn_dropout",":","float",",","optional","Attention","dropout","probability",".","Default",":","0.1.","activation",":","callable","activation","layer",",","optional","Activation","function",".","Default",":","nn.ReLU","(",")",".","Examples","--","--","--","--",">",">",">","import","torch","as","th",">",">",">","from","dgl.nn","import","GraphormerLayer",">",">",">","batch_size","=","16",">",">",">","num_nodes","=","100",">",">",">","feat_size","=","512",">",">",">","num_heads","=","8",">",">",">","nfeat","=","th.rand","(","batch_size",",","num_nodes",",","feat_size",")",">",">",">","bias","=","th.rand","(","batch_size",",","num_nodes",",","num_nodes",",","num_heads",")",">",">",">","net","=","GraphormerLayer","(","feat_size=feat_size",",","hidden_size=2048",",","num_heads=num_heads",")",">",">",">","out","=","net","(","nfeat",",","bias",")"]
code: def __init__(
      self,
      feat_size,
      hidden_size,
      num_heads,
      attn_bias_type="add",
      norm_first=False,
      dropout=0.1,
      attn_dropout=0.1,
      activation=nn.ReLU(),
  ):
      super().__init__()
  
      self.norm_first = norm_first
  
      self.attn = BiasedMHA(
          feat_size=feat_size,
          num_heads=num_heads,
          attn_bias_type=attn_bias_type,
          attn_drop=attn_dropout,
      )
      self.ffn = nn.Sequential(
          nn.Linear(feat_size, hidden_size),
          activation,
          nn.Dropout(p=dropout),
          nn.Linear(hidden_size, feat_size),
          nn.Dropout(p=dropout),
      )
  
      self.dropout = nn.Dropout(p=dropout)
      self.attn_layer_norm = nn.LayerNorm(feat_size)
      self.ffn_layer_norm = nn.LayerNorm(feat_size)
  
imports: import torch.nn
  from .biased_mha import BiasedMHA
label_desc: Use image node_id 1 to create a new GraphormerLayer object from inherited base classes: nn with example: obj = GraphormerLayer(feat_size, hidden_size, num_heads, attn_bias_type, norm_first, dropout, attn_dropout, activation)