class_docstr: Configuration class to store `BigBird` sparsity configuration.
  For more details about this sparsity config, please see `Big Bird: Transformers for
  Longer Sequences`: https://arxiv.org/pdf/2007.14062.pdf
  This class extends parent class of `SparsityConfig` and customizes it for `BigBird` sparsity.
class_docstr_tok: ["Configuration","class","to","store","`","BigBird","`","sparsity","configuration",".","For","more","details","about","this","sparsity","config",",","please","see","`","Big","Bird",":","Transformers","for","Longer","Sequences","`",":","https",":","\/\/arxiv.org\/pdf\/2007.14062.pdf","This","class","extends","parent","class","of","`","SparsityConfig","`","and","customizes","it","for","`","BigBird","`","sparsity","."]
docstr: Initialize the BigBird Sparsity Pattern Config.
  For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial
  Arguments:
       num_heads: required: an integer determining number of attention heads of the layer.
       block_size: optional: an integer determining the block size. Current implementation of
          sparse self-attention is based on blocked sparse matrices. In which this parameter
          defines size of such blocks, `Block X Block`.
       different_layout_per_head: optional: a boolean determining if each head should be assigned
          a different sparsity layout; default is false and this will be satisfied based on
          availability.
       num_random_blocks: optional: an integer determining the number of random blocks in each
          block row.
       num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding
          local attention window.
       num_global_blocks: optional: an integer determining how many consecutive blocks, starting
          from index 0, are considered as global attention. Global block tokens will be attended
          by all other block tokens and will attend to all other block tokens as well.
       attention: optional: a string determining attention type. Attention can be `unidirectional`,
          such as autoregressive models, in which tokens attend only to tokens appear before them
          in the context. Considering that, the upper triangular of attention matrix is empty as
          above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to
          any other tokens before or after them. Then, the upper triangular part of the attention
          matrix is mirror of the lower triangular in the above figure.
docstr_tok: ["Initialize","the","BigBird","Sparsity","Pattern","Config",".","For","usage","example","please","see",",","TODO","DeepSpeed","Sparse","Transformer","Tutorial","Arguments",":","num_heads",":","required",":","an","integer","determining","number","of","attention","heads","of","the","layer",".","block_size",":","optional",":","an","integer","determining","the","block","size",".","Current","implementation","of","sparse","self-attention","is","based","on","blocked","sparse","matrices",".","In","which","this","parameter","defines","size","of","such","blocks",",","`","Block","X","Block","`",".","different_layout_per_head",":","optional",":","a","boolean","determining","if","each","head","should","be","assigned","a","different","sparsity","layout",";","default","is","false","and","this","will","be","satisfied","based","on","availability",".","num_random_blocks",":","optional",":","an","integer","determining","the","number","of","random","blocks","in","each","block","row",".","num_sliding_window_blocks",":","optional",":","an","integer","determining","the","number","of","blocks","in","sliding","local","attention","window",".","num_global_blocks",":","optional",":","an","integer","determining","how","many","consecutive","blocks",",","starting","from","index","0",",","are","considered","as","global","attention",".","Global","block","tokens","will","be","attended","by","all","other","block","tokens","and","will","attend","to","all","other","block","tokens","as","well",".","attention",":","optional",":","a","string","determining","attention","type",".","Attention","can","be","`","unidirectional","`",",","such","as","autoregressive","models",",","in","which","tokens","attend","only","to","tokens","appear","before","them","in","the","context",".","Considering","that",",","the","upper","triangular","of","attention","matrix","is","empty","as","above","figure",".","Or","it","can","be","`","bidirectional","`",",","such","as","BERT",",","in","which","tokens","can","attend","to","any","other","tokens","before","or","after","them",".","Then",",","the","upper","triangular","part","of","the","attention","matrix","is","mirror","of","the","lower","triangular","in","the","above","figure","."]
code: def __init__(
      self,
      num_heads,
      block_size=16,
      different_layout_per_head=False,
      num_random_blocks=1,
      num_sliding_window_blocks=3,
      num_global_blocks=1,
      attention="bidirectional",
  ):
      """Initialize the BigBird Sparsity Pattern Config.
      For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial
      Arguments:
           num_heads: required: an integer determining number of attention heads of the layer.
           block_size: optional: an integer determining the block size. Current implementation of
              sparse self-attention is based on blocked sparse matrices. In which this parameter
              defines size of such blocks, `Block X Block`.
           different_layout_per_head: optional: a boolean determining if each head should be assigned
              a different sparsity layout; default is false and this will be satisfied based on
              availability.
           num_random_blocks: optional: an integer determining the number of random blocks in each
              block row.
           num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding
              local attention window.
           num_global_blocks: optional: an integer determining how many consecutive blocks, starting
              from index 0, are considered as global attention. Global block tokens will be attended
              by all other block tokens and will attend to all other block tokens as well.
           attention: optional: a string determining attention type. Attention can be `unidirectional`,
              such as autoregressive models, in which tokens attend only to tokens appear before them
              in the context. Considering that, the upper triangular of attention matrix is empty as
              above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to
              any other tokens before or after them. Then, the upper triangular part of the attention
              matrix is mirror of the lower triangular in the above figure.
      """
  
      super().__init__(num_heads, block_size, different_layout_per_head)
  
      self.num_random_blocks = num_random_blocks
      self.num_sliding_window_blocks = num_sliding_window_blocks
      self.num_global_blocks = num_global_blocks
  
      if attention != "unidirectional" and attention != "bidirectional":
          raise NotImplementedError(
              'only "uni/bi-directional" attentions are supported for now!'
          )
      self.attention = attention
  
imports: import random
  import torch
label_desc: Use image node_id 1 to create a new BigBirdSparsityConfig object from inherited base classes: SparsityConfig with example: obj = BigBirdSparsityConfig(num_heads, block_size, different_layout_per_head, num_random_blocks, num_sliding_window_blocks, num_global_blocks, attention)