class_docstr: Configuration class to store `Fixed` sparsity configuration.
  For more details about this sparsity config, please see `Generative Modeling with
  Sparse Transformers`: https://arxiv.org/abs/1904.10509; this has been customized.
  This class extends parent class of `SparsityConfig` and customizes it for `Fixed` sparsity.
class_docstr_tok: ["Configuration","class","to","store","`","Fixed","`","sparsity","configuration",".","For","more","details","about","this","sparsity","config",",","please","see","`","Generative","Modeling","with","Sparse","Transformers","`",":","https",":","\/\/arxiv.org\/abs\/1904.10509",";","this","has","been","customized",".","This","class","extends","parent","class","of","`","SparsityConfig","`","and","customizes","it","for","`","Fixed","`","sparsity","."]
docstr: Sets global attention layout used by the given head in the sparse attention.
  Currently we set global blocks starting from the last block of a local window to the first one.
  That means if a local window consists of 4 blocks and global attention size is one block, we use
  block #4 in each local window as global. If we have different layout per head, then other heads
  will get #3, #2, and #1. And if we have more heads (and different layout has set) than num of global
  attentions, multiple head may have same global attentions.
  Note) if horizontal_global_attention is set, global blocks will be set both horizontally and
  vertically.
  Arguments:
       h: required: an integer determining head index
       layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing
          sparsity layout of all head; may not be completely set at this step
  Return:
       layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity
          layout of all head in which global layout is set
docstr_tok: ["Sets","global","attention","layout","used","by","the","given","head","in","the","sparse","attention",".","Currently","we","set","global","blocks","starting","from","the","last","block","of","a","local","window","to","the","first","one",".","That","means","if","a","local","window","consists","of","4","blocks","and","global","attention","size","is","one","block",",","we","use","block","#","4","in","each","local","window","as","global",".","If","we","have","different","layout","per","head",",","then","other","heads","will","get","#","3",",","#","2",",","and","#","1",".","And","if","we","have","more","heads","(","and","different","layout","has","set",")","than","num","of","global","attentions",",","multiple","head","may","have","same","global","attentions",".","Note",")","if","horizontal_global_attention","is","set",",","global","blocks","will","be","set","both","horizontally","and","vertically",".","Arguments",":","h",":","required",":","an","integer","determining","head","index","layout",":","required",":","a","tensor","of","dimension","(","num_heads",",","num_blocks",",","num_blocks",")","containing","sparsity","layout","of","all","head",";","may","not","be","completely","set","at","this","step","Return",":","layout",":","a","tensor","of","dimension","(","num_heads",",","num_blocks",",","num_blocks",")","containing","sparsity","layout","of","all","head","in","which","global","layout","is","set"]
code: def set_global_layout(self, h, layout):
      """Sets global attention layout used by the given head in the sparse attention.
      Currently we set global blocks starting from the last block of a local window to the first one.
      That means if a local window consists of 4 blocks and global attention size is one block, we use
      block #4 in each local window as global. If we have different layout per head, then other heads
      will get #3, #2, and #1. And if we have more heads (and different layout has set) than num of global
      attentions, multiple head may have same global attentions.
      Note) if horizontal_global_attention is set, global blocks will be set both horizontally and
      vertically.
      Arguments:
           h: required: an integer determining head index
           layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing
              sparsity layout of all head; may not be completely set at this step
      Return:
           layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity
              layout of all head in which global layout is set
      """
  
      num_blocks = layout.shape[1]
      first_global_block_idx = (
          self.num_local_blocks
          - (1 + h % self.num_different_global_patterns)
          * self.num_global_blocks
      )
  
      # set all global blocks except the last one if (in last local window)
      end = num_blocks - (num_blocks % self.num_local_blocks)
      for i in range(
          first_global_block_idx, end, self.num_local_blocks
      ):
          # vertical global attention
          first_row = 0 if self.attention == "bidirectional" else i
          # (((i // self.num_local_blocks) + 1) * self.num_local_blocks)
          # if (first_row < num_blocks):
          layout[h, first_row:, i : i + self.num_global_blocks] = 1
  
          # horizontal global attention; only in bidirectional attention
          if self.horizontal_global_attention:
              layout[h, i : i + self.num_global_blocks, :] = 1
  
      # set last global blocks; handle possible short last local window
      if end < num_blocks:
          start = min(
              end + first_global_block_idx,
              num_blocks - self.num_global_blocks,
          )
          end = start + self.num_global_blocks
  
          # vertical global attention
          first_row = 0 if self.attention == "bidirectional" else start
          # (((start // self.num_local_blocks) + 1) * self.num_local_blocks)
          # if (first_row < num_blocks):
          layout[h, first_row:, start:end] = 1
  
          # horizontal global attention
          if self.horizontal_global_attention:
              layout[h, start:end, :] = 1
      return layout
  
imports: import random
  import torch
label_desc: Use image node_id 3 for calling the FixedSparsityConfig obj's underlying member method code with example usage: obj.set_global_layout(h, layout) and returns: layout