class_docstr: A distributed version of SparseDispatcher.
  
  Instead of one batch of input examples, we simultaneously process
  a list of num_datashards batches of input examples.  The per-expert
  `Tensor`s contain a combination of examples from the different datashards.
  
  Each datashard is associated with a particular device and each expert is
  associated with a particular device.  All per-datashard and per-expert
  `Tensor`s are created on those devices.  There is no single-device bottleneck.
class_docstr_tok: ["A","distributed","version","of","SparseDispatcher",".","Instead","of","one","batch","of","input","examples",",","we","simultaneously","process","a","list","of","num_datashards","batches","of","input","examples",".","The","per-expert","`","Tensor","`","s","contain","a","combination","of","examples","from","the","different","datashards",".","Each","datashard","is","associated","with","a","particular","device","and","each","expert","is","associated","with","a","particular","device",".","All","per-datashard","and","per-expert","`","Tensor","`","s","are","created","on","those","devices",".","There","is","no","single-device","bottleneck","."]
docstr: Create a DistributedSparseDispatcher.
  
  Args:
    data_parallelism: a Parallelism object.
    expert_parallelism: a Parallelism object.
    gates: a list of datashard_parallelism.n `Tensor`s of shapes
      `[batch_size[d], num_experts]`.
  
  Returns:
    a DistributedSparseDispatcher
docstr_tok: ["Create","a","DistributedSparseDispatcher",".","Args",":","data_parallelism",":","a","Parallelism","object",".","expert_parallelism",":","a","Parallelism","object",".","gates",":","a","list","of","datashard_parallelism.n","`","Tensor","`","s","of","shapes","`","[","batch_size","[","d","]",",","num_experts","]","`",".","Returns",":","a","DistributedSparseDispatcher"]
code: def __init__(self, data_parallelism, expert_parallelism, gates):
      """Create a DistributedSparseDispatcher.
  
      Args:
        data_parallelism: a Parallelism object.
        expert_parallelism: a Parallelism object.
        gates: a list of datashard_parallelism.n `Tensor`s of shapes
          `[batch_size[d], num_experts]`.
  
      Returns:
        a DistributedSparseDispatcher
      """
      self._gates = gates
      self._dp = data_parallelism
      self._ep = expert_parallelism
      assert len(gates) == self._dp.n
      self._dispatchers = self._dp(SparseDispatcher, self._ep.n, gates)
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import functools
  import math
  import six
  from six.moves import range
  from six.moves import zip
  from tensor2tensor.layers import common_layers
  from tensor2tensor.layers.vq_discrete import DiscreteBottleneck
  import tensorflow.compat.v1
label_desc: Use image node_id 1 to create a new DistributedSparseDispatcher object from inherited base classes: object with example: obj = DistributedSparseDispatcher(data_parallelism, expert_parallelism, gates)