class_docstr: A subclass of MXNet gluon.Trainer.
  
  There are two differences between DistributedTrainer and Trainer:
  1. DistributedTrainer calculates gradients using BytePS push pull
     API while Trainer does it using kvstore push/pull APIs;
  2. DistributedTrainer performs push_pull(summation) and average,
     while Trainer only performs push_pull(summation).
  
  Parameters
  ----------
  params : ParameterDict
      The set of parameters to optimize.
  optimizer : str or Optimizer
      The optimizer to use. See
      `help <http://mxnet.io/api/python/optimization/optimization.html#the-mxnet-optimizer-package>`_
      on Optimizer for a list of available optimizers.
  optimizer_params : dict
      Key-word arguments to be passed to optimizer constructor. For example,
      `{'learning_rate': 0.1}`. All optimizers accept learning_rate, wd (weight decay),
      clip_gradient, and lr_scheduler. See each optimizer's
      constructor for a list of additional supported arguments
  root_rank : int
      rank of root
  compression_params : dict
      Key-word arguments to be passed to gradient compression constructor. For example, 
      `{'compressor': 'onebit', 'ef': 'vanilla', 'momentum': 'nesterov', 'scaling': true}`.
      All compressor accept 'compressor', 'ef'. See each compressor's constructor for a list 
      of additional supported arguments
class_docstr_tok: ["A","subclass","of","MXNet","gluon.Trainer",".","There","are","two","differences","between","DistributedTrainer","and","Trainer",":","1",".","DistributedTrainer","calculates","gradients","using","BytePS","push","pull","API","while","Trainer","does","it","using","kvstore","push\/pull","APIs",";","2",".","DistributedTrainer","performs","push_pull","(","summation",")","and","average",",","while","Trainer","only","performs","push_pull","(","summation",")",".","Parameters","--","--","--","--","--","params",":","ParameterDict","The","set","of","parameters","to","optimize",".","optimizer",":","str","or","Optimizer","The","optimizer","to","use",".","See","`","help","<","http",":","\/\/mxnet.io\/api\/python\/optimization\/optimization.html","#","the-mxnet-optimizer-package",">","`","_","on","Optimizer","for","a","list","of","available","optimizers",".","optimizer_params",":","dict","Key-word","arguments","to","be","passed","to","optimizer","constructor",".","For","example",",","`","{","'learning_rate","'",":","0.1","}","`",".","All","optimizers","accept","learning_rate",",","wd","(","weight","decay",")",",","clip_gradient",",","and","lr_scheduler",".","See","each","optimizer's","constructor","for","a","list","of","additional","supported","arguments","root_rank",":","int","rank","of","root","compression_params",":","dict","Key-word","arguments","to","be","passed","to","gradient","compression","constructor",".","For","example",",","`","{","'compressor","'",":","'onebit","'",",","'ef","'",":","'vanilla","'",",","'momentum","'",":","'nesterov","'",",","'scaling","'",":","true","}","`",".","All","compressor","accept","'compressor","'",",","'ef","'",".","See","each","compressor","'s","constructor","for","a","list","of","additional","supported","arguments"]
code: def _allreduce_grads(self):
      # update lr
      if local_rank() == 0:
          self._f.seek(0)
          ba = struct.pack("d", self.learning_rate)
          self._f.write(ba)
          self._f.flush()
  
      for i, param in enumerate(self._params):
          if param.grad_req != "null":
              # normalized with batch_size and num_workers
              nd._internal._mul_scalar(
                  param._grad[0],
                  1.0 / self._scale / self._bps_size,
                  out=param._grad[0],
              )
              compressed, ctx = self._intra_compressors[
                  param.name
              ].compress(param._grad[0])
              byteps_push_pull(
                  compressed,
                  is_average=False,
                  name="gradient_" + str(i),
                  priority=-i,
              )
              param._grad[0][:] = self._intra_compressors[
                  param.name
              ].decompress(compressed, ctx, x=param._data[0])
  
imports: from __future__ import absolute_import, division, print_function
  import copy
  import os
  import struct
  import warnings
  import mxnet
  import mxnet.ndarray
  from byteps.mxnet.compression import Compression
  from byteps.mxnet.ops import byteps_declare_tensor, byteps_push_pull, init, local_rank, local_size, rank, resume, shutdown, size, suspend
label_desc: Use image node_id 5 for calling the DistributedTrainer obj's underlying member method code with example usage: obj._allreduce_grads() without return types