class_docstr: BytePS trainer. Supports both multi-GPU and distributed training.
  It achieves better scalability than horovod in distributed training, if the model is communication
  intensive and you have properly set up the machines following its
  `best practices <https://github.com/bytedance/byteps/blob/master/docs/best-practice.md>`_
  which requires a few extra bandwidth servers than horovod.
  
  To use it, switch the trainer, and refer to BytePS documentation on how to
  launch server/scheduler/workers.
  
  Attributes:
      hvd (module): the byteps module that contains horovod-compatible APIs
          like `rank(),size()`.
          This attribute exists so that downstream code that uses these APIs
          does not need to worry about which library is being used under the hood.
class_docstr_tok: ["BytePS","trainer",".","Supports","both","multi-GPU","and","distributed","training",".","It","achieves","better","scalability","than","horovod","in","distributed","training",",","if","the","model","is","communication","intensive","and","you","have","properly","set","up","the","machines","following","its","`","best","practices","<","https",":","\/\/github.com\/bytedance\/byteps\/blob\/master\/docs\/best-practice.md",">","`","_","which","requires","a","few","extra","bandwidth","servers","than","horovod",".","To","use","it",",","switch","the","trainer",",","and","refer","to","BytePS","documentation","on","how","to","launch","server\/scheduler\/workers",".","Attributes",":","hvd","(","module",")",":","the","byteps","module","that","contains","horovod-compatible","APIs","like","`","rank","(",")",",","size","(",")","`",".","This","attribute","exists","so","that","downstream","code","that","uses","these","APIs","does","not","need","to","worry","about","which","library","is","being","used","under","the","hood","."]
docstr: Args:
      average (bool): whether to average or sum the gradients across processes.
docstr_tok: ["Args",":","average","(","bool",")",":","whether","to","average","or","sum","the","gradients","across","processes","."]
code: def __init__(self, average=True):
      """
      Args:
          average (bool): whether to average or sum the gradients across processes.
      """
      import byteps.tensorflow as bps
  
      self.hvd = bps  # BytePS has the same interface as Horovod
      self.hvd.allreduce = (
          bps.push_pull
      )  # https://github.com/bytedance/byteps/issues/8
      assert os.environ.get("DMLC_ROLE", None) == "worker"
      assert (
          "DMLC_WORKER_ID" in os.environ
          and "DMLC_NUM_WORKER" in os.environ
      )
      bps.init()
      self.is_chief = bps.rank() == 0
  
      self._local_rank = bps.local_rank()
      self._rank = bps.rank()
      self._average = average
  
      self._compression = None
      self._has_compression = False
      logger.info(
          "[BytePSTrainer] local rank={}".format(self._local_rank)
      )
      SingleCostTrainer.__init__(self)
  
imports: import multiprocessing
  import os
  import sys
  import tensorflow
  from tensorpack.compat import tfv1
  from ..callbacks import CallbackFactory, RunOp
  from ..graph_builder.distributed import DistributedParameterServerBuilder, DistributedReplicatedBuilder
  from ..graph_builder.training import AsyncMultiGPUBuilder, SyncMultiGPUParameterServerBuilder, SyncMultiGPUReplicatedBuilder
  from ..graph_builder.utils import override_to_local_variable
  from ..input_source import FeedfreeInput, QueueInput
  from ..tfutils import get_global_step_var
  from ..tfutils.common import get_tf_version_tuple
  from ..tfutils.distributed import get_distributed_session_creator
  from ..tfutils.sesscreate import NewSessionCreator
  from ..tfutils.tower import TrainTowerContext
  from ..utils import logger
  from ..utils.argtools import map_arg
  from ..utils.develop import HIDE_DOC, deprecated
  from .tower import SingleCostTrainer
label_desc: Use image node_id 1 to create a new BytePSTrainer object from inherited base classes: HorovodTrainer with example: obj = BytePSTrainer(average)