class_docstr: Horovod trainer, support both multi-GPU and distributed training.
  
  To use for multi-GPU training:
  
  .. code-block:: bash
  
      # First, change trainer to HorovodTrainer(), then
      CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_DEBUG=INFO horovodrun -np 4 --output-filename mylog python train.py
  
  To use for distributed training:
  
  .. code-block:: bash
  
      # First, change trainer to HorovodTrainer(), then
      horovodrun -np 8 -H server1:4,server2:4 --output-filename mylog \
          python train.py
  
  Note:
      1. To reach the maximum speed in your system, there are many options to tune
         in Horovod installation, horovodrun arguments, and in the MPI command line.
         See Horovod docs for details.
  
      2. Due to a TF bug (#8136), you must not initialize CUDA context before the trainer starts training.
         Therefore TF functions like `is_gpu_available()` or `list_local_devices()`
         must be avoided.
         You can, however, use `tf.config.experimental.list_physical_devices('GPU')`, introduced in TF 1.14.
  
      3. Horovod supports both MPI and gloo. There are a few drawbacks of the MPI backend:
  
          + MPI does not like `fork()`. If your code (e.g. dataflow) contains multiprocessing, it may cause problems.
          + MPI sometimes fails to kill all processes in the end. Be sure to check it afterwards.
  
         The gloo backend is recommended though it may come with very minor slow down.
         To use gloo backend, see
         `horovod documentation <https://github.com/horovod/horovod#running-horovod>`_ for more details.
  
      4. Keep in mind that there is one process running the script per GPU, therefore:
  
         + Make sure your InputSource has reasonable randomness.
  
         + If your data processing is heavy, doing it in a single dedicated process might be
           a better choice than doing them repeatedly in each process.
  
         + You need to make sure log directories in each process won't conflict.
           You can set it only for the chief process, or set a different one for each process.
  
         + Callbacks have an option to be run only in the chief process, or in all processes.
           See :meth:`Callback.set_chief_only()`. Most callbacks have a reasonable
           default already, but certain callbacks may need your customization.
           Report an issue if you find any bad defaults.
  
         + You can use Horovod API such as `hvd.rank()` to know which process you are and choose
           different code path. Chief process has rank 0.
  
      5. Due to these caveats, see
         `ResNet-Horovod <https://github.com/tensorpack/benchmarks/tree/master/ResNet-Horovod>`_
         for a full example which has handled these common issues.
         This example can train ImageNet in roughly an hour following the paper's setup.
  
  Attributes:
      BROADCAST_EVERY_EPOCH (bool):
          Whether to broadcast the variables every epoch.
          Theoretically this is a no-op (because the variables
          are supposed to be in-sync).
          But this cheap operation may help prevent certain numerical issues in practice.
  
          Note that in cases such as BatchNorm, the variables may not be in sync:
          e.g., non-master worker may not maintain EMAs.
  
          For benchmark, disable this option.
class_docstr_tok: ["Horovod","trainer",",","support","both","multi-GPU","and","distributed","training",".","To","use","for","multi-GPU","training",":","..","code-block",":",":","bash","#","First",",","change","trainer","to","HorovodTrainer","(",")",",","then","CUDA_VISIBLE_DEVICES=0,1,2,3","NCCL_DEBUG=INFO","horovodrun","-np","4","--","output-filename","mylog","python","train.py","To","use","for","distributed","training",":","..","code-block",":",":","bash","#","First",",","change","trainer","to","HorovodTrainer","(",")",",","then","horovodrun","-np","8","-H","server1:4",",","server2:4","--","output-filename","mylog","\\","python","train.py","Note",":","1",".","To","reach","the","maximum","speed","in","your","system",",","there","are","many","options","to","tune","in","Horovod","installation",",","horovodrun","arguments",",","and","in","the","MPI","command","line",".","See","Horovod","docs","for","details",".","2",".","Due","to","a","TF","bug","(","#","8136",")",",","you","must","not","initialize","CUDA","context","before","the","trainer","starts","training",".","Therefore","TF","functions","like","`","is_gpu_available","(",")","`","or","`","list_local_devices","(",")","`","must","be","avoided",".","You","can",",","however",",","use","`","tf.config.experimental.list_physical_devices","(","'GPU","'",")","`",",","introduced","in","TF","1.14",".","3",".","Horovod","supports","both","MPI","and","gloo",".","There","are","a","few","drawbacks","of","the","MPI","backend",":","+","MPI","does","not","like","`","fork","(",")","`",".","If","your","code","(","e.g",".","dataflow",")","contains","multiprocessing",",","it","may","cause","problems",".","+","MPI","sometimes","fails","to","kill","all","processes","in","the","end",".","Be","sure","to","check","it","afterwards",".","The","gloo","backend","is","recommended","though","it","may","come","with","very","minor","slow","down",".","To","use","gloo","backend",",","see","`","horovod","documentation","<","https",":","\/\/github.com\/horovod\/horovod","#","running-horovod",">","`","_","for","more","details",".","4",".","Keep","in","mind","that","there","is","one","process","running","the","script","per","GPU",",","therefore",":","+","Make","sure","your","InputSource","has","reasonable","randomness",".","+","If","your","data","processing","is","heavy",",","doing","it","in","a","single","dedicated","process","might","be","a","better","choice","than","doing","them","repeatedly","in","each","process",".","+","You","need","to","make","sure","log","directories","in","each","process","wo","n't","conflict",".","You","can","set","it","only","for","the","chief","process",",","or","set","a","different","one","for","each","process",".","+","Callbacks","have","an","option","to","be","run","only","in","the","chief","process",",","or","in","all","processes",".","See",":","meth",":","`","Callback.set_chief_only","(",")","`",".","Most","callbacks","have","a","reasonable","default","already",",","but","certain","callbacks","may","need","your","customization",".","Report","an","issue","if","you","find","any","bad","defaults",".","+","You","can","use","Horovod","API","such","as","`","hvd.rank","(",")","`","to","know","which","process","you","are","and","choose","different","code","path",".","Chief","process","has","rank","0",".","5",".","Due","to","these","caveats",",","see","`","ResNet-Horovod","<","https",":","\/\/github.com\/tensorpack\/benchmarks\/tree\/master\/ResNet-Horovod",">","`","_","for","a","full","example","which","has","handled","these","common","issues",".","This","example","can","train","ImageNet","in","roughly","an","hour","following","the","paper","'s","setup",".","Attributes",":","BROADCAST_EVERY_EPOCH","(","bool",")",":","Whether","to","broadcast","the","variables","every","epoch",".","Theoretically","this","is","a","no-op","(","because","the","variables","are","supposed","to","be","in-sync",")",".","But","this","cheap","operation","may","help","prevent","certain","numerical","issues","in","practice",".","Note","that","in","cases","such","as","BatchNorm",",","the","variables","may","not","be","in","sync",":","e.g.",",","non-master","worker","may","not","maintain","EMAs",".","For","benchmark",",","disable","this","option","."]
docstr: Returns:
      bool: whether hvd is currently running under MPI
docstr_tok: ["Returns",":","bool",":","whether","hvd","is","currently","running","under","MPI"]
code: def mpi_enabled(self):
      """
      Returns:
          bool: whether hvd is currently running under MPI
      """
      try:
          return self.hvd.mpi_enabled()
      except AttributeError:
          return False
  
imports: import multiprocessing
  import os
  import sys
  import tensorflow
  from tensorpack.compat import tfv1
  from ..callbacks import CallbackFactory, RunOp
  from ..graph_builder.distributed import DistributedParameterServerBuilder, DistributedReplicatedBuilder
  from ..graph_builder.training import AsyncMultiGPUBuilder, SyncMultiGPUParameterServerBuilder, SyncMultiGPUReplicatedBuilder
  from ..graph_builder.utils import override_to_local_variable
  from ..input_source import FeedfreeInput, QueueInput
  from ..tfutils import get_global_step_var
  from ..tfutils.common import get_tf_version_tuple
  from ..tfutils.distributed import get_distributed_session_creator
  from ..tfutils.sesscreate import NewSessionCreator
  from ..tfutils.tower import TrainTowerContext
  from ..utils import logger
  from ..utils.argtools import map_arg
  from ..utils.develop import HIDE_DOC, deprecated
  from .tower import SingleCostTrainer
label_desc: Use image node_id 2 for calling the HorovodTrainer obj's underlying member method code with example usage: obj.mpi_enabled() and returns: self, False