class_docstr: FP16_Optimizer is designed to wrap an existing PyTorch optimizer,
  and enable an fp16 model to be trained using a main copy of fp32 weights.
  
  Args:
      optimizer (torch.optim.optimizer):  Existing optimizer containing initialized fp16 parameters.  Internally, FP16_Optimizer replaces the passed optimizer's fp16 parameters with new fp32 parameters copied from the original ones.  FP16_Optimizer also stores references to the original fp16 parameters, and updates these fp16 parameters from the main fp32 copy after each step.
      static_loss_scale (float, optional, default=1.0):  Loss scale used internally to scale fp16 gradients computed by the model.  Scaled gradients will be copied to fp32, then downscaled before being applied to the fp32 main params, so static_loss_scale should not affect learning rate.
      dynamic_loss_scale (bool, optional, default=False):  Use dynamic loss scaling.  If True, this will override any static_loss_scale option.
class_docstr_tok: ["FP16_Optimizer","is","designed","to","wrap","an","existing","PyTorch","optimizer",",","and","enable","an","fp16","model","to","be","trained","using","a","main","copy","of","fp32","weights",".","Args",":","optimizer","(","torch.optim.optimizer",")",":","Existing","optimizer","containing","initialized","fp16","parameters",".","Internally",",","FP16_Optimizer","replaces","the","passed","optimizer","'s","fp16","parameters","with","new","fp32","parameters","copied","from","the","original","ones",".","FP16_Optimizer","also","stores","references","to","the","original","fp16","parameters",",","and","updates","these","fp16","parameters","from","the","main","fp32","copy","after","each","step",".","static_loss_scale","(","float",",","optional",",","default=1.0",")",":","Loss","scale","used","internally","to","scale","fp16","gradients","computed","by","the","model",".","Scaled","gradients","will","be","copied","to","fp32",",","then","downscaled","before","being","applied","to","the","fp32","main","params",",","so","static_loss_scale","should","not","affect","learning","rate",".","dynamic_loss_scale","(","bool",",","optional",",","default=False",")",":","Use","dynamic","loss","scaling",".","If","True",",","this","will","override","any","static_loss_scale","option","."]
docstr: If no closure is supplied, step should be called after fp16_optimizer_obj.backward(loss).
  step updates the fp32 main copy of parameters using the optimizer supplied to
  FP16_Optimizer's constructor, then copies the updated fp32 params into the fp16 params
  originally referenced by Fp16_Optimizer's constructor, so the user may immediately run
  another forward pass using their model.
  
  If a closure is supplied, step may be called without a prior call to self.backward(loss).
  However, the user should take care that any loss.backward() call within the closure
  has been replaced by fp16_optimizer_obj.backward(loss).
  
  Args:
     closure (optional):  Closure that will be supplied to the underlying optimizer originally passed to FP16_Optimizer's constructor.  closure should call zero_grad on the FP16_Optimizer object, compute the loss, call .backward(loss), and return the loss.
  
  Closure example::
  
      # optimizer is assumed to be an FP16_Optimizer object, previously constructed from an
      # existing pytorch optimizer.
      for input, target in dataset:
          def closure():
              optimizer.zero_grad()
              output = model(input)
              loss = loss_fn(output, target)
              optimizer.backward(loss)
              return loss
          optimizer.step(closure)
  
  .. note::
      The only changes that need to be made compared to
      `ordinary optimizer closures`_ are that "optimizer" itself should be an instance of
      FP16_Optimizer, and that the call to loss.backward should be replaced by
      optimizer.backward(loss).
  
  .. warning::
      Currently, calling step with a closure is not compatible with dynamic loss scaling.
  
  .. _`ordinary optimizer closures`:
      http://pytorch.org/docs/master/optim.html#optimizer-step-closure
docstr_tok: ["If","no","closure","is","supplied",",","step","should","be","called","after","fp16_optimizer_obj.backward","(","loss",")",".","step","updates","the","fp32","main","copy","of","parameters","using","the","optimizer","supplied","to","FP16_Optimizer","'s","constructor",",","then","copies","the","updated","fp32","params","into","the","fp16","params","originally","referenced","by","Fp16_Optimizer","'s","constructor",",","so","the","user","may","immediately","run","another","forward","pass","using","their","model",".","If","a","closure","is","supplied",",","step","may","be","called","without","a","prior","call","to","self.backward","(","loss",")",".","However",",","the","user","should","take","care","that","any","loss.backward","(",")","call","within","the","closure","has","been","replaced","by","fp16_optimizer_obj.backward","(","loss",")",".","Args",":","closure","(","optional",")",":","Closure","that","will","be","supplied","to","the","underlying","optimizer","originally","passed","to","FP16_Optimizer","'s","constructor",".","closure","should","call","zero_grad","on","the","FP16_Optimizer","object",",","compute","the","loss",",","call",".backward","(","loss",")",",","and","return","the","loss",".","Closure","example",":",":","#","optimizer","is","assumed","to","be","an","FP16_Optimizer","object",",","previously","constructed","from","an","#","existing","pytorch","optimizer",".","for","input",",","target","in","dataset",":","def","closure","(",")",":","optimizer.zero_grad","(",")","output","=","model","(","input",")","loss","=","loss_fn","(","output",",","target",")","optimizer.backward","(","loss",")","return","loss","optimizer.step","(","closure",")","..","note",":",":","The","only","changes","that","need","to","be","made","compared","to","`","ordinary","optimizer","closures","`","_","are","that","``","optimizer","''","itself","should","be","an","instance","of","FP16_Optimizer",",","and","that","the","call","to","loss.backward","should","be","replaced","by","optimizer.backward","(","loss",")",".","..","warning",":",":","Currently",",","calling","step","with","a","closure","is","not","compatible","with","dynamic","loss","scaling",".","..","_","`","ordinary","optimizer","closures","`",":","http",":","\/\/pytorch.org\/docs\/master\/optim.html","#","optimizer-step-closure"]
code: def step(self, closure=None):  # could add clip option.
      """
      If no closure is supplied, step should be called after fp16_optimizer_obj.backward(loss).
      step updates the fp32 main copy of parameters using the optimizer supplied to
      FP16_Optimizer's constructor, then copies the updated fp32 params into the fp16 params
      originally referenced by Fp16_Optimizer's constructor, so the user may immediately run
      another forward pass using their model.
  
      If a closure is supplied, step may be called without a prior call to self.backward(loss).
      However, the user should take care that any loss.backward() call within the closure
      has been replaced by fp16_optimizer_obj.backward(loss).
  
      Args:
         closure (optional):  Closure that will be supplied to the underlying optimizer originally passed to FP16_Optimizer's constructor.  closure should call zero_grad on the FP16_Optimizer object, compute the loss, call .backward(loss), and return the loss.
  
      Closure example::
  
          # optimizer is assumed to be an FP16_Optimizer object, previously constructed from an
          # existing pytorch optimizer.
          for input, target in dataset:
              def closure():
                  optimizer.zero_grad()
                  output = model(input)
                  loss = loss_fn(output, target)
                  optimizer.backward(loss)
                  return loss
              optimizer.step(closure)
  
      .. note::
          The only changes that need to be made compared to
          `ordinary optimizer closures`_ are that "optimizer" itself should be an instance of
          FP16_Optimizer, and that the call to loss.backward should be replaced by
          optimizer.backward(loss).
  
      .. warning::
          Currently, calling step with a closure is not compatible with dynamic loss scaling.
  
      .. _`ordinary optimizer closures`:
          http://pytorch.org/docs/master/optim.html#optimizer-step-closure
      """
      if closure is not None and isinstance(
          self.loss_scaler, DynamicLossScaler
      ):
          raise TypeError(
              "Using step with a closure is currently not "
              "compatible with dynamic loss scaling."
          )
  
      scale = self.loss_scaler.loss_scale
      self._update_scale(self.overflow)
  
      if self.overflow:
          print(
              "OVERFLOW! Skipping step. Attempted loss scale: {}".format(
                  scale
              )
          )
          return
  
      if closure is not None:
          self._step_with_closure(closure)
      else:
          self.optimizer.step()
  
      self._copy_params_fp32_to_fp16()
  
      return
  
imports: import torch
  from torch import nn
  from torch.autograd import Variable
  from torch.nn.parameter import Parameter
  from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
  from common.loss_scaler import DynamicLossScaler, LossScaler
label_desc: Use image node_id 11 for calling the FP16_Optimizer obj's underlying member method code with example usage: obj.step(closure) without return types