class_docstr: FP16_Optimizer is designed to wrap an existing PyTorch optimizer,
  and enable an fp16 model to be trained using a main copy of fp32 weights.
  
  Args:
      optimizer (torch.optim.optimizer):  Existing optimizer containing initialized fp16 parameters.  Internally, FP16_Optimizer replaces the passed optimizer's fp16 parameters with new fp32 parameters copied from the original ones.  FP16_Optimizer also stores references to the original fp16 parameters, and updates these fp16 parameters from the main fp32 copy after each step.
      static_loss_scale (float, optional, default=1.0):  Loss scale used internally to scale fp16 gradients computed by the model.  Scaled gradients will be copied to fp32, then downscaled before being applied to the fp32 main params, so static_loss_scale should not affect learning rate.
      dynamic_loss_scale (bool, optional, default=False):  Use dynamic loss scaling.  If True, this will override any static_loss_scale option.
class_docstr_tok: ["FP16_Optimizer","is","designed","to","wrap","an","existing","PyTorch","optimizer",",","and","enable","an","fp16","model","to","be","trained","using","a","main","copy","of","fp32","weights",".","Args",":","optimizer","(","torch.optim.optimizer",")",":","Existing","optimizer","containing","initialized","fp16","parameters",".","Internally",",","FP16_Optimizer","replaces","the","passed","optimizer","'s","fp16","parameters","with","new","fp32","parameters","copied","from","the","original","ones",".","FP16_Optimizer","also","stores","references","to","the","original","fp16","parameters",",","and","updates","these","fp16","parameters","from","the","main","fp32","copy","after","each","step",".","static_loss_scale","(","float",",","optional",",","default=1.0",")",":","Loss","scale","used","internally","to","scale","fp16","gradients","computed","by","the","model",".","Scaled","gradients","will","be","copied","to","fp32",",","then","downscaled","before","being","applied","to","the","fp32","main","params",",","so","static_loss_scale","should","not","affect","learning","rate",".","dynamic_loss_scale","(","bool",",","optional",",","default=False",")",":","Use","dynamic","loss","scaling",".","If","True",",","this","will","override","any","static_loss_scale","option","."]
code: def _step_with_closure(self, closure):
      def wrapped_closure():
          if self.first_closure_call_this_step:
              """
              We expect that the fp16 params are initially fresh on entering self.step(),
              so _copy_params_fp32_to_fp16() is unnecessary the first time wrapped_closure()
              is called within self.optimizer.step().
              """
              self.first_closure_call_this_step = False
          else:
              """
              If self.optimizer.step() internally calls wrapped_closure more than once,
              it may update the fp32 params after each call.  However, self.optimizer
              doesn't know about the fp16 params at all.  If the fp32 params get updated,
              we can't rely on self.optimizer to refresh the fp16 params.  We need
              to handle that manually:
              """
              self._copy_params_fp32_to_fp16()
  
          """
              Our API expects the user to give us ownership of the backward() call by
              replacing all calls to loss.backward() with optimizer.backward(loss).
              This requirement holds whether or not the call to backward() is made within
              a closure.
              If the user is properly calling optimizer.backward(loss) within "closure,"
              calling closure() here will give the fp32 main params fresh gradients
              for the optimizer to play with,
              so all wrapped_closure needs to do is call closure() and return the loss.
              """
          temp_loss = closure()
          return temp_loss
  
      self.optimizer.step(wrapped_closure)
  
      self.first_closure_call_this_step = True
  
imports: import torch
  from torch import nn
  from torch.autograd import Variable
  from torch.nn.parameter import Parameter
  from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
  from common.loss_scaler import DynamicLossScaler, LossScaler
label_desc: Use image node_id 12 for calling the FP16_Optimizer obj's underlying member method code with example usage: obj._step_with_closure(closure) and returns: temp_loss