class_docstr: FP16_Optimizer is designed to wrap an existing PyTorch optimizer,
  and enable an fp16 model to be trained using a main copy of fp32 weights.
  
  Args:
      optimizer (torch.optim.optimizer):  Existing optimizer containing initialized fp16 parameters.  Internally, FP16_Optimizer replaces the passed optimizer's fp16 parameters with new fp32 parameters copied from the original ones.  FP16_Optimizer also stores references to the original fp16 parameters, and updates these fp16 parameters from the main fp32 copy after each step.
      static_loss_scale (float, optional, default=1.0):  Loss scale used internally to scale fp16 gradients computed by the model.  Scaled gradients will be copied to fp32, then downscaled before being applied to the fp32 main params, so static_loss_scale should not affect learning rate.
      dynamic_loss_scale (bool, optional, default=False):  Use dynamic loss scaling.  If True, this will override any static_loss_scale option.
class_docstr_tok: ["FP16_Optimizer","is","designed","to","wrap","an","existing","PyTorch","optimizer",",","and","enable","an","fp16","model","to","be","trained","using","a","main","copy","of","fp32","weights",".","Args",":","optimizer","(","torch.optim.optimizer",")",":","Existing","optimizer","containing","initialized","fp16","parameters",".","Internally",",","FP16_Optimizer","replaces","the","passed","optimizer","'s","fp16","parameters","with","new","fp32","parameters","copied","from","the","original","ones",".","FP16_Optimizer","also","stores","references","to","the","original","fp16","parameters",",","and","updates","these","fp16","parameters","from","the","main","fp32","copy","after","each","step",".","static_loss_scale","(","float",",","optional",",","default=1.0",")",":","Loss","scale","used","internally","to","scale","fp16","gradients","computed","by","the","model",".","Scaled","gradients","will","be","copied","to","fp32",",","then","downscaled","before","being","applied","to","the","fp32","main","params",",","so","static_loss_scale","should","not","affect","learning","rate",".","dynamic_loss_scale","(","bool",",","optional",",","default=False",")",":","Use","dynamic","loss","scaling",".","If","True",",","this","will","override","any","static_loss_scale","option","."]
docstr: fp16_optimizer_obj.backward performs the following conceptual operations:
  
  fp32_loss = loss.float() (see first Note below)
  
  scaled_loss = fp32_loss*loss_scale
  
  scaled_loss.backward(), which accumulates scaled gradients into the .grad attributes of the
  fp16 model's leaves.
  
  fp16 grads are then copied to the stored fp32 params' .grad attributes (see second Note).
  
  Finally, fp32 grads are divided by loss_scale.
  
  In this way, after fp16_optimizer_obj.backward, the fp32 parameters have fresh gradients,
  and fp16_optimizer_obj.step may be called.
  
  .. note::
      Converting the loss to fp32 before applying the loss scale provides some
      additional safety against overflow if the user has supplied an fp16 value.
      However, for maximum overflow safety, the user should
      compute the loss criterion (MSE, cross entropy, etc) in fp32 before supplying it to
      fp16_optimizer_obj.backward.
  
  .. note::
      The gradients found in an fp16 model's leaves after a call to
      fp16_optimizer_obj.backward should not be regarded as valid in general,
      because it's possible
      they have been scaled (and in the case of dynamic loss scaling,
      the scale factor may change over time).
      If the user wants to inspect gradients after a call to fp16_optimizer_obj.backward,
      only the main gradients should be regarded as valid, and can be retrieved via
      :attr:`inspect_fp32_grad_data()`.
  
  
  Args:
      loss:  The loss output by the user's model.  loss may be either float or half (but see first Note above).
      update_fp32_grads (bool, optional, default=True):  Option to copy fp16 grads to fp32 grads on this call.  By setting this to False, the user can delay this copy, which is useful to eliminate redundant fp16->fp32 grad copies if fp16_optimizer_obj.backward is being called on multiple losses in one iteration.  If set to False, the user becomes responsible for calling fp16_optimizer_obj.update_fp32_grads before calling fp16_optimizer_obj.step.
  
  Example::
  
      # Ordinary operation:
      optimizer.backward(loss)
  
      # Naive operation with multiple losses (technically valid, but less efficient):
      # fp32 grads will be correct after the second call,  but
      # the first call incurs an unnecessary fp16->fp32 grad copy.
      optimizer.backward(loss1)
      optimizer.backward(loss2)
  
      # More efficient way to handle multiple losses:
      # The fp16->fp32 grad copy is delayed until fp16 grads from all
      # losses have been accumulated.
      optimizer.backward(loss1, update_fp32_grads=False)
      optimizer.backward(loss2, update_fp32_grads=False)
      optimizer.update_fp32_grads()
docstr_tok: ["fp16_optimizer_obj.backward","performs","the","following","conceptual","operations",":","fp32_loss","=","loss.float","(",")","(","see","first","Note","below",")","scaled_loss","=","fp32_loss","*","loss_scale","scaled_loss.backward","(",")",",","which","accumulates","scaled","gradients","into","the",".grad","attributes","of","the","fp16","model","'s","leaves",".","fp16","grads","are","then","copied","to","the","stored","fp32","params","'",".grad","attributes","(","see","second","Note",")",".","Finally",",","fp32","grads","are","divided","by","loss_scale",".","In","this","way",",","after","fp16_optimizer_obj.backward",",","the","fp32","parameters","have","fresh","gradients",",","and","fp16_optimizer_obj.step","may","be","called",".","..","note",":",":","Converting","the","loss","to","fp32","before","applying","the","loss","scale","provides","some","additional","safety","against","overflow","if","the","user","has","supplied","an","fp16","value",".","However",",","for","maximum","overflow","safety",",","the","user","should","compute","the","loss","criterion","(","MSE",",","cross","entropy",",","etc",")","in","fp32","before","supplying","it","to","fp16_optimizer_obj.backward",".","..","note",":",":","The","gradients","found","in","an","fp16","model","'s","leaves","after","a","call","to","fp16_optimizer_obj.backward","should","not","be","regarded","as","valid","in","general",",","because","it","'s","possible","they","have","been","scaled","(","and","in","the","case","of","dynamic","loss","scaling",",","the","scale","factor","may","change","over","time",")",".","If","the","user","wants","to","inspect","gradients","after","a","call","to","fp16_optimizer_obj.backward",",","only","the","main","gradients","should","be","regarded","as","valid",",","and","can","be","retrieved","via",":","attr",":","`","inspect_fp32_grad_data","(",")","`",".","Args",":","loss",":","The","loss","output","by","the","user","'s","model",".","loss","may","be","either","float","or","half","(","but","see","first","Note","above",")",".","update_fp32_grads","(","bool",",","optional",",","default=True",")",":","Option","to","copy","fp16","grads","to","fp32","grads","on","this","call",".","By","setting","this","to","False",",","the","user","can","delay","this","copy",",","which","is","useful","to","eliminate","redundant","fp16-",">","fp32","grad","copies","if","fp16_optimizer_obj.backward","is","being","called","on","multiple","losses","in","one","iteration",".","If","set","to","False",",","the","user","becomes","responsible","for","calling","fp16_optimizer_obj.update_fp32_grads","before","calling","fp16_optimizer_obj.step",".","Example",":",":","#","Ordinary","operation",":","optimizer.backward","(","loss",")","#","Naive","operation","with","multiple","losses","(","technically","valid",",","but","less","efficient",")",":","#","fp32","grads","will","be","correct","after","the","second","call",",","but","#","the","first","call","incurs","an","unnecessary","fp16-",">","fp32","grad","copy",".","optimizer.backward","(","loss1",")","optimizer.backward","(","loss2",")","#","More","efficient","way","to","handle","multiple","losses",":","#","The","fp16-",">","fp32","grad","copy","is","delayed","until","fp16","grads","from","all","#","losses","have","been","accumulated",".","optimizer.backward","(","loss1",",","update_fp32_grads=False",")","optimizer.backward","(","loss2",",","update_fp32_grads=False",")","optimizer.update_fp32_grads","(",")"]
code: def backward(self, loss, update_fp32_grads=True):
      """
      fp16_optimizer_obj.backward performs the following conceptual operations:
  
      fp32_loss = loss.float() (see first Note below)
  
      scaled_loss = fp32_loss*loss_scale
  
      scaled_loss.backward(), which accumulates scaled gradients into the .grad attributes of the
      fp16 model's leaves.
  
      fp16 grads are then copied to the stored fp32 params' .grad attributes (see second Note).
  
      Finally, fp32 grads are divided by loss_scale.
  
      In this way, after fp16_optimizer_obj.backward, the fp32 parameters have fresh gradients,
      and fp16_optimizer_obj.step may be called.
  
      .. note::
          Converting the loss to fp32 before applying the loss scale provides some
          additional safety against overflow if the user has supplied an fp16 value.
          However, for maximum overflow safety, the user should
          compute the loss criterion (MSE, cross entropy, etc) in fp32 before supplying it to
          fp16_optimizer_obj.backward.
  
      .. note::
          The gradients found in an fp16 model's leaves after a call to
          fp16_optimizer_obj.backward should not be regarded as valid in general,
          because it's possible
          they have been scaled (and in the case of dynamic loss scaling,
          the scale factor may change over time).
          If the user wants to inspect gradients after a call to fp16_optimizer_obj.backward,
          only the main gradients should be regarded as valid, and can be retrieved via
          :attr:`inspect_fp32_grad_data()`.
  
  
      Args:
          loss:  The loss output by the user's model.  loss may be either float or half (but see first Note above).
          update_fp32_grads (bool, optional, default=True):  Option to copy fp16 grads to fp32 grads on this call.  By setting this to False, the user can delay this copy, which is useful to eliminate redundant fp16->fp32 grad copies if fp16_optimizer_obj.backward is being called on multiple losses in one iteration.  If set to False, the user becomes responsible for calling fp16_optimizer_obj.update_fp32_grads before calling fp16_optimizer_obj.step.
  
      Example::
  
          # Ordinary operation:
          optimizer.backward(loss)
  
          # Naive operation with multiple losses (technically valid, but less efficient):
          # fp32 grads will be correct after the second call,  but
          # the first call incurs an unnecessary fp16->fp32 grad copy.
          optimizer.backward(loss1)
          optimizer.backward(loss2)
  
          # More efficient way to handle multiple losses:
          # The fp16->fp32 grad copy is delayed until fp16 grads from all
          # losses have been accumulated.
          optimizer.backward(loss1, update_fp32_grads=False)
          optimizer.backward(loss2, update_fp32_grads=False)
          optimizer.update_fp32_grads()
      """
      self.loss_scaler.backward(loss.float())
      if update_fp32_grads:
          self.update_fp32_grads()
  
imports: import torch
  from torch import nn
  from torch.autograd import Variable
  from torch.nn.parameter import Parameter
  from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
  from common.loss_scaler import DynamicLossScaler, LossScaler
label_desc: Use image node_id 13 for calling the FP16_Optimizer obj's underlying member method code with example usage: obj.backward(loss, update_fp32_grads) without return types