class_docstr: Stochastic gradient descent with momentum.
  
  Given the parameters :math:`\theta`, the learning rate :math:`\alpha`,
  and the gradients :math:`\nabla J(\theta; x)`
  computed on the minibatch data :math:`x`, SGD updates the parameters via
  
  .. math::
      \theta' = \theta - \alpha\nabla J(\theta; x)
  
  Here we implement SGD with momentum. Momentum tracks the history of
  gradient updates to help the system move faster through saddle points.
  Given the additional parameters: momentum :math:`\gamma`, weight decay :math:`\lambda`,
  and current velocity :math:`v`, we use the following update equations
  
  .. math::
      v' = \gamma v - \alpha(\nabla J(\theta; x) + \lambda\theta)
      theta' = \theta + v'
  
  The optional `nesterov` parameter implements Nesterov Accelerated Gradient.
  If this is set, we use the following update equations instead
  .. math::
      v' = \gamma^2 v + \alpha (\gamma + 1) (\nabla J(\theta; x) + \lambda\theta)
      theta' = \theta + v'
  
  Example usage:
  
  .. code-block:: python
  
      from neon.optimizers import GradientDescentMomentum
  
      # use SGD with learning rate 0.01 and momentum 0.9, while
      # clipping the gradient magnitude to between -5 and 5.
      opt = GradientDescentMomentum(0.01, 0.9, gradient_clip_value = 5)
class_docstr_tok: ["Stochastic","gradient","descent","with","momentum",".","Given","the","parameters",":","math",":","`","\\theta","`",",","the","learning","rate",":","math",":","`","\\alpha","`",",","and","the","gradients",":","math",":","`","\\nabla","J","(","\\theta",";","x",")","`","computed","on","the","minibatch","data",":","math",":","`","x","`",",","SGD","updates","the","parameters","via","..","math",":",":","\\theta","'","=","\\theta","-","\\alpha\\nabla","J","(","\\theta",";","x",")","Here","we","implement","SGD","with","momentum",".","Momentum","tracks","the","history","of","gradient","updates","to","help","the","system","move","faster","through","saddle","points",".","Given","the","additional","parameters",":","momentum",":","math",":","`","\\gamma","`",",","weight","decay",":","math",":","`","\\lambda","`",",","and","current","velocity",":","math",":","`","v","`",",","we","use","the","following","update","equations","..","math",":",":","v","'","=","\\gamma","v","-","\\alpha","(","\\nabla","J","(","\\theta",";","x",")","+","\\lambda\\theta",")","theta","'","=","\\theta","+","v'","The","optional","`","nesterov","`","parameter","implements","Nesterov","Accelerated","Gradient",".","If","this","is","set",",","we","use","the","following","update","equations","instead","..","math",":",":","v","'","=","\\gamma^2","v","+","\\alpha","(","\\gamma","+","1",")","(","\\nabla","J","(","\\theta",";","x",")","+","\\lambda\\theta",")","theta","'","=","\\theta","+","v'","Example","usage",":","..","code-block",":",":","python","from","neon.optimizers","import","GradientDescentMomentum","#","use","SGD","with","learning","rate","0.01","and","momentum","0.9",",","while","#","clipping","the","gradient","magnitude","to","between","-5","and","5.","opt","=","GradientDescentMomentum","(","0.01",",","0.9",",","gradient_clip_value","=","5",")"]
docstr: Apply the learning rule to all the layers and update the states.
  
  Arguments:
      layer_list (list): a list of Layer objects to optimize.
      epoch (int): the current epoch, needed for the Schedule object.
docstr_tok: ["Apply","the","learning","rule","to","all","the","layers","and","update","the","states",".","Arguments",":","layer_list","(","list",")",":","a","list","of","Layer","objects","to","optimize",".","epoch","(","int",")",":","the","current","epoch",",","needed","for","the","Schedule","object","."]
code: def optimize(self, layer_list, epoch):
      """
      Apply the learning rule to all the layers and update the states.
  
      Arguments:
          layer_list (list): a list of Layer objects to optimize.
          epoch (int): the current epoch, needed for the Schedule object.
      """
      lrate = self.schedule.get_learning_rate(self.learning_rate, epoch)
      param_list = get_param_list(layer_list)
  
      scale_factor = self.clip_gradient_norm(
          param_list, self.gradient_clip_norm
      )
  
      for (param, grad), states in param_list:
          param.rounding = self.stochastic_round
          if len(states) == 0 and self.momentum_coef != 0:
              states.append(self.be.zeros_like(grad))
  
          grad = grad / self.be.bsz
          grad = self.clip_value(grad, self.gradient_clip_value)
  
          if self.momentum_coef == 0:
              param[:] = (-lrate * scale_factor) * grad + (
                  1 - lrate * self.wdecay
              ) * param
              param = self.clip_value(param, self.param_clip_value)
          else:
              grad = scale_factor * grad + self.wdecay * param
              velocity = states[0]
              velocity[:] = self.momentum_coef * velocity - lrate * grad
  
              # Nesterov accelerated gradient (NAG) is implemented the same
              # as in torch's "sgd.lua". It's a reformulation of Sutskever's
              # NAG equation found in "On the importance of initialization
              # and momentum in deep learning".
              if self.nesterov:
                  param[:] = self.clip_value(
                      param
                      + self.momentum_coef * velocity
                      - lrate * grad,
                      self.param_clip_value,
                  )
              else:
                  param[:] = self.clip_value(
                      param + velocity, self.param_clip_value
                  )
  
imports: from __future__ import division
  from neon import NervanaObject
  from neon.util.persist import load_class
  import logging
  import numpy
label_desc: Use image node_id 2 for calling the GradientDescentMomentum obj's underlying member method code with example usage: obj.optimize(layer_list, epoch) without return types