class_docstr: Adadelta optimization algorithm.
  
  Similar to RMSprop, Adadelta tracks the running average of the
  gradients, :math:`\mu_J`, over a window size :math:`1/\lambda`, where
  :math:`\lambda` is the parameter ``decay``. Adadelta also tracks an average of the
  recent update steps, which we denote as :math:`\mu_\theta`, and sets the learning rate
  as the ratio of the two averages:
  
  .. math::
  
      \mu_J' &= \lambda\mu_J + (1-\lambda) (\nabla J)^2
  
  .. math::
  
      \Delta \theta &= \sqrt{\frac{\mu_\theta + \epsilon}{\mu_J' + \epsilon}} \nabla J
  
  .. math::
  
      \mu_\theta &= \lambda \mu_\theta + (1-\rho) (\Delta \theta)^2
  
  .. math::
  
      \theta &= \theta - \Delta \theta
  
  Note that the learning rate is a ratio of the average updates from the
  previous step, :math:`\mu_\theta`, divided by the average gradients including the current
  step, :math:`\mu'_J`.
  
  Example usage:
  
  .. code-block:: python
  
      from neon.optimizers import Adadelta
      # use Adagrad with a learning rate of 0.01
      optimizer = Adadelta(decay=0.95, epsilon=1e-6)
class_docstr_tok: ["Adadelta","optimization","algorithm",".","Similar","to","RMSprop",",","Adadelta","tracks","the","running","average","of","the","gradients",",",":","math",":","`","\\mu_J","`",",","over","a","window","size",":","math",":","`","1\/\\lambda","`",",","where",":","math",":","`","\\lambda","`","is","the","parameter","``","decay","``",".","Adadelta","also","tracks","an","average","of","the","recent","update","steps",",","which","we","denote","as",":","math",":","`","\\mu_\\theta","`",",","and","sets","the","learning","rate","as","the","ratio","of","the","two","averages",":","..","math",":",":","\\mu_J","'","&","=","\\lambda\\mu_J","+","(","1-\\lambda",")","(","\\nabla","J",")","^2","..","math",":",":","\\Delta","\\theta","&","=","\\sqrt","{","\\frac","{","\\mu_\\theta","+","\\epsilon","}","{","\\mu_J","'","+","\\epsilon","}","}","\\nabla","J","..","math",":",":","\\mu_\\theta","&","=","\\lambda","\\mu_\\theta","+","(","1-\\rho",")","(","\\Delta","\\theta",")","^2","..","math",":",":","\\theta","&","=","\\theta","-","\\Delta","\\theta","Note","that","the","learning","rate","is","a","ratio","of","the","average","updates","from","the","previous","step",",",":","math",":","`","\\mu_\\theta","`",",","divided","by","the","average","gradients","including","the","current","step",",",":","math",":","`","\\mu'_J","`",".","Example","usage",":","..","code-block",":",":","python","from","neon.optimizers","import","Adadelta","#","use","Adagrad","with","a","learning","rate","of","0.01","optimizer","=","Adadelta","(","decay=0.95",",","epsilon=1e-6",")"]
docstr: Apply the learning rule to all the layers and update the states.
  
  Arguments:
      param_list (list): a list of tuples of the form ((param, grad), state),
                         corresponding to parameters, grads,
                         and states of layers to be updated
      epoch (int): the current epoch, needed for the Schedule object.
docstr_tok: ["Apply","the","learning","rule","to","all","the","layers","and","update","the","states",".","Arguments",":","param_list","(","list",")",":","a","list","of","tuples","of","the","form","(","(","param",",","grad",")",",","state",")",",","corresponding","to","parameters",",","grads",",","and","states","of","layers","to","be","updated","epoch","(","int",")",":","the","current","epoch",",","needed","for","the","Schedule","object","."]
code: def optimize(self, layer_list, epoch):
      """
      Apply the learning rule to all the layers and update the states.
  
      Arguments:
          param_list (list): a list of tuples of the form ((param, grad), state),
                             corresponding to parameters, grads,
                             and states of layers to be updated
          epoch (int): the current epoch, needed for the Schedule object.
      """
      epsilon, decay = (self.epsilon, self.decay)
  
      param_list = get_param_list(layer_list)
  
      for (param, grad), states in param_list:
          param.rounding = self.stochastic_round
  
          if len(states) == 0:
              # E[Grad^2], E[Delt^2], updates
              states.extend(
                  [self.be.zeros_like(grad) for i in range(3)]
              )
  
          grad = grad / self.be.bsz
          states[0][:] = states[0] * decay + (1.0 - decay) * grad * grad
          states[2][:] = (
              self.be.sqrt(
                  (states[1] + epsilon) / (states[0] + epsilon)
              )
              * grad
          )
          states[1][:] = (
              states[1] * decay + (1.0 - decay) * states[2] * states[2]
          )
  
          param[:] = self.clip_value(
              param - states[2], self.param_clip_value
          )
  
imports: from __future__ import division
  from neon import NervanaObject
  from neon.util.persist import load_class
  import logging
  import numpy
label_desc: Use image node_id 2 for calling the Adadelta obj's underlying member method code with example usage: obj.optimize(layer_list, epoch) without return types