class_docstr: Represents the learning rate schedule described in
  :cite:t:`https://doi.org/10.48550/arxiv.1608.03983`.
  
  **During warmup:**
  
  .. math::
      \eta_t = \eta_{base} \frac{t}{T_{warmup}}
  
  **After warmup:**
  
  .. math::
      \eta_t = \eta_{final}^i + \frac{1}{2} (\eta_{base}^i - \eta_{final}^i) (1 + \text{cos}(\pi \frac{t_{i}}{T_{i}}))
  
  where :math:`i` is the number of the current annealing cycle, :math:`t_i` is
  the number of steps taken since the last restart, and :math:`T_i` is the
  total number of steps within the :math:`i`-th cycle (i.e. *length* of the
  cycle).
  
  *Cosine Annealing* is a type of learning rate schedule that has the effect
  of starting with a large learning rate that is relatively rapidly decreased
  to a minimum value before being increased rapidly again.
  
  Please refer to the paper to learn more about the details.
  
  In addition to the original schedule, this implementation also supports a
  warmup phase where the learning rate is linearly increased for the first
  :math:`T_{warmup}` training steps to the base learning rate.
  
  .. note::
      This scheduler is not chainable.
class_docstr_tok: ["Represents","the","learning","rate","schedule","described","in",":","cite",":","t",":","`","https",":","\/\/doi.org\/10.48550\/arxiv.1608.03983","`",".","*","*","During","warmup",":","*","*","..","math",":",":","\\eta_t","=","\\eta_","{","base","}","\\frac","{","t","}","{","T_","{","warmup","}","}","*","*","After","warmup",":","*","*","..","math",":",":","\\eta_t","=","\\eta_","{","final","}","^i","+","\\frac","{","1","}","{","2","}","(","\\eta_","{","base","}","^i","-","\\eta_","{","final","}","^i",")","(","1","+","\\text","{","cos","}","(","\\pi","\\frac","{","t_","{","i","}","}","{","T_","{","i","}","}",")",")","where",":","math",":","`","i","`","is","the","number","of","the","current","annealing","cycle",",",":","math",":","`","t_i","`","is","the","number","of","steps","taken","since","the","last","restart",",","and",":","math",":","`","T_i","`","is","the","total","number","of","steps","within","the",":","math",":","`","i","`","-th","cycle","(","i.e",".","*","length","*","of","the","cycle",")",".","*","Cosine","Annealing","*","is","a","type","of","learning","rate","schedule","that","has","the","effect","of","starting","with","a","large","learning","rate","that","is","relatively","rapidly","decreased","to","a","minimum","value","before","being","increased","rapidly","again",".","Please","refer","to","the","paper","to","learn","more","about","the","details",".","In","addition","to","the","original","schedule",",","this","implementation","also","supports","a","warmup","phase","where","the","learning","rate","is","linearly","increased","for","the","first",":","math",":","`","T_","{","warmup","}","`","training","steps","to","the","base","learning","rate",".","..","note",":",":","This","scheduler","is","not","chainable","."]
docstr: :param optimizer:
      The associated optimizer.
  :param cycle_len:
      The number of steps within the first cycle.
  :param num_warmup_steps:
      The number of warmup steps.
  :param cycle_mul:
      The factor to grow the length of each cycle.
  :param lr_mul:
      The factor to scale the base and final learning rate at the end of
      each cycle.
  :param start_lr:
      The initial warmup learning rate of all parameter groups, or of each
      parameter group respectively.
  :param final_lr:
      The final learning rate of all parameter groups, or of each
      parameter group respectively, at the end of the first cycle.
  :param last_epoch:
      The index of the last epoch.
  :param verbose:
      If ``True``, prints a message to stdout for each update.
docstr_tok: [":","param","optimizer",":","The","associated","optimizer",".",":","param","cycle_len",":","The","number","of","steps","within","the","first","cycle",".",":","param","num_warmup_steps",":","The","number","of","warmup","steps",".",":","param","cycle_mul",":","The","factor","to","grow","the","length","of","each","cycle",".",":","param","lr_mul",":","The","factor","to","scale","the","base","and","final","learning","rate","at","the","end","of","each","cycle",".",":","param","start_lr",":","The","initial","warmup","learning","rate","of","all","parameter","groups",",","or","of","each","parameter","group","respectively",".",":","param","final_lr",":","The","final","learning","rate","of","all","parameter","groups",",","or","of","each","parameter","group","respectively",",","at","the","end","of","the","first","cycle",".",":","param","last_epoch",":","The","index","of","the","last","epoch",".",":","param","verbose",":","If","``","True","``",",","prints","a","message","to","stdout","for","each","update","."]
code: def __init__(
      self,
      optimizer: Optimizer,
      cycle_len: int,
      num_warmup_steps: int,
      *,
      cycle_mul: float = 1.0,
      lr_mul: float = 1.0,
      start_lr: Union[float, Sequence[float]] = 0.0,
      final_lr: Union[float, Sequence[float]] = 0.0,
      last_epoch: int = -1,
      verbose: bool = False,
  ) -> None:
      """
      :param optimizer:
          The associated optimizer.
      :param cycle_len:
          The number of steps within the first cycle.
      :param num_warmup_steps:
          The number of warmup steps.
      :param cycle_mul:
          The factor to grow the length of each cycle.
      :param lr_mul:
          The factor to scale the base and final learning rate at the end of
          each cycle.
      :param start_lr:
          The initial warmup learning rate of all parameter groups, or of each
          parameter group respectively.
      :param final_lr:
          The final learning rate of all parameter groups, or of each
          parameter group respectively, at the end of the first cycle.
      :param last_epoch:
          The index of the last epoch.
      :param verbose:
          If ``True``, prints a message to stdout for each update.
      """
      self.cycle_len = cycle_len
      self.cycle_mul = cycle_mul
      self.num_warmup_steps = num_warmup_steps
      self.lr_mul = lr_mul
  
      self.start_lrs = _get_per_param_group(
          optimizer, "start_lr", start_lr
      )
      self.final_lrs = _get_per_param_group(
          optimizer, "final_lr", final_lr
      )
  
      super().__init__(optimizer, last_epoch, verbose)
  
imports: import math
  import warnings
  from abc import ABC, abstractmethod
  from typing import List, Sequence, Union, final
  from torch.optim import Optimizer
  from torch.optim.lr_scheduler import _LRScheduler
  from typing_extensions import TypeAlias
  from fairseq2.typing import finaloverride
label_desc: Use image node_id 1 to create a new CosineAnnealingLR object from inherited base classes: LRSchedulerBase with example: obj = CosineAnnealingLR(optimizer, cycle_len, num_warmup_steps)