class_docstr: Implements the Proximal Policy Optimization (PPO) algorithm for reinforcement learning.
  
  The algorithm is described in Schulman et al, "Proximal Policy Optimization Algorithms"
  (https://openai-public.s3-us-west-2.amazonaws.com/blog/2017-07/ppo/ppo-arxiv.pdf).
  This class requires the policy to output two quantities: a vector giving the probability of
  taking each action, and an estimate of the value function for the current state.  It
  optimizes both outputs at once using a loss that is the sum of three terms:
  
  1. The policy loss, which seeks to maximize the discounted reward for each action.
  2. The value loss, which tries to make the value estimate match the actual discounted reward
      that was attained at each step.
  3. An entropy term to encourage exploration.
  
  This class only supports environments with discrete action spaces, not continuous ones.  The
  "action" argument passed to the environment is an integer, giving the index of the action to perform.
  
  This class supports Generalized Advantage Estimation as described in Schulman et al., "High-Dimensional
  Continuous Control Using Generalized Advantage Estimation" (https://arxiv.org/abs/1506.02438).
  This is a method of trading off bias and variance in the advantage estimate, which can sometimes
  improve the rate of convergance.  Use the advantage_lambda parameter to adjust the tradeoff.
  
  This class supports Hindsight Experience Replay as described in Andrychowicz et al., "Hindsight
  Experience Replay" (https://arxiv.org/abs/1707.01495).  This is a method that can enormously
  accelerate learning when rewards are very rare.  It requires that the environment state contains
  information about the goal the agent is trying to achieve.  Each time it generates a rollout, it
  processes that rollout twice: once using the actual goal the agent was pursuing while generating
  it, and again using the final state of that rollout as the goal.  This guarantees that half of
  all rollouts processed will be ones that achieved their goals, and hence received a reward.
  
  To use this feature, specify use_hindsight=True to the constructor.  The environment must have
  a method defined as follows:
  
  def apply_hindsight(self, states, actions, goal):
      ...
      return new_states, rewards
  
  The method receives the list of states generated during the rollout, the action taken for each one,
  and a new goal state.  It should generate a new list of states that are identical to the input ones,
  except specifying the new goal.  It should return that list of states, and the rewards that would
  have been received for taking the specified actions from those states.  The output arrays may be
  shorter than the input ones, if the modified rollout would have terminated sooner.
class_docstr_tok: ["Implements","the","Proximal","Policy","Optimization","(","PPO",")","algorithm","for","reinforcement","learning",".","The","algorithm","is","described","in","Schulman","et","al",",","``","Proximal","Policy","Optimization","Algorithms","''","(","https",":","\/\/openai-public.s3-us-west-2.amazonaws.com\/blog\/2017-07\/ppo\/ppo-arxiv.pdf",")",".","This","class","requires","the","policy","to","output","two","quantities",":","a","vector","giving","the","probability","of","taking","each","action",",","and","an","estimate","of","the","value","function","for","the","current","state",".","It","optimizes","both","outputs","at","once","using","a","loss","that","is","the","sum","of","three","terms",":","1",".","The","policy","loss",",","which","seeks","to","maximize","the","discounted","reward","for","each","action",".","2",".","The","value","loss",",","which","tries","to","make","the","value","estimate","match","the","actual","discounted","reward","that","was","attained","at","each","step",".","3",".","An","entropy","term","to","encourage","exploration",".","This","class","only","supports","environments","with","discrete","action","spaces",",","not","continuous","ones",".","The","''","action","''","argument","passed","to","the","environment","is","an","integer",",","giving","the","index","of","the","action","to","perform",".","This","class","supports","Generalized","Advantage","Estimation","as","described","in","Schulman","et","al.",",","``","High-Dimensional","Continuous","Control","Using","Generalized","Advantage","Estimation","''","(","https",":","\/\/arxiv.org\/abs\/1506.02438",")",".","This","is","a","method","of","trading","off","bias","and","variance","in","the","advantage","estimate",",","which","can","sometimes","improve","the","rate","of","convergance",".","Use","the","advantage_lambda","parameter","to","adjust","the","tradeoff",".","This","class","supports","Hindsight","Experience","Replay","as","described","in","Andrychowicz","et","al.",",","``","Hindsight","Experience","Replay","''","(","https",":","\/\/arxiv.org\/abs\/1707.01495",")",".","This","is","a","method","that","can","enormously","accelerate","learning","when","rewards","are","very","rare",".","It","requires","that","the","environment","state","contains","information","about","the","goal","the","agent","is","trying","to","achieve",".","Each","time","it","generates","a","rollout",",","it","processes","that","rollout","twice",":","once","using","the","actual","goal","the","agent","was","pursuing","while","generating","it",",","and","again","using","the","final","state","of","that","rollout","as","the","goal",".","This","guarantees","that","half","of","all","rollouts","processed","will","be","ones","that","achieved","their","goals",",","and","hence","received","a","reward",".","To","use","this","feature",",","specify","use_hindsight=True","to","the","constructor",".","The","environment","must","have","a","method","defined","as","follows",":","def","apply_hindsight","(","self",",","states",",","actions",",","goal",")",":","...","return","new_states",",","rewards","The","method","receives","the","list","of","states","generated","during","the","rollout",",","the","action","taken","for","each","one",",","and","a","new","goal","state",".","It","should","generate","a","new","list","of","states","that","are","identical","to","the","input","ones",",","except","specifying","the","new","goal",".","It","should","return","that","list","of","states",",","and","the","rewards","that","would","have","been","received","for","taking","the","specified","actions","from","those","states",".","The","output","arrays","may","be","shorter","than","the","input","ones",",","if","the","modified","rollout","would","have","terminated","sooner","."]
docstr: Create an object for optimizing a policy.
  
  Parameters
  ----------
  env: Environment
      the Environment to interact with
  policy: Policy
      the Policy to optimize.  It must have outputs with the names 'action_prob'
      and 'value', corresponding to the action probabilities and value estimate
  max_rollout_length: int
      the maximum length of rollouts to generate
  optimization_rollouts: int
      the number of rollouts to generate for each iteration of optimization
  optimization_epochs: int
      the number of epochs of optimization to perform within each iteration
  batch_size: int
      the batch size to use during optimization.  If this is 0, each rollout will be used as a
      separate batch.
  clipping_width: float
      in computing the PPO loss function, the probability ratio is clipped to the range
      (1-clipping_width, 1+clipping_width)
  discount_factor: float
      the discount factor to use when computing rewards
  advantage_lambda: float
      the parameter for trading bias vs. variance in Generalized Advantage Estimation
  value_weight: float
      a scale factor for the value loss term in the loss function
  entropy_weight: float
      a scale factor for the entropy term in the loss function
  optimizer: Optimizer
      the optimizer to use.  If None, a default optimizer is used.
  model_dir: str
      the directory in which the model will be saved.  If None, a temporary directory will be created.
  use_hindsight: bool
      if True, use Hindsight Experience Replay
docstr_tok: ["Create","an","object","for","optimizing","a","policy",".","Parameters","--","--","--","--","--","env",":","Environment","the","Environment","to","interact","with","policy",":","Policy","the","Policy","to","optimize",".","It","must","have","outputs","with","the","names","'action_prob'","and","'value","'",",","corresponding","to","the","action","probabilities","and","value","estimate","max_rollout_length",":","int","the","maximum","length","of","rollouts","to","generate","optimization_rollouts",":","int","the","number","of","rollouts","to","generate","for","each","iteration","of","optimization","optimization_epochs",":","int","the","number","of","epochs","of","optimization","to","perform","within","each","iteration","batch_size",":","int","the","batch","size","to","use","during","optimization",".","If","this","is","0",",","each","rollout","will","be","used","as","a","separate","batch",".","clipping_width",":","float","in","computing","the","PPO","loss","function",",","the","probability","ratio","is","clipped","to","the","range","(","1-clipping_width",",","1+clipping_width",")","discount_factor",":","float","the","discount","factor","to","use","when","computing","rewards","advantage_lambda",":","float","the","parameter","for","trading","bias","vs.","variance","in","Generalized","Advantage","Estimation","value_weight",":","float","a","scale","factor","for","the","value","loss","term","in","the","loss","function","entropy_weight",":","float","a","scale","factor","for","the","entropy","term","in","the","loss","function","optimizer",":","Optimizer","the","optimizer","to","use",".","If","None",",","a","default","optimizer","is","used",".","model_dir",":","str","the","directory","in","which","the","model","will","be","saved",".","If","None",",","a","temporary","directory","will","be","created",".","use_hindsight",":","bool","if","True",",","use","Hindsight","Experience","Replay"]
code: def __init__(
      self,
      env,
      policy,
      max_rollout_length=20,
      optimization_rollouts=8,
      optimization_epochs=4,
      batch_size=64,
      clipping_width=0.2,
      discount_factor=0.99,
      advantage_lambda=0.98,
      value_weight=1.0,
      entropy_weight=0.01,
      optimizer=None,
      model_dir=None,
      use_hindsight=False,
  ):
      """Create an object for optimizing a policy.
  
      Parameters
      ----------
      env: Environment
          the Environment to interact with
      policy: Policy
          the Policy to optimize.  It must have outputs with the names 'action_prob'
          and 'value', corresponding to the action probabilities and value estimate
      max_rollout_length: int
          the maximum length of rollouts to generate
      optimization_rollouts: int
          the number of rollouts to generate for each iteration of optimization
      optimization_epochs: int
          the number of epochs of optimization to perform within each iteration
      batch_size: int
          the batch size to use during optimization.  If this is 0, each rollout will be used as a
          separate batch.
      clipping_width: float
          in computing the PPO loss function, the probability ratio is clipped to the range
          (1-clipping_width, 1+clipping_width)
      discount_factor: float
          the discount factor to use when computing rewards
      advantage_lambda: float
          the parameter for trading bias vs. variance in Generalized Advantage Estimation
      value_weight: float
          a scale factor for the value loss term in the loss function
      entropy_weight: float
          a scale factor for the entropy term in the loss function
      optimizer: Optimizer
          the optimizer to use.  If None, a default optimizer is used.
      model_dir: str
          the directory in which the model will be saved.  If None, a temporary directory will be created.
      use_hindsight: bool
          if True, use Hindsight Experience Replay
      """
      self._env = env
      self._policy = policy
      self.max_rollout_length = max_rollout_length
      self.optimization_rollouts = optimization_rollouts
      self.optimization_epochs = optimization_epochs
      self.batch_size = batch_size
      self.clipping_width = clipping_width
      self.discount_factor = discount_factor
      self.advantage_lambda = advantage_lambda
      self.value_weight = value_weight
      self.entropy_weight = entropy_weight
      self.use_hindsight = use_hindsight
      self._state_is_list = isinstance(
          env.state_shape[0], SequenceCollection
      )
      if optimizer is None:
          self._optimizer = Adam(
              learning_rate=0.001, beta1=0.9, beta2=0.999
          )
      else:
          self._optimizer = optimizer
      output_names = policy.output_names
      self._value_index = output_names.index("value")
      self._action_prob_index = output_names.index("action_prob")
      self._rnn_final_state_indices = [
          i for i, n in enumerate(output_names) if n == "rnn_state"
      ]
      self._rnn_states = policy.rnn_initial_states
      if len(self._rnn_states) > 0 and batch_size != 0:
          raise ValueError(
              "Cannot batch rollouts when the policy contains a recurrent layer.  Set batch_size to 0."
          )
      self._model = self._build_model(model_dir)
      self._checkpoint = tf.train.Checkpoint()
      self._checkpoint.save_counter  # Ensure the variable has been created
      self._checkpoint.listed = self._model.model.trainable_variables
  
imports: import copy
  import time
  from collections.abc import Sequence
  from multiprocessing.dummy import Pool
  import numpy
  import tensorflow
  from deepchem.models import KerasModel
  from deepchem.models.optimizers import Adam
label_desc: Use image node_id 1 to create a new PPO object from inherited base classes: object with example: obj = PPO(env, policy, max_rollout_length, optimization_rollouts, optimization_epochs, batch_size, clipping_width, discount_factor, advantage_lambda, value_weight, entropy_weight, optimizer, model_dir, use_hindsight)