class_docstr: Implements the Advantage Actor-Critic (A2C) algorithm for reinforcement learning.
  
  The algorithm is described in Mnih et al, "Asynchronous Methods for Deep Reinforcement Learning"
  (https://arxiv.org/abs/1602.01783).  This class supports environments with both discrete and
  continuous action spaces.  For discrete action spaces, the "action" argument passed to the
  environment is an integer giving the index of the action to perform.  The policy must output
  a vector called "action_prob" giving the probability of taking each action.  For continuous
  action spaces, the action is an array where each element is chosen independently from a
  normal distribution.  The policy must output two arrays of the same shape: "action_mean"
  gives the mean value for each element, and "action_std" gives the standard deviation for
  each element.  In either case, the policy must also output a scalar called "value" which
  is an estimate of the value function for the current state.
  
  The algorithm optimizes all outputs at once using a loss that is the sum of three terms:
  
  1. The policy loss, which seeks to maximize the discounted reward for each action.
  2. The value loss, which tries to make the value estimate match the actual discounted reward
      that was attained at each step.
  3. An entropy term to encourage exploration.
  
  This class supports Generalized Advantage Estimation as described in Schulman et al., "High-Dimensional
  Continuous Control Using Generalized Advantage Estimation" (https://arxiv.org/abs/1506.02438).
  This is a method of trading off bias and variance in the advantage estimate, which can sometimes
  improve the rate of convergance.  Use the advantage_lambda parameter to adjust the tradeoff.
  
  This class supports Hindsight Experience Replay as described in Andrychowicz et al., "Hindsight
  Experience Replay" (https://arxiv.org/abs/1707.01495).  This is a method that can enormously
  accelerate learning when rewards are very rare.  It requires that the environment state contains
  information about the goal the agent is trying to achieve.  Each time it generates a rollout, it
  processes that rollout twice: once using the actual goal the agent was pursuing while generating
  it, and again using the final state of that rollout as the goal.  This guarantees that half of
  all rollouts processed will be ones that achieved their goals, and hence received a reward.
  
  To use this feature, specify use_hindsight=True to the constructor.  The environment must have
  a method defined as follows:
  
  def apply_hindsight(self, states, actions, goal):
      ...
      return new_states, rewards
  
  The method receives the list of states generated during the rollout, the action taken for each one,
  and a new goal state.  It should generate a new list of states that are identical to the input ones,
  except specifying the new goal.  It should return that list of states, and the rewards that would
  have been received for taking the specified actions from those states.  The output arrays may be
  shorter than the input ones, if the modified rollout would have terminated sooner.
  
  
  Note
  ----
  Using this class on continuous action spaces requires that `tensorflow_probability` be installed.
class_docstr_tok: ["Implements","the","Advantage","Actor-Critic","(","A2C",")","algorithm","for","reinforcement","learning",".","The","algorithm","is","described","in","Mnih","et","al",",","``","Asynchronous","Methods","for","Deep","Reinforcement","Learning","''","(","https",":","\/\/arxiv.org\/abs\/1602.01783",")",".","This","class","supports","environments","with","both","discrete","and","continuous","action","spaces",".","For","discrete","action","spaces",",","the","``","action","''","argument","passed","to","the","environment","is","an","integer","giving","the","index","of","the","action","to","perform",".","The","policy","must","output","a","vector","called","``","action_prob","''","giving","the","probability","of","taking","each","action",".","For","continuous","action","spaces",",","the","action","is","an","array","where","each","element","is","chosen","independently","from","a","normal","distribution",".","The","policy","must","output","two","arrays","of","the","same","shape",":","``","action_mean","''","gives","the","mean","value","for","each","element",",","and","``","action_std","''","gives","the","standard","deviation","for","each","element",".","In","either","case",",","the","policy","must","also","output","a","scalar","called","``","value","''","which","is","an","estimate","of","the","value","function","for","the","current","state",".","The","algorithm","optimizes","all","outputs","at","once","using","a","loss","that","is","the","sum","of","three","terms",":","1",".","The","policy","loss",",","which","seeks","to","maximize","the","discounted","reward","for","each","action",".","2",".","The","value","loss",",","which","tries","to","make","the","value","estimate","match","the","actual","discounted","reward","that","was","attained","at","each","step",".","3",".","An","entropy","term","to","encourage","exploration",".","This","class","supports","Generalized","Advantage","Estimation","as","described","in","Schulman","et","al.",",","``","High-Dimensional","Continuous","Control","Using","Generalized","Advantage","Estimation","''","(","https",":","\/\/arxiv.org\/abs\/1506.02438",")",".","This","is","a","method","of","trading","off","bias","and","variance","in","the","advantage","estimate",",","which","can","sometimes","improve","the","rate","of","convergance",".","Use","the","advantage_lambda","parameter","to","adjust","the","tradeoff",".","This","class","supports","Hindsight","Experience","Replay","as","described","in","Andrychowicz","et","al.",",","``","Hindsight","Experience","Replay","''","(","https",":","\/\/arxiv.org\/abs\/1707.01495",")",".","This","is","a","method","that","can","enormously","accelerate","learning","when","rewards","are","very","rare",".","It","requires","that","the","environment","state","contains","information","about","the","goal","the","agent","is","trying","to","achieve",".","Each","time","it","generates","a","rollout",",","it","processes","that","rollout","twice",":","once","using","the","actual","goal","the","agent","was","pursuing","while","generating","it",",","and","again","using","the","final","state","of","that","rollout","as","the","goal",".","This","guarantees","that","half","of","all","rollouts","processed","will","be","ones","that","achieved","their","goals",",","and","hence","received","a","reward",".","To","use","this","feature",",","specify","use_hindsight=True","to","the","constructor",".","The","environment","must","have","a","method","defined","as","follows",":","def","apply_hindsight","(","self",",","states",",","actions",",","goal",")",":","...","return","new_states",",","rewards","The","method","receives","the","list","of","states","generated","during","the","rollout",",","the","action","taken","for","each","one",",","and","a","new","goal","state",".","It","should","generate","a","new","list","of","states","that","are","identical","to","the","input","ones",",","except","specifying","the","new","goal",".","It","should","return","that","list","of","states",",","and","the","rewards","that","would","have","been","received","for","taking","the","specified","actions","from","those","states",".","The","output","arrays","may","be","shorter","than","the","input","ones",",","if","the","modified","rollout","would","have","terminated","sooner",".","Note","--","--","Using","this","class","on","continuous","action","spaces","requires","that","`","tensorflow_probability","`","be","installed","."]
docstr: Generate a rollout.
docstr_tok: ["Generate","a","rollout","."]
code: def _create_rollout(self, rnn_states):
      """Generate a rollout."""
      states = []
      actions = []
      rewards = []
      values = []
  
      # Generate the rollout.
  
      for i in range(self.max_rollout_length):
          if self._env.terminated:
              break
          state = self._env.state
          states.append(state)
          results = self._compute_model(
              self._create_model_inputs(state, rnn_states)
          )
          results = [r.numpy() for r in results]
          value = results[self._value_index]
          rnn_states = [
              np.squeeze(results[i], 0)
              for i in self._rnn_final_state_indices
          ]
          action = self._select_action_from_outputs(results, False)
          actions.append(action)
          values.append(float(value))
          rewards.append(self._env.step(action))
  
      # Compute an estimate of the reward for the rest of the episode.
  
      if not self._env.terminated:
          results = self._compute_model(
              self._create_model_inputs(self._env.state, rnn_states)
          )
          final_value = (
              self.discount_factor
              * results[self._value_index].numpy()[0]
          )
      else:
          final_value = 0.0
      values.append(final_value)
      if self._env.terminated:
          self._env.reset()
          rnn_states = self._policy.rnn_initial_states
      return (
          states,
          actions,
          np.array(rewards, dtype=np.float32),
          np.array(values, dtype=np.float32),
          rnn_states,
      )
  
imports: import time
  from collections.abc import Sequence
  import numpy
  import tensorflow
  from deepchem.models import KerasModel
  from deepchem.models.optimizers import Adam
label_desc: Use image node_id 10 for calling the A2C obj's underlying member method code with example usage: obj._create_rollout(rnn_states) and returns: states,  actions,  np,  np,  rnn_states