docstr: :param force_cpu:
      Force the neural networks to run on the CPU even if a GPU is available
  :param async_training:
      If set to True, asynchronous training will be used, meaning that each workers will progress in its own
      speed, while not waiting for the rest of the workers to calculate their gradients.
  :param shared_optimizer:
      If set to True, a central optimizer which will be shared with all the workers will be used for applying
      gradients to the network. Otherwise, each worker will have its own optimizer with its own internal
      parameters that will only be affected by the gradients calculated by that worker
  :param scale_down_gradients_by_number_of_workers_for_sync_training:
      If set to True, in synchronous training, the gradients of each worker will be scaled down by the
      number of workers. This essentially means that the gradients applied to the network are the average
      of the gradients over all the workers.
  :param clip_gradients:
      A value that will be used for clipping the gradients of the network. If set to None, no gradient clipping
      will be applied. Otherwise, the gradients will be clipped according to the gradients_clipping_method.
  :param gradients_clipping_method:
      A gradient clipping method, defined by a GradientClippingMethod enum, and that will be used to clip the
      gradients of the network. This will only be used if the clip_gradients value is defined as a value other
      than None.
  :param l2_regularization:
      A L2 regularization weight that will be applied to the network weights while calculating the loss function
  :param learning_rate:
      The learning rate for the network
  :param learning_rate_decay_rate:
      If this value is larger than 0, an exponential decay will be applied to the network learning rate.
      The rate of the decay is defined by this parameter, and the number of training steps the decay will be
      applied is defined by learning_rate_decay_steps. Notice that both parameters should be defined in order
      for this to work correctly.
  :param learning_rate_decay_steps:
      If the learning_rate_decay_rate of the network is larger than 0, an exponential decay will be applied to
      the network learning rate. The number of steps the decay will be applied is defined by this parameter.
      Notice that both this parameter, as well as learning_rate_decay_rate should be defined in order for the
      learning rate decay to work correctly.
  :param input_embedders_parameters:
      A dictionary mapping between input names and input embedders (InputEmbedderParameters) to use for the
      network. Each of the keys is an input name as returned from the environment in the state.
      For example, if the environment returns a state containing 'observation' and 'measurements', then
      the keys for the input embedders dictionary can be either 'observation' to use the observation as input,
      'measurements' to use the measurements as input, or both.
      The embedder type will be automatically selected according to the input type. Vector inputs will
      produce a fully connected embedder, and image inputs will produce a convolutional embedder.
  :param embedding_merger_type:
      The type of embedding merging to use, given by one of the EmbeddingMergerType enum values.
      This will be used to merge the outputs of all the input embedders into a single embbeding.
  :param middleware_parameters:
      The parameters of the middleware to use, given by a MiddlewareParameters object.
      Each network will have only a single middleware embedder which will take the merged embeddings from the
      input embedders and pass them through more neural network layers.
  :param heads_parameters:
      A list of heads for the network given by their corresponding HeadParameters.
      Each network can have one or multiple network heads, where each one will take the output of the middleware
      and make some additional computation on top of it. Additionally, each head calculates a weighted loss value,
      and the loss values from all the heads will be summed later on.
  :param use_separate_networks_per_head:
      A flag that allows using different copies of the input embedders and middleware for each one of the heads.
      Regularly, the heads will have a shared input, but in the case where use_separate_networks_per_head is set
      to True, each one of the heads will get a different input.
  :param optimizer_type:
      A string specifying the optimizer type to use for updating the network. The available optimizers are
      Adam, RMSProp and LBFGS.
  :param optimizer_epsilon:
      An internal optimizer parameter used for Adam and RMSProp.
  :param adam_optimizer_beta1:
      An beta1 internal optimizer parameter used for Adam. It will be used only if Adam was selected as the
      optimizer for the network.
  :param adam_optimizer_beta2:
      An beta2 internal optimizer parameter used for Adam. It will be used only if Adam was selected as the
      optimizer for the network.
  :param rms_prop_optimizer_decay:
      The decay value for the RMSProp optimizer, which will be used only in case the RMSProp optimizer was
      selected for this network.
  :param batch_size:
      The batch size to use when updating the network.
  :param replace_mse_with_huber_loss:
  :param create_target_network:
      If this flag is set to True, an additional copy of the network will be created and initialized with the
      same weights as the online network. It can then be queried, and its weights can be synced from the
      online network at will.
  :param tensorflow_support:
      A flag which specifies if the network is supported by the TensorFlow framework.
  :param softmax_temperature:
      If a softmax is present in the network head output, use this temperature
docstr_tok: [":","param","force_cpu",":","Force","the","neural","networks","to","run","on","the","CPU","even","if","a","GPU","is","available",":","param","async_training",":","If","set","to","True",",","asynchronous","training","will","be","used",",","meaning","that","each","workers","will","progress","in","its","own","speed",",","while","not","waiting","for","the","rest","of","the","workers","to","calculate","their","gradients",".",":","param","shared_optimizer",":","If","set","to","True",",","a","central","optimizer","which","will","be","shared","with","all","the","workers","will","be","used","for","applying","gradients","to","the","network",".","Otherwise",",","each","worker","will","have","its","own","optimizer","with","its","own","internal","parameters","that","will","only","be","affected","by","the","gradients","calculated","by","that","worker",":","param","scale_down_gradients_by_number_of_workers_for_sync_training",":","If","set","to","True",",","in","synchronous","training",",","the","gradients","of","each","worker","will","be","scaled","down","by","the","number","of","workers",".","This","essentially","means","that","the","gradients","applied","to","the","network","are","the","average","of","the","gradients","over","all","the","workers",".",":","param","clip_gradients",":","A","value","that","will","be","used","for","clipping","the","gradients","of","the","network",".","If","set","to","None",",","no","gradient","clipping","will","be","applied",".","Otherwise",",","the","gradients","will","be","clipped","according","to","the","gradients_clipping_method",".",":","param","gradients_clipping_method",":","A","gradient","clipping","method",",","defined","by","a","GradientClippingMethod","enum",",","and","that","will","be","used","to","clip","the","gradients","of","the","network",".","This","will","only","be","used","if","the","clip_gradients","value","is","defined","as","a","value","other","than","None",".",":","param","l2_regularization",":","A","L2","regularization","weight","that","will","be","applied","to","the","network","weights","while","calculating","the","loss","function",":","param","learning_rate",":","The","learning","rate","for","the","network",":","param","learning_rate_decay_rate",":","If","this","value","is","larger","than","0",",","an","exponential","decay","will","be","applied","to","the","network","learning","rate",".","The","rate","of","the","decay","is","defined","by","this","parameter",",","and","the","number","of","training","steps","the","decay","will","be","applied","is","defined","by","learning_rate_decay_steps",".","Notice","that","both","parameters","should","be","defined","in","order","for","this","to","work","correctly",".",":","param","learning_rate_decay_steps",":","If","the","learning_rate_decay_rate","of","the","network","is","larger","than","0",",","an","exponential","decay","will","be","applied","to","the","network","learning","rate",".","The","number","of","steps","the","decay","will","be","applied","is","defined","by","this","parameter",".","Notice","that","both","this","parameter",",","as","well","as","learning_rate_decay_rate","should","be","defined","in","order","for","the","learning","rate","decay","to","work","correctly",".",":","param","input_embedders_parameters",":","A","dictionary","mapping","between","input","names","and","input","embedders","(","InputEmbedderParameters",")","to","use","for","the","network",".","Each","of","the","keys","is","an","input","name","as","returned","from","the","environment","in","the","state",".","For","example",",","if","the","environment","returns","a","state","containing","'observation","'","and","'measurements","'",",","then","the","keys","for","the","input","embedders","dictionary","can","be","either","'observation","'","to","use","the","observation","as","input",",","'measurements","'","to","use","the","measurements","as","input",",","or","both",".","The","embedder","type","will","be","automatically","selected","according","to","the","input","type",".","Vector","inputs","will","produce","a","fully","connected","embedder",",","and","image","inputs","will","produce","a","convolutional","embedder",".",":","param","embedding_merger_type",":","The","type","of","embedding","merging","to","use",",","given","by","one","of","the","EmbeddingMergerType","enum","values",".","This","will","be","used","to","merge","the","outputs","of","all","the","input","embedders","into","a","single","embbeding",".",":","param","middleware_parameters",":","The","parameters","of","the","middleware","to","use",",","given","by","a","MiddlewareParameters","object",".","Each","network","will","have","only","a","single","middleware","embedder","which","will","take","the","merged","embeddings","from","the","input","embedders","and","pass","them","through","more","neural","network","layers",".",":","param","heads_parameters",":","A","list","of","heads","for","the","network","given","by","their","corresponding","HeadParameters",".","Each","network","can","have","one","or","multiple","network","heads",",","where","each","one","will","take","the","output","of","the","middleware","and","make","some","additional","computation","on","top","of","it",".","Additionally",",","each","head","calculates","a","weighted","loss","value",",","and","the","loss","values","from","all","the","heads","will","be","summed","later","on",".",":","param","use_separate_networks_per_head",":","A","flag","that","allows","using","different","copies","of","the","input","embedders","and","middleware","for","each","one","of","the","heads",".","Regularly",",","the","heads","will","have","a","shared","input",",","but","in","the","case","where","use_separate_networks_per_head","is","set","to","True",",","each","one","of","the","heads","will","get","a","different","input",".",":","param","optimizer_type",":","A","string","specifying","the","optimizer","type","to","use","for","updating","the","network",".","The","available","optimizers","are","Adam",",","RMSProp","and","LBFGS",".",":","param","optimizer_epsilon",":","An","internal","optimizer","parameter","used","for","Adam","and","RMSProp",".",":","param","adam_optimizer_beta1",":","An","beta1","internal","optimizer","parameter","used","for","Adam",".","It","will","be","used","only","if","Adam","was","selected","as","the","optimizer","for","the","network",".",":","param","adam_optimizer_beta2",":","An","beta2","internal","optimizer","parameter","used","for","Adam",".","It","will","be","used","only","if","Adam","was","selected","as","the","optimizer","for","the","network",".",":","param","rms_prop_optimizer_decay",":","The","decay","value","for","the","RMSProp","optimizer",",","which","will","be","used","only","in","case","the","RMSProp","optimizer","was","selected","for","this","network",".",":","param","batch_size",":","The","batch","size","to","use","when","updating","the","network",".",":","param","replace_mse_with_huber_loss",":",":","param","create_target_network",":","If","this","flag","is","set","to","True",",","an","additional","copy","of","the","network","will","be","created","and","initialized","with","the","same","weights","as","the","online","network",".","It","can","then","be","queried",",","and","its","weights","can","be","synced","from","the","online","network","at","will",".",":","param","tensorflow_support",":","A","flag","which","specifies","if","the","network","is","supported","by","the","TensorFlow","framework",".",":","param","softmax_temperature",":","If","a","softmax","is","present","in","the","network","head","output",",","use","this","temperature"]
code: def __init__(
      self,
      force_cpu=False,
      async_training=False,
      shared_optimizer=True,
      scale_down_gradients_by_number_of_workers_for_sync_training=True,
      clip_gradients=None,
      gradients_clipping_method=GradientClippingMethod.ClipByGlobalNorm,
      l2_regularization=0,
      learning_rate=0.00025,
      learning_rate_decay_rate=0,
      learning_rate_decay_steps=0,
      input_embedders_parameters={},
      embedding_merger_type=EmbeddingMergerType.Concat,
      middleware_parameters=None,
      heads_parameters=[],
      use_separate_networks_per_head=False,
      optimizer_type="Adam",
      optimizer_epsilon=0.0001,
      adam_optimizer_beta1=0.9,
      adam_optimizer_beta2=0.99,
      rms_prop_optimizer_decay=0.9,
      batch_size=32,
      replace_mse_with_huber_loss=False,
      create_target_network=False,
      tensorflow_support=True,
      softmax_temperature=1,
  ):
      """
      :param force_cpu:
          Force the neural networks to run on the CPU even if a GPU is available
      :param async_training:
          If set to True, asynchronous training will be used, meaning that each workers will progress in its own
          speed, while not waiting for the rest of the workers to calculate their gradients.
      :param shared_optimizer:
          If set to True, a central optimizer which will be shared with all the workers will be used for applying
          gradients to the network. Otherwise, each worker will have its own optimizer with its own internal
          parameters that will only be affected by the gradients calculated by that worker
      :param scale_down_gradients_by_number_of_workers_for_sync_training:
          If set to True, in synchronous training, the gradients of each worker will be scaled down by the
          number of workers. This essentially means that the gradients applied to the network are the average
          of the gradients over all the workers.
      :param clip_gradients:
          A value that will be used for clipping the gradients of the network. If set to None, no gradient clipping
          will be applied. Otherwise, the gradients will be clipped according to the gradients_clipping_method.
      :param gradients_clipping_method:
          A gradient clipping method, defined by a GradientClippingMethod enum, and that will be used to clip the
          gradients of the network. This will only be used if the clip_gradients value is defined as a value other
          than None.
      :param l2_regularization:
          A L2 regularization weight that will be applied to the network weights while calculating the loss function
      :param learning_rate:
          The learning rate for the network
      :param learning_rate_decay_rate:
          If this value is larger than 0, an exponential decay will be applied to the network learning rate.
          The rate of the decay is defined by this parameter, and the number of training steps the decay will be
          applied is defined by learning_rate_decay_steps. Notice that both parameters should be defined in order
          for this to work correctly.
      :param learning_rate_decay_steps:
          If the learning_rate_decay_rate of the network is larger than 0, an exponential decay will be applied to
          the network learning rate. The number of steps the decay will be applied is defined by this parameter.
          Notice that both this parameter, as well as learning_rate_decay_rate should be defined in order for the
          learning rate decay to work correctly.
      :param input_embedders_parameters:
          A dictionary mapping between input names and input embedders (InputEmbedderParameters) to use for the
          network. Each of the keys is an input name as returned from the environment in the state.
          For example, if the environment returns a state containing 'observation' and 'measurements', then
          the keys for the input embedders dictionary can be either 'observation' to use the observation as input,
          'measurements' to use the measurements as input, or both.
          The embedder type will be automatically selected according to the input type. Vector inputs will
          produce a fully connected embedder, and image inputs will produce a convolutional embedder.
      :param embedding_merger_type:
          The type of embedding merging to use, given by one of the EmbeddingMergerType enum values.
          This will be used to merge the outputs of all the input embedders into a single embbeding.
      :param middleware_parameters:
          The parameters of the middleware to use, given by a MiddlewareParameters object.
          Each network will have only a single middleware embedder which will take the merged embeddings from the
          input embedders and pass them through more neural network layers.
      :param heads_parameters:
          A list of heads for the network given by their corresponding HeadParameters.
          Each network can have one or multiple network heads, where each one will take the output of the middleware
          and make some additional computation on top of it. Additionally, each head calculates a weighted loss value,
          and the loss values from all the heads will be summed later on.
      :param use_separate_networks_per_head:
          A flag that allows using different copies of the input embedders and middleware for each one of the heads.
          Regularly, the heads will have a shared input, but in the case where use_separate_networks_per_head is set
          to True, each one of the heads will get a different input.
      :param optimizer_type:
          A string specifying the optimizer type to use for updating the network. The available optimizers are
          Adam, RMSProp and LBFGS.
      :param optimizer_epsilon:
          An internal optimizer parameter used for Adam and RMSProp.
      :param adam_optimizer_beta1:
          An beta1 internal optimizer parameter used for Adam. It will be used only if Adam was selected as the
          optimizer for the network.
      :param adam_optimizer_beta2:
          An beta2 internal optimizer parameter used for Adam. It will be used only if Adam was selected as the
          optimizer for the network.
      :param rms_prop_optimizer_decay:
          The decay value for the RMSProp optimizer, which will be used only in case the RMSProp optimizer was
          selected for this network.
      :param batch_size:
          The batch size to use when updating the network.
      :param replace_mse_with_huber_loss:
      :param create_target_network:
          If this flag is set to True, an additional copy of the network will be created and initialized with the
          same weights as the online network. It can then be queried, and its weights can be synced from the
          online network at will.
      :param tensorflow_support:
          A flag which specifies if the network is supported by the TensorFlow framework.
      :param softmax_temperature:
          If a softmax is present in the network head output, use this temperature
      """
      super().__init__()
      self.framework = Frameworks.tensorflow
      self.sess = None
  
      # hardware parameters
      self.force_cpu = force_cpu
  
      # distributed training options
      self.async_training = async_training
      self.shared_optimizer = shared_optimizer
      self.scale_down_gradients_by_number_of_workers_for_sync_training = (
          scale_down_gradients_by_number_of_workers_for_sync_training
      )
  
      # regularization
      self.clip_gradients = clip_gradients
      self.gradients_clipping_method = gradients_clipping_method
      self.l2_regularization = l2_regularization
  
      # learning rate
      self.learning_rate = learning_rate
      self.learning_rate_decay_rate = learning_rate_decay_rate
      self.learning_rate_decay_steps = learning_rate_decay_steps
  
      # structure
      self.input_embedders_parameters = input_embedders_parameters
      self.embedding_merger_type = embedding_merger_type
      self.middleware_parameters = middleware_parameters
      self.heads_parameters = heads_parameters
      self.use_separate_networks_per_head = (
          use_separate_networks_per_head
      )
      self.optimizer_type = optimizer_type
      self.replace_mse_with_huber_loss = replace_mse_with_huber_loss
      self.create_target_network = create_target_network
  
      # Framework support
      self.tensorflow_support = tensorflow_support
  
      # Hyper-Parameter values
      self.optimizer_epsilon = optimizer_epsilon
      self.adam_optimizer_beta1 = adam_optimizer_beta1
      self.adam_optimizer_beta2 = adam_optimizer_beta2
      self.rms_prop_optimizer_decay = rms_prop_optimizer_decay
      self.batch_size = batch_size
      self.softmax_temperature = softmax_temperature
  
imports: import inspect
  import json
  import os
  import sys
  import types
  from collections import OrderedDict
  from enum import Enum
  from typing import Dict, List, Union
  from rl_coach.core_types import TrainingSteps, EnvironmentSteps, GradientClippingMethod, RunPhase, SelectedPhaseOnlyDumpFilter, MaxDumpFilter
  from rl_coach.filters.filter import NoInputFilter
  from rl_coach.logger import screen
label_desc: Use image node_id 1 to create a new NetworkParameters object from inherited base classes: Parameters with example: obj = NetworkParameters(force_cpu, async_training, shared_optimizer, scale_down_gradients_by_number_of_workers_for_sync_training, clip_gradients, gradients_clipping_method, l2_regularization, learning_rate, learning_rate_decay_rate, learning_rate_decay_steps, input_embedders_parameters, embedding_merger_type, middleware_parameters, heads_parameters, use_separate_networks_per_head, optimizer_type, optimizer_epsilon, adam_optimizer_beta1, adam_optimizer_beta2, rms_prop_optimizer_decay, batch_size, replace_mse_with_huber_loss, create_target_network, tensorflow_support, softmax_temperature)