class_docstr: Base class for RNN-with-attention speech decoders.
  
  The decoder takes encoder_outputs, a NestedMap generated by the encoder, as
  input. The NestedMap is expected to contain the following fields:
  
    - 'encoded': encoded features
    - 'padding': padding for encoded features
  
  The decoder operates in a 'step-by-step' fashion. The model encapsulates all
  information which should persist from one step to the next in the
  DecoderStepState NestedMap, which provides a 'misc_states' NestedMap which can
  store arbitrary information required by the specific decoder sub-class.
  
  A 'step' in training consists of the following sequence of steps which compute
  the outputs from the decoder given the current input target (and the state of
  the model after making the previous predictions):
  
  1. Compute the input target at the current time step::
  
      cur_target_info = self.TargetsToBeFedAtCurrentDecodeStep(...)
  
  2. Update state and compute outputs by running SingleDecodeStep::
  
      step_outs, new_state = self.SingleDecodeStep(...)
  
  3. Update state based on the logits computed at this step::
  
      new_state = self.PostStepDecoderStateUpdate(old_state, logits)
  
  4. Display summaries based on the accumulated information across all steps::
  
      self.AddAdditionalDecoderSummaries(seq_out_tas)
  
  Sub-classes can customize behavior by implementing the following functions,
  which will modify the behavior of the decoder:
  
  For beam search decoder:
    - _InitBeamSearchStateCallback
    - _PreBeamSearchStepCallback
    - _PostBeamSearchStepCallback
  For EMBR training:
    - ComputeHypsWithBeamSearch
  
  - MiscZeroState: NestedMap which represents the initial state for the
    'misc_states' in the DecoderStepState. The default implementation returns
    an empty NestedMap.
  
  - SingleDecodeStep: This corresponds to the computation which happens in each
    step of the model. The function should return the outputs of the decoder
    as well as the updated state.
  
  - PostStepDecoderStateUpdate: A function which updates the DecoderStepState
    after the output logits from the decoder have been computed. By default,
    this returns the DecoderStepState unchanged.
  
  - TargetsToBeFedAtCurrentDecodeStep: Returns a TargetInfo namedtuple, which
    represents information about the targets which should be input at the
    current step, as well as the output label which should be predicted.
    The default implementation uses the values in the batched 'targets'
    provided by the InputGenerator.
  
  - AddAdditionalDecoderSummaries: A function which can be used to add any
    decoder specific information as part of the summaries displayed during
    training. By default this is a no-op.
  
  - CreateTargetInfoMisc: A function which can be used to store arbitrary
    information as required by a sub-classes in the target info arrays used
    to determine the current label at each step during training. By default,
    this creates an empty NestedMap.
  
  A few other functions that control how the decoder initializes and computes
  attention during the initial step, and during each step can also be
  modified, if need be:
  
  - _GetAttenContextDim: The dimensionality of the attention context vector.
  - _CreateAtten: Controls how the attention module is configured. Most
    subclasses will not have to change this unless it changes how attention
    works.
  - BaseZeroState: Returns initial state of RNNs, and attention.
  - _InitAttention: Initializes Tensors used by the attention module.
  - _GetInitialSeqStateTensorArrays: Get intitial tensor arrays for
    ComputePredictionsDynamic.
  - _GetNewAttenProbs: Update atten probs for a timestep and return the
    updated tensor array.
class_docstr_tok: ["Base","class","for","RNN-with-attention","speech","decoders",".","The","decoder","takes","encoder_outputs",",","a","NestedMap","generated","by","the","encoder",",","as","input",".","The","NestedMap","is","expected","to","contain","the","following","fields",":","-","'encoded","'",":","encoded","features","-","'padding","'",":","padding","for","encoded","features","The","decoder","operates","in","a","'step-by-step","'","fashion",".","The","model","encapsulates","all","information","which","should","persist","from","one","step","to","the","next","in","the","DecoderStepState","NestedMap",",","which","provides","a","'misc_states","'","NestedMap","which","can","store","arbitrary","information","required","by","the","specific","decoder","sub-class",".","A","'step","'","in","training","consists","of","the","following","sequence","of","steps","which","compute","the","outputs","from","the","decoder","given","the","current","input","target","(","and","the","state","of","the","model","after","making","the","previous","predictions",")",":","1",".","Compute","the","input","target","at","the","current","time","step",":",":","cur_target_info","=","self.TargetsToBeFedAtCurrentDecodeStep","(","...",")","2",".","Update","state","and","compute","outputs","by","running","SingleDecodeStep",":",":","step_outs",",","new_state","=","self.SingleDecodeStep","(","...",")","3",".","Update","state","based","on","the","logits","computed","at","this","step",":",":","new_state","=","self.PostStepDecoderStateUpdate","(","old_state",",","logits",")","4",".","Display","summaries","based","on","the","accumulated","information","across","all","steps",":",":","self.AddAdditionalDecoderSummaries","(","seq_out_tas",")","Sub-classes","can","customize","behavior","by","implementing","the","following","functions",",","which","will","modify","the","behavior","of","the","decoder",":","For","beam","search","decoder",":","-","_InitBeamSearchStateCallback","-","_PreBeamSearchStepCallback","-","_PostBeamSearchStepCallback","For","EMBR","training",":","-","ComputeHypsWithBeamSearch","-","MiscZeroState",":","NestedMap","which","represents","the","initial","state","for","the","'misc_states","'","in","the","DecoderStepState",".","The","default","implementation","returns","an","empty","NestedMap",".","-","SingleDecodeStep",":","This","corresponds","to","the","computation","which","happens","in","each","step","of","the","model",".","The","function","should","return","the","outputs","of","the","decoder","as","well","as","the","updated","state",".","-","PostStepDecoderStateUpdate",":","A","function","which","updates","the","DecoderStepState","after","the","output","logits","from","the","decoder","have","been","computed",".","By","default",",","this","returns","the","DecoderStepState","unchanged",".","-","TargetsToBeFedAtCurrentDecodeStep",":","Returns","a","TargetInfo","namedtuple",",","which","represents","information","about","the","targets","which","should","be","input","at","the","current","step",",","as","well","as","the","output","label","which","should","be","predicted",".","The","default","implementation","uses","the","values","in","the","batched","'targets'","provided","by","the","InputGenerator",".","-","AddAdditionalDecoderSummaries",":","A","function","which","can","be","used","to","add","any","decoder","specific","information","as","part","of","the","summaries","displayed","during","training",".","By","default","this","is","a","no-op",".","-","CreateTargetInfoMisc",":","A","function","which","can","be","used","to","store","arbitrary","information","as","required","by","a","sub-classes","in","the","target","info","arrays","used","to","determine","the","current","label","at","each","step","during","training",".","By","default",",","this","creates","an","empty","NestedMap",".","A","few","other","functions","that","control","how","the","decoder","initializes","and","computes","attention","during","the","initial","step",",","and","during","each","step","can","also","be","modified",",","if","need","be",":","-","_GetAttenContextDim",":","The","dimensionality","of","the","attention","context","vector",".","-","_CreateAtten",":","Controls","how","the","attention","module","is","configured",".","Most","subclasses","will","not","have","to","change","this","unless","it","changes","how","attention","works",".","-","BaseZeroState",":","Returns","initial","state","of","RNNs",",","and","attention",".","-","_InitAttention",":","Initializes","Tensors","used","by","the","attention","module",".","-","_GetInitialSeqStateTensorArrays",":","Get","intitial","tensor","arrays","for","ComputePredictionsDynamic",".","-","_GetNewAttenProbs",":","Update","atten","probs","for","a","timestep","and","return","the","updated","tensor","array","."]
docstr: Adds summary about decoder activations.
  
  For each of the args, a TensorArray can also be a Tensor representing
  the stacked array.
  
  Args:
    encoder_outputs: a NestedMap computed by encoder.
    targets: a NestedMap, usually input_batch.tgt.
    atten_probs: a TensorArray of max_target_length elements, each of shape
      [batch, max_source_length].
    rnn_outs: a list of TensorArray, one for each RNN layer. Each
      TensorArray has max_target_length elements, each of shape [batch,
      rnn_output_dim].
    softmax_input: a Tensor of shape [batch, max_target_length, vocab_size].
    additional_atten_probs: an optional list of (name, TensorArray) to display
      along with atten_probs.
    target_alignments: an optional Tensor of shape [batch, max_target_length]
      where every value is an int32 in the range of [1, max_source_length],
      representing number of source frames by which a target label should be
      emitted.
  
  Returns:
    A finalized figure.
docstr_tok: ["Adds","summary","about","decoder","activations",".","For","each","of","the","args",",","a","TensorArray","can","also","be","a","Tensor","representing","the","stacked","array",".","Args",":","encoder_outputs",":","a","NestedMap","computed","by","encoder",".","targets",":","a","NestedMap",",","usually","input_batch.tgt",".","atten_probs",":","a","TensorArray","of","max_target_length","elements",",","each","of","shape","[","batch",",","max_source_length","]",".","rnn_outs",":","a","list","of","TensorArray",",","one","for","each","RNN","layer",".","Each","TensorArray","has","max_target_length","elements",",","each","of","shape","[","batch",",","rnn_output_dim","]",".","softmax_input",":","a","Tensor","of","shape","[","batch",",","max_target_length",",","vocab_size","]",".","additional_atten_probs",":","an","optional","list","of","(","name",",","TensorArray",")","to","display","along","with","atten_probs",".","target_alignments",":","an","optional","Tensor","of","shape","[","batch",",","max_target_length","]","where","every","value","is","an","int32","in","the","range","of","[","1",",","max_source_length","]",",","representing","number","of","source","frames","by","which","a","target","label","should","be","emitted",".","Returns",":","A","finalized","figure","."]
code: def _AddDecoderActivationsSummary(
      self,
      encoder_outputs,
      targets,
      atten_probs,
      rnn_outs,
      softmax_input,
      additional_atten_probs=None,
      target_alignments=None,
  ):
      """Adds summary about decoder activations.
  
      For each of the args, a TensorArray can also be a Tensor representing
      the stacked array.
  
      Args:
        encoder_outputs: a NestedMap computed by encoder.
        targets: a NestedMap, usually input_batch.tgt.
        atten_probs: a TensorArray of max_target_length elements, each of shape
          [batch, max_source_length].
        rnn_outs: a list of TensorArray, one for each RNN layer. Each
          TensorArray has max_target_length elements, each of shape [batch,
          rnn_output_dim].
        softmax_input: a Tensor of shape [batch, max_target_length, vocab_size].
        additional_atten_probs: an optional list of (name, TensorArray) to display
          along with atten_probs.
        target_alignments: an optional Tensor of shape [batch, max_target_length]
          where every value is an int32 in the range of [1, max_source_length],
          representing number of source frames by which a target label should be
          emitted.
  
      Returns:
        A finalized figure.
      """
      source_encs = encoder_outputs.encoded
      source_paddings = encoder_outputs.padding
      if not self.cluster.add_summary:
          return
  
      def _ToTensor(t):
          return t.stack() if isinstance(t, tf.TensorArray) else t
  
      atten_probs = _ToTensor(atten_probs)
      rnn_outs = [_ToTensor(ta) for ta in rnn_outs]
      if additional_atten_probs:
          additional_atten_probs = [
              (name, _ToTensor(ta))
              for name, ta in additional_atten_probs
          ]
  
      num_cols = 2 + len(rnn_outs)
      fig = plot.MatplotlibFigureSummary(
          "decoder_example",
          figsize=(2.3 * (3 + num_cols - 1), 6),
          max_outputs=1,
          subplot_grid_shape=(2, num_cols),
          gridspec_kwargs=dict(
              width_ratios=[3] + [1] * (num_cols - 1),
              height_ratios=(4, 1),
          ),
      )
  
      # Attention needs a custom plot_func to allow for clean y-axis label for
      # very long transcripts
      def PlotAttention(fig, axes, transcript, atten_probs, title):
          plot.AddImage(fig, axes, atten_probs, title=title)
          axes.set_ylabel(
              plot.ToUnicode(transcript + "\nOutput token"),
              size="x-small",
              wrap=True,
              fontproperties=self._font_properties,
          )
  
      index = 0
      if "transcripts" not in targets:
          return
      transcript = targets.transcripts[: index + 1]
  
      srclen = tf.cast(
          tf.round(tf.reduce_sum(1 - source_paddings[:, index])),
          tf.int32,
      )
      tgtlen = tf.cast(
          tf.round(tf.reduce_sum(1 - targets.paddings[index, :])),
          tf.int32,
      )
  
      def PlotAttentionForOneExample(
          atten_probs, target_fig, title, alignments=None
      ):
          """Plots attention for one example."""
          tf.logging.info(
              "Plotting attention for %s: %s %s",
              title,
              atten_probs.shape,
              alignments,
          )
          atten_probs = atten_probs[:tgtlen, index, :srclen]
          if alignments is not None:
              # [tgtlen].
              alignment_positions = alignments[index, :tgtlen] - 1
              # [tgtlen, srclen].
              alignment_probs = tf.one_hot(
                  alignment_positions, depth=srclen, axis=-1
              )
  
              # The summary image will use red bars to represent target label
              # alignments and purple shades for attention probabilities.
              atten_probs = 1 - tf.stack(
                  [
                      atten_probs,
                      # Overlay atten_probs and alignment_probs on the green channel
                      # so that colors are visible on a white background.
                      tf.minimum(atten_probs + alignment_probs, 1.0),
                      alignment_probs,
                  ],
                  axis=-1,
              )
          probs = tf.expand_dims(atten_probs, 0)
          target_fig.AddSubplot(
              [transcript, probs], PlotAttention, title=title
          )
  
      PlotAttentionForOneExample(atten_probs, fig, title="atten_probs")
      # rnn_outs and softmax_input have transposed shapes of [tgtlen, dim]
      # compared to source_encs [dim, srclen].
      for i in range(len(rnn_outs)):
          rnn_out = tf.expand_dims(rnn_outs[i][:tgtlen, index, :], 0)
          fig.AddSubplot([rnn_out], title="rnn_outs/%d" % i)
      fig.AddSubplot(
          [softmax_input[: index + 1, :tgtlen, :]],
          title="softmax_input",
      )
      source_encs = tf.expand_dims(
          tf.transpose(source_encs[:srclen, index, :]), 0
      )
      fig.AddSubplot(
          [source_encs], title="source_encs", xlabel="Encoder frame"
      )
      finalized_fig = fig.Finalize()
  
      if additional_atten_probs:
          all_atten_probs = [
              ("atten_probs", atten_probs)
          ] + additional_atten_probs
          num_atten_images = len(all_atten_probs)
          atten_fig = plot.MatplotlibFigureSummary(
              "decoder_attention",
              figsize=(6, 3 * num_atten_images),
              max_outputs=1,
          )
          for key, probs in all_atten_probs:
              PlotAttentionForOneExample(
                  probs,
                  atten_fig,
                  title=key,
                  alignments=target_alignments,
              )
          atten_fig.Finalize()
      return finalized_fig
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import math
  import lingvo.compat
  from lingvo.core import attention
  from lingvo.core import base_decoder
  from lingvo.core import base_layer
  from lingvo.core import cluster_factory
  from lingvo.core import layers
  from lingvo.core import plot
  from lingvo.core import py_utils
  from lingvo.core import recurrent
  from lingvo.core import rnn_cell
  from lingvo.core import summary_utils
  from lingvo.core import symbolic
  from lingvo.tasks.asr import contextualizer_base
  from lingvo.tasks.asr import decoder_utils
  from lingvo.tasks.asr import fusion
  from matplotlib import font_manager
  import six
  from six.moves import range
label_desc: Use image node_id 11 for calling the AsrDecoderBase obj's underlying member method code with example usage: obj._AddDecoderActivationsSummary(encoder_outputs, targets, atten_probs, rnn_outs, softmax_input, additional_atten_probs, target_alignments) and returns: finalized_fig, unknown