class_docstr: MegatronGPTInfusedAdapterModel is a model that combines a base model (GPTModel) with a "Infused Adapter that can Inhibiting and Amplify Inner Activations", known as IA3.
  This class supports the addition of IA3 into a transformer based LM as described in Liu et al. (https://arxiv.org/pdf/2205.05638.pdf)
  
  Three adapter's are inserted into each Transformer layer in the base GPT Model. Each adapter is basically a vector that simply scales the key, value or ffn hidden representations.
  
  It is assumed that these set of adapters will then be trained for a specific task.
  Once trained, the adapter weights will be saved and can be re-loaded 
  and infused into the same GPT Model for inference. 
class_docstr_tok: ["MegatronGPTInfusedAdapterModel","is","a","model","that","combines","a","base","model","(","GPTModel",")","with","a","``","Infused","Adapter","that","can","Inhibiting","and","Amplify","Inner","Activations","''",",","known","as","IA3",".","This","class","supports","the","addition","of","IA3","into","a","transformer","based","LM","as","described","in","Liu","et","al",".","(","https",":","\/\/arxiv.org\/pdf\/2205.05638.pdf",")","Three","adapter","'s","are","inserted","into","each","Transformer","layer","in","the","base","GPT","Model",".","Each","adapter","is","basically","a","vector","that","simply","scales","the","key",",","value","or","ffn","hidden","representations",".","It","is","assumed","that","these","set","of","adapters","will","then","be","trained","for","a","specific","task",".","Once","trained",",","the","adapter","weights","will","be","saved","and","can","be","re-loaded","and","infused","into","the","same","GPT","Model","for","inference","."]
code: def __init__(self, cfg: DictConfig, trainer: Trainer):
      super().__init__(cfg, trainer)
      self.adapter_name_keys = [
          AdapterName.KEY_INFUSED,
          AdapterName.VALUE_INFUSED,
          AdapterName.MLP_INFUSED,
      ]
      for _, layer in self.frozen_model.named_modules():
          if hasattr(layer, "activations_checkpoint_method"):
              layer.activations_checkpoint_method = None  # (@adithyare) adapter learning does not support activations checkpointing atm.
  
      logging.info(
          f"Before adding adapters:\n{self.frozen_model.summarize()}"
      )
  
      self.frozen_model.freeze()
      for _, module in self.frozen_model.named_modules():
          if isinstance(module, adapter_mixins.AdapterModuleMixin):
              for adapter_key in self.adapter_name_keys:
                  if adapter_key == AdapterName.MLP_INFUSED:
                      cfg = MLPInfusedAdapterConfig(
                          in_features=self.frozen_model_cfg.ffn_hidden_size
                          // self.frozen_model_cfg.tensor_model_parallel_size
                      )
                  elif adapter_key in [
                      AdapterName.KEY_INFUSED,
                      AdapterName.VALUE_INFUSED,
                  ]:
                      cfg = InfusedAdapterConfig(
                          in_features=self.frozen_model_cfg.hidden_size
                          // self.frozen_model_cfg.tensor_model_parallel_size
                      )
                  else:
                      raise ValueError(
                          f"Adapter Key {adapter_key} is unknown."
                      )
                  if (
                      model_utils.import_class_by_path(cfg._target_)
                      in module.get_accepted_adapter_types()
                  ):
                      module.add_adapter(name=adapter_key, cfg=cfg)
  
      logging.info(
          f"After adding adapters:\n{self.frozen_model.summarize()}"
      )
  
imports: import itertools
  import os
  import torch
  from omegaconf.dictconfig import DictConfig
  from pytorch_lightning.trainer.trainer import Trainer
  from nemo.collections.common.parts.adapter_modules import LinearAdapterConfig
  from nemo.collections.nlp.models.language_modeling.megatron_gpt_model import MegatronGPTModel
  from nemo.collections.nlp.models.language_modeling.megatron_gpt_prompt_learning_model import MegatronGPTPromptLearningModel
  from nemo.collections.nlp.modules.common import VirtualPromptStyle
  from nemo.collections.nlp.modules.common.megatron.adapters.parallel_adapters import AdapterName, InfusedAdapterConfig, MLPInfusedAdapterConfig, ParallelLinearAdapterConfig
  from nemo.collections.nlp.modules.common.megatron.utils import average_losses_across_data_parallel_group
  from nemo.collections.nlp.parts.nlp_overrides import NLPSaveRestoreConnector
  from nemo.collections.nlp.parts.utils_funcs import get_last_rank
  from nemo.core.classes.mixins import adapter_mixins
  from nemo.utils import logging, model_utils
label_desc: Use image node_id 1 to create a new MegatronGPTInfusedAdapterModel object from inherited base classes: MegatronGPTBaseAdapterModel with example: obj = MegatronGPTInfusedAdapterModel(cfg, trainer)