class_docstr: A `Metric` takes the results of execution and produces `Stat`s for a
  scenario.
  
  Note: `Metric` actually right now is a bit of misnomer because it produces many
  `Stat`s, that might be distinct but are computed together.  Eventually we
  might move to a world where there is one (or very few metrics that are domain-independent).
class_docstr_tok: ["A","`","Metric","`","takes","the","results","of","execution","and","produces","`","Stat","`","s","for","a","scenario",".","Note",":","`","Metric","`","actually","right","now","is","a","bit","of","misnomer","because","it","produces","many","`","Stat","`","s",",","that","might","be","distinct","but","are","computed","together",".","Eventually","we","might","move","to","a","world","where","there","is","one","(","or","very","few","metrics","that","are","domain-independent",")","."]
docstr: For each instance, we compute the worst case perfomance between each perturbation and the non-perturbed input
  (perturbation=None). This allows us to reason about the invariances of a model as opposed to just looking
  at its performance on perturbed inputs. We also compute the worst case performance across all robustness-related
  and fairness-related perturbations (including the original input in both).
  
  For each such worst-case metric, we record a `before_` metric that aggregates the performance on the
  non-perturbed version of the corresponding inputs.
  
  We return the aggregate metrics across instances. Note that none of these metrics make a lot of sense if the
  original, un-perturbed version of an Instance is not included in a scenario (i.e., we want
  `include_original=True`).
docstr_tok: ["For","each","instance",",","we","compute","the","worst","case","perfomance","between","each","perturbation","and","the","non-perturbed","input","(","perturbation=None",")",".","This","allows","us","to","reason","about","the","invariances","of","a","model","as","opposed","to","just","looking","at","its","performance","on","perturbed","inputs",".","We","also","compute","the","worst","case","performance","across","all","robustness-related","and","fairness-related","perturbations","(","including","the","original","input","in","both",")",".","For","each","such","worst-case","metric",",","we","record","a","`","before_","`","metric","that","aggregates","the","performance","on","the","non-perturbed","version","of","the","corresponding","inputs",".","We","return","the","aggregate","metrics","across","instances",".","Note","that","none","of","these","metrics","make","a","lot","of","sense","if","the","original",",","un-perturbed","version","of","an","Instance","is","not","included","in","a","scenario","(","i.e.",",","we","want","`","include_original=True","`",")","."]
code: def compute_worst_case_metrics(
      self, per_instance_stats: Dict[Instance, List[Stat]]
  ) -> List[Stat]:
      """
      For each instance, we compute the worst case perfomance between each perturbation and the non-perturbed input
      (perturbation=None). This allows us to reason about the invariances of a model as opposed to just looking
      at its performance on perturbed inputs. We also compute the worst case performance across all robustness-related
      and fairness-related perturbations (including the original input in both).
  
      For each such worst-case metric, we record a `before_` metric that aggregates the performance on the
      non-perturbed version of the corresponding inputs.
  
      We return the aggregate metrics across instances. Note that none of these metrics make a lot of sense if the
      original, un-perturbed version of an Instance is not included in a scenario (i.e., we want
      `include_original=True`).
      """
      # Collect statistics per input-metric pair across perturbations
      per_instance_perturbation_stats: Dict[
          Tuple[MetricName, str], List[Stat]
      ] = defaultdict(list)
      for instance, stats in per_instance_stats.items():
          for stat in stats:
              assert instance.id is not None
              # Group all perturbations for a specific metric name together
              per_instance_perturbation_stats[
                  (replace(stat.name, perturbation=None), instance.id)
              ].append(stat)
  
      # Compute worst perturbation stats
      derived_stats_dict: Dict[MetricName, Stat] = {}
      for (
          metric_name,
          instance_id,
      ), stats in per_instance_perturbation_stats.items():
          original_stat: Optional[Stat] = None
          robustness_stat = Stat(
              replace(
                  metric_name,
                  perturbation=PerturbationDescription(
                      name="robustness", robustness=True
                  ),
              )
          )
          fairness_stat = Stat(
              replace(
                  metric_name,
                  perturbation=PerturbationDescription(
                      name="fairness", fairness=True
                  ),
              )
          )
          individual_perturbation_stats: Dict[
              PerturbationDescription, Stat
          ] = {}
  
          for (
              stat
          ) in (
              stats
          ):  # go through all the perturbations of the instance and merge relevant stats
              perturbation = stat.name.perturbation
              if perturbation is None:
                  assert (
                      original_stat is None
                  )  # we should only have one original stat
                  original_stat = stat
              else:
                  if perturbation.robustness:
                      robustness_stat.merge(stat)
                  if perturbation.fairness:
                      fairness_stat.merge(stat)
                  assert (
                      perturbation not in individual_perturbation_stats
                  ), perturbation
                  individual_perturbation_stats[perturbation] = Stat(
                      stat.name
                  ).merge(
                      stat
                  )  # copy
  
          for stat in [
              robustness_stat,
              fairness_stat,
              *individual_perturbation_stats.values(),
          ]:
              perturbation = stat.name.perturbation
              assert perturbation is not None
  
              if original_stat is not None:
                  stat.merge(original_stat)
                  if perturbation.name not in [
                      "robustness",
                      "fairness",
                  ]:
                      before = replace(
                          perturbation,
                          computed_on=PERTURBATION_ORIGINAL,
                      )
                      merge_stat(
                          derived_stats_dict,
                          Stat(
                              replace(stat.name, perturbation=before)
                          ).merge(original_stat),
                      )
  
              # keep the minimum performance for each input
              worst = replace(
                  perturbation, computed_on=PERTURBATION_WORST
              )
              if stat.count > 0:
                  # TODO: take stat.max if lower_is_better = True
                  merge_stat(
                      derived_stats_dict,
                      Stat(replace(stat.name, perturbation=worst)).add(
                          stat.min
                      ),
                  )
      return list(derived_stats_dict.values())
  
imports: from abc import ABC
  from dataclasses import dataclass, replace
  from collections import defaultdict
  from typing import List, Dict, Tuple, Optional, Iterable, Set
  from helm.common.object_spec import ObjectSpec, create_object
  from helm.common.general import singleton, parallel_map
  from helm.benchmark.augmentations.perturbation_description import PerturbationDescription, PERTURBATION_ORIGINAL, PERTURBATION_WORST
  from helm.benchmark.adaptation.adapters.adapter_factory import ADAPT_LANGUAGE_MODELING
  from helm.benchmark.adaptation.scenario_state import ScenarioState
  from helm.benchmark.adaptation.request_state import RequestState
  from helm.benchmark.adaptation.adapter_spec import AdapterSpec
  from helm.benchmark.scenarios.scenario import Instance
  from .metric_name import MetricName, MetricContext
  from .metric_service import MetricService
  from .statistic import Stat, merge_stat
label_desc: Use image node_id 8 for calling the Metric obj's underlying member method code with example usage: obj.compute_worst_case_metrics(per_instance_stats) and returns: list