class_docstr: This class is for testing directly via get_regression_trainer
  
  It mixes in `TrainerIntegrationCommon` which already has a lot of helper validation methods
  which we can re-use here.
  
  Important: this class' setup can only work with a single gpu because it runs within the current
  pytest worker. For multi-gpu tests use TestDeepSpeedWithLauncher.
  
  Note: if any of the tests of this class get run there will be at least one gpu occupied by them
  until this pytest worker exits. This is because the gpu memory allocated by the cuda-kernels
  won't be released until this pytest worker exits.
  
  This may appear as some run-away tests if you watch `nvidia-smi` while other tests that fork new
  processes are run. So there will be one or two "stale" processes reported in `nvidia-smi`. This
  is not a bug.
class_docstr_tok: ["This","class","is","for","testing","directly","via","get_regression_trainer","It","mixes","in","`","TrainerIntegrationCommon","`","which","already","has","a","lot","of","helper","validation","methods","which","we","can","re-use","here",".","Important",":","this","class","'","setup","can","only","work","with","a","single","gpu","because","it","runs","within","the","current","pytest","worker",".","For","multi-gpu","tests","use","TestDeepSpeedWithLauncher",".","Note",":","if","any","of","the","tests","of","this","class","get","run","there","will","be","at","least","one","gpu","occupied","by","them","until","this","pytest","worker","exits",".","This","is","because","the","gpu","memory","allocated","by","the","cuda-kernels","wo","n't","be","released","until","this","pytest","worker","exits",".","This","may","appear","as","some","run-away","tests","if","you","watch","`","nvidia-smi","`","while","other","tests","that","fork","new","processes","are","run",".","So","there","will","be","one","or","two","``","stale","''","processes","reported","in","`","nvidia-smi","`",".","This","is","not","a","bug","."]
code: def test_hyperparameter_search(self):
      with mockenv_context(**self.dist_env_1_gpu):
          ds_config_zero3_dict = self.get_config_dict(ZERO3)
  
          # hyperparameter_search requires model_init() to recreate the model for each trial
          def model_init():
              config = RegressionModelConfig(
                  a=0, b=0, double_output=False
              )
              model = RegressionPreTrainedModel(config)
              return model
  
          trainer = get_regression_trainer(
              local_rank=0,
              fp16=True,
              model_init=model_init,
              deepspeed=ds_config_zero3_dict,
          )
  
          n_trials = 3
          with CaptureLogger(deepspeed_logger) as cl:
              with CaptureStd() as cs:
                  trainer.hyperparameter_search(
                      direction="maximize", n_trials=n_trials
                  )
          self.assertIn(
              "DeepSpeed info",
              cl.out,
              "expected DeepSpeed logger output but got none",
          )
          self.assertIn(
              f"Trial {n_trials-1} finished with value",
              cs.err,
              "expected hyperparameter_search output",
          )
          self.assertIn(
              "Best is trial",
              cs.err,
              "expected hyperparameter_search output",
          )
  
imports: import dataclasses
  import io
  import itertools
  import json
  import os
  import unittest
  from copy import deepcopy
  import datasets
  from parameterized import parameterized
  from tests.trainer.test_trainer import TrainerIntegrationCommon
  from transformers import AutoModel, TrainingArguments, is_torch_available, logging
  from transformers.deepspeed import HfDeepSpeedConfig, is_deepspeed_available, unset_hf_deepspeed_config
  from transformers.testing_utils import CaptureLogger, CaptureStd, CaptureStderr, LoggingLevel, TestCasePlus, execute_subprocess_async, get_gpu_count, mockenv_context, require_deepspeed, require_optuna, require_torch_gpu, require_torch_multi_gpu, slow
  from transformers.trainer_utils import get_last_checkpoint, set_seed
  from transformers.utils import WEIGHTS_NAME, is_torch_bf16_gpu_available
label_desc: Use image node_id 6 for calling the TrainerIntegrationDeepSpeed obj's underlying member method code with example usage: obj.test_hyperparameter_search() and returns: model