class_docstr: Run bayesian optimisation loop
  
  An `Optimizer` represents the steps of a bayesian optimisation loop. To use it you need to
  provide your own loop mechanism. The various optimisers provided by `skopt` use this class
  under the hood. Use this class directly if you want to control the iterations of your bayesian
  optimisation loop
  
  Parameters
  ----------
  dimensions: List
      List of shape (n_dims,) containing search space dimensions. Each search dimension can be
      defined as any of the following:
  
      * Instance of a `Dimension` object (`Real`, `Integer` or `Categorical`)
      * (<lower_bound>, <upper_bound>) tuple (for `Real` or `Integer` dimensions)
      * (<lower_bound>, <upper_bound>, <prior>) tuple (for `Real` dimensions)
      * List of categories (for `Categorical` dimensions)
  base_estimator: {SKLearn Regressor, "GP", "RF", "ET", "GBRT", "DUMMY"}, default="GP"
      If not string, should inherit from `sklearn.base.RegressorMixin`. In addition, the `predict`
      method should have an optional `return_std` argument, which returns `std(Y | x)`,
      along with `E[Y | x]`.
  
      If `base_estimator` is a string in {"GP", "RF", "ET", "GBRT", "DUMMY"}, a surrogate model
      corresponding to the relevant `X_minimize` function is created
  n_initial_points: Int, default=10
      Number of evaluations of `func` with initialization points before approximating it with
      `base_estimator`. Points provided as `x0` count as initialization points.
      If len(`x0`) < `n_initial_points`, additional points are sampled at random
  acq_func: {"LCB", "EI", "PI", "gp_hedge", "EIps", "PIps"}, default="gp_hedge"
      Function to minimize over the posterior distribution. Can be any of the following:
  
      * "LCB": Lower confidence bound
      * "EI": Negative expected improvement
      * "PI": Negative probability of improvement
      * "gp_hedge": Probabilistically choose one of the above three acquisition functions at
        every iteration
  
          * The gains `g_i` are initialized to zero
          * At every iteration,
  
              * Each acquisition function is optimised independently to propose a candidate point
                `X_i`
              * Out of all these candidate points, the next point `X_best` is chosen by
                `softmax(eta g_i)`
              * After fitting the surrogate model with `(X_best, y_best)`, the gains are updated
                such that `g_i -= mu(X_i)`
  
      * "EIps": Negated expected improvement per second to take into account the function compute
        time. Then, the objective function is assumed to return two values, the first being the
        objective value and the second being the time taken in seconds
      * "PIps": Negated probability of improvement per second. The return type of the objective
        function is identical to that of "EIps"
  acq_optimizer: {"sampling", "lbfgs", "auto"}, default="auto"
      Method to minimize the acquisition function. The fit model is updated with the optimal
      value obtained by optimizing `acq_func` with `acq_optimizer`
  
      * "sampling": `acq_func` is optimized by computing `acq_func` at `n_initial_points`
        randomly sampled points.
      * "lbfgs": `acq_func` is optimized by
  
            * Randomly sampling `n_restarts_optimizer` (from `acq_optimizer_kwargs`) points
            * "lbfgs" is run for 20 iterations with these initial points to find local minima
            * The optimal of these local minima is used to update the prior
  
      * "auto": `acq_optimizer` is configured on the basis of the `base_estimator` and the search
        space. If the space is `Categorical` or if the provided estimator is based on tree-models,
        then this is set to "sampling"
  random_state: Int, or RandomState instance (optional)
      Set random state to something other than None for reproducible results
  acq_func_kwargs: Dict (optional)
      Additional arguments to be passed to the acquisition function.
  acq_optimizer_kwargs: Dict (optional)
      Additional arguments to be passed to the acquisition optimizer
  warn_on_re_ask: Boolean, default=False
      If True, and the internal `optimizer` recommends a point that has already been evaluated
      on invocation of `ask`, a warning is logged before recommending a random point. Either
      way, a random point is used instead of already-evaluated recommendations. However,
      logging the fact that this has taken place can be useful to indicate that the optimizer
      may be stalling, especially if it repeatedly recommends the same point. In these cases,
      if the suggested point is not optimal, it can be helpful to switch a different OptPro
      (especially `DummyOptPro`), which will suggest points using different criteria
  
  Attributes
  ----------
  Xi: List
      Points at which objective has been evaluated
  yi: List
      Values of objective at corresponding points in `Xi`
  models: List
      Regression models used to fit observations and compute acquisition function
  space: `hyperparameter_hunter.space.space_core.Space`
      Stores parameter search space used to sample points, bounds, and type of parameters
  n_initial_points_: Int
      Original value passed through the `n_initial_points` kwarg. The value of this attribute
      remains unchanged along the lifespan of `Optimizer`, unlike :attr:`_n_initial_points`
  _n_initial_points: Int
      Number of remaining points that must be evaluated before fitting a surrogate estimator and
      using it to recommend incumbent search points. Initially, :attr:`_n_initial_points` is set
      to the value of the `n_initial_points` kwarg, like :attr:`n_initial_points_`. However,
      :attr:`_n_initial_points` is decremented for each point `tell`-ed to `Optimizer`
class_docstr_tok: ["Run","bayesian","optimisation","loop","An","`","Optimizer","`","represents","the","steps","of","a","bayesian","optimisation","loop",".","To","use","it","you","need","to","provide","your","own","loop","mechanism",".","The","various","optimisers","provided","by","`","skopt","`","use","this","class","under","the","hood",".","Use","this","class","directly","if","you","want","to","control","the","iterations","of","your","bayesian","optimisation","loop","Parameters","--","--","--","--","--","dimensions",":","List","List","of","shape","(","n_dims",",",")","containing","search","space","dimensions",".","Each","search","dimension","can","be","defined","as","any","of","the","following",":","*","Instance","of","a","`","Dimension","`","object","(","`","Real","`",",","`","Integer","`","or","`","Categorical","`",")","*","(","<","lower_bound",">",",","<","upper_bound",">",")","tuple","(","for","`","Real","`","or","`","Integer","`","dimensions",")","*","(","<","lower_bound",">",",","<","upper_bound",">",",","<","prior",">",")","tuple","(","for","`","Real","`","dimensions",")","*","List","of","categories","(","for","`","Categorical","`","dimensions",")","base_estimator",":","{","SKLearn","Regressor",",","``","GP","''",",","``","RF","''",",","``","ET","''",",","``","GBRT","''",",","``","DUMMY","''","}",",","default=","''","GP","''","If","not","string",",","should","inherit","from","`","sklearn.base.RegressorMixin","`",".","In","addition",",","the","`","predict","`","method","should","have","an","optional","`","return_std","`","argument",",","which","returns","`","std","(","Y","|","x",")","`",",","along","with","`","E","[","Y","|","x","]","`",".","If","`","base_estimator","`","is","a","string","in","{","``","GP","''",",","``","RF","''",",","``","ET","''",",","``","GBRT","''",",","``","DUMMY","''","}",",","a","surrogate","model","corresponding","to","the","relevant","`","X_minimize","`","function","is","created","n_initial_points",":","Int",",","default=10","Number","of","evaluations","of","`","func","`","with","initialization","points","before","approximating","it","with","`","base_estimator","`",".","Points","provided","as","`","x0","`","count","as","initialization","points",".","If","len","(","`","x0","`",")","<","`","n_initial_points","`",",","additional","points","are","sampled","at","random","acq_func",":","{","``","LCB","''",",","``","EI","''",",","``","PI","''",",","``","gp_hedge","''",",","``","EIps","''",",","``","PIps","''","}",",","default=","''","gp_hedge","''","Function","to","minimize","over","the","posterior","distribution",".","Can","be","any","of","the","following",":","*","``","LCB","''",":","Lower","confidence","bound","*","``","EI","''",":","Negative","expected","improvement","*","``","PI","''",":","Negative","probability","of","improvement","*","``","gp_hedge","''",":","Probabilistically","choose","one","of","the","above","three","acquisition","functions","at","every","iteration","*","The","gains","`","g_i","`","are","initialized","to","zero","*","At","every","iteration",",","*","Each","acquisition","function","is","optimised","independently","to","propose","a","candidate","point","`","X_i","`","*","Out","of","all","these","candidate","points",",","the","next","point","`","X_best","`","is","chosen","by","`","softmax","(","eta","g_i",")","`","*","After","fitting","the","surrogate","model","with","`","(","X_best",",","y_best",")","`",",","the","gains","are","updated","such","that","`","g_i","-=","mu","(","X_i",")","`","*","``","EIps","''",":","Negated","expected","improvement","per","second","to","take","into","account","the","function","compute","time",".","Then",",","the","objective","function","is","assumed","to","return","two","values",",","the","first","being","the","objective","value","and","the","second","being","the","time","taken","in","seconds","*","``","PIps","''",":","Negated","probability","of","improvement","per","second",".","The","return","type","of","the","objective","function","is","identical","to","that","of","``","EIps","''","acq_optimizer",":","{","``","sampling","''",",","``","lbfgs","''",",","``","auto","''","}",",","default=","''","auto","''","Method","to","minimize","the","acquisition","function",".","The","fit","model","is","updated","with","the","optimal","value","obtained","by","optimizing","`","acq_func","`","with","`","acq_optimizer","`","*","``","sampling","''",":","`","acq_func","`","is","optimized","by","computing","`","acq_func","`","at","`","n_initial_points","`","randomly","sampled","points",".","*","``","lbfgs","''",":","`","acq_func","`","is","optimized","by","*","Randomly","sampling","`","n_restarts_optimizer","`","(","from","`","acq_optimizer_kwargs","`",")","points","*","``","lbfgs","''","is","run","for","20","iterations","with","these","initial","points","to","find","local","minima","*","The","optimal","of","these","local","minima","is","used","to","update","the","prior","*","``","auto","''",":","`","acq_optimizer","`","is","configured","on","the","basis","of","the","`","base_estimator","`","and","the","search","space",".","If","the","space","is","`","Categorical","`","or","if","the","provided","estimator","is","based","on","tree-models",",","then","this","is","set","to","``","sampling","''","random_state",":","Int",",","or","RandomState","instance","(","optional",")","Set","random","state","to","something","other","than","None","for","reproducible","results","acq_func_kwargs",":","Dict","(","optional",")","Additional","arguments","to","be","passed","to","the","acquisition","function",".","acq_optimizer_kwargs",":","Dict","(","optional",")","Additional","arguments","to","be","passed","to","the","acquisition","optimizer","warn_on_re_ask",":","Boolean",",","default=False","If","True",",","and","the","internal","`","optimizer","`","recommends","a","point","that","has","already","been","evaluated","on","invocation","of","`","ask","`",",","a","warning","is","logged","before","recommending","a","random","point",".","Either","way",",","a","random","point","is","used","instead","of","already-evaluated","recommendations",".","However",",","logging","the","fact","that","this","has","taken","place","can","be","useful","to","indicate","that","the","optimizer","may","be","stalling",",","especially","if","it","repeatedly","recommends","the","same","point",".","In","these","cases",",","if","the","suggested","point","is","not","optimal",",","it","can","be","helpful","to","switch","a","different","OptPro","(","especially","`","DummyOptPro","`",")",",","which","will","suggest","points","using","different","criteria","Attributes","--","--","--","--","--","Xi",":","List","Points","at","which","objective","has","been","evaluated","yi",":","List","Values","of","objective","at","corresponding","points","in","`","Xi","`","models",":","List","Regression","models","used","to","fit","observations","and","compute","acquisition","function","space",":","`","hyperparameter_hunter.space.space_core.Space","`","Stores","parameter","search","space","used","to","sample","points",",","bounds",",","and","type","of","parameters","n_initial_points_",":","Int","Original","value","passed","through","the","`","n_initial_points","`","kwarg",".","The","value","of","this","attribute","remains","unchanged","along","the","lifespan","of","`","Optimizer","`",",","unlike",":","attr",":","`","_n_initial_points","`","_n_initial_points",":","Int","Number","of","remaining","points","that","must","be","evaluated","before","fitting","a","surrogate","estimator","and","using","it","to","recommend","incumbent","search","points",".","Initially",",",":","attr",":","`","_n_initial_points","`","is","set","to","the","value","of","the","`","n_initial_points","`","kwarg",",","like",":","attr",":","`","n_initial_points_","`",".","However",",",":","attr",":","`","_n_initial_points","`","is","decremented","for","each","point","`","tell","`","-ed","to","`","Optimizer","`"]
docstr: Request point (or points) at which objective should be evaluated next
  
  Parameters
  ----------
  n_points: Int (optional)
      Number of points returned by the ask method. If `n_points` not given, a single point
      to evaluate is returned. Otherwise, a list of points to evaluate is returned of size
      `n_points`. This is useful if you can evaluate your objective in parallel, and thus
      obtain more objective function evaluations per unit of time
  strategy: {"cl_min", "cl_mean", "cl_max"}, default="cl_min"
      Method used to sample multiple points if `n_points` is an integer. If `n_points` is not
      given, `strategy` is ignored.
  
      If set to "cl_min", then "Constant Liar" strategy (see reference) is used with lie
      objective value being minimum of observed objective values. "cl_mean" and "cl_max"
      correspond to the mean and max of values, respectively.
  
      With this strategy, a copy of optimizer is created, which is then asked for a point,
      and the point is told to the copy of optimizer with some fake objective (lie), the
      next point is asked from copy, it is also told to the copy with fake objective and so
      on. The type of lie defines different flavours of "cl..." strategies
  
  Returns
  -------
  List
      Point (or points) recommended to be evaluated next
  
  References
  ----------
  .. [1] Chevalier, C.; Ginsbourger, D.: "Fast Computation of the Multi-points Expected
      Improvement with Applications in Batch Selection".
      https://hal.archives-ouvertes.fr/hal-00732512/document
docstr_tok: ["Request","point","(","or","points",")","at","which","objective","should","be","evaluated","next","Parameters","--","--","--","--","--","n_points",":","Int","(","optional",")","Number","of","points","returned","by","the","ask","method",".","If","`","n_points","`","not","given",",","a","single","point","to","evaluate","is","returned",".","Otherwise",",","a","list","of","points","to","evaluate","is","returned","of","size","`","n_points","`",".","This","is","useful","if","you","can","evaluate","your","objective","in","parallel",",","and","thus","obtain","more","objective","function","evaluations","per","unit","of","time","strategy",":","{","``","cl_min","''",",","``","cl_mean","''",",","``","cl_max","''","}",",","default=","''","cl_min","''","Method","used","to","sample","multiple","points","if","`","n_points","`","is","an","integer",".","If","`","n_points","`","is","not","given",",","`","strategy","`","is","ignored",".","If","set","to","``","cl_min","''",",","then","``","Constant","Liar","''","strategy","(","see","reference",")","is","used","with","lie","objective","value","being","minimum","of","observed","objective","values",".","``","cl_mean","''","and","``","cl_max","''","correspond","to","the","mean","and","max","of","values",",","respectively",".","With","this","strategy",",","a","copy","of","optimizer","is","created",",","which","is","then","asked","for","a","point",",","and","the","point","is","told","to","the","copy","of","optimizer","with","some","fake","objective","(","lie",")",",","the","next","point","is","asked","from","copy",",","it","is","also","told","to","the","copy","with","fake","objective","and","so","on",".","The","type","of","lie","defines","different","flavours","of","``","cl","...","''","strategies","Returns","--","--","--","-","List","Point","(","or","points",")","recommended","to","be","evaluated","next","References","--","--","--","--","--","..","[","1","]","Chevalier",",","C.",";","Ginsbourger",",","D.",":","``","Fast","Computation","of","the","Multi-points","Expected","Improvement","with","Applications","in","Batch","Selection","''",".","https",":","\/\/hal.archives-ouvertes.fr\/hal-00732512\/document"]
code: def ask(
      self, n_points=None, strategy="cl_min"
  ):  # TODO: Try `n_points` default=1
      """Request point (or points) at which objective should be evaluated next
  
      Parameters
      ----------
      n_points: Int (optional)
          Number of points returned by the ask method. If `n_points` not given, a single point
          to evaluate is returned. Otherwise, a list of points to evaluate is returned of size
          `n_points`. This is useful if you can evaluate your objective in parallel, and thus
          obtain more objective function evaluations per unit of time
      strategy: {"cl_min", "cl_mean", "cl_max"}, default="cl_min"
          Method used to sample multiple points if `n_points` is an integer. If `n_points` is not
          given, `strategy` is ignored.
  
          If set to "cl_min", then "Constant Liar" strategy (see reference) is used with lie
          objective value being minimum of observed objective values. "cl_mean" and "cl_max"
          correspond to the mean and max of values, respectively.
  
          With this strategy, a copy of optimizer is created, which is then asked for a point,
          and the point is told to the copy of optimizer with some fake objective (lie), the
          next point is asked from copy, it is also told to the copy with fake objective and so
          on. The type of lie defines different flavours of "cl..." strategies
  
      Returns
      -------
      List
          Point (or points) recommended to be evaluated next
  
      References
      ----------
      .. [1] Chevalier, C.; Ginsbourger, D.: "Fast Computation of the Multi-points Expected
          Improvement with Applications in Batch Selection".
          https://hal.archives-ouvertes.fr/hal-00732512/document"""
      if n_points is None:
          return self._ask()
  
      #################### Validate Parameters ####################
      if not (isinstance(n_points, int) and n_points > 0):
          raise ValueError(
              f"`n_points` must be int > 0. Got {n_points}"
          )
  
      supported_strategies = ["cl_min", "cl_mean", "cl_max"]
      if strategy not in supported_strategies:
          raise ValueError(
              f"Expected `strategy` in {supported_strategies}. Got {strategy}"
          )
  
      #################### Check Cache ####################
      # If repeated parameters given to `ask`, return cached entry
      if (n_points, strategy) in self.cache_:
          return self.cache_[(n_points, strategy)]
  
      #################### Constant Liar ####################
      # Copy of the optimizer is made in order to manage the deletion of points with "lie"
      #   objective (the copy of optimizer is simply discarded)
      opt = self.copy(
          random_state=self.rng.randint(0, np.iinfo(np.int32).max)
      )
  
      points = []
      for i in range(n_points):
          x = opt.ask()
          points.append(x)
  
          # TODO: Put below section into `how_to_lie` helper function for easier testing
          ti_available = (
              self.acq_func.endswith("ps") and len(opt.yi) > 0
          )
          ti = [t for (_, t) in opt.yi] if ti_available else None
  
          # TODO: Do below `y_lie` lines directly calculate min/max/mean on `opt.yi` when it could also contain times?
          if strategy == "cl_min":
              y_lie = np.min(opt.yi) if opt.yi else 0.0  # CL-min lie
              t_lie = (
                  np.min(ti)
                  if ti is not None
                  else log(sys.float_info.max)
              )
          elif strategy == "cl_mean":
              y_lie = np.mean(opt.yi) if opt.yi else 0.0  # CL-mean lie
              t_lie = (
                  np.mean(ti)
                  if ti is not None
                  else log(sys.float_info.max)
              )
          else:
              y_lie = np.max(opt.yi) if opt.yi else 0.0  # CL-max lie
              t_lie = (
                  np.max(ti)
                  if ti is not None
                  else log(sys.float_info.max)
              )
  
          #################### Lie to Optimizer ####################
          # Use `_tell` (not `tell`) to prevent repeated log transformations of computation times
          if self.acq_func.endswith("ps"):
              opt._tell(x, (y_lie, t_lie))
          else:
              opt._tell(x, y_lie)
  
      #################### Cache and Return Result ####################
      self.cache_ = {(n_points, strategy): points}
      return points
  
imports: from hyperparameter_hunter.settings import G
  from hyperparameter_hunter.space.space_core import Space, normalize_dimensions
  from joblib import Parallel, delayed
  from math import log
  from numbers import Number
  import numpy
  import sys
  import warnings
  from scipy.optimize import fmin_l_bfgs_b
  from sklearn.base import clone, is_regressor
  from sklearn.ensemble import GradientBoostingRegressor
  from sklearn.multioutput import MultiOutputRegressor
  from sklearn.utils import check_random_state
  from skopt.acquisition import _gaussian_acquisition, gaussian_acquisition_1D
  from skopt.learning import ExtraTreesRegressor, GaussianProcessRegressor, GradientBoostingQuantileRegressor, RandomForestRegressor
  from skopt.learning.gaussian_process.kernels import ConstantKernel, HammingKernel, Matern
  from skopt.utils import create_result, has_gradients
label_desc: Use image node_id 6 for calling the Optimizer obj's underlying member method code with example usage: obj.ask(n_points, strategy) and returns: points, self, self