class_docstr: Run bayesian optimisation loop
  
  An `Optimizer` represents the steps of a bayesian optimisation loop. To use it you need to
  provide your own loop mechanism. The various optimisers provided by `skopt` use this class
  under the hood. Use this class directly if you want to control the iterations of your bayesian
  optimisation loop
  
  Parameters
  ----------
  dimensions: List
      List of shape (n_dims,) containing search space dimensions. Each search dimension can be
      defined as any of the following:
  
      * Instance of a `Dimension` object (`Real`, `Integer` or `Categorical`)
      * (<lower_bound>, <upper_bound>) tuple (for `Real` or `Integer` dimensions)
      * (<lower_bound>, <upper_bound>, <prior>) tuple (for `Real` dimensions)
      * List of categories (for `Categorical` dimensions)
  base_estimator: {SKLearn Regressor, "GP", "RF", "ET", "GBRT", "DUMMY"}, default="GP"
      If not string, should inherit from `sklearn.base.RegressorMixin`. In addition, the `predict`
      method should have an optional `return_std` argument, which returns `std(Y | x)`,
      along with `E[Y | x]`.
  
      If `base_estimator` is a string in {"GP", "RF", "ET", "GBRT", "DUMMY"}, a surrogate model
      corresponding to the relevant `X_minimize` function is created
  n_initial_points: Int, default=10
      Number of evaluations of `func` with initialization points before approximating it with
      `base_estimator`. Points provided as `x0` count as initialization points.
      If len(`x0`) < `n_initial_points`, additional points are sampled at random
  acq_func: {"LCB", "EI", "PI", "gp_hedge", "EIps", "PIps"}, default="gp_hedge"
      Function to minimize over the posterior distribution. Can be any of the following:
  
      * "LCB": Lower confidence bound
      * "EI": Negative expected improvement
      * "PI": Negative probability of improvement
      * "gp_hedge": Probabilistically choose one of the above three acquisition functions at
        every iteration
  
          * The gains `g_i` are initialized to zero
          * At every iteration,
  
              * Each acquisition function is optimised independently to propose a candidate point
                `X_i`
              * Out of all these candidate points, the next point `X_best` is chosen by
                `softmax(eta g_i)`
              * After fitting the surrogate model with `(X_best, y_best)`, the gains are updated
                such that `g_i -= mu(X_i)`
  
      * "EIps": Negated expected improvement per second to take into account the function compute
        time. Then, the objective function is assumed to return two values, the first being the
        objective value and the second being the time taken in seconds
      * "PIps": Negated probability of improvement per second. The return type of the objective
        function is identical to that of "EIps"
  acq_optimizer: {"sampling", "lbfgs", "auto"}, default="auto"
      Method to minimize the acquisition function. The fit model is updated with the optimal
      value obtained by optimizing `acq_func` with `acq_optimizer`
  
      * "sampling": `acq_func` is optimized by computing `acq_func` at `n_initial_points`
        randomly sampled points.
      * "lbfgs": `acq_func` is optimized by
  
            * Randomly sampling `n_restarts_optimizer` (from `acq_optimizer_kwargs`) points
            * "lbfgs" is run for 20 iterations with these initial points to find local minima
            * The optimal of these local minima is used to update the prior
  
      * "auto": `acq_optimizer` is configured on the basis of the `base_estimator` and the search
        space. If the space is `Categorical` or if the provided estimator is based on tree-models,
        then this is set to "sampling"
  random_state: Int, or RandomState instance (optional)
      Set random state to something other than None for reproducible results
  acq_func_kwargs: Dict (optional)
      Additional arguments to be passed to the acquisition function.
  acq_optimizer_kwargs: Dict (optional)
      Additional arguments to be passed to the acquisition optimizer
  warn_on_re_ask: Boolean, default=False
      If True, and the internal `optimizer` recommends a point that has already been evaluated
      on invocation of `ask`, a warning is logged before recommending a random point. Either
      way, a random point is used instead of already-evaluated recommendations. However,
      logging the fact that this has taken place can be useful to indicate that the optimizer
      may be stalling, especially if it repeatedly recommends the same point. In these cases,
      if the suggested point is not optimal, it can be helpful to switch a different OptPro
      (especially `DummyOptPro`), which will suggest points using different criteria
  
  Attributes
  ----------
  Xi: List
      Points at which objective has been evaluated
  yi: List
      Values of objective at corresponding points in `Xi`
  models: List
      Regression models used to fit observations and compute acquisition function
  space: `hyperparameter_hunter.space.space_core.Space`
      Stores parameter search space used to sample points, bounds, and type of parameters
  n_initial_points_: Int
      Original value passed through the `n_initial_points` kwarg. The value of this attribute
      remains unchanged along the lifespan of `Optimizer`, unlike :attr:`_n_initial_points`
  _n_initial_points: Int
      Number of remaining points that must be evaluated before fitting a surrogate estimator and
      using it to recommend incumbent search points. Initially, :attr:`_n_initial_points` is set
      to the value of the `n_initial_points` kwarg, like :attr:`n_initial_points_`. However,
      :attr:`_n_initial_points` is decremented for each point `tell`-ed to `Optimizer`
class_docstr_tok: ["Run","bayesian","optimisation","loop","An","`","Optimizer","`","represents","the","steps","of","a","bayesian","optimisation","loop",".","To","use","it","you","need","to","provide","your","own","loop","mechanism",".","The","various","optimisers","provided","by","`","skopt","`","use","this","class","under","the","hood",".","Use","this","class","directly","if","you","want","to","control","the","iterations","of","your","bayesian","optimisation","loop","Parameters","--","--","--","--","--","dimensions",":","List","List","of","shape","(","n_dims",",",")","containing","search","space","dimensions",".","Each","search","dimension","can","be","defined","as","any","of","the","following",":","*","Instance","of","a","`","Dimension","`","object","(","`","Real","`",",","`","Integer","`","or","`","Categorical","`",")","*","(","<","lower_bound",">",",","<","upper_bound",">",")","tuple","(","for","`","Real","`","or","`","Integer","`","dimensions",")","*","(","<","lower_bound",">",",","<","upper_bound",">",",","<","prior",">",")","tuple","(","for","`","Real","`","dimensions",")","*","List","of","categories","(","for","`","Categorical","`","dimensions",")","base_estimator",":","{","SKLearn","Regressor",",","``","GP","''",",","``","RF","''",",","``","ET","''",",","``","GBRT","''",",","``","DUMMY","''","}",",","default=","''","GP","''","If","not","string",",","should","inherit","from","`","sklearn.base.RegressorMixin","`",".","In","addition",",","the","`","predict","`","method","should","have","an","optional","`","return_std","`","argument",",","which","returns","`","std","(","Y","|","x",")","`",",","along","with","`","E","[","Y","|","x","]","`",".","If","`","base_estimator","`","is","a","string","in","{","``","GP","''",",","``","RF","''",",","``","ET","''",",","``","GBRT","''",",","``","DUMMY","''","}",",","a","surrogate","model","corresponding","to","the","relevant","`","X_minimize","`","function","is","created","n_initial_points",":","Int",",","default=10","Number","of","evaluations","of","`","func","`","with","initialization","points","before","approximating","it","with","`","base_estimator","`",".","Points","provided","as","`","x0","`","count","as","initialization","points",".","If","len","(","`","x0","`",")","<","`","n_initial_points","`",",","additional","points","are","sampled","at","random","acq_func",":","{","``","LCB","''",",","``","EI","''",",","``","PI","''",",","``","gp_hedge","''",",","``","EIps","''",",","``","PIps","''","}",",","default=","''","gp_hedge","''","Function","to","minimize","over","the","posterior","distribution",".","Can","be","any","of","the","following",":","*","``","LCB","''",":","Lower","confidence","bound","*","``","EI","''",":","Negative","expected","improvement","*","``","PI","''",":","Negative","probability","of","improvement","*","``","gp_hedge","''",":","Probabilistically","choose","one","of","the","above","three","acquisition","functions","at","every","iteration","*","The","gains","`","g_i","`","are","initialized","to","zero","*","At","every","iteration",",","*","Each","acquisition","function","is","optimised","independently","to","propose","a","candidate","point","`","X_i","`","*","Out","of","all","these","candidate","points",",","the","next","point","`","X_best","`","is","chosen","by","`","softmax","(","eta","g_i",")","`","*","After","fitting","the","surrogate","model","with","`","(","X_best",",","y_best",")","`",",","the","gains","are","updated","such","that","`","g_i","-=","mu","(","X_i",")","`","*","``","EIps","''",":","Negated","expected","improvement","per","second","to","take","into","account","the","function","compute","time",".","Then",",","the","objective","function","is","assumed","to","return","two","values",",","the","first","being","the","objective","value","and","the","second","being","the","time","taken","in","seconds","*","``","PIps","''",":","Negated","probability","of","improvement","per","second",".","The","return","type","of","the","objective","function","is","identical","to","that","of","``","EIps","''","acq_optimizer",":","{","``","sampling","''",",","``","lbfgs","''",",","``","auto","''","}",",","default=","''","auto","''","Method","to","minimize","the","acquisition","function",".","The","fit","model","is","updated","with","the","optimal","value","obtained","by","optimizing","`","acq_func","`","with","`","acq_optimizer","`","*","``","sampling","''",":","`","acq_func","`","is","optimized","by","computing","`","acq_func","`","at","`","n_initial_points","`","randomly","sampled","points",".","*","``","lbfgs","''",":","`","acq_func","`","is","optimized","by","*","Randomly","sampling","`","n_restarts_optimizer","`","(","from","`","acq_optimizer_kwargs","`",")","points","*","``","lbfgs","''","is","run","for","20","iterations","with","these","initial","points","to","find","local","minima","*","The","optimal","of","these","local","minima","is","used","to","update","the","prior","*","``","auto","''",":","`","acq_optimizer","`","is","configured","on","the","basis","of","the","`","base_estimator","`","and","the","search","space",".","If","the","space","is","`","Categorical","`","or","if","the","provided","estimator","is","based","on","tree-models",",","then","this","is","set","to","``","sampling","''","random_state",":","Int",",","or","RandomState","instance","(","optional",")","Set","random","state","to","something","other","than","None","for","reproducible","results","acq_func_kwargs",":","Dict","(","optional",")","Additional","arguments","to","be","passed","to","the","acquisition","function",".","acq_optimizer_kwargs",":","Dict","(","optional",")","Additional","arguments","to","be","passed","to","the","acquisition","optimizer","warn_on_re_ask",":","Boolean",",","default=False","If","True",",","and","the","internal","`","optimizer","`","recommends","a","point","that","has","already","been","evaluated","on","invocation","of","`","ask","`",",","a","warning","is","logged","before","recommending","a","random","point",".","Either","way",",","a","random","point","is","used","instead","of","already-evaluated","recommendations",".","However",",","logging","the","fact","that","this","has","taken","place","can","be","useful","to","indicate","that","the","optimizer","may","be","stalling",",","especially","if","it","repeatedly","recommends","the","same","point",".","In","these","cases",",","if","the","suggested","point","is","not","optimal",",","it","can","be","helpful","to","switch","a","different","OptPro","(","especially","`","DummyOptPro","`",")",",","which","will","suggest","points","using","different","criteria","Attributes","--","--","--","--","--","Xi",":","List","Points","at","which","objective","has","been","evaluated","yi",":","List","Values","of","objective","at","corresponding","points","in","`","Xi","`","models",":","List","Regression","models","used","to","fit","observations","and","compute","acquisition","function","space",":","`","hyperparameter_hunter.space.space_core.Space","`","Stores","parameter","search","space","used","to","sample","points",",","bounds",",","and","type","of","parameters","n_initial_points_",":","Int","Original","value","passed","through","the","`","n_initial_points","`","kwarg",".","The","value","of","this","attribute","remains","unchanged","along","the","lifespan","of","`","Optimizer","`",",","unlike",":","attr",":","`","_n_initial_points","`","_n_initial_points",":","Int","Number","of","remaining","points","that","must","be","evaluated","before","fitting","a","surrogate","estimator","and","using","it","to","recommend","incumbent","search","points",".","Initially",",",":","attr",":","`","_n_initial_points","`","is","set","to","the","value","of","the","`","n_initial_points","`","kwarg",",","like",":","attr",":","`","n_initial_points_","`",".","However",",",":","attr",":","`","_n_initial_points","`","is","decremented","for","each","point","`","tell","`","-ed","to","`","Optimizer","`"]
docstr: Method to minimize the acquisition function. See documentation for the `acq_optimizer`
  kwarg in :meth:`Optimizer.__init__` for additional information
  
  Returns
  -------
  {"lbfgs", "sampling"}
      String in {"lbfgs", "sampling"}. If originally "auto", one of the two aforementioned
      strings is selected based on :attr:`base_estimator`
docstr_tok: ["Method","to","minimize","the","acquisition","function",".","See","documentation","for","the","`","acq_optimizer","`","kwarg","in",":","meth",":","`","Optimizer.__init__","`","for","additional","information","Returns","--","--","--","-","{","``","lbfgs","''",",","``","sampling","''","}","String","in","{","``","lbfgs","''",",","``","sampling","''","}",".","If","originally","``","auto","''",",","one","of","the","two","aforementioned","strings","is","selected","based","on",":","attr",":","`","base_estimator","`"]
code: def acq_optimizer(self) -> str:
      """Method to minimize the acquisition function. See documentation for the `acq_optimizer`
      kwarg in :meth:`Optimizer.__init__` for additional information
  
      Returns
      -------
      {"lbfgs", "sampling"}
          String in {"lbfgs", "sampling"}. If originally "auto", one of the two aforementioned
          strings is selected based on :attr:`base_estimator`"""
      return self._acq_optimizer
  
imports: from hyperparameter_hunter.settings import G
  from hyperparameter_hunter.space.space_core import Space, normalize_dimensions
  from joblib import Parallel, delayed
  from math import log
  from numbers import Number
  import numpy
  import sys
  import warnings
  from scipy.optimize import fmin_l_bfgs_b
  from sklearn.base import clone, is_regressor
  from sklearn.ensemble import GradientBoostingRegressor
  from sklearn.multioutput import MultiOutputRegressor
  from sklearn.utils import check_random_state
  from skopt.acquisition import _gaussian_acquisition, gaussian_acquisition_1D
  from skopt.learning import ExtraTreesRegressor, GaussianProcessRegressor, GradientBoostingQuantileRegressor, RandomForestRegressor
  from skopt.learning.gaussian_process.kernels import ConstantKernel, HammingKernel, Matern
  from skopt.utils import create_result, has_gradients
label_desc: Use image node_id 4 for calling the Optimizer obj's underlying member method code with example usage: obj.acq_optimizer() and returns: self