yikit.models package#

Machine learning models and hyperparameter optimization utilities.

This module provides scikit-learn compatible regressors (an ensemble of regressors and, when lightgbm is installed, a gradient boosting regressor) and, when optuna is installed, utilities for hyperparameter optimization.

class yikit.models.EnsembleRegressor(estimators: Sequence[Any] = (RandomForestRegressor(),), method: str = 'blending', cv: int | BaseCrossValidator | Iterable[Any] = 5, n_jobs: int | None = None, random_state: int | np.random.RandomState | None = None, scoring: str | Callable[..., float] | None = 'neg_mean_squared_error', verbose: int = 0, opt: bool = True, n_trials: int = 100)#

Bases: RegressorMixin, BaseEstimator

Regressor that combines regressors, optionally tuned with optuna.

fit builds a VotingRegressor or a StackingRegressor of scikit-learn from the given models, fits it on X and y as they are (without converting them to arrays, so that pipelines that use the column names of a pandas.DataFrame work), and predict delegates to it. With opt=True, each model is first wrapped in an OptunaSearchCV of optuna-integration that searches the parameters of yikit.models.ParamDistributions. In all the methods, the predictions are made by the models fitted again on all the training data (never by the average of the models of the folds).

Parameters:
  • estimators (sequence of estimators or of (str, estimator) tuples, default=(RandomForestRegressor(),)) – Regressors to combine. The given objects are not changed: fit works on copies, on which it applies the random_state rule (see random_state). The other parameters of the copies, such as n_jobs, are the given ones, except those that the tuning searches when opt=True. A (name, estimator) pair keeps its name. A model without a name is named after its class in lower case, and the models that share a name get -1, -2, … in their order (as in sklearn.pipeline.make_pipeline), e.g. [Ridge(), SVR(), Ridge()] gives "ridge-1", "svr" and "ridge-2". A Pipeline whose last step is a regressor is accepted, e.g. to select the features of a model with yikit.feature_selection.BorutaPy before it.

  • method ({"blending", "average", "stacking"}, default="blending") –

    How to combine the models.

    • "average": VotingRegressor, the average of the predictions of the models.

    • "stacking": StackingRegressor whose final estimator is LinearRegression(), fitted on the predictions of the models for the test part of each fold of cv.

    • "blending": as "stacking", with LinearRegression(positive=True, fit_intercept=False): the weights of the models are non-negative and are not scaled to sum to 1.

  • cv (int, cross-validation generator or iterable, default=5) – Splitting of the cross-validation of StackingRegressor (for "stacking" and "blending") and of the tuning of each model (when opt=True). Pass an int or a splitter such as KFold when both are used: an iterable of (train, test) indices refers to the samples of the whole data, which the tuning in the folds of StackingRegressor does not see.

  • n_jobs (int or None, default=None) – Number of jobs of VotingRegressor or StackingRegressor, which fit the models (and the folds) in parallel. None means 1 unless in a joblib.parallel_backend context. The models keep their own n_jobs, so the number of jobs running at once is up to this n_jobs times the n_jobs of each model. To run the jobs in parallel here, set n_jobs=1 on each model; otherwise keep None and let the models run in parallel. The n_jobs of OptunaSearchCV is not set, so the trials of the tuning run one after another. With "stacking" and "blending", this n_jobs also runs in parallel the splits of cv, in each of which the models are fitted (and tuned, see opt) again.

  • random_state (int, RandomState instance or None, default=None) – Seeds the models and the tuning. fit first draws one integer from it, model_random_state_, and sets it on every random_state parameter left unspecified (None, or the global RandomState that ngboost stores in its place) in each model and in the estimators nested in it (the steps of a Pipeline, the regressor of a TransformedTargetRegressor, the Base of ngboost’s NGBRegressor, …), the same rule as yikit.models.Objective. The random_state values given in the models are kept, and models without a random_state parameter (such as SVR) get none. With opt=True, it then draws one integer for each model, in the order of estimators, as the random_state of its OptunaSearchCV (the seed of the sampler). Pass an int for reproducible results.

  • scoring (str, callable or None, default="neg_mean_squared_error") – Score that the tuning maximizes (the scoring of OptunaSearchCV). Not used when opt=False.

  • verbose (int, default=0) – Verbosity of StackingRegressor (VotingRegressor gets bool(verbose)) and of the tuning. With 0, the logs of the trials of optuna are hidden while the models are tuned.

  • opt (bool, default=True) – Whether to tune each model before combining them. Each model is wrapped in an OptunaSearchCV that runs n_trials trials on the parameters of ParamDistributions(model, n_features=X.shape[1]) with cv and scoring, and fits the model with the best parameters on its training data. It needs optuna and optuna-integration; with opt=False they are not imported. With "stacking" and "blending", each model is tuned n_splits + 1 times (cv + 1 for an int cv), where n_splits is the number of splits of cv: in each split, on its training part only, and on all the data. Each tuning runs n_trials trials of n_splits fits, so the model is fitted about n_trials * n_splits * (n_splits + 1) times (3000 times with the defaults); with "average" it is tuned once (n_trials * n_splits fits).

  • n_trials (int, default=100) – Number of trials of the tuning of each model. Not used when opt=False.

estimator_#

The fitted ensemble of scikit-learn that predict uses.

Type:

VotingRegressor or StackingRegressor

estimators_#

The models fitted on all the training data, in the order of estimators. With opt=True they are the fitted OptunaSearchCV, whose study_, best_params_ and best_estimator_ hold the results of the tuning.

Type:

list of estimators

named_estimators_#

The models of estimators_ by name.

Type:

sklearn.utils.Bunch

final_estimator_#

The final estimator fitted on the cross-validated predictions of the models. Only with "stacking" and "blending".

Type:

LinearRegression

weights_#

The weights of the models (final_estimator_.coef_) with "stacking" and "blending"; None with "average".

Type:

ndarray of shape (n_estimators,) or None

model_random_state_#

The integer drawn from random_state and set on the random_state parameters left unspecified (None, or the global RandomState that ngboost stores in its place) in the models.

Type:

int

n_features_in_#

Number of features seen during fit.

Type:

int

feature_names_in_#

Names of the features seen during fit. Those of estimator_ when it has them (read from the models). When no model records them (e.g. a Pipeline whose first step is BorutaPy), the names of the columns of X (such as a pandas.DataFrame) as an ndarray of dtype object, if they are all of type str (the rule of scikit-learn). Not defined (removed after a previous fit) when X has no such names.

Type:

ndarray of shape (n_features_in_,)

See also

sklearn.ensemble.VotingRegressor

The ensemble of "average".

sklearn.ensemble.StackingRegressor

The ensemble of "stacking" and "blending".

yikit.models.ParamDistributions

The parameters searched with opt=True.

Notes

yikit 0.4.0 removed the boruta parameter and the results_ attribute. To select features, pass each model in a Pipeline whose first step selects them, such as yikit.feature_selection.BorutaPy (see Examples). To measure the scores of the ensemble, cross-validate it (e.g. with sklearn.model_selection.cross_validate).

NGBRegressor of ngboost is not recognized as a regressor by scikit-learn >= 1.6, nor by older scikit-learn with ngboost before 0.5.2, so fit rejects it (as VotingRegressor and StackingRegressor do). Wrap it as TransformedTargetRegressor(regressor=NGBRegressor(...)). With opt=True, OptunaSearchCV cannot tune the parameters of its Base (regressor__Base__max_depth, …), because set_params of ngboost 0.4.0 or later ignores nested names: ParamDistributions warns about them, and their values are still sampled and listed in best_params_ but have no effect.

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import Ridge
>>> from sklearn.svm import SVR
>>> from yikit.models import EnsembleRegressor
>>> X, y = make_regression(
...     n_samples=50, n_features=4, noise=1.0, random_state=334
... )
>>> ensemble = EnsembleRegressor(
...     estimators=[Ridge(), SVR()], method="blending", cv=3, opt=False
... )
>>> ensemble = ensemble.fit(X, y)
>>> list(ensemble.named_estimators_)
['ridge', 'svr']
>>> bool((ensemble.weights_ >= 0).all())
True
>>> ensemble.predict(X[:3]).shape
(3,)

Select the features of a model with BorutaPy of yikit (it needs boruta) as the first step of a Pipeline. This replaces the boruta parameter, which yikit 0.4.0 removed, and lets each model have its own selection, fitted on the same data as the model (on the training part of each split of cv too). perc="auto" works in a Pipeline. The small n_estimators, max_iter and max_shuf only keep this example fast:

>>> from sklearn.ensemble import RandomForestRegressor
>>> from sklearn.pipeline import Pipeline
>>> from yikit.feature_selection import BorutaPy
>>> selector = BorutaPy(
...     RandomForestRegressor(max_depth=5, n_jobs=1, random_state=334),
...     n_estimators=20,
...     max_iter=20,
...     max_shuf=100,
...     verbose=0,
...     random_state=334,
... )
>>> boruta_ridge = Pipeline([("boruta", selector), ("ridge", Ridge())])
>>> ensemble = EnsembleRegressor(
...     estimators=[("boruta_ridge", boruta_ridge), SVR()],
...     cv=3,
...     opt=False,
... ).fit(X, y)
>>> fitted = ensemble.named_estimators_["boruta_ridge"]
>>> fitted.named_steps["boruta"].support_.shape  # mask of the 4 features
(4,)
>>> ensemble.predict(X[:3]).shape
(3,)

With opt=True (the default; it needs optuna and optuna-integration), each model is tuned with n_trials trials before the models are combined. estimators_ holds the fitted OptunaSearchCV, and the parameters of a model in a Pipeline are prefixed with the name of its step:

>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> ensemble = EnsembleRegressor(
...     estimators=[Ridge(), make_pipeline(StandardScaler(), SVR())],
...     cv=3,
...     n_trials=5,
...     random_state=334,
... ).fit(X, y)
>>> search = ensemble.named_estimators_["pipeline"]
>>> sorted(search.best_params_)
['svr__C', 'svr__epsilon']
>>> len(search.study_.trials)
5

A Pipeline of BorutaPy is tuned in the same way, on the parameters of its regressor only (BorutaPy has no search space). The selection then runs in every fit of the tuning, which makes it slow:

>>> ensemble = EnsembleRegressor(
...     estimators=[("boruta_ridge", boruta_ridge), SVR()],
...     cv=3,
...     n_trials=5,
...     random_state=334,
... ).fit(X, y)
>>> search = ensemble.named_estimators_["boruta_ridge"]
>>> sorted(search.best_params_)
['ridge__alpha']
fit(X: ArrayLike, y: ArrayLike) → EnsembleRegressor#

Fit the ensemble of the (tuned) models.

Parameters:
  • X (array-like of shape (n_samples, n_features)) – Training data. It is passed to the models as it is.

  • y (array-like of shape (n_samples,)) – Target values.

Returns:

self – The fitted ensemble.

Return type:

EnsembleRegressor

Raises:
  • ValueError – If method is not one of "average", "stacking" and "blending", if estimators is empty, if one of the models is not a regressor, or if X is not 2-dimensional.

  • ImportError – If opt=True and optuna or optuna-integration cannot be imported. It tells the user to install them or to pass opt=False.

  • NotImplementedError – If opt=True and ParamDistributions has no search space for one of the models. It names the model and its type.

predict(X: ArrayLike) → NDArray[Any]#

Predict with the ensemble of the models fitted on all the data.

Parameters:

X (array-like of shape (n_samples, n_features)) – Samples to predict. They are passed to the models as they are.

Returns:

Predicted values.

Return type:

ndarray of shape (n_samples,)

Raises:

sklearn.exceptions.NotFittedError – If the ensemble is not fitted yet.

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → EnsembleRegressor#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

class yikit.models.GBDTRegressor(boosting_type: str = 'gbdt', num_leaves: int = 31, max_depth: int = -1, learning_rate: float = 0.1, n_estimators: int = 100, subsample_for_bin: int = 200000, objective: Any = None, class_weight: Any = None, min_split_gain: float = 0.0, min_child_weight: float = 0.001, min_child_samples: int = 20, subsample: float = 1.0, subsample_freq: int = 0, colsample_bytree: float = 1.0, reg_alpha: float = 0.0, reg_lambda: float = 0.0, random_state: Any = None, n_jobs: int = -1, silent: bool = True, importance_type: str = 'split')#

Bases: RegressorMixin, BaseEstimator

Gradient Boosting Decision Tree regressor using LightGBM.

fit holds out 20% of the training data as a validation set, trains lightgbm.LGBMRegressor on the rest, and stops adding trees when neither the mean squared error nor the mean absolute error on the validation set improves for 20 rounds. Predictions use the best iteration. It works with LightGBM 4.0 and later.

Only the parameters below are accepted. To pass other LightGBM parameters, use lightgbm.LGBMRegressor directly; it is searched by yikit.models.Objective with the same search space.

Parameters:
  • boosting_type (str, default='gbdt') – Type of boosting algorithm to use.

  • num_leaves (int, default=31) – Maximum tree leaves for base learners.

  • max_depth (int, default=-1) – Maximum tree depth for base learners, <=0 means no limit.

  • learning_rate (float, default=0.1) – Boosting learning rate.

  • n_estimators (int, default=100) – Maximum number of boosted trees to fit.

  • subsample_for_bin (int, default=200000) – Number of samples for constructing bins.

  • objective (str, callable or None, default=None) – Learning objective. None means LightGBM’s default for regression.

  • class_weight (dict, 'balanced' or None, default=None) – Passed to LGBMRegressor; it has no effect on regression.

  • min_split_gain (float, default=0.0) – Minimum loss reduction required to make a further partition.

  • min_child_weight (float, default=0.001) – Minimum sum of instance weight (hessian) needed in a child.

  • min_child_samples (int, default=20) – Minimum number of data needed in a child (leaf).

  • subsample (float, default=1.0) – Subsample ratio of the training instances.

  • subsample_freq (int, default=0) – Frequency of subsample, <=0 means no enable.

  • colsample_bytree (float, default=1.0) – Subsample ratio of columns when constructing each tree.

  • reg_alpha (float, default=0.0) – L1 regularization term on weights.

  • reg_lambda (float, default=0.0) – L2 regularization term on weights.

  • random_state (int, RandomState instance or None, default=None) – Controls the split into training and validation data and the randomness of LightGBM.

  • n_jobs (int, default=-1) – Number of parallel threads used by LightGBM.

  • silent (bool, default=True) – If True, LightGBM prints nothing (verbosity=-1); otherwise it prints information messages (verbosity=1).

  • importance_type (str, default='split') – The type of feature importance in feature_importances_.

estimator_#

The fitted LightGBM regressor.

Type:

lightgbm.LGBMRegressor

best_iteration_#

Number of trees chosen by early stopping and used by predict. It is the best_iteration_ of estimator_.

Type:

int

feature_importances_#

The feature importances.

Type:

ndarray of shape (n_features,)

n_features_in_#

Number of features seen during fit.

Type:

int

rng_#

Random state used for the split and passed to LightGBM.

Type:

RandomState

Examples

>>> import numpy as np
>>> from yikit.models import GBDTRegressor
>>> rng = np.random.RandomState(0)
>>> X = rng.normal(size=(100, 5))
>>> y = X @ rng.normal(size=5) + rng.normal(scale=0.1, size=100)
>>> model = GBDTRegressor(n_estimators=500, random_state=0).fit(X, y)
>>> model.predict(X).shape
(100,)
>>> model.best_iteration_ < 500
True

Notes

This class requires the ‘lightgbm’ package to be installed.

fit(X: ArrayLike, y: ArrayLike) → GBDTRegressor#

Fit the model with early stopping on a held-out split.

Parameters:
  • X (array-like of shape (n_samples, n_features)) – Training data.

  • y (array-like of shape (n_samples,)) – Target values.

Returns:

The fitted estimator.

Return type:

GBDTRegressor

predict(X: ArrayLike) → NDArray[Any]#

Predict with the best iteration found by early stopping.

Parameters:

X (array-like of shape (n_samples, n_features)) – Samples.

Returns:

Predicted values.

Return type:

ndarray of shape (n_samples,)

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → GBDTRegressor#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

class yikit.models.Objective(estimator: Any, X: ArrayLike, y: ArrayLike, custom_params: Callable[[optuna.trial.BaseTrial], Mapping[str, Any]] | None = None, fixed_params: Mapping[str, Any] | None = None, cv: int | BaseCrossValidator | Iterable[Any] = 5, random_state: int | np.random.RandomState | None = None, scoring: str | Callable[..., float] | None = None, n_jobs: int | None = None)#

Bases: object

Objective function of optuna that cross-validates an estimator.

Each trial copies estimator and sets on the copy only the values searched in the trial, the integer of the random_state rule and fixed_params, in this order of priority from the lowest. Every other parameter keeps the value of estimator: in particular n_jobs and the values that yikit recommends (see RecommendedParams) are never set unless they are in fixed_params. The copy is cross-validated, and the mean of its test scores is the value of the trial.

The searched parameters come from the search space table of yikit, which knows LGBMRegressor of LightGBM, GBDTRegressor of yikit, RandomForestRegressor, SVR, LinearSVR, MLPRegressor, NGBRegressor of ngboost, PLSRegression, Ridge, Lasso and ElasticNet (and their subclasses). The last step of a Pipeline and the regressor of a TransformedTargetRegressor are followed, so a nested model is searched with prefixed names such as regressor__svr__C. custom_params replaces the table.

random_state rule: every random_state parameter of estimator and of the estimators nested in it whose value is None (or the global RandomState of NumPy, which ngboost puts in place of None) is set to model_random_state, one integer drawn from random_state when the objective is created. The values given by the user are kept, and models without a random_state parameter get none. An estimator given in fixed_params follows the same rule in place of the one it replaces.

A trial whose model fails to fit in every fold of the cross-validation is reported with a FitFailedWarning and returns nan, as does a trial whose fit fails in some folds; optuna records such a trial as failed, never chooses it as the best trial, and goes on with the search.

Parameters:
  • estimator (object) – scikit-learn compatible estimator instance to search, e.g. sklearn.ensemble.RandomForestRegressor() or a Pipeline. It is never modified.

  • X (array-like of shape (n_samples, n_features)) – Features. Its number of columns bounds the parameters limited by the data, such as n_components of PLSRegression.

  • y (array-like of shape (n_samples,)) – Target.

  • custom_params (callable or None, default=None) – Function that takes an optuna trial and returns a dict from parameter names to values, suggesting the values with the trial, e.g. lambda trial: {"C": trial.suggest_float("C", 0.1, 10)}. When it returns a non-empty dict, the dict is used instead of the search space table, and its names are used as they are (no prefix is added). An empty dict means that the table is used.

  • fixed_params (Mapping or None, default=None) – Parameter values to set on every model, prefixed like the searched names for nested estimators. They take priority over the searched values and the random_state rule. A fixed name covers itself and every name under it, and the names of the table that it covers are not searched. When it holds an estimator, such as {"Base": DecisionTreeRegressor(max_depth=5)} for NGBRegressor or {"svr": SVR(C=10.0)} for a Pipeline, the random_state rule does not set the names under it either (Base__max_depth, svr__C, … are neither searched nor changed), so the parameters of the given estimator are kept. The rule is applied to that estimator instead: its random_state parameters left as None get model_random_state (e.g. Base__random_state), and those given are kept. Names under a fixed estimator that custom_params returns are still applied, to the copy of that estimator. None and {} mean no fixed value.

  • cv (int, cross-validation generator or iterable, default=5) – Cross-validation splitting strategy passed to check_cv.

  • random_state (int, RandomState instance or None, default=None) – Seed of the TPESampler exposed as sampler and of model_random_state, drawn in this order.

  • scoring (str, callable or None, default=None) – Scoring passed to check_scoring. None means the score method of estimator.

  • n_jobs (int or None, default=None) – Number of jobs of cross_validate. It is not set on the model.

estimator#

The estimator passed by the user.

Type:

object

X#

Features validated by check_X_y.

Type:

numpy.ndarray of shape (n_samples, n_features)

y#

Target validated by check_X_y.

Type:

numpy.ndarray of shape (n_samples,)

custom_params#

The custom_params passed by the user.

Type:

callable or None

fixed_params#

A copy of fixed_params ({} when None was passed).

Type:

dict

cv#

Cross-validator returned by check_cv.

Type:

cross-validation generator

scoring#

Scorer returned by check_scoring.

Type:

callable

n_jobs#

Number of jobs of cross_validate.

Type:

int or None

sampler#

Sampler seeded from random_state, to pass to optuna.create_study.

Type:

optuna.samplers.TPESampler

model_random_state#

Integer set on the unspecified random_state parameters, the same for every trial and for get_best_params.

Type:

int

param_distributions#

Prefixed distributions of the search space table for estimator and the number of columns of X, without the names covered by fixed_params (the fixed names and the names under them). None when the table has no row for the model.

Type:

dict or None

estimator_#

Unfitted model of the last trial, with the parameters of that trial set. It exists after the first trial.

Type:

object

Raises:
  • NotImplementedError – If the table has no search space for the (resolved) model and custom_params is None.

  • ValueError – If a name in fixed_params is not a parameter of the targeted estimator.

Examples

>>> import optuna
>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import ElasticNet
>>> from yikit.models import Objective
>>> X, y = make_regression(n_samples=50, n_features=5, random_state=334)
>>> objective = Objective(ElasticNet(), X, y, random_state=334)
>>> sorted(objective.param_distributions)
['alpha', 'l1_ratio']
>>> study = optuna.create_study(
...     direction="maximize", sampler=objective.sampler
... )
>>> study.optimize(objective, n_trials=10)
>>> best_estimator = objective.get_best_estimator(study).fit(X, y)

Ridge and Lasso are searched in the same way (they replace the removed LinearModelRegressor):

>>> from sklearn.linear_model import Lasso, Ridge
>>> sorted(Objective(Ridge(), X, y).param_distributions)
['alpha']
>>> sorted(Objective(Lasso(), X, y).param_distributions)
['alpha']

An SVR scaled by a Pipeline and a TransformedTargetRegressor (which replace the removed SupportVectorRegressor) is searched with prefixed names. fixed_params sets the gamma that SupportVectorRegressor used:

>>> from sklearn.compose import TransformedTargetRegressor
>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> from sklearn.svm import SVR
>>> estimator = TransformedTargetRegressor(
...     regressor=make_pipeline(StandardScaler(), SVR()),
...     transformer=StandardScaler(),
... )
>>> objective = Objective(
...     estimator,
...     X,
...     y,
...     fixed_params={"regressor__svr__gamma": "auto"},
...     random_state=334,
... )
>>> sorted(objective.param_distributions)
['regressor__svr__C', 'regressor__svr__epsilon']
>>> study = optuna.create_study(
...     direction="maximize", sampler=objective.sampler
... )
>>> study.optimize(objective, n_trials=10)
>>> best_params = objective.get_best_params(study)
>>> sorted(best_params)
['regressor__svr__C', 'regressor__svr__epsilon', 'regressor__svr__gamma']
>>> best_estimator = objective.get_best_estimator(study)
>>> best_estimator.regressor.named_steps["svr"].gamma
'auto'
>>> best_estimator = best_estimator.fit(X, y)
get_best_estimator(study: Study) → Any#

Return an unfitted copy of estimator with the best parameters.

Parameters:

study (optuna.study.Study) – Study optimized with this objective. It must have a completed trial.

Returns:

Unfitted copy of estimator with get_best_params(study) set. The other parameters keep the values of estimator.

Return type:

object

Raises:

ValueError – If study has no completed trial (raised by optuna).

get_best_params(study: Study) → dict[str, Any]#

Return the parameters of the model of the best trial.

They are built again from the values of the best trial, in the same way as in the trial (through optuna.trial.FixedTrial), so they hold the names returned by custom_params even when they differ from the names given to suggest_*.

Parameters:

study (optuna.study.Study) – Study optimized with this objective. It must have a completed trial.

Returns:

The values searched in the best trial, the integer of the random_state rule for the unspecified random_state parameters, and fixed_params, which win over the others. The other parameters of estimator are not included.

Return type:

dict of str to object

Raises:

ValueError – If study has no completed trial (raised by optuna).

class yikit.models.ParamDistributions(estimator: Any, custom_params: Mapping[str, BaseDistribution] | Callable[[optuna.trial.BaseTrial], Mapping[str, BaseDistribution]] | None = None, *, n_features: int | None = None)#

Bases: dict

Distributions of the search space to pass to OptunaSearchCV.

A dict from parameter names to optuna distributions, built from the same search space table as Objective: for the same estimator and the same number of features, it holds the same names and the same distributions as the trials of Objective. It is passed as it is as param_distributions of OptunaSearchCV of optuna-integration, and keeps its contents and type through copy.deepcopy and sklearn.base.clone.

The table knows LGBMRegressor of LightGBM, GBDTRegressor of yikit, RandomForestRegressor, SVR, LinearSVR, MLPRegressor, NGBRegressor of ngboost, PLSRegression, Ridge, Lasso and ElasticNet (and their subclasses). The last step of a Pipeline and the regressor of a TransformedTargetRegressor are followed, so a nested model gets prefixed names such as regressor__svr__C, which OptunaSearchCV passes to set_params of the estimator. Only searched parameters are included: fixed values, such as n_jobs, random_state or the values of RecommendedParams, are set on the estimator itself, whose other parameters OptunaSearchCV keeps.

set_params of NGBRegressor (ngboost 0.4.0 or later) ignores nested names such as Base__max_depth, so OptunaSearchCV cannot apply them. A UserWarning lists such names; Objective applies them.

Parameters:
  • estimator (object) – scikit-learn compatible estimator instance to search, e.g. sklearn.svm.SVR() or a Pipeline whose last step is one. It is not modified.

  • custom_params (Mapping, callable or None, default=None) – Mapping from parameter names to optuna distributions, e.g. {"C": FloatDistribution(0.1, 10.0, log=True)}, or a function that takes an optuna trial and returns such a mapping. The function is called once with optuna.trial.FixedTrial({}), so it must return distributions instead of suggesting values (a function that suggests values, as custom_params of Objective does, raises the error of FixedTrial). A non-empty mapping is used instead of the search space table, and its names are used as they are (no prefix is added). None or an empty mapping means that the table is used.

  • n_features (int or None, default=None) – Number of features of the data passed to estimator (the outermost one when nested). Keyword-only. When given, the upper bound of the parameters limited by the data, n_components of PLSRegression, becomes min(10, n_features). When None, the bound of the table (10) is kept, and OptunaSearchCV records the trials that exceed the number of features as failed and goes on with the search.

estimator#

The estimator passed by the user.

Type:

object

custom_params#

The custom_params passed by the user.

Type:

Mapping, callable or None

n_features#

The n_features passed by the user.

Type:

int or None

Raises:
  • NotImplementedError – If the table has no search space for the (resolved) model and custom_params gives no distribution.

  • TypeError – If custom_params, or what its function returns, is not a mapping, or if one of its values is not an optuna distribution. The fixed_params and random_state arguments of yikit 0.4.0-rc.0 were removed, and passing them raises a TypeError too.

  • ValueError – If n_features is smaller than 1, or if a Pipeline on the way has no steps.

Warns:

UserWarning – If the resolved model ignores nested names in its own set_params (NGBRegressor of ngboost 0.4.0 or later) and the distributions hold such names, e.g. Base__max_depth.

See also

Objective

Objective function of optuna that searches the same table.

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.linear_model import ElasticNet
>>> from yikit.models import ParamDistributions
>>> from optuna_integration import OptunaSearchCV
>>> X, y = make_regression(n_samples=50, n_features=5, random_state=334)
>>> estimator = ElasticNet()
>>> param_distributions = ParamDistributions(
...     estimator, n_features=X.shape[1]
... )
>>> sorted(param_distributions)
['alpha', 'l1_ratio']
>>> search = OptunaSearchCV(
...     estimator,
...     param_distributions=param_distributions,
...     n_trials=10,
...     random_state=334,
... ).fit(X, y)
>>> sorted(search.best_params_)
['alpha', 'l1_ratio']

Ridge and Lasso are searched in the same way (they replace the removed LinearModelRegressor):

>>> from sklearn.linear_model import Lasso, Ridge
>>> sorted(ParamDistributions(Ridge()))
['alpha']
>>> sorted(ParamDistributions(Lasso()))
['alpha']

An SVR scaled by a Pipeline and a TransformedTargetRegressor (which replace the removed SupportVectorRegressor) gets prefixed names. The gamma that SupportVectorRegressor used is set on the SVR itself, and OptunaSearchCV keeps it:

>>> from sklearn.compose import TransformedTargetRegressor
>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> from sklearn.svm import SVR
>>> estimator = TransformedTargetRegressor(
...     regressor=make_pipeline(StandardScaler(), SVR(gamma="auto")),
...     transformer=StandardScaler(),
... )
>>> param_distributions = ParamDistributions(
...     estimator, n_features=X.shape[1]
... )
>>> sorted(param_distributions)
['regressor__svr__C', 'regressor__svr__epsilon']
>>> search = OptunaSearchCV(
...     estimator,
...     param_distributions=param_distributions,
...     n_trials=5,
...     random_state=334,
... ).fit(X, y)
>>> sorted(search.best_params_)
['regressor__svr__C', 'regressor__svr__epsilon']
>>> search.best_estimator_.regressor_.named_steps["svr"].gamma
'auto'

The upper bound of n_components of PLSRegression follows n_features, and custom_params replaces the table:

>>> from optuna.distributions import FloatDistribution
>>> from sklearn.cross_decomposition import PLSRegression
>>> ParamDistributions(PLSRegression())["n_components"].high
10
>>> ParamDistributions(PLSRegression(), n_features=3)["n_components"].high
3
>>> sorted(
...     ParamDistributions(
...         SVR(), {"C": FloatDistribution(0.1, 10.0, log=True)}
...     )
... )
['C']
class yikit.models.RecommendedParams(estimator: Any)#

Bases: dict

Values that yikit recommends to fix for an estimator, as a dict.

A dict from parameter names to the values that yikit recommends for estimator. The model is found in the same way as for the search space of Objective and ParamDistributions: the last step of a Pipeline and the regressor of a TransformedTargetRegressor are followed, and the names get the same prefix, such as regressor__svr__gamma. It is empty, without an error, when yikit recommends nothing for the model, including models without a search space.

The values are opt-in. Objective and ParamDistributions never set them by themselves, and this class does not modify estimator. Pass them yourself, as fixed_params of Objective or to set_params of the estimator (OptunaSearchCV keeps the parameters of the estimator that it does not search). The table starts small: it only holds gamma="auto" for SVR (and its subclasses), the value that the removed SupportVectorRegressor used, and may get more rows in later versions.

As a plain dict, it can be unpacked with **, compares equal to a dict with the same items, and keeps its contents and type through copy.deepcopy and pickle. Each instance holds new copies of the values, so changing it does not change the table.

Parameters:

estimator (object) – scikit-learn compatible estimator instance to look up, e.g. sklearn.svm.SVR() or a Pipeline whose last step is one. It is not modified.

estimator#

The estimator passed by the user.

Type:

object

Raises:

ValueError – If a Pipeline on the way has no steps.

See also

Objective

Objective function of optuna that takes them as fixed_params.

ParamDistributions

Distributions of the search space to pass to OptunaSearchCV.

Examples

>>> from sklearn.svm import SVR
>>> from yikit.models import RecommendedParams
>>> RecommendedParams(SVR())
RecommendedParams({'gamma': 'auto'})
>>> RecommendedParams(SVR()) == {"gamma": "auto"}
True
>>> from sklearn.ensemble import RandomForestRegressor
>>> RecommendedParams(RandomForestRegressor())
RecommendedParams({})

A nested SVR gets prefixed names:

>>> from sklearn.compose import TransformedTargetRegressor
>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> estimator = TransformedTargetRegressor(
...     regressor=make_pipeline(StandardScaler(), SVR()),
...     transformer=StandardScaler(),
... )
>>> RecommendedParams(estimator) == {"regressor__svr__gamma": "auto"}
True

Set the values on the estimator, e.g. before passing it to OptunaSearchCV with ParamDistributions:

>>> estimator = estimator.set_params(**RecommendedParams(estimator))
>>> estimator.regressor.named_steps["svr"].gamma
'auto'

Or fix them in every trial of Objective, which leaves the passed estimator as it is:

>>> import optuna
>>> from sklearn.datasets import make_regression
>>> from yikit.models import Objective
>>> X, y = make_regression(n_samples=50, n_features=5, random_state=334)
>>> estimator = SVR()
>>> objective = Objective(
...     estimator,
...     X,
...     y,
...     fixed_params=RecommendedParams(estimator),
...     random_state=334,
... )
>>> sorted(objective.param_distributions)
['C', 'epsilon']
>>> study = optuna.create_study(
...     direction="maximize", sampler=objective.sampler
... )
>>> study.optimize(objective, n_trials=5)
>>> objective.get_best_estimator(study).gamma
'auto'
>>> estimator.gamma
'scale'