yikit.models package#
Machine learning models and hyperparameter optimization utilities.
This module provides scikit-learn compatible regressors (an ensemble of regressors and, when lightgbm is installed, a gradient boosting regressor) and, when optuna is installed, utilities for hyperparameter optimization.
- class yikit.models.EnsembleRegressor(estimators: Sequence[Any] = (RandomForestRegressor(),), method: str = 'blending', cv: int | BaseCrossValidator | Iterable[Any] = 5, n_jobs: int | None = None, random_state: int | np.random.RandomState | None = None, scoring: str | Callable[..., float] | None = 'neg_mean_squared_error', verbose: int = 0, opt: bool = True, n_trials: int = 100)#
Bases:
RegressorMixin,BaseEstimatorRegressor that combines regressors, optionally tuned with optuna.
fitbuilds aVotingRegressoror aStackingRegressorof scikit-learn from the given models, fits it onXandyas they are (without converting them to arrays, so that pipelines that use the column names of apandas.DataFramework), andpredictdelegates to it. Withopt=True, each model is first wrapped in anOptunaSearchCVof optuna-integration that searches the parameters ofyikit.models.ParamDistributions. In all the methods, the predictions are made by the models fitted again on all the training data (never by the average of the models of the folds).- Parameters:
estimators (sequence of estimators or of (str, estimator) tuples, default=(RandomForestRegressor(),)) – Regressors to combine. The given objects are not changed:
fitworks on copies, on which it applies therandom_staterule (seerandom_state). The other parameters of the copies, such asn_jobs, are the given ones, except those that the tuning searches whenopt=True. A(name, estimator)pair keeps its name. A model without a name is named after its class in lower case, and the models that share a name get-1,-2, … in their order (as insklearn.pipeline.make_pipeline), e.g.[Ridge(), SVR(), Ridge()]gives"ridge-1","svr"and"ridge-2". APipelinewhose last step is a regressor is accepted, e.g. to select the features of a model withyikit.feature_selection.BorutaPybefore it.method ({"blending", "average", "stacking"}, default="blending") –
How to combine the models.
"average":VotingRegressor, the average of the predictions of the models."stacking":StackingRegressorwhose final estimator isLinearRegression(), fitted on the predictions of the models for the test part of each fold ofcv."blending": as"stacking", withLinearRegression(positive=True, fit_intercept=False): the weights of the models are non-negative and are not scaled to sum to 1.
cv (int, cross-validation generator or iterable, default=5) – Splitting of the cross-validation of
StackingRegressor(for"stacking"and"blending") and of the tuning of each model (whenopt=True). Pass an int or a splitter such asKFoldwhen both are used: an iterable of(train, test)indices refers to the samples of the whole data, which the tuning in the folds ofStackingRegressordoes not see.n_jobs (int or None, default=None) – Number of jobs of
VotingRegressororStackingRegressor, which fit the models (and the folds) in parallel. None means 1 unless in ajoblib.parallel_backendcontext. The models keep their ownn_jobs, so the number of jobs running at once is up to thisn_jobstimes then_jobsof each model. To run the jobs in parallel here, setn_jobs=1on each model; otherwise keep None and let the models run in parallel. Then_jobsofOptunaSearchCVis not set, so the trials of the tuning run one after another. With"stacking"and"blending", thisn_jobsalso runs in parallel the splits ofcv, in each of which the models are fitted (and tuned, seeopt) again.random_state (int, RandomState instance or None, default=None) – Seeds the models and the tuning.
fitfirst draws one integer from it,model_random_state_, and sets it on everyrandom_stateparameter left unspecified (None, or the globalRandomStatethat ngboost stores in its place) in each model and in the estimators nested in it (the steps of aPipeline, theregressorof aTransformedTargetRegressor, theBaseof ngboost’sNGBRegressor, …), the same rule asyikit.models.Objective. Therandom_statevalues given in the models are kept, and models without arandom_stateparameter (such asSVR) get none. Withopt=True, it then draws one integer for each model, in the order ofestimators, as therandom_stateof itsOptunaSearchCV(the seed of the sampler). Pass an int for reproducible results.scoring (str, callable or None, default="neg_mean_squared_error") – Score that the tuning maximizes (the
scoringofOptunaSearchCV). Not used whenopt=False.verbose (int, default=0) – Verbosity of
StackingRegressor(VotingRegressorgetsbool(verbose)) and of the tuning. With 0, the logs of the trials of optuna are hidden while the models are tuned.opt (bool, default=True) – Whether to tune each model before combining them. Each model is wrapped in an
OptunaSearchCVthat runsn_trialstrials on the parameters ofParamDistributions(model, n_features=X.shape[1])withcvandscoring, and fits the model with the best parameters on its training data. It needs optuna and optuna-integration; withopt=Falsethey are not imported. With"stacking"and"blending", each model is tunedn_splits + 1times (cv + 1for an intcv), wheren_splitsis the number of splits ofcv: in each split, on its training part only, and on all the data. Each tuning runsn_trialstrials ofn_splitsfits, so the model is fitted aboutn_trials * n_splits * (n_splits + 1)times (3000 times with the defaults); with"average"it is tuned once (n_trials * n_splitsfits).n_trials (int, default=100) – Number of trials of the tuning of each model. Not used when
opt=False.
- estimator_#
The fitted ensemble of scikit-learn that
predictuses.- Type:
VotingRegressor or StackingRegressor
- estimators_#
The models fitted on all the training data, in the order of
estimators. Withopt=Truethey are the fittedOptunaSearchCV, whosestudy_,best_params_andbest_estimator_hold the results of the tuning.- Type:
list of estimators
- named_estimators_#
The models of
estimators_by name.- Type:
sklearn.utils.Bunch
- final_estimator_#
The final estimator fitted on the cross-validated predictions of the models. Only with
"stacking"and"blending".- Type:
LinearRegression
- weights_#
The weights of the models (
final_estimator_.coef_) with"stacking"and"blending"; None with"average".- Type:
ndarray of shape (n_estimators,) or None
- model_random_state_#
The integer drawn from
random_stateand set on therandom_stateparameters left unspecified (None, or the globalRandomStatethat ngboost stores in its place) in the models.- Type:
int
- n_features_in_#
Number of features seen during
fit.- Type:
int
- feature_names_in_#
Names of the features seen during
fit. Those ofestimator_when it has them (read from the models). When no model records them (e.g. aPipelinewhose first step isBorutaPy), the names of thecolumnsofX(such as apandas.DataFrame) as an ndarray of dtype object, if they are all of typestr(the rule of scikit-learn). Not defined (removed after a previous fit) whenXhas no such names.- Type:
ndarray of shape (n_features_in_,)
See also
sklearn.ensemble.VotingRegressorThe ensemble of
"average".sklearn.ensemble.StackingRegressorThe ensemble of
"stacking"and"blending".yikit.models.ParamDistributionsThe parameters searched with
opt=True.
Notes
yikit 0.4.0 removed the
borutaparameter and theresults_attribute. To select features, pass each model in aPipelinewhose first step selects them, such asyikit.feature_selection.BorutaPy(see Examples). To measure the scores of the ensemble, cross-validate it (e.g. withsklearn.model_selection.cross_validate).NGBRegressorof ngboost is not recognized as a regressor by scikit-learn >= 1.6, nor by older scikit-learn with ngboost before 0.5.2, sofitrejects it (asVotingRegressorandStackingRegressordo). Wrap it asTransformedTargetRegressor(regressor=NGBRegressor(...)). Withopt=True,OptunaSearchCVcannot tune the parameters of itsBase(regressor__Base__max_depth, …), becauseset_paramsof ngboost 0.4.0 or later ignores nested names:ParamDistributionswarns about them, and their values are still sampled and listed inbest_params_but have no effect.Examples
>>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import Ridge >>> from sklearn.svm import SVR >>> from yikit.models import EnsembleRegressor >>> X, y = make_regression( ... n_samples=50, n_features=4, noise=1.0, random_state=334 ... ) >>> ensemble = EnsembleRegressor( ... estimators=[Ridge(), SVR()], method="blending", cv=3, opt=False ... ) >>> ensemble = ensemble.fit(X, y) >>> list(ensemble.named_estimators_) ['ridge', 'svr'] >>> bool((ensemble.weights_ >= 0).all()) True >>> ensemble.predict(X[:3]).shape (3,)
Select the features of a model with
BorutaPyof yikit (it needs boruta) as the first step of aPipeline. This replaces theborutaparameter, which yikit 0.4.0 removed, and lets each model have its own selection, fitted on the same data as the model (on the training part of each split ofcvtoo).perc="auto"works in aPipeline. The smalln_estimators,max_iterandmax_shufonly keep this example fast:>>> from sklearn.ensemble import RandomForestRegressor >>> from sklearn.pipeline import Pipeline >>> from yikit.feature_selection import BorutaPy >>> selector = BorutaPy( ... RandomForestRegressor(max_depth=5, n_jobs=1, random_state=334), ... n_estimators=20, ... max_iter=20, ... max_shuf=100, ... verbose=0, ... random_state=334, ... ) >>> boruta_ridge = Pipeline([("boruta", selector), ("ridge", Ridge())]) >>> ensemble = EnsembleRegressor( ... estimators=[("boruta_ridge", boruta_ridge), SVR()], ... cv=3, ... opt=False, ... ).fit(X, y) >>> fitted = ensemble.named_estimators_["boruta_ridge"] >>> fitted.named_steps["boruta"].support_.shape # mask of the 4 features (4,) >>> ensemble.predict(X[:3]).shape (3,)
With
opt=True(the default; it needs optuna and optuna-integration), each model is tuned withn_trialstrials before the models are combined.estimators_holds the fittedOptunaSearchCV, and the parameters of a model in aPipelineare prefixed with the name of its step:>>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import StandardScaler >>> ensemble = EnsembleRegressor( ... estimators=[Ridge(), make_pipeline(StandardScaler(), SVR())], ... cv=3, ... n_trials=5, ... random_state=334, ... ).fit(X, y) >>> search = ensemble.named_estimators_["pipeline"] >>> sorted(search.best_params_) ['svr__C', 'svr__epsilon'] >>> len(search.study_.trials) 5
A
PipelineofBorutaPyis tuned in the same way, on the parameters of its regressor only (BorutaPyhas no search space). The selection then runs in every fit of the tuning, which makes it slow:>>> ensemble = EnsembleRegressor( ... estimators=[("boruta_ridge", boruta_ridge), SVR()], ... cv=3, ... n_trials=5, ... random_state=334, ... ).fit(X, y) >>> search = ensemble.named_estimators_["boruta_ridge"] >>> sorted(search.best_params_) ['ridge__alpha']
- fit(X: ArrayLike, y: ArrayLike) EnsembleRegressor#
Fit the ensemble of the (tuned) models.
- Parameters:
X (array-like of shape (n_samples, n_features)) – Training data. It is passed to the models as it is.
y (array-like of shape (n_samples,)) – Target values.
- Returns:
self – The fitted ensemble.
- Return type:
- Raises:
ValueError – If
methodis not one of"average","stacking"and"blending", ifestimatorsis empty, if one of the models is not a regressor, or ifXis not 2-dimensional.ImportError – If
opt=Trueand optuna or optuna-integration cannot be imported. It tells the user to install them or to passopt=False.NotImplementedError – If
opt=TrueandParamDistributionshas no search space for one of the models. It names the model and its type.
- predict(X: ArrayLike) NDArray[Any]#
Predict with the ensemble of the models fitted on all the data.
- Parameters:
X (array-like of shape (n_samples, n_features)) – Samples to predict. They are passed to the models as they are.
- Returns:
Predicted values.
- Return type:
ndarray of shape (n_samples,)
- Raises:
sklearn.exceptions.NotFittedError – If the ensemble is not fitted yet.
- set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') EnsembleRegressor#
Configure whether metadata should be requested to be passed to the
scoremethod.Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with
enable_metadata_routing=True(seesklearn.set_config()). Please check the User Guide on how the routing mechanism works.The options for each parameter are:
True: metadata is requested, and passed toscoreif provided. The request is ignored if metadata is not provided.False: metadata is not requested and the meta-estimator will not pass it toscore.None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.str: metadata should be passed to the meta-estimator with this given alias instead of the original name.
The default (
sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.Added in version 1.3.
- Parameters:
sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for
sample_weightparameter inscore.- Returns:
self – The updated object.
- Return type:
object
- class yikit.models.GBDTRegressor(boosting_type: str = 'gbdt', num_leaves: int = 31, max_depth: int = -1, learning_rate: float = 0.1, n_estimators: int = 100, subsample_for_bin: int = 200000, objective: Any = None, class_weight: Any = None, min_split_gain: float = 0.0, min_child_weight: float = 0.001, min_child_samples: int = 20, subsample: float = 1.0, subsample_freq: int = 0, colsample_bytree: float = 1.0, reg_alpha: float = 0.0, reg_lambda: float = 0.0, random_state: Any = None, n_jobs: int = -1, silent: bool = True, importance_type: str = 'split')#
Bases:
RegressorMixin,BaseEstimatorGradient Boosting Decision Tree regressor using LightGBM.
fitholds out 20% of the training data as a validation set, trainslightgbm.LGBMRegressoron the rest, and stops adding trees when neither the mean squared error nor the mean absolute error on the validation set improves for 20 rounds. Predictions use the best iteration. It works with LightGBM 4.0 and later.Only the parameters below are accepted. To pass other LightGBM parameters, use
lightgbm.LGBMRegressordirectly; it is searched byyikit.models.Objectivewith the same search space.- Parameters:
boosting_type (str, default='gbdt') – Type of boosting algorithm to use.
num_leaves (int, default=31) – Maximum tree leaves for base learners.
max_depth (int, default=-1) – Maximum tree depth for base learners, <=0 means no limit.
learning_rate (float, default=0.1) – Boosting learning rate.
n_estimators (int, default=100) – Maximum number of boosted trees to fit.
subsample_for_bin (int, default=200000) – Number of samples for constructing bins.
objective (str, callable or None, default=None) – Learning objective. None means LightGBM’s default for regression.
class_weight (dict, 'balanced' or None, default=None) – Passed to
LGBMRegressor; it has no effect on regression.min_split_gain (float, default=0.0) – Minimum loss reduction required to make a further partition.
min_child_weight (float, default=0.001) – Minimum sum of instance weight (hessian) needed in a child.
min_child_samples (int, default=20) – Minimum number of data needed in a child (leaf).
subsample (float, default=1.0) – Subsample ratio of the training instances.
subsample_freq (int, default=0) – Frequency of subsample, <=0 means no enable.
colsample_bytree (float, default=1.0) – Subsample ratio of columns when constructing each tree.
reg_alpha (float, default=0.0) – L1 regularization term on weights.
reg_lambda (float, default=0.0) – L2 regularization term on weights.
random_state (int, RandomState instance or None, default=None) – Controls the split into training and validation data and the randomness of LightGBM.
n_jobs (int, default=-1) – Number of parallel threads used by LightGBM.
silent (bool, default=True) – If True, LightGBM prints nothing (
verbosity=-1); otherwise it prints information messages (verbosity=1).importance_type (str, default='split') – The type of feature importance in
feature_importances_.
- estimator_#
The fitted LightGBM regressor.
- Type:
lightgbm.LGBMRegressor
- best_iteration_#
Number of trees chosen by early stopping and used by
predict. It is thebest_iteration_ofestimator_.- Type:
int
- feature_importances_#
The feature importances.
- Type:
ndarray of shape (n_features,)
- n_features_in_#
Number of features seen during fit.
- Type:
int
- rng_#
Random state used for the split and passed to LightGBM.
- Type:
RandomState
Examples
>>> import numpy as np >>> from yikit.models import GBDTRegressor >>> rng = np.random.RandomState(0) >>> X = rng.normal(size=(100, 5)) >>> y = X @ rng.normal(size=5) + rng.normal(scale=0.1, size=100) >>> model = GBDTRegressor(n_estimators=500, random_state=0).fit(X, y) >>> model.predict(X).shape (100,) >>> model.best_iteration_ < 500 True
Notes
This class requires the ‘lightgbm’ package to be installed.
- fit(X: ArrayLike, y: ArrayLike) GBDTRegressor#
Fit the model with early stopping on a held-out split.
- Parameters:
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,)) – Target values.
- Returns:
The fitted estimator.
- Return type:
- predict(X: ArrayLike) NDArray[Any]#
Predict with the best iteration found by early stopping.
- Parameters:
X (array-like of shape (n_samples, n_features)) – Samples.
- Returns:
Predicted values.
- Return type:
ndarray of shape (n_samples,)
- set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') GBDTRegressor#
Configure whether metadata should be requested to be passed to the
scoremethod.Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with
enable_metadata_routing=True(seesklearn.set_config()). Please check the User Guide on how the routing mechanism works.The options for each parameter are:
True: metadata is requested, and passed toscoreif provided. The request is ignored if metadata is not provided.False: metadata is not requested and the meta-estimator will not pass it toscore.None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.str: metadata should be passed to the meta-estimator with this given alias instead of the original name.
The default (
sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.Added in version 1.3.
- Parameters:
sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for
sample_weightparameter inscore.- Returns:
self – The updated object.
- Return type:
object
- class yikit.models.Objective(estimator: Any, X: ArrayLike, y: ArrayLike, custom_params: Callable[[optuna.trial.BaseTrial], Mapping[str, Any]] | None = None, fixed_params: Mapping[str, Any] | None = None, cv: int | BaseCrossValidator | Iterable[Any] = 5, random_state: int | np.random.RandomState | None = None, scoring: str | Callable[..., float] | None = None, n_jobs: int | None = None)#
Bases:
objectObjective function of optuna that cross-validates an estimator.
Each trial copies
estimatorand sets on the copy only the values searched in the trial, the integer of therandom_staterule andfixed_params, in this order of priority from the lowest. Every other parameter keeps the value ofestimator: in particularn_jobsand the values that yikit recommends (seeRecommendedParams) are never set unless they are infixed_params. The copy is cross-validated, and the mean of its test scores is the value of the trial.The searched parameters come from the search space table of yikit, which knows
LGBMRegressorof LightGBM,GBDTRegressorof yikit,RandomForestRegressor,SVR,LinearSVR,MLPRegressor,NGBRegressorof ngboost,PLSRegression,Ridge,LassoandElasticNet(and their subclasses). The last step of aPipelineand theregressorof aTransformedTargetRegressorare followed, so a nested model is searched with prefixed names such asregressor__svr__C.custom_paramsreplaces the table.random_staterule: everyrandom_stateparameter ofestimatorand of the estimators nested in it whose value is None (or the globalRandomStateof NumPy, which ngboost puts in place of None) is set tomodel_random_state, one integer drawn fromrandom_statewhen the objective is created. The values given by the user are kept, and models without arandom_stateparameter get none. An estimator given infixed_paramsfollows the same rule in place of the one it replaces.A trial whose model fails to fit in every fold of the cross-validation is reported with a
FitFailedWarningand returns nan, as does a trial whose fit fails in some folds; optuna records such a trial as failed, never chooses it as the best trial, and goes on with the search.- Parameters:
estimator (object) – scikit-learn compatible estimator instance to search, e.g.
sklearn.ensemble.RandomForestRegressor()or aPipeline. It is never modified.X (array-like of shape (n_samples, n_features)) – Features. Its number of columns bounds the parameters limited by the data, such as
n_componentsofPLSRegression.y (array-like of shape (n_samples,)) – Target.
custom_params (callable or None, default=None) – Function that takes an optuna trial and returns a dict from parameter names to values, suggesting the values with the trial, e.g.
lambda trial: {"C": trial.suggest_float("C", 0.1, 10)}. When it returns a non-empty dict, the dict is used instead of the search space table, and its names are used as they are (no prefix is added). An empty dict means that the table is used.fixed_params (Mapping or None, default=None) – Parameter values to set on every model, prefixed like the searched names for nested estimators. They take priority over the searched values and the
random_staterule. A fixed name covers itself and every name under it, and the names of the table that it covers are not searched. When it holds an estimator, such as{"Base": DecisionTreeRegressor(max_depth=5)}forNGBRegressoror{"svr": SVR(C=10.0)}for aPipeline, therandom_staterule does not set the names under it either (Base__max_depth,svr__C, … are neither searched nor changed), so the parameters of the given estimator are kept. The rule is applied to that estimator instead: itsrandom_stateparameters left as None getmodel_random_state(e.g.Base__random_state), and those given are kept. Names under a fixed estimator thatcustom_paramsreturns are still applied, to the copy of that estimator.Noneand{}mean no fixed value.cv (int, cross-validation generator or iterable, default=5) – Cross-validation splitting strategy passed to
check_cv.random_state (int, RandomState instance or None, default=None) – Seed of the
TPESamplerexposed assamplerand ofmodel_random_state, drawn in this order.scoring (str, callable or None, default=None) – Scoring passed to
check_scoring.Nonemeans thescoremethod ofestimator.n_jobs (int or None, default=None) – Number of jobs of
cross_validate. It is not set on the model.
- estimator#
The estimator passed by the user.
- Type:
object
- X#
Features validated by
check_X_y.- Type:
numpy.ndarray of shape (n_samples, n_features)
- y#
Target validated by
check_X_y.- Type:
numpy.ndarray of shape (n_samples,)
- custom_params#
The
custom_paramspassed by the user.- Type:
callable or None
- fixed_params#
A copy of
fixed_params({}when None was passed).- Type:
dict
- cv#
Cross-validator returned by
check_cv.- Type:
cross-validation generator
- scoring#
Scorer returned by
check_scoring.- Type:
callable
- n_jobs#
Number of jobs of
cross_validate.- Type:
int or None
- sampler#
Sampler seeded from
random_state, to pass tooptuna.create_study.- Type:
optuna.samplers.TPESampler
- model_random_state#
Integer set on the unspecified
random_stateparameters, the same for every trial and forget_best_params.- Type:
int
- param_distributions#
Prefixed distributions of the search space table for
estimatorand the number of columns ofX, without the names covered byfixed_params(the fixed names and the names under them).Nonewhen the table has no row for the model.- Type:
dict or None
- estimator_#
Unfitted model of the last trial, with the parameters of that trial set. It exists after the first trial.
- Type:
object
- Raises:
NotImplementedError – If the table has no search space for the (resolved) model and
custom_paramsis None.ValueError – If a name in
fixed_paramsis not a parameter of the targeted estimator.
Examples
>>> import optuna >>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import ElasticNet >>> from yikit.models import Objective >>> X, y = make_regression(n_samples=50, n_features=5, random_state=334) >>> objective = Objective(ElasticNet(), X, y, random_state=334) >>> sorted(objective.param_distributions) ['alpha', 'l1_ratio'] >>> study = optuna.create_study( ... direction="maximize", sampler=objective.sampler ... ) >>> study.optimize(objective, n_trials=10) >>> best_estimator = objective.get_best_estimator(study).fit(X, y)
RidgeandLassoare searched in the same way (they replace the removedLinearModelRegressor):>>> from sklearn.linear_model import Lasso, Ridge >>> sorted(Objective(Ridge(), X, y).param_distributions) ['alpha'] >>> sorted(Objective(Lasso(), X, y).param_distributions) ['alpha']
An
SVRscaled by aPipelineand aTransformedTargetRegressor(which replace the removedSupportVectorRegressor) is searched with prefixed names.fixed_paramssets thegammathatSupportVectorRegressorused:>>> from sklearn.compose import TransformedTargetRegressor >>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import StandardScaler >>> from sklearn.svm import SVR >>> estimator = TransformedTargetRegressor( ... regressor=make_pipeline(StandardScaler(), SVR()), ... transformer=StandardScaler(), ... ) >>> objective = Objective( ... estimator, ... X, ... y, ... fixed_params={"regressor__svr__gamma": "auto"}, ... random_state=334, ... ) >>> sorted(objective.param_distributions) ['regressor__svr__C', 'regressor__svr__epsilon'] >>> study = optuna.create_study( ... direction="maximize", sampler=objective.sampler ... ) >>> study.optimize(objective, n_trials=10) >>> best_params = objective.get_best_params(study) >>> sorted(best_params) ['regressor__svr__C', 'regressor__svr__epsilon', 'regressor__svr__gamma'] >>> best_estimator = objective.get_best_estimator(study) >>> best_estimator.regressor.named_steps["svr"].gamma 'auto' >>> best_estimator = best_estimator.fit(X, y)
- get_best_estimator(study: Study) Any#
Return an unfitted copy of
estimatorwith the best parameters.- Parameters:
study (optuna.study.Study) – Study optimized with this objective. It must have a completed trial.
- Returns:
Unfitted copy of
estimatorwithget_best_params(study)set. The other parameters keep the values ofestimator.- Return type:
object
- Raises:
ValueError – If
studyhas no completed trial (raised by optuna).
- get_best_params(study: Study) dict[str, Any]#
Return the parameters of the model of the best trial.
They are built again from the values of the best trial, in the same way as in the trial (through
optuna.trial.FixedTrial), so they hold the names returned bycustom_paramseven when they differ from the names given tosuggest_*.- Parameters:
study (optuna.study.Study) – Study optimized with this objective. It must have a completed trial.
- Returns:
The values searched in the best trial, the integer of the
random_staterule for the unspecifiedrandom_stateparameters, andfixed_params, which win over the others. The other parameters ofestimatorare not included.- Return type:
dict of str to object
- Raises:
ValueError – If
studyhas no completed trial (raised by optuna).
- class yikit.models.ParamDistributions(estimator: Any, custom_params: Mapping[str, BaseDistribution] | Callable[[optuna.trial.BaseTrial], Mapping[str, BaseDistribution]] | None = None, *, n_features: int | None = None)#
Bases:
dictDistributions of the search space to pass to
OptunaSearchCV.A dict from parameter names to optuna distributions, built from the same search space table as
Objective: for the same estimator and the same number of features, it holds the same names and the same distributions as the trials ofObjective. It is passed as it is asparam_distributionsofOptunaSearchCVof optuna-integration, and keeps its contents and type throughcopy.deepcopyandsklearn.base.clone.The table knows
LGBMRegressorof LightGBM,GBDTRegressorof yikit,RandomForestRegressor,SVR,LinearSVR,MLPRegressor,NGBRegressorof ngboost,PLSRegression,Ridge,LassoandElasticNet(and their subclasses). The last step of aPipelineand theregressorof aTransformedTargetRegressorare followed, so a nested model gets prefixed names such asregressor__svr__C, whichOptunaSearchCVpasses toset_paramsof the estimator. Only searched parameters are included: fixed values, such asn_jobs,random_stateor the values ofRecommendedParams, are set on the estimator itself, whose other parametersOptunaSearchCVkeeps.set_paramsofNGBRegressor(ngboost 0.4.0 or later) ignores nested names such asBase__max_depth, soOptunaSearchCVcannot apply them. AUserWarninglists such names;Objectiveapplies them.- Parameters:
estimator (object) – scikit-learn compatible estimator instance to search, e.g.
sklearn.svm.SVR()or aPipelinewhose last step is one. It is not modified.custom_params (Mapping, callable or None, default=None) – Mapping from parameter names to optuna distributions, e.g.
{"C": FloatDistribution(0.1, 10.0, log=True)}, or a function that takes an optuna trial and returns such a mapping. The function is called once withoptuna.trial.FixedTrial({}), so it must return distributions instead of suggesting values (a function that suggests values, ascustom_paramsofObjectivedoes, raises the error ofFixedTrial). A non-empty mapping is used instead of the search space table, and its names are used as they are (no prefix is added). None or an empty mapping means that the table is used.n_features (int or None, default=None) – Number of features of the data passed to
estimator(the outermost one when nested). Keyword-only. When given, the upper bound of the parameters limited by the data,n_componentsofPLSRegression, becomesmin(10, n_features). When None, the bound of the table (10) is kept, andOptunaSearchCVrecords the trials that exceed the number of features as failed and goes on with the search.
- estimator#
The estimator passed by the user.
- Type:
object
- custom_params#
The
custom_paramspassed by the user.- Type:
Mapping, callable or None
- n_features#
The
n_featurespassed by the user.- Type:
int or None
- Raises:
NotImplementedError – If the table has no search space for the (resolved) model and
custom_paramsgives no distribution.TypeError – If
custom_params, or what its function returns, is not a mapping, or if one of its values is not an optuna distribution. Thefixed_paramsandrandom_statearguments of yikit 0.4.0-rc.0 were removed, and passing them raises aTypeErrortoo.ValueError – If
n_featuresis smaller than 1, or if aPipelineon the way has no steps.
- Warns:
UserWarning – If the resolved model ignores nested names in its own
set_params(NGBRegressorof ngboost 0.4.0 or later) and the distributions hold such names, e.g.Base__max_depth.
See also
ObjectiveObjective function of optuna that searches the same table.
Examples
>>> from sklearn.datasets import make_regression >>> from sklearn.linear_model import ElasticNet >>> from yikit.models import ParamDistributions >>> from optuna_integration import OptunaSearchCV >>> X, y = make_regression(n_samples=50, n_features=5, random_state=334) >>> estimator = ElasticNet() >>> param_distributions = ParamDistributions( ... estimator, n_features=X.shape[1] ... ) >>> sorted(param_distributions) ['alpha', 'l1_ratio'] >>> search = OptunaSearchCV( ... estimator, ... param_distributions=param_distributions, ... n_trials=10, ... random_state=334, ... ).fit(X, y) >>> sorted(search.best_params_) ['alpha', 'l1_ratio']
RidgeandLassoare searched in the same way (they replace the removedLinearModelRegressor):>>> from sklearn.linear_model import Lasso, Ridge >>> sorted(ParamDistributions(Ridge())) ['alpha'] >>> sorted(ParamDistributions(Lasso())) ['alpha']
An
SVRscaled by aPipelineand aTransformedTargetRegressor(which replace the removedSupportVectorRegressor) gets prefixed names. ThegammathatSupportVectorRegressorused is set on theSVRitself, andOptunaSearchCVkeeps it:>>> from sklearn.compose import TransformedTargetRegressor >>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import StandardScaler >>> from sklearn.svm import SVR >>> estimator = TransformedTargetRegressor( ... regressor=make_pipeline(StandardScaler(), SVR(gamma="auto")), ... transformer=StandardScaler(), ... ) >>> param_distributions = ParamDistributions( ... estimator, n_features=X.shape[1] ... ) >>> sorted(param_distributions) ['regressor__svr__C', 'regressor__svr__epsilon'] >>> search = OptunaSearchCV( ... estimator, ... param_distributions=param_distributions, ... n_trials=5, ... random_state=334, ... ).fit(X, y) >>> sorted(search.best_params_) ['regressor__svr__C', 'regressor__svr__epsilon'] >>> search.best_estimator_.regressor_.named_steps["svr"].gamma 'auto'
The upper bound of
n_componentsofPLSRegressionfollowsn_features, andcustom_paramsreplaces the table:>>> from optuna.distributions import FloatDistribution >>> from sklearn.cross_decomposition import PLSRegression >>> ParamDistributions(PLSRegression())["n_components"].high 10 >>> ParamDistributions(PLSRegression(), n_features=3)["n_components"].high 3 >>> sorted( ... ParamDistributions( ... SVR(), {"C": FloatDistribution(0.1, 10.0, log=True)} ... ) ... ) ['C']
- class yikit.models.RecommendedParams(estimator: Any)#
Bases:
dictValues that yikit recommends to fix for an estimator, as a dict.
A dict from parameter names to the values that yikit recommends for
estimator. The model is found in the same way as for the search space ofObjectiveandParamDistributions: the last step of aPipelineand theregressorof aTransformedTargetRegressorare followed, and the names get the same prefix, such asregressor__svr__gamma. It is empty, without an error, when yikit recommends nothing for the model, including models without a search space.The values are opt-in.
ObjectiveandParamDistributionsnever set them by themselves, and this class does not modifyestimator. Pass them yourself, asfixed_paramsofObjectiveor toset_paramsof the estimator (OptunaSearchCVkeeps the parameters of the estimator that it does not search). The table starts small: it only holdsgamma="auto"forSVR(and its subclasses), the value that the removedSupportVectorRegressorused, and may get more rows in later versions.As a plain dict, it can be unpacked with
**, compares equal to a dict with the same items, and keeps its contents and type throughcopy.deepcopyandpickle. Each instance holds new copies of the values, so changing it does not change the table.- Parameters:
estimator (object) – scikit-learn compatible estimator instance to look up, e.g.
sklearn.svm.SVR()or aPipelinewhose last step is one. It is not modified.
- estimator#
The estimator passed by the user.
- Type:
object
- Raises:
ValueError – If a
Pipelineon the way has no steps.
See also
ObjectiveObjective function of optuna that takes them as
fixed_params.ParamDistributionsDistributions of the search space to pass to
OptunaSearchCV.
Examples
>>> from sklearn.svm import SVR >>> from yikit.models import RecommendedParams >>> RecommendedParams(SVR()) RecommendedParams({'gamma': 'auto'}) >>> RecommendedParams(SVR()) == {"gamma": "auto"} True >>> from sklearn.ensemble import RandomForestRegressor >>> RecommendedParams(RandomForestRegressor()) RecommendedParams({})
A nested
SVRgets prefixed names:>>> from sklearn.compose import TransformedTargetRegressor >>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import StandardScaler >>> estimator = TransformedTargetRegressor( ... regressor=make_pipeline(StandardScaler(), SVR()), ... transformer=StandardScaler(), ... ) >>> RecommendedParams(estimator) == {"regressor__svr__gamma": "auto"} True
Set the values on the estimator, e.g. before passing it to
OptunaSearchCVwithParamDistributions:>>> estimator = estimator.set_params(**RecommendedParams(estimator)) >>> estimator.regressor.named_steps["svr"].gamma 'auto'
Or fix them in every trial of
Objective, which leaves the passed estimator as it is:>>> import optuna >>> from sklearn.datasets import make_regression >>> from yikit.models import Objective >>> X, y = make_regression(n_samples=50, n_features=5, random_state=334) >>> estimator = SVR() >>> objective = Objective( ... estimator, ... X, ... y, ... fixed_params=RecommendedParams(estimator), ... random_state=334, ... ) >>> sorted(objective.param_distributions) ['C', 'epsilon'] >>> study = optuna.create_study( ... direction="maximize", sampler=objective.sampler ... ) >>> study.optimize(objective, n_trials=5) >>> objective.get_best_estimator(study).gamma 'auto' >>> estimator.gamma 'scale'