yikit.feature_selection package#
Feature selection methods for machine learning.
This module provides filter methods and wrapper methods for feature selection, including correlation-based filtering and Boruta algorithm.
- class yikit.feature_selection.BorutaPy(estimator, n_estimators: int | Literal['auto'] = 'auto', perc: float | Literal['auto'] = 'auto', alpha: float = 0.05, two_step: bool = True, max_iter: int = 100, random_state: RandomState | int | None = None, verbose: int = 1, max_shuf: int = 10000, n_jobs: int | None = None)#
Bases:
BorutaPyBoruta all-relevant feature selection with an automatic percentile.
This class extends
boruta.BorutaPyof scikit-learn-contrib/boruta_py (BSD 3-Clause License), and this docstring is adapted from the one of boruta. Boruta is an all-relevant feature selection method: it tries to find all the features that carry information usable for prediction, rather than a compact subset on which a model has a minimal error. In each iteration,estimatoris fitted on the features and on their shuffled copies (the shadow features), and a feature scores a hit when its importance is above thepercpercentile of the importances of the shadow features. The features with significantly many hits are confirmed, and those with significantly few hits are rejected. Withtwo_step=True, the p-values are corrected with the Benjamini Hochberg FDR across the features still in the selection in each iteration, and then with Bonferroni across the iterations.Compared with boruta, this class adds:
perc="auto"(the default), which sets the percentile from the largest Pearson correlation between shuffled features and the target (seepercandmax_shuf), in bothfitandfit_transform(and therefore in aPipeline);n_estimators="auto"as the default;a progress bar (tqdm, when installed) and the logger of yikit instead of
print(seeverbose);transformof apandas.DataFramewithoutreturn_df, which returns a NumPy array of the selected columns.
It follows the conventions of scikit-learn: the parameters are stored as passed (
get_paramsreturns them, also after a fit), and they are read in every fit. Pruned trees with a depth of 3 to 7 are recommended asestimator.- Parameters:
estimator (object) – A supervised estimator with a
fitmethod that setsfeature_importances_. Important features must have high absolute values infeature_importances_. boruta setsn_estimatorsandrandom_stateof this estimator in each iteration.n_estimators (int or "auto", default="auto") – The number of estimators of
estimator. If “auto”, it is set in each iteration from the number of features that are not rejected and themax_depthofestimator(10 when it is None), so that fewer trees are used as features are rejected.perc (float or "auto", default="auto") – The percentile of the importances of the shadow features that a feature must exceed to score a hit. 100 is the maximum, as in the original Boruta; it tends to be too strict. A lower
percselects more features, including more irrelevant ones. If “auto”, it is calculated from the data in every fit as100 * (1 - r_ccmax_), wherer_ccmax_is the largest Pearson correlation between randomly shuffled features and the target (seemax_shuf). The value used is stored inperc_; this parameter is never overwritten, soget_paramsandclone()keep “auto”.alpha (float, default=0.05) – The level at which the corrected p-values are rejected in both correction steps.
two_step (bool, default=True) – Whether to use the two-step correction. If False, only the Bonferroni correction of the original Boruta is used.
max_iter (int, default=100) – The maximum number of iterations.
random_state (int, RandomState instance or None, default=None) – Controls the shuffles of
perc="auto", the shuffles of the shadow features and therandom_stategiven toestimator. The value is stored as passed, and each fit makes its random number generator from it withcheck_random_state(). Pass an int to get the same result in every fit (also of a clone). A RandomState instance is used and advanced, and None uses the global random number generator of NumPy.verbose (int, default=1) –
Controls what is shown during a fit. It is read at the start of each fit, so
set_params(verbose=...)applies to the next fit.0: nothing.
1: a progress bar of the iterations (and of the shuffles of
perc="auto") when tqdm is installed, otherwise the iteration number is logged. The value ofperc="auto"and a summary at the end are logged.2: as 1, and the numbers of confirmed, tentative and rejected features are logged in each iteration.
max_shuf (int, default=10000) – The number of shuffles used to calculate
r_ccmax_whenpercis “auto”. Each shuffle has its own seed, drawn fromrandom_statebefore the shuffles are split among the jobs, soperc_does not depend onn_jobs.n_jobs (int or None, default=None) – The number of jobs that run the shuffles of
perc="auto"in parallel with joblib. None means 1 unless in ajoblib.parallel_backendcontext, and -1 means all processors.
- n_features_#
The number of selected (confirmed) features.
- Type:
int
- support_#
The mask of the selected features; only the confirmed ones are True.
- Type:
ndarray of shape (n_features,)
- support_weak_#
The mask of the tentative features, which have not gained enough support during the
max_iteriterations.- Type:
ndarray of shape (n_features,)
- ranking_#
The ranking of the features: 1 for the confirmed features, 2 for the tentative ones, and from 3 (2 when there are no tentative features) for the rejected ones, based on their importances through the iterations.
- Type:
ndarray of shape (n_features,)
- importance_history_#
The importances of the features in each iteration, after a first row of zeros. A feature has NaN in the iterations after it was rejected.
- Type:
ndarray of shape (n_iterations + 1, n_features)
- perc_#
The percentile used in the last fit: the float calculated from the data when
perc="auto", otherwisepercitself as passed.- Type:
float or int
- r_ccmax_#
The largest Pearson correlation between the shuffled features and the target. Only set by a fit with
perc="auto".- Type:
float
- pears_#
The Pearson correlations between the shuffled features and the target, one array for each of the
max_shufshuffles. Only set by a fit withperc="auto".- Type:
list of array.array of shape (n_features,)
See also
FilterSelectorRemove features that are highly correlated with another feature.
References
Kursa M., Rudnicki W., “Feature Selection with the Boruta Package”, Journal of Statistical Software, Vol. 36, Issue 11, Sep 2010. https://doi.org/10.18637/jss.v036.i11
scikit-learn-contrib/boruta_py: scikit-learn-contrib/boruta_py
Examples
The first 3 of the 6 features are informative. The small
max_iterandmax_shufonly keep this example fast:>>> from sklearn.datasets import make_regression >>> from sklearn.ensemble import RandomForestRegressor >>> from yikit.feature_selection import BorutaPy >>> X, y = make_regression( ... n_samples=100, ... n_features=6, ... n_informative=3, ... shuffle=False, ... random_state=334, ... ) >>> selector = BorutaPy( ... RandomForestRegressor(max_depth=5, n_jobs=1, random_state=334), ... max_iter=20, ... max_shuf=100, ... verbose=0, ... random_state=334, ... ) >>> X_selected = selector.fit_transform(X, y) >>> selector.support_ array([ True, True, True, False, False, False]) >>> X_selected.shape (100, 3)
perc="auto"is resolved inperc_, and the parameters keep the passed values, so a fit of a clone gives the same result:>>> from sklearn.base import clone >>> selector.get_params()["perc"], selector.get_params()["random_state"] ('auto', 334) >>> bool(0 < selector.perc_ < 100) True >>> bool(clone(selector).fit(X, y).perc_ == selector.perc_) True
- fit(X, y)#
Fit the Boruta feature selection with
estimator.This docstring is adapted from the one of scikit-learn-contrib/boruta_py (BSD 3-Clause License).
perc="auto"is resolved in_fit, the same path thatfit_transformtakes. The resolved value is stored inperc_and thepercparameter is left unchanged.- Parameters:
X (array-like of shape (n_samples, n_features)) – The training input samples.
y (array-like of shape (n_samples,)) – The target values.
- Returns:
self – The fitted selector.
- Return type:
- get_support(weak: bool = False) ndarray#
Return a boolean mask of the selected features.
The mask marks the same features as the columns that
transform(X, weak=weak)keeps.- Parameters:
weak (bool, default=False) – If False, only the confirmed features are True (
support_). If True, the tentative features are also True (support_ | support_weak_).- Returns:
support – The boolean mask of the selected features. With
weak=False, this issupport_itself (not a copy).- Return type:
ndarray of shape (n_features,)
- Raises:
sklearn.exceptions.NotFittedError – If the selector is not fitted yet.
- set_transform_request(*, return_df: bool | None | str = '$UNCHANGED$', weak: bool | None | str = '$UNCHANGED$') BorutaPy#
Configure whether metadata should be requested to be passed to the
transformmethod.Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with
enable_metadata_routing=True(seesklearn.set_config()). Please check the User Guide on how the routing mechanism works.The options for each parameter are:
True: metadata is requested, and passed totransformif provided. The request is ignored if metadata is not provided.False: metadata is not requested and the meta-estimator will not pass it totransform.None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.str: metadata should be passed to the meta-estimator with this given alias instead of the original name.
The default (
sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.Added in version 1.3.
- Parameters:
return_df (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for
return_dfparameter intransform.weak (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for
weakparameter intransform.
- Returns:
self – The updated object.
- Return type:
object
- class yikit.feature_selection.FilterSelector(r: float = 0.9, alpha: float = 0.05, verbose: int | bool = True, n_jobs: int | None = None)#
Bases:
SelectorMixin,BaseEstimatorRemove features that are strongly correlated with other features.
A pair of features is strongly correlated when the absolute value of their Pearson correlation coefficient is larger than
rand its p-value is smaller thanalpha, so a strong negative correlation counts as well as a strong positive one. The feature in the largest number of strongly correlated pairs is removed (the first one on ties), and this is repeated with the remaining features until no strongly correlated pair is left.A constant feature is treated as uncorrelated with every other feature, so it is never removed by this selector.
- Parameters:
r (float, default=0.9) – The threshold of the absolute value of the correlation coefficient. Pairs whose absolute correlation is larger than this are tested for significance and may be removed.
alpha (float, default=0.05) – The significance level. The correlation of a pair is significant when its p-value is smaller than this. It is recommended not to change it without a specific reason.
verbose (bool or int, default=True) – If positive, show the progress of computing the p-values: a tqdm progress bar when tqdm is installed, otherwise log messages.
n_jobs (int or None, default=None) – The number of jobs to compute the p-values in parallel with joblib. None means 1 unless in a
joblib.parallel_backendcontext; -1 means all the processors.
- corr_#
corr_[i, j, 0]is the Pearson correlation coefficient of the featuresiandjandcorr_[i, j, 1]is its p-value. The p-values are computed only for the pairs whose absolute correlation is larger thanr; the others are 1.0. The diagonal holds 1.0 and 0.0, and a pair with a constant feature holds 0.0 and 1.0.- Type:
numpy.ndarray of shape (n_features_in_, n_features_in_, 2)
- support_#
The boolean mask of the selected (kept) features.
- Type:
numpy.ndarray of shape (n_features_in_,)
- n_features_in_#
The number of features seen during
fit.- Type:
int
- feature_names_in_#
The names of the features seen during
fit. Defined only whenXhas feature names that are all strings.- Type:
numpy.ndarray of shape (n_features_in_,)
Examples
>>> import numpy as np >>> import pandas as pd >>> from yikit.feature_selection import FilterSelector >>> rng = np.random.RandomState(334) >>> x = rng.normal(size=50) >>> X = pd.DataFrame( ... { ... "a": x, ... "b": x + 0.1 * rng.normal(size=50), ... "c": -x + 0.1 * rng.normal(size=50), ... "d": rng.normal(size=50), ... } ... )
a,bandcare strongly correlated with each other (cnegatively), so only one of them is kept.>>> selector = FilterSelector(verbose=False).fit(X) >>> selector.get_support() array([False, False, True, True]) >>> selector.get_feature_names_out() array(['c', 'd'], dtype=object) >>> selector.transform(X).shape (50, 2)
- fit(X: Any, y: Any = None) FilterSelector#
Find the features to keep.
- Parameters:
X (array-like of shape (n_samples, n_features)) – The feature matrix, such as a pandas DataFrame or a 2D numpy array.
y (None) – Ignored. Present for API consistency.
- Returns:
self – The fitted selector.
- Return type: