yikit.feature_selection package#

Feature selection methods for machine learning.

This module provides filter methods and wrapper methods for feature selection, including correlation-based filtering and Boruta algorithm.

class yikit.feature_selection.BorutaPy(estimator, n_estimators: int | Literal['auto'] = 'auto', perc: float | Literal['auto'] = 'auto', alpha: float = 0.05, two_step: bool = True, max_iter: int = 100, random_state: RandomState | int | None = None, verbose: int = 1, max_shuf: int = 10000, n_jobs: int | None = None)#

Bases: BorutaPy

Boruta all-relevant feature selection with an automatic percentile.

This class extends boruta.BorutaPy of scikit-learn-contrib/boruta_py (BSD 3-Clause License), and this docstring is adapted from the one of boruta. Boruta is an all-relevant feature selection method: it tries to find all the features that carry information usable for prediction, rather than a compact subset on which a model has a minimal error. In each iteration, estimator is fitted on the features and on their shuffled copies (the shadow features), and a feature scores a hit when its importance is above the perc percentile of the importances of the shadow features. The features with significantly many hits are confirmed, and those with significantly few hits are rejected. With two_step=True, the p-values are corrected with the Benjamini Hochberg FDR across the features still in the selection in each iteration, and then with Bonferroni across the iterations.

Compared with boruta, this class adds:

  • perc="auto" (the default), which sets the percentile from the largest Pearson correlation between shuffled features and the target (see perc and max_shuf), in both fit and fit_transform (and therefore in a Pipeline);

  • n_estimators="auto" as the default;

  • a progress bar (tqdm, when installed) and the logger of yikit instead of print (see verbose);

  • transform of a pandas.DataFrame without return_df, which returns a NumPy array of the selected columns.

It follows the conventions of scikit-learn: the parameters are stored as passed (get_params returns them, also after a fit), and they are read in every fit. Pruned trees with a depth of 3 to 7 are recommended as estimator.

Parameters:
  • estimator (object) – A supervised estimator with a fit method that sets feature_importances_. Important features must have high absolute values in feature_importances_. boruta sets n_estimators and random_state of this estimator in each iteration.

  • n_estimators (int or "auto", default="auto") – The number of estimators of estimator. If “auto”, it is set in each iteration from the number of features that are not rejected and the max_depth of estimator (10 when it is None), so that fewer trees are used as features are rejected.

  • perc (float or "auto", default="auto") – The percentile of the importances of the shadow features that a feature must exceed to score a hit. 100 is the maximum, as in the original Boruta; it tends to be too strict. A lower perc selects more features, including more irrelevant ones. If “auto”, it is calculated from the data in every fit as 100 * (1 - r_ccmax_), where r_ccmax_ is the largest Pearson correlation between randomly shuffled features and the target (see max_shuf). The value used is stored in perc_; this parameter is never overwritten, so get_params and clone() keep “auto”.

  • alpha (float, default=0.05) – The level at which the corrected p-values are rejected in both correction steps.

  • two_step (bool, default=True) – Whether to use the two-step correction. If False, only the Bonferroni correction of the original Boruta is used.

  • max_iter (int, default=100) – The maximum number of iterations.

  • random_state (int, RandomState instance or None, default=None) – Controls the shuffles of perc="auto", the shuffles of the shadow features and the random_state given to estimator. The value is stored as passed, and each fit makes its random number generator from it with check_random_state(). Pass an int to get the same result in every fit (also of a clone). A RandomState instance is used and advanced, and None uses the global random number generator of NumPy.

  • verbose (int, default=1) –

    Controls what is shown during a fit. It is read at the start of each fit, so set_params(verbose=...) applies to the next fit.

    • 0: nothing.

    • 1: a progress bar of the iterations (and of the shuffles of perc="auto") when tqdm is installed, otherwise the iteration number is logged. The value of perc="auto" and a summary at the end are logged.

    • 2: as 1, and the numbers of confirmed, tentative and rejected features are logged in each iteration.

  • max_shuf (int, default=10000) – The number of shuffles used to calculate r_ccmax_ when perc is “auto”. Each shuffle has its own seed, drawn from random_state before the shuffles are split among the jobs, so perc_ does not depend on n_jobs.

  • n_jobs (int or None, default=None) – The number of jobs that run the shuffles of perc="auto" in parallel with joblib. None means 1 unless in a joblib.parallel_backend context, and -1 means all processors.

n_features_#

The number of selected (confirmed) features.

Type:

int

support_#

The mask of the selected features; only the confirmed ones are True.

Type:

ndarray of shape (n_features,)

support_weak_#

The mask of the tentative features, which have not gained enough support during the max_iter iterations.

Type:

ndarray of shape (n_features,)

ranking_#

The ranking of the features: 1 for the confirmed features, 2 for the tentative ones, and from 3 (2 when there are no tentative features) for the rejected ones, based on their importances through the iterations.

Type:

ndarray of shape (n_features,)

importance_history_#

The importances of the features in each iteration, after a first row of zeros. A feature has NaN in the iterations after it was rejected.

Type:

ndarray of shape (n_iterations + 1, n_features)

perc_#

The percentile used in the last fit: the float calculated from the data when perc="auto", otherwise perc itself as passed.

Type:

float or int

r_ccmax_#

The largest Pearson correlation between the shuffled features and the target. Only set by a fit with perc="auto".

Type:

float

pears_#

The Pearson correlations between the shuffled features and the target, one array for each of the max_shuf shuffles. Only set by a fit with perc="auto".

Type:

list of array.array of shape (n_features,)

See also

FilterSelector

Remove features that are highly correlated with another feature.

References

Kursa M., Rudnicki W., “Feature Selection with the Boruta Package”, Journal of Statistical Software, Vol. 36, Issue 11, Sep 2010. https://doi.org/10.18637/jss.v036.i11

scikit-learn-contrib/boruta_py: scikit-learn-contrib/boruta_py

Examples

The first 3 of the 6 features are informative. The small max_iter and max_shuf only keep this example fast:

>>> from sklearn.datasets import make_regression
>>> from sklearn.ensemble import RandomForestRegressor
>>> from yikit.feature_selection import BorutaPy
>>> X, y = make_regression(
...     n_samples=100,
...     n_features=6,
...     n_informative=3,
...     shuffle=False,
...     random_state=334,
... )
>>> selector = BorutaPy(
...     RandomForestRegressor(max_depth=5, n_jobs=1, random_state=334),
...     max_iter=20,
...     max_shuf=100,
...     verbose=0,
...     random_state=334,
... )
>>> X_selected = selector.fit_transform(X, y)
>>> selector.support_
array([ True,  True,  True, False, False, False])
>>> X_selected.shape
(100, 3)

perc="auto" is resolved in perc_, and the parameters keep the passed values, so a fit of a clone gives the same result:

>>> from sklearn.base import clone
>>> selector.get_params()["perc"], selector.get_params()["random_state"]
('auto', 334)
>>> bool(0 < selector.perc_ < 100)
True
>>> bool(clone(selector).fit(X, y).perc_ == selector.perc_)
True
fit(X, y)#

Fit the Boruta feature selection with estimator.

This docstring is adapted from the one of scikit-learn-contrib/boruta_py (BSD 3-Clause License).

perc="auto" is resolved in _fit, the same path that fit_transform takes. The resolved value is stored in perc_ and the perc parameter is left unchanged.

Parameters:
  • X (array-like of shape (n_samples, n_features)) – The training input samples.

  • y (array-like of shape (n_samples,)) – The target values.

Returns:

self – The fitted selector.

Return type:

BorutaPy

get_support(weak: bool = False) → ndarray#

Return a boolean mask of the selected features.

The mask marks the same features as the columns that transform(X, weak=weak) keeps.

Parameters:

weak (bool, default=False) – If False, only the confirmed features are True (support_). If True, the tentative features are also True (support_ | support_weak_).

Returns:

support – The boolean mask of the selected features. With weak=False, this is support_ itself (not a copy).

Return type:

ndarray of shape (n_features,)

Raises:

sklearn.exceptions.NotFittedError – If the selector is not fitted yet.

set_transform_request(*, return_df: bool | None | str = '$UNCHANGED$', weak: bool | None | str = '$UNCHANGED$') → BorutaPy#

Configure whether metadata should be requested to be passed to the transform method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
  • return_df (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for return_df parameter in transform.

  • weak (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for weak parameter in transform.

Returns:

self – The updated object.

Return type:

object

class yikit.feature_selection.FilterSelector(r: float = 0.9, alpha: float = 0.05, verbose: int | bool = True, n_jobs: int | None = None)#

Bases: SelectorMixin, BaseEstimator

Remove features that are strongly correlated with other features.

A pair of features is strongly correlated when the absolute value of their Pearson correlation coefficient is larger than r and its p-value is smaller than alpha, so a strong negative correlation counts as well as a strong positive one. The feature in the largest number of strongly correlated pairs is removed (the first one on ties), and this is repeated with the remaining features until no strongly correlated pair is left.

A constant feature is treated as uncorrelated with every other feature, so it is never removed by this selector.

Parameters:
  • r (float, default=0.9) – The threshold of the absolute value of the correlation coefficient. Pairs whose absolute correlation is larger than this are tested for significance and may be removed.

  • alpha (float, default=0.05) – The significance level. The correlation of a pair is significant when its p-value is smaller than this. It is recommended not to change it without a specific reason.

  • verbose (bool or int, default=True) – If positive, show the progress of computing the p-values: a tqdm progress bar when tqdm is installed, otherwise log messages.

  • n_jobs (int or None, default=None) – The number of jobs to compute the p-values in parallel with joblib. None means 1 unless in a joblib.parallel_backend context; -1 means all the processors.

corr_#

corr_[i, j, 0] is the Pearson correlation coefficient of the features i and j and corr_[i, j, 1] is its p-value. The p-values are computed only for the pairs whose absolute correlation is larger than r; the others are 1.0. The diagonal holds 1.0 and 0.0, and a pair with a constant feature holds 0.0 and 1.0.

Type:

numpy.ndarray of shape (n_features_in_, n_features_in_, 2)

support_#

The boolean mask of the selected (kept) features.

Type:

numpy.ndarray of shape (n_features_in_,)

n_features_in_#

The number of features seen during fit.

Type:

int

feature_names_in_#

The names of the features seen during fit. Defined only when X has feature names that are all strings.

Type:

numpy.ndarray of shape (n_features_in_,)

Examples

>>> import numpy as np
>>> import pandas as pd
>>> from yikit.feature_selection import FilterSelector
>>> rng = np.random.RandomState(334)
>>> x = rng.normal(size=50)
>>> X = pd.DataFrame(
...     {
...         "a": x,
...         "b": x + 0.1 * rng.normal(size=50),
...         "c": -x + 0.1 * rng.normal(size=50),
...         "d": rng.normal(size=50),
...     }
... )

a, b and c are strongly correlated with each other (c negatively), so only one of them is kept.

>>> selector = FilterSelector(verbose=False).fit(X)
>>> selector.get_support()
array([False, False,  True,  True])
>>> selector.get_feature_names_out()
array(['c', 'd'], dtype=object)
>>> selector.transform(X).shape
(50, 2)
fit(X: Any, y: Any = None) → FilterSelector#

Find the features to keep.

Parameters:
  • X (array-like of shape (n_samples, n_features)) – The feature matrix, such as a pandas DataFrame or a 2D numpy array.

  • y (None) – Ignored. Present for API consistency.

Returns:

self – The fitted selector.

Return type:

FilterSelector