yikit.metrics package

Contents

yikit.metrics package#

Evaluation metrics that scikit-learn does not provide.

log_likelihood scores NGBoost models and is available only when ngboost is installed. The root mean squared error was removed in 0.4.0; use scikit-learn’s implementation instead.

yikit.metrics.log_likelihood(estimator, X, y)#

Return the mean log-likelihood of y under an NGBoost model.

The value is the average, over the samples, of the log of the predicted probability density (or mass) of each target for a regressor, or of the predicted probability of the true class for a classifier. It does not depend on the scoring rule (Score) the model was trained with. Higher is better.

Parameters:
  • estimator (ngboost.NGBRegressor or ngboost.NGBClassifier) – A fitted NGBoost regressor or classifier.

  • X (array-like of shape (n_samples, n_features)) – Input samples.

  • y (array-like of shape (n_samples,)) – True target values or class labels.

Returns:

The mean log-likelihood. For a model trained with the default Score=LogScore, it equals -estimator.score(X, y).

Return type:

float

Raises:
  • TypeError – If estimator is not an NGBRegressor or an NGBClassifier.

  • ValueError – If ngboost has no LogScore implementation for the model’s distribution.

Notes

estimator.score returns the mean of the scoring rule the model was trained with, so -estimator.score(X, y) is the log-likelihood only for LogScore (for CRPScore it is the negated mean CRPS). This function instead evaluates ngboost’s LogScore implementation for the model’s distribution at the predicted parameters, with equal weights for all samples. For the distributions that ngboost implements with censored scores only (such as LogNormal and Exponential), every target is treated as observed (not censored).

Examples

>>> from ngboost import NGBRegressor
>>> from ngboost.scores import CRPScore
>>> from sklearn.datasets import make_regression
>>> from yikit.metrics import log_likelihood
>>> X, y = make_regression(n_samples=50, random_state=334)
>>> ngb = NGBRegressor(Score=CRPScore, verbose=False)
>>> ngb = ngb.fit(X, y)
>>> log_likelihood(ngb, X, y)