Note

Click here to download the full example code or to run this example in your browser via Binder

Metrics¶

Auto-sklearn supports various built-in metrics, which can be found in the metrics section in the API. However, it is also possible to define your own metric and use it to fit and evaluate your model. The following examples show how to use built-in and self-defined metrics for a classification problem.

import numpy as np

import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

import autosklearn.classification
import autosklearn.metrics

Custom Metrics¶

def accuracy(solution, prediction):
    # custom function defining accuracy
    return np.mean(solution == prediction)


def error(solution, prediction):
    # custom function defining error
    return np.mean(solution != prediction)


def accuracy_wk(solution, prediction, extra_argument):
    # custom function defining accuracy and accepting an additional argument
    assert extra_argument is None
    return np.mean(solution == prediction)


def error_wk(solution, prediction, extra_argument):
    # custom function defining error and accepting an additional argument
    assert extra_argument is None
    return np.mean(solution != prediction)


def metric_which_needs_x(solution, prediction, X_data, consider_col, val_threshold):
    # custom function defining accuracy
    assert X_data is not None
    rel_idx = X_data[:, consider_col] > val_threshold
    return np.mean(solution[rel_idx] == prediction[rel_idx])

Data Loading¶

X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
    X, y, random_state=1
)

Print a list of available metrics¶

print("Available CLASSIFICATION metrics autosklearn.metrics.*:")
print("\t*" + "\n\t*".join(autosklearn.metrics.CLASSIFICATION_METRICS))

print("Available REGRESSION autosklearn.metrics.*:")
print("\t*" + "\n\t*".join(autosklearn.metrics.REGRESSION_METRICS))

Available CLASSIFICATION metrics autosklearn.metrics.*:
        *accuracy
        *balanced_accuracy
        *roc_auc
        *average_precision
        *log_loss
        *precision_macro
        *precision_micro
        *precision_samples
        *precision_weighted
        *recall_macro
        *recall_micro
        *recall_samples
        *recall_weighted
        *f1_macro
        *f1_micro
        *f1_samples
        *f1_weighted
Available REGRESSION autosklearn.metrics.*:
        *mean_absolute_error
        *mean_squared_error
        *root_mean_squared_error
        *mean_squared_log_error
        *median_absolute_error
        *r2

First example: Use predefined accuracy metric¶

print("#" * 80)
print("Use predefined accuracy metric")
scorer = autosklearn.metrics.accuracy
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60,
    seed=1,
    metric=scorer,
)
cls.fit(X_train, y_train)

predictions = cls.predict(X_test)
score = scorer(y_test, predictions)
print(f"Accuracy score {score:.3f} using {scorer.name}")

################################################################################
Use predefined accuracy metric

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:50<00:00,  1.20it/s, The total time budget for this task is 0:01:00]
Accuracy score 0.944 using accuracy

Second example: Use own accuracy metric¶

print("#" * 80)
print("Use self defined accuracy metric")
accuracy_scorer = autosklearn.metrics.make_scorer(
    name="accu",
    score_func=accuracy,
    optimum=1,
    greater_is_better=True,
    needs_proba=False,
    needs_threshold=False,
)
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60,
    seed=1,
    metric=accuracy_scorer,
)
cls.fit(X_train, y_train)

predictions = cls.predict(X_test)
score = accuracy_scorer(y_test, predictions)
print(f"Accuracy score {score:.3f} using {accuracy_scorer.name:s}")

################################################################################
Use self defined accuracy metric

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:50<00:00,  1.19it/s, The total time budget for this task is 0:01:00]
Accuracy score 0.958 using accu

Third example: Use own error metric¶

print("#" * 80)
print("Use self defined error metric")
error_rate = autosklearn.metrics.make_scorer(
    name="error",
    score_func=error,
    optimum=0,
    greater_is_better=False,
    needs_proba=False,
    needs_threshold=False,
)
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60,
    seed=1,
    metric=error_rate,
)
cls.fit(X_train, y_train)

cls.predictions = cls.predict(X_test)
score = error_rate(y_test, predictions)
print(f"Error score {score:.3f} using {error_rate.name:s}")

################################################################################
Use self defined error metric

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  85%|########5 | 51/60 [00:51<00:09,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  87%|########6 | 52/60 [00:52<00:08,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  88%|########8 | 53/60 [00:53<00:07,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  90%|######### | 54/60 [00:54<00:06,  1.04s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  92%|#########1| 55/60 [00:55<00:05,  1.03s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:55<00:00,  1.08it/s, The total time budget for this task is 0:01:00]
Error score -0.042 using error

Fourth example: Use own accuracy metric with additional argument¶

print("#" * 80)
print("Use self defined accuracy with additional argument")
accuracy_scorer = autosklearn.metrics.make_scorer(
    name="accu_add",
    score_func=accuracy_wk,
    optimum=1,
    greater_is_better=True,
    needs_proba=False,
    needs_threshold=False,
    extra_argument=None,
)
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60, per_run_time_limit=30, seed=1, metric=accuracy_scorer
)
cls.fit(X_train, y_train)

predictions = cls.predict(X_test)
score = accuracy_scorer(y_test, predictions)
print(f"Accuracy score {score:.3f} using {accuracy_scorer.name:s}")

################################################################################
Use self defined accuracy with additional argument

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.02s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  85%|########5 | 51/60 [00:51<00:09,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  87%|########6 | 52/60 [00:52<00:08,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  88%|########8 | 53/60 [00:53<00:07,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  90%|######### | 54/60 [00:54<00:06,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:54<00:00,  1.11it/s, The total time budget for this task is 0:01:00]
Accuracy score 0.958 using accu_add

Fifth example: Use own accuracy metric with additional argument¶

print("#" * 80)
print("Use self defined error with additional argument")
error_rate = autosklearn.metrics.make_scorer(
    name="error_add",
    score_func=error_wk,
    optimum=0,
    greater_is_better=True,
    needs_proba=False,
    needs_threshold=False,
    extra_argument=None,
)
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60,
    seed=1,
    metric=error_rate,
)
cls.fit(X_train, y_train)

predictions = cls.predict(X_test)
score = error_rate(y_test, predictions)
print(f"Error score {score:.3f} using {error_rate.name:s}")

################################################################################
Use self defined error with additional argument

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:00,361:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:03,553:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:07,312:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:08,564:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:50,  1.06s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:48,  1.04s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:47,  1.03s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:12,512:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.02s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:13,415:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.02s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:15,768:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:19,584:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:23,676:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:28,532:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:31,917:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:36,948:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.02s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:41,057:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:41,958:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:46,703:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:49<00:00,  1.21it/s, The total time budget for this task is 0:01:00]
Error score 0.615 using error_add

Sixth example: Use a metric with additional argument which also needs xdata¶

"""
Finally, *Auto-sklearn* also support metric that require the train data (aka X_data) to
compute a value. This can be useful if one only cares about the score on a subset of the
data.
"""

accuracy_scorer = autosklearn.metrics.make_scorer(
    name="accu_X",
    score_func=metric_which_needs_x,
    optimum=1,
    greater_is_better=True,
    needs_proba=False,
    needs_X=True,
    needs_threshold=False,
    consider_col=1,
    val_threshold=18.8,
)
cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60,
    seed=1,
    metric=accuracy_scorer,
)
cls.fit(X_train, y_train)

predictions = cls.predict(X_test)
score = metric_which_needs_x(
    y_test,
    predictions,
    X_data=X_test,
    consider_col=1,
    val_threshold=18.8,
)
print(f"Error score {score:.3f} using {accuracy_scorer.name:s}")

Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.01s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:36,313:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:37,670:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:38,890:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:41,358:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:42,354:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
Fitting to the training data: 100%|##########| 60/60 [00:49<00:00,  1.22it/s, The total time budget for this task is 0:01:00]
Error score 0.919 using accu_X

Total running time of the script: ( 6 minutes 0.720 seconds)

Gallery generated by Sphinx-Gallery