Note
Click here to download the full example code or to run this example in your browser via Binder
Metrics¶
In Auto-sklearn, model is optimized over a metric, either built-in or custom metric. Moreover, it is also possible to calculate multiple metrics per run. The following examples show how to calculate metrics built-in and self-defined metrics for a classification problem.
import autosklearn.classification
import numpy as np
import pandas as pd
import sklearn.datasets
import sklearn.metrics
from autosklearn.metrics import balanced_accuracy, precision, recall, f1
def error(solution, prediction):
# custom function defining error
return np.mean(solution != prediction)
def get_metric_result(cv_results):
results = pd.DataFrame.from_dict(cv_results)
results = results[results["status"] == "Success"]
cols = ["rank_test_scores", "param_classifier:__choice__", "mean_test_score"]
cols.extend([key for key in cv_results.keys() if key.startswith("metric_")])
return results[cols]
Data Loading¶
X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
X, y, random_state=1
)
Build and fit a classifier¶
error_rate = autosklearn.metrics.make_scorer(
name="custom_error",
score_func=error,
optimum=0,
greater_is_better=False,
needs_proba=False,
needs_threshold=False,
)
cls = autosklearn.classification.AutoSklearnClassifier(
time_left_for_this_task=120,
per_run_time_limit=30,
scoring_functions=[balanced_accuracy, precision, recall, f1, error_rate],
)
cls.fit(X_train, y_train, X_test, y_test)
Fitting to the training data: 0%| | 0/120 [00:00<?, ?it/s, The total time budget for this task is 0:02:00]
Fitting to the training data: 1%| | 1/120 [00:01<01:59, 1.01s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 2%|1 | 2/120 [00:02<01:58, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 2%|2 | 3/120 [00:03<01:57, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 3%|3 | 4/120 [00:04<01:56, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 4%|4 | 5/120 [00:05<01:55, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 5%|5 | 6/120 [00:06<01:54, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 6%|5 | 7/120 [00:07<01:53, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 7%|6 | 8/120 [00:08<01:52, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 8%|7 | 9/120 [00:09<01:51, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 8%|8 | 10/120 [00:10<01:50, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 9%|9 | 11/120 [00:11<01:49, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 10%|# | 12/120 [00:12<01:48, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 11%|# | 13/120 [00:13<01:47, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 12%|#1 | 14/120 [00:14<01:46, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 12%|#2 | 15/120 [00:15<01:45, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 13%|#3 | 16/120 [00:16<01:44, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 14%|#4 | 17/120 [00:17<01:43, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 15%|#5 | 18/120 [00:18<01:42, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 16%|#5 | 19/120 [00:19<01:41, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 17%|#6 | 20/120 [00:20<01:40, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 18%|#7 | 21/120 [00:21<01:39, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 18%|#8 | 22/120 [00:22<01:38, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 19%|#9 | 23/120 [00:23<01:37, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 20%|## | 24/120 [00:24<01:36, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 21%|## | 25/120 [00:25<01:35, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 22%|##1 | 26/120 [00:26<01:34, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 22%|##2 | 27/120 [00:27<01:33, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 23%|##3 | 28/120 [00:28<01:32, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 24%|##4 | 29/120 [00:29<01:31, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 25%|##5 | 30/120 [00:30<01:30, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 26%|##5 | 31/120 [00:31<01:29, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 27%|##6 | 32/120 [00:32<01:28, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 28%|##7 | 33/120 [00:33<01:27, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 28%|##8 | 34/120 [00:34<01:26, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 29%|##9 | 35/120 [00:35<01:25, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 30%|### | 36/120 [00:36<01:24, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 31%|### | 37/120 [00:37<01:23, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 32%|###1 | 38/120 [00:38<01:22, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 32%|###2 | 39/120 [00:39<01:21, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 33%|###3 | 40/120 [00:40<01:20, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 34%|###4 | 41/120 [00:41<01:19, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 35%|###5 | 42/120 [00:42<01:18, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 36%|###5 | 43/120 [00:43<01:17, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 37%|###6 | 44/120 [00:44<01:16, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 38%|###7 | 45/120 [00:45<01:15, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 38%|###8 | 46/120 [00:46<01:14, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 39%|###9 | 47/120 [00:47<01:13, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 40%|#### | 48/120 [00:48<01:12, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 41%|#### | 49/120 [00:49<01:11, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 42%|####1 | 50/120 [00:50<01:10, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 42%|####2 | 51/120 [00:51<01:09, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 43%|####3 | 52/120 [00:52<01:08, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 44%|####4 | 53/120 [00:53<01:07, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 45%|####5 | 54/120 [00:54<01:06, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 46%|####5 | 55/120 [00:55<01:05, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 47%|####6 | 56/120 [00:56<01:04, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 48%|####7 | 57/120 [00:57<01:03, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 48%|####8 | 58/120 [00:58<01:02, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 49%|####9 | 59/120 [00:59<01:01, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 50%|##### | 60/120 [01:00<01:00, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 51%|##### | 61/120 [01:01<00:59, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 52%|#####1 | 62/120 [01:02<00:58, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 52%|#####2 | 63/120 [01:03<00:57, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 53%|#####3 | 64/120 [01:04<00:56, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 54%|#####4 | 65/120 [01:05<00:55, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 55%|#####5 | 66/120 [01:06<00:54, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 56%|#####5 | 67/120 [01:07<00:53, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 57%|#####6 | 68/120 [01:08<00:52, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 57%|#####7 | 69/120 [01:09<00:51, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 58%|#####8 | 70/120 [01:10<00:50, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 59%|#####9 | 71/120 [01:11<00:49, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 60%|###### | 72/120 [01:12<00:48, 1.01s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 61%|###### | 73/120 [01:13<00:47, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 62%|######1 | 74/120 [01:14<00:46, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 62%|######2 | 75/120 [01:15<00:45, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 63%|######3 | 76/120 [01:16<00:44, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 64%|######4 | 77/120 [01:17<00:43, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 65%|######5 | 78/120 [01:18<00:42, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 66%|######5 | 79/120 [01:19<00:41, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 67%|######6 | 80/120 [01:20<00:40, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 68%|######7 | 81/120 [01:21<00:39, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 68%|######8 | 82/120 [01:22<00:38, 1.01s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 69%|######9 | 83/120 [01:23<00:37, 1.01s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 70%|####### | 84/120 [01:24<00:36, 1.01s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 71%|####### | 85/120 [01:25<00:35, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 72%|#######1 | 86/120 [01:26<00:34, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 72%|#######2 | 87/120 [01:27<00:33, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 73%|#######3 | 88/120 [01:28<00:32, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 74%|#######4 | 89/120 [01:29<00:31, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 75%|#######5 | 90/120 [01:30<00:30, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 76%|#######5 | 91/120 [01:31<00:29, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 77%|#######6 | 92/120 [01:32<00:28, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 78%|#######7 | 93/120 [01:33<00:27, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 78%|#######8 | 94/120 [01:34<00:26, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 79%|#######9 | 95/120 [01:35<00:25, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 80%|######## | 96/120 [01:36<00:24, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 81%|######## | 97/120 [01:37<00:23, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 82%|########1 | 98/120 [01:38<00:22, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 82%|########2 | 99/120 [01:39<00:21, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 83%|########3 | 100/120 [01:40<00:20, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 84%|########4 | 101/120 [01:41<00:19, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 85%|########5 | 102/120 [01:42<00:18, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 86%|########5 | 103/120 [01:43<00:17, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 87%|########6 | 104/120 [01:44<00:16, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 88%|########7 | 105/120 [01:45<00:15, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 88%|########8 | 106/120 [01:46<00:14, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 89%|########9 | 107/120 [01:47<00:13, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 90%|######### | 108/120 [01:48<00:12, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 91%|######### | 109/120 [01:49<00:11, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 92%|#########1| 110/120 [01:50<00:10, 1.00s/it, The total time budget for this task is 0:02:00]
Fitting to the training data: 100%|##########| 120/120 [01:50<00:00, 1.09it/s, The total time budget for this task is 0:02:00]
AutoSklearnClassifier(ensemble_class=<class 'autosklearn.ensembles.ensemble_selection.EnsembleSelection'>,
per_run_time_limit=30,
scoring_functions=[balanced_accuracy, precision, recall,
f1, custom_error],
time_left_for_this_task=120)
Get the Score of the final ensemble¶
predictions = cls.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))
print("#" * 80)
print("Metric results")
print(get_metric_result(cls.cv_results_).to_string(index=False))
Accuracy score 0.958041958041958
################################################################################
Metric results
rank_test_scores param_classifier:__choice__ mean_test_score metric_balanced_accuracy metric_precision metric_recall metric_f1 metric_custom_error
6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369
6 mlp 0.971631 0.961538 0.956989 1.000000 0.978022 0.028369
26 mlp 0.943262 0.935069 0.945055 0.966292 0.955556 0.056738
16 random_forest 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461
6 mlp 0.971631 0.961538 0.956989 1.000000 0.978022 0.028369
1 extra_trees 0.985816 0.984767 0.988764 0.988764 0.988764 0.014184
16 random_forest 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461
22 extra_trees 0.957447 0.954300 0.966292 0.966292 0.966292 0.042553
6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369
6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369
16 gradient_boosting 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461
6 gradient_boosting 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369
6 mlp 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369
24 mlp 0.950355 0.948682 0.965909 0.955056 0.960452 0.049645
3 gradient_boosting 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277
16 gradient_boosting 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461
16 random_forest 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461
6 extra_trees 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369
29 passive_aggressive 0.921986 0.894231 0.890000 1.000000 0.941799 0.078014
3 extra_trees 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277
6 gradient_boosting 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369
24 mlp 0.950355 0.940687 0.945652 0.977528 0.961326 0.049645
27 random_forest 0.929078 0.923833 0.943820 0.943820 0.943820 0.070922
22 adaboost 0.957447 0.950303 0.956044 0.977528 0.966667 0.042553
6 extra_trees 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369
1 extra_trees 0.985816 0.984767 0.988764 0.988764 0.988764 0.014184
27 bernoulli_nb 0.929078 0.927831 0.954023 0.932584 0.943182 0.070922
31 mlp 0.865248 0.817308 0.824074 1.000000 0.903553 0.134752
3 extra_trees 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277
16 random_forest 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461
29 gradient_boosting 0.921986 0.930207 0.975610 0.898876 0.935673 0.078014
Total running time of the script: ( 1 minutes 57.849 seconds)
