.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/40_advanced/example_metrics.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_examples_40_advanced_example_metrics.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_40_advanced_example_metrics.py:


=======
Metrics
=======

*Auto-sklearn* supports various built-in metrics, which can be found in the
:ref:`metrics section in the API <api:Built-in Metrics>`. However, it is also
possible to define your own metric and use it to fit and evaluate your model.
The following examples show how to use built-in and self-defined metrics for a
classification problem.

.. GENERATED FROM PYTHON SOURCE LINES 13-24

.. code-block:: default


    import numpy as np

    import sklearn.model_selection
    import sklearn.datasets
    import sklearn.metrics

    import autosklearn.classification
    import autosklearn.metrics









.. GENERATED FROM PYTHON SOURCE LINES 25-27

Custom Metrics
==============

.. GENERATED FROM PYTHON SOURCE LINES 27-56

.. code-block:: default

    def accuracy(solution, prediction):
        # custom function defining accuracy
        return np.mean(solution == prediction)


    def error(solution, prediction):
        # custom function defining error
        return np.mean(solution != prediction)


    def accuracy_wk(solution, prediction, extra_argument):
        # custom function defining accuracy and accepting an additional argument
        assert extra_argument is None
        return np.mean(solution == prediction)


    def error_wk(solution, prediction, extra_argument):
        # custom function defining error and accepting an additional argument
        assert extra_argument is None
        return np.mean(solution != prediction)


    def metric_which_needs_x(solution, prediction, X_data, consider_col, val_threshold):
        # custom function defining accuracy
        assert X_data is not None
        rel_idx = X_data[:, consider_col] > val_threshold
        return np.mean(solution[rel_idx] == prediction[rel_idx])









.. GENERATED FROM PYTHON SOURCE LINES 57-59

Data Loading
============

.. GENERATED FROM PYTHON SOURCE LINES 59-65

.. code-block:: default


    X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
        X, y, random_state=1
    )








.. GENERATED FROM PYTHON SOURCE LINES 66-68

Print a list of available metrics
=================================

.. GENERATED FROM PYTHON SOURCE LINES 68-75

.. code-block:: default


    print("Available CLASSIFICATION metrics autosklearn.metrics.*:")
    print("\t*" + "\n\t*".join(autosklearn.metrics.CLASSIFICATION_METRICS))

    print("Available REGRESSION autosklearn.metrics.*:")
    print("\t*" + "\n\t*".join(autosklearn.metrics.REGRESSION_METRICS))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Available CLASSIFICATION metrics autosklearn.metrics.*:
            *accuracy
            *balanced_accuracy
            *roc_auc
            *average_precision
            *log_loss
            *precision_macro
            *precision_micro
            *precision_samples
            *precision_weighted
            *recall_macro
            *recall_micro
            *recall_samples
            *recall_weighted
            *f1_macro
            *f1_micro
            *f1_samples
            *f1_weighted
    Available REGRESSION autosklearn.metrics.*:
            *mean_absolute_error
            *mean_squared_error
            *root_mean_squared_error
            *mean_squared_log_error
            *median_absolute_error
            *r2




.. GENERATED FROM PYTHON SOURCE LINES 76-78

First example: Use predefined accuracy metric
=============================================

.. GENERATED FROM PYTHON SOURCE LINES 78-93

.. code-block:: default


    print("#" * 80)
    print("Use predefined accuracy metric")
    scorer = autosklearn.metrics.accuracy
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60,
        seed=1,
        metric=scorer,
    )
    cls.fit(X_train, y_train)

    predictions = cls.predict(X_test)
    score = scorer(y_test, predictions)
    print(f"Accuracy score {score:.3f} using {scorer.name}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ################################################################################
    Use predefined accuracy metric

    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:50<00:00,  1.20it/s, The total time budget for this task is 0:01:00]
    Accuracy score 0.944 using accuracy




.. GENERATED FROM PYTHON SOURCE LINES 94-96

Second example: Use own accuracy metric
=======================================

.. GENERATED FROM PYTHON SOURCE LINES 96-118

.. code-block:: default


    print("#" * 80)
    print("Use self defined accuracy metric")
    accuracy_scorer = autosklearn.metrics.make_scorer(
        name="accu",
        score_func=accuracy,
        optimum=1,
        greater_is_better=True,
        needs_proba=False,
        needs_threshold=False,
    )
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60,
        seed=1,
        metric=accuracy_scorer,
    )
    cls.fit(X_train, y_train)

    predictions = cls.predict(X_test)
    score = accuracy_scorer(y_test, predictions)
    print(f"Accuracy score {score:.3f} using {accuracy_scorer.name:s}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ################################################################################
    Use self defined accuracy metric

    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:50<00:00,  1.19it/s, The total time budget for this task is 0:01:00]
    Accuracy score 0.958 using accu




.. GENERATED FROM PYTHON SOURCE LINES 119-121

Third example: Use own error metric
===================================

.. GENERATED FROM PYTHON SOURCE LINES 121-143

.. code-block:: default


    print("#" * 80)
    print("Use self defined error metric")
    error_rate = autosklearn.metrics.make_scorer(
        name="error",
        score_func=error,
        optimum=0,
        greater_is_better=False,
        needs_proba=False,
        needs_threshold=False,
    )
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60,
        seed=1,
        metric=error_rate,
    )
    cls.fit(X_train, y_train)

    cls.predictions = cls.predict(X_test)
    score = error_rate(y_test, predictions)
    print(f"Error score {score:.3f} using {error_rate.name:s}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ################################################################################
    Use self defined error metric

    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  85%|########5 | 51/60 [00:51<00:09,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  87%|########6 | 52/60 [00:52<00:08,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  88%|########8 | 53/60 [00:53<00:07,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  90%|######### | 54/60 [00:54<00:06,  1.04s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  92%|#########1| 55/60 [00:55<00:05,  1.03s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:55<00:00,  1.08it/s, The total time budget for this task is 0:01:00]
    Error score -0.042 using error




.. GENERATED FROM PYTHON SOURCE LINES 144-146

Fourth example: Use own accuracy metric with additional argument
================================================================

.. GENERATED FROM PYTHON SOURCE LINES 146-167

.. code-block:: default


    print("#" * 80)
    print("Use self defined accuracy with additional argument")
    accuracy_scorer = autosklearn.metrics.make_scorer(
        name="accu_add",
        score_func=accuracy_wk,
        optimum=1,
        greater_is_better=True,
        needs_proba=False,
        needs_threshold=False,
        extra_argument=None,
    )
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60, per_run_time_limit=30, seed=1, metric=accuracy_scorer
    )
    cls.fit(X_train, y_train)

    predictions = cls.predict(X_test)
    score = accuracy_scorer(y_test, predictions)
    print(f"Accuracy score {score:.3f} using {accuracy_scorer.name:s}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ################################################################################
    Use self defined accuracy with additional argument

    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.02s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  83%|########3 | 50/60 [00:50<00:10,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  85%|########5 | 51/60 [00:51<00:09,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  87%|########6 | 52/60 [00:52<00:08,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  88%|########8 | 53/60 [00:53<00:07,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  90%|######### | 54/60 [00:54<00:06,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:54<00:00,  1.11it/s, The total time budget for this task is 0:01:00]
    Accuracy score 0.958 using accu_add




.. GENERATED FROM PYTHON SOURCE LINES 168-170

Fifth example: Use own accuracy metric with additional argument
===============================================================

.. GENERATED FROM PYTHON SOURCE LINES 170-194

.. code-block:: default


    print("#" * 80)
    print("Use self defined error with additional argument")
    error_rate = autosklearn.metrics.make_scorer(
        name="error_add",
        score_func=error_wk,
        optimum=0,
        greater_is_better=True,
        needs_proba=False,
        needs_threshold=False,
        extra_argument=None,
    )
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60,
        seed=1,
        metric=error_rate,
    )
    cls.fit(X_train, y_train)

    predictions = cls.predict(X_test)
    score = error_rate(y_test, predictions)
    print(f"Error score {score:.3f} using {error_rate.name:s}")






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ################################################################################
    Use self defined error with additional argument

    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:00,361:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:03,553:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:07,312:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:08,564:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:50,  1.06s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:48,  1.04s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:47,  1.03s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:12,512:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.02s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:13,415:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.02s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:15,768:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:19,584:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:23,676:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:28,532:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:31,917:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:36,948:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.02s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:41,057:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:41,958:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.01s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:51:46,703:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:49<00:00,  1.21it/s, The total time budget for this task is 0:01:00]
    Error score 0.615 using error_add




.. GENERATED FROM PYTHON SOURCE LINES 195-197

Sixth example: Use a metric with additional argument which also needs xdata
===========================================================================

.. GENERATED FROM PYTHON SOURCE LINES 197-230

.. code-block:: default

    """
    Finally, *Auto-sklearn* also support metric that require the train data (aka X_data) to
    compute a value. This can be useful if one only cares about the score on a subset of the
    data.
    """

    accuracy_scorer = autosklearn.metrics.make_scorer(
        name="accu_X",
        score_func=metric_which_needs_x,
        optimum=1,
        greater_is_better=True,
        needs_proba=False,
        needs_X=True,
        needs_threshold=False,
        consider_col=1,
        val_threshold=18.8,
    )
    cls = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=60,
        seed=1,
        metric=accuracy_scorer,
    )
    cls.fit(X_train, y_train)

    predictions = cls.predict(X_test)
    score = metric_which_needs_x(
        y_test,
        predictions,
        X_data=X_test,
        consider_col=1,
        val_threshold=18.8,
    )
    print(f"Error score {score:.3f} using {accuracy_scorer.name:s}")




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Fitting to the training data:   0%|          | 0/60 [00:00<?, ?it/s, The total time budget for this task is 0:01:00]
    Fitting to the training data:   2%|1         | 1/60 [00:01<00:59,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   3%|3         | 2/60 [00:02<00:58,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   5%|5         | 3/60 [00:03<00:57,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   7%|6         | 4/60 [00:04<00:56,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:   8%|8         | 5/60 [00:05<00:55,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  10%|#         | 6/60 [00:06<00:54,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  12%|#1        | 7/60 [00:07<00:53,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  13%|#3        | 8/60 [00:08<00:52,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  15%|#5        | 9/60 [00:09<00:51,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  17%|#6        | 10/60 [00:10<00:50,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  18%|#8        | 11/60 [00:11<00:49,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  20%|##        | 12/60 [00:12<00:48,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  22%|##1       | 13/60 [00:13<00:47,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  23%|##3       | 14/60 [00:14<00:46,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  25%|##5       | 15/60 [00:15<00:45,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  27%|##6       | 16/60 [00:16<00:44,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  28%|##8       | 17/60 [00:17<00:43,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  30%|###       | 18/60 [00:18<00:42,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  32%|###1      | 19/60 [00:19<00:41,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  33%|###3      | 20/60 [00:20<00:40,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  35%|###5      | 21/60 [00:21<00:39,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  37%|###6      | 22/60 [00:22<00:38,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  38%|###8      | 23/60 [00:23<00:37,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  40%|####      | 24/60 [00:24<00:36,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  42%|####1     | 25/60 [00:25<00:35,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  43%|####3     | 26/60 [00:26<00:34,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  45%|####5     | 27/60 [00:27<00:33,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  47%|####6     | 28/60 [00:28<00:32,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  48%|####8     | 29/60 [00:29<00:31,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  50%|#####     | 30/60 [00:30<00:30,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  52%|#####1    | 31/60 [00:31<00:29,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  53%|#####3    | 32/60 [00:32<00:28,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  55%|#####5    | 33/60 [00:33<00:27,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  57%|#####6    | 34/60 [00:34<00:26,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  58%|#####8    | 35/60 [00:35<00:25,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  60%|######    | 36/60 [00:36<00:24,  1.01s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  62%|######1   | 37/60 [00:37<00:23,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  63%|######3   | 38/60 [00:38<00:22,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:36,313:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  65%|######5   | 39/60 [00:39<00:21,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:37,670:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  67%|######6   | 40/60 [00:40<00:20,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  68%|######8   | 41/60 [00:41<00:19,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:38,890:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  70%|#######   | 42/60 [00:42<00:18,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  72%|#######1  | 43/60 [00:43<00:17,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:41,358:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  73%|#######3  | 44/60 [00:44<00:16,  1.00s/it, The total time budget for this task is 0:01:00][WARNING] [2022-11-24 12:52:42,354:smac.runhistory.runhistory2epm.RunHistory2EPM4LogCost] Got cost of smaller/equal to 0. Replace by 0.000010 since we use log cost.

    Fitting to the training data:  75%|#######5  | 45/60 [00:45<00:15,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  77%|#######6  | 46/60 [00:46<00:14,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  78%|#######8  | 47/60 [00:47<00:13,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  80%|########  | 48/60 [00:48<00:12,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data:  82%|########1 | 49/60 [00:49<00:11,  1.00s/it, The total time budget for this task is 0:01:00]
    Fitting to the training data: 100%|##########| 60/60 [00:49<00:00,  1.22it/s, The total time budget for this task is 0:01:00]
    Error score 0.919 using accu_X





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 6 minutes  0.720 seconds)


.. _sphx_glr_download_examples_40_advanced_example_metrics.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/automl/auto-sklearn/development?urlpath=lab/tree/notebooks/examples/40_advanced/example_metrics.ipynb
        :alt: Launch binder
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: example_metrics.py <example_metrics.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: example_metrics.ipynb <example_metrics.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_