.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/40_advanced/example_calc_multiple_metrics.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_40_advanced_example_calc_multiple_metrics.py: ======= Metrics ======= In *Auto-sklearn*, model is optimized over a metric, either built-in or custom metric. Moreover, it is also possible to calculate multiple metrics per run. The following examples show how to calculate metrics built-in and self-defined metrics for a classification problem. .. GENERATED FROM PYTHON SOURCE LINES 12-34 .. code-block:: default import autosklearn.classification import numpy as np import pandas as pd import sklearn.datasets import sklearn.metrics from autosklearn.metrics import balanced_accuracy, precision, recall, f1 def error(solution, prediction): # custom function defining error return np.mean(solution != prediction) def get_metric_result(cv_results): results = pd.DataFrame.from_dict(cv_results) results = results[results["status"] == "Success"] cols = ["rank_test_scores", "param_classifier:__choice__", "mean_test_score"] cols.extend([key for key in cv_results.keys() if key.startswith("metric_")]) return results[cols] .. GENERATED FROM PYTHON SOURCE LINES 35-37 Data Loading ============ .. GENERATED FROM PYTHON SOURCE LINES 37-43 .. code-block:: default X, y = sklearn.datasets.load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split( X, y, random_state=1 ) .. GENERATED FROM PYTHON SOURCE LINES 44-46 Build and fit a classifier ========================== .. GENERATED FROM PYTHON SOURCE LINES 46-62 .. code-block:: default error_rate = autosklearn.metrics.make_scorer( name="custom_error", score_func=error, optimum=0, greater_is_better=False, needs_proba=False, needs_threshold=False, ) cls = autosklearn.classification.AutoSklearnClassifier( time_left_for_this_task=120, per_run_time_limit=30, scoring_functions=[balanced_accuracy, precision, recall, f1, error_rate], ) cls.fit(X_train, y_train, X_test, y_test) .. rst-class:: sphx-glr-script-out .. code-block:: none Fitting to the training data: 0%| | 0/120 [00:00, per_run_time_limit=30, scoring_functions=[balanced_accuracy, precision, recall, f1, custom_error], time_left_for_this_task=120) .. GENERATED FROM PYTHON SOURCE LINES 63-65 Get the Score of the final ensemble =================================== .. GENERATED FROM PYTHON SOURCE LINES 65-72 .. code-block:: default predictions = cls.predict(X_test) print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions)) print("#" * 80) print("Metric results") print(get_metric_result(cls.cv_results_).to_string(index=False)) .. rst-class:: sphx-glr-script-out .. code-block:: none Accuracy score 0.958041958041958 ################################################################################ Metric results rank_test_scores param_classifier:__choice__ mean_test_score metric_balanced_accuracy metric_precision metric_recall metric_f1 metric_custom_error 6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369 6 mlp 0.971631 0.961538 0.956989 1.000000 0.978022 0.028369 26 mlp 0.943262 0.935069 0.945055 0.966292 0.955556 0.056738 16 random_forest 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461 6 mlp 0.971631 0.961538 0.956989 1.000000 0.978022 0.028369 1 extra_trees 0.985816 0.984767 0.988764 0.988764 0.988764 0.014184 16 random_forest 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461 22 extra_trees 0.957447 0.954300 0.966292 0.966292 0.966292 0.042553 6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369 6 random_forest 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369 16 gradient_boosting 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461 6 gradient_boosting 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369 6 mlp 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369 24 mlp 0.950355 0.948682 0.965909 0.955056 0.960452 0.049645 3 gradient_boosting 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277 16 gradient_boosting 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461 16 random_forest 0.964539 0.959918 0.966667 0.977528 0.972067 0.035461 6 extra_trees 0.971631 0.969533 0.977528 0.977528 0.977528 0.028369 29 passive_aggressive 0.921986 0.894231 0.890000 1.000000 0.941799 0.078014 3 extra_trees 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277 6 gradient_boosting 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369 24 mlp 0.950355 0.940687 0.945652 0.977528 0.961326 0.049645 27 random_forest 0.929078 0.923833 0.943820 0.943820 0.943820 0.070922 22 adaboost 0.957447 0.950303 0.956044 0.977528 0.966667 0.042553 6 extra_trees 0.971631 0.965536 0.967033 0.988764 0.977778 0.028369 1 extra_trees 0.985816 0.984767 0.988764 0.988764 0.988764 0.014184 27 bernoulli_nb 0.929078 0.927831 0.954023 0.932584 0.943182 0.070922 31 mlp 0.865248 0.817308 0.824074 1.000000 0.903553 0.134752 3 extra_trees 0.978723 0.975151 0.977778 0.988764 0.983240 0.021277 16 random_forest 0.964539 0.963915 0.977273 0.966292 0.971751 0.035461 29 gradient_boosting 0.921986 0.930207 0.975610 0.898876 0.935673 0.078014 .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 57.849 seconds) .. _sphx_glr_download_examples_40_advanced_example_calc_multiple_metrics.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/automl/auto-sklearn/development?urlpath=lab/tree/notebooks/examples/40_advanced/example_calc_multiple_metrics.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: example_calc_multiple_metrics.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: example_calc_multiple_metrics.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_