.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/40_advanced/example_multi_objective.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_40_advanced_example_multi_objective.py: ============== Classification ============== The following example shows how to fit *auto-sklearn* to optimize for two competing metrics: `precision` and `recall` (read more on this tradeoff in the `scikit-learn docs `_. Auto-sklearn uses `SMAC3's implementation of ParEGO `_. Multi-objective ensembling and proper access to the full Pareto set will be added in the near future. .. GENERATED FROM PYTHON SOURCE LINES 15-26 .. code-block:: default from pprint import pprint import matplotlib.pyplot as plt import numpy as np import sklearn.datasets import sklearn.metrics import autosklearn.classification import autosklearn.metrics .. GENERATED FROM PYTHON SOURCE LINES 27-29 Data Loading ============ .. GENERATED FROM PYTHON SOURCE LINES 29-39 .. code-block:: default X, y = sklearn.datasets.fetch_openml(data_id=31, return_X_y=True, as_frame=True) # Change the target to align with scikit-learn's convention that # ``1`` is the minority class. In this example it is predicting # that a credit is "bad", i.e. that it will default. y = np.array([1 if val == "bad" else 0 for val in y]) X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split( X, y, random_state=1 ) .. GENERATED FROM PYTHON SOURCE LINES 40-42 Build and fit a classifier ========================== .. GENERATED FROM PYTHON SOURCE LINES 42-50 .. code-block:: default automl = autosklearn.classification.AutoSklearnClassifier( time_left_for_this_task=120, metric=[autosklearn.metrics.precision, autosklearn.metrics.recall], delete_tmp_folder_after_terminate=False, ) automl.fit(X_train, y_train, dataset_name="German Credit") .. rst-class:: sphx-glr-script-out .. code-block:: none AutoSklearnClassifier(delete_tmp_folder_after_terminate=False, ensemble_class=, metric=[precision, recall], per_run_time_limit=12, time_left_for_this_task=120) .. GENERATED FROM PYTHON SOURCE LINES 51-53 Compute the two competing metrics ================================= .. GENERATED FROM PYTHON SOURCE LINES 53-58 .. code-block:: default predictions = automl.predict(X_test) print("Precision", sklearn.metrics.precision_score(y_test, predictions)) print("Recall", sklearn.metrics.recall_score(y_test, predictions)) .. rst-class:: sphx-glr-script-out .. code-block:: none Precision 0.6060606060606061 Recall 0.2702702702702703 .. GENERATED FROM PYTHON SOURCE LINES 59-62 View the models found by auto-sklearn ===================================== They are by default sorted by the first metric given to *auto-sklearn*. .. GENERATED FROM PYTHON SOURCE LINES 62-65 .. code-block:: default print(automl.leaderboard()) .. rst-class:: sphx-glr-script-out .. code-block:: none rank ensemble_weight type cost_0 cost_1 duration model_id 21 1 1.0 random_forest 0.205128 0.586667 2.151786 .. GENERATED FROM PYTHON SOURCE LINES 66-70 ``cv_results`` also contains both metrics ========================================= Similarly to the leaderboard, they are sorted by the first metric given to *auto-sklearn*. .. GENERATED FROM PYTHON SOURCE LINES 70-73 .. code-block:: default pprint(automl.cv_results_) .. rst-class:: sphx-glr-script-out .. code-block:: none {'budgets': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'mean_fit_time': array([ 2.06615281, 2.0024941 , 1.83083034, 2.021312 , 2.59403086, 3.43858409, 1.91830468, 1.10799742, 1.91203833, 12.01424766, 1.40986538, 1.96470523, 1.53796887, 0.86812377, 2.05662704, 1.04536629, 4.48269749, 1.19880962, 8.15736961, 2.15178585, 1.83902264, 0.82527328, 1.87171245, 1.05902672, 1.67635489, 2.04935241, 0.96880317, 2.52337337, 2.07653451, 2.00908375]), 'mean_test_precision': array([0.68888889, 0.61403509, 0.42156863, 0.55072464, 0.48275862, 0.70588235, 0.67391304, 0.38235294, 0.51401869, 0. , 0.49090909, 0.55263158, 0.50526316, 0.59016393, 0.625 , 0. , 0.47222222, 0.47272727, 0.54117647, 0.79487179, 0.44444444, 0. , 0.39534884, 0. , 0.51376147, 0.48148148, 0.34782609, 0.57534247, 0.6440678 , 0. ]), 'mean_test_recall': array([0.41333333, 0.46666667, 0.57333333, 0.50666667, 0.18666667, 0.48 , 0.41333333, 0.17333333, 0.73333333, 0. , 0.72 , 0.28 , 0.64 , 0.48 , 0.06666667, 0. , 0.45333333, 0.34666667, 0.61333333, 0.41333333, 0.69333333, 0. , 0.22666667, 0. , 0.74666667, 0.69333333, 0.53333333, 0.56 , 0.50666667, 0. ]), 'param_balancing:strategy': masked_array(data=['none', 'none', 'weighting', 'weighting', 'weighting', 'weighting', 'none', 'weighting', 'weighting', 'weighting', 'weighting', 'none', 'weighting', 'none', 'none', 'weighting', 'weighting', 'none', 'weighting', 'none', 'weighting', 'none', 'none', 'weighting', 'weighting', 'weighting', 'weighting', 'weighting', 'none', 'none'], mask=[False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], fill_value='N/A', dtype='` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: example_multi_objective.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_