.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/40_advanced/example_early_stopping_and_callbacks.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_40_advanced_example_early_stopping_and_callbacks.py: ============================ Early stopping and Callbacks ============================ The example below shows how we can use the ``get_trials_callback`` parameter of auto-sklearn to implement an early-stopping mechanism through a callback. These callbacks give access to the result of each model + hyperparameter configuration optimized by SMAC, the underlying optimizer for autosklearn. By checking the cost of a result, we can implement a simple yet effective early stopping mechanism! Do note however, this does not provide any access to the ensembles that autosklearn produces, only the individual models. You may wish to perform a more sophisticated early stopping mechanism such that there are enough good models for autosklearn to build and ensemble with. This is here to provide a simple example. .. GENERATED FROM PYTHON SOURCE LINES 18-31 .. code-block:: default from __future__ import annotations from pprint import pprint import sklearn.datasets import sklearn.metrics import autosklearn.classification from smac.optimizer.smbo import SMBO from smac.runhistory.runhistory import RunInfo, RunValue .. GENERATED FROM PYTHON SOURCE LINES 32-34 Build and fit a classifier ========================== .. GENERATED FROM PYTHON SOURCE LINES 34-64 .. code-block:: default def callback( smbo: SMBO, run_info: RunInfo, result: RunValue, time_left: float, ) -> bool | None: """Stop early if we get a very low cost value for a single run The return value indicates to SMAC whether to stop or not. False will stop the search process while any other value will mean it continues. """ # You can find out the parameters in the SMAC documentation # https://automl.github.io/SMAC3/main/ if result.cost <= 0.02: print("Stopping!") print(run_info) print(result) return False X, y = sklearn.datasets.load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split( X, y, random_state=1 ) automl = autosklearn.classification.AutoSklearnClassifier( time_left_for_this_task=120, per_run_time_limit=30, get_trials_callback=callback ) automl.fit(X_train, y_train, dataset_name="breast_cancer") .. rst-class:: sphx-glr-script-out .. code-block:: none Fitting to the training data: 0%| | 0/120 [00:00, starttime=1669292713.2752733, endtime=1669292715.0874944, additional_info={'duration': 1.6978273391723633, 'num_run': 7, 'train_loss': 0.0, 'configuration_origin': 'Initial design'}) Fitting to the training data: 12%|#1 | 14/120 [00:14<01:46, 1.00s/it, The total time budget for this task is 0:02:00] Fitting to the training data: 100%|##########| 120/120 [00:14<00:00, 8.55it/s, The total time budget for this task is 0:02:00] AutoSklearnClassifier(ensemble_class=, get_trials_callback=, per_run_time_limit=30, time_left_for_this_task=120) .. GENERATED FROM PYTHON SOURCE LINES 65-67 View the models found by auto-sklearn ===================================== .. GENERATED FROM PYTHON SOURCE LINES 67-70 .. code-block:: default print(automl.leaderboard()) .. rst-class:: sphx-glr-script-out .. code-block:: none rank ensemble_weight type cost duration model_id 7 1 0.68 extra_trees 0.014184 1.788108 2 2 0.10 random_forest 0.028369 2.042358 3 3 0.22 mlp 0.028369 1.138246 .. GENERATED FROM PYTHON SOURCE LINES 71-73 Print the final ensemble constructed by auto-sklearn ==================================================== .. GENERATED FROM PYTHON SOURCE LINES 73-76 .. code-block:: default pprint(automl.show_models(), indent=4) .. rst-class:: sphx-glr-script-out .. code-block:: none { 2: { 'balancing': Balancing(random_state=1), 'classifier': , 'cost': 0.028368794326241176, 'data_preprocessor': , 'ensemble_weight': 0.1, 'feature_preprocessor': , 'model_id': 2, 'rank': 2, 'sklearn_classifier': RandomForestClassifier(max_features=5, n_estimators=512, n_jobs=1, random_state=1, warm_start=True)}, 3: { 'balancing': Balancing(random_state=1), 'classifier': , 'cost': 0.028368794326241176, 'data_preprocessor': , 'ensemble_weight': 0.22, 'feature_preprocessor': , 'model_id': 3, 'rank': 3, 'sklearn_classifier': MLPClassifier(activation='tanh', alpha=0.0001363185819149026, beta_1=0.999, beta_2=0.9, early_stopping=True, hidden_layer_sizes=(115, 115, 115), learning_rate_init=0.00018009776276177523, max_iter=32, n_iter_no_change=32, random_state=1, verbose=0, warm_start=True)}, 7: { 'balancing': Balancing(random_state=1), 'classifier': , 'cost': 0.014184397163120588, 'data_preprocessor': , 'ensemble_weight': 0.68, 'feature_preprocessor': , 'model_id': 7, 'rank': 1, 'sklearn_classifier': ExtraTreesClassifier(max_features=34, min_samples_leaf=3, min_samples_split=11, n_estimators=512, n_jobs=1, random_state=1, warm_start=True)}} .. GENERATED FROM PYTHON SOURCE LINES 77-79 Get the Score of the final ensemble =================================== .. GENERATED FROM PYTHON SOURCE LINES 79-82 .. code-block:: default predictions = automl.predict(X_test) print("Accuracy score:", sklearn.metrics.accuracy_score(y_test, predictions)) .. rst-class:: sphx-glr-script-out .. code-block:: none Accuracy score: 0.9440559440559441 .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 20.441 seconds) .. _sphx_glr_download_examples_40_advanced_example_early_stopping_and_callbacks.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/automl/auto-sklearn/development?urlpath=lab/tree/notebooks/examples/40_advanced/example_early_stopping_and_callbacks.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: example_early_stopping_and_callbacks.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: example_early_stopping_and_callbacks.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_