Fit a single configuration

Auto-sklearn searches for the best combination of machine learning algorithms and their hyper-parameter configuration for a given task, using Scikit-Learn Pipelines. To further improve performance, this pipelines are ensemble together using Ensemble Selection from Caruana (2004).

This example shows how one can fit one of this pipelines, both, with an user defined configuration, and a randomly sampled one form the configuration space.

The pipelines that Auto-Sklearn fits are compatible with Scikit-Learn API. You can get further documentation about Scikit-Learn models here: <https://scikit-learn.org/stable/getting_started.html`>_

import numpy as np
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

from ConfigSpace.configuration_space import Configuration

import autosklearn.classification

Data Loading

X, y = sklearn.datasets.fetch_openml(data_id=3, return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
    X, y, test_size=0.5, random_state=3
)

Define an estimator

cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=60,
    memory_limit=4096,
    # We will limit the configuration space only to
    # have RandomForest as a valid model. We recommend enabling all
    # possible models to get a better performance.
    include={
        'classifier': ['random_forest']
    },
    delete_tmp_folder_after_terminate=False,
)

Fit an user provided configuration

# We will create a configuration that has a user defined
# min_samples_split in the Random Forest. We recommend you to look into
# how the ConfigSpace package works here:
# https://automl.github.io/ConfigSpace/master/
cs = cls.get_configuration_space(X, y, dataset_name='kr-vs-kp')
config = cs.sample_configuration()
config._values['classifier:random_forest:min_samples_split'] = 11

# Make sure that your changed configuration complies with the configuration space
config.is_valid_configuration()

pipeline, run_info, run_value = cls.fit_pipeline(X=X_train, y=y_train,
                                                 dataset_name='kr-vs-kp',
                                                 config=config,
                                                 X_test=X_test, y_test=y_test)

# This object complies with Scikit-Learn Pipeline API.
# https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html
print(pipeline.named_steps)

# The fit_pipeline command also returns a named tuple with the pipeline constraints
print(run_info)

# The fit_pipeline command also returns a named tuple with train/test performance
print(run_value)

# We can make sure that our pipeline configuration was honored as follows
print("Passed Configuration:", pipeline.config)
print("Random Forest:", pipeline.named_steps['classifier'].choice.estimator)

# We can also search for new configurations using the fit() method
# Any configurations found by Auto-Sklearn -- even the ones created using
# fit_pipeline() are stored to disk and can be used for Ensemble Selection
cs = cls.fit(X, y, dataset_name='kr-vs-kp')

Out:

{'data_preprocessor': <autosklearn.pipeline.components.data_preprocessing.DataPreprocessorChoice object at 0x7f937c5a9ca0>, 'balancing': Balancing(random_state=1, strategy='weighting'), 'feature_preprocessor': <autosklearn.pipeline.components.feature_preprocessing.FeaturePreprocessorChoice object at 0x7f937b0d0e50>, 'classifier': <autosklearn.pipeline.components.classification.ClassifierChoice object at 0x7f937b0d0a00>}
RunInfo(config=Configuration:
  balancing:strategy, Value: 'weighting'
  classifier:__choice__, Value: 'random_forest'
  classifier:random_forest:bootstrap, Value: 'False'
  classifier:random_forest:criterion, Value: 'gini'
  classifier:random_forest:max_depth, Constant: 'None'
  classifier:random_forest:max_features, Value: 0.40665293991189455
  classifier:random_forest:max_leaf_nodes, Constant: 'None'
  classifier:random_forest:min_impurity_decrease, Constant: 0.0
  classifier:random_forest:min_samples_leaf, Value: 13
  classifier:random_forest:min_samples_split, Value: 11
  classifier:random_forest:min_weight_fraction_leaf, Constant: 0.0
  data_preprocessor:__choice__, Value: 'feature_type'
  data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__, Value: 'no_encoding'
  data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__, Value: 'minority_coalescer'
  data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction, Value: 0.0012559712452161484
  data_preprocessor:feature_type:numerical_transformer:imputation:strategy, Value: 'median'
  data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__, Value: 'quantile_transformer'
  data_preprocessor:feature_type:numerical_transformer:rescaling:quantile_transformer:n_quantiles, Value: 204
  data_preprocessor:feature_type:numerical_transformer:rescaling:quantile_transformer:output_distribution, Value: 'normal'
  feature_preprocessor:__choice__, Value: 'polynomial'
  feature_preprocessor:polynomial:degree, Value: 2
  feature_preprocessor:polynomial:include_bias, Value: 'False'
  feature_preprocessor:polynomial:interaction_only, Value: 'True'
, instance=None, instance_specific=None, seed=1, cutoff=60, capped=False, budget=0.0, source_id=0)
RunValue(cost=0.012322274881516604, time=2.914306402206421, status=<StatusType.SUCCESS: 1>, starttime=1636487580.5942185, endtime=1636487583.5312755, additional_info={'duration': 2.8303675651550293, 'num_run': 2, 'train_loss': 0.012143858010275621, 'configuration_origin': None})
Passed Configuration: Configuration:
  balancing:strategy, Value: 'weighting'
  classifier:__choice__, Value: 'random_forest'
  classifier:random_forest:bootstrap, Value: 'False'
  classifier:random_forest:criterion, Value: 'gini'
  classifier:random_forest:max_depth, Constant: 'None'
  classifier:random_forest:max_features, Value: 0.40665293991189455
  classifier:random_forest:max_leaf_nodes, Constant: 'None'
  classifier:random_forest:min_impurity_decrease, Constant: 0.0
  classifier:random_forest:min_samples_leaf, Value: 13
  classifier:random_forest:min_samples_split, Value: 11
  classifier:random_forest:min_weight_fraction_leaf, Constant: 0.0
  data_preprocessor:__choice__, Value: 'feature_type'
  data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__, Value: 'no_encoding'
  data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__, Value: 'minority_coalescer'
  data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction, Value: 0.0012559712452161484
  data_preprocessor:feature_type:numerical_transformer:imputation:strategy, Value: 'median'
  data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__, Value: 'quantile_transformer'
  data_preprocessor:feature_type:numerical_transformer:rescaling:quantile_transformer:n_quantiles, Value: 204
  data_preprocessor:feature_type:numerical_transformer:rescaling:quantile_transformer:output_distribution, Value: 'normal'
  feature_preprocessor:__choice__, Value: 'polynomial'
  feature_preprocessor:polynomial:degree, Value: 2
  feature_preprocessor:polynomial:include_bias, Value: 'False'
  feature_preprocessor:polynomial:interaction_only, Value: 'True'

Random Forest: RandomForestClassifier(bootstrap=False, max_features=14, min_samples_leaf=13,
                       min_samples_split=11, n_estimators=512, n_jobs=1,
                       random_state=1, warm_start=True)

Total running time of the script: ( 2 minutes 24.877 seconds)

Gallery generated by Sphinx-Gallery