.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/40_advanced/example_plot_over_time.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_examples_40_advanced_example_plot_over_time.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_40_advanced_example_plot_over_time.py:


==============================
Plot the Performance over Time
==============================

Auto-Pytorch uses SMAC to fit individual machine learning algorithms
and then ensembles them together using `Ensemble Selection
<https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf>`_.

The following examples shows how to plot both the performance
of the individual models and their respective ensemble.

Additionally, as we are compatible with matplotlib,
you can input any args or kwargs that are compatible with ax.plot.
In the case when you would like to create multipanel visualization,
please input plt.Axes obtained from matplotlib.pyplot.subplots.

.. GENERATED FROM PYTHON SOURCE LINES 19-36

.. code-block:: default

    import warnings

    import numpy as np
    import pandas as pd

    from sklearn import model_selection

    import matplotlib.pyplot as plt

    from autoPyTorch.api.tabular_classification import TabularClassificationTask
    from autoPyTorch.utils.results_visualizer import PlotSettingParams


    warnings.simplefilter(action='ignore', category=UserWarning)
    warnings.simplefilter(action='ignore', category=FutureWarning)









.. GENERATED FROM PYTHON SOURCE LINES 37-39

Task Definition
===============

.. GENERATED FROM PYTHON SOURCE LINES 39-47

.. code-block:: default

    n_samples, dim = 100, 2
    X = np.random.random((n_samples, dim)) * 2 - 1
    y = ((X ** 2).sum(axis=-1) < 2 / np.pi).astype(np.int32)
    print(y)

    X, y = pd.DataFrame(X), pd.DataFrame(y)
    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [0 0 0 1 0 0 0 1 1 1 1 0 1 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 1 1 0 1 0 0 1 1
     0 1 0 1 1 0 1 0 0 1 0 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 1 0 0 1 1 0 1 0 0 0 0
     1 0 1 1 1 0 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 1 0 1 0]




.. GENERATED FROM PYTHON SOURCE LINES 48-50

API Instantiation and Searching
===============================

.. GENERATED FROM PYTHON SOURCE LINES 50-55

.. code-block:: default

    api = TabularClassificationTask(seed=42)

    api.search(X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test,
               optimize_metric='accuracy', total_walltime_limit=120, func_eval_time_limit_secs=10)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f9aa68dbe80>



.. GENERATED FROM PYTHON SOURCE LINES 56-58

Create Setting Parameters Object
================================

.. GENERATED FROM PYTHON SOURCE LINES 58-70

.. code-block:: default

    metric_name = 'accuracy'

    params = PlotSettingParams(
        xscale='log',
        xlabel='Runtime',
        ylabel='Accuracy',
        title='Toy Example',
        figname='example_plot_over_time.png',
        savefig_kwargs={'bbox_inches': 'tight'},
        show=False  # If you would like to show, make it True and set figname=None
    )








.. GENERATED FROM PYTHON SOURCE LINES 71-74

Plot with the Specified Setting Parameters
==========================================
_, ax = plt.subplots()  <=== You can feed it to post-process the figure.

.. GENERATED FROM PYTHON SOURCE LINES 74-82

.. code-block:: default


    # You might need to run `export DISPLAY=:0.0` if you are using non-GUI based environment.
    api.plot_perf_over_time(
        metric_name=metric_name,
        plot_setting_params=params,
        marker='*',
        markersize=10
    )



.. image-sg:: /examples/40_advanced/images/sphx_glr_example_plot_over_time_001.png
   :alt: Toy Example
   :srcset: /examples/40_advanced/images/sphx_glr_example_plot_over_time_001.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 2 minutes  16.039 seconds)


.. _sphx_glr_download_examples_40_advanced_example_plot_over_time.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/automl/Auto-PyTorch/development?urlpath=lab/tree/notebooks/examples/40_advanced/example_plot_over_time.ipynb
        :alt: Launch binder
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: example_plot_over_time.py <example_plot_over_time.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: example_plot_over_time.ipynb <example_plot_over_time.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_