mdp_playground.analysis.analysis.MDPP_Analysis

class mdp_playground.analysis.analysis.MDPP_Analysis[source]

Bases: object

Utility class to load and plot data for analysis of experiments from MDP Playground

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__()

Initialize self.

gather_stats(list_exp_data, train, …)

get_exp_data(dir_name, exp_name[, exp_type, …])

Get training and evaluation data from a single set of recorded CSV stats files.

load_data(experiments[, load_eval, exp_type])

Loads training and evaluation data from given multiple files

plot_1d_dimensions(list_exp_data[, …])

Plots 1-D bar plots across a single dimension with mean and std.

plot_2d_heatmap(list_exp_data[, save_fig, …])

Plots 2 2-D heatmaps: 1 for mean and 1 for std.

plot_bar(ax, stats_data[, save_fig, …])

Plots 1-D bar plots across a single dimension with mean and std.

plot_learning_curves(list_exp_data[, …])

Plots learning curves: Either across 1 or 2 meta-features of MDP Playground.

plot_radar(list_exp_data[, save_fig, train, …])

Plots radar (spider) chart across different dimensions
get_exp_data(dir_name, exp_name, exp_type='grid', num_metrics=3, load_eval=True, threshold=0.05, sample_freq=1)[source]

Get training and evaluation data from a single set of recorded CSV stats files.

Parameters

  • dir_name (str) – The location where the training and evaluation CSV files were written

  • exp_name (str) – The name of the experiment: the training and evaluation CSV filenames are formed using this string

  • exp_type (str) – One of [‘grid’, ‘random’]. If it’s ‘grid’, it’s assumed that a grid of configurations was run and data loading takes place in a manner specific to grids. Otherwise, data loading tries to look for different unique configurations run.

  • num_metrics (int) – The number of metrics that were written to CSV stats files. Default is 3 (timesteps_total, episode_reward_mean, episode_len_mean).

  • load_eval (bool) – Whether to load evaluation stats CSV or not.

  • threshold (float) – The fault tolerance threshold while loading data. Show warnings for files where the number of data rows deviates by more than the threshold value (from the mean number of rows across the loaded files), e.g., if the threshold = 0.05 and mean number of rows across the loaded files = 20, a warning is displayed when for a loaded file the number of rows < 19.

  • sample_freq (int) – The subsampling frequency when loading data. Sub-select data lines based on this value, e.g., if sample_freq = 5, load every 5th row in a file.

Returns

  • experiment data (dictionary type with following key-value)

  • train_stats (np.ndarray) – Training stats at end of training: 8-D tensor with 1st 6 dims the meta-features of MDP Playground, 7th dim is across the seeds, 8th dim is across different stats saved

  • eval_stats (np.ndarray) – Training stats at end of training: 8-D tensor with 1st 6 dims the meta-features of MDP Playground, 7th dim is across the seeds, 8th dim is across different stats saved

  • train_curves (np.ndarray) – The loaded training CSV with the last 3 columns the train stats that were saved and the initial columns are various setting for the algorithm and environment.

  • eval_curves (np.ndarray) – The loaded evaluation CSV with the columns the evaluation stats that were saved

load_data(experiments: dict, load_eval=True, exp_type='grid')[source]

Loads training and evaluation data from given multiple files

Parameters

experiments (dict<str,str>) –

per experiment -> key-value pair of exp_name & dir_name

dir_namestr

The location where the training and evaluation CSV files were written

exp_namestr

The name of the experiment: the training and evaluation CSV filenames are formed using this string

eg: experiments = {

}

Returns

  • list of experiment data of type <dict>

  • [ for more details refer get_exp_data() ]

plot_1d_dimensions(list_exp_data, save_fig=False, train=True, use_aucs=False, metric_num=- 2, plot_type='agent')[source]

Plots 1-D bar plots across a single dimension with mean and std. dev.

Parameters

  • list_exp_data (list of experiment data of type <dict>) – [ for more details refer load_data(), get_exp_data() ]

  • save_fig (bool, optional) – A flag used to save a PDF (default is False)

  • train (bool, optional) – A flag used to insert either _train or _eval in the filename of the PDF (default is True)

  • use_aucs (bool, optional) – A flag used to insert _aucs in the filename of the PDF (default is False)

  • metric_num –

    allowed values-> ‘-1’ to plot episode mean lengths

    ’-2’ to plot episode reward

  • plot_type (string describing how to group data and plot, say based on agent or metric) – allowed values-> [‘agent’ ,’metric’]

plot_2d_heatmap(list_exp_data, save_fig=False, train=True, metric_num=- 2)[source]

Plots 2 2-D heatmaps: 1 for mean and 1 for std. dev. across 2 meta-features of MDP Playground

Parameters

  • list_exp_data (list of experiment data of type <dict>) – [ for more details refer load_data(), get_exp_data() ]

  • save_fig (bool, optional) – A flag used to save a PDF (default is False)

  • train (bool, optional) – A flag used to insert either _train or _eval in the filename of the PDF (default is True)

plot_bar(ax, stats_data, save_fig=False, metric_num=- 2, bar_color='blue')[source]

Plots 1-D bar plots across a single dimension with mean and std. dev.

Parameters

  • ax (matplotlib axes instance to plot) –

  • stats_data (dictionary type with data related to ['train_stats', 'eval_stats', ..]) – [ for more details refer load_data(), get_exp_data() ]

  • save_fig (bool, optional) – A flag used to save a PDF (default is False)

  • metric_num –

    allowed values-> ‘-1’ to plot episode mean lengths

    ’-2’ to plot episode reward

  • bar_color (the color of bars in plots) –

plot_learning_curves(list_exp_data, save_fig=False, train=True, metric_num=- 2)[source]

Plots learning curves: Either across 1 or 2 meta-features of MDP Playground. Different colours represent learning curves for different seeds.

Parameters

  • list_exp_data (list of experiment data of type <dict>) – [ for more details refer load_data(), get_exp_data() ]

  • save_fig (bool, optional) – A flag used to save a PDF (default is False)

  • train (bool, optional) – A flag used to insert either _train or _eval in the filename of the PDF (default is True)

plot_radar(list_exp_data, save_fig=False, train=True, metric_num=- 2, plot_type='agent', weights={}, use_aucs=False)[source]

Plots radar (spider) chart across different dimensions

Parameters

  • list_exp_data (list of experiment data of type <dict>) – [ for more details refer load_data(), get_exp_data() ]

  • save_fig (bool, optional) – A flag used to save a PDF (default is False)

  • train (bool, optional) – A flag used to insert either _train or _eval in the filename of the PDF (default is True)

  • metric_num –

    allowed values-> ‘-1’ to plot episode mean lengths

    ’-2’ to plot episode reward

  • use_aucs (bool, optional) – A flag used to insert _aucs in the filename of the PDF (default is False)

  • plot_type (string describing how to group data and plot, say based on agent or metric) – allowed values-> [‘agent’ ,’metric’]

  • weights (dictionary of weights associated per dimension (plot_type) data) – eg weights[‘reward_noise’] = [.25, .25, .25, .25]

mdp_playground.analysis.analysis mdp_playground.analysis.radar_chart