smac.epm.gaussian_process_mcmc module

class smac.epm.gaussian_process_mcmc.GaussianProcessMCMC(configspace: ConfigSpace.configuration_space.ConfigurationSpace, types: List[int], bounds: List[Tuple[float, float]], seed: int, kernel: skopt.learning.gaussian_process.kernels.Kernel, n_mcmc_walkers: int = 20, chain_length: int = 50, burnin_steps: int = 50, normalize_y: bool = True, mcmc_sampler: str = 'emcee', average_samples: bool = False, instance_features: Optional[numpy.ndarray] = None, pca_components: Optional[int] = None)[source]

Bases: smac.epm.base_gp.BaseModel

Gaussian process model.

The GP hyperparameters are integrated out by MCMC. If you use this class make sure that you also use an integrated acquisition function to integrate over the GP’s hyperparameter as proposed by Snoek et al.

This code is based on the implementation of RoBO:

Klein, A. and Falkner, S. and Mansur, N. and Hutter, F. RoBO: A Flexible and Robust Bayesian Optimization Framework in Python In: NIPS 2017 Bayesian Optimization Workshop

Parameters
  • types (List[int]) – Specifies the number of categorical values of an input dimension where the i-th entry corresponds to the i-th input dimension. Let’s say we have 2 dimension where the first dimension consists of 3 different categorical choices and the second dimension is continuous than we have to pass [3, 0]. Note that we count starting from 0.

  • bounds (List[Tuple[float, float]]) – bounds of input dimensions: (lower, uppper) for continuous dims; (n_cat, np.nan) for categorical dims

  • seed (int) – Model seed.

  • kernel (george kernel object) – Specifies the kernel that is used for all Gaussian Process

  • n_mcmc_walkers (int) – The number of hyperparameter samples. This also determines the number of walker for MCMC sampling as each walker will return one hyperparameter sample.

  • chain_length (int) – The length of the MCMC chain. We start n_mcmc_walkers walker for chain_length steps and we use the last sample in the chain as a hyperparameter sample.

  • burnin_steps (int) – The number of burnin steps before the actual MCMC sampling starts.

  • normalize_y (bool) – Zero mean unit variance normalization of the output values

  • mcmc_sampler (str) – Choose a self-tuning MCMC sampler. Can be either emcee or nuts.

  • instance_features (np.ndarray (I, K)) – Contains the K dimensional instance features of the I different instances

  • pca_components (float) – Number of components to keep when using PCA to reduce dimensionality of instance features. Requires to set n_feats (> pca_dims).

_get_gp() → skopt.learning.gaussian_process.gpr.GaussianProcessRegressor[source]
_ll(theta: numpy.ndarray) → float[source]

Returns the marginal log likelihood (+ the prior) for a hyperparameter configuration theta.

Parameters

theta (np.ndarray(H)) – Hyperparameter vector. Note that all hyperparameter are on a log scale.

Returns

lnlikelihood + prior

Return type

float

_ll_w_grad(theta: numpy.ndarray) → Tuple[float, numpy.ndarray][source]

Returns the marginal log likelihood (+ the prior) for a hyperparameter configuration theta.

Parameters

theta (np.ndarray(H)) – Hyperparameter vector. Note that all hyperparameter are on a log scale.

Returns

lnlikelihood + prior

Return type

float

_predict(X_test: numpy.ndarray, cov_return_type: Optional[str] = 'diagonal_cov') → Tuple[numpy.ndarray, numpy.ndarray][source]

Returns the predictive mean and variance of the objective function at X average over all hyperparameter samples. The mean is computed by: :math mu(x) = frac{1}{M}sum_{i=1}^{M}mu_m(x) And the variance by: :math sigma^2(x) = (frac{1}{M}sum_{i=1}^{M}(sigma^2_m(x) + mu_m(x)^2) - mu^2

Parameters
  • X_test (np.ndarray (N, D)) – Input test points

  • cov_return_type (typing.Optional[str]) – Specifies what to return along with the mean. Refer predict() for more information.

Returns

  • np.array(N,) – predictive mean

  • np.array(N,) – predictive variance

_train(X: numpy.ndarray, y: numpy.ndarray, do_optimize: bool = True)smac.epm.gaussian_process_mcmc.GaussianProcessMCMC[source]

Performs MCMC sampling to sample hyperparameter configurations from the likelihood and trains for each sample a GP on X and y

Parameters
  • X (np.ndarray (N, D)) – Input data points. The dimensionality of X is (N, D), with N as the number of points and D is the number of features.

  • y (np.ndarray (N,)) – The corresponding target values.

  • do_optimize (boolean) – If set to true we perform MCMC sampling otherwise we just use the hyperparameter specified in the kernel.