This project focuses on Restricted Boltzmann Machines (RBM), which are graphical
probabilistic models that can be used in the pre-training step of deep learning
models. Due to its probabilistic nature and its nice mathematical formulation,
it di ers from many other machine learning algorithms. In particular, once the
model is trained, you can obtain additional information not given by the other
algorithms, such as the probabilities of new instances. Nevertheless, there are
numerous terms that are hard to compute when training Restricted Boltzmann
Machines, such as the partition function and the derivatives of the log-likelihood.
These drawbacks have prevented RBMs to show their real potential as truly probabilistic
models. Therefore, in this project we investigate the e ects of previously
presented techniques and propose a new training algorithm, Weighted Parallel
Tempering, that ts the RBM to the training set closer, and allows us to better
understand the behavior of RBMs. We compare the performance of each training
algorithm in multiple application scenarios: ranging from low-dimensional arti cial
datasets to high-dimensional real problems. Our experiments demonstrate that a
weighted version of Parallel Tempering is likely to be a very promising procedure
to learn the parameters of Restricted Boltzmann Machines. Last, we describe a
new approach to evaluate the sampling power over Restricted Boltzmann Machines
with a low computational cost. The experiments demonstrate that Parallel Tempering
is likely to be a better sampling algorithm than Gibbs sampling and, for
this reason, the weighted version of Parallel Tempering often outperforms other
training algorithms proposed for learning the parameters of Restricted Boltzmann
Machines.
