Zóna pre zamestnancov
a študentov FMFI UK

A Multi-Batch L-BFGS Method for Machine Learning - Martin Takáč (11.7.2016)

v pondelok 11.7.2016 o 10:30 hod. v miestnosti M/125


06. 07. 2016 13.05 hod.
Od: Pavol Brunovský

Prednášajúci: Martin Takáč (Lehigh University USA)

Názov prednášky: A Multi-Batch L-BFGS Method for Machine Learning

Termín: 11.7.2016, 10:30 hod., M/125


Abstrakt:
The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each iteration to facilitate parallelism, and that employ second-order information. In order to improve the learning process, we follow a multi-batch approach in which the batch changes at each iteration. This inherently gives the algorithm a stochastic flavor that can cause instability in L-BFGS, a popular batch method in machine learning. These difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations; when these gradients are computed using different data points the process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, illustrates the behavior of the algorithm in a distributed computing platform, and studies its convergence properties for both the convex and nonconvex cases.