Accelerating generalized iterative scaling using componentwise extrapolations for on-line conditional random fields

In this paper, a simple and globally convergent method based on penalized generalized iterative scaling (GIS) with staggered Aitken acceleration is proposed to efficiently estimate the parameters for an on-line conditional random field (CRF). The staggered Aitken acceleration method, which alternates between an acceleration step and a non-acceleration step, provides numerical stability and computational simplicity in analyzing the incompleteness of data. The proposed method is based on stochastic gradient descent (SGD) and it has the following advantages: (1) it can approximate parameters close to the empirical optimum in a single pass through the training examples; (2) it can reduce the computing time by eliminating computation of the inverse of the objective function's Hessian matrix with staggered Aitken acceleration. We show the convergence of penalized GIS based on the staggered Aitken acceleration method, compare its speed of convergence with that of other stochastic optimization methods, and also illustrate experimental results with public data sets.