Segmentation of heterogeneous document images : an ... - Tel

incorrect [100]. Several sufficient (but not necessary) conditions for convergence
of loopy belief propagation to a unique fixed point exist [6]. Even so, the precise
conditions under which loopy belief propagation will converge are still not well
understood in its application for computer vision. As a consequence, through
our experiments we found that the usage of some feature functions will result
to no answer from L-BFGS optimization.
tel-00912566, version 1 - 2 Dec 2013
4.6.3 L-BFGS
Perhaps the simplest approach to optimize l(λ) is by using a steepest ascent
algorithm, but in practice, this requires too many iterations. Newton’s method
converges much faster because it takes into account the second derivatives of the
likelihood function, but it needs to compute Hessian, the matrix of all second
derivatives. The size of the Hessian matrix is quadratic in the number of parameters
and since real-world applications often use thousands (e.g. computer
vision) or even millions (e.g. natural language processing) of parameters, even
storing the full Hessian matrix in memory is impractical.
Instead, the latest optimization methods make approximate use of secondorder
information. Quasi-Newton methods such as BFGS, which compute an
approximation of Hessian from only the first derivative of the objective function,
have been successfully applied [55]. A full K × K approximation to the Hessian
still requires quadratic size. Therefore, a limited-memory version of the method
(L-BFGS) is developed [20].
L-BFGS can simply be treated as a black-box optimization procedure, requiring
only that one provides the value and first-derivative of the function to
be optimized. The important note is that the use of L-BFGS does not make
it easier to compute the partition function. Recall that the partition function
Z(x, λ) and the marginal distributions p(y s |x s , λ) in the gradients, depend on λ;
thus, every training instance in each iteration has a different partition function
and marginal. Even in linear-chain CRFs that the partition function Z can be
computed efficiently by forward-backward algorithms, it is reported that on a
part-of-speech (POS) tagging data set, with 45 labels and one million words of
training data, training requires over a week [94].
Our implementation of L-BFGS is based on libLBFGS library [72] in C++.
4.7 Experimental results
Our training dataset consists of 28 pages from a collection of our corpus. It
contains a large variety of page layouts including articles, journals, forms and
multi-columns documents with or without side notes and graphical components.
For each page in our dataset we generated a ground-truth image and a map
that indicates the location of between-columns in multi-column documents. The
former is used only for tracking the rate of errors in those areas and is not part
of a training routine. Figure 4.9 shows two pages in our training dataset along
with their corresponding ground-truth. Note that three colors are used as part
71