Segmentation of heterogeneous document images : an ... - Tel

∂l λ
= ∑ (
∑y∈Y
f k (y s , x s ) −
f k(y s , x s ) exp ∑ F
i=1 λ )
if i (y s , x s )
∂λ k Z(x s , λ)
s∈S
⎛
⎞
= ∑ ⎝f k (y s , x s ) − ∑ f k (y s , x s )P (y|x s ) ⎠
s∈S
y∈Y
= ∑ s∈S
(
fk (y s , x s ) − E P (y|xs)[f k (y s , x s )] )
tel-00912566, version 1 - 2 Dec 2013
where Y = {text, non-text} and E p() [.] is the expected value of the model
under the conditional probability distribution. For maximum likelihood solution,
the equation will equal zero, and therefore the expectation of the feature
f k with respect to the model distribution must be equal to the expected value
of f k with respect to the empirical distribution. However, calculating the expectation
requires the enumeration of all the y labels. In linear-chain models,
inference techniques based on a variation of forward-backward algorithms can be
performed to efficiently compute this expectation. However, in two-dimensional
CRFs, approximation techniques are needed to simplify the computations. One
solution is to use a Voted Perceptron Method.
4.6.1 Collin’s voted perceptron method
Perceptrons [81] use an approximation of the gradient of the unregularized conditional
log-likelihood. Perceptron-based training methods consider one misclassified
instance at a time, along with its contribution to the gradient. The
expectation of features are further approximated by a point estimate of the feature
function vector at the best possible labeling. The approximation for the
i th instance and the k th feature function can be written as:
∇ k l(λ) ≈ ( f k (y i , x i ) − f k (ŷ i , x k ) )
where
ŷ i = arg max λ k f k (y, x i )
y
Using this approximate gradient, the following first order update rule can
be used for maximization:
λ t+1
k
= λ t k + α ( f k (y i , x i ) − f k (ŷ i , x i ) ) .
where α is the learning rate. This update step is applied once for each missclassified
instance x i in the training set and multiple passes are made over the
training dataset. Thought, it has been noted that the final obtained weights
suffer from over-fitting. As a solution, Collins [26] suggests a voting scheme,
where, in a particular pass of the training data, all the updates are collected,
69