A Time-Dependent Enhanced Support Vector Machine For Time ...

Table 2: Support Vector Regression and Time-Dependent Support Vector Regression empirical loss and its
derivative
Support Vector Regression Time-Dependent Support Vector Regression
Empirical loss l 1(x i,y i,w) max(0, |w T x i – y i| – ǫ) max(0, |w T x i – y i| – ǫ)
Derivative l 1(x ′ i,y i,w) sign(w T x i–y i) if |w T x i–y i| > ǫ; sign(w T x i – y i) if |w T x i – y i| > ǫ;
otherwise 0 otherwise 0
Time-Dependent loss 0 for all samples max (0, |(w T x i – y i) – (w T x i−1 – y i−1)| - ǫ t )
l 2(x i,y i,w,x i−1,y i−1) for distribution shift samples; otherwise 0
Derivative 0 for all samples sign((w T x i–y i)–(w T x i−1–y i−1))
l 2(x ′ i,y i,w,x i−1,y i−1)
if |(w T x i–y i)–(w T x i−1 – y i−1)| > ǫ t
for distribution shift samples, otherwise 0
l 2(x i,y i,w, x i−1,y i−1) is
{ | (w T x i −y i )−(w T x i−1 −y i−1 ) | x i−1 is dist. shift
0 otherwise
(5)
where it can be seen that if x i−1 is not a distribution shift
sample, then no loss is incurred. Otherwise, the loss is equal
to the amount of difference between the prediction error for
x i and theprediction error for x i−1. Wecan further incorporate
a tolerable error term ǫ t to yield a soft time-dependent
loss. This is shown in Table 2. By linearly combining our
time-dependentloss function with the standardloss function
for SVM regression, we formulate a new type of SVM, which
we henceforth refer to as TiSe SVM (time sensitive SVM).
Observe that if no samples are classified as distribution shift
samples, then a TiSE SVM is exactly the same as SVM. Using
the definitions from Table 2, the modified empirical loss
and regularized risk function for TiSE SVM have the form
R TiSe
emp (w) = 1 n
n∑
(l 1 (x i ,y i ,w)+λ∗l 2 (x i ,y i ,w,x i−1 ,y i−1 ))
i=1
(6)
L(w ∗ ) = argmin w φw T w+R TiSe
emp (w) (7)
The λ parameter is a regularization parameter that determines
the extent to which we want to minimize the timedependent
loss function. Larger values of λ will result in the
time-dependent loss function having more influence in the
overall error. We investigate the effect of this parameter in
the experimental section, in order to determine values suitable
for our experimental work.
3.4 SVM Regression time-dependent loss optimization
The new SVM regression which incorporates the timedependent
loss will now have the following form:
L TiSe (w ∗ ) = 1 ∑ n
∑ n
2 ||w||2 +C 1 (ξ + i +ξ − i )+C 2 (ζ + i +ζ − i )
i=1
i=2
subject to
⎧
〈w,x i〉+b+ǫ+ξ + i ≥ y i
〈w,x ⎪⎨ i〉+b−ǫ−ξ − i
≤ y i
(〈w,x i〉−y i)−(〈w,x i−1〉−y i−1)+ǫ t +ζ + i
≥ 0
(〈w,x i〉−y i)−(〈w,x i−1〉−y i−1)−ǫ t −ζ − i
≤ 0
⎪⎩
ξ + i ,ξ− i ,ζ+ i ,ζ− i
≥ 0
(8)
(9)
where ǫ t is the allowed value of the differences in the time
sensitiveerror, andζ + andζ − areslackvariablesforthetime
sensitive loss. In order to solve this, we introduce Lagrange
multipliers α + i
≥ 0, α − i
≥ 0, µ + i
≥ 0 and µ − i
≥ 0 for all i,
β + i , β− i , η+ i
and η − i
for i = 2...n:
∑ n n∑
L P =C 1 (ξ + i +ξ − i )+C 2 (ζ + i +ζ − i )+ 1 2 ||w||2
i=1 i=2
n∑
n
− (µ + i ξ+ i +µ − i ξ− i )− ∑
α + i (ǫ+ξ+ i −y i +〈w,x i 〉+b)
i=1
i=1
n∑
− α − i (ǫ+ξ− i +y i −〈w,x i 〉−b)
i=1
n∑
−λ (η + i ζ+ i +η − i ζ− i )
i=2
n∑
−λ β + i ((〈w,x i〉−y i )−(〈w,x i−1 〉−y i−1 )+ǫ t +ζ + )
i=2
n∑
−λ β − i (−(〈w,x i〉−y i )+(〈w,x i−1 〉−y i−1 )+ǫ t +ζ − )
i=2
(10)
Differentiating with respect to w, b, ξ + i , ξ− i , ζ+ i
and ζ − i
and
setting the derivatives to 0, we will get the dual form:
n∑
n∑
L D = (α + i −α − i )y i +λ (β + i −β − i )y i
i=1
i=2
n∑
n∑
−λ (β + i −β − i )y i−1 −ǫ (α + i +α − ∑ n
i )−λǫt (β + i +β − i )
i=2
i=1
− 1 n∑
(α + i −α − i
2
)(α+ j −α− j )x ix j
i,j=1
i=2
− 1 ∑ n
2 λ2 (β + i −β − i )(β+ j −β− j )(x ix j −2x i x j−1 +x i−1 x j−1 )
−λ
i,j=2
n∑
(α + i −α − i )(β+ j −β− j )(x ix j −x i x j−1 )
i=1,j=2
(11)
This form allows for a Quadratic Programming to be applied
in order to find w and b. Also, it can be noticed that if
we need to move to a higher dimensionality space x → ψ(x),
such that a kernel function exists k(x ix j) = ψ(x i)ψ(x j), we
can do so as L D can be kernelized.