Tensor decompositions in statistical signal processing

Introduction Statistics Tensors TheEnd AppTensor decompositionsin statistical signal processingPierre ComonI3S, CNRS, University of Nice, Sophia-Antipolis, FranceJuly 2, 2009Pierre Comon MMDS - July 2009 1

Introduction Statistics Tensors TheEnd App Model Goals ApplLinear statistical modelwith⎧y :⎪⎨ s :A :b : ⎪⎩y = A s + b (1)K × 1 randomP × 1 random stat. independentK × P deterministicerrorsPierre Comon MMDS - July 2009 2

Introduction Statistics Tensors TheEnd App Model Goals ApplLinear statistical modely = A s + b (1)with⎧y : K × 1 random “sensors”⎪⎨ s : P × 1 random stat. independent “sources”A : K × P deterministic “mixing matrix”b : errors “noise”⎪⎩Pierre Comon MMDS - July 2009 2

Introduction Statistics Tensors TheEnd App Model Goals ApplLinear statistical modely = A s + b (1)with⎧y : K × 1 random “sensors”⎪⎨ s : P × 1 random stat. independent “sources”A : K × P deterministic “mixing matrix”b : errors “noise”⎪⎩(may be removed for P large enough)Pierre Comon MMDS - July 2009 2

Introduction Statistics Tensors TheEnd App Model Goals ApplOther writing[ sy = [A, I]b]Pierre Comon MMDS - July 2009 3

Introduction Statistics Tensors TheEnd App Model Goals ApplOther writing[ sy = [A, I]b]which is of the noiseless formy = A s (2)with a larger dimension PPierre Comon MMDS - July 2009 3

Introduction Statistics Tensors TheEnd App Model Goals ApplTaxinomyProblems addressed in the literature:Pierre Comon MMDS - July 2009 4

Introduction Statistics Tensors TheEnd App Model Goals ApplTaxinomyProblems addressed in the literature:K ≥ P: “over-determined”can be reduced to a square P × P regular mixtureA orthogonal or unitaryA square invertiblePierre Comon MMDS - July 2009 4

Introduction Statistics Tensors TheEnd App Model Goals ApplTaxinomyProblems addressed in the literature:K ≥ P: “over-determined”can be reduced to a square P × P regular mixtureA orthogonal or unitaryA square invertibleK < P: “under-determined”A rectangular with pairwise lin. independent columnsPierre Comon MMDS - July 2009 4

Introduction Statistics Tensors TheEnd App Model Goals ApplTaxinomyProblems addressed in the literature:K ≥ P: “over-determined” → Approximation Pbcan be reduced to a square P × P regular mixtureA orthogonal or unitaryA square invertibleK < P: “under-determined”A rectangular with pairwise lin. independent columnsPierre Comon MMDS - July 2009 4

Introduction Statistics Tensors TheEnd App Model Goals ApplTaxinomyProblems addressed in the literature:K ≥ P: “over-determined” → Approximation Pbcan be reduced to a square P × P regular mixtureA orthogonal or unitaryA square invertibleK < P: “under-determined” → Exact decompostionA rectangular with pairwise lin. independent columnsPierre Comon MMDS - July 2009 4

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yPierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Pierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Estimate matrix A: Blind identificationPierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Estimate matrix A: Blind identificationEstimate realizations of the “source” vector s: Blindseparation/extraction/equalizationPierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Estimate matrix A: Blind identificationEstimate realizations of the “source” vector s: Blindseparation/extraction/equalizationIn the deterministic framework:Pierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Estimate matrix A: Blind identificationEstimate realizations of the “source” vector s: Blindseparation/extraction/equalizationIn the deterministic framework:Build a data tensor Y thanks to a diversity k, and decomposeit:Y ijk = ∑ A ip s jp B kppPierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplGoalsSolely from realizations of observation vector yIn the stochastic framework:Estimate matrix A: Blind identificationEstimate realizations of the “source” vector s: Blindseparation/extraction/equalizationIn the deterministic framework:Build a data tensor Y thanks to a diversity k, and decomposeit:Y ijk = ∑ A ip s jp B kppA and s are jointly estimatedPierre Comon MMDS - July 2009 5

Introduction Statistics Tensors TheEnd App Model Goals ApplApplication areas1 Telecommunications (Cellular, Satellite, Military),2 Radar, Sonar,3 Biomedical (EchoGraphy, ElectroEncephaloGraphy,ElectroCardioGraphy)...4 Speech, Audio,5 Machine learning,6 Data mining,7 Control...Pierre Comon MMDS - July 2009 6

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionIdentifiability & UniquenessUniqueness/Identifiability up to inherent ambiguitiesFinite number of solutionsPierre Comon MMDS - July 2009 7

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionEquivalent representationsLet y admit two representationsy = A s and y = B zwhere s (resp. z) have independent components, and A (resp. B)have pairwise noncollinear columns.Pierre Comon MMDS - July 2009 8

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionEquivalent representationsLet y admit two representationsy = A s and y = B zwhere s (resp. z) have independent components, and A (resp. B)have pairwise noncollinear columns.These representations are equivalent if every column of A isproportional to some column of B, and vice versa.Pierre Comon MMDS - July 2009 8

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionEquivalent representationsLet y admit two representationsy = A s and y = B zwhere s (resp. z) have independent components, and A (resp. B)have pairwise noncollinear columns.These representations are equivalent if every column of A isproportional to some column of B, and vice versa.If all representations of y are equivalent, they are said to beessentially unique (permutation & scale ambiguities only).Pierre Comon MMDS - July 2009 8

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionIdentifiability & uniqueness theoremsLet y be a random vector of the form y = A s, where s p areindependent, and A has non pairwise collinear columns.Pierre Comon MMDS - July 2009 9

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionIdentifiability & uniqueness theoremsLet y be a random vector of the form y = A s, where s p areindependent, and A has non pairwise collinear columns.Identifiability theorem y can be represented asy = A 1 s 1 + A 2 s 2 , where s 1 is non Gaussian, s 2 is Gaussianindependent of s 1 , and A 1 is essentially unique.Pierre Comon MMDS - July 2009 9

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionIdentifiability & uniqueness theoremsLet y be a random vector of the form y = A s, where s p areindependent, and A has non pairwise collinear columns.Identifiability theorem y can be represented asy = A 1 s 1 + A 2 s 2 , where s 1 is non Gaussian, s 2 is Gaussianindependent of s 1 , and A 1 is essentially unique.Uniqueness theorem If in addition the columns of A 1 arelinearly independent, then the distribution of s 1 is unique upto scale and location indeterminacies.Pierre Comon MMDS - July 2009 9

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionExample of uniquenessLet s i be independent with no Gaussian component, and b i beindependent Gaussian. Then the linear model below is identifiable,but A 2 is not essentially unique whereas A 1 is:( )( )( )s1 + s 2 + 2 b 1b1b1 + b= As 1 + 2 b 1 s+A 2 = A2 b 1 s+A 232 b 1 − b 2withA 1 =( 1 11 0) ( 2 0, A 2 =0 2)and A 3 =( 1 11 −1)Hence the distribution of s is essentially unique.But (A 1 , A 2 ) not equivalent to (A 1 , A 3 ).Pierre Comon MMDS - July 2009 10

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionExample of non uniquenessLet s i be independent with no Gaussian component, and b i beindependent Gaussian. Then the linear model below is identifiable,but the distribution of s is not unique because a 2 × 4 matrixcannot be full column rank:( )s1 + s 3 + s 4 + 2 b 1= As 2 + s 3 − s 4 + 2 b 2⎛⎜⎝⎞s 1s 2⎟s 3 + b 1 + b 2s 4 + b 1 − b 2⎛⎠ = A ⎜⎝s 1 + 2 b 1s 2 + 2 b 2s 3s 4⎞⎟⎠withA =( 1 0 1 10 1 1 −1)Pierre Comon MMDS - July 2009 11

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Real Scalar: Φ x (t) def = E{e ı tx } = ∫ u eı tu dF x (u)Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Real Scalar: Φ x (t) def = E{e ı tx } = ∫ u eı tu dF x (u)Real Multivariate: Φx(t) def = E{e ı tT x } =∫u eı tT u dF x(u)Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Real Scalar: Φ x (t) def = E{e ı tx } = ∫ u eı tu dF x (u)Real Multivariate: Φx(t) def = E{e ı tT x } =∫u eı tT u dF x(u)Second c.f.Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Real Scalar: Φ x (t) def = E{e ı tx } = ∫ u eı tu dF x (u)Real Multivariate: Φx(t) def = E{e ı tT x } =∫u eı tT u dF x(u)Second c.f.Ψ(t) def = log Φ(t)Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functionsFirst c.f.Real Scalar: Φ x (t) def = E{e ı tx } = ∫ u eı tu dF x (u)Real Multivariate: Φx(t) def = E{e ı tT x } =∫u eı tT u dF x(u)Second c.f.Ψ(t) def = log Φ(t)Properties:Always exists in the neighborhood of 0Uniquely defined as long as Φ(t) ≠ 0Pierre Comon MMDS - July 2009 12

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functions (cont’d)Properties of the 2nd Characteristic function (cont’d):Pierre Comon MMDS - July 2009 13

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functions (cont’d)Properties of the 2nd Characteristic function (cont’d):if a c.f. Ψ x (t) is a polynomial, then its degree is at most 2 andx is Gaussian (Marcinkiewicz’1938)Pierre Comon MMDS - July 2009 13

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functions (cont’d)Properties of the 2nd Characteristic function (cont’d):if a c.f. Ψ x (t) is a polynomial, then its degree is at most 2 andx is Gaussian (Marcinkiewicz’1938)Proof.complicated...Pierre Comon MMDS - July 2009 13

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functions (cont’d)Properties of the 2nd Characteristic function (cont’d):if a c.f. Ψ x (t) is a polynomial, then its degree is at most 2 andx is Gaussian (Marcinkiewicz’1938)Proof.complicated...if (x, y) statistically independent, thenΨ x,y (u, v) = Ψ x (u) + Ψ y (v) (3)Pierre Comon MMDS - July 2009 13

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionCharacteristic functions (cont’d)Properties of the 2nd Characteristic function (cont’d):if a c.f. Ψ x (t) is a polynomial, then its degree is at most 2 andx is Gaussian (Marcinkiewicz’1938)Proof.complicated...if (x, y) statistically independent, thenΨ x,y (u, v) = Ψ x (u) + Ψ y (v) (3)Proof.Ψ x,y (u, v) = log[E{exp ı(ux + vy)}]= log[E{exp ı(ux)} E{exp ı(vy)}].Pierre Comon MMDS - July 2009 13

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionProblem posed in terms of Characteristic FunctionsIf s p independent and y = A s, we have the core equation:Ψ y (u) = ∑ ( )∑ψ p u q A qpp q(4)Proof.where ψ p (w) def = log E{exp ı(s p w)}.Pierre Comon MMDS - July 2009 14

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionProblem posed in terms of Characteristic FunctionsIf s p independent and y = A s, we have the core equation:Ψ y (u) = ∑ ( )∑ψ p u q A qpp q(4)Proof.where ψ p (w) def = log E{exp ı(s p w)}.Plug y = A s, in definition of Ψ y and getΦ y (u) def = E{exp ı(u T A s)} = E{exp ı( ∑ u q A qp s p )}p,qPierre Comon MMDS - July 2009 14

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionProblem posed in terms of Characteristic FunctionsIf s p independent and y = A s, we have the core equation:Ψ y (u) = ∑ ( )∑ψ p u q A qpp q(4)Proof.where ψ p (w) def = log E{exp ı(s p w)}.Plug y = A s, in definition of Ψ y and getΦ y (u) def = E{exp ı(u T A s)} = E{exp ı( ∑ u q A qp s p )}p,qSince s p independent, φ y (u) = ∏ p E{exp ı(∑ q u q A qp s p )}Pierre Comon MMDS - July 2009 14

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionFunction decompositionProblem:Equation (4) shows that we have to decompose a mutlivariatefunction into a sum of univariate ones.Pierre Comon MMDS - July 2009 15

Introduction Statistics Tensors TheEnd App Uniqueness c.f. DecompositionFunction decompositionProblem:Equation (4) shows that we have to decompose a mutlivariatefunction into a sum of univariate ones.Weierstrass (1885), Stone (1948), Hilbert (1900), Kolmogorov(1957), Sprecher (1965), Hornik (1989), Cybenko (1989)...➽ But here, Ψ y and ψ p ’s are characteristic functions.Pierre Comon MMDS - July 2009 15

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as non symmetric tensor decompositionBack to core equation (4):Ψ y (u) = ∑ pψ p( ∑qu q A qp), u ∈ GAssumption: functions ψ p , 1 ≤ p ≤ P admit finite derivativesup to order d in a neighborhood of the origin, containing G.Pierre Comon MMDS - July 2009 16

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as non symmetric tensor decompositionBack to core equation (4):Ψ y (u) = ∑ pψ p( ∑qu q A qp), u ∈ GAssumption: functions ψ p , 1 ≤ p ≤ P admit finite derivativesup to order d in a neighborhood of the origin, containing G.Taking d = 3 as a working example:∂ 3 Ψ y∂u i ∂u j ∂u k(u) =P∑p=1A ip A jp A kp ψ (3)p (K∑u q A qp )q=1Pierre Comon MMDS - July 2009 16

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as non symmetric tensor decompositionBack to core equation (4):Ψ y (u) = ∑ pψ p( ∑qu q A qp), u ∈ GAssumption: functions ψ p , 1 ≤ p ≤ P admit finite derivativesup to order d in a neighborhood of the origin, containing G.Taking d = 3 as a working example:∂ 3 Ψ y∂u i ∂u j ∂u k(u) =P∑p=1A ip A jp A kp ψ (3)p (K∑u q A qp )q=1of the form T ijkl = ∑ p A ip A jp A kp F lp , if L points u l ∈ G.Pierre Comon MMDS - July 2009 16

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as symmetric tensor decomposition (1/2)If only the origin in G, i.e. u l = 0, thenC ijk = ∑ pA ip A jp A kp f pMulti-linear relation relating cumulants.Pierre Comon MMDS - July 2009 17

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as symmetric tensor decomposition (2/2)Equivalent writing (still with d = 3 as working example)C =P∑f p h(p) ⊗ h(p) ⊗ h(p)p=1where h(p) def = pth column of A.Pierre Comon MMDS - July 2009 18

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as symmetric tensor decomposition (2/2)Equivalent writing (still with d = 3 as working example)C =P∑f p h(p) ⊗ h(p) ⊗ h(p)p=1where h(p) def = pth column of A.If vectors h(p) are not pairwise collinear, P is hopefullyminimal, i.e. equal to the rank of tensor CPierre Comon MMDS - July 2009 18

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicProblem seen as symmetric tensor decomposition (2/2)Equivalent writing (still with d = 3 as working example)C =P∑f p h(p) ⊗ h(p) ⊗ h(p)p=1where h(p) def = pth column of A.If vectors h(p) are not pairwise collinear, P is hopefullyminimal, i.e. equal to the rank of tensor C → conditionPierre Comon MMDS - July 2009 18

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicExpected rankFor an order-d symmetric tensor of dimension KNumber of equations: ( )K+d−1dNumber of unknowns: PKFor what P can one have an exact fit?Pierre Comon MMDS - July 2009 19

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicExpected rankFor an order-d symmetric tensor of dimension KNumber of equations: ( )K+d−1dNumber of unknowns: PKFor what P can one have an exact fit?“Expected rank”:R(K, d) def =⌈( K+d−1dK) ⌉(5)Pierre Comon MMDS - July 2009 19

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicClebsch’s statementAlfred Clebsch (1833-1872)For (d, K) = (4, 3), a generic tensor cannot be written as the sumof 5 rank-1 terms, even if #unknowns = 15 = #equationsPierre Comon MMDS - July 2009 20

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicHirschowitz theoremFrom Alexander-Hirschowitz thm (1995), one deduces [CGLM08]:Theorem For d > 2, the generic rank of a dth order symmetrictensor of dimension K is always equal to the expected rank¯R s = R(K, d) (6)except for (d, K) ∈ {(3, 5), (4, 3), (4, 4), (4, 5)}➽ Only a finite number of exceptions !Pierre Comon MMDS - July 2009 21

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityIdentifiability in wider sense: finite number of solutionsPierre Comon MMDS - July 2009 22

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityIdentifiability in wider sense: finite number of solutionsNecessary condition for identifiability: P ≤ R(K, d).Not sufficient, cf. Clebsch (1861), Hirschowitz (1995)Pierre Comon MMDS - July 2009 22

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityIdentifiability in wider sense: finite number of solutionsNecessary condition for identifiability: P ≤ R(K, d).Not sufficient, cf. Clebsch (1861), Hirschowitz (1995)Sufficient condition for almost sure identifiability:P < R(K, d).Pierre Comon MMDS - July 2009 22

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityIdentifiability in wider sense: finite number of solutionsNecessary condition for identifiability: P ≤ R(K, d).Not sufficient, cf. Clebsch (1861), Hirschowitz (1995)Sufficient condition for almost sure identifiability:P < R(K, d).NS condition for almost sure identifiability:( K+d−1)dP = = R(K, d) =K¯R sPierre Comon MMDS - July 2009 22

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicGeneric rank of symmetric tensorsSymmetric tensors of order d and dimension K:d K 2 3 4 5 6 7 82 2 3 4 5 6 7 83 2 4 5 8 10 12 154 3 6 10 15 21 30 42Pierre Comon MMDS - July 2009 23

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicDeterministic approachesModel:Y ijk = ∑ pA ip s jp B kpY = ∑ Pp=1a(p) ⊗ s(p) ⊗ b(p)Pierre Comon MMDS - July 2009 24

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicExpected rankFor an order-d symmetric tensor of dimensions K l , 1 ≤ l ≤ dNumber of equations: ∏ l K lNumber of unknowns: ∑ l K lP − (d − 1)PPierre Comon MMDS - July 2009 25

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicExpected rankFor an order-d symmetric tensor of dimensions K l , 1 ≤ l ≤ dNumber of equations: ∏ l K lNumber of unknowns: ∑ l K lP − (d − 1)P“Expected rank” again given by the ceil rule:⌈ ∏R(K 1 , .., K d ) defl=K ⌉l∑l K l − d + 1(7)Pierre Comon MMDS - July 2009 25

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityNecessary condition for identifiability:P ≤ R(K 1 , .., K d )Pierre Comon MMDS - July 2009 26

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityNecessary condition for identifiability:P ≤ R(K 1 , .., K d )Sufficient condition for almost sure identifiability:P < R(K 1 , .., K d )Pierre Comon MMDS - July 2009 26

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicIdentifiabilityNecessary condition for identifiability:P ≤ R(K 1 , .., K d )Sufficient condition for almost sure identifiability:P < R(K 1 , .., K d )For general tensors, generic rank ¯R not yet known everywhere.Pierre Comon MMDS - July 2009 26

Introduction Statistics Tensors TheEnd App Non symmetric Symmetric DeterministicGeneric rank of order 3 free tensors (1)K 2 3 4J 2 3 4 5 3 4 5 4 52 2 3 4 4 3 4 5 4 53 3 3 4 5 5 5 5 6 64 4 4 4 5 5 6 6 7 85 4 5 5 5 5 6 8 8 96 4 6 6 6 6 7 8 8 10I 7 4 6 7 7 7 7 9 9 108 4 6 8 8 8 8 9 10 119 4 6 8 9 9 9 9 10 1210 4 6 8 10 9 10 10 10 1211 4 6 8 10 9 11 11 11 1312 4 6 8 10 9 12 12 12 13Pierre Comon MMDS - July 2009 27

Introduction Statistics Tensors TheEnd AppPerspectivesEfficient algorithms to compute finite set of solutionsPierre Comon MMDS - July 2009 28

Introduction Statistics Tensors TheEnd AppPerspectivesEfficient algorithms to compute finite set of solutionsDetection of weak defectivity: rare cases when ∞ manysolutionsPierre Comon MMDS - July 2009 28

Introduction Statistics Tensors TheEnd AppPerspectivesEfficient algorithms to compute finite set of solutionsDetection of weak defectivity: rare cases when ∞ manysolutionsConstruction of additional diversity: increases orderPierre Comon MMDS - July 2009 28

Introduction Statistics Tensors TheEnd AppPerspectivesEfficient algorithms to compute finite set of solutionsDetection of weak defectivity: rare cases when ∞ manysolutionsConstruction of additional diversity: increases orderThanks for your attentionPierre Comon MMDS - July 2009 28

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refDarmois-Skitovich theorem (1953)TheoremLet s i be statistically independent random variables, and two linearstatistics:y 1 = ∑ a i s i and y 2 = ∑ b i s iiiIf y 1 and y 2 are statistically independent, then random variables s kfor which a k b k ≠ 0 are Gaussian.NB: holds in both R or CPierre Comon MMDS - July 2009 29

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refProof of Darmois-Skitovich thmAssume [a k , b k ] not collinear and ψ p differentiable.Pierre Comon MMDS - July 2009 30

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refProof of Darmois-Skitovich thmAssume [a k , b k ] not collinear and ψ p differentiable.1 Hence ∑ Pk=1 ψ p(u a k + v b k ) = ∑ Pk=1 ψ k(u a k ) + ψ k (v b k )Trivial for terms for which a k b k = 0.From now on, restrict the sum to terms a k b k ≠ 0Pierre Comon MMDS - July 2009 30

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refProof of Darmois-Skitovich thmAssume [a k , b k ] not collinear and ψ p differentiable.1 Hence ∑ Pk=1 ψ p(u a k + v b k ) = ∑ Pk=1 ψ k(u a k ) + ψ k (v b k )Trivial for terms for which a k b k = 0.From now on, restrict the sum to terms a k b k ≠ 02 Write this at u + α/a P and v − α/b P :P∑k=1(ψ k u a k + v b k + α( a k− b )k) = f (u) + g(v)a P b PPierre Comon MMDS - July 2009 30

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refProof of Darmois-Skitovich thmAssume [a k , b k ] not collinear and ψ p differentiable.1 Hence ∑ Pk=1 ψ p(u a k + v b k ) = ∑ Pk=1 ψ k(u a k ) + ψ k (v b k )Trivial for terms for which a k b k = 0.From now on, restrict the sum to terms a k b k ≠ 02 Write this at u + α/a P and v − α/b P :P∑k=1(ψ k u a k + v b k + α( a k− b )k) = f (u) + g(v)a P b P3 Subtract to cancel Pth term, divide by α, and let α → 0:P−1∑( a k− b k) ψ (1)a P bk(u a k + v b k ) = f (1) (u) + g (1) (v)Pk=1for some univariate functions f (1) (u) and g (1) (u).Pierre Comon MMDS - July 2009 30

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refProof of Darmois-Skitovich thmAssume [a k , b k ] not collinear and ψ p differentiable.1 Hence ∑ Pk=1 ψ p(u a k + v b k ) = ∑ Pk=1 ψ k(u a k ) + ψ k (v b k )Trivial for terms for which a k b k = 0.From now on, restrict the sum to terms a k b k ≠ 02 Write this at u + α/a P and v − α/b P :P∑k=1(ψ k u a k + v b k + α( a k− b )k) = f (u) + g(v)a P b P3 Subtract to cancel Pth term, divide by α, and let α → 0:P−1∑( a k− b k) ψ (1)a P bk(u a k + v b k ) = f (1) (u) + g (1) (v)Pk=1for some univariate functions f (1) (u) and g (1) (u).Conclusion: We have one term lessPierre Comon MMDS - July 2009 30

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=2Pierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=25 Hence ψ (P−1)1(u a 1 + v b 1 ) is linear, as a sum of two univariatefunctions (ψ (P)1is a constant because a 1 b 1 ≠ 0).Pierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=25 Hence ψ (P−1)1(u a 1 + v b 1 ) is linear, as a sum of two univariatefunctions (ψ (P)1is a constant because a 1 b 1 ≠ 0).6 Eventually ψ 1 is a polynomial.Pierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=25 Hence ψ (P−1)1(u a 1 + v b 1 ) is linear, as a sum of two univariatefunctions (ψ (P)1is a constant because a 1 b 1 ≠ 0).6 Eventually ψ 1 is a polynomial.7 Lastly invoke Marcinkiewicz theorem to conclude that s 1 isGaussian.Pierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=25 Hence ψ (P−1)1(u a 1 + v b 1 ) is linear, as a sum of two univariatefunctions (ψ (P)1is a constant because a 1 b 1 ≠ 0).6 Eventually ψ 1 is a polynomial.7 Lastly invoke Marcinkiewicz theorem to conclude that s 1 isGaussian.8 Same is true for any ψ p such that a p b p ≠ 0: s p is Gaussian.Pierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm ref4 Repeat the procedure (P − 1) times and get:P∏( a 1− b 1) ψ (P−1)1(u a 1 + v b 1 ) = f (P−1) (u) + g (P−1) (v)a j b jj=25 Hence ψ (P−1)1(u a 1 + v b 1 ) is linear, as a sum of two univariatefunctions (ψ (P)1is a constant because a 1 b 1 ≠ 0).6 Eventually ψ 1 is a polynomial.7 Lastly invoke Marcinkiewicz theorem to conclude that s 1 isGaussian.8 Same is true for any ψ p such that a p b p ≠ 0: s p is Gaussian.NB: also holds if ψ p not differentiablePierre Comon MMDS - July 2009 31

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refDefinition of CumulantsMoments:defµ r = E{x r } = (−ı) r ∂ r ∣Φ(t) ∣∣∣t=0∂t rPierre Comon MMDS - July 2009 32

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refDefinition of CumulantsMoments:Cumulants:C x (r)defdefµ r = E{x r } = (−ı) r ∂ r ∣Φ(t) ∣∣∣t=0∂t r= Cum{x, . . . , x} = (−ı) r ∂ r Ψ(t)} {{ }∂t rr times∣∣∣∣t=0Pierre Comon MMDS - July 2009 32

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refDefinition of CumulantsMoments:Cumulants:C x (r)defdefµ r = E{x r } = (−ı) r ∂ r ∣Φ(t) ∣∣∣t=0∂t r= Cum{x, . . . , x} = (−ı) r ∂ r Ψ(t)} {{ }∂t rr times∣∣∣∣t=0Relationship between Moments and Cumulants obtained byexpanding both sides in Taylor series:log Φ x (t) = Ψ x (t)Pierre Comon MMDS - July 2009 32

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refPolynomial interpolationAlexander-Hirschowitz Theorem (1995) Let L(d, m) be thespace of hypersurfaces of degree at most d in m variables. Thisspace is of dimension D(m, d) def = ( )m+d − 1.Theorem Denote {p i } K given distinct points in the complexprojective space P m . The dimension of the linear subspace ofhypersurfaces of L(d, m) having multiplicity at least 2 at everypoint p i is:D(m, d) − K(m + 1)except for the following cases:• d = 2 and 2 ≤ K ≤ m• d ≥ 3 and (m, d, K) ∈ {(2, 4, 5), (3, 4, 9), (4, 1, 14), (4, 3, 7)}dIn other words, there are a finite number of exceptions.Pierre Comon MMDS - July 2009 33

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refReferencesF. L. HITCHCOCK. “Multiple invariants and generalized rank of a p-way matrix or tensor.” J. Math. andPhys., 7(1):39–79, 1927.L. R. TUCKER. “Some mathematical notes for three-mode factor analysis.” Psychometrika, 31:279–311,1966.J. B. KRUSKAL. “Three-way arrays: Rank and uniqueness of trilinear decompositions.” Linear Algebra andApplications, 18:95–138, 1977.V. STRASSEN. “Rank and optimal computation of generic tensors.” Linear Algebra Appl., 52:645–685, July1983.P. COMON and B. MOURRAIN. “Decomposition of quantics in sums of powers of linear forms.” SignalProcessing, Elsevier, 53(2):93–107, September 1996.N. D. SIDIROPOULOS and R. BRO. “On the uniqueness of multilinear decomposition of N-way arrays.” J.Chemometrics, 14:229–239, 2000.P. COMON. “Tensor decompositions.” In J. G. McWhirter and I. K. Proudler eds, Mathematics in SignalProcessing V, pages 1–24. Clarendon Press, Oxford, UK, 2002.A. SMILDE, R. BRO, and P. GELADI. Multi-Way Analysis. Wiley, 2004.P. COMON, G. GOLUB, L-H. LIM, and B. MOURRAIN. “Symmetric tensors and symmetric tensor rank.”SIAM Journal on Matrix Analysis Appl., 30(3):1254–1279, 2008.P. COMON, J. M. F. ten BERGE, L. De LATHAUWER and J. CASTAING. “Generic and Typical Ranks ofMulti-Way Arrays.” Linear Algebra Appl., 2009. See also http://arxiv.org/abs/0802.2371P. COMON, X. LUCIANI and A. de ALMEIDA. “Tensor decompositions, alternating least squares and othertales.” J. Chemometrics. 2009, special issue.Pierre Comon MMDS - July 2009 34

Introduction Statistics Tensors TheEnd App Darmois proof Cumulants AH-thm refReferencesParticipants in ANR-06-BLAN-0074 “DECOTES” project:Signal team, Lab. I3S UMR6070, Univ. Nice and CNRSGalaad team, INRIA, UR Sophia-Antipolis - Méditerranée.Lab. LTSI U642, INSERM and Univ. Rennes 1Thalès CommunicationsPierre Comon MMDS - July 2009 35