Wei-Lun Chao, Jun-Zuo Liu, and Jian-Jiun Ding

This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institutionand sharing with colleagues.Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to personal, institutional or third partywebsites are prohibited.In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website orinstitutional repository. Authors requiring further informationregarding Elsevier’s archiving and manuscript policies areencouraged to visit:http://www.elsevier.com/copyright

Author's personal copyPattern Recognition 46 (2013) 628–641Contents lists available at SciVerse ScienceDirectPattern Recognitionjournal homepage: www.elsevier.com/locate/prFacial age estimation based on label-sensitive learning andage-oriented regressionWei-Lun Chao n , Jun-Zuo Liu, Jian-Jiun DingGraduate Institute of Communication Engineering, National Taiwan University, No. 1, Section 4, Roosevelt Rd., Taipei 10617, Taiwanarticle infoArticle history:Received 16 March 2012Received in revised form9 September 2012Accepted 17 September 2012Available online 24 September 2012Keywords:Machine learningPattern recognitionManifold learningDimensionality reductionDistance metric learningLocal regressionAge estimationabstractThis paper provides a new age estimation approach, which distinguishes itself with the following threecontributions. First, we combine distance metric learning and dimensionality reduction to betterexplore the connections between facial features and age labels. Second, to exploit the intrinsic ordinalrelationship among human ages and overcome the potential data imbalance problem, a label-sensitiveconcept and several imbalance treatments are introduced in the system training phase. Finally, anage-oriented local regression is presented to capture the complicated facial aging process for agedetermination. The simulation results show that our approach achieves the lowest estimation erroragainst existing methods.& 2012 Elsevier Ltd. All rights reserved.1. IntroductionIn the last few decades, with the increasing need of automaticrecognition and surveillance systems, researches on human faces– including face detection, face recognition, gender classification,and facial expression recognition – have attracted significantattention in both computer vision and pattern recognition.Compared to these face-related researches, facial age estimationis a relatively new topic yet with several interesting and importantpotential usage. For example, an automatic age estimationsystem can not only facilitate the human–computer interface, butalso prevent under ages from accessing cigarettes, bears, andpornographic websites. In addition, the age attribute has beenapplied in face verification and retrieval [24] to enhance theoverall performance.Estimating human ages is intrinsically a challenging task dueto its multi-class nature: An age label can be seen as an individualclass. This nature makes age estimation much harder than genderclassification and face detection from the perspective of machinelearning, where well-studied binary classifiers cannot be directlyapplied. This nature also makes age estimation easily suffer theover-fitting problem when the size of the training set is insufficient.Moreover, because of the diversity of facial aging processes acrosspeople, it is very difficult to design and determine the type of facialn Corresponding author. Tel./fax: þ886 2 33663662.E-mail addresses: weilunchao760414@gmail.com (W.-L. Chao),jinzuo2011@gmail.com (J.-Z. Liu), djj@cc.ee.ntu.edu.tw (J.-J. Ding).features that can directly represents human ages. To solve theseproblems, several age estimation algorithms have been published inthe past decade; these algorithms are generally composed of twosteps: feature extraction [10,12,14,20,22,25,26,31,35,46,47] andagedetermination [5,6,18,19,32,40,42–44,51].Apart from the challenges mentioned above, three importantfactors should also be regarded for building a robust age estimationsystem. First, there exist the ordinal relationship and correlationsamong age labels; for example, age 30 is closer to age 25 than toage 10. This relationship leads age estimation to a more difficultlearning task than the traditional multi-class classification problem,which assumes no correlations among classes. Althoughtechniques like regression and cost-sensitive learning could modelthe ordinal relationship in their objective functions, most of thedistance metric learning and dimensionality reduction algorithmsare designed only for the traditional classification case and ignorethe correlations among labels. Second, as mentioned in [18], theaging process on human faces is rather complicated and may not befully captured by a single classifier or regressor. Finally in agedatabases, the number of images of each age label can be highlydifferent, which would result in the serious imbalance problemduring the training phase and degrade the overall performance ofage estimation.In this paper, a new facial age estimation approach is proposed,which takes all the above factors and challenges into consideration: To avoid over-fitting and explore the connections betweenfacial features and age labels, the locality preserving projection(LPP) [21] is utilized to reduce the dimensionality of features0031-3203/$ - see front matter & 2012 Elsevier Ltd. All rights reserved.http://dx.doi.org/10.1016/j.patcog.2012.09.011

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 629drastically and preserve the most important information forage estimation. To better exploit the ordinal relationship among age labels, alabel-sensitive concept is introduced, which regards the labelsimilarity during the training phase of LPP. To capture the complicated facial aging process, an age-orientedlocal regression algorithm named KNN-SVR (K nearest neighborssupportvector regression) is presented. To alleviate the imbalance problem, several imbalance treatmentsare proposed for some steps in our approach.In addition, to further enhance the performance of LPP, a distancemetric adjustment step is introduced (just before LPP) to achieve asuitable space for neighbor searching, an essential operation in LPP.The label-sensitive concept could also be employed in this step.The proposed approach is examined under several experimentalsettings and evaluation criteria on the most widely-usedFG-NET aging database [53], and the simulation results demonstratethe effectiveness and efficiency of our approach. Besides, tofurther illustrate the availability of the label-sensitive concept ondimensionality reduction algorithms other than LPP, we applythis concept to another popular algorithm called the marginalfisher analysis (MFA) [45]; the results also show significantimprovements in the age estimation performance.This paper is organized as follows: In Section 2, we broadlyreview the previous work on facial age estimation and somerelated techniques of our approach. In Section 3, an overview ofthe proposed approach is presented, and the label-sensitive conceptis introduced in Section 4. InSections 5 and 6, the proposedalgorithms of distance metric adjustment and dimensionalityreduction as well as their corresponding imbalance treatmentsare described; the proposed age-oriented local regression KNN-SVR and a short summary of our approach are presented inSections 7 and 8. Finally, the simulation results and conclusionare given in Sections 9 and 10.presented in [13], this model was claimed not suitable for actualage estimation.The active appearance model (AAM) [7], originally proposed forface detection, was first exploited for age estimation by Lanitis et al.[26] in 2002. AAM can jointly extract the shape and texturevariations from human faces – the variations that indeed showsome clues of the facial aging process – so it soon became the mostwidely-used feature type in age estimation [5,6,17–19,27,29,32,40,42–44,48,51]. Later according to AAM, Geng et al. proposedthe aging pattern subspace (AGES) [14–16], which further considersthe identity information and the ordinal relationship of ages duringfeature extraction.To enhance the extraction of local facial information, Yan et al.[46] proposed the spatially flexible patch (SFP), which encodesthe age label, the local appearance, and the corresponding spatialcoordinate into a SFP vector; a unified Gaussian mixture model(GMM) is then trained to model the variations of all SFP vectors.The SFP method is claimed robust against slight pose change andface misalignment [13], whereas a unified probability model isunlikely to represent all possible local appearances. This problemwas later considered and alleviated in [47,50].Manifold learning has also been applied for feature extraction onhuman faces. In [10,12], Fu et al. directly performed manifoldlearning algorithms [11,21] on the resized face images to extractcompact facial features. This approach shows plausible performanceon the private UIUC-IFP database [10], but is not suitable for the FG-NET database [53], which contains significant pose and expressionvariations. In our approach, we first extract the AAM features, whichcould effectively record the pose and expression variations on faces;manifold learning algorithms are then applied for dimensionalityreduction and learning the connections between facial features andage labels, not for direct feature extraction from face images.Besides the feature extraction methods mentioned above,well-known texture descriptors like the Gabor wavelets [20]and the local binary patterns (LBP) [9] have been utilized forage estimation as well.2. Previous and related work2.1. Previous work on facial age estimationThere were several age estimation algorithms published in thelast decade, and the goals of these algorithms can be separated intotwo categories: One is to estimate the actual age (e.g., 30-year old);the other is to classify a person into an age range, such as baby,teenager, middle-ager, or elder. In this paper, we aim at the firstgoal, which is the main focus in previous work.An age estimation algorithm can be simply divided into twosteps: feature extraction and age determination. In the first step,facial features related to human ages or facial appearance changeare extracted from human faces for compact representation; inthe second step, an age determination function is built to estimatethe age based on the extracted features. In the following, we givea broad review of the previous work in each step respectively.2.1.1. Feature extractionAmong all kinds of facial features, the first one utilized in ageestimation is probably the anthropometric model, which is basedon the domain knowledge of facial aging processes, such as theoccurrence of wrinkles and the change of face shapes. In [25],Kwon et al. exploited the snake algorithm [23] for wrinkledetection, and combined this information with some measuresof facial geometry for age range classification. The anthropometricmodel was also applied in [22,31] for the same task and resultedin acceptable accuracy. However, according to the experiments2.1.2. Age determinationAs described in Section 1, age estimation can be seen as a multiclassclassification problem, so traditional classification algorithmssuch as the nearest centroid classifier [26,27,50] and the supportvector machine (SVM) [18–20] can be directly applied for agedetermination. These algorithms, nevertheless, do not take theordinal relationship and correlations among age labels intoaccount. Namely, they assign in their objective functions a uniformpenalty to any misclassification case; however, in age estimation,wrongly predicting a 30-year-old person as 10-year-old is muchmore serious than predicting him/her as 25-year-old.Regressors, on the other hand, indigenously consider the ordinalrelationship and could probably result in better estimation results.Regression algorithms such as the least square regression [10,12],the kernel regression [42,47], the Gaussian process regression(GPR) [32,51], and the support vector regression (SVR) [18–20]all have been applied and examined in previous work. In [43],Yan et al. proposed an auto-structured regression according to theclaim: The age label given to each image should be a nonnegativeinterval, rather than a fixed value. This regression model was laterimproved in [44] by utilizing the expectation maximization (EM)algorithm for optimization, leading to better age estimation performancethan in [43].Compared to using a single regressor or classifier for agedetermination, combining several regressors and classifiers couldbetter capture the complicated aging process and improve theestimation performance [18]. In[26,27], Lanitis et al. proposed theappearance- and age-specific combinations, which first classify a

Author's personal copy630W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641face into a similar-appearance group or an age group, and thenutilize the age determination function built specifically for thatgroup to predict the age. The age-specific combination was alsoapplied in [18,19,29] by concatenating SVM and SVR.Recently, some new techniques of machine learning have beenbrought into age estimation. In [5,52], the ordinal regressiontechnique is exploited to model the ordinal relationship amongage labels; and in [6], Chang et al. applied the cost-sensitiveclassification technique, where the correlations among age labelscan be flexibly modeled according to different evaluation criteria.2.2. Related techniques of the proposed approach2.2.1. Manifold learningThe proposed approach improves LPP [21], alinearized manifoldlearning algorithm, for dimensionality reduction on the extractedfacial features, as mentioned in Section 1. Manifold learning hasbeen experimented to successfully discover the intrinsic and latentvariables from data samples [28,39]; since the number of thesevariables is generally much smaller than that of the input features,manifold learning could therefore be used for dimensionalityreduction.Algorithms of manifold learning are generally composed of threesteps: The first two steps, neighbor searching and local geometrymodeling, explore certain kinds of nonlinear properties from theinput data samples; the third step, embedding computation, thenembeds these input samples into the output feature space (with thereduced dimensionality) through optimizing the objective functionbuilt on such explored properties. In early manifold learningalgorithms (such as [4,33,36]), the embedding computation stepdirectly generates the output sample for each input sample in agiven training set, but cannot handle the testing (or unseen) data.To solve this problem, linearization – a modification that restrictsthe input–output relationship by a linear projection matrix – isproposed in [21], where LPP is the linearized form of the Laplacianeigenmap [4]. Based on the learned linear matrix, a testing samplecan thus be projected into the output space for dimensionalityreduction. Linearization was later applied in many manifold learningalgorithms [11,45], and in the next subsection, the algorithm ofLPP is presented.2.2.2. Locality preserving projection (LPP)Given a training set X ¼fx ðnÞ AR d g N n ¼ 1with N d-dimensionalsamples and a desired output dimensionality p, LPP aims to learna matrix W LPP AR dp that minimizes the average neighbor distanceamong the projected output samples Z ¼fz ðnÞ AR p g N n ¼ 1 ,where z ðnÞ ¼ W T LPP xðnÞ AR p .The algorithm of LPP comprises the three steps mentionedin Section 2.2.1. In the first step, each sample in X searches itsk 1 nearest neighbors via the Euclidean distance. And in the secondstep, an N N matrix B þ ¼½b þ ij Š 1 r i,j r N is constructed to recordthe feature similarityb þij ¼ expð :x ðiÞ x ðjÞ : 2 =tÞ ð1Þbetween any pair of neighboring samples for local geometrymodeling; if samples x (i) and x (j) are not neighbors, b þ ij ¼ b þji ¼ 0.Finally in the third step, the projection matrix W LPP is reachedthrough minimizing the following objective function E LPP (W):E LPP ðWÞ¼ XNX Ni ¼ 1 j ¼ 1b þ ij :W T ðx ðiÞ x ðjÞ Þ: 2 ¼ XNX Ni ¼ 1 j ¼ 1b þij :z ðiÞ z ðjÞ : 2 ,ð2Þwith the constraint W T XD þ X T W¼I (for preventing trivial solutions),where X is a d N matrix containing each training samplein the corresponding column, and D þ is a diagonal matrix withd þii ¼ P b þij . To sum up, LPP tries to bring the neighbors (searchedjin X) as closer as possible in the output p-dimensional space,according to the neighbor similarity B þ . Furthermore, the learnedmatrix W LPP can be used directly for dimensionality reduction onthe testing data.When the class label y (n) of each training sample x (n) is providedin the training phase, the algorithm of LPP could be adjusted into thesupervised version by searching neighbors with only the same classlabel in the first step [45]; the other parts of the LPP algorithmremained unchanged. Since now the neighbors are defined not onlyby feature similarity, but also by the same-label constraint, theconnection between features and labels can be explicitly linked,hence leading to more representative features for classification afterdimensionality reduction. The implementation details of LPP couldbe traced from the proposed algorithm in Table 3.3. The proposed age estimation approach3.1. The problem setting of age estimationIn this section, we give an overview about the proposed ageestimation approach. The problem setting used in this paper isdefined as follows: Given a training set fi ðnÞ g N n ¼ 1 with N faceimages, and its corresponding label set Y ¼fy ðnÞ ALg N n ¼ 1 withL ¼fl 1 ,...,l c g, building an age estimation system can be modeledas a supervised machine learning task, where the symbol c indicatesthe total number of age labels concerned.3.2. The overview of the proposed approachThe proposed approach basically adopts the traditional twostepframework of age estimation algorithms (feature extractionþagedetermination), but further include two new steps –distance metric adjustment and dimensionality reduction –asillustrated in Fig. 1. There are two main purposes for performingdimensionality reduction on the extracted features before feedingthem into the age determination step: One is to alleviate the overfittingproblem in training the age determination function; theother is to better explore the connection between facial featuresand age labels—via the supervised version of dimensionalityreduction.Suggested by the previous work, the active appearance model(AAM) [7], which jointly considers the texture and shape informationfrom human faces, is utilized for feature extraction in ourapproach and outputs a d-dimensional feature vector xAR d for aninput image i. Then, a distance metric adjustment step is appliedto transfer the AAM feature space into a suitable space that canenhance the performance of the following dimensionality reductionstep. The resulting feature vectors after distance metricadjustment and dimensionality reduction are denoted as x adjustARd and zAR p respectively; x adjust has the same dimensionality asthe extracted AAM features and z has a lower dimensionality p.Finally, an age determination function is built to estimate theage label ^y (for the input image i) based on the p-dimensionalvector zAR p .In our approach, we improve the relevant component analysis(RCA) [3] and LPP [21] for distance metric adjustment anddimensionality reduction, respectively; both of them are machinelearning algorithms, and each requires a training phase to generatea projection matrix that can be directly applied on thetesting data. To our best knowledge, this is the first time tocombine supervised distance metric learning and dimensionality

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 631Fig. 1. The flowchart of our approach: The proposed algorithms will be introduced in later sections.reduction algorithms for facial age estimation; the motivationwill be described in Section 5.3.3. Main concerns and contributionsAside from the combinations of distance metric learning anddimensionality reduction, we also emphasize the following concernsand propose the corresponding treatments: The label-sensitive concept is proposed (in Section 4) to takethe ordinal relationship among age labels into account indistance metric adjustment and dimensionality reduction. As mentioned in [18], the facial aging process is rathercomplicated. To overcome this problem and also fit thecriterion used for evaluating the performance of age estimation,an age-oriented local regression algorithm, called KNN-SVR,is proposed. In age databases, the number of images of each age label canbe highly different; if these databases are directly used fortraining, the resulting age estimation system will be biased bythe labels with relatively more samples, leading to the imbalancedlearning problem. To alleviate this problem, severalimbalance treatments are considered in our approach. Through incorporating the label-sensitive concept and theimbalance treatments, new algorithms for the 2nd and 3rdsteps in our approach are presented and marked in Fig. 1.4. The label-sensitive conceptBefore describing in more detail about the proposed approach,we first introduce the label-sensitive concept. In traditionalmulti-class classification, a class is treated independently of theother classes, and a uniform penalty is given when a sample ismisclassified into any other class. This condition also occurs intraining the conventional supervised dimensionality reductionand distance metric learning algorithms, where a uniform penaltyis given to each different-label sample pair and only samples withexactly the same label are to be pulled closer. However, in thetask of age estimation, there intrinsically exists the ordinalrelationship and correlations among age labels; therefore, thepenalties should be varied according to different misclassificationcases and different degrees of label dissimilarity.In the existing literature, there are indeed some algorithmsthat can handle the ordinal relationship in the age determinationstep, such as the regression algorithms (as described in Section2.1.2) and the modified forms of multi-class classification[2,5,6,8,37]. For the distance metric learning and dimensionalityreduction steps, unfortunately, few previous algorithms (e.g. [30])have taken the ordinal relationship into account. To achieve this,the label-sensitive concept is proposed in paper, and in Section 9.7,the differences between our algorithms and the one in [30] arecompared and discussed.In the training phases of supervised distance metric learningand dimensionality reduction, several statistical quantities (e.g.,the scatter matrix) are required to compute for every single class(age label), as will be mentioned in the next two sections. Insteadof treating each class independently with only the same-labelsamples, the proposed label-sensitive concept claims that sampleswith similar class labels (defined from the ordinal relationship)should also be considered; the weight of each sample in computingthe quantities of a specific class is assigned based on thelabel similarity. For example, when computing the scatter matrixof age 30, samples with ages around 30 are also regarded. In thefollowing two sections, we show how to embed the label-sensitiveconcept into distance metric learning and dimensionalityreduction.5. Distance metric adjustment5.1. The drawbacks of AAM features and manifold learningalgorithmsThe shape and texture variations extracted from human facesby AAM may not directly reflect the corresponding age labels:These variations may result from different personalities, poses,genders, races, expressions, and living environments, not justfrom ages. Besides, the dimensionality of AAM features is usuallytoo high (simply over a hundred) to train a robust age classifier orregressor. To overcome these problems, we apply LPP [21] toexplore the connections between facial features and age labels,and drastically reduce the dimensionality of features. LPP, asintroduced in Section 2.2.2, is a manifold learning algorithm andcomposed of three steps in the training phase. In the first step,LPP and most manifold learning algorithms [21,45] assumes theinput space to be locally Euclidean, and utilizes the Euclideanmetric for neighbor searching; this distance metric, however, maynot be suitable in practical cases.5.2. The reason for performing distance metric adjustmentIn age estimation, the desired neighbors of a given sample aresamples with similar facial appearance change caused by humanages. To judge the availability of the Euclidean metric for

Author's personal copy632W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641Table 1The algorithm of RCA (relevant component analysis) [3].Presetting Training set: X ¼fx ðnÞ AR d g N n ¼ 1 , Y ¼fyðnÞ ALg N n ¼ 1 Define X i as the feature set containing all feature samples with age label l i . Thenumber of samples in X i is denoted as N i . The goal of RCA is to find the projection matrix WRCA AR dd ; thenx adjust ¼ W T RCA xARd .Algorithm For each age label li , compute the mean vector l i ¼ 1 PN ix ðnÞ and the intraclassscatter matrix S i ¼ 1 Px ðnÞ A X iN iðx ðnÞ l i Þðx ðnÞ l i Þ T .x ðnÞ A X i P Compute the total intra-class scatter matrixS ¼ 1cN i S i .Ni ¼ 1 Perform the eigendecomposition: S ¼ VLV T ; then W RCA ¼ VL ð 1=2Þ .Table 2The proposed lsRCA algorithm (without the imbalance treatment). Define the sample-to-class weight (from sample n to class i;s and e aretunable):(e ðnÞ ¼ expð ðyðnÞ l i Þ 2 =sÞ ify ðnÞ l ii0 otherwise rewhere e is the label-sensitive threshold to define the range of similar labels. Modify the computation of l i ,S i , and S as:l i ¼ 1 o iX Nn ¼ 1,e ðnÞ x ðnÞ , S i i ¼ 1 X Neo ðnÞ ðx ðnÞ li i Þðx ðnÞ l i Þ T , and S ¼ Xco i S ii n ¼ 1i ¼ 1where o i ¼ PN e ðnÞ is the sum of sample-to-class weights from all the samplesin ¼ 1to the label i. Follow the last step of RCA to compute WRCA AR dd .X ci ¼ 1o i ,searching neighbors in the AAM feature space, we simply checkthe standard deviation (STD) of each feature: AAM utilizes theprincipal component analysis (PCA) [38] for features generation,and each STD value (equivalent to the square root of the eigenvaluein PCA) represents how sharply the corresponding feature variesacross different face images. Besides, the usage of PCA also makesthe AAM features uncorrelated, suggesting that each feature couldbe independently considered. Namely, the higher the STD is, thestronger influence the feature has in computing the Euclideandistance.The STDs of the AAM features extracted from the FG-NETdatabase [53] are shown in Fig. 2; as presented, there exists astrong variation among the STD values. However, from both ourobservations and the experiments in [7,26,27], most of the AAMfeatures with higher STDs are not relevant to human ages, but toposes and expressions, indicating the problem of applying theEuclidean metric for neighbor searching. That is, the searchedneighbors are mainly with similar poses or expressions, but notwith similar change of appearances caused by human ages. Thisproblem would degrade not only the performance of LPP, but alsothe overall accuracy of age estimation.To deal with the above issue, a distance metric adjustmentstep is introduced in our work (just before LPP) for re-weighingthe strengths of features.5.3. The proposed usage of distance metric learning algorithmsIn the existing literature of pattern recognition, normalizingor scaling the STDs of features is the most widely-used methodto achieve distance metric adjustment. Nevertheless, withoutregarding the label information, this method again could notproduce a suitable space for searching the desired neighbors viathe Euclidean distance. In our approach, we adopt the relevantcomponent analysis (RCA) [3] for distance metric adjustmentbecause of its efficiency and supervised nature: RCA is a superviseddistance metric learning algorithm.Given a training set X ¼fx ðnÞ AR d g N n ¼ 1 with the label set Y ¼fy ðnÞ ALg N n ¼ 1 , RCA first computes the intra-class scatter matrix S i ofeach class l i , and then performs the weighted sum on thesematrices to produce the total intra-class scatter matrix S. Finally,the d d matrix W that leads to W T SW¼I is selected as theadjustment matrix W RCA ; the sample after distance metric adjustmentis denoted as x adjust , where x adjust ¼ W T RCA xARd .The objective function of RCA, W T SW¼I, is a whitening operation;in other words, RCA aims to whiten the intra-class scattermatrices of all classes simultaneously. The resulting feature spaceafter RCA, though with no theoretical guarantee as the desiredspace for neighbor searching, does lead to a (nearly) globallyEuclidean space inside each age class—through the whiteningoperation in RCA with the label information. Therefore, when fedwith this feature space, the neighbor searching step of supervisedLPP (with the same-label constraint) now explores the k 1 nearestneighbors in a globally Euclidean space, which better matches theusage of the Euclidean metric for neighbor searching, henceimproving the overall performance (as shown in Section 9). Thedetailed algorithm of RCA is presented in Table 1; to be noticed,x adjust has the same dimensionality as the input AAM features.5.4. The proposed label-sensitive relevant component analysisTo further exploit the ordinal relationship among age labels,the label-sensitive concept is applied in RCA: When computing theintra-class scatter matrix S i of class l i , samples with labels similarto l i are also involved. The weight of each sample to l i is modeledthrough a radial basis function, and the range of similar labels isdefined by a label-sensitive threshold e: Two ages with the gapno larger than e are viewed as similar ages. The resultinglabel-sensitive form of RCA, called lsRCA, is summarized inTable 2, where the formulas of the mean vector l i, the intraclassscatter S i , and the total intra-class scatter S defined inTable 1 are modified.Eventually, to balance the influence of each label in lsRCA – theage labels with more samples are likely to dominate the trainingof lsRCA – an imbalance-compensated version called C-lsRCA isproposed, where the formula of S in Table 2 is replaced byS ¼ 1 cX ci ¼ 1S i :This modification equalizes the influence of each intra-classscatter matrix on the total intra-class scatter matrix.6. Dimensionality reduction6.1. The proposed label-sensitive locality preserving projectionRCA or its modified versions result in a suitable feature setfx ðnÞadjust ¼ WT RCA xðnÞ AR d g N n ¼ 1 , where the Euclidean metric now canbe utilized in the neighbor searching step of LPP. As mentioned inSection 2.2.2, LPP is originally an unsupervised technique, yetcan be extended into the supervised version by searchingð3Þ

Author's personal copyTable 3The proposed lsLPP algorithm (label-sensitive LPP).Presetting Training set: X ¼fxðnÞadjust ARd g N n ¼ 1 , Y ¼fyðnÞ ALg N n ¼ 1 (X is represented as a d Nmatrix) Define the similar-label set N þ ðiÞ for each sample x ðiÞadjust :N þ ðiÞ¼fx ðjÞy ðjÞ y ðiÞ adjustre,jaig,where e is the label-sensitive threshold to define the range of similar labels. Create an N N sample similarity matrix B þ ¼½b þ ij ¼ 0Š 1 r i,j r N . The goal of lsLPP is to find the projection matrix WLPP AR dp ; thenz ¼ W T LPP x adjust AR p .Algorithm For each sample xðiÞadjust , find the k 1-nearestsamples in N þ ðiÞ, and denote thesesamples as KNN þ ðiÞ. The parameter k 1 defines the number of neighboringsamples. For each sample pair fxðiÞadjust ,xðjÞ adjust g,ifxðjÞ adjust AKNNþ ðiÞor x ðiÞadjust AKNNþ ðjÞ, set:b þ ij ¼ expð :x ðiÞadjustx ðjÞadjust :2 =tÞexpð ðy ðiÞ y ðjÞ Þ 2 =sÞ: Compute L þ ¼ D þ B , where D þ is a diagonal matrix with d þ ii ¼ P b þij .j Solve the generalized eigendecomposition problem: XL þ X T v ðiÞ ¼ l ðiÞ XD þ X T v ðiÞ - XL þ X T V ¼ XD þ X T VL,Where L is arranged in the descending order. WLPP ¼ v ðN p þ 1Þ ,v ðN p þ 2Þ ,...,v ðNÞ ¼ VOpðN pÞ 9I pp T :W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 633Fig. 3. The schematic illustration of the proposed lsLPP compared to the conventionalsupervised extension of LPP [45]: Here we only take x ðiÞ ’ s neighborhoodadjustas an example. In both (a) and (b), the left ones are the distributions ofneighboring samples before dimensionality reduction; the right ones depict thedistributions after applying supervised LPP and lsLPP. As shown, lsLPP can not onlypull samples with similar labels together, but also preserve the order of labelsimilarity with respect to x ðiÞadjust .6.2. The proposed label-sensitive marginal fisher analysisFig. 2. The STD of each AAM feature (127 features are extracted from the FG-NETdatabase).neighbors with the same class label. To further take the ordinalrelationship among age labels into consideration, the labelsensitiveconcept is applied in LPP to achieve an improved versionnamed label sensitive LPP (lsLPP), where the same-label constraintis replaced by a similar-label one (according to the label-sensitivethreshold e). That is, a training sample now searches the k 1nearest neighbors with the similar labels.In addition, lsLPP defines a new neighbor weighting function:b þij ¼ expð :x ðiÞadjustx ðjÞadjust :2 =tÞexpð ðy ðiÞ y ðjÞ Þ 2 =sÞ, ð4Þwhich regards both the feature similarity (former part) and thelabel similarity (later part) between neighbors: The degree of labelsimilarity is again modeled through a radial basis function.In Fig. 3, we show the schematic illustration of lsLPP comparedto the conventional supervised extension of LPP; the detailedalgorithm of lsLPP is described in Table 3, where e, s, and t aretunable parameters for defining the feature and label similarity.The learned matrix W LPP by lsLPP then can be applied to theunseen or testing data.To illustrate the availability of the label-sensitive concept ondimensionality reduction algorithms other than LPP, we alsoapply this concept to another popular algorithm called themarginal fisher analysis (MFA) [45]. MFA, contrary to LPP, isintrinsically a supervised manifold learning algorithm. In theneighbor searching step, MFA searches the k 1 same-label and k 2different-label nearest neighbors for each sample via the Euclideanmetric. Then in the local geometry modeling step, MFA builds twoN N matrices, an intrinsic matrix B þ and a penalty matrix B ,torecord the geometrical information for each type of neighbors,respectively; the entries in B þ and B are defined by (1). Finally inthe embedding computation step, MFA seeks the matrixW MFA AR dp that minimizes E þ MFA ðWÞ and maximizes E MFAðWÞ, asdefined below, simultaneously:E þ XNMFAðWÞ¼ E MFA ðWÞ¼ XNX Ni ¼ 1 j ¼ 1X Ni ¼ 1 j ¼ 1b þij :W T x ðiÞ x: ðjÞ 2 ¼ XNb ij :W T x ðiÞ x: ðjÞ 2 ¼ XNX Ni ¼ 1 j ¼ 1X Ni ¼ 1 j ¼ 1b þij :z ðiÞ z ðjÞ : 2 ,b ij :z ðiÞ z ðjÞ : 2 :In other words, MFA tries to pull the same-label neighborscloser while push the different-label neighbors away for eachsample. The last step of MFA, just like LPP, can be formulated as ageneralized eigendecomposition problem.ð5Þð6Þ

Author's personal copy634W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641Following the modifications proposed in Section 6.1 for LPP,we replace the constraints of the same- and different-labelneighbors in MFA by the similar- and dissimilar-label neighbors(again based on the label-sensitive threshold e). Moreover, thedegrees of label similarity and label dissimilarity, which are usedin computing the modified neighbor weights b þij and b ij as in (4),are modeled by a radial basis and a sigmoid function, respectively(shown in Fig. 4). The improved algorithm is called the labelsensitiveMFA (lsMFA) and summarized in Table 4, wheree, s, g, Z, and t are tunable parameters for defining the featureand label similarity.6.3. The proposed imbalance treatments for dimensionalityreductionTo alleviate the imbalance problems when training lsLPP andlsMFA, three imbalance treatments are proposed: sample weightmodification (SWM), neighbor size modification (NSM), andneighbor range modification (NRM).The sample weight modification (SWM) is to balance theinfluence of each label in the optimization problems of lsLPP andlsMFA. Namely, when computing b þ ij ,b ij in Tables 3 and 4, the labelinfluence of y ðiÞ and y ðjÞ should also be considered. For this purpose,an SWM weighting function f SWM ðyÞ is defined, which giveshigher weights to age labels with fewer samples. The computationof b þij ,b ij are then modified asfb þ ij ,b ij g : ¼fb þ ij ,b ij gf SWMðy ðiÞ Þf SWM ðy ðjÞ Þ:ð7ÞThe neighbor size modification (NSM) is another way tobalance the influence of each label in the optimization problemsof lsLPP and lsMFA. Instead of directly changing the values ofb þij and b ij , NSM changes the number of neighbors searched foreach sample based on the corresponding label: The more neighborsare searched, the higher influence that sample has. In ourimplementation, an NSM function is defined, which also giveshigher weights to labels with fewer samples. The computation ofk 1 and k 2 in Tables 3 and 4 are then modified asfk 1 ðiÞ,k 2 ðiÞg : ¼fk 1 ,k 2 gf NSM ðy ðiÞ Þ:ð8ÞIn addition to the influence of each label, the quality of thesearched neighbors, which define the local geometry to bepreserved, is also an important issue of lsLPP and lsMFA. That is,the searched neighbors should be close (according to the featuresimilarity) to the target sample to represent the local geometryfaithfully. For a sample x ðiÞadjust , if the similar-label set N þ (i) definedin Tables 3 and 4 contains sufficient samples, it is more likely tofind the k 1 neighbors close to x ðiÞadjust ; on contrary, if N þ (i) containsinsufficient samples, the k 1 searched neighbors may not be closeto x ðiÞ , resulting in poor local geometry. To compensate thisadjustproblem, we propose the neighbor range modification (NRM),which sets the label-sensitive threshold e and the parameter s ofthe radial basis function in Tables 3 and 4 individually for eachsample based on the corresponding label:feðiÞ,sðiÞg : ¼fe,sgf NRM ðy ðiÞ Þ:ð9ÞThe weighting function f NRM ðyÞ also gives higher weights to labelswith fewer samples, making the size of N þ (i) much stable. Fig. 5illustrates the concept of NRM.In our implementation, we apply NSMþNRM to lsLPP andSWMþNRM to lsMFA according to the experimental results. Theresulting imbalance-compensated versions are then called C-lsLPPand C-lsMFA respectively. To be noticed, the matrices B þ , B , L þ ,and L defined in Tables 3 and 4 may become asymmetric afterthese three treatments, disobeying the standard form of LPP andFig. 4. Illustrations of the degrees of label similarity and label dissimilaritydefined in Table 4 for lsMFA. (a) The degree of label similarity is modeled by aradial basis function (e.g. with s ¼ 12). (b) The degree of label dissimilarity ismodeled by a sigmoid function (e.g. with e ¼ 5, g ¼ 2, and Z ¼ 2).MFA. To deal with this problem, we replace the entries of B þ andB by the entries of ðB þ þB þ T Þ=2andðB þB T Þ=2 before computingD þ , D and L þ , L .7. Age determinationAfter dimensionality reduction, the resulting p-dimensionalfeature set Z ¼fz ðnÞ AR p g N n ¼ 1 (zðnÞ ¼ W T LPP xðnÞ or W T adjust MFA xðnÞ adjust ; p isusually much smaller than d) now can be used for training the agedetermination function. Instead of applying the global regressiontechniques for age determination, we propose to utilize localregression mainly for two reasons: Local regression can generatecomparably sophisticated mapping functions; hence is morelikely to capture the complicated facial aging process. Besides,from the perspective of manifold learning [21,45], the outputspace of LPP, MFA, or their modified versions is locally Euclideanrather than globally Euclidean, meaning that only the localstatistical properties are reliable to exploit.Inspired by the local classification algorithm in [49] and theL 1 loss of support vector regression (SVR) emphasized in [18],an age-oriented local regression algorithm named KNN-SVR isproposed: Given a testing sample z, KNN-SVR first searches thek-nearest neighbors of z in the training set Z ¼fz ðnÞ AR p g N n ¼ 1 , andtrains an RBF-kernel SVR regressor based on these neighboringsamples; the learned regressor is then applied to the testingsample z for age estimation. The algorithm of KNN-SVR issummarized in Table 5. Notice that, the neighbor searching stepin KNN-SVR is with no prior distance metric adjustment step.8. Summarization of the proposed approachIn this section, a short summary of the proposed approach ispresented, where the training phase for building the age estimationsystem and the testing phase for predicting the age of atesting image are described separately.8.1. The training phaseGiven a training set fi ðnÞ g N n ¼ 1 with N face images, our approachfirst applies AAM to extract the facial features from each image,resulting in a d-dimensional feature set X ¼fx ðnÞ AR d g N n ¼ 1 . Then,through jointly considering X with its corresponding label setY ¼fy ðnÞ ALg N n ¼ 1 ,ad d matrix W RCA is learned by the proposedC-lsRCA algorithm for distance metric adjustment; the adjustedfeature set is denoted as X ¼fx ðnÞadjust ¼ WT RCA xðnÞ AR d g N n ¼ 1 . With thisadjusted feature set X and the label set Y, the proposed C-lsLPP

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 635Table 4The proposed lsMFA algorithm (label-sensitive MFA).Presetting Training set: X ¼fxðnÞadjust ARd g N n ¼ 1 , Y ¼fyðnÞ ALg N n ¼ 1 (X is represented as a d N matrix) Define the similar-label set N þ ðiÞ and dissimilar-label set N ðiÞ for each sample x ðiÞadjust :N þ ðiÞ¼fx ðjÞy ðjÞ y ðiÞ adjust re,jaig and N ðiÞ¼fx ðjÞy ðjÞ y ðiÞ adjust4e,jaig,where e is the label-sensitive threshold to define the range of similar labels. Create an N N intrinsic matrix B þ ¼½b þij ¼ 0Š 1 r i,j r N and a penalty one B ¼½b ij ¼ 0Š 1 r i,j r N . The goal of lsMFA is to find the projection matrixWMFA AR dp , then z ¼ W T MFA x adjust AR p .Algorithm For each sample xðiÞadjust , find the k 1- and k 2 -nearestsamples in N þ ðiÞ and N ðiÞ, and denote these samples as KNN þ ðiÞ and KNN ðiÞ. For each sample pair fxðiÞadjust ,xðjÞ adjust g,ifxðjÞ adjust AKNNþ ðiÞor x ðiÞadjust AKNNþ ðjÞ, set:b þ ij ¼ expð :x ðiÞadjustx ðjÞadjust :2 =tÞexpð ðy ðiÞ y ðjÞ Þ 2 =sÞ: For each sample pair fxðiÞadjust ,xðjÞ adjust g,ifxðjÞ AKNN ðiÞor xðiÞ AKNN ðjÞ, set:adjust adjustb ij ¼ expð:x ðiÞadjust x ðjÞadjust :2 =tÞ 1þexp g y ðiÞ y ðjÞ1: ðeþZÞ Compute L þ ¼ D þ B and L ¼ D B , where D þ and D are both diagonal matrices with d þ ii ¼ P j Solve the generalized eigendecomposition problem: XL þ X T v ðiÞ ¼ l ðiÞ XL X T v ðiÞ - XL þ X T V ¼ XL X T VL,where L is arranged in the descending order. T:WMFA ¼ v ðN p þ 1Þ ,v ðN p þ 2Þ ,...,v ðNÞ ¼ VOpðN pÞ 9I ppb þijand d ii ¼ P b ij .jFig. 5. The schematic illustration of the neighbor range modification (NRM): Before NRM, the 6 nearest samples with similar labels are sparsely distributed; after NRM, theyellow samples are considered similar to the red sample, and the 6 nearest samples can be searched in a much smaller area around the target red sample, resulting in amuch faithful local geometry. (For interpretation of the reference to color in this figure, the reader is referred to the web version of this article.)algorithm (or C-lsMFA) is performed to achieve a d pmatrix W LPP (or W MFA ) for dimensionality reduction, leadingto the output feature set Z ¼fz ðnÞ ¼ W T LPP xðnÞ adjust ARp g N n ¼ 1 (orZ ¼fz ðnÞ ¼ W T MFA xðnÞ AR p g N adjust n ¼ 1 ) with dimensionality p.8.2. The testing phaseNow given a testing image i, the AAM features x are againextracted at the first step. Then based on the learned matricesW RCA and W LPP (or W MFA ), both the distance metric adjustmentand dimensionality reduction steps can be simply accomplishedby linear projection, resulting in the p-dimensional vector z:z ¼ W T LPP ðW T RCA xÞARp or z ¼ W T MFA ðWT RCA xÞARp . Finally, accordingto the testing sample z and the feature set Z, which is acquired inthe training phase, the proposed KNN-SVR is performed toestimate the actual age for the input face image i. Notice that,there is also an neighbor searching step in KNN-SVR, just like LPPand MFA; however, this neighbor searching step is performed onthe p-dimensional data in the testing phase.9. Simulation results9.1. DatabaseThe age estimation experiments are performed on the mostwidely-used FG-NET aging database [53], which contains 1002

Author's personal copy636W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641face images from 82 individuals and provides 68 landmarks oneach face. These images are ranging from age 0 to age 69, butmore than 700 of them are under age 20, making the FG-NETaging database highly imbalanced. Fig. 6 shows some exampleimages of the FG-NET database, and the number of images in eachage range is listed in Table 10.9.2. Experimental settings and evaluation criteriaSuggested by the experimental setups in previous work, theleave-one-person-out (LOPO) testing strategy is adopted, wherethe age estimation algorithm is repeatedly trained on images of81 people and tested on images of the remaining person. Afterrepeating 82 times, each image will be the testing sample for onceand receive an estimated age.To evaluate the performance, two popular measures, the meanabsolute error (MAE) and the cumulative score (CS), proposed in[14] are computed in our experiments. MAE stands for theaverage L 1 loss during testing, which fits the loss function of SVR,and that is why we adopt SVR in the proposed local regressionalgorithm; CS calculates the percentage of images with L 1 losseslower than a given threshold. The formulations of MAE and CS aredefined in Table 6.9.3. Implementation details of the proposed approachThe proposed age estimation approach is composed of foursteps as illustrated in Fig. 1. In the feature extraction step, the 68landmark points provided on each FG-NET face image are used forAAM training (implemented by the AAM-API tool [34]). Under thememory limitation during our implementation, all face imagesare first down-sampled, and totally 127 features are extracted torepresent each image.In the subsequent three steps, besides the proposed algorithms,other existing algorithms (e.g., SVR for age determination)could also be applied; therefore, we perform different algorithmcombinations to demonstrate the improvements achieved by theusage of RCA and the proposed C-lsRCA, C-lsLPP, C-lsMFA, andKNN-SVR algorithms. The tunable parameters of the proposedalgorithms, such as the label-sensitive threshold e and the numberof neighbors, are selected via cross validation. The three functionsfor imbalance treatments proposed in Section 6.3 are simplydefined in Table 7 to balance the influence of each age label.The LOPO MAE results of different algorithm combinations inour four-step framework are listed in Table 8. As presented, theusage of RCA outperforms the widely-used standard deviationnormalization (STDN) for distance metric adjustment; theimproved version C-lsRCA further reduces the MAEs in most ofthe combinations. The proposed lsLPP and lsMFA with the labelsensitiveconcept obviously surpass their original versions, and theimbalance treatments (NSMþNRM for lsLPP and SWMþNRM forlsMFA) improve the performance in several combinations. Finally,the proposed KNN-SVR definitely outperforms SVR in all thecases, demonstrating the usage of local regression after performingmanifold learning algorithms. To sum up, the combinationswith our propositions and modifications – including distancemetric adjustment, local regression, the imbalance treatments,and the label-sensitive concept – achieve the lowest age estimationerrors against other algorithm combinations.Furthermore, the optimal output dimensionality p ofC-lsLPP, C-lsMFA and the parameter k of KNN-SVR are reachedaround 10 and 15 (as shown in Fig. 7), illustrating the effectivenessof dimensionality reduction and local statistics in ageestimation: Fig. 7 shows the MAEs achieved by the two bestcombinations in Table 8 – C-lsRCAþC-lsLPPþKNN-SVR andC-lsRCAþC-lsMFAþKNN-SVR – with different choices of dimensionalityp and the parameter k of KNN-SVR.9.4. Performance comparisons with existing algorithmsWe further compare the two best combinations, C-lsRCAþC-lsLPPþKNN-SVR and C-lsRCAþC-lsMFAþKNN-SVR, with otherexisting age estimation algorithms; the LOPO MAE results of theexisting and the proposed algorithms are listed in Table 9, alongwith brief algorithm descriptions. As presented, both the twoproposed combinations reach the lowest MAEs, even better thanTable 6Definitions of MAE and CS (^y: the predicted age).Testing setMAECSD test : fi ðnÞ g Ntn ¼ 1 ,Y ¼fyðnÞ ALg Ntn ¼ 1MAE ¼ 1 P Nt^y ðnÞ y ðnÞ Ntn ¼ 1(CSðjÞ¼ 1 P hiNt1 ^y ðnÞ y ðnÞ 1½trueŠ¼ 1rj , where Nt1 f alsen ¼ 1¼ 0Table 5The proposed KNN-SVR algorithm.Presetting Training set: Z ¼fz ðnÞ AR p g N n ¼ 1 , Y ¼fyðnÞ ALg N n ¼ 1Algorithm: (k is tunable) Given an input query z, find its k-nearest Euclidean neighbors fzðiÞKNN ,yðiÞ KNN gk n ¼ 1in Z. Train an RBF-kernel SVR regressor based on fzðiÞKNN ,yðiÞ KNN gk n ¼ 1 to predict the age of z.Fig. 6. Example images of the FG-NET aging database: Each row shows part of the face images of one subject from younger to older.

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 637Table 7Definitions of the three imbalance treatments proposed in Section 6.3.SWM NSM NRM(1, for yr20f SWM ðyÞ¼1þ ðy 20Þ50 , for y42081:0, for yr20>< 1:3, for 20oyr30f NSM ðyÞ¼1:8, for 30oyr40>: 2:5, for y44081:0, for yr20>< 1:2, for 20oyr30f NRM ðyÞ¼1:6, for 30oyr40>: 2:0, for y440Table 8Comparisons of the LOPO MAEs with different algorithm combinations in our four-step framework. For the distance metric adjustment step, four different algorithms areconsidered: No change, STDN, RCA, and C-lsRCA. For the dimensionality reduction step, six algorithms are compared: LPP, lsLPP, C-lsLPP, MFA, lsMFA, and C-lsMFA. For theage determination step, SVR and KNN-SVR are used and compared.sLPP a lsLPP C-lsLPP MFA lsMFA C-lsMFASVRNo change b 7.35 6.95 7.28 8.07 7.62 7.58STDN c 5.79 5.82 5.85 5.62 5.71 5.84RCA 5.43 5.38 5.44 5.49 5.43 5.48C-lsRCA 5.55 5.32 5.34 5.89 5.46 5.37KNN-SVRNo change 4.84 4.85 4.81 5.06 4.61 4.68STDN 5.10 4.95 4.75 5.07 4.97 5.12RCA 4.67 4.53 4.49 4.95 4.60 4.62C-lsRCA 4.74 4.43 4.38 5.40 4.67 4.44a sLPP: supervised LPP.b No change: no distance metric adjustment.c STDN: STD normalization.Table 9Comparisons of the LOPO MAE results on the FG-NET database.Category Algorithm name MAE Algorithm descriptionAAMþClassifiers/Regressors KNN 8.24 AAMþK nearest neighborsSVM 7.25 AAMþSupport vector machineMLP 6.95 AAMþMulti-layer perceptronKNN Regression 6.44 AAMþKNN RegressionRUN1 [43] 5.78 AAMþRUN1 aGaussian IIS-LLD [17] 5.77 AAMþLearning from label distributionSVR 5.68 AAMþSupport vector regressionRUN2 [44] 5.33 AAMþRUN2RED-SVM [5] 5.24 AAMþRED-SVMMHR [6] 4.87 AAMþMultiple hyperplanes rankerOHR [6] 4.48 AAM þOrdinal hyperplanes rankerAAMþHybrid combination WAS [26] 8.06 AAMþWeighted appearance-specificLARR [18] 5.07 AAMþLARR bPFA [19] 4.97 AAMþProbabilistic fusion approachAGES AGES [14] 6.77 AAMþAging pattern subspaceAGES with LDA [14] 6.22 AAMþAGES with LDA cKAGES [15] 6.18 AAMþKernel AGESMSA [16] 5.36 AAMþMulti-linear subspace analysisManifold learning LEA [48] 7.65 AAMþLocally embedded analysisSSE [48] 5.21 AAMþSSE dDistance metric learning mGPR [32] 5.08 AAMþDML e þGPR fmKNN [42] 4.93 AAMþDMLþKNN RegressionGaussian process regression WGP [51] 4.95 AAMþWarped GPRMTWGP [51] 4.83 AAMþMulti-task warped GPROther features BIF [20] 4.77 BIF g þPCAþSupport vector regressionRPK [47] 4.95 SFP h þPatch kernelþKernel regressionProposed C-IsRCAþC-lsMFA (SWMþNRM) 4.44 AAMþC-lsRCAþC-lsMFAþKNN-SVRC-IsRCAþC-lsLPP (NSMþNRM) 4.38 AAMþC-lsRCAþC-lsLPPþKNN-SVRa RUN: regression with uncertain nonnegative label.b LARR: locally adjusted robust regression.c LDA: linear discriminant analysis.d SSE: synchronized submanifold embedding.e DML: distance metric learning.f GPR: Gaussian process regression.g BIF: biologically inspired features.h SFP: Spatially flexible patch.

Author's personal copy638W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641Table 10Comparisons of the LOPO MAEs at different age ranges on the FG-Net database.Range # images LEA [48] MLP QF KNN Reg. a RUN1 [43] SSE [48]0–9 371 3.89 5.25 5.67 4.76 2.51 2.0610–19 339 4.85 5.24 5.54 3.43 3.76 3.2620–29 144 8.67 5.85 5.92 5.46 6.38 6.0330–39 70 13.02 11.29 10.27 13.51 12.51 9.5340–49 46 19.46 16.48 12.24 22.2 20.09 11.1750–59 15 26.13 28.80 18.60 31.17 28.07 16.0060–69 8 39.00 39.50 28.00 43.47 42.50 26.88Average 1002 7.65 6.95 6.70 6.44 5.78 5.21Range # images mGPR[32] RPK[47] mKNN[42] BIF[20] C-lsMFA C-lsLPP0–9 371 2.99 2.3 2.29 2.99 1.965 1.91110–19 339 4.19 4.86 3.65 3.39 3.7021 3.526420–29 144 5.34 4.02 5.44 4.30 5.5278 5.32630–39 70 9.28 7.32 10.55 8.24 10.06 10.6740–49 46 13.52 15.24 15.81 14.98 10.80 10.1150–59 15 17.79 22.2 25.18 20.49 14.53 15.0760–69 8 22.68 33.15 36.80 31.62 22.25 23.37Average 1002 5.08 4.95 4.93 4.77 4.44 4.38a KNN Reg.: KNN regression.Fig. 7. The LOPO MAEs achieved by C-lsRCAþC-lsLPPþKNN-SVR and C-lsRCAþC-lsMFAþKNN-SVR with different choices of dimensionality p and the parameter k of KNN-SVR. The first row shows the MAEs achieved by varied p, with either a fixed k or the best k under the corresponding p. The second row shows the MAEs achieved by variedd, with either a fixed p or the best p under the corresponding k. From our observation, the best p under each varied k in the combination C-lsRCAþC-lsLPPþKNN-SVR isalways 7.the state-of-art ordinal hyperplanes ranker (OHR) proposed byChang et al. [6]. Also from Table 9, we found that our approach,combining distance metric learning and dimensionality reduction,definitely outperform the work in [32,42,48], which only appliesone of these two techniques, supporting the usage of the fourstepframework in our approach. To further illustrate the performanceimprovement achieved by the proposed imbalancetreatments, the LOPO MAEs at different age ranges are computedand listed in Table 10; both the proposed combinations reachsignificantly lower MAEs at higher ages.Finally in Fig. 8, we show the comparison of the cumulativescore (CS), where totally 11 thresholds are examined. As presented,the proposed approaches achieve the highest CSs at eachthreshold against other algorithms.

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 6399.5. Performance comparisons under other experimental settingsIn addition to the LOPO, there are still other experimentalsettings adopted in previous work. In [9], only the images underage 30 (totally 873 images) are utilized for training and testingwith the LOPO evaluation; in [35,52], the whole FG-NET databaseare randomly divided into 4 folders, and the 4-fold cross validationis performed for the MAE evaluation. This cross validationsetting disregards the identity information, where a single personmay have images in the training (base) and testing (validation)sets simultaneously.To justify the effectiveness of the proposed approach, we alsoperform these two settings for comparison. The MAE results arelisted in Tables 11 and 12, and our approaches still achieve thelowest MAEs in both settings. Furthermore, the cross validationcase is observed to result in lower MAEs than the LOPO case withthe whole FG-NET database, revealing that if images of one personare included in both the training and testing sets, the accuracy ofage estimation could be further improved. Besides, this observationalso tells that under different experimental settings, an ageestimation algorithm would produce different estimation results.9.6. Further discussions and illustrationsThe proposed approach can be efficiently trained and tested. Ittakes only 6 s for training the C-lsRCA and C-lsLPP/ C-lsMFAmatrices by using Matlab on a duo-core PC. Although KNN-SVRis an on-line algorithm, in the testing phase it requires only0.003 s for searching neighbors in the low-dimensional spaceafter dimensionality reduction and training the SVR regressorwith only the k nearest neighbors: The training complexity of thedual-form SVR is generally from O(m) toO(m 2 ), where m is thenumber of training samples. Compared to other age estimationalgorithms, such as the ones requiring nearly a half minute fortraining the SVR regressor or the multi-class SVM classifier, thecomputational complexity of our approach seems acceptable.Even with a larger training set, the efficiency of the neighborsearching step in KNN-SVR could still be kept by incorporating theindexing techniques [1,41].Besides, to demonstrate the effectiveness of the proposeddimensionality reduction algorithms on learning the featurelabelconnection, we depict in Fig. 9 the distributions of the firstTable 11Comparisons of the LOPO MAE results on the FG-NET database (with ages smallerthan 30).Algorithm name MAE Algorithm descriptionLBPþOLPPþMLP [9] 4.28 LBP a þOLPP b þMLP cLBPþOLPPþQF [9] 3.65 LBPþOLPPþQuadratic functionC-lsRCAþlsMFA (SWMþNRM) 3.10 AAMþC-lsRCAþC-lsMFAþKNN-SVRC-lsRCAþlsLPP (NSMþNRM) 3.06 AAMþC-lsRCAþC-lsLPPþKNN-SVRa LBP: local binary pattern.b OLPP: orthogonal LPP.c MLP: multi-layer perceptron.Table 12Comparisons of the 4-fold cross validation MAE results on the FG-NET database.Algorithm name MAE Algorithm descriptionHFM [35] 5.97 Hierarchical face modelþMLPRankBoost [52] 5.67 Haar-like featuresþRankBoostþSVRC-lsRCAþlsMFA (SWMþNRM) 4.23 AAMþC-lsRCAþC-lsMFAþKNN-SVRC-lsRCAþlsLPP (NSMþNRM) 4.11 AAMþC-lsRCAþC-lsLPPþKNN-SVRtwo C-lsLPP and C-lsMFA features from the whole FG-NET database.As shown, obvious feature-label dependences have beenreached by the proposed algorithms. This result also providessupport to the use of local regression: Neighboring samples in thefeature space after performing C-lsRCAþC-lsLPP or C-lsRCAþC-lsMFA are with similar age labels.Furthermore, to justify the improvement (on the neighborsearching step in manifold learning) achieved by distance metricadjustment, we show in Fig. 10 the searched neighbors (via theEuclidean metric) of an arbitrary target sample, in either theoriginal AAM feature space or the feature space adjusted byC-lsRCA, and with or without the similar-label constraint in lsLPP.On the condition with no similar-label constraint, the searchedneighbors in the AAM feature space (though with similar expressionsand poses) are with large age gaps to the target sample; oncontrary, the searched neighbors in the adjusted space are of closeages to the target sample and with similar appearance changecaused by human ages. Even with the similar-label constraint(e.g., label-sensitive threshold e¼3), the searched neighbors in theAAM feature space are still with similar poses and expressions;the influence of these factors can be much suppresses in theadjusted space, demonstrating the effectiveness of C-lsRCA onreaching a suitable space for neighbor searching in LPP (MFA),lsLPP (lsMFA), and C-lsLPP (C-lsMFA).9.7. Algorithm comparisons with the class distance baseddiscriminant analysis [30]Fig. 8. The comparison of LOPO CS (cumulative score) results on the FG-NETdatabase.The class distance based discriminant analysis (CDDA) presentedby Ma et al. [30] also considers the relationship amonglabels in supervised dimensionality reduction; however, there aresome fundamental differences between CDDA and the proposedC-lsLPP and C-lsMFA. First, though the objective functions ofCDDA can be written in the forms of (5) and (6), it is not amanifold algorithm indeed: The neighbor searching step for settingthe entries of B þ and B (in Tables 3 and 4) are not included inCDDA. Second, any sample pair in CDDA could receive weight inB þ without considering the similar-label constraint. That is, asample pair with a large label difference (e.g. over 20 years)could still get weight in B þ . Third, the purpose for building thepenalty matrix B in CDDA is contrary to the one in C-lsMFA:According to the claim that similar labels are easily to be

Author's personal copy640W.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641Fig. 9. The dimensionality reduction results after C-lsRCAþC-lsLPP/C-lsMFA based on the whole FG-NET database: The distributions of the 1st and 2nd dimensions(features) are shown for visualization.Fig. 10. The influence of distance metric adjustment on the neighbor searching step in manifold learning algorithms: (a), (c) show the 5 nearest neighbors searched in theAAM feature space, either with or without the similar-label constraint; (b), (d) show the 5 nearest neighbors searched in the C-lsRCA adjusted space. As presented, theneighbor searching process in the adjusted space is affected less by expressions and poses; therefore, the searched neighbors are with more similar ages and facial agingprocesses to the target sample than the ones in the AAM feature space.misclassified, CDDA gives a larger penalty b ij to the sample pairwith a smaller label difference (the largest penalty is given tosample pairs with y ðiÞ y ðjÞ ¼ 1). Our label-sensitive constraint, onthe other hand, aims to pull the similar-label neighbors closerwhile push the dissimilar-label neighbors away for each sample;therefore, the smaller the label difference is, the smaller thepenalty is assigned. In more detail, CDDA simultaneously givesthe similar-label sample pair a large b þij and b ij , whereas C-lsMFAassign a small (even no) b ij to this pair. Finally, the distancemetric adjustment step and the potential imbalance problem arenot considered in CDDA. Since the age estimation experiments in[30] are not performed on the FG-NET database, we implementCDDA and substitute it for the proposed C-lsLPP (C-lsMFA) in ourfour-step framework; the resulting LOPO MAE is 5.80, demonstratingthe effectiveness of our label-sensitive concept on exploitingthe label information.10. ConclusionIn this paper, a new age estimation approach consideringthe intrinsic factors of human ages is proposed. After featureextraction, RCA is utilized to achieve a suitable space for neighborsearching. Then based on this adjusted space, LPP and MFA aretrained to drastically reduce the feature dimensionality and learnthe connections between features and age labels. To furtherconsider the ordinal relationship of human ages as well as theimbalanced learning problem in RCA, LPP, and MFA, the labelsensitiveconcept and several imbalance treatments are proposed,resulting in new algorithms called C-lsRCA, C-lsLPP, and C-lsMFA.In addition, an age-oriented local regression algorithm calledKNN-SVR is presented to capture the complicated facial agingprocess for age determination. From the simulation resultsperformed on the most widely-used FG-NET aging database, theproposed approach achieves the lowest MAE against the state-ofartalgorithms under several experimental settings.References[1] A. Andoni, P. Indyk, Near-optimal hashing algorithms for approximatenearest neighbor in high dimensions, Communications of the ACM 51 (1)(2008) 117–122.[2] J. Avigad, R. Sommer, A model-theoretic approach to ordinal analysis, TheBulletin of Symbolic Logic 3 (1997) 17–52.[3] A. Bar-Hillel, T. Hertz, N. Shental, D. Weinshall, Learning distance functionsusing equivalence relations, in: Proceedings of the International Conferenceon Machine Learning (ICML), 2003, pp. 11–18.[4] M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques forembedding and clustering, in: Proceedings of the Conference on NeuralInformation Processing Systems (NIPS), 2001, pp. 585–591.

Author's personal copyW.-L. Chao et al. / Pattern Recognition 46 (2013) 628–641 641[5] K.Y. Chang, C.S. Chen, Y.P. Hung, A ranking approach for human ageestimation based on face images, in: Proceedings of the InternationalConference on Pattern Recognition (ICPR), 2010, pp. 3396–3399.[6] K.Y. Chang, C.S. Chen, Y.P. Hung, Ordinal hyperplanes ranker with costsensitivities for age estimation, in: Proceedings of the Computer Vision andPattern Recognition (CVPR), 2001, pp. 585–592.[7] T.F. Cootes, G.J. Edwards, C.J. Taylor, Active appearance models, in: Proceedingsof the European Conference on Computer Vision (ECCV), vol. 2, 1998,pp. 484–498.[8] J. Coste, E. Walter, D. Wasserman, A. Venot, Optimal discriminant analysis forordinal responses, Statistics in Medicine 16 (5) (1997) 561–569.[9] H. Fang, P. Grant, M. Chen, Discriminant feature manifold for facial agingestimation, in: Proceedings of the International Conference on PatternRecognition (ICPR), 2010, pp. 593–596.[10] Y. Fu, Y. Xu, T.S. Huang, Estimating human ages by manifold analysis offace pictures and regression on aging features, in: Proceedings of theIEEE International Conference on Multimedia & Expo (ICME), 2007, pp.1383–1386.[11] Y. Fu, M. Liu, T.S. Huang, Conformal embedding analysis with local graphmodeling on the unit hypersphere, in: Proceedings of the Computer Visionand Pattern Recognition Workshop (CVPRW), 2007.[12] Y. Fu, T.S. Huang, Human age estimation with regression on discriminativeaging manifold, IEEE Transactions on Multimedia (TMM) 10 (4) (2008)578–584.[13] Y. Fu, G. Guo, T.S. Huang, Age synthesis and estimation via faces: a survey,IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)32 (11) (2010) 1955–1976.[14] X. Geng, Z.H. Zhou, K. Smith-Miles, Automatic age estimation based on facialaging patterns, IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI) 29 (12) (2007) 2234–2240.[15] X. Geng, K. Smith-Miles, Z.H. Zhou, Facial age estimation by nonlinear agingpattern subspace, in: Proceedings of the ACM Multimedia (ACM-MM), 2008,pp. 721–724.[16] X. Geng, K. Smith-Miles, Facial age estimation by multilinear subspaceanalysis, in: Proceedings of the IEEE International Conference on Acoustics,Speech, and Signal Processing (ICASSP), 2009, pp. 865–868.[17] X. Geng, K. Smith-Miles, Z.H. Zhou, Facial age estimation by learning fromlabeled distributions, in: Proceedings of the AAAI Conference on ArtificialIntelligence, 2010, pp. 451–456.[18] G. Guo, Y. Fu, C. Dyer, T.S. Huang, Image-based human age estimation bymanifold learning and locally adjusted robust regression, IEEE Transactionson Image Processing (TIP) 17 (7) (2008) 1178–1188.[19] G. Guo, Y. Fu, T.S. Huang, C. Dyer, A probabilistic fusion approach to humanage prediction, in: Proceedings of the Computer Vision and Pattern RecognitionWorkshop (CVPRW), 2008.[20] G. Guo, G. Mu, Y. Fu, T.S. Huang, Human age estimation using bio inspiredfeatures, in: Proceedings of the Computer Vision and Pattern Recognition(CVPR), 2009, pp. 112–119.[21] X. He, P. Niyogi, Locality preserving projections, in: Proceedings ofthe Conference on Neural Information Processing Systems (NIPS), 2003,pp. 153–160.[22] W.B. Horng, C.P. Lee, C.W. Chen, Classification of age groups based onfacial features, Tamkang Journal of Science and Engineering 4 (3) (2001)183–192.[23] M. Kass, A. Witkin, D. Terzopoulos, Snakes: active contour models, InternationalJournal of Computer Vision (IJCV) 1 (4) (1988) 321–331.[24] N. Kumar, A. Berg, P.N. Belhumeur, S. Nayar, Describable visual attributes forface verification and image search, IEEE Transactions on Pattern Analysis andMachine Intelligence (PAMI) 33 (10) (2011) 1962–1977.[25] Y. Kwon, N. Lobo, Age classification from facial images, Computer Vision andImage Understanding 74 (1) (1999) 1–21.[26] A. Lanitis, C. Taylor, T. Cootes, Toward automatic simulation of aging effectson face images, IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI) 24 (4) (2002) 442–455.[27] A. Lanitis, C. Draganova, C. Christodoulou, Comparing different classifiers forautomatic age estimation, IEEE Transactions on Systems, Man, and Cybernetics,Part B 34 (1) (2004) 621–628.[28] J.A. Lee, M. Verleysen, Nonlinear Dimensionality Reduction, first ed., Springer,New York, 2007.[29] K. Luu, K. Ricanek, T.D. Bui, C.Y. Suen, Age estimation using active appearancemodels and support vector machine regression, in: Proceedings of the IEEEInternational Conference on Biometrics, 2009, pp. 314–318.[30] B. Ma, S. Shan, X. Chen, W. Gao, Discriminant analysis for perceptionallycomparable classes, in: Proceedings of the IEEE Conference on AutomaticFace and Gesture Recognition, 2008.[31] A. Montillo, H. Ling, Age regression from faces using random forests, in:Proceedings of the IEEE International Conference on Image Processing (ICIP),2009, pp. 2465–2468.[32] L. Pan, Human age estimation by metric learning for regression problems,Energy Minimization Methods in Computer Vision and Pattern Recognition(EMM-CVPR) (2009) 455–465.[33] S.T. Roweis, L.K. Saul, Nonlinear dimensionality reduction by locally linearembedding, Science 290 (5500) (2000) 2323–2326.[34] M.B. Stegmann, The AAM-API: an open source active appearance modelimplementation, in: Proceedings of the International Conference on Medicalimage computing and computer-assisted intervention—MICCAI, 2003,pp. 951–952.[35] J. Suo, T. Wu, S. Zhu, S. Shan, X. Chen, W. Gao, Design sparse features for ageestimation using hierarchical face model, in: Proceedings of the IEEE Conferenceon Automatic Face and Gesture Recognition, 2008, pp. 1–6.[36] J.B. Tenenbaum, V. de Silva, J.C. Langford, A global geometric framework fornonlinear dimensionality reduction, Science 290 (5500) (2000) 2319–2323.[37] Y. Toren, Ordinal risk-group classification, arXiv: 1012.5487v4, 2011.[38] M.A Turk, A.P. Pentland, Eigenfaces for recognition, Journal of CognitiveNeuroscience 3 (1) (1991) 71–86.[39] L.J.P. van der Maaten, E.O. Postma, H.J. van den Herik, Dimensionalityreduction: a comparative review, Technical Report, Tilburg University, 2009.[40] C. Wang, Y. Su, C. Hsu, C. Lin, H. Liao, Bayesian age estimation on face images,in: Proceedings of the IEEE International Conference on Multimedia & Expo(ICME), 2009, pp. 282–285.[41] Y. Weiss, A. Torralba, R. Fergus, Spectral hashing, in: Proceedings of theConference on Neural Information Processing Systems (NIPS), 2008.[42] B. Xiao, X. Yang, Y. Xu, Learning distance metric for regression by semidefiniteprogramming with application to human age estimation, in: Proceedingsof the ACM Multimedia (ACM-MM), 2009, pp. 451–460.[43] S. Yan, H. Wang, X. Tang, T.S. Huang, Learning auto-structured regressor fromuncertain nonnegative labels, in: Proceedings of the International Conferenceon Computer Vision (ICCV), 2007, pp. 1–8.[44] S. Yan, H. Wang, T.S. Huang, X. Tang, Ranking with uncertain labels, in:Proceedings of the IEEE International Conference on Multimedia & Expo(ICME), 2007, pp. 96–99.[45] S. Yan, D. Xu, B. Zhang, H.J. Zhang, Q. Yang, S. Lin, Graph embedding andextension: a general framework for dimensionality reduction, IEEE Transactionson Pattern Analysis and Machine Intelligence (PAMI) 29 (1) (2007)40–51.[46] S. Yan, M. Liu, T.S. Huang, Extracting age information from local spatiallyflexible patches, in: Proceedings of the IEEE International Conference onAcoustics, Speech, and Signal Processing (ICASSP), 2008, pp. 737–740.[47] S. Yan, X. Zhou, M. Liu, M. Hasegawa-Johnson, T.S. Huang, Regression frompatch-kernel, in: Proceedings of the Computer Vision and Pattern Recognition(CVPR), 2008, pp. 1–8.[48] S. Yan, H. Wang, Y. Fu, J. Yan, X. Tang, T.S. Huang, Synchronized submanifoldembedding for person independent pose estimation and beyond, IEEETransactions on Image Processing (TIP) 18 (1) (2009) 202–210.[49] H. Zhang, A. Berg, M. Maire, J. Malik, SVM-KNN: discriminative nearestneighbor classification for visual category recognition, in: Proceedings of theComputer Vision and Pattern Recognition (CVPR), 2006, pp. 2126–2136.[50] X. Zhuang, X. Zhou, M. Hasegawa-Johnson, T.S. Huang, Face age estimationusing patch-based Hidden Markov Model supervectors, in: Proceedings of theInternational Conference on Pattern Recognition (ICPR), 2008, pp. 1–4.[51] Y. Zhang, D. Yeung, Multi-task warped Gaussian process for personalized ageestimation, in: Proceedings of the Computer Vision and Pattern Recognition(CVPR), 2010, pp. 2622–2629.[52] P. Yang, L. Zhong, D. Metaxas, Ranking model for facial age estimation, in:Proceedings of the International Conference on Pattern Recognition (ICPR),2010, pp. 3404–3407.[53] The FG-NET Aging Database, /http://www.fgnet.rsunit.com/S.Wei-Lun Chao received the M.S. degree from the Graduate Institute of Communication Engineering (GICE), National Taiwan University (NTU), Taiwan, in 2011, and he iscurrently working towards the Ph.D. degree, also in GICE, NTU. His research interests include machine learning, computer vision, multimedia signal processing, and patternrecognition.Jun-Zuo Liu received the B.S. degree from the Department of Communication Engineering, National Central University, Taiwan, in 2010. He is currently working towardsthe M.S. degree in Graduate Institute of Communication Engineering, National Taiwan University, Taiwan. His research interests include machine learning, patternrecognition, and face-related topics.Jian-Jiun Ding was born in 1973 in Taiwan. He received the Ph.D. degree in 2001. He is currently an assistant professor with the Graduate Institute of CommunicationEngineering, National Taiwan University. His current research areas include time–frequency analysis, linear canonical transforms, image compression, image processing,integer transforms, pattern recognition, etc.