Addressing database mismatch in forensic speaker ... - ATVS

More documents

Recommendations

Info

dencies or wire-tapped. This is exactly the same case asfor a recovered wire-tap.¤ Speech recorded using a microphone. Although therecording conditions are much more controlled than inthe hidden microphone case for questioned recordings,variability in the room layout, environmental conditionsand microphone types makes extremely difficult to predictwhich kind of speech will be needed for a representativebackground database.Although database mismatch is commonly seen as a problemin the community and among forensic experts, to theauthors’ knowledge, rigorous experimental studies regardingthis important issue in forensic speaker recognition have beenmainly focused on telephonic databases. With the release ofNIST multi-microphone databases in SRE 2005 and 2006 [1],a richer dataset has been available for research in the topic. Infact, realistic databases for forensic speaker recognition are importantfor two main reasons: first, more data will be availablefor research in database mismatch; and second, forensic laboratorieswill be able to improve the robustness of their speakerrecognition systems in casework conditions.Driven by these needs, this paper presents the Ahumada IIIdatabase, a real-casework publicly available corpus in Spanish,which has been acquired by the Acoustics and Image ProcessingDepartment of Spanish Guardia Civil. In its current release,Ahumada III Release 1 (Ah3R1), it includes speech data fromreal forensic cases recovered using one of the typical recordingsystems from Guardia Civil, namely analog magnetic tapescontaining GSM tappings. Moreover, the database is being extendedwith more material under this platform and also usingSITEL, a Spanish nationwide digital tapping system. Ah3R1includes variability in conditions such as noise, environmentalcharacteristics, emotional state, country and region of originand dialect of speakers, etc. Next releases will significantly increasethe amount of data presenting strong variability from realcases.This work also explores the database mismatch problem usingthe presented Ahumada III database and a corpus generatedfrom multi-microphone data from NIST SRE 2006, namely theNIST4M database (NIST MultiMic MisMatch). This paper isorganized as follows. First, the Ahumada III database is describedin the context of Guardia Civil operative procedures forforensic speaker recognition. The experimental section thenpresents results which illustrate the impact of database mismatchin system performance in two different ways: databasemismatch using NIST multi-microphone corpora, and robustnessin real-casework conditions using Ahumada III. Resultsshow the importance of the database mismatch problem andencourages research in the field and data collection. Finally,conclusions are drawn.2. Addressing database mismatch in realcasework: the Ahumada III databaseIn the last years, Spanish Guardia Civil has done a significanteffort on the application of automatic speaker recognition systemsto forensic voice evidences [2]. Following a Bayesianframework, and pursuing transparency in their reports, muchof this effort has concentrated in the assessment of their systemin the sake of testability, as a demanding requirement inthe new paradigm of forensic identification sciences [6]. Mostvoice evidences arriving to Guardia Civil laboratories have twopossible origins. First, digitized analog magnetic recordingsfrom GSM mobile calls are typical in cases between 1995 and2004. From those recordings of this type received in the lastten years, those authorized (case by case) by the correspondingjudge after a trial, have been added to a database registered inthe Spanish Ministerio del Interior, known as Base de Datos deRegistros Acústicos (BDRA) 1 . Second, nationwide digital interceptionsystem (SITEL) has been used since 2005 by the twoSpanish State Police forces. This system records digital wiretapsdirectly connected to all mobile telephone operators.With the purpose of robustness in real-case conditions andproper assessment of systems, Guardia Civil has recorded severalspeech databases in the last decade. Back in 1998, the AhumadaI corpus was collected [7] containing 100 male speakersin telephone and microphone recordings. A subcorpus ofAhumada I spontaneous telephone speech was used in NISTSpeaker Recognition Evaluations both in 2000 and 2001.complementary database known as Gaudi with 100 femalespeakers was recorded in 2001 2 . From 2004 to 2006, the Baezadatabase was recorded, including GSM and microphone spontaneousconversational speech sessions recorded at the sametime, from ¡¢£ males and ¡¢ females. Baeza has shown, upto this moment, the most relevant reference population to beused in real cases, as it was recorded through the same SpanishGSM network as in actual cases. In 2008, all 100 male speakersfrom Ahumada I are still available and are to be again recordedthrough GSM and SITEL. In this way, Ahumada II (as is to beknown) will constitute a major contribution for the analysis oflong-term stability and degradation of speaker features after tenyears. As most new cases come through SITEL recordings andare to be, by this time, evaluated with Baeza as reference population,also in 2008 almost 100 speakers from Baeza (differentfrom Ahumada speakers) are to be recorded again through SI-TEL. This database will be known as Ahumada IV and will beused for system assessment.2.1. The Ahumada III databaseAhumada III consists of authorized conversational speech fromreal cases both from BDRA and SITEL. The expected size ofthe database in number of speakers and variety of conditionsaddressed is huge both in terms of number of available calls andamount of data. However, as conditions are not uniform, andspeech recordings have to be authorized one by one, differentreleases of the database will be progressively available.Ahumada III Release 1 (Ah3R1) consists ¢£ of speakersfrom a number of real cases with GSM BDRA calls acrossSpain, with a variety of country of origin of speakers, emotionaland acoustic conditions, and dialects in the case of Spanishspeech.The unique low variability dimension is gender,as all of them are male speakers. All ¢£ speakers in Ah3R1have two minutes of speech available from a single phone callto be used as unquestioned (control) recording, with the purposeof model training or voice characterization. Additionally,ten speech segments for £ speakers and five segments for £speakers are included for testing issues, each one from a differentcall. Such fragments present between ¡ and ¢¤ secondsof speech, with an average duration of £ seconds. An evaluationprotocol, equivalent to that of NIST SRE [1], is availableand ready to be used. In this way, experienced NIST SRE par-1 With reference public scientific file number 1981420003 fromSpanish Guardia Civil, Orden Ministerial INT/3764/2004 de 11 denoviembre.2 A 25 Ahumada male subcorpus and an equivalent 25 Gaudi femalesubcorpus are freely available in http://atvs.ii.uam.es/databases.jsp.A
ticipants can run and assess immediately their systems on theAh3R1 data. Ah3R1 is publicly available for research purposesunder a license agreement 3 to be signed with Guardia Civil(contact: crim-acustica@guardiacivil.es). Sample speech segmentsare ready to be directly heard in ATVS website in orderto perceive the quality and diversity of Ah3R1 recordings.3. Experiments on database mismatchIn this section we present results illustrating the database mismatchproblem, and how Ahumada III will improve accuracyin real forensic casework. For score computation, the ATVSGMM system has been used, where speech data known to comefrom a given speaker is represented using Gaussian MixtureModels (GMM) adapted from an Universal Background Model(UBM). GMM of ££¢ mixtures have been used for modellingMFCC-based features with channel compensation based on featurewarping and factor analysis [4].3.1. Database mismatch with NIST4M databaseAn evaluation corpus, namely NIST4M (NIST MultiMic Mis-Match), has been constructed using NIST SRE 2006 data.This corpus aims at simulating unfavourable session variability.Moreover, utterances in NIST SRE 2006 from all speakers havingmicrophone conversations have been combined in NIST4Mfor trial generation, having multimicrophone data both in theenrolled models and in the test segments. Database mismatchhas been simulated by eliminating all the data for a given microphonefrom the background data, using such microphone onlyin NIST4M evaluation data. Thus, trials for that microphonewill be performed without considering such kind of data in thebackground set, and therefore under database mismatch. Thissituation is usual in forensic speaker recognition.For the construction of NIST4M, half of the telephone utterancesof each speaker were selected as testing segments andthe rest as training data. For microphone data, microphonesin one session per speaker has been used for training data, and¢ microphones from each of the rest of sessions are used fortesting. The microphone types changed with different speakers,in such a way that the data for all microphone types tend tobe balanced. Trials were generated by performing all possiblecombinations between the same speaker utterances for targettrials, and the next £¢ speakers for non-target trials, avoidingdifferent-gender comparisons.Development data has been selected from past NIST SRE.Microphone data from NIST SRE 2005 has been divided inthree sub-sets, separately used for training factor analysis parameters,UBM and T-Norm cohorts. This data has been complementedwith NIST SRE 2005, 2004 and 2002 telephone data,as well as speech from Switchboard I 4 .For experiments presented here microphones (¡ ) and¤ (¡¤) as quoted by NIST [1] have been chosen. Both microphonespresent different characteristics, being ¡ a distantmicrophone ¡¤ and a higher-quality device. Two different testingconditions have been defined. On the one hand, we definea same-microphone condition where trials containing onlya given microphone both in the training and testing speech datahave been selected. For this condition, in the case of ¡ thereare target and ¤¢¤ non-target scores, and in the case of ¡¤there are £ target and ¤ £ non-target scores. Such numbersare quite small, and therefore results obtained for this condition3 Available at http://atvs.ii.uam.es/databases.jsp4 See LDC website http://www.ldc.upenn.edu/should be carefully interpreted. However, we have consideredthem here in order to illustrate database mismatch when sessionvariability is controlled. On the other hand, we define amicrophone-model condition, where trials containing a givenmicrophone in the models and every possible channel in thetest segment have been considered. For this condition, when¡ is selected there are £¢ target and ¢££¤ non-target scores;whereas ¤£¤ target and ¤¢£¢ non-target scores have been generatedfor ¡¤.For database mismatch simulation, on the one hand, thescores for the ¡ microphone-model condition in a databasematching situation will have data from NIST SRE 2005 ¡microphone in the UBM, factor analysis data and T-Norm cohort.On the other hand, in database mismatch conditions datarecorded over NIST SRE 2005 ¡ microphone will be eliminatedfrom the UBM, factor analysis data and T-Norm cohort.The same procedure is applied to ¡¤. It is needed to remark thatthe data from a given microphone represents only a £¢¤ ¤¥ ofthe whole data used for training UBM and factor analysis, andonly £ up to £ models in the whole T-Norm cohort. This isimportant in order to evaluate the impact of the elimination ofsuch a relatively small amount of data from the background set.3.1.1. ResultsTable 1 shows the effect of database mismatch for the samemicrophonecondition described above. Performance is measuredin the form of equal error rate (EER) and Detection CostFunction (DCF) as defined by NIST [1]. Results show thatdatabase mismatch leads to a significant degradation in performance,especially for the DCF performance measure. Howeverthese results should be carefully interpreted, as the number ofscores in each condition is scarce.DB match DB mismatchEER DCF EER DCF¡ ¡ ¤ ¡¤vs. 15.2137 0.0391 17.094 0.0472vs. 1.4815 0.0033 2.2222 0.0143Table 1: Performance of same-microphone condition underdatabase mismatch.Figure 1 illustrates the effects of database mismatch for themicrophone-model condition. This aims to simulate a typicalforensic situation, where speech from the suspect may havebeen recorded in particular microphone conditions for whicha background database is not available. DET curves show asignificant degradation in discrimination performance when themicrophone of the considered model is eliminated from thebackground data.3.2. Experiments with Ahumada IIIIn this section, the use of Ahumada III for compensatingdatabase mismatch in real-casework data is illustrated. The simulationhas been conducted by adding Ahumada III data to theUBM trained with NIST SRE data described in Section 3.1. Althoughmore accurate results can be obtained by using Baeza orBDRA databases from Guardia Civil, we selected a NIST-SREbasedbackground dataset for the illustration of database mismatch,and also for transparency, because any laboratory havingthe NIST SRE databases can test the same results in AhumadaIII. The same ATVS GMM system has been used for performingthe experiments, except for session variability compensa-
Page 1: Addressing database mismatch in for

Addressing database mismatch in forensic speaker ... - ATVS

Create successful ePaper yourself

Delete template?

Save as template?