SPIDIA-RNA: First external quality assessment ... - TATAA Biocenter

M. Pazzagli et al. / Methods 59 (2013) 20–31 23Table 1QUESTIONNAIRE distribution frequencies: usual procedures used by participant labs (N =92 a ).Questions N %1. In which tube do you usually perform blood collection? K 2 EDTA 60 65.22NaCitrate 2 2.17LiEparine 1 1.09PAXgene blood RNA tube 22 23.91Other 7 7.612. How many milliliters of blood do you collect? 1.5 ml < ml < 2.5 ml 6 6.522.5 ml 6 ml < 5 ml 41 44.575ml6 ml 6 10 ml 38 41.30>10 ml 7 7.613. How long is the time interval between the612 h 50 54.35blood collection and the RNA extraction?12 h < h 6 24 h 29 31.52> 24 h 13 14.134. At what temperature is stored the collected blood? 80 °C 9 9.7820 °C 11 11.964 °C 48 52.17Room temperature 24 26.095. What is the procedure for RNA extraction?Do you use a kit? Yes 77 83.70The method to isolated RNA is based on... No 15 16.30Silica membrane 63 81.82Magnetic beads 3 3.90Precipitation 6 7.79Not available 5 6.496. How many microliters do you use to resuspend/elute650 ll 60 65.22the extracted RNA from blood?50 ll100 ll 6 6.527. Do you evaluate the concentration of extracted RNA?What is the method? Yes 81 88.04No 11 11.96Spectrophotometer 61 75.31Agilent Bioanalyzer 1 1.24PicoGreen 1 1.24Other 5 6.17Agilent + other 1 1.24Spectrophotometer + Bioanalyzer + other 12 14.808. How long is the time interval between the RNA extraction66 h 78 84.78and concentration evaluation?6h24 h 2 2.179. What kind of analysis do you usually perform on your extracted RNA?(multiple answers) rt-PCR 7 7.61rt-qPCR 35 38.04microarray 1 1.09rt-PCR + rt-qPCR 34 36.96RIN + rt-qPCR 6 6.52RIN + rt-PCR + rt-qPCR 3 3.26RIN + microarray 1 1.09rt-PCR + microarray 1 1.09RIN + rt-qPCR + microarray 3 3.26rt-PCR + rt-qPCR + microarray 1 1.0910. How long is the time interval between the RNA66 h 21 22.83extraction and the analysis of RNA?6h120 h 9 9.7811. At what temperature do you usually store the extracted RNA? 80 °C 77 83.7020 °C 15 16.3012. For how long time do you usually store RNA? No storage 3 3.26Days 4 4.35Months 8 8.69Years 77 83.70aOne participant did not answer to the questionnaire.the laboratory’s performance for this specific parameter. A zoomwindow was also displayed for each box-plot to aid visualizationof the participant’s result when the distribution of the variableswas highly skewed.3.2.1. Spectrophotometric dataTables A.1 and A.2 summarize the spectrophotometric measurementresults provided by the participants along with some detailsconcerning times, methods, and reagents used by each participant.Sections A.3 and A.4 show box-plots of the distributions of RNAyield and purity reported by the participants in the Results formfor the two extracted blood samples.For purity (A.3), the median value and inter-quartile range (IQR)are similar for RNA A and RNA B (median RNA A: 1.97, median RNAB: 2.00, IQR RNA A 0.20, and IQR RNA B 0.20). The warning limitsindicate that the performance of a laboratory can be classified as

24 M. Pazzagli et al. / Methods 59 (2013) 20–31being in control with respect to the results of the other participants(having excluded outliers) if their reported purity values are in therange 1.7–2.1.For total RNA yield (ng/ll blood) (A.4), the distribution of RNA Ayields is broader than that of RNA B due to outlier values. The medianvalues and the control limits of the two RNAs, however, aresimilar because computation of these numbers is not influencedby outlier values. The median yield of RNA A is 2.23 ng/ll bloodand of RNA B it is 2.25 ng/ll blood, and the IQR’s are very similar:2.07 for RNA A and 1.82 for RNA B. The warning limits indicate thatthe performance of a specific laboratory can be classified in controlwith respect to the results of the majority of participants if themeasured RNA total yield is more than about 1.00 ng/ll of bloodfor both RNA A and B.3.2.2. RIN valueSection B.1 reports the distributions of the RIN values measuredon the two RNA samples extracted by the participants. Due to atypicalelectropherograms observed for several of the extracted RNAsamples, the evaluation of RIN values was possible only for 77 RNAA and 78 RNA B samples. The median value was similar for the twoRNAs (median RNA A 7.6 and RNA B 7.4), while the variability representedby the IQR was higher in RNA B (IQR = 4.2) than for RNA A(IQR = 3.5). The limits calculated (B.1) indicate that the performanceof a specific laboratory can be classified in control with respect to theresults of the majority of the participants when the RIN value isgreater than approximately 4 for RNA A as well as RNA B.The electropherograms of the RNA integrity analysis for RNA Aand B are reported in Section B.2.Finally, to explore the association between RNA integrity andgene expression on our data, we dichotomized the continuous variableRIN according to the cut-off value of 5 as suggested by Fleigeet al. [17,18]. The box plots in Fig. 2, obtained by considering theGAPDH expression, are representative of the decreasing trend inexpression that we observed, regardless of the statistical significance,for all the other three considered genes (data not shown).3.2.3. Gene expression profileThe distributions of the gene expression [Log 10 (copies/lg totalRNA)] of the four genes tested are graphically represented in SectionsC.1 and C.2. The distributions of the expression levels for allthe gene transcripts are similar in RNA A and RNA B.3.2.4. Real time PCR kinetics analysisSections D.1 and D.2 report the distribution of the kinetics distance(Kd) obtained by the analysis of the qPCR kinetics data withTable 2Classification of the performance of the laboratories.Categories N %All in control or warning performance 24 25.81One out of control performance and/or one or more missing 38 40.86Two or more out of control performance with or without 31 33.33missingTotal 93 100.00Description of the categories: all in control or warning performance: labs with all performances in control orwarning, without missing; one out of control performance and/or one or more missing: labs with only one outof control (N = 12); labs with only one missing (N = 7) or only more than onemissing (N = 7); labs with one out of control and one missing (N = 8); labs withone out of control and more than one missing (N = 4); two or more out of control performance with or without missing: labs with two outof control with at least one missing (N = 8) or without missing (N = 4); labs withmore than two out of control with at least one missing (N = 11) or without missing(N = 8).the Kineret software described by Tichopad et al. [12]. The twolines depicted in the figures correspond to the theoretical limitsused to detect strong (>9.21) and weak outliers (5.99–9.21). Forall transcripts the median value is below the defined thresholds.Only for GAPDH were no outliers observed.3.2.5. Summary of the lab performance evaluationTable E (Appendix A) summarizes the performance of the laboratoriesfor the evaluated RNA quality parameters. The table shows theresults using three colors: green if the performance is ‘‘in control’’,yellow if the performance is ‘‘warning’’ or ‘‘weak outlier’’, and redif the performance is ‘‘out of control’’ or ‘‘strong outlier’’. When itwas not possible to evaluate the performance due to a missing value,‘‘missing’’ is reported in the summary table with an explanation inthe ‘‘comments’’ column. All data are visually summarized as a radargraph with performance symbolized by a colored square (same colorsas in the summary table) and the distance between the coloredsquare and the center of the radar indicates the performance (furtheraway from the center, the worse is the performance).3.3. Overall performance of the laboratories participating in the RNASPIDIA EQA programOn the basis of the RNA quality parameters and of the statisticalapproach used in this EQA program, for 24/93 laboratories (26%)the performance was not critical (all parameters classified as ‘‘incontrol’’ or ‘‘warning’’). For 38/93 (41%) laboratories only one qualityparameter was ‘‘out of control’’ (plus possibly one or more missingdata point). For the remaining 31/93 (33%) laboratories two ormore ‘‘out of control’’ quality parameters were noted, plus possiblyone or more missing data points (see Table 2).4. DiscussionFig. 2. GAPDH expression in RNA A and RIN variable dichotomized according to thecut-off value of 5 (N lab = 20 for RIN 6 5; N lab = 57 for RIN > 5; Kruskal–Wallis test,p < 0.0001).Molecular methods are today widely used in laboratory medicine.For genomics analysis in the last few years, it has been possibleto implement an adequate quality assurance framework. This isbased on knowledge acquired from specific surveys [13], theimplementation of quality assurance and proficiency programs(www.cap.org; www.emqn.org; www.dgkl.dc; www.eurogentest.org),and by the development of dedicated guidelines [14].For transcriptome analysis, the situation is different. Due to thewell known intrinsic instability of RNA and therefore the criticalrole the pre-analytical phase plays in RNA analysis, there is a muchhigher level of complexity of factors that affect accurate results.Only a few surveys and EQA programs have been implemented[9,15,16], and the quality of assays is currently not monitored.

M. Pazzagli et al. / Methods 59 (2013) 20–31 25One aim of the EU granted project SPIDIA is to develop evidence-basedguidelines and quality assurance schemes for the collection,transportation and processing of blood samples for RNAbasedanalysis. As one step towards this goal, a survey and a proficiencytesting program has been implemented to evaluate currentsample technologies for RNA-based analysis from blood amongroutine laboratories in Europe.The survey asked questions about what sorts of laboratory policiesand practices are in place (Fig. 1) for specimen handling, andrespondents were requested to provide information on aspects relatedto blood sampling and extraction protocols (Table 1). A cleartendency to use commercially available extraction kits (predominantlyusing silica membrane technology) was noted. The vastmajority of the laboratories performed the evaluation of yield andquality of RNA by UV analysis and stored extracted RNA at80 °C. The dominant downstream analytical methods were PCRtechnologies (qualitative and quantitative RT-PCR). Other aspectsof sample handling and analysis protocols were more variable, suchas the volume of blood used and the kind of collection tube, timeand storage temperature between blood collection and RNA extractionand analysis. The wide range of answers to these questionsconfirmed the need to develop guidelines for this area of analysis.Together with the survey, SPIDIA also implemented an EQA programfocused on the pre-analytical phase of analyzing RNA inblood. Details on the experimental set-up and the RNA qualityparameters used in this EQA were investigated in a pilot studyand have been reported elsewhere [3].Using this approach, we have evaluated the quality of RNA extractedby the participants and produced an individual reportdescribing the performance of the laboratory classified in respectto the results of the majority of participants. Looking at the distributionof the results obtained by the participants (see the report inAppendix A), we observed a median value of measured RNA purityclose to 2.0 for both RNA A and RNA B, which is typical for high qualityRNA [17,18]. The median RNA yield was about 2 ng/ll blood. TheAL and WL of the RIN values were 2.61 and 4.77 for RNA A (median7.6) and 2.49 and 4.25 for RNA B (median 7.4), respectively. Fleigeet al. [18] report that a RIN value higher than 8 corresponds to highquality total RNA, while a RIN value between 5 and 8 reflects partlydegraded RNA. According to the experimental model and statisticalanalysis proposed in this study, we classified ‘‘in control’’ some RNAsamples that according to Fleige et al., should be considered ‘‘partlydegraded’’. This is not an inconsistency for three reasons. First, wedefined valid result based on consensus with all testing participants’results and not by comparison to any absolute target, Second,participants’ RNA samples were obtained by different extractionprocedures, and Third, the survey material may have been heterogeneousat the time of extraction due to other pre-analytical variables(time and temperature of shipment/storage).Following the RIN cut-off reported by Fleige et al. we categorizedthe RNA A samples and analyzed the gene expression of thefour target genes; all of them showed an expression decreased insamples with RIN below 5. This trend underlines the relevance ofthe RIN in gene expression studies: RIN under 5 can induce misleadingin the gene expression evaluation, even if the ampliconlength is less than 130 bp.The distribution of the expression levels of the four gene transcriptsinvestigated as well as the Kd values obtained from theKineret analysis are similar in RNA A and RNA B. Taken together,these data show reliable values for the RNA quality parameters,but we must emphasize that they were obtained without consideringthe missing data and after removing outliers. For example, statisticalanalysis of RIN score data, only 82% of all of the samplesreturned by participating laboratories were considered becauseof aberrant electropherograms.Analysis of individual reports of the participants after theassessment of the performance of the laboratories participatingin the SPIDIA EQA program. revealed that 31/93 laboratories (about33%) had at least two quality parameters ‘‘out of control’’ and only24/93 (26%) of the laboratories had a ‘‘not critical’’ (‘‘in control’’ or‘‘warning’’) assessment for all the considered quality parameters.The goal of this first SPIDIA-RNA EQA was to gather and collateinformation on resulting RNA quality when the participants followedthe protocols and reagents in use in their laboratories. Dueto the high variability in protocols and reagents used by the laboratoriesthat received an EDTA tube, it was not possible to furtheranalyze the data of this survey with regard to the impact on a singlequality parameter of certain sample preparation methods andthe time interval between collection and extraction. Direct comparisonsof individual quality parameter of the laboratories thatused the PAXgene Blood RNA tubes and the laboratories that receivedEDTA tubes are in process.As mentioned earlier, gene expression stability is a prerequisitefor good performance of the isolated RNA in RNA based analyticalmethods, and the use of blood collection tubes without stabilizercan lead to artificially induced post sampling gene expressionchanges for certain genes [5–8]. A more detailed analysis of the differentsample collection technologies will be necessary to investigateand compare the results from the laboratories that receivedthe PAXgene Blood RNA tubes to those who received EDTA tubesto answer whether the use of a blood RNA stabilization technologyimproves the results of RNA based analysis from blood samples.In preparation for the second SPIDIA-RNA EQA, we have performedadditional studies on a small scale to further optimizethe protocols for the implementation of this next step. The use ofimproved protocols for blood collection and shipment, togetherwith more rigorous and defined protocols for the participants is expectedto allow a more in-depth analysis of the critical steps of thepre-analytical phase which will allow for the development of evidence-basedquality guidelines.AcknowledgementsGrant/funding support. The SPIDIA RNA study has been supportedby European Union (FP7. Title ‘‘SPIDIA: Standardizationand improvement of generic pre-analytical tools and proceduresfor in vitro diagnostic’’, Grant agreement no. 222916).We acknowledge Labonnet Ltd. Company for providing theKineret software.We gratefully acknowledge the contribution of all partners andparticipant laboratories.We are very grateful to Lynne Rainen for the scientific contributionand language editing of this manuscript.Appendix A. Purity and Quantity of RNA A and RNA B.A.1. Spectrophotometric data provided by your labSample 260 nm 280 nm 320 nm Purity Quantity (ng/ll blood) Dilution factor Extraction vol. (ll) Elution vol. (ll) BufferRNA A 0.051 0.025 0.001 2.083 0.600 1 5000 30 –RNA B 0.114 0.055 0.000 2.073 1.368 1 5000 30 –

26 M. Pazzagli et al. / Methods 59 (2013) 20–31A.2. Additional information provided by your labSample Time interval (hours) Temperature of RNA storage Extraction SpectrophotometerArrival toextractionExtraction toanalysisArrival toextractionExtraction toanalysisProducer Supplier Producer SupplierRNA A 4.75 h 164.00 h 4 °C 20 °C Qiagen 74134/RNeasyPlus Mini kitRNA B 28.75 h 140.00 h 4 °C 20 °CIMPLEMNanophotometerA.3. Your lab () versus overall distribution – PurityIn the figures the blue lines represent the Action Limits (ALs) and the gray lines represent the Warning Limits (WLs).3.02.5zoom 3.02.5zoom2.52.52.02.0Purity2.0Purity2.01.51.51.51.5Lower AL=1.58; Upper AL=2.20Lower WL=1.74; Upper WL=2.14Lower AL=1.66; Upper AL=2.22Lower WL=1.74; Upper WL=2.151.01.0RNA A (N=85)in controlRNA B (N=87)in controlA.4. Your lab () versus overall distribution – QuantityIn the figures the blue line represents the Action Limit (AL) and the gray line represents the Warning Limit (WL).55505zoom55505zoom454454Quantity (ng/µl blood )4035302520153210Quantity (ng/µl blood )4035302520153210105AL=0.34; WL=1.06105AL=0.26; WL=0.9200RNA A (N=87)warningRNA B (N=88)in control

M. Pazzagli et al. / Methods 59 (2013) 20–31 27Appendix B. Integrity of RNA A and RNA BB.1. Your lab () versus overall distribution – RIN numberIn the figures the blue line represents the Action Limit (AL) and the gray line represents the Warning Limit (WL).101099887766RIN5RIN54433221100RNA A (N=77)RNA B (N=78)AL=2.61 WL=4.77AL=2.49 WL=4.25in controlin controlB.2. Electropherogram Agilent

28 M. Pazzagli et al. / Methods 59 (2013) 20–31Appendix C. Quantification of four genes by real-time PCR on RNA A and RNA BC.1. Your lab () versus overall distribution – RNA AIn the figures the blue lines represent the Action Limits (ALs) and the gray lines represent the Warning Limits (WLs).C-FOSGAPDHIL1bIL8101010109999Log 10 (copies/μg total RNA)876548765487654876543RNA A (N=91) RNA A (N=91) RNA A (N=91) RNA A (N=91)Lower AL=6.74; Upper AL=8.98Lower WL=7.04; Upper WL=8.803Lower AL=5.63; Upper AL=7.99Lower WL=6.09; Upper WL=7.68Lower AL=5.64; Upper AL=7.06Lower WL=5.81; Upper WL=6.90Lower AL=6.64; Upper AL=8.91Lower WL=6.73; Upper WL=8.70in control in control warning in control33C.2. Your lab () versus overall distribution – RNA BIn the figures the blue lines represent the Action Limits (ALs) and the gray lines represent the Warning Limits (WLs).C-FOSGAPDHIL1b10 10 10 10IL89999Log 10 (cop pies/μg total RNA)876548765487654876543333RNA B (N=92) RNA B (N=92) RNA B (N=92) RNA B (N=92)Lower AL=6.79; Upper AL=9.02Lower WL=7.13; Upper WL=8.79Lower AL=5.68; Upper AL=7.78Lower WL=6.14; Upper WL=7.48Lower AL=5.51; Upper AL=7.11Lower WL=5.78; Upper WL=6.85Lower AL=6.57; Upper AL=8.92Lower WL=6.69; Upper WL=8.67warning out of control warning in control

8M. Pazzagli et al. / Methods 59 (2013) 20–31 29Appendix D. Interferences in quantification of four genes by real-time PCR on RNA A and RNA BD.1. Your lab () versus overall distribution – RNA AIn the figures the two lines represent the two Kineret threshold for outliers identification: 5.99 (weak outlier) and 9.21 (strong outlier).C-FOSGAPDHIL1bIL810zoom10zoom10zoom10zoom99994087408740874087666635535535535il8544443033033033032222Kin neret distance252015101025201510102520151010il8252015101055550000RNA A (N=89) RNA A (N=76) RNA A (N=90) RNA A (N=91)in control in control in control in controlD.2. Your lab () versus overall distribution – RNA BIn the figures the two lines represent the two Kineret threshold for outliers identification: 5.99 (weak outlier) and 9.21 (strong outlier).C-FOSGAPDHIL1bIL810zoom10zoom10zoom10zoom99998888407640764076407635543554355435il854istanceKineret di302520153210302520153210302520153210il83025201532101010101055550 0 0 0RNA B (N=91) RNA B (N=71) RNA B (N=90) RNA B (N=91)in control missing in control in control

321030 M. Pazzagli et al. / Methods 59 (2013) 20–31Appendix E. SummaryRNA A –Purityin controlRNA B – PurityRNA A – QuantityRNA B – QuantityRNA A – IntegrityRNA B – IntegrityRNA A –C-FOS quantRNA A – GAPDH quantRNA A – IL1b quantRNA A – IL8 quantRNA B – C-FOS quantin controlin controlin controlin controlin controlin controlin controlPerformance Missing CommentswarningwarningwarningRNA B – GAPDH quantRNA B – IL1b quantRNA B – IL8 quantRNA A – C-FOS interferencesRNA A – GAPDH interferencesRNA A – IL1b interferencesRNA A – IL8 interferencesin controlin controlin controlin controlin controlwarningout of controlRNA B – C-FOS interferences in controlRNA B – GAPDH interferences missing there isn't enought data forkineret analysisRNA B – IL1b interferences in controlRNA B – IL8 interferences in controlRNA A – PurityRNA B – IL8 interferencesRNA B – PurityRNA B – IL1b interferencesRNA A – QuantityRNA B – GAPDH interferencesRNA B – QuantityRNA B – C-FOS interferencesRNA A – IntegrityRNA A – IL8 interferencesRNA A – IL1b interferencesRNA B – IntegrityRNA A – C-FOS quantRNA A – GAPDH interferencesRNA A – GAPDH quantRNA A – C-FOS interferencesRNA A – IL1b quantRNA B – IL8 quantRNA A – IL8 quantRNA B – IL1b quantRNA B – C-FOS quantRNA B – GAPDH quantThis report has been produced with the collaboration of the University of Florence (M.Pazzagli, S.Gelmini,C.Orlando, L.Simi, F.Malentacchi), Fondazione IRCCS Istituto Nazionale dei Tumori of Milan (P.Verderio,S.Pizzamiglio, C.Ciniselli) and TATAA BIOCENTER (A.Tichopad).

M. Pazzagli et al. / Methods 59 (2013) 20–31 31Appendix A. Supplementary materialSupplementary data associated with this article can be found,in the online version, at http://dx.doi.org/10.1016/j.ymeth.2012.10.007.References[1] C. Nussbaumer, E. Gharehbaghi-Schnell, I. Korschineck, Forensic Sci. Int. 157(2006) 181–186.[2] R. Ovstebo, K. Lande, P. Kierulf, K.B.F. Haug, Clin. Chim. Med. 45 (2007) 171–176.[3] K. Günther, F. Malentacchi, P. Verderio, S. Pizzamiglio, C.M. Ciniselli, A.Tichopad, M. Kubista, R. Wyrich, M. Pazzagli, S. Gelmini, Clin. Chim. Acta 413(2012) 779–786.[4] S.A. Bustin, Methods 50 (2010) 217–226.[5] L. Rainen, U. Oelmueller, S. Jurgensen, R. Wyrich, C. Ballas, J. Schram, C.Herdman, D. Bankaitis-Davis, N. Nicholls, D. Trollinger, V. Tryon, Clin. Chem. 48(2002) 1883–1890.[6] C. Härtel, G. Bein, M. Müller-Steinhardt, H. Klüter, J. Immunol. Methods 249(2001) 63–71.[7] A. Pahl, K. Brune, Blood 100 (2002) 1094–1095.[8] S.J. Kim, D.J. Dix, K.E. Thompson, et al., Clin. Chem. 53 (2007) 1038–1045.[9] C. Orlando, P. Verderio, R. Maatman, J. Danneberg, S. Ramsden, M. Neumaier, D.Taruscio, V. Falbo, R. Jansen, C. Casini-Raggi, F. Malentacchi, E. Marubini, S.Pizzamiglio, K. Vernelen, J.C. Libeer, V. Palicka, M. Pazzagli, Clin. Chem. 53(2007) 1349–1357.[10] B. Efron, R.J. Tibshirana, An Introduction to the Bootstrap, Chapman and Hall,London, 1993. 436 pp.[11] B. Iglewicz, D.C. Hoaglin, How to Detect and Handle Outliers, ASCQ QualityPress, Milwaukee, WI, 1993.[12] A. Tichopad, T. Bar, L. Pecen, R.R. Kitchen, M. Kubista, M.W. Pfaffl, Methods 50(2010) 308–312.[13] OECD Genetic Testing: a survey of Quality Assurance and Proficiency Testing,2007, 134pp. ISBN-978-92-64-03201.[14] R.M. Madej, Z. Cao, L. Hall, P.D. Neuwald, L.O. Williams, Proficiency Testing(External Quality Assessment) for Molecular Methods, Approved Guideline.CLSI document MM14-A, vol. 25, no. 24. Clinical and Laboratory StandardsInstitute, 2005, pp. 1–51.[15] S.C. Ramsden, S. Daly, W.J. Eilenkeuser, G. Duncan, F. Hermitte, E. Marubini, M.Neumaier, C. Orlando, V. Palicka, A. Paradiso, M. Pazzagli, S. Pizzamiglio, P.Verderio, Clin. Chem. 52 (2006) 1584–1591.[16] C.C. Raggi, P. Verderio, M. Pazzagli, E. Marubini, L. Simi, P. Pinzani, A. Paradiso,C. Orlando, Clin. Chem. Lab. Med. 43 (2005) 542–548.[17] S. Fleige, M. Pfaffl, Mol. Asp. Med. 27 (2006) 126–139.[18] S. Fleige, V. Walf, S. Huch, C. Prgomet, J. Sehm, M.W. Pfaffl, Biotechnol. Lett. 28(2006) 1601–1613.