Research Priorlties 'in Educatlonal Technology: A Delphi Study

Research Priorlties 'in EducatlonalTechnology: A Delphi StudyConstance Pollardloie St,ar UJniversityRichard IPollardUJniversq)j oflda/ivAbstract1/sarticre !,eports un the findings qf'a De1sphi study conducted to derennine l e;anea need's inedutational teckleinogy over the next five yearc fin I?elphi pant C on'isted o'f 30 edttcatinoiatechnoalogy e.xpetts f,ron throughtout the United Sates .ho participated in a three-routndconsefsusbuiiiigigpttiPerfs viaf tie Internet. he results vis e-resetrch study pratnide afiwntework qfeigft reae/rdiprtority areas and spefic tesa re/rih topks fin tibose engnged in educationsil tehnology.u'or't reseacrh priorities, lerniung teachersnodels, sratengies.,INTRODUCTIONSince the introduEction oa the compuiter inito cducation ill the i970s, researchershave investigated its effects on students, teachers, and learning environiimentsFarly studies were focused on demiionstrating the impact of a particulartechnology or software on student achiievemenit and student motivation. Withincreased access to computers and the Internet, researchers extendedi their- cffortsto investigate the role Of tchtology in the educational setting includingits innpact on teachers and the learnini£g process. In the past decade, the study ofthe comnputer as an instructional delivery miecdtitun has been expanded to investigatingtechnolo)gyv as 'a transformational1 tool anid an integral part of the lear-nitgenvironmient' (Fouts, 2(100, p. 9)A review of the litcrature discloses an abundanice of educational technologyresearch studies employing a v,ari£ty of research methodologies in a variety ofeducational settings. JI an cfto t to synti hesize and analyze the results of thesestu6dies, a number of meta-analyses haive beeni conndttctedi. Onle of the Inorecornprehensive meta-analytie studies (Kutlik, 1994) surmnarized more than 97of the comnputer-based inistrLuction studies condLocted in the 1980s, notinig thatstudenits typically learn mnore and iaster in courses involving cornputer-based instiruetonand have mnore pOsitive attitucdes. The results of his ineta-arialysis supporttihe use of compiuters as a means to improve studenit achievement.Scharer (2001) analyzed a numbiber of large-scale studi6es of educational technologyand concluded that studenits in techinoleogy-rich elivironiminelits experteneedpositive effects on achievetmerit ancd a consistent irnprovemnent in selfconcept.Moreover. his rescarch supports the use of computer assistedinistructioni and collaborative networked technologies as a means to teachhigher-order thinkicng.Waxmari, Connell, and Gray (20t)2), £n a tepit eoifllSsittlcc Ly the U.S.Department of Education, presented a quantitative synthesis of recenit researchjournal of'Research on Technolgy in lEducation 145

oin the effects of teacking and learninig withi technology. Using the statistical restiltsof experimental and quasi- experimcintal published research, they conlcludecdthat there is a modest, positive effect of teaching and learninlg with technologyon studenit outcomes. 'the authlors rioted a lack of quality, refereedquantitative studies in the area of educational techiology and allutdetd to the seriousproblem of the lack of empirical evidcnce that certain programs or approachesutilizing techiiology adc effective.Although thc results of many research studies and ineta-analyses do showsome positive results in the use of techniology, policymrakers want to see it decisivelydemonstrated that technologys value measures up to the cost. Tlhey aredemnandling evidence that their investments in cducationial technology have beenworthwhile. The UJ.S. iDepartmnent of Fducationi has been rallyillg educatorsand researchers to mnarshal evidence thiat shows that stu(dents beniefit fronm ahigh-tech cnvironiment. A recent paper commrlissioned by the U.S. Departmentof Education titled A Retrospective on 'iventy Years ofh½ucation Technologjgy to/iny(Culp, Honey, & Mandinach, 2003) examined twenty years of key policy reports.Providing an overview and analysis of 28 reports, the authors concLudthat, "The call for research on the impact of educational technology on schoolsand teaching and learntilng activities is a final constant theme found over thepast twenty years of reports" (p. 15).This mandate for research on the cffects of teclhnology on teaching and learningis clearly confirmed in the National diducationa tlechnology Plan (U.S. I)epartrnentof Education, 2000), " I'here is a pressing need for a high-quality,long-term national agenda for collecting, analyzing, an(d disseminatinig informatio)non the use and effectiveness of technology in education" (p. 44). Moreover,one of the five goals outlined in the national plait is to provide "research andevaluation activities that will improve the next generation of techinology applhcationsfor teaching and learning" (p. 4).The quaestion is, however, "what researclh will mnost benefit educators andwhere should research efforts be dlirected?" TFhe call for research it edtucationaltechnology is clearly evident, but there is a need for a f'raniework for that research.Ihis paper reports on rthe results of a national study that was conductdctto provide guidance l-or the development of an educational technology researchagenida.Ihe study; utilizing the Delphii technique, was tindertaken to identif, categorize,anti Prioritize research needs that should be addresscd in educational technologyover the niext five years. A D)elphi panel of 30 educational tecthnologyexperts throuighout the United States was fornmed to generate. discuss, and rateresearch priorities in educational technology. This c-research (Anderson &Kanuka, 2003) study featured a three-rouind Delphi process comnpleted via theInternet.METHODOLOGYThe study employed the D)elphi technique to obtain a coinsensus fromzl educaionialtechnology experts about areas/issues that are tnost in neecd of researchover the next five ycars. Initially developed by the RAND Corporation in the146Winteir 2004-2005: Veilune 37Nurnber 2

early 195ts to predict future defc'nse nieeds (C'lope, 1981), the Dclphi has nowbeen implemented aciross disciplinie areas-and in educatiorn as earl y as 1971-as a means of obtainiing opirnions from persons without ph)ysically bringinigthiem together (Cyphlert & Cant, 1971).L.itnstone and 'lr roff (1978) described the lltility of the Delphi as a researchtechnique particularly in the, following sittua[ioins1. 1Fhe problemt does not lend itself to precise analytical techniques btut canbernefit fromn subjective judgments on a coliective basis.2. 'k-he individuals who nieed to interact cannot be brought together in a faceto-faceexchange becauLsC of time or cost constraints. Further, a convenitional confetenee tenids to be dlominated by particularly strong personalitiesor to give rise to an undesirable bandwagon effect. (p. 275)A st-udy examining thc effecti-veness of the l)elphi in comparison to traditionlaldistcussion groItps determined t-hat the Delphi was the more effective becauisethie "...anonyirity and isolatiO m of the participants facilitated a freedorifrozm conformity pressures" ('Van de Ven & Delbecq, 1974, p. 61 9). An additionalperceived bernefit of utsing the Delphi is the belief that the writing processenables participants to thoroughly deliberate an-d reflect tupotn all aspects of theproblem.T he result is participants' submission of precise, distinctive ideas.Whlile there has been considerable variance in administerinig the Delphi pfOcess,prescribed methodology reqUireS that two cycles of questionnaires andfeedback reports be used (Van de Ven &c Delbecq, 19743. lor this study, fornierPreparinglorniorrow'sl,'eachers to Use '['chnology (PT") Grant Directorsthroughout the LUsnited States were e-mnailed an invitation to participate in athree- round Delphi Process involvinig two cycles of online questionnaires andfeedback reports. l'rocedural steps Nvere as follows:Round]: Participants were directed to rhi l)elphi study Web site and askecdto generate respoinses to the qitestion, "What should tire research priorities flrecdutcational techn0o0logy, over the next five years he?" Roundi 1 staternents werearranged in caregories according to research focus. identified rescarch statenientsand categories wvere thiei used to develop tirc Round 2 instrument.Round 2: Participants wxeie asked to ,ate the researcl statenients and categoriesidentified in Round 1 as to researchl need. In additiorn to rating each researclrstatemen-t, they ranked the major researcxh categories in order of theirperceived importance. Once returned, descriptive statistics for the group ratingswere catculated:i meani, rnediin, and standard deviatiotn.Roun.d3: The ratings of research statements and rankings of major researchcategories by the group in Round 2 were comnpiled. Participants in Round 3again ranked the mrajor research categories as they dicl in Round 2, btut this timedescriptive infinmationi about h1ow the grgoup responded, as a whole, was pr-ovided. larticipatitig experts were asked to review each itenm, consider the groupresponse and tlhen re-rate the items, taking the information into account.1 he three-round i)elphi process enabled the participanits to generate theirown opinions about necessary educational technology researceh areas, prioritize]ournai oJ Research oni Techlnoogy in Edtication4 147

esearch focus categories, and then to finalize their views based upon considerationof the entire group's opinions. IThis process, engendering thie dynlamics ofeff-ective group interactions, cnabled researchers to gain a consensus from apanel of expert participanits in diverse geographical locations about cducationaltechnology research priorities for the twevnty-first century.Delphi Patiel ExpertsI'he educational technology experts inlvited to participatc in this Delphi studywere individuals whio had served as grant directors for a Preparing Tomnorrow'sTeachers to TUse JIchnology (PT3) grant. '1'he PT grant program (U.S. Departmnentof Education, 2004) was created by Congress in response to the urgentcall for teacher quality improvemeent and the need to reform teachiing and learningin cducation. Sinice 1999. P1'3 has awarded nmore thani 400 grants to inniovativeprogramis focusinig on technology-infused learninlg in the K-l6 environinent,with formnal and informal evaluation of grant interventions as integralelemiients. Although specific program componenits differed, each of the P1ITgrant awardees worked to transform education so that technology became integratedthroughout the reachling and learning process. PT' grant awardees wererequiredt to provide a research basis for all grant activities, develop strong grantevaluation components, and use sould research methodology in examining theeffects of technology interventions. [lus, PT 3 grantees were viewecd as the mostappropriate choices for participating on thc Delphi panel based: on their proactiveinvolveincnr in promoting technolog,-infused learning in the K-16 environmrentand in examininig the cifects of those efforts.As the Delphi process is a time-consuminlg ole potential expert participantswere contacted in advanice to deterrmine if they were willing tO commit to allthree roundls of the research stuidy. Sixty-three grant directors fromn the 1999PT' grant program were contacted by e-mnail and asked to particpate or to rccommnendsomeone friom their grant (e.g., the grant evaluator) who was activelyengaged in the grant techniology research activities to take part in the studv.Thirty-two educational technology experts initially agreed to participate, loutonlv 30 wcrc able to complete all three rounds of the Lelphi process. A reviewof lDelphi studies reveals that Delphi panels are uisually comnprised of 10 to 20members and that "few n1ew ideas are generated withini a homiogeneous grouponce the size excecds 30 well-chosen particpanits" (Delbecq.Van de Ven. &Gustafson, 1975, p. 89).' able 1 provides the demographic characteristics of the 30 Delphi expert participants,who wcre primarily (90%) comprised of universitv professors and admitnistrators.The 19 males (63%) and II females (37%) were experienced technologyusers, with 26 participants (87%) reporting tnore than 10 years oftechnology experience.FINDINGSThe results of the three-round Delphi study reflect the consenisus of opirnionsfrom 30 professiornals in educational techlnology, all individuials who hadworked as grant directors or gratnt evaluators for PT' grants. The. Rounid 1 ques-148Winter 2004--2005: Vo4lu,ae 37 Nunzber 2

Table 1: Demographic Information of Educational Technology ExpertsCompleting the Delphi StudyDeinographic em N noGenderMale 19 63'emale 11 37Educ5natio--PhD 19 63EdD 8 27D)l?A 1 3Mvlasters 2 7Current Position -leldProfessoi 16 53Adn-inistrator 9 30Professor/Adminiisti ator 5 17Years of Technology lExperience5-9 years 4 1310- l 5 years 8 2716-20 vears 13 43Over 20 years 5 17Institutiolnal ftflTiationUniversity 27 90State Department 2 7Business 1 3Geographical LocationEastern U.S. 11 40Central U.S. 9 27NWesternl U.S. 10 33tionnaire asked Delphi panel nmemlbers to respond to th-e question, "Whatshould be the researchl priorities for educational technology over the next fiveyears?" In Round 1, expert participants generated 167 responses as to the researchneeds iii educational technlology over the next five years. Tihese responseswere rev iewed by he researchers and coinsolidated so as to elitnirnate duplicGtion.Common research theme areas with similar responses were conmpiled withas much of the respondent's original wording as possible retained., IT) total, 84statemnents were coffipiled and then categorized under eight research prioritytheme areas: Assessmnent, External influcnices, Current Issues, Learniing, Models/Strategies,Schools, Teachers, ancd Web-Based Irnvironnients,For Round 2, panel members were asked to rate the 84 statements on aLik-erttype scale as to the degree of need (I = No Need; 2 = Low Need; 3 MediurniNeed; 4 = High Need: 5 Very -ligh Need) ftor research that each statementrepresents. Addlitionally, panel mermibers were asked to prioritize the eightresearch areas and encouraged to comnrictit on any of the research statenmentsand/or ireas. This was their first exposure to tthe research statements and areasgenierated by the panel, arid they were informied that they would ihave anotherJouezal of, Researhb on Jechnology it ,Edua tion1 149

opportunity to assign ratings ini Round 3. This round allowed thleml to assignltheir initial perceptioni of the need for the research knowing full well that theywould finalize their ratings based on thei perceptionis and the panel's ratings inthe£ next round. In this way, a group consensus oni research priorities could bereachcd.The Round 3 questionnaire fcatured the panel ratings for the 84 statementsand eight research areas listing the mnean, mtiedian, mode, and stancdard deviation.For this round, paniel memhbers were asked to review the research priorities,con1sider the group response and then re-rate the iteEns, taking the inforniationinto accountt. The eight priority areas were also reviewed arid participantsre-ranked the areas from I to 8, with I beiing the top priority for research ineducationial tcchnology.lFable 2 presents the LDClphi pan1els ranking of the rescarch priority areas foredticational technlology over the next five years. Research itn all aspects of learni£1gandc the effccts of technology on the learning process was assigned the toppriority, with research focused on teacher use of technology and teacher trainitnga,s the second priority.Table 2: Ranking of Research Priority Areas-Round 3 Delphi ResponsesRanlk Category Mean Mode N %I Learning 1.32 ] 22 732 Teachers 2.68 2 1. 373 Models/Strategies 3.63 3 19 334 Assessment 3.89 3 10 335 Current Issues 5.21 5 11 376 Schools 5.57 6 13 '137 Web-Bascd :nvinieoss 6.21 7 11 378 External Influenices 7.47 8 27 90fhe rest of this paper will discuss the research priority catcgories and the specificresearch statements within cach priority area. Table 3 lists the 12 researchstatements inciudcd in the learning category with the Round 3 descriptive statisticalsummary.LearningAlthotughl the nieed( to investigate the imipact of techniology oti studet achicveinienthas beeni a cornmon focus of technology research, the Delphi panel membersstill considered it a high priority for the next five years. Pan1l members commentedon thie establishnient of federal, staite, atnd local standards and accountability issuesas additional reasons wlhy it is so important to examnine student achievement.Ani examinationi of the m-ean ratings of the 12 iesearch statements generateddiscloses a wide range of perceived liced, with1 the panel nienibcrs empphasizingbroader, niiore comprehensive research on learninig and student achievemtient(mcan = 4.47) and less emphasis on investigations tO deterninie the differencenmade by eniploying specific technologies (mean - 2.57). Thiey advocated researchexaniiing how people learn utsinig technology, unlike mariy past researchl150150Winter 2004 ---2005: Voluroe 37.Numbir2

studies that have focused on specific techntiologies and theit imnpact on learninga particular set of skills or content (HEoniey, CuLip, & (C'arrigg, 2000). TheDelphi paniel recomnmended that research efforts be directed toward thie role oftechnology in helping students to becomre better probleml solvers and to accomplishlearning tasks. Coupled with this examination was the need to designl effectivetechnology-enhaniced learning environments to determiine how technologycontributes to student learniing.Panel membitners cited the need to coniduct research that examines the impactof technology in relatison to learnitng principles, brain research. arid multi-r iodallearning. T hey discussed the neect to develop an understanding of how peoplelearn Usinig technology and the impact of technology on the various domnaitns ofleartning (cognitive, affective, and psychmorntoi). The emphasis was on "learning"and riot oni scores onI achievement tests.Table 3. Learning Research Category-Round 3 Delphi ResponsesResearch Need _Meani SD Md-nInvesxdgatc the inpact of technology on sttideritachieverinent 4.47 .69 5Develop anid use learning priniciples to designtechnlology-enhanced learning environn)ents 4 10 .71 4Examine the role of mr£ulti-mnodal learning to providejutst in imne" and only "just as needed" soltitionsto training and in'formational needs 3.85 1.08 4l.,ink knowledge about teachting and learningwith appropriate technologies 3.95 60 4Determrinc the efficacy of learrning content usinglethnologh3y versis more traditional Iieth'ods 3.45 1t .09Exanimne miethods to learn techniology contextually 3.40 .99 3Determnine if the level of technology use relates tothe engagement of activities in the learning process 3030 .57 3Deteinmine the impact of technology on varioutsdomains (cognitive, aiffective, psyc.hornotorl 3.25 .71 3Identify a taxonomy that best represents the issuesand understandings of learner cenereednessand deeper learning 3.25 .78 3Exaiinen methods to make stUdients better at solvingproblemis and accomrplishing tasks usinig I "r(btloimationi ant i'oimleationI'innlogyt 3.25 .98 3Examline how the brain leai ns sod ccse brainscience-based understistinding C£ofhoWv to make effectivetisc of C(T tC) heip improve learning 3.15 .93 3Investigate the cognitive paths of lear2lers in interacntivecourses 3.10 .96 3Determine whvich technologies are making a difference 2.5' 1.3 2Not. J¾reiczpans iere as t o n,e earbreseayvh•sarentrn tire =e nfdl'ie u/¾sn a 5-pointLikenr-tjpe scale (I = N"ees Aleed and 5 = lreiy Zhigh5 Alerdf).Jou rnal of Researcb on Ikehnoiory in Education1 .5

TeachersResearch designed to develop models for preparinig iniservice and preserviceteachers to be more effective users of technology was seen as a high priority area.Iviost of the parnel menmbers were (lirectly involved in tcacher training and perceiveda need for rescarch-based mnodels for teacher training and professional developmentactivities. 'Iable 4 lists the eight rcconuniendationis generated by the group.Table 4: Teachers Research Category - Round 3 Delphi ResponsesResearch Need Mean SI) MdnDevelop models for preparing inservice and preserviceteachers to be more effective users of' technology 4.(05 .68 4Examine approaclhes that apply technologies toindividualize teacher profcssionzal devclopment basedotn real problems and opportuLnlities in the teacher'sclassroom 4.0 .72 4Examine techniiology barriers to assist inservicc andprcservice teachers when integrating educationaltechnologies into the daily teaching and learninlgenvironiment 3.85 .S1 4Examnin£e the nature andl extent of CT1 (Informationand Commurication Technology) in preserviceteacher cducatiotn to produce teachers whot effectivelyintegrate techrnology (instruiction and assessment)as soon as they get on the job 3.6 .82 3.5Examine effective. technology professioinal developmentpractices in K -12 schools 3.4+5 .83 3Examine teaching styles in relation to the use oftcchnology 3.35 .75 3Determine barriers to using technology in theclassroomi (lhardware availability, time. etc.) 3.15 .93 3Dctermuine the influcnce of handhelds (Palms) oncritical thinkinig and teacher behavior 2.95 .88 3NVote. Participants were asked to rare each reearch statement on rtse bass o .;need using a 5-pointlfkert-type scale (1 = No iVeed and 5 = Very Hiti: Need).The Delphi panel experts not otily expressed a rieed to develop tiodecls forpreparing technology-proficient preservice teachers, but wanted to examiine thenature and extrnrt of IC T (Inform-lation anti( Communication Technology) inpreservice teacher programs to produce teac'hers who effectively integrate technologyas soon as they begin their teachiing duties.Acknowledging that there are barriers to integrating tcchnologys the panel recommiendedresearch to examine those barriers in order to develop ways to assistinservicc and preservice teachers. As one panel tMetnber wrote, "Tcachers are tiekcy to technology ititegration. If wc want themii to integrate technology they needto help in overconming the many challenges associated with tising technology."152Winter 2004-2005: Volze 37NNmnber 2

Models/Strategies'fe focus of the tcwrninetndationis in this cate;gore , was t) examine wax tO ineprove traching/learning throughi the use of techinology-based instructiolial modelsancd strategies.The Delphi panel generated 15 research statements that were includedin a ategory titled Moclels/'Stiattgies. Although the members of the Delphipatiel geneerated m-ore rcsearch statemenits witbiti this categorv thai a3ny other inthe studvy it should be noted that nio mean ratinpg for individtial research statemenitswas over 4J), as w'as evideniced in soime of the othet research categories. 'T he factthat the patnel meibers were, oni the whole, heascvly invoLveed ill teacther traininocould accouLtt for thle preponderance ofsuggcsted research activities able 5 liststhe 15 statemnents alicn dicscriptive statistical summary for each statemenit. This rcsearchpriority arca feattired recommendations frotil the ceducational technology expertsto examinetk thc tSe of tcchnologTy to) develop instru tnonal tmodels and strate-Table 5: Models/Strategies Research Category-Round 3 Delphi ResponsesResearch Needl Mean SI) Mcidnt xaniileeand develop nlodels lor inregriing techlnologyinmcK--2 instructionl 3'95 .82 4Bxamine tne use of techitology to sUppoit distribtuo c,learn ing and projeect-bascd Icarning 85 .87 4D)ete£ mine "best practiCes' scenarios for r tehchinglVith technology 3.8 1(30Examriile the tse of tcehnotoiogy enhanced ilistrttctonialsttategies to ben7eiit specific grOups ol stUdcnts 3. 6 .S')Investigate wa's to tnse technology to extend thelearning exp rictce heyoncl the traditional classroron 3.() .88 4Develop effect:Ve edcteationaimulatiotns3 .4 .94 3.5De)termne effectiv use tof tool sofiware to support,he reaching and .earning process 3.3 3 3(reate interopet tble technology applicatioris 3.2 I95 3ID)ete tnine thie impact of digtal engageteni: practices(i.e. digital strs itelling artd 'I'ninkQuest) on K--12stttdet achilieWI tIeClt 3.3 .110 3Develop neiore etfectiv sootwcare to promote stuCnteoLearningi, 3.IS .87 3Dctero tioc areas in wnticlh I tighlhl 11tcraCeiVe (Colni it e-Assisted L.eartitog .s ngcore eff ctive than tratdittot-na`teachind 31 1 .02 3Restructure "tutoring sor£tware o lIcart fi-om the stLdellt I3'5 1.05U .Ex.Ilmioe the role of cxperrDev'elop inteiligenit tutoring svstett sesEt ysms" O etlance learniitg 22.95 .82 3fllSftapproprit-tehigherl COi COLSCS and prognra£ns 2195 1.05 3Deteririne effEcrfivse use of peripherlI devitcs forsuippotting teac hing/learning 2.75 .71 3,'rhz "articipants were asked to m'ate ea,h n fnlI tat2n or ;t an /t'mth iieet usiii a po aziI zikert- y,I esra,'f (I = A'o .A`ed d 5 Very it f7ig5 Need).Journ al ef Res,ea-ch oJI fechnioIotUy iu Edit'ftioz1 1_513

gies to support learinillg. Moreover, recommendatiotns to determine thie best use oftechnology to suppoft specific learning models (distributed anid project basedlearniing) were cited by the experts.Although the investigation of specific technologies was giveni less priority,Delphli panel members dLid suggest researchl ini determining the effective UsC oftool software anid peripheral devices. Examininig the effects of digital engagemweritpractices (i.e., cigital storytelling and Th;nkqluest), tutorinig softwaae!sysrenis,and expert systeiis were also stIggested as research topics.Assessmenthlie needL to determine methods and criteria for tvaluating the efctriveness of technologyenhanuieed instrutionI, inelnciing assessnment that is miultifaected, were priority topicsincludcd in the assessment category. Lablc 6 presents the I0 rcsearch statenicrits generatedby the Delphi participants, wsith mean ratings fronm 2.95 to 4.435.'Table 60 Assessment Research Category - Round 3 Delphi ResponsesResearch NeedI:)ererminc moethods and criteria mr esaluaring tceffectiven£ess oft echnolog' -enhanced instriactioiiEniplos assessnienlt that is mantifaceted (llot ottfocused onI high-stakes testing)Letermine the effectiveness of using electronicteaching portfolios to assess pertorniance in mtciringthe VaiiOtiS standardsCreate an infrastrticture for nsing technology to slipportteaching rigorouis content and fornmatively assessstudent learning wtilo respect. to this contenitFxamine the tise of tecinology tci enhaance alternativeformls of a.ssessmentDerermine perl tiin.lnce otttct mes of studenlt use oftectthnology tools in acadermic areasDevelop appropria t e (fair, valid, reliable, cost-eifeccivc)assessmtnt C)t of'stutent use of it( F as an aic to solvingproblems and accotnpliBting tasks in all subject areasExamrine the role of non intrusive, t-cal-time assessi-l iiiof student learning in opti£11;Zin1g timne-oa-tnisk andteacher efficacyean SD) Mdt£4.35 .74 4.54.25 .71 43.8 1.0 43.65 .98 43.6 .75 433 .7 3 33.2 .89 33,.7 8') .3Validlate assessntcnts for deterinining technologicalknow ledge, skills, ant dispositionisFxainiie the use ort 1 )As for assessnietn2.9tS2.95.6s.8233Naote.hte 2Ivert;i taske;a5d to nie each? 1 Je'atl staotmeatw the Osi'/ of 'n'd uat or~ a a cojzttI tie; r-tipt' 't/c (I .Me A'] V n! a 1", i 11ioh V= t).(Current IssuiesIn this category, Delphi panel itiernbc-rs recniomniedeti that research be conIcentratecdprimiarily on digital dividte isUss, inctluditig wavs to eliminate orlesseni the efi'ccts of the digital divide. )tther issic s insclurdeti changwcs resulting54 9Winter 2704-2005: oflumte37 .Number 2

from the uLse of tcchology, iclud0ing the eltects onE social interaction anid collaboration.Tls Studenlit use of comptluters outside of' the school cday was alsosutggested as ati ares in need of further research). T'hc seven research statemneutsare listedi in T'ablbh below.'lable 7: Current Issues Research Category-Round 3 Delphi ResponsesRiescarch NeedMearin SI iMdtiDevelop strategies to eliminat or lessen the efiectot the digital diviice3.95 .82 4xattn;ne digital equiity issues ri ating to low SES,gender and cthnicits3.x65 93 4ELxanllmne the effect ol ftechtnoiog' on social inltei ractinI id coliahorationl3.6 .68 3.5i)ectsenie ksfactots irflunc ing die digital divide3.4 1.(0i 3Esxaminte human-comrnutter interactions (Ontormat11oretrievai, imonersivc environments, osabiliit, etc.) .35 1 03 3Deitertmine the ahility of -ec-hniology to Cengage stodentsitl itmportantt ac 1t5i OuitisItde of'tim 'trnttal sChoo liours 3.2 .95 3Deteruitie tilhe ffects, of eduictional technologies i1ivte itV of oric -to-otet studntidtcoEt/ puttretenvirotnme3ts (horme and school) 3.1 5 .87 3otnc Pli-artcipants w-er aIked ro nutt akc *ewt,bh starer;ient on the 'ii t7Tdt (Ag a *t y wari/kt,, 'tr cale (I =;N Au7fed ei/ad 5 = t Hry i ih J.fSchoolsJImr)le of teehitoif>logS in fiostcting schcol ifnprovetnmnt atd ultti-mlatey student le-arniogwete emphasizedin die Schools tcs, arch area (see Tlabk 8). Of ighiest prionigywasthe need to invcstidate the ro3e of tecnltiology in the cilan,gc proIcss, witb Delpht panlcxperts citing the lsed to examine ways in wvhich techot0>loog can facilitate educationaltransforttnation, assist inl assessiig student. growth, andS help in making efdective datadrivendecisions. The I)clpi patiel advsatoded researcn to ceterminfe the ccial rolc Iftechnolok, in facilitating ediucational ti-ansformation and school reforno.Table 8: Schools Research Category-Round 3 Delphi ResponsesReseatrc]h Ne\dtInivestigate changes in classrotom1, teachsF Oles, andscthools duc to technologg integrationInivestigate the es thCts OF ne.v mnodels of sctIooliUg(using technology) to des 7lop highcr-orsetr outcomesand criical attribustes (i.e., crea.tivity, problemi solving.te'allwork, etc )Eivalinle the role of teehnolovy in helpinl"g SChIoolslook at more than studenit petiortmance ini schoolirprovex'lemnt pianningiExxamoine the role of informationis techntiologsy infacilitsatirg education.ll traitsfornsationMea~n Si) :V ctii4.2 .61 44.0 t '7/5 43.5 .82 33.5 .82 3Journa of Research on lechnology in Educoation1I i55

Table 8 coni'tResearch NeedMe inExamiiine the role of technology in broadening thedefinition of Annual YearLv Progress and buildingschool 'ireport cards" that reflect student growthDcternmine the effect of adeministrative data-driveninstrructional decisions on scudenit/schoolt/classes)ectermine the best uses of computer mediatedcornmitnication among schlools to encourage andsuppoI- innovation/research3.53.53.45.88.94.83333Examniie the role of change thlicoI and models to fosterclanges in schools when ititegrating technologyDeterniine the use and effectivetness of'technologyin rmral schoolsDetertmine tie impact of admninmstrative support ontechnology implementationExaminie the role of fuinding in the support andl Luse3.43.33725 .75.97.85333of technologs 2.95 .82 3MIdniA t ote. i areuripants ern askeel to t ate raeh resemrch stateoeot on the basis o/ nreed 2sog a S-poin07tL.ikert- tpe sat ak (I - Neo Aed and 5 = 7eIa3 High redj).Web-Based EnivironmenitsITable 9 lists the Delphi panel recommendations for research to be conductedwxithin Web-based envirotmientts. Of top priority is the need to investigate ttheonline collaborative leaning process in an effort to develop collaborative tools tosupport that proccss. Research to determiinc efctive online instructional miiodelsand the effects of online instruction on studcnt Icarning were advocated as well.Table 9: Web-Based Environments Researchi Category-Round 3 DelphiResponsesResearclh NeedInvestigate the online collaborative learning processExamrine the challetiges faccd in nIovinig frons tcachercentrictransfer mnodel of learnintg to the design of rich,NVeb-based learning- ccntered environmenitsDevelop collaborative tools to support collaborativeprocessesLxamine the role anti( struccture of effective onlinenmentoringExamine the integration of modcls of instructioni intoWeb- based instructional strategiesT)eselop models for online instructioniDetermine thie effect of virttial classrooms on stidenltleai ningDetermine the imipact of c-learn1ing on educationalprogramsS1:)Mcan SI)3.-)5 .683.85 .74 >13.8 .89 43.7 .86 4Mdn3.65 .74 3.53.55 .68 43.55 .

Table 9 con'tRescarcih Nerd Meati SD) Mda'fetetrEne the effectcs o coriecrtig at-risk studentsswith advocates who care aboutl and commsuniratewithl the stldent electrorneically and tlOt face-to-fact 3.2 .61 3L`xanr-ie the role of video literacy tC) suppof thereplacemenit of curr-tt text-based Internietcomni n-, Lications 3-05 .82 3N'ote. PRui ticiants e, re asked1 to air,i each reer k stear,'nen,t on the basis of 'ied cZ- ng a 5-poinI Lert- )e .s,cale f A( N t 'i ed tN{ = I ef lig'h ALeeel)External InaflueeicesAlthough iexternal influences weitc recogni'ied as having an effect onl thle use oftechnolo,,y in the scliools, ther, -wcre few ar-eas considered as priority for researcch.able I0 lists the recommendations, whi icih receiv d meam, ratings of1.65 to 3.A, the loswest ratitigs of all the researcli categorics.Table 10 External Influenceis Research C.ategory-Round 3 Delphi ResponsesResearch Need Meani SI) MdnP sarnine the irntpat of Fdeeral and state policv on theuse oftcchnology for teaching and le.irning 3.4 1.0(9 4Identifv uses of the internet to iipr(ove eduscational-eformii and [QA .Are word/s missing here?) 3 .2 .89) 3Drvelop policies to tacilihate comipat ihility andintcropcrabilitv hetwseen state accr rnirab0iitysystems practices 2.75 1.02 3De)terminr the impiact of top down approaches, atthe tesderal and state levels, that lead to appropriateuse of CT( I (Inforrnmation and (CrnmunuicationTechiology) in school's 2.; .830 3Define and deterrnine supportD fnr rechnology literacy 1.65 .81 2f.ote. lPrttic zver 'ipan aikeel to vlt' e,7.h research statellent on the bacs of ned tiinfig7 5-PointL ikert- pie scale (I = No Need and S = Veriy lkigl., AVeem).Highest P1riority Research ActivitiesAn examination of the staterments included w-ithini each of the eight researchpriority categories reveals six speitic research activities considered is the greatestpriority. Thiese six researchi statements, as linsed in "Iable I 1, received a mieaiof over 4.0, placing them in a very higih-need rating.SUMMARYA review of eduecational technology literatuEe over the past three decades revealsa proliferationi of resear. h articles and national reports detailing the effectsof computer technology in the classroomn. In the l970s and '80s the majority ofresearchi efforts examiinled the effects of particular technologies or software onlstLudenit learniing (rtypically evidenrced bv studentL scores on a particular subject-Journael of Researclh oni Technwolevy in Eklucafion 157

Table I1: Specific Research Statemnernts of Highiest Prioritv - Round 3Delphi ResponsesResearch Need MCan S) MdrnInnvestigate the impact of technology on stuto iachievement 4i47 .69 5Determine methods anti criteria for evaluating theeffectiven ess of technology enhawced instruction 4 35 .74 4 .5lmploy assessrnenit htiat is multifaceted 4.25 .'1 4investigate changes in classroorn, tcaclher roles, antischos)Is dee to technology integration 4.2 .61 4Develop and LIsc scarniig principles to dcsigntechnology-cilieanced learning enviroitrmcnts r. 10 .7: 4Dcvelop models for preparing inservicc and pseservicetcaceicls to be more cfeeCtive uscrs Of ccclilology 4.05 .68 4A'otri. Rlortiants zjaLike rt-t pe scale ,e ie asked to ro r,te each i' sear'h satzteno,rt On- No Need etnd S b-Very /ikb \iedrA basis b a! o ' a Ig/Otrelated tcst), student attitudes, and attendance (I lon0ey, Culp, & (Carrigg, 200')0;Robiyer, 1988). The '90s brought about scOmc changes: the etCUs began3 to shiftfronii research oni specific kinrds of tecthnology anid thIeir cffcts on student leartlitgto a broader exarinlationi of the effects of techttology oni the analytical andcreative abilities of students. Nioreover, nIew graplic-ricih interactive techlnoltogieshave helped shape the direction of research efforts, withi researchers investigatirigall facets of the oiitinc learning etvixironnenit antid tle opporturtities providedby the Internet for teaching and learning.For the last 20 years, governmncnt-feninded policy reports havc repeatedly iden--tified the r"ee' for research on the cffcct of'educationial techinology cii teaehing.leairning, and schools to substantiate increased techinology fullding. Recclit reports((ulnp et al., 2003) advocate tc need to "establish a dcilinitioni of conditionsfor effcctive use of technology: create nlew measures of progress and indicatorsof effective use; atid design new approaches to) assessrnent and iioresensitive cvaluitiots tools" (p. 16). TPhese research pirioirities were identified bythe Delphi pancl mcmbers as well.'Ilihe Tea/chers' iko/l for tmle 21 C'entruy eport (Nationial 'enter fo0 l1diucatiotsStatistics. 2000) espoused nine research cjuestiorns to guide educationaltechnology researchi effcrts. Amlitrng tEle research focuis areas were the needto determine types of technologies avail'sble in schools, changes to enableincreased use of technology, fiscal expendittures on teclnology at all lcvels,and the beniefit of technology in tcrnis of costs. Althoutgh the f.)clphi panldnienfibers did not address these ioore monctary issucs, they' did echo thereport's stated neced for research in techelenology as it relates to leatrning,tcacher training, assessncent, anti the effect of techriology on schlooIs antdeducational reform.Spectilcallys thc members of the Delphi pancl ac;vocated a research agendathiat incltudes ani exanination of the followving areas:158'Winter 20)4-2005: Volumine 37 Number 2

1. Learning examirli the relationship of technology and ohow pcoplc leatrn inclhiding an investigatilon of the leart1iimg process, leariner engagement, andcontexttial leartl-ing.2. Teachers--dcevclop modtels for preparing inservice 'and preservicc teachlers 10be miore effectiVe Ltsirs of technology.3. lodels/S&traegies----devx lop tCchnology-rich instructional n2odels to support studcnrt leai niig in the classroom and in) the online emivironnielit,4. Assessment-"develop appropriate miethods anii criteria for evaluating theeffectiveness of'technology-ehbamnced instruc'ion, particularly for moreconipiex learnitig tasks.5. Schools ine'i stigate CJ usgcs in Lhe classroom, teachicr roles, atd scboo 0 scluCe tO technology integration and determiine ioV. technology might bestfacilitate educational rieforn.6. Social ssues-investigate factors inifluencing tIle digital divide atid t! e efects of techlniology on social interaction anld collaboration.IThc neicd litr enipirical research dcterminig the eifects of techliiology Oti teachin3g anmdlearning has been wdli tecognizd. 1 he meEnmbers of tEIC IDJelil panel in thlis nationalstudy tot only supportedi this premise. btit advocated that theme be a "rigoroits docunientationof the liik etvwcven techlt30oog, usc and learning. Given the complex nature oftec hoclc,_);, thcy agreed that there shotild be a complex antI bin ded approach to rcsearchthat secks unidetstanding asout a brsad rmngc of factors. Moreover, thei panel membtersrecot-tllended that educationlil technology research niethodologs iiclude the tue ofS iotgitudirlJand nii-xcd-inethoods rcsearch involving teachers and classtooriss. T'he tiSe ofrntlutiple methodologies and a triangulation of findings in educational research Will providcthe "rigorotls documentation" acdvocated by paane'l. mlenibers. Furtherniorec. dlestltsof this studc providle specific research priority aresas uand topics for those engaged ineducdtionsal tcchnology.Constrisbuto rs(Coustanice Pollard is a protessot' itl the Ieparttmlet co' f,dtieatonal tchnology at Boise State Utniversity. (Address' (,onstance Pollard, Phi), Departiment ofEducationlal leclhnologn4, Botse State IUniversitr, 1910 1Univecsity) Drive, Boise,Ii) 83725; epohltrY oiscssat'.c.edmsRichard PollatndI is a professor in the kacher Ed ducation departient u at theUnixversity of Idaho. (Addtrss: Richard Polliarsi PhD, Jlacher Edi crsration, lJniversitvofliatio, B(cisc ( Centt, 800 Park Blved. Boise, T.) 83 712;rpoHal>>rd(utlidaho(.cdult.)ReferencesAnIderson, 1 & RKanutk., F. (22003). e- Rflevse'c' AMero&,nd sluGZis adOnd isuet".kostatn: Alslv ailtid acuon.(",opt- R. fiR (9'81).Iducatiotn 1990: A delphi study of possibl,e futuire piobleCnSor issuCs FOE polMlic dt" atton its Missosuri (Doctoral disscriation, L'ioiveisityof Missouri, 1981). D)is.'tatioi Abrv. -trt, ,Lo-u nadtioe , 43(06), 61 A.mournatl oj Res ech; i oT Ie'hnology in Ednieation01 159

What thye inost cUrfent resear'h hozs to si/j. Santa Monica, CA: NMilkeii Exchangeon Education Technology. .vailable ornline at httU://wxsw nifforg/ptbs/ME1 i .pdfU.S. Dipartiiert of Edueation. (2000). Ilh nbtioazl techno/o(y iediaonp/an. e-tLearnhii/g: Puttinyza wont 1 c/lass edhucation at the fainge;tips o cf/ichilGWId;n.sVashilngton, DC: U .S. (overniment Printing Ofilee. Available: http://www.ed .gov/ahotLt/offices/list/os/technologv/re)otrts/e-learning.pdf1; TS. )eparttment of Fdutation. (2004). Pripalemig tonrIo J te,ie'he;s to inctechnology progranti (P1' ). Available: http:/!lwwv.ed.gov/p)rogramus/tcaehteehiindex.hltmlVan tic yen. A. I . & I)elb(cq, A.H. (l 974). T he effeerivet ess of nontinal,delphi, and intecracting group decision nidckisng processes. Acne/i inj' osVlWanage-605-621.WNaxuan, H.-., ( onunil, M. & ( tray. . (2002). A quantitative snilthesis ofrecent research on the effects of teaching and learning with teehnologv on stLi-ft/ent]joiura/, 17(4),Culp, K. M., Honrc -M., & Mandinach. L. (2003). A ret rospeCtive on twottyyears ofedu-ecation Lecbno/ogypo/ic. 'Washington, l)(:: Education Development(enter t(r Children antdT Technology. A:vailable: http://w-\Fzwv.naitiotaledtechplan.org/docs-and_pdf/20yearsdocrcvVised..pifivphert, F., & Cant, WX. (1971). The delphi tcchnique: A case study. PhiL)e/to Jvappan, 42, 272- 27.Delbecq, A. L., Vani dc Venl, A. I., & (iustafison, 1). Hi. (1Y95). (Goup techniquesforprt1ornnzp/annln-in A guidz`e to non/inalgroup and dl/phi proceisus(icliview. II: Scott, loresmani, and (Conipany,.Fouts, J. TI (2000). Research on cornputers and Ce/Ucalion: Pest, prese1nt an t!/4-ture. Bill and Melinda (ates Ioundatiori. Available: http://urwwwesd1189.org/tlp/irnages/TotalReport3.pdfHoney, NM., (Culp K., & Carrigg, E (2000). Perspectives ot tecEhnology andceducation research: L cssons fi-omi the past and present. Jocn;rnal af Ldariona/(Go;nputiugJ?nesearch, 2,3(1), 5-14.iKulik, . A (I 994). Meta-analytic studies o findinlgs on computerizcd instruction.In E. Baker & H. (O'Neil (Ects). ltc/mUoIogy asscssjen,t in ediuclii'oni/ aidt/,in/ing (pp. 9 33) Hillsdale, NJ: LaawrenLe Elribaumll Associates.I instone, H. A., & ituroff, MI. (1978). 71fe dtelph/ method.. Reading, MAA:Addisoni-Wesley Publis}ling Cornmpany.National Center for EduLcation Statistics. (2000). leachers tools fin the 2]- eentury:A ieport on teachers' use o/teihnol/ogy lVC(;7. 2000-102 W'OCslhimngtoll, )C:U.S. Department of Edtceation. AVxailablc: httpl)://uiecs.ed.gu:v!pubseareh/pu-bsinfo.asp?ptubid=200010G 2Robhvcr, M . 13. (1988). rhe ecflketivencss of nicrocompLuters in education: Ar viewv of the rese arch fiom 1980- 1987. TeubiolzoAicl Horizons in [d.tucation,16(2), 85-89.Schacter, J. (200 1). The izniact of editucatioi/ tec/n-olocy on7 stndent utchiei;uent:det itOtotcOtes. North Cenitral Regiont 7'Cducation, Laboratory. Available: http://hwww.necrel.org/teclh/e'ffects/160s1Winter 200 -2005. Volcunl e .31 Number 2

COPYRIGHT INFORMATIONTITLE: Research Priorities in Educational Technology: A DelphiStudySOURCE: J Res Technol Educ 37 no2 Wint 2004/2005WN: 0436007424003The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited. To contact the publisher:http://www.iste.org/Copyright 1982-2004 The H.W. Wilson Company. All rights reserved.