View Research - Promethean Technology in Education, Classroom ...
View Research - Promethean Technology in Education, Classroom ...
View Research - Promethean Technology in Education, Classroom ...
- No tags were found...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
Executive SummaryA Randomized Controlled Trial of the Effects of <strong>Promethean</strong> ActivExpression onStudent AchievementPrepared by Marzano <strong>Research</strong> Laboratoryfor<strong>Promethean</strong>, Ltd.December 2012MARZANO RESEARCH LABORATORYwww.Marzano<strong>Research</strong>.comPhone: 888-849-0851 | Fax: 866-801-1447
OUR MISSION OUR VISION OUR GOALTo provide the best research, themost useful actions, and the highestlevel of services to educators.To cont<strong>in</strong>uously develop tools thattranslate high quality educationalresearch <strong>in</strong>to practical applicationseducators can put to immediateuse.To be the place educators go for thelatest <strong>in</strong>formation and data,synthesized <strong>in</strong>to clear, conciseresources that facilitate immediateaction.REPORT AUTHORMark W. Haystead & Robert J. MarzanoFor citation purposes, please refer to this document as:Haystead, M. W., & Marzano, R. J. (2012). Executive summary: A randomized controlled trial of the effects of<strong>Promethean</strong> ActivExpression on student achievement. Centennial, CO: Marzano <strong>Research</strong> Laboratory.Copyright © 2012. All rights reserved. No part of this publication may be reproduced or transmitted <strong>in</strong> any form orby any means without permission from <strong>Promethean</strong>, Ltd. http://www.prometheanworld.com.BUSINESS DEVELOPMENT OFFICE555 N. Morton StreetBloom<strong>in</strong>gton, IN 47404Phone: 888-849-0851Fax: 866-801-1447RESEARCH CENTER9000 E. Nichols Ave. Ste. 112Centennial, CO 80112Phone: 303-766-9199Fax: 303-694-1778www.Marzano<strong>Research</strong>.com
Dur<strong>in</strong>g the 2008/2009 school year, Marzano <strong>Research</strong> Laboratory (MRL) was commissioned by<strong>Promethean</strong> Ltd. to conduct an evaluation study of the effects of <strong>Promethean</strong> Activ<strong>Classroom</strong> tools andtechnologies on student academic achievement. Those f<strong>in</strong>d<strong>in</strong>gs were reported <strong>in</strong> F<strong>in</strong>al Report: AnEvaluation Study of the Effects of <strong>Promethean</strong> Activ<strong>Classroom</strong> on Student Achievement (Marzano &Haystead, 2009). The f<strong>in</strong>d<strong>in</strong>gs from a second year evaluation study conducted dur<strong>in</strong>g the 2009/2010school year were reported <strong>in</strong> F<strong>in</strong>al Report: A Second Year Evaluation Study of <strong>Promethean</strong>Activ<strong>Classroom</strong> (Marzano & Haystead, 2010a). As their titles <strong>in</strong>dicate, both reports focused on theeffects of the suite of <strong>Promethean</strong> Activ<strong>Classroom</strong> tools and technologies on student achievement.Dur<strong>in</strong>g the 2009/2010 school year, MRL was also commissioned by <strong>Promethean</strong> Ltd. to conduct a studyon the unique effects of the <strong>Promethean</strong> ActivExpression Learner Response System on studentachievement. This <strong>in</strong>volved a re-analysis of the data collected from the first year study ofActiv<strong>Classroom</strong> along with an analysis of new treatment-control studies conducted solely for thispurpose. Those f<strong>in</strong>d<strong>in</strong>gs were reported <strong>in</strong> F<strong>in</strong>al Report: An Evaluation Study of <strong>Promethean</strong>ActivExpression Learner Response System (Marzano & Haystead, 2010b).Dur<strong>in</strong>g the 2010/2011 and 2011/2012 school years, MRL was aga<strong>in</strong> commissioned by <strong>Promethean</strong> Ltd.to further exam<strong>in</strong>e the effects of <strong>Promethean</strong> ActivExpression on student achievement (hereafterreferred to as the ActivExpression study). The ActivExpression Study was conducted <strong>in</strong> two phases.Phase I <strong>in</strong>volved an analysis of student learn<strong>in</strong>g with and without <strong>Promethean</strong> ActivExpression. Phase II<strong>in</strong>volved an analysis of student learn<strong>in</strong>g with and without <strong>Promethean</strong> ActivExpression related toteacher behaviors as evidenced <strong>in</strong> video record<strong>in</strong>gs of teachers us<strong>in</strong>g the technology <strong>in</strong> their classrooms.In the previous evaluation studies, each treatment-control study utilized a quasi-experimental researchdesign. Teachers acted as their own control and collected pretest and posttest scores for two groups ofstudents—one treatment and one control. The pretest scores were used as a covariate <strong>in</strong> an analysis ofcovariance (ANCOVA) to statistically equate the students and partially control for differ<strong>in</strong>g levels ofbackground knowledge and skill. The current study sought to strengthen <strong>in</strong>ternal validity by employ<strong>in</strong>ga form of random assignment. As before, teachers acted as their own control and collected pretest andposttest scores from two groups of students. However, <strong>in</strong>stead of us<strong>in</strong>g two <strong>in</strong>tact classrooms, teacherswere asked to split a s<strong>in</strong>gle class <strong>in</strong>to two groups—Group 1 and Group 2. For example, a teacher mighthave used their class list to assign the first student to Group 1, the second student to Group 2, the thirdstudent to Group 1, the fourth student to Group 2, and so on. Additionally, teachers were arbitrarilyassigned by MRL to one of two groups—Group A or Group B. The first teacher on the master list ofpotential participants was assigned to Group A, the second teacher was assigned to Group B, the thirdteacher was assigned to Group A, the fourth teacher was assigned to Group B, and so on.Regardless of the group assignment, participat<strong>in</strong>g teachers were asked to select a specific unit of<strong>in</strong>struction on a s<strong>in</strong>gle topic (hereafter referred to as the unit) to be adm<strong>in</strong>istered to their students dur<strong>in</strong>ga typical “contact hour” (or class period) as determ<strong>in</strong>ed by their schedule. In addition, teachers wereasked to design a comprehensive test to be given to their students as the pretest and posttest. Thepurpose of the test was to determ<strong>in</strong>e what their students knew about the topic before and after<strong>in</strong>struction. On Day 1, teachers <strong>in</strong> both groups were asked to adm<strong>in</strong>ister the pretest to the entire class.Executive Summary 1 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012
Then, teachers <strong>in</strong> Group A were asked to teach the unit to students <strong>in</strong> Group 1 with ActivExpression(treatment group <strong>in</strong> Group A) and teachers <strong>in</strong> Group B were asked to teach the unit to students <strong>in</strong> Group1 without ActivExpression (control group <strong>in</strong> Group B). On Day 2, teachers <strong>in</strong> Group A were asked toteach the unit to students <strong>in</strong> Group 2 without ActivExpression (control group <strong>in</strong> Group A) and teachers<strong>in</strong> Group B were asked to teach the unit to students <strong>in</strong> Group 2 with ActivExpression (treatment group <strong>in</strong>Group B). (For detailed directions provided to teachers, see Haystead & Marzano, 2012, Appendix A).Instructional activities <strong>in</strong> both groups were to be as similar as possible except for the fact that<strong>Promethean</strong> ActivExpression was used <strong>in</strong> the treatment group only. In other words, teachers were askedto use ActivExpression with the students <strong>in</strong> the treatment group to check for student comprehension ofthe content presented and to help ma<strong>in</strong>ta<strong>in</strong> student engagement. With the control group students,teachers were asked to use nontechnology-based methods (e.g., rais<strong>in</strong>g hands, choral response) to checkfor comprehension and to help ma<strong>in</strong>ta<strong>in</strong> engagement. In both groups, other <strong>Promethean</strong> Activ<strong>Classroom</strong>tools and technologies were used to augment <strong>in</strong>structional practices. Specific <strong>in</strong>formation regard<strong>in</strong>g the<strong>Promethean</strong> ActivExpression Learner Response System and other Activ<strong>Classroom</strong> tools available foruse <strong>in</strong> the classroom can be found at <strong>Promethean</strong>’s website (see http://www.prometheanworld.com).Phase I of the ActivExpression study attempted to answer the follow<strong>in</strong>g question through a metaanalysisof effect sizes from the treatment-control studies:Evaluation Question 1: What effect does <strong>Promethean</strong> ActivExpression have on students’achievement regard<strong>in</strong>g the subject matter content taught by their teachers?Evaluation Question 2: Does the effect of <strong>Promethean</strong> ActivExpression differ based on gradelevel, academic content area, and <strong>in</strong>struction sequence?In addition, the follow<strong>in</strong>g questions were considered through a correlation analysis of effect sizes andteacher and student survey responses:Evaluation Question 3: Is there a relationship between the effect of <strong>Promethean</strong> ActivExpressionand teacher survey responses related to age, experience, confidence, frequency, and professionaldevelopment?Evaluation Question 4: Is there a relationship between the effect of <strong>Promethean</strong> ActivExpressionand treatment group student survey responses related to enjoyment, attention, frequency,learn<strong>in</strong>g, and effort?The weighted mean effect size for the use of <strong>Promethean</strong> ActivExpression <strong>in</strong> 73 treatment-controlstudies was d = 0.01, p = .904, 95% CI [−0.12, 0.14]. (Unless otherwise noted, two-tailed significancetests were employed.)It should be noted that teachers were asked to submit their students’ answers for the purpose ofcalculat<strong>in</strong>g Cronbach’s alpha (a measure of the <strong>in</strong>ternal consistency of responses) as an estimate of thereliability of the teacher-generated assessments. Cronbach’s alpha was negative for 7 treatment-controlstudies (it is important to note that the effect sizes were also negative). Rather than exclude their datafrom the meta-analysis, the lowest positive value <strong>in</strong> the distribution (Cronbach’s alpha = 0.05) was usedExecutive Summary 2 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012
as an estimate of the low reliability of the teacher-generated assessments <strong>in</strong> the 7 treatment-controlstudies. In addition, there was <strong>in</strong>sufficient data to calculate Cronbach’s alpha for 14 treatment-controlstudies. The mean value of the revised distribution (Cronbach’s alpha = 0.53) was used as an estimate ofreliability of the teacher-generated assessments <strong>in</strong> the 14 studies.When the effect sizes from the treatment-control studies were corrected for attenuation due tounreliability of the dependent measure and then comb<strong>in</strong>ed <strong>in</strong>to a more accurate estimate of effect size,the corrected weighted mean effect size was d c = 0.04, p = .672, 95% CI [−0.14, 0.22], k = 73. Four ofthe corrected effect sizes were found to be potential statistical outliers—one positive and three negative.When the four effect sizes were replaced with less extreme values, the corrected weighted mean effectsize <strong>in</strong>creased slightly to d c = 0.05, p = .590, 95% CI [−0.13, 0.22], k = 73.It is worth not<strong>in</strong>g that the corrected weighted mean effect size reported by Marzano and Haystead(2010a) for the use of <strong>Promethean</strong> Activ<strong>Classroom</strong> tools and technologies was d c = 0.41, p < .001,95% CI [0.29, 0.56], k = 131. In that study, teachers used <strong>Promethean</strong> Activ<strong>Classroom</strong> with theirtreatment group students to augment their <strong>in</strong>structional practices. In contrast, teachers usednontechnology strategies and materials to facilitate <strong>in</strong>struction with their control group students withoutthe use of <strong>Promethean</strong> Activ<strong>Classroom</strong>. Participat<strong>in</strong>g teachers were asked to report the <strong>Promethean</strong>Learner Response System that was available for use <strong>in</strong> their classrooms (e.g., ActiVote,ActivExpression). Effect sizes from 14 treatment-control studies were found <strong>in</strong> which no learnerresponse system was reported on the teacher survey. When corrected for attenuation, the weighted meaneffect size for those treatment-control studies was d c = 0.39, p = .028, 95% CI [0.04, 0.74], k = 14.Therefore, the use of <strong>Promethean</strong> Activ<strong>Classroom</strong> tools—without ActivExpression—was associatedwith a 15.2 percentile po<strong>in</strong>t ga<strong>in</strong> over what was expected when teachers did not use <strong>Promethean</strong>Activ<strong>Classroom</strong>. The f<strong>in</strong>d<strong>in</strong>gs of the ActivExpression study (d = 0.01, d c = 0.05) would suggest anadditional slight <strong>in</strong>crease <strong>in</strong> student academic achievement when <strong>Promethean</strong> ActivExpression is usedalong with other Activ<strong>Classroom</strong> tools.No significant mediator <strong>in</strong>fluence was found for the effects of <strong>Promethean</strong> ActivExpression and thefollow<strong>in</strong>g variables: grade level, academic content area, and <strong>in</strong>struction sequence. These f<strong>in</strong>d<strong>in</strong>gs implythat grade level, academic content area, and <strong>in</strong>struction sequence were not significant contributors to thevariance between the 73 effect sizes.No significant correlations were found between the corrected effect sizes associated with the use of<strong>Promethean</strong> ActivExpression and the follow<strong>in</strong>g teacher survey response variables: age (years), amountof experience as a classroom teacher (years), amount of experience with learner response systems(years), self-reported confidence <strong>in</strong> the ability to use <strong>Promethean</strong> ActivExpression, how often<strong>Promethean</strong> ActivExpression is used dur<strong>in</strong>g a class period, and the amount of professional developmenttra<strong>in</strong><strong>in</strong>g on the use of <strong>Promethean</strong> ActivExpression. Taken at face value, a reasonable <strong>in</strong>ference can bemade that <strong>Promethean</strong> ActivExpression can be effective for teachers regardless of their age, theiramount of experience as a classroom teacher, their amount of experience with learner response systems,their confidence <strong>in</strong> their ability to use <strong>Promethean</strong> ActivExpression, how often they use <strong>Promethean</strong>ActivExpression dur<strong>in</strong>g a class period, or the amount of professional development tra<strong>in</strong><strong>in</strong>g they receiveon the use of <strong>Promethean</strong> ActivExpression.Executive Summary 3 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012
No significant correlations were found between the corrected effect sizes associated with the use of<strong>Promethean</strong> ActivExpression and the mean treatment group student survey responses related to thefollow<strong>in</strong>g variables: enjoyment, attention, frequency, learn<strong>in</strong>g, and effort. Taken at face value, areasonable <strong>in</strong>ference can be made that <strong>Promethean</strong> ActivExpression can be effective for teachersregardless of student perceptions of enjoyment, attention, frequency, learn<strong>in</strong>g, and effort perta<strong>in</strong><strong>in</strong>g totheir teachers’ use of ActivExpression.Aga<strong>in</strong>, Phase II <strong>in</strong>volved an analysis of student learn<strong>in</strong>g with and without <strong>Promethean</strong> ActivExpressionfrom the perspective of teacher behavior as evidenced <strong>in</strong> video record<strong>in</strong>gs of teachers us<strong>in</strong>g <strong>Promethean</strong>ActivExpression along with other Activ<strong>Classroom</strong> tools <strong>in</strong> their classrooms. Teachers <strong>in</strong> theActivExpression Study were asked to submit a video record<strong>in</strong>g of one class period <strong>in</strong> which they used<strong>Promethean</strong> ActivExpression along with other Activ<strong>Classroom</strong> tools (i.e., treatment group) and oneclass period <strong>in</strong> which they did not use <strong>Promethean</strong> ActivExpression along with other Activ<strong>Classroom</strong>tools (i.e., control group). This allowed for the exam<strong>in</strong>ation of teachers’ classroom behaviors andpedagogical skills that were not available without the video record<strong>in</strong>gs. The general focus of Phase IIwas to determ<strong>in</strong>e the behaviors and skills that differentiated those teachers who obta<strong>in</strong>ed positive effectsfrom <strong>Promethean</strong> ActivExpression from those who did not.In the previous evaluation study on the effects of <strong>Promethean</strong> Activ<strong>Classroom</strong> tools and technologies,Marzano and Haystead (2010a) reported significant medium to large correlations (rang<strong>in</strong>g from r = .29to r = .73) between the corrected effect sizes from the treatment-control studies and teachers’observation scores (rang<strong>in</strong>g from 1 to 7) for the follow<strong>in</strong>g variables:• student engagement – high scores on this variable are exhibited by students consistentlyattend<strong>in</strong>g to the activities orchestrated by the teacher• learner response systems – high scores on this variable are <strong>in</strong>dicated when teachers presentstudents with questions and have each student vote on the questions• nonl<strong>in</strong>guistic representation – high scores on this variable are <strong>in</strong>dicated by use of pictures,pictographs, graphic organizers and the like to present new <strong>in</strong>formation• preview<strong>in</strong>g – high scores on this variable are manifested by teachers beg<strong>in</strong>n<strong>in</strong>g a lesson withactivities like mak<strong>in</strong>g l<strong>in</strong>kages with what students already know about the topic, provid<strong>in</strong>g sometype of advance organizer for the content, etc.• chunk<strong>in</strong>g – high scores on this variable are <strong>in</strong>dicated by teachers present<strong>in</strong>g content <strong>in</strong>“digestible bites” and then stopp<strong>in</strong>g so that students can process the content• scaffold<strong>in</strong>g – high scores on this variable are manifest as content be<strong>in</strong>g organized <strong>in</strong> a clear,logical progression• pac<strong>in</strong>g – high scores on this variable are manifest as the teacher slow<strong>in</strong>g down or speed<strong>in</strong>g up apresentation <strong>in</strong> reaction to students’ needs and engagement levels• monitor<strong>in</strong>g – high scores on this variable are manifest as the teacher assess<strong>in</strong>g students (formallyor <strong>in</strong>formally) and us<strong>in</strong>g the results to alter <strong>in</strong>struction• clarity – high scores on this variable are manifest as the teacher keep<strong>in</strong>g each flipchart free ofdistract<strong>in</strong>g content and images or the teacher us<strong>in</strong>g highlight<strong>in</strong>g features to mark the criticalversus peripheral contentExecutive Summary 4 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012
• student <strong>in</strong>teraction – high scores on this variable are <strong>in</strong>dicated when the teacher <strong>in</strong>cludesactivities like students answer<strong>in</strong>g questions <strong>in</strong> groups, students summariz<strong>in</strong>g new content <strong>in</strong>groups, students mak<strong>in</strong>g predictions <strong>in</strong> groups, etc.• student response rates – high scores on this variable manifest as multiple students respond<strong>in</strong>g toteacher questions (as opposed to a small group of students) and responses be<strong>in</strong>g elicited by theteacher frequently throughout a class period• classroom management – high scores on this variable are manifest as teachers hav<strong>in</strong>g clear rulesand procedures and respond<strong>in</strong>g appropriately to classroom <strong>in</strong>terruptions and violations of rulesand proceduresA prelim<strong>in</strong>ary analysis of video record<strong>in</strong>gs from 20 teachers revealed no significant difference (i.e.,p > .05) between the treatment and control group mean observation scores for the 12 variables.Therefore, Phase II of the ActivExpression study sought to answer the follow<strong>in</strong>g questions:Evaluation Question 1: What is the relationship between the extent to which teachers use<strong>Promethean</strong> ActivExpression features to query students and positive versus negative effects forthe use of <strong>Promethean</strong> ActivExpression?Evaluation Question 2: What is the relationship between the extent to which teachers andstudents <strong>in</strong>teract with each other about the answers provided by students us<strong>in</strong>g <strong>Promethean</strong>ActivExpression and positive versus negative effects for the use of <strong>Promethean</strong>ActivExpression?Two types of analyses were performed for the two evaluation questions. It should be noted that, prior toconduct<strong>in</strong>g the analyses, the corrected effect sizes were used to place the treatment-control studies <strong>in</strong>tofour groups based on the quartiles with<strong>in</strong> the distribution of effect sizes. The treatment-control studieswith<strong>in</strong> the first and fourth quartiles were selected for the analyses. In the first analysis, treatment-controlstudies were separated <strong>in</strong>to two groups—teachers with positive effects and teachers with negative effectsfor their use of <strong>Promethean</strong> ActivExpression. Membership <strong>in</strong> these groups was considered the<strong>in</strong>dependent variable and observation scores for the variables described by the two Phase II evaluationquestions were considered the dependent variables. An <strong>in</strong>dependent-samples t test was conductedcompar<strong>in</strong>g the mean observation score from the group with positive effects and the mean observationscore from the group with negative effects. In the second analysis, observation scores for the two PhaseII variables were correlated with the corrected effect sizes from Phase I (with outliers adjusted to lessextreme values).The difference between mean observation scores for Evaluation Question 1 was statistically significantat the standard alpha of .05 (i.e., p < .05) and favored teachers with positive effects for their use of<strong>Promethean</strong> ActivExpression. The effect size was d = 0.76, p = .046, 95% CI [0.04, 1.49] with apercentile ga<strong>in</strong> of 27.6 po<strong>in</strong>ts. The percentile ga<strong>in</strong> <strong>in</strong>dicates that the teacher with the mean observationscore <strong>in</strong> the positive-effects group (i.e., the teacher at the 50 th percentile <strong>in</strong> the distribution) scoredhigher than 77.6% of the teachers <strong>in</strong> the negative-effects group.Executive Summary 5 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012
The difference between the mean observation scores for Evaluation Question 2 was also significant andfavored teachers with positive effects for their use of <strong>Promethean</strong> ActivExpression. The effect size wasd = 2.07, p < .001, 95% CI [1.21, 2.93] with a percentile ga<strong>in</strong> of 48.0 po<strong>in</strong>ts. The percentile ga<strong>in</strong><strong>in</strong>dicates that the teacher with the mean observation score <strong>in</strong> the positive-effects group (i.e., the teacherat the 50 th percentile <strong>in</strong> the distribution) scored higher than 98.0% of the teachers <strong>in</strong> the negative-effectsgroup.Statistically significant correlations were also found between the observation scores and corrected effectsizes (with outliers adjusted to less extreme values) for Evaluation Question 1, r(30) = .41, p = .018, andEvaluation Question 2, r(30) = .77, p < .001.Taken at face value, these f<strong>in</strong>d<strong>in</strong>gs might suggest a moderate effect on student achievement whenteachers query students and have students provide a short written response us<strong>in</strong>g the text feature of<strong>Promethean</strong> ActivExpression, and an even stronger effect when teachers and students <strong>in</strong>teract with eachother about the answers provided by students us<strong>in</strong>g the text feature of <strong>Promethean</strong> ActivExpression.ReferencesHaystead, M. W., & Marzano, R. J. (2012). F<strong>in</strong>al report: A randomized controlled trial of the effects of<strong>Promethean</strong> ActivExpression on student achievement. Centennial, CO: Marzano <strong>Research</strong>Laboratory.Marzano, R. J., & Haystead, M. W. (2009). F<strong>in</strong>al report: An evaluation study of the effects of<strong>Promethean</strong> Activ<strong>Classroom</strong> on student achievement. Centennial, CO: Marzano <strong>Research</strong>Laboratory.Marzano, R. J., & Haystead, M. W. (2010a). F<strong>in</strong>al report: A second year evaluation study of<strong>Promethean</strong> Activ<strong>Classroom</strong>. Centennial, CO: Marzano <strong>Research</strong> Laboratory.Marzano, R. J., & Haystead, M. W. (2010b). F<strong>in</strong>al report: An evaluation study of <strong>Promethean</strong>ActivExpression Learner Response System. Centennial, CO: Marzano <strong>Research</strong> LaboratoryExecutive Summary 6 Marzano <strong>Research</strong> Laboratory2011/2012 Randomized Controlled Trial of <strong>Promethean</strong> ActivExpression December 2012