View Research - Promethean Technology in Education, Classroom ...

Executive SummaryA Randomized Controlled Trial of the Effects of Promethean ActivExpression onStudent AchievementPrepared by Marzano Research LaboratoryforPromethean, Ltd.December 2012MARZANO RESEARCH LABORATORYwww.MarzanoResearch.comPhone: 888-849-0851 | Fax: 866-801-1447

OUR MISSION OUR VISION OUR GOALTo provide the best research, themost useful actions, and the highestlevel of services to educators.To continuously develop tools thattranslate high quality educationalresearch into practical applicationseducators can put to immediateuse.To be the place educators go for thelatest information and data,synthesized into clear, conciseresources that facilitate immediateaction.REPORT AUTHORMark W. Haystead & Robert J. MarzanoFor citation purposes, please refer to this document as:Haystead, M. W., & Marzano, R. J. (2012). Executive summary: A randomized controlled trial of the effects ofPromethean ActivExpression on student achievement. Centennial, CO: Marzano Research Laboratory.Copyright © 2012. All rights reserved. No part of this publication may be reproduced or transmitted in any form orby any means without permission from Promethean, Ltd. http://www.prometheanworld.com.BUSINESS DEVELOPMENT OFFICE555 N. Morton StreetBloomington, IN 47404Phone: 888-849-0851Fax: 866-801-1447RESEARCH CENTER9000 E. Nichols Ave. Ste. 112Centennial, CO 80112Phone: 303-766-9199Fax: 303-694-1778www.MarzanoResearch.com

During the 2008/2009 school year, Marzano Research Laboratory (MRL) was commissioned byPromethean Ltd. to conduct an evaluation study of the effects of Promethean ActivClassroom tools andtechnologies on student academic achievement. Those findings were reported in Final Report: AnEvaluation Study of the Effects of Promethean ActivClassroom on Student Achievement (Marzano &Haystead, 2009). The findings from a second year evaluation study conducted during the 2009/2010school year were reported in Final Report: A Second Year Evaluation Study of PrometheanActivClassroom (Marzano & Haystead, 2010a). As their titles indicate, both reports focused on theeffects of the suite of Promethean ActivClassroom tools and technologies on student achievement.During the 2009/2010 school year, MRL was also commissioned by Promethean Ltd. to conduct a studyon the unique effects of the Promethean ActivExpression Learner Response System on studentachievement. This involved a re-analysis of the data collected from the first year study ofActivClassroom along with an analysis of new treatment-control studies conducted solely for thispurpose. Those findings were reported in Final Report: An Evaluation Study of PrometheanActivExpression Learner Response System (Marzano & Haystead, 2010b).During the 2010/2011 and 2011/2012 school years, MRL was again commissioned by Promethean Ltd.to further examine the effects of Promethean ActivExpression on student achievement (hereafterreferred to as the ActivExpression study). The ActivExpression Study was conducted in two phases.Phase I involved an analysis of student learning with and without Promethean ActivExpression. Phase IIinvolved an analysis of student learning with and without Promethean ActivExpression related toteacher behaviors as evidenced in video recordings of teachers using the technology in their classrooms.In the previous evaluation studies, each treatment-control study utilized a quasi-experimental researchdesign. Teachers acted as their own control and collected pretest and posttest scores for two groups ofstudents—one treatment and one control. The pretest scores were used as a covariate in an analysis ofcovariance (ANCOVA) to statistically equate the students and partially control for differing levels ofbackground knowledge and skill. The current study sought to strengthen internal validity by employinga form of random assignment. As before, teachers acted as their own control and collected pretest andposttest scores from two groups of students. However, instead of using two intact classrooms, teacherswere asked to split a single class into two groups—Group 1 and Group 2. For example, a teacher mighthave used their class list to assign the first student to Group 1, the second student to Group 2, the thirdstudent to Group 1, the fourth student to Group 2, and so on. Additionally, teachers were arbitrarilyassigned by MRL to one of two groups—Group A or Group B. The first teacher on the master list ofpotential participants was assigned to Group A, the second teacher was assigned to Group B, the thirdteacher was assigned to Group A, the fourth teacher was assigned to Group B, and so on.Regardless of the group assignment, participating teachers were asked to select a specific unit ofinstruction on a single topic (hereafter referred to as the unit) to be administered to their students duringa typical “contact hour” (or class period) as determined by their schedule. In addition, teachers wereasked to design a comprehensive test to be given to their students as the pretest and posttest. Thepurpose of the test was to determine what their students knew about the topic before and afterinstruction. On Day 1, teachers in both groups were asked to administer the pretest to the entire class.Executive Summary 1 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012

Then, teachers in Group A were asked to teach the unit to students in Group 1 with ActivExpression(treatment group in Group A) and teachers in Group B were asked to teach the unit to students in Group1 without ActivExpression (control group in Group B). On Day 2, teachers in Group A were asked toteach the unit to students in Group 2 without ActivExpression (control group in Group A) and teachersin Group B were asked to teach the unit to students in Group 2 with ActivExpression (treatment group inGroup B). (For detailed directions provided to teachers, see Haystead & Marzano, 2012, Appendix A).Instructional activities in both groups were to be as similar as possible except for the fact thatPromethean ActivExpression was used in the treatment group only. In other words, teachers were askedto use ActivExpression with the students in the treatment group to check for student comprehension ofthe content presented and to help maintain student engagement. With the control group students,teachers were asked to use nontechnology-based methods (e.g., raising hands, choral response) to checkfor comprehension and to help maintain engagement. In both groups, other Promethean ActivClassroomtools and technologies were used to augment instructional practices. Specific information regarding thePromethean ActivExpression Learner Response System and other ActivClassroom tools available foruse in the classroom can be found at Promethean’s website (see http://www.prometheanworld.com).Phase I of the ActivExpression study attempted to answer the following question through a metaanalysisof effect sizes from the treatment-control studies:Evaluation Question 1: What effect does Promethean ActivExpression have on students’achievement regarding the subject matter content taught by their teachers?Evaluation Question 2: Does the effect of Promethean ActivExpression differ based on gradelevel, academic content area, and instruction sequence?In addition, the following questions were considered through a correlation analysis of effect sizes andteacher and student survey responses:Evaluation Question 3: Is there a relationship between the effect of Promethean ActivExpressionand teacher survey responses related to age, experience, confidence, frequency, and professionaldevelopment?Evaluation Question 4: Is there a relationship between the effect of Promethean ActivExpressionand treatment group student survey responses related to enjoyment, attention, frequency,learning, and effort?The weighted mean effect size for the use of Promethean ActivExpression in 73 treatment-controlstudies was d = 0.01, p = .904, 95% CI [−0.12, 0.14]. (Unless otherwise noted, two-tailed significancetests were employed.)It should be noted that teachers were asked to submit their students’ answers for the purpose ofcalculating Cronbach’s alpha (a measure of the internal consistency of responses) as an estimate of thereliability of the teacher-generated assessments. Cronbach’s alpha was negative for 7 treatment-controlstudies (it is important to note that the effect sizes were also negative). Rather than exclude their datafrom the meta-analysis, the lowest positive value in the distribution (Cronbach’s alpha = 0.05) was usedExecutive Summary 2 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012

as an estimate of the low reliability of the teacher-generated assessments in the 7 treatment-controlstudies. In addition, there was insufficient data to calculate Cronbach’s alpha for 14 treatment-controlstudies. The mean value of the revised distribution (Cronbach’s alpha = 0.53) was used as an estimate ofreliability of the teacher-generated assessments in the 14 studies.When the effect sizes from the treatment-control studies were corrected for attenuation due tounreliability of the dependent measure and then combined into a more accurate estimate of effect size,the corrected weighted mean effect size was d c = 0.04, p = .672, 95% CI [−0.14, 0.22], k = 73. Four ofthe corrected effect sizes were found to be potential statistical outliers—one positive and three negative.When the four effect sizes were replaced with less extreme values, the corrected weighted mean effectsize increased slightly to d c = 0.05, p = .590, 95% CI [−0.13, 0.22], k = 73.It is worth noting that the corrected weighted mean effect size reported by Marzano and Haystead(2010a) for the use of Promethean ActivClassroom tools and technologies was d c = 0.41, p < .001,95% CI [0.29, 0.56], k = 131. In that study, teachers used Promethean ActivClassroom with theirtreatment group students to augment their instructional practices. In contrast, teachers usednontechnology strategies and materials to facilitate instruction with their control group students withoutthe use of Promethean ActivClassroom. Participating teachers were asked to report the PrometheanLearner Response System that was available for use in their classrooms (e.g., ActiVote,ActivExpression). Effect sizes from 14 treatment-control studies were found in which no learnerresponse system was reported on the teacher survey. When corrected for attenuation, the weighted meaneffect size for those treatment-control studies was d c = 0.39, p = .028, 95% CI [0.04, 0.74], k = 14.Therefore, the use of Promethean ActivClassroom tools—without ActivExpression—was associatedwith a 15.2 percentile point gain over what was expected when teachers did not use PrometheanActivClassroom. The findings of the ActivExpression study (d = 0.01, d c = 0.05) would suggest anadditional slight increase in student academic achievement when Promethean ActivExpression is usedalong with other ActivClassroom tools.No significant mediator influence was found for the effects of Promethean ActivExpression and thefollowing variables: grade level, academic content area, and instruction sequence. These findings implythat grade level, academic content area, and instruction sequence were not significant contributors to thevariance between the 73 effect sizes.No significant correlations were found between the corrected effect sizes associated with the use ofPromethean ActivExpression and the following teacher survey response variables: age (years), amountof experience as a classroom teacher (years), amount of experience with learner response systems(years), self-reported confidence in the ability to use Promethean ActivExpression, how oftenPromethean ActivExpression is used during a class period, and the amount of professional developmenttraining on the use of Promethean ActivExpression. Taken at face value, a reasonable inference can bemade that Promethean ActivExpression can be effective for teachers regardless of their age, theiramount of experience as a classroom teacher, their amount of experience with learner response systems,their confidence in their ability to use Promethean ActivExpression, how often they use PrometheanActivExpression during a class period, or the amount of professional development training they receiveon the use of Promethean ActivExpression.Executive Summary 3 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012

No significant correlations were found between the corrected effect sizes associated with the use ofPromethean ActivExpression and the mean treatment group student survey responses related to thefollowing variables: enjoyment, attention, frequency, learning, and effort. Taken at face value, areasonable inference can be made that Promethean ActivExpression can be effective for teachersregardless of student perceptions of enjoyment, attention, frequency, learning, and effort pertaining totheir teachers’ use of ActivExpression.Again, Phase II involved an analysis of student learning with and without Promethean ActivExpressionfrom the perspective of teacher behavior as evidenced in video recordings of teachers using PrometheanActivExpression along with other ActivClassroom tools in their classrooms. Teachers in theActivExpression Study were asked to submit a video recording of one class period in which they usedPromethean ActivExpression along with other ActivClassroom tools (i.e., treatment group) and oneclass period in which they did not use Promethean ActivExpression along with other ActivClassroomtools (i.e., control group). This allowed for the examination of teachers’ classroom behaviors andpedagogical skills that were not available without the video recordings. The general focus of Phase IIwas to determine the behaviors and skills that differentiated those teachers who obtained positive effectsfrom Promethean ActivExpression from those who did not.In the previous evaluation study on the effects of Promethean ActivClassroom tools and technologies,Marzano and Haystead (2010a) reported significant medium to large correlations (ranging from r = .29to r = .73) between the corrected effect sizes from the treatment-control studies and teachers’observation scores (ranging from 1 to 7) for the following variables:• student engagement – high scores on this variable are exhibited by students consistentlyattending to the activities orchestrated by the teacher• learner response systems – high scores on this variable are indicated when teachers presentstudents with questions and have each student vote on the questions• nonlinguistic representation – high scores on this variable are indicated by use of pictures,pictographs, graphic organizers and the like to present new information• previewing – high scores on this variable are manifested by teachers beginning a lesson withactivities like making linkages with what students already know about the topic, providing sometype of advance organizer for the content, etc.• chunking – high scores on this variable are indicated by teachers presenting content in“digestible bites” and then stopping so that students can process the content• scaffolding – high scores on this variable are manifest as content being organized in a clear,logical progression• pacing – high scores on this variable are manifest as the teacher slowing down or speeding up apresentation in reaction to students’ needs and engagement levels• monitoring – high scores on this variable are manifest as the teacher assessing students (formallyor informally) and using the results to alter instruction• clarity – high scores on this variable are manifest as the teacher keeping each flipchart free ofdistracting content and images or the teacher using highlighting features to mark the criticalversus peripheral contentExecutive Summary 4 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012

• student interaction – high scores on this variable are indicated when the teacher includesactivities like students answering questions in groups, students summarizing new content ingroups, students making predictions in groups, etc.• student response rates – high scores on this variable manifest as multiple students responding toteacher questions (as opposed to a small group of students) and responses being elicited by theteacher frequently throughout a class period• classroom management – high scores on this variable are manifest as teachers having clear rulesand procedures and responding appropriately to classroom interruptions and violations of rulesand proceduresA preliminary analysis of video recordings from 20 teachers revealed no significant difference (i.e.,p > .05) between the treatment and control group mean observation scores for the 12 variables.Therefore, Phase II of the ActivExpression study sought to answer the following questions:Evaluation Question 1: What is the relationship between the extent to which teachers usePromethean ActivExpression features to query students and positive versus negative effects forthe use of Promethean ActivExpression?Evaluation Question 2: What is the relationship between the extent to which teachers andstudents interact with each other about the answers provided by students using PrometheanActivExpression and positive versus negative effects for the use of PrometheanActivExpression?Two types of analyses were performed for the two evaluation questions. It should be noted that, prior toconducting the analyses, the corrected effect sizes were used to place the treatment-control studies intofour groups based on the quartiles within the distribution of effect sizes. The treatment-control studieswithin the first and fourth quartiles were selected for the analyses. In the first analysis, treatment-controlstudies were separated into two groups—teachers with positive effects and teachers with negative effectsfor their use of Promethean ActivExpression. Membership in these groups was considered theindependent variable and observation scores for the variables described by the two Phase II evaluationquestions were considered the dependent variables. An independent-samples t test was conductedcomparing the mean observation score from the group with positive effects and the mean observationscore from the group with negative effects. In the second analysis, observation scores for the two PhaseII variables were correlated with the corrected effect sizes from Phase I (with outliers adjusted to lessextreme values).The difference between mean observation scores for Evaluation Question 1 was statistically significantat the standard alpha of .05 (i.e., p < .05) and favored teachers with positive effects for their use ofPromethean ActivExpression. The effect size was d = 0.76, p = .046, 95% CI [0.04, 1.49] with apercentile gain of 27.6 points. The percentile gain indicates that the teacher with the mean observationscore in the positive-effects group (i.e., the teacher at the 50 th percentile in the distribution) scoredhigher than 77.6% of the teachers in the negative-effects group.Executive Summary 5 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012

The difference between the mean observation scores for Evaluation Question 2 was also significant andfavored teachers with positive effects for their use of Promethean ActivExpression. The effect size wasd = 2.07, p < .001, 95% CI [1.21, 2.93] with a percentile gain of 48.0 points. The percentile gainindicates that the teacher with the mean observation score in the positive-effects group (i.e., the teacherat the 50 th percentile in the distribution) scored higher than 98.0% of the teachers in the negative-effectsgroup.Statistically significant correlations were also found between the observation scores and corrected effectsizes (with outliers adjusted to less extreme values) for Evaluation Question 1, r(30) = .41, p = .018, andEvaluation Question 2, r(30) = .77, p < .001.Taken at face value, these findings might suggest a moderate effect on student achievement whenteachers query students and have students provide a short written response using the text feature ofPromethean ActivExpression, and an even stronger effect when teachers and students interact with eachother about the answers provided by students using the text feature of Promethean ActivExpression.ReferencesHaystead, M. W., & Marzano, R. J. (2012). Final report: A randomized controlled trial of the effects ofPromethean ActivExpression on student achievement. Centennial, CO: Marzano ResearchLaboratory.Marzano, R. J., & Haystead, M. W. (2009). Final report: An evaluation study of the effects ofPromethean ActivClassroom on student achievement. Centennial, CO: Marzano ResearchLaboratory.Marzano, R. J., & Haystead, M. W. (2010a). Final report: A second year evaluation study ofPromethean ActivClassroom. Centennial, CO: Marzano Research Laboratory.Marzano, R. J., & Haystead, M. W. (2010b). Final report: An evaluation study of PrometheanActivExpression Learner Response System. Centennial, CO: Marzano Research LaboratoryExecutive Summary 6 Marzano Research Laboratory2011/2012 Randomized Controlled Trial of Promethean ActivExpression December 2012