Module 1: Oral feedback
Module 1: Oral feedback
Module 1: Oral feedback
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
Unit<br />
4<br />
<strong>Module</strong> 1: <strong>Oral</strong> <strong>feedback</strong><br />
Objectives<br />
• To highlight how effective oral <strong>feedback</strong> develops pupils’ learning<br />
• To consider a range of strategies for giving effective oral <strong>feedback</strong> which can<br />
be both formal and informal<br />
• To help teachers plan for giving effective oral <strong>feedback</strong><br />
• To help schools develop a whole-school policy linking and clarifying the<br />
relationship between oral and written <strong>feedback</strong><br />
Resources<br />
Slides 4.1.1–4.1.9<br />
Handouts 4.1.1–4.1.7<br />
Video sequence for this module<br />
Session outline<br />
4.1 Introduction 20 minutes<br />
4.2 Using oral <strong>feedback</strong> for different purposes 25 minutes<br />
4.3 Making the most of <strong>feedback</strong> 10 minutes<br />
4.4 Planning for oral <strong>feedback</strong> 15 minutes<br />
4.5 Ready for more? 5 minutes<br />
Total<br />
75 minutes<br />
4.1 Introduction 20 minutes<br />
Show slide 4.1.1 to introduce the objectives of the module. (5 minutes)<br />
Slide 4.1.1<br />
Explain the following.<br />
• <strong>Module</strong>s 1 and 2 of this unit, ‘<strong>Oral</strong> <strong>feedback</strong>’ and ‘Written <strong>feedback</strong>’, are<br />
closely interrelated. Both focus on improving the quality of regular<br />
1 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
constructive, informative and focused <strong>feedback</strong> in order to help pupils take<br />
the next steps in their learning and make progress towards their targets.<br />
• <strong>Module</strong> 2, ‘Written <strong>feedback</strong>’, focuses on providing high-quality, detailed<br />
and informative marking on a selective basis rather than cursory surface<br />
marking for all pupils on every piece of work. This will involve a significant<br />
change in marking policy for many departments and schools. This approach<br />
to providing written <strong>feedback</strong> cannot work unless oral <strong>feedback</strong> is detailed,<br />
personalised and routinely provides the information pupils need to<br />
make progress.<br />
• The relationship between oral and written <strong>feedback</strong> needs to be agreed and<br />
embodied in whole-school policy. It should be explicitly shared with pupils<br />
and parents so that it is clear what pupils can expect to receive.<br />
• Schools must continue to ensure that approaches to providing <strong>feedback</strong><br />
and its quality remain consistent across departments.<br />
Tell participants that:<br />
• oral <strong>feedback</strong> is a powerful force for moving pupils on and will be the most<br />
regular and interactive form of <strong>feedback</strong>;<br />
• oral <strong>feedback</strong> is direct (targeted to individuals or groups), but also indirect<br />
(others listen and reflect on what has been said);<br />
• oral <strong>feedback</strong> is implicit in all lessons. This module aims to make<br />
spontaneous oral <strong>feedback</strong> more effective and help teachers plan for oral<br />
<strong>feedback</strong> more explicitly;<br />
• effective oral <strong>feedback</strong> takes time to develop, needs to be planned and<br />
requires fostering within a supportive learning environment;<br />
• in their study of the effects of oral <strong>feedback</strong>, which examined 3000 reports<br />
covering nearly 13,000 students, Kluger and DeNisi (1996) found that oral<br />
<strong>feedback</strong> raised achievement in 60% of cases, but in 40% was ineffective.<br />
The challenge, therefore, is to identify what makes oral <strong>feedback</strong> effective.<br />
Show slide 4.1.2.<br />
Slide 4.1.2<br />
Explain that <strong>feedback</strong> works in three directions:<br />
• Teacher to pupil – teacher giving information to pupils regarding current<br />
achievement, and next steps.<br />
• Pupil to teacher – teacher receiving <strong>feedback</strong> from pupils about<br />
their learning.<br />
2 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
• Pupil to pupil – in peer assessment.<br />
All of these are important, necessary and very clearly interlinked. In offering oral<br />
<strong>feedback</strong> the teacher should model the language of <strong>feedback</strong> that pupils can<br />
then employ themselves. Point out that further support and training can be<br />
found as follows:<br />
• Improving the quality of responses gained from pupils: modules on<br />
‘Questioning’, ‘Challenge’ and ‘Plenaries’ in the Training materials for the<br />
foundation subjects (DfES 0350/2002)<br />
• Improving pupils’ oral responses: module 7, ‘The management of group<br />
talk’ in Literacy across the curriculum (DfEE 0235/2001). This deals with<br />
improving and structuring pupils’ speaking and listening skills, and suggests<br />
appropriate ways to group pupils for different types of support<br />
and interaction.<br />
• Peer and self assessment: unit 5 of this folder.<br />
Tell participants that:<br />
• although all three areas are crucial to the learning process, this module<br />
focuses on the quality of teachers’ responses to pupils at various points in<br />
the teaching and learning sequence;<br />
• this module will look at how to improve the quality of spontaneous<br />
interactive response, helping the teacher to know when and how to<br />
intervene, and how to move on, refocus and refine learning.<br />
Task A (5 minutes)<br />
Working in threes or fours, ask participants to discuss the advantages of giving<br />
oral <strong>feedback</strong> in lessons. Without showing slide 4.1.3 offer one example.<br />
Take brief <strong>feedback</strong>. Show slide 4.1.3 and use the comments below to expand<br />
on any points not covered. Note any further suggestions.<br />
Slide 4.1.3<br />
Use the following points to support <strong>feedback</strong>:<br />
• Immediate and context-specific – able to deal with misconceptions<br />
as they occur and in the context of the specific learning.<br />
• Dynamic and adaptable – allows for exchange of ideas and is adaptable<br />
to respective learning needs at the point of receiving.<br />
• Ongoing – a continuous and familiar part of a lesson.<br />
3 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
• Episodic – a lesson can be punctuated with plenary moments to structure<br />
learning with reference to the objectives threaded through the lesson.<br />
• Stimulating – oral <strong>feedback</strong> can encourage, enthuse and stimulate.<br />
• Personalised – facial expressions and body language enhance <strong>feedback</strong>.<br />
• Versatile – questioning, modelling and explaining can all be used as means<br />
of providing <strong>feedback</strong>.<br />
• Motivating – pupils like immediate oral <strong>feedback</strong> and respond easily as the<br />
learning is fresh, and quick improvements build a feeling of success.<br />
Task B (10 minutes)<br />
Introduce this task by saying that teachers need to plan carefully to make their<br />
<strong>feedback</strong> more effective and reduce the pitfalls. In this activity show slide 4.1.4.<br />
Allocate two or three bullet points to pairs or groups. Ask participants to<br />
discuss possible ways of avoiding the suggested difficulties. Allow 5 minutes for<br />
discussing the issues and suggesting some possible solutions.<br />
Slide 4.1.4<br />
Briefly take <strong>feedback</strong> and draw on handout 4.1.1, selecting some examples.<br />
Handout 4.1.1<br />
Handout 4.1.1<br />
Some solutions to possible difficulties<br />
• Instant responses from the teacher may not always be well considered or<br />
sufficiently focused on learning objectives.<br />
– Feedback should always be very clearly related to the lesson objectives and avoid<br />
comment on extraneous factors or areas; the objective needs to be fully explored at the<br />
outset of the lesson, task or episode.<br />
– Wait time before and after questions or required responses encourages pupils to consider<br />
carefully and expand on their response. Other pupils can be asked to reflect or comment<br />
on the response before the teacher responds.<br />
– Encourage pupils to use specific learning outcomes (success criteria) to assess their<br />
own or others’ responses, before or after your <strong>feedback</strong>.<br />
– Using prompts such as Would you like to say a bit more about that? gives the teacher the<br />
opportunity to see what stage of thinking the learner has reached and to fine-tune<br />
<strong>feedback</strong> as appropriate. Such teacher responses also make the process developmental.<br />
– Teachers should attempt to pre-empt possible misconceptions and issues in explanation<br />
of the task.<br />
– Teachers can use pre-emptive strategies, e.g. discuss possible pitfalls and success<br />
criteria when setting tasks.<br />
– Targeting questions or plenary activities at specific pupils can reduce the likelihood of<br />
inappropriate responses from pupils.<br />
– Teachers need to consider the style of <strong>feedback</strong> given as well as the content and its<br />
implication for the way the lesson progresses, e.g. the choice between simple<br />
acknowledgement that an answer is correct or a response that urges further<br />
amplification or exploration.<br />
• Pupils do not act on oral <strong>feedback</strong> and do not take it seriously. Other pupils do not<br />
listen to it.<br />
– It takes time to develop a supportive culture in the classroom and to establish that your<br />
own and other pupils’ errors are learning opportunities.<br />
– It takes time to establish the value and importance of giving and receiving <strong>feedback</strong>.<br />
– While focusing on specific areas of a response, be prepared to say when an answer is<br />
wrong in order to avoid confusion or reinforcing misconceptions.<br />
– Pupils need explicit training and practice in the ground rules of speaking and listening.<br />
– Ask another pupil to follow up on <strong>feedback</strong>.<br />
• Unplanned responses can become random and fail to develop pupils’ learning in a<br />
structured way.<br />
– Questions and tasks need to be planned to encourage higher-level thinking and responses<br />
so that <strong>feedback</strong> can help develop it.<br />
– Structure chains of questions and tasks incrementally so that <strong>feedback</strong> also moves<br />
learning on.<br />
– Ensure that some time is built in for more extended or structured <strong>feedback</strong> at an<br />
appropriate time.<br />
• <strong>Oral</strong> <strong>feedback</strong> is ephemeral.<br />
– This is largely true but what is important is to ensure pupils have time to respond and act<br />
on the <strong>feedback</strong> if necessary.<br />
Handout 4.1.1 cont.<br />
– It may be helpful in some instances to get pupils to reflect on the <strong>feedback</strong> and write their<br />
next steps or targets in their exercise books, or to revise their work/performance/learning<br />
behaviours in the light of <strong>feedback</strong> there and then.<br />
– Make a mental (or actual) note of pupils’ responses and return to points later to check on<br />
understanding and follow up.<br />
• Pupils may feel exposed by public <strong>feedback</strong>.<br />
– The key is to establish a supportive environment in which pupils are happy to contribute to<br />
the lesson. Share the process by encouraging pupils to talk through their thinking and<br />
identify points at which they went wrong or identify ways to improve.<br />
– Establish the idea that if you are not making mistakes you are not learning.<br />
– Create a challenge culture over time where genuine errors are perceived as positive<br />
learning opportunities, and where challenge is welcomed and pupils do not automatically<br />
choose the easiest option.<br />
– Provide opportunities to give private or small group <strong>feedback</strong>.<br />
– Establish routines for group and guided work.<br />
– Always reinforce the value of the pupils’ contribution.<br />
– On occasions, be willing to acknowledge your own errors or lack of specific knowledge.<br />
• Pupils may not have the time needed to reflect on the <strong>feedback</strong> and respond to it.<br />
– Not all responses will require reflection or adjustment; some simply confirm, acknowledge<br />
or encourage.<br />
– Provide more time by echoing the point, e.g. So what you’re saying is …<br />
– Build in time for pupils to amend work.<br />
– Build in regular structured review points.<br />
• Feedback may not be immediately understood.<br />
– Use pupils or their work to exemplify your comments.<br />
– If you are unsure check later.<br />
• Individuals can dominate question and answer sessions and extended oral<br />
<strong>feedback</strong> slows down learning for others in the class.<br />
– ‘No hands up’ questioning is an excellent way of ensuring time for reflection, involving all<br />
pupils in thinking and enabling teachers to target and support individuals.<br />
– If instant oral <strong>feedback</strong> reveals the need for more extended interaction with an individual,<br />
which will not be of benefit to the class, make time later, such as during independent work.<br />
– Offer more extended <strong>feedback</strong> in small groups.<br />
– Feed back to the whole class when it is relevant, using methods other than questioning,<br />
e.g. demonstration, or check pupils’ understanding by silent methods, e.g. whiteboards or<br />
‘show me’ activities.<br />
• Time for individual <strong>feedback</strong> is limited.<br />
– Try to offer collective <strong>feedback</strong> to the class where there are shared learning tasks.<br />
– Utilise small group or guided work on a regular basis to offer <strong>feedback</strong> to pupils with<br />
similar needs.<br />
– Establish understanding that periodic reviews or coaching sessions will be available to<br />
all pupils on a rotation basis.<br />
It is not necessary to cover all the points, as handout 4.1.1 should be given out<br />
to teachers at the end of the session to use as an aide-memoire to support their<br />
own classroom practice.<br />
After <strong>feedback</strong> and discussion say that:<br />
• teachers need to plan alternatives to random questioning and cursory<br />
responses using approaches which encourage pupils to develop their<br />
thinking and learning;<br />
• alternative approaches will be discussed later in the session.<br />
4 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Feedback for Examples of Examples observed in the lesson Impact on<br />
different oral prompts learning ✔<br />
4.2 Using oral <strong>feedback</strong> 25 minutes<br />
for different purposes<br />
Explain that:<br />
• teachers should use different types of oral prompt for different purposes in<br />
their lessons;<br />
• research has shown that few questions asked in classrooms prompt pupils<br />
to use higher-order thinking skills and that in instant <strong>feedback</strong> the teachers’<br />
comments focus mainly on praise and encouragement, rather than moving<br />
learning on.<br />
Task C<br />
Give out handout 4.1.2, which shows ways of providing oral <strong>feedback</strong> which<br />
concentrate on encouraging pupils to develop and move on their thinking and<br />
learning. Explain to participants that the handout provides examples of<br />
<strong>feedback</strong> for different purposes together with exemplar oral prompts. The<br />
suggested list is not exhaustive.<br />
Handout 4.1.2<br />
<strong>Oral</strong> <strong>feedback</strong> for different purposes<br />
Handout 4.1.2<br />
purposes<br />
Correcting<br />
an error<br />
Good try, but that’s not<br />
correct, actually it’s …<br />
Providing<br />
information<br />
Yes, what you’re talking<br />
about is called …<br />
Appraising and That would make sense,<br />
praising good thinking …<br />
Has anyone else tried<br />
this or something similar?<br />
Challenging<br />
Try that again. This time<br />
include/vary …<br />
Seeking<br />
clarification<br />
Encouraging<br />
exploration,<br />
elaboration or<br />
development<br />
Redirecting<br />
learning or<br />
activity<br />
Focusing or<br />
orienting<br />
learning<br />
Confirmation<br />
and moving<br />
learning on<br />
Crystallising<br />
next steps<br />
Distilling and<br />
summarising<br />
learning<br />
What do you mean when<br />
you say it needs more<br />
detail?<br />
How might you take that<br />
argument even further?<br />
Which would be the<br />
best way to …?<br />
That’s a detailed<br />
illustration. Now move on<br />
to the explanation as it<br />
is also important.<br />
All this is important but<br />
it’s really your use of…<br />
that will really improve<br />
your work.<br />
Yes, that’s right,<br />
now you can …<br />
So spend a few minutes<br />
deciding on two changes<br />
you will make to your …<br />
Let’s think about what<br />
we’ve learned so far.<br />
Firstly, we’ve found out …<br />
Encouraging Let’s just think about<br />
pupils to reflect what we’ve discussed<br />
– is there anything else<br />
you might do?<br />
Focusing on Excellent, now how<br />
learning exactly did you manage<br />
approaches and to improve on …?<br />
strategies<br />
Ask participants in pairs or groups of three to discuss the effects the different<br />
types of oral <strong>feedback</strong> will have on pupils’ learning. Explain that the purpose of<br />
this activity is to foster group discussion rather than seeking correct answers.<br />
Allow only 5 minutes.<br />
Explain to participants that they are going to use handout 4.1.2 to record<br />
information about oral <strong>feedback</strong> in a video lesson sequence. Provide the<br />
following information:<br />
• Cannock Chase High School is an 11–18 mixed comprehensive school in<br />
Staffordshire with 1800 pupils.<br />
• The socio-economic circumstances of the pupils are broadly average and<br />
attainment on entry to the school is below the national average.<br />
• The sequence shows a Year 9 geography lesson in which pupils are learning<br />
to improve their writing of geographical descriptions. Make the following<br />
points to prevent follow up discussion being diverted away from AfL.<br />
– In the National Curriculum for geography the specific focus for describing<br />
is on geographical patterns and processes. This requires a distinctive<br />
approach to writing information text, the characteristics of which include<br />
the use of technical terms and supporting facts and figures to create<br />
5 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Feedback for Examples of Examples of oral prompts Impact on<br />
different purposes oral prompts observed in the lesson learning ✔<br />
complex sentences which interrelate variables and factors important in<br />
geography. Participants should not be sidetracked by comparing this<br />
example to descriptions in other curricular areas.<br />
– This lesson focuses on sentence and word level work. The pupils will<br />
consider text level descriptions in a later lesson.<br />
– In humanities subjects it is important that pupils understand the difference<br />
between describing and explaining, which is why teaching for this is made<br />
explicit in the learning objective.<br />
– One of the Year 9 English Framework objectives for sentence level work is<br />
to ‘review and develop the meaning, clarity, organisation and impact of<br />
complex sentences in their own writing.’<br />
• This is the fourth lesson in a unit entitled ‘Local actions, global effects’.<br />
Before the lesson, pupils carried out research about pollution along the River<br />
Rhine and drafted a description about this for homework.<br />
Ask participants to identify examples of different types of oral prompt observed<br />
in the lesson and to enter these in the third column in handout 4.1.2. They<br />
should also tick the fourth column where they can recognise that the oral<br />
prompt has had an impact on pupil learning.<br />
Show the lesson sequence, which is 11 minutes long.<br />
After the video sequence has been viewed, ask participants to spend 5<br />
minutes, in pairs, discussing which aspects were identified and which were<br />
seen to impact on learning.<br />
Use handout 4.1.3, which provides some suggestions from the lesson, to<br />
debrief the activity.<br />
Handout 4.1.3<br />
<strong>Oral</strong> <strong>feedback</strong> for different purposes<br />
Handout 4.1.3<br />
Correcting an error Good try, but that’s not correct. If he’s saying due to… is he describing?<br />
Actually it’s …<br />
…Explaining<br />
Providing Yes, what you’re talking about So these are all descriptions … this is<br />
information is called … telling you why, it’s an explanation.<br />
Appraising and That would make sense, good Let’s just pull that… that’s really<br />
praising thinking … Has anyone else tried important … thank you.<br />
something similar?<br />
Challenging Try that again. This time Do you agree with what he underlined?<br />
include/vary …<br />
Seeking What do you mean when you say So is using numbers not better than<br />
clarification it needs more detail? comparing the two places?<br />
Urging amplification, How might you take that What’s different about this sentence from<br />
exploration or argument even further? Which the one above?<br />
development would be the best way to …?<br />
Redirecting That’s a detailed illustration. Move Now you’ve named places think about<br />
learning or activity on to the explanation as it is how you could compare them.<br />
also important.<br />
Focusing or All this is important but it’s really So she (Lucy) really focused you in on one<br />
orienting learning your use of… that will really thing you could use… actually using<br />
improve your work.<br />
numbers.<br />
Confirmation Yes, that’s right, We’re making quite a lot of headway<br />
and moving now you can … here… does anyone have an even better<br />
learning on<br />
sentence?<br />
Crystallising So spend a few minutes deciding So her target would be to add numbers<br />
next steps on two changes you will make to her comparisons.<br />
to your …<br />
Distilling and Let’s think about what we’ve So we’ve agreed that comparing is good,<br />
summarising learned so far. Firstly, we’ve using numbers to compare is even better.<br />
learning<br />
found out …<br />
Encouraging Let’s just think about what we’ve I want you to read it carefully and say what<br />
pupils to reflect discussed – is there anything else you think is your best sentence.<br />
you might do?<br />
Focusing on Excellent, now how exactly did Discuss any time during the lesson that<br />
learning you manage to improve on …? you feel the other person helped you<br />
approaches and<br />
improve.<br />
strategies<br />
Round off the activity by making the point that the different types of oral prompt<br />
in handout 4.1.2:<br />
• acknowledge what pupils have learned and encourage them to reflect on<br />
and extend their learning still further;<br />
• recognise that pupils need time to reflect on their learning;<br />
• encourage pupils to pose further questions to clarify or further develop their<br />
own or each other’s thinking;<br />
• mostly encourage pupils to make next steps.<br />
6 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
4.3 Making the most of <strong>feedback</strong> 10 minutes<br />
<strong>Oral</strong> <strong>feedback</strong> should always be both positive, i.e. recognising pupils’ efforts<br />
and achievements to date, and developmental, i.e. offering specific details of<br />
ways forward.<br />
Show slide 4.1.5 and give out handout 4.1.4. Read aloud the teacher<br />
comments below and ask participants to place the letter for each statement on<br />
the diagram which is made up of one vertical and one horizontal continuum.<br />
Suggested answers are provided in brackets and should not be read out.<br />
Slide 4.1.5<br />
Handout 4.1.4<br />
A Your long jump was poor. Put in more effort next go. (Negative/Non-specific)<br />
B Your long jump has really come on. You may be in the team next term.<br />
(Positive/Non-specific)<br />
C Your long jump was a disaster. You didn’t touch the board, your legs were<br />
too straight and I can hardly make out your shoulders from your chin.<br />
(Negative/Specific)<br />
D Your long jump was good but you should touch the board and keep your<br />
chin forward. (Positive/Specific)<br />
E Well done, your long jump has really improved. To increase still further you<br />
need to push off from the board and keep your chin well forward. Try these<br />
two things next round and let’s see if you can make 5 metres. You could get<br />
in the team soon. (Positive/Specific)<br />
Teachers will easily place statements A, B, C and E, but there may be some<br />
discussion regarding the place of D. This has been included to illustrate the<br />
negative effect that the word but can infuse into an otherwise positive<br />
statement. It is better to avoid but in most cases and simply state what needs to<br />
be done to improve. Take brief <strong>feedback</strong> after the activity.<br />
7 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Task D<br />
Show slide 4.1.6 which lists some examples of comments used when teachers<br />
provide oral <strong>feedback</strong>.<br />
Slide 4.1.6<br />
Ask participants to:<br />
• place the comments on the continuum diagram on handout 4.1.4;<br />
• improve the comments so that they are more positive and specific (some of<br />
their additions may need to be subject-specific);<br />
• think of some typical comments in their subject and phrase them in a<br />
positive/specific way.<br />
Take brief <strong>feedback</strong> and finally make the point that specific positive <strong>feedback</strong><br />
both improves motivation and moves pupils on.<br />
4.4 Planning for oral <strong>feedback</strong> 15 minutes<br />
Show slide 4.1.7.<br />
Slide 4.1.7<br />
<strong>Oral</strong> <strong>feedback</strong> about pupils’ learning occurs in a range of situations on<br />
a continuum from the instant, informal reply to the more formally planned<br />
reviews and conferences. The following points can be used to support<br />
discussion of slide 4.1.7.<br />
• Instant, informal reply is ephemeral and immediate and tends to be<br />
used for validating, motivating, trouble shooting, or moving the lesson on<br />
quickly. Teachers may need to follow up comments with some pupils at a<br />
later stage.<br />
• Interim review at the end of key episodes during a lesson should be<br />
planned with <strong>feedback</strong> designed to build on learning. This needs to be done<br />
in the light of pupils’ needs and incorporated into the teacher’s short-term<br />
8 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
planning. Teachers may plan to focus on specific pupils or groups and<br />
would also address problems at the point of learning.<br />
• Planned review with individual or groups of pupils focusing on their<br />
targets will need to be considered at whole-school and departmental level<br />
and incorporated into the department’s medium-term plans.<br />
If teachers are to give effective oral <strong>feedback</strong> they need to plan ahead and<br />
organise pupil groupings in their lessons to facilitate the process. Give out<br />
handout 4.1.5, which shows various types of oral <strong>feedback</strong> and the group<br />
composition used.<br />
Handout 4.1.5<br />
Handout 0 cont. 4.1.5<br />
Composition of group<br />
Type of oral <strong>feedback</strong><br />
Whole class<br />
As a starter or within an episode, the teacher uses pupils’ previous<br />
work to feed back teaching points for individuals and the whole class.<br />
The teacher models the language of <strong>feedback</strong> for pupils.<br />
(Planned <strong>feedback</strong> relating to learning objectives)<br />
Whole class<br />
The teacher feeds back points for learning to model improvement or<br />
next steps. Pupils’ contributions are commented on and shaped as the<br />
development is shared with the class. (Planned <strong>feedback</strong> using<br />
modelling of content/techniques or learning strategies, followed by<br />
instant <strong>feedback</strong> on pupil contributions related to objectives)<br />
Whole class<br />
The teacher invites contributions from selected pupils asking for<br />
evaluation and review. This is summed up with teacher <strong>feedback</strong><br />
related to learning objectives and to the quality of the class<br />
review. (Instant <strong>feedback</strong> related to objectives as part of a plenary)<br />
Group<br />
In guided ability or targeted group work the teacher gives instant<br />
<strong>feedback</strong> to pupils as they work on a task and make adjustments in<br />
response. (Some planning in relation to learning objectives or<br />
learning strategies, but some instant <strong>feedback</strong> necessary)<br />
Group<br />
In guided ability or targeted group work the teacher gives planned<br />
<strong>feedback</strong> on work completed previously in response to objectives<br />
common to the group. (Planned review)<br />
Paired work<br />
In pairs pupils assess each other’s work, then the teacher gives oral<br />
<strong>feedback</strong> to the pairs. (Instant <strong>feedback</strong> but related to pupils’ targets/<br />
success criteria)<br />
Paired work<br />
In pairs pupils work on a task. During the process the teacher offers<br />
<strong>feedback</strong> to which pupils respond by making adjustments. (Instant<br />
<strong>feedback</strong> related to tasks/success criteria)<br />
Individual<br />
The teacher gives individual <strong>feedback</strong> during the course of independent<br />
work in process. (Instant <strong>feedback</strong> in relation to objectives/criteria/self<br />
assessment/learning strategies)<br />
Individual<br />
The teacher gives oral <strong>feedback</strong> in an individual review while rest of<br />
class work independently on a task (Planned review)<br />
Task E (5 minutes)<br />
Ask participants to read handout 4.1.5 and, working in small groups:<br />
• decide which types they use most frequently;<br />
• identify any types they have not used before and decide if they might use<br />
these in the future and where;<br />
• suggest any other way of giving oral <strong>feedback</strong> within the lesson and add it to<br />
the grid, completing the details in both columns.<br />
Take brief <strong>feedback</strong> focusing on other suggestions.<br />
We have noted above the place of planned periodic reviews of pupils’ work.<br />
At this point show slide 4.1.8 and say that it may be useful to consider the<br />
following simple sequence when setting up planned periodic reviews with<br />
individual pupils. This will also be distributed at the end of the session as<br />
handout 4.1.6.<br />
Slide 4.1.8<br />
Handout 4.1.6<br />
9 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
4.5 Ready for more? 5 minutes<br />
Refer back to the opening remarks about the need to develop and embed<br />
routine, high-quality oral <strong>feedback</strong> before marking policy can become more<br />
focused, more informative and more selective. This will help lead into future<br />
training in developing written <strong>feedback</strong> should the school choose to<br />
undertake this.<br />
Stress that effective oral <strong>feedback</strong> is an important tool for developing pupils’<br />
learning. Put up slide 4.1.9 (also handout 4.1.7) and point out that, as this<br />
module is to be a focus for whole-school development, teachers need to adopt<br />
one or more of the suggestions to improve their skills in using oral <strong>feedback</strong>.<br />
Slide 4.1.9<br />
Handout 4.1.7<br />
Distribute handouts 4.1.6 and 4.1.7.<br />
Explain that the ‘Ready for more?’ section provides a number of possible<br />
activities that encourage teachers to trial the principles outlined in the training<br />
unit in their teaching.<br />
To encourage a consistent approach in subject areas direct participants to the<br />
Subject Development Tasks. Say that these build on the generic principles in<br />
this training unit and are aimed at embedding this practice in a subject-specific<br />
context. Some of the development tasks will be long-term and need to be<br />
incorporated in the department’s improvement plan.<br />
Emphasise that these are best coordinated by the subject leader to ensure that<br />
there is an agreed focus in the department for the area of development.<br />
The subject development tasks are aimed at addressing, improving and<br />
embedding AfL within subject practice.<br />
10 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
The tasks are designed according to the following process:<br />
Identify<br />
Plan<br />
Do<br />
Review<br />
Refine and improve practice<br />
The presenter should refer to pages 13 and 14 of the Guidance for Senior<br />
Leaders for details relating to how developments in subjects may be<br />
approached.<br />
11 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
PowerPoint slides<br />
Unit 4.1 – <strong>Oral</strong> <strong>feedback</strong><br />
12 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
PowerPoint slides<br />
13 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Slide 4.1.1<br />
Objectives<br />
• To highlight how effective oral <strong>feedback</strong><br />
develops pupils’ learning<br />
• To consider a range of strategies for giving<br />
effective oral <strong>feedback</strong> which can be both formal<br />
and informal<br />
• To help teachers plan for giving effective oral<br />
<strong>feedback</strong><br />
• To help schools develop a whole-school policy<br />
linking and clarifying the relationships between oral<br />
and written <strong>feedback</strong><br />
14 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Slide 4.1.2<br />
<strong>Oral</strong> <strong>feedback</strong><br />
Teacher<br />
Pupil<br />
Pupil<br />
Pupil<br />
Teacher<br />
Pupil<br />
15 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Slide 4.1.3<br />
Some advantages of oral <strong>feedback</strong><br />
to pupils<br />
• immediate and context-specific<br />
• dynamic and adaptable<br />
• ongoing<br />
• episodic<br />
• stimulating<br />
• personalised<br />
• versatile<br />
• motivating<br />
16 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Slide 4.1.4<br />
Some possible difficulties of<br />
oral <strong>feedback</strong><br />
• Instant responses from the teacher may not always<br />
be well considered<br />
• Pupils may not act on oral <strong>feedback</strong> and may not<br />
take it seriously. Other pupils may not listen to it<br />
• Unplanned responses can become random and fail<br />
to develop pupils’ learning in a structured way<br />
• <strong>Oral</strong> <strong>feedback</strong> is ephemeral<br />
• Pupils may feel exposed by public <strong>feedback</strong><br />
• Pupils may not have the time needed to reflect on<br />
the <strong>feedback</strong> and respond to it<br />
• Feedback may not be immediately understood<br />
• Individuals can dominate question and<br />
answer sessions<br />
• Fast-paced question and answer sessions may<br />
be mistaken for fast-paced learning<br />
• Time for individual <strong>feedback</strong> is limited<br />
17 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Slide 4.1.5<br />
<strong>Oral</strong> <strong>feedback</strong><br />
Specific<br />
- +<br />
Non-specific<br />
18 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Slide 4.1.6<br />
Examples of oral <strong>feedback</strong><br />
comments<br />
• You are not making the most of working in a group.<br />
• I don’t think you have the foreground right.<br />
• Well done — a well-constructed answer.<br />
• Your results were not very consistent. You need to<br />
make sure you use the same volume of liquid.<br />
• An excellent answer, but you could develop<br />
it further.<br />
• Yes, good. Your accent is really improving.<br />
19 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Slide 4.1.7<br />
Continuum of oral <strong>feedback</strong><br />
Instant, informal reply<br />
Interim review at the end of<br />
key episodes during a lesson to build<br />
on learning<br />
Planned review with individual<br />
or groups of pupils<br />
focusing on their targets<br />
20 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Slide 4.1.8<br />
A suggested sequence for<br />
giving individual oral <strong>feedback</strong> in a<br />
planned review<br />
1. Plan <strong>feedback</strong> which is positive and specific<br />
2. Reinforce the value and importance of the<br />
pupils’ contribution<br />
3. Focus on recent learning objectives and learning<br />
outcomes in the context of pupil targets<br />
4. Give the pupil(s) time to reflect and respond<br />
5. Encourage the pupil(s) to ask questions to clarify<br />
their understanding of the progress they have made<br />
6. Identify and agree the most important next steps in<br />
learning and revise pupil targets if necessary<br />
7. Agree immediate and longer-term actions. Clarify<br />
when these will be reviewed, by whom, and what<br />
evidence will be sought<br />
21 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Slide 4.1.9<br />
Giving effective oral <strong>feedback</strong> –<br />
ready for more?<br />
• With reference to handout 4.1.2, plan to improve<br />
your questioning techniques as a method of giving<br />
effective oral <strong>feedback</strong>.<br />
• Review which techniques are most successful and<br />
consider their use in your short-term planning.<br />
• With a colleague, observe each other teaching, with<br />
an agreed focus on oral <strong>feedback</strong>. Use the grid on<br />
handout 4.1.2 to record which <strong>feedback</strong> techniques<br />
are used. Identify a technique that was effective and<br />
discuss together why it was.<br />
• Experiment with using the suggested sequence for<br />
planning a formal review (on handout 4.1.6).<br />
Compare experiences with a colleague.<br />
• In a lesson, plan and explain how pupils might use<br />
handout 4.1.4 with pupils to guide their oral<br />
<strong>feedback</strong> to each other.<br />
22 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Handout 4.1.1<br />
Some solutions to possible difficulties<br />
• Instant responses from the teacher may not always be well considered or<br />
sufficiently focused on learning objectives.<br />
– Feedback should always be very clearly related to the lesson objectives and avoid<br />
comment on extraneous factors or areas; the objective needs to be fully explored at the<br />
outset of the lesson, task or episode.<br />
– Wait time before and after questions or required responses encourages pupils to consider<br />
carefully and expand on their response. Other pupils can be asked to reflect or comment<br />
on the response before the teacher responds.<br />
– Encourage pupils to use specific learning outcomes (success criteria) to assess their<br />
own or others’ responses, before or after your <strong>feedback</strong>.<br />
– Using prompts such as Would you like to say a bit more about that? gives the teacher the<br />
opportunity to see what stage of thinking the learner has reached and to fine-tune<br />
<strong>feedback</strong> as appropriate. Such teacher responses also make the process developmental.<br />
– Teachers should attempt to pre-empt possible misconceptions and issues in explanation<br />
of the task.<br />
– Teachers can use pre-emptive strategies, e.g. discuss possible pitfalls and success<br />
criteria when setting tasks.<br />
– Targeting questions or plenary activities at specific pupils can reduce the likelihood of<br />
inappropriate responses from pupils.<br />
– Teachers need to consider the style of <strong>feedback</strong> given as well as the content and its<br />
implication for the way the lesson progresses, e.g. the choice between simple<br />
acknowledgement that an answer is correct or a response that urges further<br />
amplification or exploration.<br />
• Pupils do not act on oral <strong>feedback</strong> and do not take it seriously. Other pupils do not<br />
listen to it.<br />
– It takes time to develop a supportive culture in the classroom and to establish that your<br />
own and other pupils’ errors are learning opportunities.<br />
– It takes time to establish the value and importance of giving and receiving <strong>feedback</strong>.<br />
– While focusing on specific areas of a response, be prepared to say when an answer is<br />
wrong in order to avoid confusion or reinforcing misconceptions.<br />
– Pupils need explicit training and practice in the ground rules of speaking and listening.<br />
– Ask another pupil to follow up on <strong>feedback</strong>.<br />
• Unplanned responses can become random and fail to develop pupils’ learning in a<br />
structured way.<br />
– Questions and tasks need to be planned to encourage higher-level thinking and responses<br />
so that <strong>feedback</strong> can help develop it.<br />
– Structure chains of questions and tasks incrementally so that <strong>feedback</strong> also moves<br />
learning on.<br />
– Ensure that some time is built in for more extended or structured <strong>feedback</strong> at an<br />
appropriate time.<br />
• <strong>Oral</strong> <strong>feedback</strong> is ephemeral.<br />
– This is largely true but what is important is to ensure pupils have time to respond and act<br />
on the <strong>feedback</strong> if necessary.<br />
23 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Handout 4.1.1 cont.<br />
– It may be helpful in some instances to get pupils to reflect on the <strong>feedback</strong> and write their<br />
next steps or targets in their exercise books, or to revise their work/performance/learning<br />
behaviours in the light of <strong>feedback</strong> there and then.<br />
– Make a mental (or actual) note of pupils’ responses and return to points later to check on<br />
understanding and follow up.<br />
• Pupils may feel exposed by public <strong>feedback</strong>.<br />
– The key is to establish a supportive environment in which pupils are happy to contribute to<br />
the lesson. Share the process by encouraging pupils to talk through their thinking and<br />
identify points at which they went wrong or identify ways to improve.<br />
– Establish the idea that if you are not making mistakes you are not learning.<br />
– Create a challenge culture over time where genuine errors are perceived as positive<br />
learning opportunities, and where challenge is welcomed and pupils do not automatically<br />
choose the easiest option.<br />
– Provide opportunities to give private or small group <strong>feedback</strong>.<br />
– Establish routines for group and guided work.<br />
– Always reinforce the value of the pupils’ contribution.<br />
– On occasions, be willing to acknowledge your own errors or lack of specific knowledge.<br />
• Pupils may not have the time needed to reflect on the <strong>feedback</strong> and respond to it.<br />
– Not all responses will require reflection or adjustment; some simply confirm, acknowledge<br />
or encourage.<br />
– Provide more time by echoing the point, e.g. So what you’re saying is …<br />
– Build in time for pupils to amend work.<br />
– Build in regular structured review points.<br />
• Feedback may not be immediately understood.<br />
– Use pupils or their work to exemplify your comments.<br />
– If you are unsure check later.<br />
• Individuals can dominate question and answer sessions and extended oral<br />
<strong>feedback</strong> slows down learning for others in the class.<br />
– ‘No hands up’ questioning is an excellent way of ensuring time for reflection, involving all<br />
pupils in thinking and enabling teachers to target and support individuals.<br />
– If instant oral <strong>feedback</strong> reveals the need for more extended interaction with an individual,<br />
which will not be of benefit to the class, make time later, such as during independent work.<br />
– Offer more extended <strong>feedback</strong> in small groups.<br />
– Feed back to the whole class when it is relevant, using methods other than questioning,<br />
e.g. demonstration, or check pupils’ understanding by silent methods, e.g. whiteboards or<br />
‘show me’ activities.<br />
• Time for individual <strong>feedback</strong> is limited.<br />
– Try to offer collective <strong>feedback</strong> to the class where there are shared learning tasks.<br />
– Utilise small group or guided work on a regular basis to offer <strong>feedback</strong> to pupils with<br />
similar needs.<br />
– Establish understanding that periodic reviews or coaching sessions will be available to<br />
all pupils on a rotation basis.<br />
24 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Handout 4.1.2<br />
<strong>Oral</strong> <strong>feedback</strong> for different purposes<br />
Feedback for Examples of Examples observed in the lesson Impact on<br />
different oral prompts learning ✔<br />
purposes<br />
Correcting Good try, but that’s not<br />
an error correct, actually it’s …<br />
Providing<br />
information<br />
Yes, what you’re talking<br />
about is called …<br />
Appraising and<br />
praising<br />
Challenging<br />
That would make sense,<br />
good thinking …<br />
Has anyone else tried<br />
this or something similar?<br />
Try that again. This time<br />
include/vary …<br />
Seeking<br />
clarification<br />
Encouraging<br />
exploration,<br />
elaboration or<br />
development<br />
Redirecting<br />
learning or<br />
activity<br />
Focusing or<br />
orienting<br />
learning<br />
Confirmation<br />
and moving<br />
learning on<br />
Crystallising<br />
next steps<br />
Distilling and<br />
summarising<br />
learning<br />
Encouraging<br />
pupils to reflect<br />
Focusing on<br />
learning<br />
approaches and<br />
strategies<br />
What do you mean when<br />
you say it needs more<br />
detail?<br />
How might you take that<br />
argument even further?<br />
Which would be the<br />
best way to …?<br />
That’s a detailed<br />
illustration. Now move on<br />
to the explanation as it<br />
is also important.<br />
All this is important but<br />
it’s really your use of…<br />
that will really improve<br />
your work.<br />
Yes, that’s right,<br />
now you can …<br />
So spend a few minutes<br />
deciding on two changes<br />
you will make to your …<br />
Let’s think about what<br />
we’ve learned so far.<br />
Firstly, we’ve found out …<br />
Let’s just think about<br />
what we’ve discussed<br />
– is there anything else<br />
you might do?<br />
Excellent, now how<br />
exactly did you manage<br />
to improve on …?<br />
25 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Handout 4.1.3<br />
<strong>Oral</strong> <strong>feedback</strong> for different purposes<br />
Feedback for Examples of Examples of oral prompts Impact on<br />
different purposes oral prompts observed in the lesson learning ✔<br />
Correcting an error Good try, but that’s not correct. If he’s saying due to… is he describing?<br />
Actually it’s …<br />
…Explaining<br />
Providing Yes, what you’re talking about So these are all descriptions … this is<br />
information is called … telling you why, it’s an explanation.<br />
Appraising and That would make sense, good Let’s just pull that… that’s really<br />
praising thinking … Has anyone else tried important … thank you.<br />
something similar?<br />
Challenging Try that again. This time Do you agree with what he underlined?<br />
include/vary …<br />
Seeking What do you mean when you say So is using numbers not better than<br />
clarification it needs more detail? comparing the two places?<br />
Urging amplification, How might you take that What’s different about this sentence from<br />
exploration or argument even further? Which the one above?<br />
development would be the best way to …?<br />
Redirecting That’s a detailed illustration. Move Now you’ve named places think about<br />
learning or activity on to the explanation as it is how you could compare them.<br />
also important.<br />
Focusing or All this is important but it’s really So she (Lucy) really focused you in on one<br />
orienting learning your use of… that will really thing you could use… actually using<br />
improve your work.<br />
numbers.<br />
Confirmation Yes, that’s right, We’re making quite a lot of headway<br />
and moving now you can … here… does anyone have an even better<br />
learning on<br />
sentence?<br />
Crystallising So spend a few minutes deciding So her target would be to add numbers<br />
next steps on two changes you will make to her comparisons.<br />
to your …<br />
Distilling and Let’s think about what we’ve So we’ve agreed that comparing is good,<br />
summarising learned so far. Firstly, we’ve using numbers to compare is even better.<br />
learning<br />
found out …<br />
Encouraging Let’s just think about what we’ve I want you to read it carefully and say what<br />
pupils to reflect discussed – is there anything else you think is your best sentence.<br />
you might do?<br />
Focusing on Excellent, now how exactly did Discuss any time during the lesson that<br />
learning you manage to improve on …? you feel the other person helped you<br />
approaches and<br />
improve.<br />
strategies<br />
26 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Handout 0 cont. 4.1.4<br />
<strong>Oral</strong> <strong>feedback</strong><br />
Specific<br />
<br />
<br />
Non-specific<br />
27 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Handout 0 cont. 4.1.5<br />
Composition of group<br />
Whole class<br />
Type of oral <strong>feedback</strong><br />
As a starter or within an episode, the teacher uses pupils’ previous<br />
work to feed back teaching points for individuals and the whole class.<br />
The teacher models the language of <strong>feedback</strong> for pupils.<br />
(Planned <strong>feedback</strong> relating to learning objectives)<br />
Whole class<br />
The teacher feeds back points for learning to model improvement or<br />
next steps. Pupils’ contributions are commented on and shaped as the<br />
development is shared with the class. (Planned <strong>feedback</strong> using<br />
modelling of content/techniques or learning strategies, followed by<br />
instant <strong>feedback</strong> on pupil contributions related to objectives)<br />
Whole class<br />
The teacher invites contributions from selected pupils asking for<br />
evaluation and review. This is summed up with teacher <strong>feedback</strong><br />
related to learning objectives and to the quality of the class<br />
review. (Instant <strong>feedback</strong> related to objectives as part of a plenary)<br />
Group<br />
In guided ability or targeted group work the teacher gives instant<br />
<strong>feedback</strong> to pupils as they work on a task and make adjustments in<br />
response. (Some planning in relation to learning objectives or<br />
learning strategies, but some instant <strong>feedback</strong> necessary)<br />
Group<br />
In guided ability or targeted group work the teacher gives planned<br />
<strong>feedback</strong> on work completed previously in response to objectives<br />
common to the group. (Planned review)<br />
Paired work<br />
In pairs pupils assess each other’s work, then the teacher gives oral<br />
<strong>feedback</strong> to the pairs. (Instant <strong>feedback</strong> but related to pupils’ targets/<br />
success criteria)<br />
Paired work<br />
In pairs pupils work on a task. During the process the teacher offers<br />
<strong>feedback</strong> to which pupils respond by making adjustments. (Instant<br />
<strong>feedback</strong> related to tasks/success criteria)<br />
Individual<br />
The teacher gives individual <strong>feedback</strong> during the course of independent<br />
work in process. (Instant <strong>feedback</strong> in relation to objectives/criteria/self<br />
assessment/learning strategies)<br />
Individual<br />
The teacher gives oral <strong>feedback</strong> in an individual review while rest of<br />
class work independently on a task (Planned review)<br />
28 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004
Handout 0 cont. 4.1.6<br />
A suggested sequence for giving individual<br />
oral <strong>feedback</strong> in a planned review<br />
1. Plan <strong>feedback</strong> which is positive and specific.<br />
2. Reinforce the value and importance of the pupils’ contribution.<br />
3. Focus on recent learning objectives and learning outcomes in the context<br />
of pupil targets.<br />
4. Give the pupil(s) time to reflect and respond.<br />
5. Encourage the pupil(s) to ask questions to clarify their understanding of<br />
the progress they have made.<br />
6. Identify and agree the most important next steps in learning and revise<br />
pupil targets if necessary.<br />
7. Agree immediate and longer-term actions. Clarify when these will be<br />
reviewed, by whom, and what evidence will be sought.<br />
29 Unit 4, <strong>Module</strong>1: <strong>Oral</strong> <strong>feedback</strong> © Crown copyright 2004
Handout 0 cont. 4.1.7<br />
Giving effective oral <strong>feedback</strong><br />
– ready for more?<br />
• With reference to handout 4.1.2, plan to improve your use of oral prompts<br />
for a range of different purposes.<br />
• With a colleague, observe each other teaching, with an agreed focus on<br />
oral <strong>feedback</strong>. Use the grid on handout 4.1.2 to record which <strong>feedback</strong><br />
techniques were used. Identify which types of oral prompt were most<br />
effective and discuss why they were successful.<br />
• Experiment with using the suggested sequence for planning a formal<br />
review (on handout 4.1.6). Compare experiences with a colleague.<br />
• In a lesson, plan and explain how pupils might use handout 4.1.4 with their<br />
peers to guide their oral <strong>feedback</strong> to each other.<br />
30 Whole-school development in assessment for learning | Presenter’s Notes © Crown copyright 2004