1 EVAL Manual EVAL is a CLAN program that analyzes ... - CHILDES

EVAL Manual
EVAL is a CLAN program that analyzes transcripts and yields a spreadsheet displaying
25 language characteristics based on a streamlined system for transcribing and coding
discourse tasks. The program can be used in several ways. For example: 1) have a person
with aphasia (PWA) perform a discourse task, then transcribe it and run EVAL on the
transcription. This will produce a spreadsheet displaying the analysis of that PWA's
language performance; 2) have the PWA perform a discourse task from the AphasiaBank
Protocol and run EVAL on the transcript, directing it to compare the results to those of a
comparison group from the AphasiaBank database. This will produce a spreadsheet
displaying the PWA's analysis side-by-side with the mean scores of the comparison
group, and indicating where the PWA and the comparison group differ by one standard
deviation or more; or 3) have the PWA perform a discourse task during a baseline
evaluation, then re-administer the discourse task after a period of therapy, and run EVAL
on the two transcripts. This will produce a spreadsheet displaying the pre and post
therapy results side-by-side, so that performance at different time points can be
compared.
This manual describes how to install the CLAN application and run the EVAL program,
and how to create CHAT transcripts for EVAL analysis.
The first section describes how to download CLAN, run the EVAL program on a
transcript and compare the results to those of a comparison group of people with aphasia
(PWAs) in the AphasiaBank database. A sample transcript (eval_demo.cha) comes in the
"work" folder that downloads in the CLAN folder so that users can quickly learn to run
EVAL and see what it does. The next sections tell how to create your own transcripts for
EVAL analysis, including how to set up the software, how to transcribe, and how to code
aphasic errors for EVAL. It then gives some tips on transcribing aphasic language.
Finally it gives instructions on comparing transcriptions of language samples from two or
more time points. Two additional sample files ready for analysis (eval_demo02a and
eval_demo02b) will download in the work folder of your CLAN application, for those
who would like to experiment with comparing transcripts before creating your own.
If you want to create more detailed transcriptions for analysis by other CLAN programs
should consult the Transcription Training Manual and the CHAT and CLAN Manuals, all
available at the TalkBank/AphasiaBank website.
1. DOWNLOADING CLAN
The first step is to download CLAN. Even if you have previously downloaded CLAN to
your computer, you should download a new version before running EVAL for the first
time. Type http://childes.psy.cmu.edu/clan/ in the address window and press return.
Choose the system you'll be using and click on the appropriate link -- CLANWin for
Windows, CLAN for MAC, or UnixCLAN for Unix.
1

PC users need to download QuickTime if it is not already installed on your computer;
MAC users already have QuickTime.
For MAC users, click on the CLAN link, and follow the web browser instructions for
downloading. The exact download process depends on which browser you are using and
how your browser is configured to handle downloads (where downloads are saved and
whether files are automatically opened after downloading).
If just the CLAN.dmg file appears, double click on it. Ultimately you want to see a folder
named CLAN with the Install.mpkg file in it. If you don’t see it, locate the icon named
CLAN that looks like a hard drive and double click it. Then you will see the folder
mentioned above. Double click on Install.mpkg and follow the instructions for installing
CLAN.
For MAC, installer will install CLAN in your Applications folder. Once the installation
is complete, you can delete the CLAN.dmg folder and eject the CLAN icon that looks
like a hard drive.
For PC users, after you click on CLANWin, you will see a dialogue box asking whether
you want to install or download. If you select download, it will ask you where to put the
download. Choose where you want to put it (e.g., Desktop) and clanwinU.exe (the
installer) will appear in that location. Find it, double click on it, and follow the installer
instructions. If you select install, it will start automatically and you simply need to follow
the installer instructions.
You will also need to have the English MOR grammar on your computer. Go to
(at the CHILDES website), and you will see a
list of MOR grammars for various languages. Click on English and a folder called eng
will download to your desktop (or wherever your downloads go). The eng folder can be
kept as a subfolder of the work folder that came with your CLAN program (see Section
4.a. Creating Folders).
You may need to restart your computer after all these downloads and setups.
Updating CLAN: The CLAN program is changed quite often, so update your CLAN
frequently. To do this, simply go to http://childes.psy.cmu.edu/clan/ and click on the
CLAN program for Windows or Mac and follow the instructions for installing. Once the
program is installed, you can delete any icons left on your desktop from the download
process.
Once the CLAN program is installed on a MAC computer, it will be in your Applications
folder, as shown in the screen shot below. It is convenient to have a CLAN icon in your
dock, so that you can easily access CLAN. You can drag the icon from the Applications
folder to your desktop.
2

On a PC, CLAN will be located in C:/CHILDES/CLAN, as shown below. The installer
will place a CLAN icon on your desktop.
3

On both MAC and PC, you will see a folder named "work" in the CLAN application
folder, as shown in the screen shots above. You will be using the materials in this folder
in the process of learning to use the EVAL program.
2. COMPARING A TRANSCRIPT TO AN APHASIABANK SAMPLE
Open CLAN, and the Command window, shown below, will appear. In this window, you
will see that your working directory (top line in the box) is set at "work", the name of the
folder mentioned above that came with your CLAN program. In this folder is a sample
file, eval_demo.cha formatted and checked for EVAL analysis. You will be using this file
as you work through the following example, which demonstrates how to compare the
language characteristics in a transcript with those of a comparison group you select from
the AphasiaBank database.
To start the comparison, push the progs (for programs) button, and select eval from the
drop down menu of CLAN programs. Push the option button, and you will see a list of
the files in your working directory folder. Double click on eval_demo.cha and it will be
displayed on the right side of the box. Below is an example of a Command window with
eval_demo.cha selected for analysis. When you have chosen the file for analysis, push
Done.
4

A new window, shown below, will appear for selecting database options. Under the
heading, Database types, you are asked to either Deselect Database or Update Database.
For this demonstration, you want to compare eval_demo.cha to a sample from the
AphasiaBank database, so push the Update Database button. This will download a copy
of the AphasiaBank transcript data to the work folder, making it available for comparison
with eval_demo.cha.
5

Next you need to make choices about the part of the AphasiaBank database you want to
compare to eval_demo.cha. For this example, since eval_demo.cha is a transcript of an
anomic aphasic person, we suggest choosing Anomic as the aphasia type for comparison.
Under Age range, since the eval_demo participant is 55 years old, we suggest choosing
45-65 (in this format). If you want only males or only females, choose the appropriate
circle. If you want both genders included, leave those circles blank, which is the choice
we suggest here.
Next select the AphasiaBank discourse task you want to compare to
eval_demo.cha. The eval_demo.cha file is a transcript of the Sandwich Gem from the
AphasiaBank protocol. Select Sandwich under Gem choices. Press the OK button, and a
command with your choices will appear in the Command box as shown below.
Press Run in the bottom right hand corner of the Command box, and the CLAN Output
will appear on your desktop as shown below.
6

Click three times on the last line of the CLAN Output to open the Excel spreadsheet
displaying the results of your analysis. Below are the first two parts of the spreadsheet
showing the analysis of eval_demo.cha, compared to the selected sample from the
AphasiaBank database. This is shown in four parts in the interest of space.
7

Data from the eval_demo transcript are displayed across the first row, and the
information for the comparison database is displayed in the rows below. The top line of
the first column identifies the name of the file being analyzed. Below that is a row
specifying the percentage of a standard deviation that eval_demo differs from the
database. When the difference is a standard deviation or more, the next row indicates this
with one asterisk for one standard deviation or more, and two asterisks for two or more.
The next three rows provide information about the comparison database: the mean, the
minimum, the maximum, and the standard deviation for each of the language measures
EVAL produces. Below this the characteristics of the comparison database are listed (in
this case the Sandwich Gem and Anomic participants from 45-65 years old). The next
line specifies the number of files in the AphasiaBank database that met the criteria for the
comparison database; in this case it was 25. The final line is the CLAN command that
was used to generate this spreadsheet. The second column is the ID information for the
person whose language is in the transcript. The next column, Duration, is listed as 0,
because this session was not timed (see Section 4.c. for timing instructions). The next
columns show the number of utterances, MLU in words and morphemes, number of
types, number of tokens, type/token ratio, clauses per utterance, word errors and utterance
errors, the number of nouns, plurals, and verbs, person and tenses, parts of speech and
repetition and retracing.
Now that you know the basics of running the EVAL program to compare a transcript to
the AphasiaBank database, the next step is to learn to use CHAT to create your own
transcripts and prepare them for EVAL analysis.
3. CONFIGURING CLAN AND COMPUTER
8

The first time you use CLAN for transcribing after installing it on your computer, it will
require some configuring.
• Go to the CLAN menu bar, click on Edit and select CLAN Options.
• In the box next to Checkpoint Every, set the number to 15.
• In the box next to Limit of lines in CLAN Output, set the number to 500.
• In the box next to Tier for disambiguation enter %MOR: (case sensitive).
• Make sure the following 2 boxes are checked: No file backup, Restore cursor on
file open.
• All the other boxes should be un-checked. (To check or uncheck a box, place the
curser in the box and left click.) Then click OK.
CLAN also needs to know where the depfile is located because it contains standards
against which your transcripts are verified. First open CLAN, and make sure that the
Commands window is visible. If it is not, go to the menu bar, click on Windows and
select Commands or use the COMMAND-D keys on Mac and CTRL-D on PC. Click on
the lib button to select the lib folder. You will see a box asking you to locate this folder.
The lib folder is located in the same folder as the CLAN application. Navigate to it,
double click on the file called lib and then click on Select Folder. Now you can close that
window.
You will also need to have the English MOR grammar on your computer. Go to
(at the CHILDES website), and you will see a
list of MOR grammars for various languages. Click on English and a folder called eng
will download to your desktop (or wherever your downloads go). The eng folder can be
kept as a subfolder of the work folder that came with your CLAN program (see Section
4.a. Creating Folders).
Mac users: Please make sure that your function keys operate correctly. Click on the
APPLE icon in the top left corner of your screen. Select System Preferences and then
click on Mission Control (or Dashboard & Expose on older Macs). In the lower half of
the window, if you see F9 in one of the boxes, click on it and select the dash located
somewhere in the pop-up menu to replace the F9. Now the F9 button should work
properly for transcribing using Walker Controller, which you need to use when creating a
transcription for EVAL analysis.
For any Mac keyboard with function keys that perform multiple functions (e.g., a
function key may also control the brightness of the computer screen): You need to enable
your function keys. Click on the APPLE icon in the top left corner of your screen. Select
System Preferences and then click on Keyboard. Make sure there is a checkmark in the
box next to the words that say Use all F1, F2, etc. keys as standard function keys.
4. TRANSCRIBING
9

Note: These instructions are written for Mac computers. Aside from some navigational
differences, the only difference for PC users is using the CTRL key instead of the
Command key.
4.a. Creating Folders
In the CLAN program folder, you have the work folder you have already used with
eval_demo in it. This is where your CHAT transcription files will go. The names of these
files should have no spaces.
Now create a media subfolder in the work folder, named "media" (without the quotation
marks), for audio/video files. This file is case sensitive. When you prepare to start
transcribing, you should put the audio/video file you plan to use into this media
subfolder.
Close all your open windows and you're ready to start.
4.b. Creating a New Transcription File: Obligatory Headers
We strongly urge you to transcribe within the CLAN program (as opposed to Word or
some other word processing program). For EVAL you will need to use the Walker
Controller method, which is explained below.
To start, open the CLAN program. Go to File in the menu bar and click on New (or hit
COMMAND-N). This will open a new blank document called newfile.cha.
At the TalkBank/AphasiaBank website, you will find a link to Transcript Template. Click
on it, and the template for the headers needed for a CHAT transcription will download to
your computer. You can copy the headers into the newfile.cha document, and you will
then need to modify the information in the headers to identify your specific file, as
detailed below. Please note that: 1) each line must end with a carriage return and have no
extra spaces; and 2) after the colon, you must hit the tab button (do not use the space bar).
The first three lines are shown below. The first line needs no changes. The second needs
no changes unless you are transcribing a language other than English. At this point
EVAL has not been tested in other languages, but if you wish to try this, the Language
codes are from the International Organization for Standardization (ISO) and are listed in
the CHAT manual under the section entitled Obligatory Headers. They must be entered
as shown -- lower case, 3 letters. The code for English is eng.
@Begin
@Languages: eng
@Participants: PAR adler02a Participant, INV Investigator
10

Only the corpus name and the participant's code name need to be changed on the
Participants line. After PAR change the ID to your participant's ID (usually your name or
the corpus/investigator name and a 2 digit number indicating that this is the first (01),
second (02), etc. person you have tested, followed by the letter a for initial testing and b,
c, etc. for repeat administrations to the same participant, e.g., adler02a.
Next, identifying information needs to be added to the ID lines. Inside the succeeding sets
of vertical lines in the template you downloaded, type the language code, your corpus
name, participant's age in the format 69;9, participant's gender, participant's aphasia type
according to the Western Aphasia Battery (for aphasic people), or just Aphasia, if the
type is unavailable, and the participant ID (e.g., adler02a). That should complete the
changes on the Participant line. Make sure that each of the four ID entries you made
according to the above instructions is preceded and followed by a vertical line, with no
spaces.
Example:
@ID: eng|Adler|PAR|69;9.|male|Conduction|adler02a|Participant|||
Follow the format in the example below for the investigator ID line. (You can omit this
line if you decide to transcribe only the participant's utterances.)
@ID: eng|Adler|INV||||adler02a|Investigator|||
The final obligatory header is for Media and includes the name of the media file followed
by a comma, a space, and the type of media (video or audio).
@Media: adler02a, video (for video files)
@Media: adler02a, audio (for audio only files)
When you are ready to start working on a new transcript, you can save the template file
renamed with the name of the new participant and the appropriate headers and ID lines.
Follow this procedure: go to File, Save As and name the file using the participant's ID
information. All CHAT files must end with the .cha extension (e.g. adler02a.cha).
This file should go in your work folder, within the CLAN application folder.
4.c. Additional Headers
You have now completed what are called the Obligatory Headers. The following headers
should also be used for transcripts to be analyzed by EVAL: 1) date of testing; 2) Gem
markings to indicate the start of each task within the transcript; and 3) duration of each
Gem.
Use this format for the date:
11

@Date: 05-JUN-2012
If you are administering and transcribing only your own task, you do not need to use
Gem headers. If you are administering one or more AphasiaBank Protocol discourse
tasks, or Gems, use the format below to insert a Gem header before each Gem that you
are transcribing. If you want to compare your transcript to those in the AphasiaBank
database, you must use a discourse task from the AphasiaBank Protocol. The Gems in the
Protocol have the following names: Speech, Stroke, Important_Event, Window,
Umbrella, Cat, Flood, Cinderella and Sandwich. Instructions on how to administer them
in the same way as those in the database are posted at the TalkBank/AphasiaBank
website under Script/Instructions-Aphasia. If you need information on how to record your
session, that can also be found at the AphasiaBank website.
Gem headers are entered with a colon and a tab, as below:
@G: Cinderella
@G: Sandwich
If you want the duration of your transcript included in the EVAL analysis, you will need
to time each Gem with a stopwatch, and then enter the Time Duration header in the
format below under the @G header for each Gem. So, a duration of 12 minutes 15
seconds is represented as in the example below:
@G: Cinderella
@Time Duration: 00:12:15
If your transcription is of a non-­‐AphasiaBank task, you do not need to use @G
headers, and you can put your @Time Duration header at the top of your transcript
below the obligatory headers.
4.d. Using Walker Controller Window to Transcribe
Once you have the headers in your new file, you're ready to begin transcribing. Walker
Controller is like having an audio/video playback system programmed into the computer
so you can stop, start, and move forward and backward as needed. In addition, you can
set the playback to play a certain segment length and repeat segments automatically.
With your CHAT file open, go to the CLAN menu, click on Windows and then choose
Walker Controller. A box will appear on your screen.
Click on the button at the top of that box that says Open media. This will open your
computer's file directory. Find and select your audio/video file, which we suggested that
you put in your work folder within the CLAN application folder, in a subfolder called
media.
Before starting, you need to make some decisions about the settings for Walker
12

Controller.
Walk length determines how long a segment you want to play at a time, so you can set it
to play shorter segments or longer segments. You can start with 4000 msec and then
decide if you want to increase or decrease it.
Loop number specifies how many times you'd like the segment to be automatically
repeated. If you set it for a Walk length of 4000 msec and a Loop number of 2, the
audio/video would automatically play a 4000 msec segment at a time and then
automatically repeat it before going on to the next 4000 msec segment, repeating it, etc.
We recommend setting the Loop number at 1, which plays the segment only once.
Backspace determines how many msec you want the recording to back up before the next
new segment plays. We recommend leaving this at 0.
Walk pause length determines the pause between loops and segments, but only applies to
audio not video files.
Playback speed can be adjusted. Reducing it can be helpful in making aphasic speech
more easily understandable. Just click on the setting you want to change, and type in the
change you want. Then put your cursor back on your transcript page and left click the
mouse to resume your work there.
To transcribe using Walker Controller, use the buttons listed in the box -- F6 makes it
play, F5 or F6 makes it stop, F7 makes it rewind to the previous segment, and F9 fast
forwards to the next segments. Note: If you find that your function keys don't operate
correctly, make sure to follow the instructions in Section 3 for Configuring your
Computer.
To begin the audio/video and start transcribing, press F6.
Each utterance goes on its own separate line that begins with an asterisk and the speaker
ID (the 3 capital letter ID from the Participants line) followed by a colon and a tab.
*PAR: mother is washing the dishes.
As a shortcut for entering the start of each line, go to the CLAN Menu and click on
Tiers, then click on Update. That will automatically assign your first speaker ID (from the
Participants line) to the COMMAND 1 button and your second speaker ID to
COMMAND 2. So, instead of typing the asterisk, 3 capital letter ID, colon and tab each
time you start a new utterance, you can just hit COMMAND 1 or COMMAND 2 and
you'll be ready to transcribe that line.
Capital letters are used only for proper nouns and the word I, and are not used at the
beginning of utterances.
13

Each transcription line ends with some form of punctuation, usually a period, a question
mark, or an exclamation mark. Other symbols may also be used to capture incomplete
utterances (see Section 6 for explanation and examples). Guidelines for making decisions
about utterance segmentation are given in Section 6. The only other obligatory part of the
transcript is the final line, which is @End with no punctuation afterwards.
So, a simple transcript could look like this:
@Begin
@Languages: eng
@Participants: PAR holland13a Participant, INV Investigator
@ID: eng|Holland|PAR|58;11.|male|Anomic|holland13a|Participant||
@ID: eng|Holland|INV||||holland13a|Investigator||
@Media: holland13a, video
@Date: 26-JUN-2007
@Time Duration: 00:12:15
@G: Cinderella
*PAR: Cinderella was a pretty girl.
*PAR: her mother and father died.
@End
5. CHECKING A TRANSCRIPT
The Check program will likely find errors regardless of how carefully you've transcribed.
Here are a couple of ways to run this program. The goal is to get the Check Program to
report this message: Success! No errors found.
5.a. Using Esc-L
You can check your open CHAT file by hitting the ESC key, releasing it, then hitting the
L key. Make sure the CLAN application is active when you do this. You can also check
your file by going to the CLAN menu bar and clicking on Mode and then selecting the
option Check Opened File. Both methods will bring up one error at a time across the
bottom of your transcript page. The cursor will go to the error location. You can correct
that error and then hit ESC then L again (or go to Mode, Check Opened File) and keep
going through your transcript. As you change things, remember to hit COMMAND-S to
save (or go to File, Save on the menu bar). Continue the Check process until you get the
Success message.
Note that sometimes a space is required before a period at the end of the utterance line. If
you can't find anything else wrong with a line, try inserting a space before the period.
5.b. MOR Check
Another useful check for your transcript is to make sure that the MOR program
recognizes all the words in the file. MOR analyzes morphological and syntactic
14

categories. If you have not yet downloaded the MOR grammar, you can get it at the
CHILDES website (http://childes.psy.cmu.edu/morgrams/) and follow the instructions in
Section 3, Configuring CLAN and Computer.
Your transcript file may contain words that are not in the MOR grammar for the
following reasons: 1) the word has never been used before in any other transcript; 2) you
misspelled the word; or 3) the word is being used as a different part of speech.
To check your transcript for words that the English MOR grammar does not recognize,
go to CLAN and open the Commands window (type COMMAND-D or go to Windows,
and select Commands). In the Commands window dialog box, type mor +xl
filename.cha. Make sure that the working directory (the one listed next to the word
working at the top of the Commands box) is the correct one for where your CHAT file is
located. If it is not, click on the working button and select the proper directory and folder.
Also make sure that your mor lib directory (also in the Commands window) is set to
where your eng file is located. Then, back in the Commands window, click on the Run
button in the bottom right corner. This will generate a new file that will be placed in the
same folder as your CHAT file. It will have the same name but the endings will be
.ulx.cex. If you open that file, you will see a list of words the program does not recognize.
This can be a good way to fix typos or other kinds of missed errors. If no words are
listed, MOR recognizes all of the words and you can proceed.
6. TRANSCRIBING FOR EVAL ANALYSIS
Transcription and coding are significantly streamlined for those who want to run only the
EVAL analysis. It is not necessary to transcribe what the clinician/investigator says, and
the transcription can be limited to what the patient says in direct response to the requested
discourse task, omitting any social exchanges or remarks. Use conventional spelling and
omit fillers such as "uh, er," etc., as well as word fragments. If the patient is being asked
to perform a single discourse task, the transcription may well be quite brief.
6.a. Utterance Segmentation
The first decision to be made when you transcribe is how to segment the material you are
transcribing into utterances. As a reminder: each utterance goes on a new line; do not use
initial capital letters, which are reserved for proper names; and each utterance must end
with a terminal marker, such as a period or a question mark, or an indication that it is
incomplete, as explained below.
Utterance segmentation decisions can be challenging. In general, an utterance can be
defined as a segment of running speech that appears to form a coherent unit (Saffran et
al., 1989). When in doubt, err on the side of creating shorter rather than longer
utterances. The following indicators are recommended, with primary weight given to
syntax and intonation:
1. Syntax: Unless there are strong prosodic counter-indications, a well-formed sentence is
15

considered to be an utterance. However, an utterance need not necessarily be
grammatically correct to be considered an utterance.
2. Intonation: Falling intonation (or rising intonation in the case of a question) suggests
the end of an utterance.
3. Pauses: Pauses may not be a reliable guide to utterance boundaries, especially for
aphasic people. When pauses occur in what appear to be otherwise well-formed
utterances, disregard them.
4. Semantics: Semantic criteria cannot be stringently applied in marking utterance
boundaries in this population.
6.b. Incomplete Utterances
Incomplete utterances may occur for a number of reasons. For EVAL analysis, indicating
incomplete utterances as shown below is sufficient.
If the speaker trails off or fails to complete an utterance, use +... as the utterance
terminator.
*PAR: but the story goes on from +...
*PAR: oh gosh.
Utterances with brief interruptions (e.g. INV says "oh I see") can just be transcribed as
single uninterrupted utterances.
6c. Repetitions and Retracings
Repetitions are called Retracing Without Correction. If the speaker repeats material
within a sentence without changing it at all, the material that is repeated is enclosed in
angle brackets (< >) and followed immediately by the square brackets ([/]) with one slash
mark enclosed. If only one word has been repeated once, angle brackets are not needed
and CLAN will assume that the one word before the square brackets with the slash was
repeated.
*PAR: [/] it was so bad.
*PAR: and the [/] the window was open.
*PAR: and she spilled [/] the water on the floor.
Revisions, or Retracings With Correction occur when the speaker changes something
(usually the syntax) in an utterance but maintains the same idea. The material being
retraced is enclosed in angle brackets, followed immediately by the square brackets with
2 slash marks enclosed. If only one word has been changed, angle brackets are not
needed and CLAN will assume that the one word before the square brackets with the
slash was revised. A change, or correction, should be something clearly identifiable that
changes the syntax but maintains the same idea of the phrase.
16

*PAR: well [//] Cinderella is a nice girl.
*PAR: and then sometimes we [//] I was scared about the traffic.
Self-interruptions occur when a speaker breaks off an utterance and starts up another.
These are coded using +//. (or +//? for a question).
*PAR: well then the [//] you know [/] the airplane that [/] that his +//.
*PAR: no the airplane that landed +//.
*PAR: no [/] no that's not right.
Shortenings occur when a speaker drops sounds out of words. Make sure you type the
intended words in this case, because Clan does not recognize the shortened words. For
example, a speaker may leave the final "g" off of "running", saying "runnin" instead. Or a
speaker may say "cause" for "because", "probly" for "probably", "member for
"remember", and so forth.
Assimilations include words such as "gonna" and "lotsa" and should be transcribed as the
intended target, such as "going to", "lots of", and so forth.
6.d. EVAL Error Coding
Once you have completed your transcription you are ready to code the words and
utterances in your transcription. The full AphasiaBank system of error coding is not
required for EVAL. Only two error codes, [*] for word-level errors, and [+ *] for
utterance-level errors are needed.
Word-level errors include phonological and semantic paraphasias, neologisms,
dysfluencies, errors in the use of bound morphemes, incorrect use of articles or parts of
speech, and missing words or parts of speech. Examples of these different types of wordlevel
errors can be found at the TalkBank.org/AphasiaBank/ website under Error Coding.
After each word error in a transcription, type a space, followed by an asterisk in square
brackets, like this:
*PAR you cook on the sink [*] in the kitchen.
If the error is not a real word and you consider it an aphasic error, it should be transcribed
as accurately as possible using the regular alphabet, and followed by the symbol @n,
with no space added (For example, schna@n). Following the @n, a space is added, and
then the symbol [*], to indicate a word-level error, as shown below.
*PAR I wanted to buy a schna@n [*] for her birthday.
EVAL will count this as a word-level error. Words and strings of words that simply
cannot be understood well enough to transcribe should be represented in the transcript
with xxx to indicate that they are unintelligible.
17

*PAR she never knew xxx.
Utterance-level errors include grammatical errors, jargon, empty speech, perseveration
and circumlocution. Examples of these utterance-level errors can be found at the
talkbank.org/AphasiaBank/ website under Error Coding. Place a space followed by the
[+ *] symbol after utterance-level errors:
I'm think of have a sandwich. [+ *].
7. MOR PROGRAM
Once your transcription has passed the Check procedure described above (Section 5), the
next step is to run the MOR program, which tags parts of speech and morphology in each
utterance, and creates a %mor line that lists the tagged utterances under each line of a
transcript.
Save and close your transcript file. Make sure that both the output directory and the
working directory in the Command box are set to where your transcription file is located,
and the mor lib is set to where the eng folder is located. In the Command box, enter the
command:
mor filename.cha +1
Press Run. MOR will add a %mor line below each utterance. MOR labels words by
syntactic category, first listing the category, then a pipe separator, then the word itself,
broken into morphemes. The MOR program looks at each word on the main tier, without
regard to context, and provides all possible grammatical categories and morphological
analyses. The example below illustrates the ambiguities that can occur in this kind of
tagging. For the words “to” and “back” in this case, MOR places the caret ^ to denote the
possibilities for each word.
*PAR: I want to go back.
%mor: pro|I v|want inf|to^prep|to v|go adv|back^n|back^v|back .
8. POST PROGRAM
To resolve these ambiguities, the next step is to run the POST program. Post decides
which tag is correct for ambiguous words, using a database of disambiguation rules
appropriate for English. To run POST on your transcript file, type this command and then
press Run. :
post filename.cha +1
POST chooses among the possibilities for each word, and places a new %mor line below
each utterance. This new line replaces the initial %mor line. The accuracy of
18

disambiguation by POST for English has been measured as above 95 percent.
9. COMPARING TRANSCRIPTS
Once all of these steps are completed, your CHAT file(s) are ready to be analyzed by
EVAL. If you want to run EVAL on your completed transcript of an AphasiaBank Gem
and compare it to a sample from the AphasiaBank database, follow the instructions in
Section 1 above. Instructions will be given here for comparing files (for example, pre-
and post-therapy transcripts) without reference to the AphasiaBank database. The
discourse tasks you transcribe can be AphasiaBank discourse tasks or other tasks, but if
you might later want to compare your patient's results with those in the AphasiaBank
database, you need to use an AphasiaBank Protocol discourse task. You can also compare
several files, so you could measure change in performance at the beginning, the midpoint,
and the end of therapy, for example. If you have not yet prepared transcriptions for
analysis, you can use the sample files eval_demo02a.cha and eval_demo02b.cha that are
already in your work folder to try this way of using EVAL.
Open CLAN, and the Command window will appear. If it doesn't, go to Windows on the
tool bar and choose Commands. If you followed the instructions given earlier, your
CHAT files to be analyzed by EVAL will be in your work folder, and you will have run
CHECK, MOR, and POST on them to prepare them for analysis. The sample files ready
to be analyzed are already in this folder.
Next, push the progs button, and select eval from the drop down menu of CLAN
programs. Push the option button, and you will see a list of the files in your working
directory folder. Double click on the files you want to analyze and they will be displayed
on the right side of the box. When you have finished your selections, click the Done
button.
A new window, shown below, will appear for selecting database options. Under the
heading Database types, choose Deselect Database. If your transcriptions contain only
one Gem, you can bypass the rest of the choices in the box, and just press OK. If your
transcriptions contain more than one Gem, and you want to choose one for analysis,
choose the circle next to the one you want to analyze and then push OK. A command box
will appear showing the appropriate command for your selections. Press run, and the
CLAN output will appear on your desktop, as it did in the example in Section 1. Click
three times on the last line of the CLAN output to open the Excel spreadsheet displaying
your data.
19

EVAL will analyze each file you have chosen, and display the results side-by-side in one
spreadsheet, so that you can directly compare them.
10. ACCESSING APHASIABANK DATA
You can also access AphasiaBank data for your own research and teaching. The easiest
way to view AphasiaBank data is to go to talkbank.org and click on Browsable Database.
A page will appear that lists the accessible data files and gives instructions for accessing
them. If you wish to download AphasiaBank data, click on the Downloadable Database
link. Because the data are password protected, you will be asked to enter username and
password that we previously suggested you request from Brian MacWhinney
(macw@cmu.edu).
Detailed instructions on ways to download and work with AphasiaBank files are provided
in the Transcription Training Manual at the TalkBank.org/AphasiaBank
website.
20