Reasoning and Problem Solving - matrics

Reasoning and Problem Solving 

General Instructions: Please review each candidate test within the domain of Reasoning and Problem Solving. A test description is 

provided for each test, followed by data relevant to the five test selection criteria: a) test-retest reliability, b) utility as a repeated measure, c) 

relationship to functional outcome, d) potential changeability in response to pharmacological agents, and e) practicality and tolerability. The 

list of tests being considered in this domain appears below. The order in which the tests are presented was randomized within each domain. 

At the end of the Reasoning and Problem Solving section, there is a rating form to be completed for the selection criteria for the candidate 

tests in this cognitive domain. Please complete the ratings on the candidate tests one selection criterion at a time before beginning the next 

section (e.g., complete ratings for test-retest reliability on all of the tests before going on to utility as a repeated measure). In making your 

ratings, please give due consideration to the data available. Avoid use of global impressions across the selection criteria for any given test. 

We prefer that you enter your ratings directly onto the web-based data entry system, and you will receive instructions and your password for 

this system shortly. Alternatively, you can complete the forms at the end of each section and fax them to us at 310-268-4056, or send them 

via overnight mail to Robert S. Kern, Ph.D. at the address listed on the cover letter. 

Reasoning and Problem Solving Candidate Tests 

1. WAIS-III Block Design 

2. Brief Assessment of Cognition in Schizophrenia (BACS) - Tower of London 

3. Neuropsychology Assessment Battery (NAB) – Mazes 

4. Penn Conditional Exclusion Test (PCET) 

5. Neuropsychology Assessment Battery (NAB) – Categories 

6. Delis-Kaplan Executive Functioning Scale (D-KEFS) - Sorting 

Test Description 

A brief description of each candidate test is provided below. The information includes: (a) a description of test procedures, (b) sample test 

stimuli (if available), (c) scoring procedures, (d) key dependent variables, and (e) approximate administration time for clinical and control 

samples. 


As stated in the WAIS-III Manual, “The examinee is asked to replicate models or pictures of two-color designs with blocks. The 

designs progress in difficulty from simple two-block designs to more complex, nine-block designs. Each block has two white sides, two red 

sides, and two half-red and half-white sides.” 

- 1 -

The test stimuli materials include the Stimulus Booklet and Block Design Blocks (9 total). 

As stated in the WAIS-III Manual, “On the Record Form and in [the] Manual, the designs are shown from [the examiner’s] 

perspective. For incorrect constructions, [the examiner has] the option of recording the examinee’s final design in the blank grids in the 

Incorrect Design column. For each item, record in the Completion Time column the number of seconds the examinee takes to construct the 

design. For correct constructions, circle Y in the Correct Design column to indicate that the examinee constructed the full design correctly. 

Record the score by circling the number of points.” 

Block Design raw scores range from 0 to 64 points. Raw scores may be converted to scaled scores using the normative tables in the 

WAIS-3 Manual. Scaled scores range from 1 to 19. 

Total administration time for Block Design is between 10 and 15 minutes for both clinical and control subjects. 


Patients are shown two pictures simultaneously. Each picture shows 3 balls of different colors arranged on 3 pegs, with the balls in a 

unique arrangement in each picture. Patients are instructed to determine the fewest possible moves of the balls in one picture to make the 

arrangement of balls identical to that of the other, opposing picture. There are twenty test items. The items are of variable difficulty, with a 

general tendency for later items to be more difficult. The test is discontinued if patients make five consecutive incorrect responses. If 

patients respond correctly to all twenty trials, two additional trials of greater difficulty are administered. There are two alternate forms. The 

key dependent variable is the number of correct responses (range: 0-22). Test administration requires approximately 8 minutes. 


The NAB Mazes test is based on a maze-tracing paradigm and was designed to avoid both floor and ceiling effects found in existing 

maze tests. The NAB Mazes test involves single-trial attempts for each of 7 mazes, with time to completion being the primary variable; 

mazes are not discontinued at wrong turns (i.e., “dead ends”). Seven mazes were initially created, with increasing complexity from very 

simple to very difficult. All mazes were 9 in. wide and, with the exception of the first and easiest maze (which was 3 in. high), were 6 in. 

high. The alley width for the first two mazes was kept constant at 1 in. The third maze has 3/4 in. alleys. The fourth maze has 1/2 in. alleys. 

The last three mazes have 1/4 in. alleys. Care was taken so that the paths traversed all four quadrants of the maze. The “start” and “end” 

points for the mazes were divided between right and left and between center and perimeter. Initial designs were pilot tested and 

subsequently modified to ensure increasing completion times across the seven mazes. Once the seven mazes were finalized, two additional 

alternate forms of each maze were created in the following manner. The first alternate form was created by rotating the original maze 180 

degrees along its vertical axis (keeping the “start” and “end” points in the original, unrotated position). The second alternate form was 

created by rotating the original maze 180 degrees along its horizontal axis (again, keeping the “start” and “end” points in the original, 

unrotated position). In this manner, a total of 21 mazes was created. The 21 mazes (three alternate forms of 7 mazes each) were rated by the 

NAB Advisory Council members for difficulty level and overall task satisfaction. The two mazes from each three-maze group with the best 

combination of highest satisfaction and similar difficulty levels were retained. That is, seven mazes were eliminated, and two forms with 

seven mazes each were retained. The results of pilot testing were used to empirically determine the difficulty level of each item, to order the 

items in ascending difficulty, and to equate items across the two equivalent forms. 

- 2 -

As with all NAB tests, Mazes raw scores are quickly and easily calculated on the record form itself, with no need to refer to a 

manual. Standard scores can be derived either by hand, using the NAB Normative Manuals, or with the NAB Software Portfolio, an easy-to 

use computerized scoring program that calculates standardized scores from raw scores using either of the two sets of norms 

(Demographically Corrected or U.S. Census-Matched). 

There is one key Mazes test variable, based on the total scores for all of the mazes administered; each maze is scored based on 

whether it is completed or not, with bonus points provided for speed. 

The total maximum duration for administration is 15 minutes, though most examinees, both clinical and control subjects, require 

much less time to complete (with discontinuation rules substantially decreasing administration time). 


The PCET is a computerized test of executive function with four alternate forms in which the subject is visually presented with four 

figures oriented horizontally and must use a mouse to click on the figure that is different than the other three. The task begins by instructing 

the subject, “In this task you will be shown four objects. You will need to determine which object does not belong with the other three. 

When you correctly choose the object that does not belong you will be told ‘correct’. If you choose an incorrect item you will be told 

‘incorrect’”. For each presentation of four stimuli there is text at the bottom of the page that states, “click on the object that does not 

belong”. The task progresses through 3 correctness criteria and the subject is presented with immediate visual feedback stating “correct” or 

“incorrect” for 500 msec. These sorting criteria are non-overlapping for the four test forms. For form 1, the first category is line thickness, 

the second is shape and the third is size. For form 2, the first category is location, the second is color and the third is type of shape. For form 

3, the first category is line angle, the second is solid versus dashed lines and the third is presence or absence of a border. For form 4, the first 

category is sharp versus rounded edges, the second is angle of the stimulus box and the third is whether the box is filled or empty. Number 

of categories, number of errors, number of perseverative errors, trials to first category, and total number of trials are primary dependent 

measures. Because the task is self-paced, the length is variable, but in the validation study it took both patients and controls approximately 

10 minutes to complete. If a participant is unable to achieve a single category, the test ends after 144 trials. 


In the design of the NAB Categories test, the strengths of existing sorting and classification tasks were incorporated, along with 

additional features that allow for an alternate form with little practice effects, no additional manipulatives or cards, and face-valid and 

visually attractive stimuli. Three complete forms of the Categories test were created initially. Each form consisted of two separate panels; 

each panel contained photographs of six adults along with associated identifying information. To perform the task, the examinee is 

instructed to come up with as many different two-group categories as they can, based on any of the information on the page; each group 

must contain at least two people, and all six people must fit into one group or the other. To create the original three forms, hundreds of 

photographs were initially culled from catalogs of digital stock photography. The selection of the final 36 photographs used in the initial 

three forms was made through an iterative process involving extensive pilot testing and regrouping and reselecting photographs in order to 

yield forms with similar possible categorization solutions based on the visual information included in the photographs (e.g., eye glasses vs. 

no eye glasses, round shirt collar vs. button-down shirt collar). Once photographs were selected, the background of each photograph was 

digitally modified to be either blue-gray or yellow (according to categorization decision rules). For each of the six panels (i.e., two panels 

- 3 -

per form), additional categorization rules were used to create distinct outlines (e.g., three punch holes vs. spiral-binding holes) and 

backgrounds (thin-lined border vs. thick-lined border) for each photograph, in addition to the identification information printed below each 

photograph. For each form, the first panel included six identifying information categories (e.g., name, occupation, place of birth), and the 

second panel included five identifying information categories. Each category (except marital status) was designed to have several possible 

solutions for 1- and 2-point scores; the possible correct responses for the first panel were not possible correct responses for the second panel. 

Once again, pilot testing results guided modifications to the category information such that each panel had a similar number of possible 

solutions across the three forms. The three forms (6 panels) were rated by the NAB Advisory Council members for difficulty level, 

ethnic/racial/cultural bias, educational bias, U.S. regional bias, sex bias, and overall task satisfaction. The two forms (two panels each) with 

the lowest bias ratings, highest task satisfaction ratings, and most similar difficulty ratings were retained. The results of pilot testing were 

used to empirically determine the difficulty level of each panel and to equate panels across the two equivalent forms. 

As with all NAB tests, Categories raw scores are quickly and easily calculated on the record form itself, with no need to refer to a 

manual. Standard scores can be derived either by hand, using the NAB Normative Manuals, or with the NAB Software Portfolio, an easy-to 

use computerized scoring program that calculates standardized scores from raw scores using either of the two sets of norms 

(Demographically Corrected or U.S. Census-Matched). 

There is one key Categories test variable based on the total number of points received (0-1-2) for all categories produced by the 

examinee over the two panels. 

The total maximum duration for administration is 8 minutes, though most examinees, both clinical and control subjects, require less 

time to complete. 

6. Delis-Kaplan Executive Functioning Scale (D-KEFS) - Sorting 

The D-KEFS Sorting Test consists of two testing conditions: Free Sorting and Sort Recognition. For the MATRICS battery, the 

Free Sorting condition is being considered. For the Free Sorting condition, the examinee is presented with six mixed-up cards that display 

both stimulus words and various perceptual features. The examinee is asked to sort the cards into two groups, three cards per group, 

according to as many different categorization rules or concepts as possible, and to describe the concepts he or she used to generate each sort. 

Each of the two card sets can be grouped into a maximum of eight target sorts: three sorts are based on verbal-semantic information from the 

stimulus words, and five are based on visual-spatial features or patterns on the cards. For the Sort Recognition condition, the same sets of 

cards are each sorted by the examiner into two groups, three cards per group, according to the eight target sorts. After each sort make by the 

examiner, the examinee attempts to describe the correct categorization rule or concept used to generate the sort. Because no right or wrong 

feedback is provided by the test, the possible adverse effect of repetitive negative feedback is minimized. This test affords an assessment of 

several executive-function components, including (a) initiating of problem-solving behavior; (b) concept-formation skills; (c) modalityspecific 

problem-solving skills (verbal versus nonverbal); (d) the ability to explain the sorting concepts abstractly; (e) the ability to transfer 

sorting concepts into action; (f) the ability to inhibit previous sorting responses; and (g) the ability to inhibit previous descriptions of sorting 

rules. The test is scored using a scoring-assisted software program. The Free Sorting takes about 10-15 minutes to administer and it can be 

administered by itself. The test has an alternate form. 

- 4 -

Test-Retest Reliability 

- 5 - 

Reasoning and Problem Solving: Test-Retest Reliability 

The tables below provide information about test-retest reliability for each candidate test. Data are provided for normal samples and, when 

available, for schizophrenia and other clinical samples. In making your ratings, please try to take into account several factors, including the 

type of statistic used to measure reliability, the time interval between testings, and whether the data are from normal or clinical samples. As 

a general guide for immediate test-retest reliability, consider r values of 0.60 = modest, 0.70 = good, and 0.80 = very good. For the 

generalizability statistic, G, consider 0.50 = modest, 0.60 = good, and 0.70 = very good. Intraclass correlations (ICCs) are identical to r 

when no mean differences are present. However, unlike r, ICCs will be somewhat lower if mean differences are present, so should be 

viewed as potentially more similar to G when considering comparisons that may include practice effects. Remember that longer test-retest 

intervals would be expected to yield lower values than would immediate test-retest evaluations. 


Author(s) & year Sample type Sample size Interval between testings Reliability coefficient 

1) Wechsler (1997) 

Healthy adults: 16-29 y.o. 100 2-12 weeks r = 0.83 

Healthy adults: 30-54 y.o. 102 2-12 weeks r = 0.88 

Healthy elderly: 55-74 y.o. 104 2-12 weeks r = 0.80 

Healthy elderly: 75-89 y.o. 88 2-12 weeks r = 0.82 



1) Keefe et al. (submitted) Healthy adults Version A repeated: 18 1 – 3 days Version A repeated: 

Version B repeated: 16 

ICC=0.83; Version B 

repeated: ICC=0.73 

Schizophrenia patients Version A repeated: 50 1 – 3 days Version A repeated: 

Version B repeated: 51 

ICC=0.66; Version B 

repeated: ICC=0.77 



1) White & Stern, 2003 Healthy adults 95 193 (+ 20) days (same form 

on both occasions) 

r = 0.55 

Healthy adults 100 25 (+ 6) days (different form 

on each occasion) 

G coefficient= 0.95


- 6 - 

Reasoning and Problem Solving: Test-Retest Reliability 


1) Gur RE et al., in press Healthy adults 24 6 weeks r = 0.69 

Schizophrenia patients 16 6 weeks r = 0.66 



1) White & Stern, 2003 Healthy adults 95 193 (+ 20) days (same form 

on both occasions) 

r = 0.60 

Healthy adults 100 25 (+ 6) days (different form 

on each occasion) 

G coefficient=0.89 

6. Delis-Kaplan Executive Functioning Scale - Sorting 


1) Delis et al., 2001 Healthy adults 286 3 weeks Correct Confirmed Sorts: 

r = 0.60 

Correct Description: r = 0.61

Utility as a Repeated Measure 

- 7 - 

Reasoning and Problem Solving: Utility as a Repeated Measure 

The tables below provide information regarding practice effects with repeated administration. Data are provided for normal samples and, 

when available, for schizophrenia and other clinical samples. Data from schizophrenia samples are most relevant. The primary 

consideration is whether the magnitude of practice effects compromises the test’s utility in a clinical trials design. For example, a test’s 

utility may be compromised by practice effects if they are of a size that they would lead to reduced range, ceiling effects, or reduced 

variance (i.e., smaller SDs over succeeding trials). If low scores on a measure reflect good performance, this is noted in the table. The first 

table under each test contains data on practice effects. The second table contains data on alternate form reliability, if the test is available in 

alternate forms. Alternate forms are generally not needed for tests with minimal practice effects, or ones that do not approach ceiling. If a 

test has practice effects of sufficient magnitude to compromise its utility in a clinical trials design with repeated measurements, then 

consideration of alternate form reliability is appropriate. 


Practice Effects 

Author(s) & year Sample type Sample 

1) Wechsler 

(1997) 

Healthy adults: 

16-29 y.o. 

Healthy adults: 

30-54 y.o. 

Healthy elderly: 

55-74 y.o. 

Healthy elderly: 

75-89 y.o. 

Alternate Form Reliability 

No alternate forms 

size 

Range of possible 

scores 

Mean (SD) at 

T1 

Interval between T1 and T2 

(weeks) 

Mean (SD) at T2 

100 1-19 (scaled scores) 10.3 (2.6) 2-12 weeks (mean = 4) 11.3 (2.7) 



88 1-19 (scaled scores) 9.7 (2.9) 2-12 weeks (mean = 4) 10.0 (3.2)




Author(s) & year Sample type Sample size Range of 

possible 

1) Keefe et al. 

(submitted) 

Healthy adults Total=50 

-Version A repeated = 18 

Schizophrenia 

patients 

-Version B repeated = 16 

Total = 150 

-Version A repeated = 50 

-Version B repeated = 51 

scores 

0-22 

(raw 

scores) 

0-22 

(raw 

scores) 

- 8 - 

Mean (SD) at T1 Interval between 

15.39 (3.18) 

17.19 (2.93) 

12.46 (4.72) 

10.90 (5.52) 

T1 and T2 

(weeks) 

1-3 days 

1-3 days 

16.83 (2.75) 

18.69 (2.5) 

13.75 (4.80) 

11.78 (5.72) 

Mean (SD) at T2 



(specify coefficient type, e.g. 

r (ICC) = .89) 

1) Keefe, et al. 

Healthy adults Version A, then B: 8 1 – 3 days Version A, then B: ICC=0.97 

Version B, then A: 8 

Schizophrenia patients Version A, then B: 23 

Version B, then A: 25 

3. Neuropsychology Assessment Battery (NAB) - Mazes 


Author(s) & 

year 

1) White & 

Stern, 2003 

Version B, then A: ICC =0.93 

1 – 3 days Version A, then B: ICC=0.81 

Version B, then A: ICC=0.89 

Sample type Sample Range of Mean (SD) Interval between Mean (SD) Interval between Mean (SD) at 

size possible scores at T1 T1 and T2 

(weeks) 

at T2 T2 and T3 

(weeks) 

T3 

Healthy adults 95 19-81 (T-scores) 49.0 (11.5) 28 50.5 (8.7) N/A N/A 




r (ICC) = .89) 

1) White & Stern, 2003 Healthy adults 100 25 (+ 6) days G coefficient= 0.95




size 

1) Gur RE, et al., 

in press 

Range of 

possible 

scores 

Healthy adults 24 (z-scores) 

-3 to +3 

Schizophrenia 16 (z-scores) 

patients 

-3 to +3 

Mean (SD) at 



size 

T1 

- 9 - 


Interval between 


Mean (SD) at 

(weeks) 

(weeks) 

0.07 (0.40) 6 weeks 0.13 (0.41) 24 weeks 0.24 (0.40) 

T2 



Mean (SD) at 

-0.50 (1.68) 6 weeks -0.75 (1.57) 24 weeks -0.19 (0.93) 

Interval between testings Reliability coefficient 


r (ICC) = .89) 

1) Kurtz et al, in press Healthy adults 80 Same test session (1-2 minutes) ICC=0.39; p



Author(s) & 

year 

1) D-KEFS Test 

Manual 

Sample type Sample 

size 

Range of possible 

scores 

Healthy adults 101 Confirmed sorts 

Corr. description 

Range = 1-19 

Mean 

(SD) at T1 

10.2 (2.8) 

10.1 (2.7) 

- 10 - 




(weeks) 

Mean (SD) 

at T2 

25 (+12.8) days 11.3 (2.4) 

11.3 (2.5) 



(weeks) 

N/A N/A 

Mean (SD) at 



(specify coefficient type, e.g. r 

(ICC) = .89) 

1) D-KEFS Test Manual Healthy adults 101 25 (+12.8) days r (Correct confirmed sorts) = .57 

r (Correct description) = .39 

T3

Reasoning and Problem Solving: Relationship to Functional Outcome 

Relationship to Functional Outcome 

The tables below provide information on the relationship between performance on candidate tests and functional outcome. Functional 

outcome measures include measures of community functioning, social problem-solving ability, or psychosocial rehabilitation. Data from 

schizophrenia samples are most important and are listed first in each table. If available, data from other groups, such as other clinical 

samples, are also listed. Demonstrations of prospective relationships to functional outcome are especially powerful and should be given 

greater weight than cross-sectional findings, even though the strength of the relationship between cognition and functional outcome might 

be lower in prospective studies than cross-sectional ones. As a general guide to effect sizes, correlation coefficients of 0.1 are considered 

small, 0.3 are considered medium, and 0.5 large. 


Studies Using This Specific Test 

Author(s) & year Sample type Sample size Functional outcome 

measure(s) 

1) Dickerson et al., 

1999 

2) Gold, J. 

(INS, 2003) 

3) Brekke et al., 

1997 

4) Buchanan et al., 

1994 

Schizophrenia 

patients 

Stable 

Schizophrenia 

outpatients 

Schizophrenia and 

Schizoaffective 

Disorder patients 

Schizophrenia 

patients 

- 11 - 

Findings (r or other magnitude of 

relationship statistic) 

Cross-sectional vs. 

prospective study (if 

prospective, provide 

interval) 

72 Social functioningoccupational 

activity 

r = 0.24, p=0.045 Prospective, 2 year follow-up 

77 Total Vocational outcome r (with employment status ) = 0.262, Cross-sectional 

(39 competitively 

employed; 38 

unemployed) 

p=0.021 

40 Vocational, Social, 

Independent Living 

r = 0.35, p

8) Lysaker et al., 

2001 

Schizophrenia and 

Schizoaffective 


49 


Self-efficacy r = ?, ns 

Cross-sectional 

Well-being r = -0.30, p




size 

1) Kurtz et al. (in 

press) 

Schizophrenia 

patients 

Functional outcome 

measure(s) 


- 13 - 

Findings (r or other magnitude of 

relationship statistic) 

28 Work competence PCET categories and errors related to 

“cooperativeness” on the job (r=0.47 and 

r=-0.49 respectively). PCET errors related 

to “work quality” (r=-0.44) and “general 

impression” (r=-0.42). 

Cross-sectional vs. 

prospective study (if 

prospective, provide interval) 

Cross-sectional 

Meta-Analysis of Similar Tests 

Author(s) & year Type of tests Sample type Sample size Pooled estimated r p value 

1) Green et al., 2000 Card sorting tests Schizophrenia and Schizoaffective 


1002 r = 0.23 p

Reasoning and Problem Solving: Potential Changeability in Response to Pharmacological Agents 

Potential Changeability in Response to Pharmacological Agents 

The following tables provide information about changeability in response to pharmacological agents. The data in this section are 

illustrative, and not comprehensive. The findings listed below were provided by test developers, or come from representative reviews and 

meta-analyses. Data for schizophrenia samples are listed first in the tables below. Data from other groups, such as other clinical samples, 

are also listed. The studies listed include those that used novel cognitive-enhancing agents, ones that examined newer versus conventional 

antipsychotic medications, or ones that compared two or more newer antipsychotic medications. In examining the data, give greater weight 

to between-group effects except in those cases in which all treatment arms would be expected to yield cognitive enhancing effects. In 

evaluating the tests for this particular criterion, consideration should be given to the fact that this area of investigation is relatively new. 




size 

Drug comparison Findings (specify within- and between-group effects) 

1) Buchanan et al., 1994 Schizophrenia 38 Clozapine vs. haloperidol Significant group difference on block design (p < 0.05) 

2) Meltzer and McGurk 

1999 

patients 

Schizophrenia 

patients 

Review of 

literature 

Studies Using Similar Tests (similar in format and assessing same construct) 

No data available 




1) Allen et al., 2003 Schizophrenia 

patients 

size 

Clozapine 2/3 studies showed improvement with clozapine (withingroup) 


20 Galantamine (procholinergic 

agent and 

modulator at the nicotinic 

receptor) vs. placebo 



- 14 - 

Tower of London change score, galantamine vs placebo at 

trend level (p=0.073)




The NAB is a newly released test and has no data available. 



size 


1) Gallhofer et al., 1996 Schizophrenia 64 Clozapine vs. risperidone Clozapine and risperidone treated patients performed 

patients 

vs. conventional 

significantly better than unmedicated patients, or those taking 

medications vs. no meds conventional meds (contrasts, p > 0.05) 

2) Meltzer and McGurk Schizophrenia Review of Clozapine and risperidone 1/2 studies showed improvement with clozapine; 0/1 showed 

1999 

patients 

literature 

improvement risperidone 




size 

1) Gur RE, et al., in press Schizophrenia 

patients 

16 

Healthy adults 24 


Patients off vs. on 

Olanzapine, vs. controls 

- 15 - 

Between-group, diagnosis by time interaction: F(5,66) = 2.72, 

p = 0.0268; patients improved more than controls 

Meta-Analyses and Reviews of Similar Tests 

Author(s) & year Type of tests Sample type No. of studies Effect size (within-group 

1) Harvey & Keefe, 2001 Tests of executive functioning Schizophrenia 

patients 

2) Keefe et al. 1999 Tests of executive functioning Schizophrenia 

patients 

3) Meltzer & McGurk, Wisconsin Card Sorting Test Schizophrenia 

1999 

patients 

effect), or box score 

N = 20 for all d = 0.18 

tests, no info on 

number of 

executive tests 

12 8/12 showed improvement 

9 2/7 studies showed improvement 

with clozapine; 1/2 showed 

improvement with risperidone




The NAB is a newly released test and has no data available. 




patients 

2) Keefe et al., 1999 Tests of executive functioning Schizophrenia 

patients 


1999 

patients 




- 16 - 


N = 20 for all d = 0.18 


number of 





improvement with risperidone 




patients 

2) Keefe et al. 1999 Tests of executive functioning Schizophrenia 

patients 


1999 

patients 


N = 20 for all d = 0.18 


number of 





improvement with risperidone

Practicality and Tolerability 

- 17 - 

Reasoning and Problem Solving: Practicality and Tolerability 

The following narratives provide information about the steps that are needed to set up each test at a new site. In evaluating each test, take 

into consideration: (a) ease of set-up including use of any specialized equipment (e.g., particular computer requirements), (b) duration of 

test, (c) the length of time to train testers to administer and score the test, (d) the length of time to score an individual test after testers have 

been trained, (e) subjective reactions of the individuals who are taking the test (e.g., do they find it boring, frustrating, intrinsically 

interesting)? It may be helpful to refer back to the Test Description section (pages 1-4) to review specific information on a given test. 


The materials needed to administer the test are: WAIS-III Manual, WAIS-III Stimulus Booklet, Block Design Blocks (9), stopwatch, 

and a Record Form. 

Administration and scoring are readily learned by testers, although testers must practice manipulating multiple materials at once. 

Total administration time for Block Design is between 10 and 15 minutes for both clinical and control subjects. Scoring time is minimal 

requiring the tester to simply add the total correct and look up a scaled score in the manual. Additional scoring time (minimal) is required if 

the option to draw incorrect designs is implemented. Both clinical and controls groups find the task mildly to moderately challenging. 

Clinical subjects tend to be more easily frustrated and tolerate the task less well than controls. 


The procedures for setting up this test at new sites have been established through its use in nine multi-site clinical trials. Each site 

receives the test materials, along with an instruction and scoring manual and an instructional CD. It is very simple to set up this test at a 

new site. The only materials needed are the Tower of London stimulus book, the one page response sheet, a stopwatch and a pen or pencil. 

It usually requires less than 15 minutes to train the tester on the administration and scoring of the test. The test is scored during 

administration. Patients may become frustrated as the items become more complex but frustration is limited due to the discontinuation 

criteria. Most controls find the test to be interesting and increasingly more challenging as complexity increases. 


The NAB Mazes test is quite easy to set up at a new site. The only materials required are: 1) the NAB Executive Functions Module 

Record Forms (one for each participant; separate forms for each of the two equivalent forms); 2) the NAB Executive Functions Module 

Response Booklets (one for each participant; separate forms for each of the two equivalent forms); and 3) a stop watch. Testers can be 

trained to administer and score the Mazes test in less than an hour. It is recommended that they practice administering the test with at least 

two examinees prior to using it for actual data collection. Calculation of raw scores takes 1-2 minutes. Calculation of the standardized Tscore 

using the NAB Software Portfolio computerized scoring package takes 1 minute. In the standardization sample (N = 1448) and in the

Reasoning and Problem Solving: Practicality and Tolerability 

numerous clinical samples examined for validity studies (including patients with MS, TBI, ADHD, HIV/AIDS), respondents reacted 

favorably to the test. It was viewed as neither too easy nor too challenging/frustrating. 


This is a computerized task originally developed on the PowerLaboratory platform. It runs on older Macintosh computers using OS9 

system software, or on newer Macintosh computers running MAC OSX using the Classic environment. Portability has been a concern so the 

task has also been implemented with a Web-based interface using FLASH MacroMedia. The Web-based version of the task can be run on or 

off-line with any computer. Either version of the task outputs performance data as a text file that gets scored using a Python script running 

on Linux. 

Training on task administration and scoring requires about 1 hour. The majority of the training involves learning how to set-up and 

launch the task. Once the task is launched, the administrator reads the on-screen instructions along with the subject and answers any 

questions to insure that the subject understands the task. However, once the task starts, it runs and saves data automatically with no 

additional input from the administrator. Scoring of the data requires a simple one-line command to launch the scoring program that scores 

and databases the results automatically. Scoring requires less than a minute. 

Subjects have reacted very favorably to taking this task. For patients, the computer interface is particularly welcome as it requires 

less social interaction than standard paper and pencil administration. In some cases, older control subjects who are not familiar with 

computers are initially intimidated. For this reason, we have developed a separate “Mouse Practice” task that familiarizes subjects with the 

computer at the beginning of our computerized test battery. 


The NAB Categories test is quite easy to set up at a new site. The only materials required are: 1) the NAB Executive Functions 

Module Stimulus Book (one for each of the two equivalent forms); 2) NAB Executive Functions Module Record Forms (one for each 

participant; separate forms for each of the two equivalent forms); and 3) a stop watch. Testers can be trained to administer and score the 

Categories test in less than an hour. It is recommended that they practice administering the test with at least two examinees prior to using it 

for actual data collection. Calculation of raw scores takes 3-5 minutes. Calculation of the standardized T-score using the NAB Software 

Portfolio computerized scoring package takes 1 minute. In the standardization sample (N = 1448) and in the numerous clinical samples 

examined for validity studies (including patients with MS, TBI, ADHD, HIV/AIDS), respondents reacted favorably to the test. It was 

viewed as neither too easy nor too challenging/frustrating. 


The sorting test is a paper-and-pencil test, though computer scoring is available. The free sorting condition, which is being 

considered for the MATRICS battery, takes considerably less time than administering both the free sort and recognition conditions. Free 

sorting alone takes 10-15 minutes to administer. No information on tolerability. 

- 18 -


Name: ______________________________ Date: _________________ 

General Instructions: Please indicate your rating for each test on the test criteria by placing an “X” in the box below the 

desired numeric rating. Descriptors are provided to help anchor the ratings. Use the “5” anchor point to indicate, “There 

are just enough data on this criterion to consider the test useful for clinical trials.” Ratings greater than “5” indicate that 

there is more than enough support for the test on this criterion; ratings less than “5” indicate that there is less than 

adequate support. 

Test-Retest Reliability 


2. BACS - Tower of London 

3. NAB – Mazes 

4. Penn Conditional 

Exclusion Test (PCET) 

5. NAB – Categories 

6. DKEFS – Sorting 

Comments: 

Utility as a Repeated 

Measure 








Comments: 

Poor 

1 

Poor 

1 

2 

2 

Fair 

3 

Fair 

3 

- 19 - 

4 

4 

Rating 

Good 

5 

Rating 

Good 

5 

6 

6 

Very 

Good 

7 

Very 

Good 

7 

8 

8 

Superb 

9 

Superb 

9

Relationship to 

Functional Outcome 


2. BACS - Tower of 

London 






Comments: 

Potential Changeability 

in Response to 

Pharmacological 

Agents 








Comments: 

None 

1 


2 

Weak 

3 

4 

- 20 - 

Rating 

Moderate 

5 

Rating 

6 

Strong 

7 

Unlikely to be Sensitive Difficult to Judge Likely to be Sensitive 

1 

2 

3 

4 

5 

6 

7 

8 

8 

Very 

Strong 

9 

9

Practicality and 

Tolerability 








Comments: 

Poor 

1 


2 

Fair 

3 

- 21 - 

4 

Rating 

Good 

5 

6 

Very 

Good 

7 

8 

Superb 

9

Reasoning and Problem Solving - matrics

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?