CCRS Grant Review Orientation Slides - Grantreviewinfo.net

Presenters• OPRE Priority Area Manager (PAM)• Akilah Swinton:• (202) 205‐3589, akilah.swinton@acf.hhs.gov• Office of Grants Management• Odessa Crocker, Grants Management Specialist• ICF International• Carolyn Swaney and Renee Pangburn• (877) 350‐4624, ChildCareScholars@icfi.com2

Office of Planning, Research andEvaluation• Unit within the Administration for Children and Families (ACF),U.S. Department of Health and Human Services• Provides guidance, analysis, technical assistance, and oversight toACF programs on performance measurement; research andevaluation methods; synthesis and dissemination of research anddemonstration findings4

Individual Roles and Responsibilities forOff‐Site Review of Grants7

Key Players in Grant Review• Reviewers• Panel Chairs• Priority Area Manager (PAM): Akilah Swinton• OPRE Contractor:ICF International, Carolyn Swaney and Renee Pangburn8

Reviewer’s Responsibilities• Read and understand the funding announcement, evaluationcriteria, and the assigned applications• For each application, assign a score for each of the evaluationcriteria. Clearly identify strengths/weaknesses in comments• Submit scores and comments in the Application Review Module(ARM) System• Participate in the panel discussion (modify scores in ARM, ifnecessary)• Work with the Panel Chair and other reviewers to complete theFinal Summary Reports• Maintain strict confidentiality before, during, and after the review9

Panel Chair’s Responsibilities• Read and understand the funding announcement, evaluationcriteria, and all applications (do not score)• Reinforce reviewers’ understanding of the announcement andevaluation criteria• During panel discussion:• Facilitate a thorough discussion of each application• Take notes of reviewers’ comments• Act as a timekeeper• Complete the Final Summary Report for each assigned applicationin ARM• Verify the forms reflect the panel’s discussions• Promote and maintain compliance with confidentiality and conflictof interest guidelines 10

PAM Responsibilities• Possess relevant content expertise related to the fundingannouncement• Select panel reviewers• Provide technical assistance to support reviewers and panel chairsin their work• Maintain integrity of the review• Review and approve Final Summary Reports• Authorize disbursement of payments11

Contractor’s (ICF) Responsibilities• Contact and confirm potential Reviewers and Panel Chairs• Provide technical assistance to applicants• Develop and produce the necessary review forms• Track received applications and prepare them for the review• Lend general assistance to OPRE staff and reviewers12

Overview of the Grant Process13

Overview of Grant Review Process1. Screen grants for eligibility (PAMs)2. Off‐site review of grants (Reviewers and Chairs)• Phase I (June 4‐June 14): Individual Panel Member Review: Reviewers readapplications; assign initial scores, and prepare draft comments• Phase II: Panel Discussion (June 15): Reviewers discuss applications and assignfinal scores• Phase III: Finalize Scoring & Comment Forms: Chair prepares Final SummaryReports– requires approval by the PAM and reviewers (may require revisionsand resubmission)• Phase IV: Review Close‐Out (June 22): Sign forms and return materials to thecontractor3. Federal review and award (PAMs & OPRE Staff)14

Tips, Tricks, and Checklistsfor each Grant Review Phase15

Phase I:Reviewer Tips forScoring and Writing CommentsPanelReviewPanelDiscussionFinalScoring &ReportingReviewClose‐Out16

Reviewers:Scoring: Evaluation CriteriaSection V.1, pages 32‐3517

Reviewers:Scoring: Total ScoresTotal ScoreGivenQualitative Description96‐100 Outstanding – Definitely recommend for funding90‐95 Excellent –Highly recommend for funding85‐89 Very Good – Recommend for funding with a few modifications80‐84 Good –Could be funded if some changes are made75‐79 Fair –Could possibly be funded if some major changes are madeBelow 75Poor – Cannot recommend funding under any circumstance18

Reviewers:Helpful Hints for Scoring• Enter scores for each evaluation criterion• Use whole numbers only• Score applications independently, not against one another• Ensure scores are consistent with comments (i.e., score quantifiesyour comments)• Scores must be entered in ARM before comments can be enteredARM Access: http://www.grantsolutions.gov/ar19

Reviewers:Purpose and Importance of Comments• To provide feedback to applicants by identifying strengths,weaknesses, and areas for improvement• To enable OPRE staff to appropriately determine funding decision• To explain and validate OPRE funding decisions20

Reviewers:Writing Evaluative Comments• Comment on the strengths and weaknesses of applicants’application relative to each criterion• Be specific, detailed, and concise• Document evidence for your evaluation (note application’s page numbers foreach comment)• Do not mix strengths and weaknesses in the same comment• Consider each sub criterion while preparing comments• Comments must support scores21

Reviewers:Writing Evaluative Comments (cont.)• Comments must pertain to stated criteria, not to your personalexperience or expectations• Be tactful and professional• Use complete sentences and check grammar and spelling• Write comments in the present tense (e.g., “the applicantindicates”)• The person reading the comments may not have seen theapplication. Do your comments make sense as a stand alonedocument?22

Reviewers:Evaluating Your Comments• Routinely ask yourself:• “Do my comments evaluate “the extent to which the applicant…” respondsto each sub‐criterion?”• “Do I identify information in the application relative to the criterion I amcommenting on?• “Will my comments help the applicant to prepare a better application nexttime?”• Consider prioritizing your comments and scores: “What is mostimportant to me?”• This may help justify your score to fellow reviewers during later discussionsof that application23

Reviewers:Evaluating Your CommentsWhen discussing the applications, beware of red flag statements,such as:• “I gave them a score of 15 out of 20. They forgot to answer onecriterion. Each criterion is worth 5 points…”• “I don’t like what she was proposing…”• “She’s from Lake Woebegone University so she must be aboveaverage and extremely competent ...”• “The faculty mentor is not well‐known in the field...”24

Reviewers:Sample Evaluative Comments: StrengthsNot Acceptable:• The applicant has indicated they can be fully operational in the required 30 daytimeframe.Better:• The applicant has indicated they can be fully operational in the required 30 daytimeframe, and has provided a chart showing a proposed timeline. (page 9)Best:• The applicant has indicated they can be fully operational in the required 30 daytimeframe. The transition timeline in the application incorporates the varioustasks they will need to attend to during this time, including the screening ofexisting Head Start staff. (pages 9‐10)25

Reviewers:Sample Evaluative Comments: WeaknessesNot Acceptable:• Of the resumes included, there is a lack of evidence that key staff members arequalified or knowledgeable regarding Head Start. (Appendix)Better:• A review of the resumes provided for key staff reveals no direct experience orfamiliarity with Head Start programs, or with the Head Start PerformanceStandards. (Appendix, pages 12‐24)Best:• A review of the resumes provided for key staff reveals no direct experience orfamiliarity with Head Start programs, or with the Head Start PerformanceStandards. In addition, only one proposed key staff member has experience inEarly Childhood Education. (Appendix, pages 12‐24)26

Reviewers:Commonalities Across Unacceptable Comments• Comments are too general; do not address specific evaluationcriteria provided in the program announcement• Page numbers are not referenced• Comments are descriptive, rather than evaluative27

Reviewers:Referencing Pages in ARM• When entering comments in ARM, Reviewers are required to enterthe page number (or range of page numbers) from the grantapplication associated with each comment• When comments refer to non‐specific page numbers, Reviewersshould use the following standard page numbering terms:Term to UsePage not foundRelevant section title(e.g., “BudgetJustification”)AllTerms Related to Page Number IdentificationTerm DefinedIf an applicant did not address a particular topic or criteria inthe grant submissionIf an applicant did not include page numbers in the grantsubmission, then enter the section title in ARM for referenceIf a particular topic occurs throughout the grant submission28

Reviewers:Prioritizing Comments in ARM• The ARM system defaults to reporting comments in the FinalSummary Report based on page number• If the PAM requests prioritization of comments, Reviewers shouldinsert “0.” in front of the page number associated with eachpriority comment. These priority comments will then appear atthe top of the list of comments• Method for prioritizing comments:• Standard comment: Page 57• High priority comment: Page 0.5729

Phase II:Panel Discussion Preparation Checklist&Panel Discussion ProcessPanelReviewPanelDiscussionFinalScoring &ReportingReviewClose‐Out31

Reviewers:Panel Discussion Preparation Checklist• You have all applications available (ARM or hard copy)• You have the program announcement available (Abbreviatedannouncement is Handbook Appendix A; complete announcementavailable on ARM)• You have the evaluation criteria available (Handbook Appendix B)• You have your initial scores and comments available for eachapplication• You are prepared to take notes during the discussion32

Panel Chairs:Panel Discussion Preparation Checklist• You have all applications available (ARM or hard copy)• You have the program announcement available (Abbreviatedannouncement is Handbook Appendix A; complete announcementavailable on ARM)• You have the evaluation criteria available (Handbook Appendix B)• You have read the reviewer comments in ARM• You are prepared to facilitate the discussion by:• Reviewing roles and responsibilities• Defining rules for panel meeting• Establishing schedule for reviewing applications during the discussion• Discussing differences in scoring, areas of agreement, etc.• You are prepared to take notes during the discussion33

Panel Discussion Process• Calls are estimated to last 3 hours• Recommended timeframe for each application:• Five (5) minutes for introduction of the proposal• Twenty‐five (25) minutes of discussion• Five (5) minutes reporting back and recording scores34

Panel Chairs:Panel Discussion Process• Provide an overview of the responsibilities and structure of thepanel discussion• Lead the discussion for the application under review• Ensure the discussion addresses sub criteria• Identify discrepancies among Reviewers’ scores and commentsand guide the panel’s discussion toward consensus• To return individual evaluations to reviewers for updating scores,click “Return Evaluation” in ARM35

Reviewers:Panel Discussion Process• Discuss each assigned application and determine final scores• If your score has changed as a result of the discussion, update yourscore in the ARM systemNote: final scores can be different from initial scores36

Phase III:Finalize Scores and Comment FormsPanelReviewPanelDiscussionFinalScoring &ReportingReviewClose‐Out37

Panel Chairs:Finalizing Scoring and Comments• In the ARM System, complete the Final Summary Report for eachapplication, incorporating comments written by each Reviewerand relevant comments made in the panel discussion (click “Viewand Include Reviewer Comments”)• Review each Final Summary Report to clarify inconsistencies andensure that it accurately reflects the panel discussion• Click “View Report” to create a pdf of the first draft of the FinalSummary Reports to email to panel members for review andapproval as soon as they are completed; revise as needed toincorporate Reviewer commentsARM Access: http://www.grantsolutions.gov/ar38

Panel Chairs:Finalizing Scoring and Comments (cont.)• Submit the Final Summary Reports to the PAM via the ARM systemas soon as possible• PAM will submit comments to Chair by email• Review edits requested by the PAM:• Make changes if possible• If questions require input from the panel, return the comments to the panelmembers, make additional edits, and resubmit to PAM• If necessary, lead the Final Panel Discussion with Reviewers forapplications with outstanding questions from the PAM and tofinalize the Final Summary Reports39

Reviewers:Finalize Scoring and Comments• Be available via email to review the Final Summary Reports toclarify inconsistencies and ensure it accurately reflects the paneldiscussion• Participate in the Final Panel Discussion to address outstandingquestions from the PAM on the Final Summary Reports• Be available via phone or email to respond to queries from thePanel Chair or the PAM until all Final Summary Reports areapproved by the PAM40

Phase IV:Grant Review Close‐OutPanelReviewPanelDiscussionFinalScoring &ReportingReviewClose‐Out41

Panel Chairs:Grant Review Close‐Out• Once approved by the PAM, print the Final Summary Report foreach application from ARM• Click “View Report” to access the pdf version of the report whichhas a signature line• Print, sign and date each Final Summary Report using black ink• Return all review materials, including applications, print‐outs,drafts, notes, meaningful or identifiable doodles42

Reviewers:Grant Review Close‐Out• If changed after panel discussion, enter final scores into ARMsystem• Click “View Score Report” to access the pdf version of eachReviewer Scoring Form which has a signature line• Print, sign and date each Reviewer Scoring Form (one for eachapplication) using black ink• Return all review materials, including applications, print‐outs,drafts, notes, meaningful or identifiable doodles43

All Panel Members:Returning Grant Review Materials• All Reviewers and Panel Chairs must return all materials pertainingto the grant review to ICF International• Ship all materials no later than the due date by 2‐day deliveryusing the supplied return label44

TimelineandDeadlines45

Reviewers’ Timeline46

Panel Chairs’ Timeline47

ARMDemonstration48

Any Questions?49

THANK YOU SO MUCH• For giving your time• For applying your skills and expertise to this effort• For caring about child care research and the next generation ofchild care researchersAkilah Swinton, OPRE(202) 205‐3589, akilah.swinton@acf.hhs.govAnn Rivera, OPRE(202) 401‐5506, ann.rivera@acf.hhs.govChildCareScholars@icfi.com(877) 350‐462450