AFCARS Assessment Review Report - Administration for Children ...

MarylandAFCARSAssessment ReviewReportMarch 2009Prepared by:Children’s Bureau, Administration on Children, Youth and FamiliesandOffice of Information ServicesAdministration for Children and FamiliesU.S. Department of Health and Human Services

TABLE OF CONTENTSBackground 1Rating Factors 2Findings 3General Requirements 4Data Elements 6Conclusion 9Tab ADetailed FindingsSection 1: General RequirementsSection 2: Foster Care and Adoption ElementsSection 3: Case FilesTab B AFCARS Improvement PlanSection 1: General RequirementsSection 2: Foster Care and Adoption Elements

BACKGROUNDData for the Adoption and Foster Care Analysis and Reporting System (AFCARS) are requiredby Federal law and regulation. The data are to be collected on children in foster care and thosewho have been adopted under the auspices of the State child welfare agency. States that fail tomeet any of the standards set forth in 45 CFR 1355.40(a-d) are considered not to be in substantialcompliance (i.e., are lacking in substantial conformity) with the requirements of the title IV-EState plan, and are subject to penalties 1 . Additionally, States that received funding to develop,implement, and operate a Statewide Automated Child Welfare Information System (SACWIS)under Federal regulations at 45 CFR 1355.53 are to produce a comprehensive, effective, andefficient system to improve the program management and administration of the State plans fortitles IV-B and IV-E of the Social Security Act. At a minimum, the system must provide foreffective management, tracking, and reporting by providing automated procedures and processesto, among other things, meet the adoption and foster care reporting requirements through thecollection, maintenance, integrity checking, and electronic transmission of the data elementsspecified by the AFCARS requirements.The Children’s Bureau is committed to assisting States to develop statewide child welfareinformation systems and to collect quality data. To this end, SACWIS and AFCARSAssessment Reviews were developed to assure that the systems support the management of theprograms under titles IV-B and IV-E and can produce accurate and reliable foster care andadoption data. AFCARS Assessment Reviews (AAR) are conducted in every State, regardless ofwhether a State operates a SACWIS. The State’s information system is assessed against theAFCARS requirements in the Federal regulations, policy issuances, and the AFCARS TechnicalBulletins. The AAR evaluates a State’s information system’s capability to collect, extract, andtransmit the AFCARS data accurately to the Children’s Bureau. A second focus of the AAR isto assess the accuracy of the collection and documentation of information related to the fostercare and/or adoption case of a child.The review process goes beyond the edit checks that must be met by a State in order to pass theAFCARS compliance error standards. The review also ascertains the extent to which a Statemeets all of the AFCARS requirements and examines the quality of its data. Additionally, whilethe review is an assessment of the State agency’s collection and reporting of AFCARS data, it isalso an opportunity for Federal staff to provide substantive technical assistance to State agencystaff.Each AAR consists of a thorough analysis of the State’s system technical documentation for thecollection, extraction and reporting of the AFCARS data. In addition to this review ofdocumentation, the Federal AFCARS team reviews each data element with the State team to gaina better understanding of the State’s child welfare practice and policy and State staff’sunderstanding of the data elements. The data are also compared against a small, randomlyselected number of hard copy case files. Through this exercise, the accuracy of the State’s dataconversion process and understanding of the information reported to AFCARS is tested.1 The Administration for Children and Families is not assessing AFCARS penalties at this time (seeACYF-CB-IM-02-03) and will not take penalties until new, final AFCARS regulations are issuedimplementing P.L. 108-145 (The Adoption Promotion Act of 2003).US DHHS/ACF/ACYF/Children’s Bureau 1

RATING FACTORSTwo major areas are evaluated during an AFCARS assessment review: the AFCARS generalrequirements, and the data elements. The general requirements include the population that is tobe reported to AFCARS and the technical requirements for constructing a data file. The dataelements are assessed for overall data quality, to determine whether the State is meeting theAFCARS definitions for the information required, and to determine whether the correct data arebeing entered and extracted.AFCARS data submissions are subject to a minimal number of edit checks, as listed inAppendix E of 45 CFR Part 1355. Based on these edit checks, substantial compliance can bedetermined for the timely submission of the data files, the timely entry of certain data elements,and whether the data meets a 90 percent level of tolerance for missing data and internalconsistency checks. However, “substantial” compliance does not mean a State has fullyimplemented the requirements in the regulations. This explains why a State formerly may havebeen penalty-free, and yet does not have accurate and reliable quality data. For example, editchecks of the data cannot determine whether the State submitted the correct foster carepopulation required by the Federal regulations.Information collected from each component of the assessment review is used to rate each dataelement. The general requirements are assessed and rated separately using the same scale. Ascale of zero (the system is not collecting the AFCARS data elements and the data are nottransmitted) to four (fully meets the AFCARS standards) is used to assign a rating factor. Belowis a chart that lists the factors that were used for the analysis of the State’s AFCARS.AFCARS Rating FactorsRATING FACTORDEFINITION4 All of the AFCARS requirements have been met. The informationsystem is functioning as required, and the information is beingaccurately collected and extracted.3 There are data quality issues. For example: The data are underreported due to inconsistent data entry. The data are not being entered. Data entry is unreliable due to incorrect or ambiguousinstructions, definitions, and/or data entry screens. There are no supervisory controls for ensuring data entry, oraccurate data entry. There is incorrect data entry due to training or design issues. There are missing or incomplete data due to conversion errors.2 The technical requirements for AFCARS reporting are not fullymet. For example: The State information system has the capability to collect thedata, but the program logic is incorrect. The State uses defaults for blank information. Information is coming from the wrong module or field in theUS DHHS/ACF/ACYF/Children’s Bureau 2

system. Information is located in the wrong place on the system, i.e., itshould be in foster care screens, not adoption screens. The system needs modification to encompass more conditions,e.g., disability information. The extraction code for the AFCARS report selects and reportsincorrect data.1 An AFCARS requirement(s) has not been implemented in theinformation system. For example: The State information system does not have the capability tocollect the correct information (i.e., there is no data field on thescreens). There is no program logic to extract the information. There is 100% missing data according to the frequency reportor DCU/DQU reports.0 States operating an automated information system for which theyreceived SACWIS-level FFP were found to be using an externalautomated information system, or a database (such as Excel orAccess), and are not collecting and reporting the AFCARS datafrom the SACWIS system. In addition, there is no program codefor the extraction of data from the SACWIS.For data elements and general requirements that do not meet existing AFCARS standards(factors 0 through 3), the State is required to make the corrections identified by the review team.It is possible that the problem with a data element and data are due to both system issues andcaseworker data entry issues. In such instances, the element will be rated a “2” to denote theneed for modification to the system. Once the corrections are made to the system, the data willbe re-analyzed. If problems related to caseworker training or data entry still exist, then a “3” willbe assigned to the requirement. A rating factor of “4” (compliant) will not be given to theelement until all system issues and/or data quality issues have been addressed.The State is required to make the changes to the information system and/or data entry in order tobe compliant with the applicable requirements and standards. Since the AFCARS data are usedfor several significant activities at the Federal and State levels, the State must implement theAFCARS Improvement Plan, under Tab B of this report, as a way to improve the quality of itsdata.FINDINGSThis section contains a summary of the significant reporting and data quality issues that werefound during the AAR. For additional information on specific AFCARS data element reportingissues and the general requirements, please see the attached matrices. The AFCARS data usedfor the review were from the report period October 1, 2007 - March 31, 2008 (2008A).Tab A contains the findings matrices for the general requirements, the data elements, and thecase file review. If as a result of the post site-visit analysis of the State’s information the ratingUS DHHS/ACF/ACYF/Children’s Bureau 3

factor changed from the preliminary on-site rating, the original is marked out and the new ratingis included.General RequirementsInformation System and ConversionMaryland’s SACWIS is called the Children’s Electronic Social Services Information Exchange(CHESSIE). CHESSIE was implemented in all Maryland localities as of January 2007. At thetime of the onsite review, the State was not making foster care payments from CHESSIE. As ofNovember 30, 2008, the State has implemented its financial payments processing module andnow considers CHESSIE to be a fully implemented SACWIS.A significant area of concern as a result of the AAR is the amount of information that is stillmissing from the CHESSIE system. The amount of missing data is due mostly to conversionfrom the State’s legacy information system to CHESSIE. While Maryland staff have worked toensure that data from the legacy system was migrated to CHESSIE, the amount of missinginformation a year after conversion is not acceptable.The attached matrix outlines data elements that contain a large percentage of missing data. Thefindings of the case file review indicate this information does exist, but not in the official file inCHESSIE. Of particular concern is the amount of missing child placement information; it isexpected that this will improve when provider payments are made out of CHESSIE, though theState will still need to ensure that historical placement information is entered accurately in thesystem.Another reason that some data fields are missing information is the court reports that areproduced by CHESSIE do not meet Maryland’s court documentation requirements. Workers aretherefore using a separate template to complete and submit court reports, including adoptionfinalization paperwork. Maryland is currently working on modifying the report templates inCHESSIE to rectify this issue.We remind the State that, as a fully implemented SACWIS, CHESSIE is considered to be thesingle system of record for child welfare cases and must be capable of reporting all relevantAFCARS information. All data relative to AFCARS reporting must be derived solely fromCHESSIE.The amount of missing data has serious consequences for the State’s Child and Family ServicesReview. For example, the child’s case plan goal is missing for 3,846 children, or 45.65% of thepopulation for the 2008A reporting period. Additionally, some data fields do not show in theAFCARS frequency report as missing per se, but are likely missing historical information. Forexample, 7,399 children (or 87.82% of the foster care population in the report period) werereported as having one placement setting in the current removal episode. The case file reviewfindings indicated that of the 56 cases in the final analysis, 27 records were found to have errorsdue to the reviewers finding more placement moves than what was reported to AFCARS.US DHHS/ACF/ACYF/Children’s Bureau 4

Maryland staff have continued efforts to increase the use of CHESSIE by field staff andsupervisors. Some recommendations made during the March 2007 technical assistance site visitby the National Resource Center for Child Welfare Data and Technology (NRC-CWDT), such asa regular newsletter about CHESSIE and training that integrates both CHESSIE and Maryland’schild welfare policy, have been implemented.Maryland worked with the University of Maryland this past summer to perform data entry ofcase information. Graduate students helped complete data entry for out-of-home placementsover a period of several months. Currently, frontline staff are focusing on updating and ensuringthe accuracy of current placements. Maryland should continue to monitor for missing data andprovide feedback to local offices for correction.At the time of the visit, Maryland staff were also preparing to begin distributing a report thatcontains missing AFCARS data elements by child and unit. This should help facilitate data entryefforts and allow supervisors to monitor data completeness for cases in their units.Reporting PopulationIn general, the State is correctly reporting the required foster care and adoption population fromthe child welfare system. One exception is that States are required to report information on alladoptions in which the agency is involved, including all adopted children for whom the Stateagency is providing adoption assistance (either ongoing or for nonrecurring expenses), care orservices directly or by contract or agreement with other private or public agencies. (45 CFR1355.40(a)(3)). The State indicated it has adoption agreements for a subsidy and/or services withfamilies who adopted a child from a private adoption agency. Currently, these records are notbeing reported in AFCARS because the State’s information system does not have the capacity tocollect the information on cases that were not previous foster care cases. The system’s designrequires there to be an open foster care case before an adoption case can be created. At the timeof the AFCARS review the State team indicated they were reviewing possible solutions. TheState’s initial AFCARS Improvement Plan must include the State’s plan for making themodifications to the system and the timeline for implementing the changes.Technical RequirementsThere are two technical requirement items that were found to be non-compliant with AFCARSreporting. One is related to the design and structure of CHESSIE and how the adoption recordsare selected in the extraction code. The code does not exclude adoption cases that do not have afinal adoption legalization date. The primary problem is with the structure of the system.Currently, all adoption records that are in the adoption (“Case 3”) module in CHESSIE arereported. These cases include records in which an adoption payment is made to the family priorto the adoption finalization. Some of these adoptions have been finalized and some have not.The State staff discussed possible solutions to address this problem. The State may need todevelop an interim short-term solution to prevent records without a finalized adoption date frombeing submitted.US DHHS/ACF/ACYF/Children’s Bureau 5

The other technical area needing correction is the extraction and submission of subsequent 2 datafiles. One of the technical requirements for reporting the foster care and adoption files is that thedata must be extracted from the data system as of the last day of the reporting period (45 CFR1355.40(b)(1)). This means that data that is entered after the last day of the data collection cycleis not to be included in the regular 3 file. For the purposes of files that are resubmitted (i.e. asubsequent file), the data extracted are to reflect the circumstances of the case for the reportperiod being submitted. For instance, if a State is re-submitting the data for the period April 1,2006 – September 30, 2006 (2006B) on July 2, 2007, the data is to reflect the events of the casefor the 2006B report period. This means that if the child had a periodic review that occurred onMay 5, 2006, that is the date to be reported for foster care element #5, date of recent periodicreview. If the child’s case plan goal was reunification during the 2006B timeframe, but as ofJuly 2, 2007 the goal is adoption, the case plan goal for 2006B must be reported as reunification.The State’s current extraction process would be reporting newer information for many of the dataelements. This has implications for the data used in the Child and Family Services Review DataProfile, the Child Welfare Outcomes Report, and other analysis done by the Children’s Bureau.Data ElementsThis section highlights those areas that require the more significant changes to the informationsystem and/or the extraction routine. In regard to the adoption data elements, all of the elementsexcept for the State identification number, report period, record number, and State agencyinvolvement elements received a rating factor of “2.” This is primarily due to the neededchanges to CHESSIE and the extraction code for recording and extracting private agencyadoptions of a special needs child. The adoption matrix does indicate what the rating wouldhave been if the State were correctly recording and reporting these children.Race InformationThe most significant issue with the collection of race data is the use of fields for “primary” and“secondary” race. States are to collect and report all races that a person identifies. If a personidentifies with one or more races the worker may be hesitant to enter the information and make adetermination as to which race is the “primary” race. The State should consider consolidatingthese fields into one multi-select field.Also, there is a need to ensure that workers are asking individuals this information directly andincluding all races with which a person identifies, so that workers are not selecting a race basedon their own observations. The Federal team also provided the State with suggestions of otheroptions to add to the screen for workers to select. For instance, if the caseworker asks theindividual to identify all races, but the person does not want to give the worker that information,then the worker could select the option “declined.” In this way, it is clear to the worker whatoption should be selected and the State can track the frequency of this response.2 Submissions that are received after the AFCARS due dates for a regular report (May 15 or November 14) areconsidered subsequent data files.3 An AFCARS State data file for a current report period that is submitted during the appropriate transmission timeframe, April 1- May 15 and October 1-November 14.US DHHS/ACF/ACYF/Children’s Bureau 6

The State will also have to modify the collection of this information to meet the new NationalYouth in Transition Database (NYTD) race requirements. Guidance on these options and how tomap them for AFCARS purposes is provided in the attached matrices.Child’s Diagnosed ConditionsThis also is an area with a significant amount of missing data; 40% of the records in the 2008Areport were missing information. For AFCARS reporting purposes, the information reported forthese elements reflects chronic and/or significant diagnosed conditions that the child may have.CHESSIE contains several screens that include information pertaining to these elements (see theFindings document for more information). One area is a health section that contains severalitems related to these elements. Caseworkers are supposed to complete each screen/field.Consequently, this results in duplicative data entry, which results in issues related to dataaccuracy and SACWIS compliance. The State needs to make modifications to consolidate allthis information into one section of the system and make the corrections as noted above.Additionally, this correction will address the issue of data duplication raised by the Children’sBureau for SACWIS compliance. A resource list of conditions to be mapped to AFCARS islisted on the Children’s Bureau’s web page athttp://www.acf.hhs.gov/programs/cb/systems/afcars/resourcesIn addition, the system does not contain the question “has this child been diagnosed with adisability?” The AFCARS responses to this question have specific definitions. These are:“Yes” indicates that a qualified professional has clinically diagnosed thechild as having at least one of the disabilities listed below.“No” indicates that a qualified professional has conducted a clinicalassessment of the child and has determined that the child has no disabilities.“Not Yet Determined” indicates that a clinical assessment of the child by aqualified professional has not been conducted.Another significant issue with the State methodology for collecting these data is that there are nodates associated with the child’s diagnosed conditions. Consequently, a complete history is notmaintained to reflect the time frame the child may have each condition. This is especiallyproblematic for files that the State resubmits.Removal EpisodesFor AFCARS purposes, a removal episode is the period of time in which the State has care andplacement responsibility for a child. In AFCARS this is the time between the child’s “Date ofLatest Removal” (foster care element #21) and the child’s “Date of Most Recent Discharge”(foster care element #56). A child may have more than one episode if he or she was removedfrom home more than once. This is not to be confused with the number of placement settingsthat a child has experienced. Within one removal episode, a child may have multiple placementsettings.US DHHS/ACF/ACYF/Children’s Bureau 7

In CHESSIE, there are two screens on which the child’s discharge date and reason may beentered. The worker may enter a placement end date with a placement end type of “permanentlyleaving custody,” and enter a placement end reason that is identical to a custody end reason. Theworker may also enter a custody end date and reason on a separate screen. However, the workermay end a placement setting with a reason of “adoption” while the custody remains open.Therefore, the worker may think that the removal episode has been end-dated, when in fact it isstill open. Maryland staff indicated that they would develop a plan to streamline this process.We recommend that the placement end date be kept separate from the removal episode (orcustody end date) to the extent possible. This may mean that if a worker ends a placement with areason that corresponds with a custody end reason, the worker is taken to the custody screen toenter that this placement end does in fact constitute the child’s discharge from the care andplacement responsibility of the State. This helps to prevent a worker from erroneously entering adischarge when a placement change should have been recorded. On the other hand, the workermay think that the child has been recorded as discharged, when really only the placement hasbeen ended in CHESSIE.Also, for AFCARS reporting purposes, when a child’s only living arrangement during a removalepisode is a placement outside the scope of the Federal definition of foster care (i.e., detention ora hospital), this child is not to be considered for the AFCARS reporting population and theremoval episode is never included when reporting AFCARS. This affects foster care elements:#18, date of first removal from home; #19, total number of removals from home; #20, date ofdischarge from the previous foster care episode; and, #21, date of current removal episode.Additionally, for AFCARS reporting purposes, if the child’s first living arrangement was onethat is outside the scope of foster care and then the child moves to a foster care setting, theremoval episode start date is the start date of the foster care living arrangement. This date wouldalways be considered the start date of the the removal episode.The State is not correctly reporting the start date of removal episodes for children/youth that areeither on runaway status when the agency received responsibility for placement and/or runs awayprior to the first placement. If the child is still on runaway status as of the end of the AFCARSreport period, the child should be included in the AFCARS population with a placement status of“runaway” (foster care element #41). Runaway is considered a temporary absence from an ongoingplacement; children whose only placement setting is runaway during the removal episodewould be reported as having zero placement settings in the current episode (foster care element#24). The date the runaway status began will be reported for the date of the current placementsetting (foster care element #23). This date is also the start date of the removal episode (fostercare element #21).Lastly, one area that serves as contextual information for why children are entering foster careand what preventive services may be needed is in the area of “circumstances associated with achild’s removal from home.” This area appears to have significant underreporting. The casefile review findings indicate there were generally more conditions present that contributed to thechild’s removal than what is being recorded in the system. The State needs to provide additionalUS DHHS/ACF/ACYF/Children’s Bureau 8

training and oversight to ensure all conditions, alleged or substantiated, that were thecircumstances associated with the child’s removal are entered into CHESSIE.Placement Count (foster care element #24)The State is incorrectly excluding certain living arrangements in the number of placement movesfor the current removal episode. If a child is placed in a locked facility or in a hospital for nonacutecare, then these are to be counted as placement moves.Also, the extraction code is incorrectly counting as a placement move instances in which thefoster care provider’s identification number changes, but the child remains in the home. TheState will have to identify a method to maintain a link between the old number and the newnumber.Determination of a child’s special needs and the basis of the special needThere are multiple locations in CHESSIE where caseworkers can document whether the agencyhas determined that a child has a special need and the basis of special need. Additionally, notevery possible field is checked by the extraction code for reporting elements #9 and #10. TheState needs to modify the extraction code to check all fields in the system for this information.The State should also consider reviewing the system design to streamline the data entry processonto one screen.CONCLUSIONIn general, the State needs to incorporate the review of the AFCARS data as part of its ongoingquality assurance processes. Supervisors should also be instructed to review the data with thecaseworker and check for accuracy and timely data entry. One time that this may be convenientis when the caseworker is reviewing the case and preparing it for the six-month periodic review.The overall accuracy of the AFCARS data is crucial for the Children’s Bureau’s use of it in theChild and Family Services Review, the Child Welfare Outcomes Report to Congress, as well asmany other purposes.It will be especially important to stress supervisory oversight to ensure that this information iskept up-to-date. CHESSIE now contains the official documented case record for each child inthe care and placement responsibility of the State agency, and it is critical that this message isconveyed to all users.Once changes are made to the system and the extraction code, underlying data quality issues maysurface. These elements and those that received a rating factor of “3” will require additionaltraining and supervisory oversight for the timeliness and accuracy of data entry.Tab B contains the AFCARS Improvement Plan (AIP). The AIP contains the AFCARS generalrequirements and data elements that do not meet the requirements in the Federal regulations.Each matrix contains a column that identifies the task(s), the date the task is to be completed, anda column for comments.US DHHS/ACF/ACYF/Children’s Bureau 9

Within 30 calendar days after the receipt of this report and the attached AIP, the State staff mustsubmit the AIP electronically to the Children’s Bureau with estimated due dates for completingthe tasks in the AIP. An electronic copy of the final matrices will be e-mailed to your staff. TheState should provide electronic quarterly updates of its progress to the Children’s Bureau. Oncethe Children’s Bureau and the State agree that the quality of the data has improved, and all tasksand revisions to the extraction code have been reviewed and approved, the State will receive aletter summarizing the final results of the review.Additionally, the State’s plan for implementing the changes to the system and for caseworkertraining must be included in the State’s title IV-B Child and Family Services Plan and AnnualProgress and Services Report as part of the information required by 45 CFR 1357.15(t) and 45CFR 1357.16(a)(5).The Regional Office will work with the State to determine if technical assistance is needed, andavailable, to implement the AFCARS Improvement Plan. The State may obtain technicalassistance from the Children’s Bureau’s Network of Training and Technical Assistance Centers.US DHHS/ACF/ACYF/Children’s Bureau 10