Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY22?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s evaluation officer or equivalent (example: Evidence Act 313)?

• The director of OPE serves as the ACL evaluation officer. Responsible for overseeing the agency’s performance and evaluation work, OPE has seven full time staff positions and three full-time onsite contractors. The ACL OPE budget for evaluation was $9,600,000 in FY22.  The director of OPE has the education, skill, and experience to meet the evaluation officer requirements listed in the Evidence Act and routinely gauges the coverage, quality, methods, consistency, effectiveness, independence, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency as well as agency capacity to support the development and use of evaluation. The director is also the designated ACL performance officer.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer or equivalent? [Example: Evidence Act 202(e)]?

• The director of OPE serves as ACL’s chief data officer.  The director of OPE leads ACL’s data governance body, including facilitating collaborative activities among the numerous actors with responsibilities and needs for data within the agency; has demonstrated training and experience in data management, governance, collection, analysis, protection, use, and dissemination; and fulfills the aspects of this role that are relevant to ACL. These include coordinating with ACL’s chief information officer and chief privacy officer on use, protection, dissemination, and generation of data to ensure that the data needs of the agency are met; ensuring that agency data conform with data management best practices; engaging agency employees, the public, and contractors in using public data assets; and encouraging collaborative approaches on improving data use. The director of OPE also acts as the agency liaison to other federal entities through, for example, serving as the ACL representative to the HHS data council, and serving on the Federal Interagency Council on Evaluation Policy as well as the HHS Data Governance Board.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, performance improvement officer, and other related officials in order to support Evidence Act implementation and improve the agency’s major programs?

• The director of OPE serves the functions of evaluation officer, chief data officer, and performance officer. In order to coordinate activities relevant to these positions, the OPE director and staff coordinate the support, improvement, and evaluation of agency programs through implementation of an agency performance strategy, learning agenda, annual agency-wide evaluation plan, and through the National Institute for Disability, Independent Living, and Rehabilitation Research. This structure requires semiannual meetings with ACL leadership and management staff and annual consultation with all program managers. In FY19 ACL instituted a council to improve its data governance and quality, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data collected by ACL through its evaluations, grant reporting, and other administrative data collections. This robust governance structure ensures cohesive collection and use of evidence across ACL regarding program performance, evaluation, and improvement. It also ensures that data are gathered, processed, and curated so as to produce evidence that program staff and agency leadership use for program and operational improvement. As an operating division without a statistical unit, ACL does not have a statistical officer.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence building plan) and did it publicly release the findings of all completed program evaluations in FY22?

2.1 Did the agency have an agency-wide evaluation policy [Example: Evidence Act 313(d)]?

• The agency’s public evaluation policy confirms its commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. As addressed in this policy, ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. The policy was updated in 2021 to better reflect U.S. Office of Management and Budget (OMB) guidance provided in OMB memo M-20-12 and to more explicitly affirm ACL’s commitment to equity in evaluation.

2.2 Did the agency have an agency-wide evaluation plan [example: Evidence Act 312(b)]?

• An agency-wide evaluation plan was submitted to HHS in support of HHS’s requirement to submit an annual evaluation plan to OMB in conjunction with its Agency Performance Plan. The agency’s annual evaluation plan includes evaluation activities related to the learning agenda and any other “significant” evaluation, such as those required by statute. The plan describes the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or to inform decision-makers about current and future activities.

2.3 Did the agency have a learning agenda (evidence building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda (example: Evidence Act 312)?

• Based on the learning agenda approach that it adopted in 2018, ACL published an FY20–FY22 learning agenda in FY20. In developing the plan, ACL engaged stakeholders through meetings with program staff and grantees as required under OMB guidance provided in memo M-19-23. Most meetings with stakeholder groups, such as those in conjunction with conference sessions, were put on hold for 2020 due to COVID-19 travel restrictions. In 2021, ACL communicated with stakeholder groups to contribute to its learning activities. For example, ACL worked with members of the RAISE Family Caregiving Advisory Council and a range of stakeholders to inform changes to the 2021 data collection under the National Survey of Older Americans Act Participants. In 2021, ACL also released a request for information directed at small businesses to solicit research approaches related to its current research priorities.

2.4 Did the agency publicly release all completed program evaluations?

• The Administration for Community Living releases all completed evaluation reports and studies, ongoing studies, and evaluation design projects according to its evaluation policy.

2.5 Did the agency conduct an Evidence Capacity Assessment that addressed the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts [example: Evidence Act 315, subchapter II (c)(3)(9)]?

• Staff from OPE play an active role in HHS’s capacity assessment efforts, serving on the Capacity Assessment and Learning Agenda Subcommittees of the HHS Evidence and Evaluation Council. The HHS 2023-2026 Capacity Assessment discusses ACL’s contributions to the coverage, quality, methods, effectiveness, and independence of the agency’s statistics, evaluation, research, and analysis efforts.  The agency’s self-assessment results were provided to HHS to support its ability to submit the required information to OMB. These results  provided information about planning and implementing evaluation activities, disseminating best practices and findings, incorporating employee views and feedback, and carrying out capacity building activities in order to use evaluation research and analysis approaches and data in day-to-day operations. Based on this information, in 2021 ACL focused on developing educational materials for its staff and data improvement tools for ACL grantees. In 2021 the ACL Data Council published a guide to evaluation system change initiatives,  as well as additional documents to promote responsible data usage: Data Quality 201: Data Visualization and Data Quality 202: Data Quality Standards. While designed initially for ACL staff, these documents are available on the ACL website and have been promoted through several industry conferences.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• Starting in 2020 and continuing into 2021, ACL is funding contracts to design the most rigorous evaluations appropriate to measure the return on investment of the Aging Network, the extent to which ACL services address social determinants of health, and the value of volunteers to ACL programs. The agency sometimes funds evaluation design contracts, such as those for the Older Americans Act Title VI Tribal Grants Program evaluation and the Long Term Care Ombudsman evaluation that are used to determine the most rigorous evaluation approach that is feasible given the structure of a particular program. While the Ombudsman program is full coverage programs, where comparison groups are not possible, ACL most frequently uses propensity score matching to identify comparison group members. This was the case for the Older Americans Act Nutrition Services Program and National Family Caregivers Support Program evaluations and the Wellness Prospective Evaluation Final Report conducted by Centers for Medicare and Medicare Services in partnership with ACL.
• The agency’s  NIDILRR funds the largest percentage of its randomized control trials (151 of 659  or 23%) for research projects employing a randomized clinical trial. To ensure adequate quality, NIDILRR adheres to strict peer reviewer evaluation criteria in the grant award process. In addition, ACL’s evaluation policy states that “in assessing the effects of programs or services, ACL evaluations will use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative.”

Did the agency invest at least 1% of program funds in evaluations in FY22 (examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance; and rigorous evaluations, including random assignments)?

3.1 ____ invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY22 budget.

• The amount invested in evaluation technical assistance, and evaluation capacity-building as a percentage of the agency’s $3,009,000,000 FY22 enacted budget was not reported. However, from FY17 to FY21 ACL funded evaluation at a consistent level.

3.2 Did the agency have a budget for evaluation and how much was it (were there any changes in this budget from the previous fiscal year)?

• The ACL OPE budget for evaluation was $9,600,000 in FY22, down from $13,600,000 in FY21.   The bulk of OPE’s evaluation funds are based on a set-aside required in Title II, section 206 of the Older Americans Act (OAA): “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1/2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.” In addition, in 2017 ACL’s OPE established a mechanism that allows ACL programs not covered by the OAA set-aside to transfer funds to OPE to support evaluations of their programs. In FY17, FY18, FY19, FY20,  FY21, and FY22 OPE added approximately $1,000,000, $1,700,000, $3,200,000, $1,200,000, $2,300,000, and $1,600,000 from these programs to its evaluation budget, respectively.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• The Administration for Community Living primarily provides information resources to grantees to build their evaluation and evidence building capacity. Staff record trainings on evaluation topics, including an overview of performance measurement. Several resources and technical assistance centers focus on evidence building including one contract dedicated to improving performance data provided by Older Americans Act Title III, VI, and VII grantees that offers live and prerecorded webinars and a range of manuals and technical assistance supports. The agency also published toolkits for strategic planning, data quality, performance measures, logic model development, and more. It provides technical assistance to grantees related to using evidence-based programs and building evidence. For example, the National Resource Center on Nutrition and Aging provides different programs and approaches that deliver nutrition-related home- and community-based services (HCBS) administered through grants to the states and territories. Access to Respite Care and Help (ARCH) provides training and technical assistance to the Lifespan Respite Network with a focus on performance measurement, sustainability, best practices, and research. The National Alzheimer’s and Dementia Resource Center supports grantees as they implement evidence-based interventions and innovative practices designed to empower and assist caregivers of persons with Alzheimer’s disease and related disorders.
• In an effort to better support Older Americans Act’s Title VI Native American Programs grantees with their data needs, OPE created several tools for grantee use. One tool is a template for collecting program satisfaction survey data from both nutrition and caregiver clients about the services they are receiving. This template is customizable, has a user guide and an Excel spreadsheet for easy data tabulation and visualization as it creates charts from the entered data. The office also worked to create an infographic for Title VI grantees to use by plugging in their own data (into an Excel template with the help of a user guide) from a variety of sources to which they have ready access so that they might create a two-page visual document to share with stakeholders to showcase the services they are providing to their communities. In addition, OPE created a Title VI Data Tracking Workbook, built with grantee input, to help grantees track their data on a daily basis. This workbook provides monthly and quarterly statistics and produces an annual roll-up of data. Based on grantee feedback, it has some customization built in and has a budget feature to help grantees manage their income and expenditures, data that ACL does not currently collect for this program. In an effort to make these tools more user friendly for grantees and to encourage their wide adoption, OPE has held webinars, trainings, and question sessions.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY22?

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• As part of the HHS Annual Performance Plan and Report, ACL reports on the following two HHS Agency Priority Goals: (1) to Increase the success rate of the Protection and Advocacy Program’s individual or systemic advocacy, thereby advancing the right of individuals with developmental disabilities to receive appropriate community based services, resulting in community integration and independence, and to have other rights enforced, retained, restored, and/or expanded; and (2) to Improve dementia capability of long-term support systems to create dementia friendly, livable communities (ACL as lead agency). Outcomes measures are available, by program, in ACL’s annual Congressional Budget Justification and include measures of program efficiency. Annual reports to Congress are submitted by ACL’s Administration on Disability, Administration on Aging, and NIDILRR. In addition, ACL contributes to other department-wide reports to Congress such as the HHS Report to Congress on Minority Health.

4.2 Did the agency use data/evidence to improve outcomes and return on investment?

• The Administration for Community Living employs a moderate approach for analyzing evidence to find ways to improve return on investment that address multiple parts of the agency. In FY20, as part of its ongoing effort to ensure that agency funds are used effectively, ACL funded a three-year contract, focused on ACL’s Administration on Aging, to identify approaches to measure how and to what extent parts of the Aging Network leverage Older Americans Act funds to increase their available resources as well as how the Aging Network uses resources to measure and improve the quality of services available/provided. This evaluation is ongoing. As part of its new employment research agenda, NIDILRR conducts research to continue development of return-on-investment models that can be used by vocational rehabilitation agencies to optimize the services they provide. In addition, in January 2021 ACL announced a new phase for the Innovative Technology Solutions for Social Care Referrals challenge competition. It also recently published the results of a study measuring the economic value of volunteerism for Older Americans Act programs.

4.3 Did the agency have continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement (examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)?

• As part of ACL’s performance strategy and learning approach, OPE staff present performance data to ACL leadership several times a year. Performance data are reviewed by ACL leadership as part of the budget justification process that informs program funding decisions. To report performance measure data and results and to discuss methods for incorporating performance and evaluation findings into funding and operational decision-making, OPE staff conducts annual meetings with ACL staff. As part of annual evaluation planning efforts, OPE staff consult with ACL center directors to identify evaluation priorities and review proposed evaluation approaches to ensure that the evaluation questions identified will provide information that will be useful for program improvement. Two projects started in late 2020 with the goal of improving agency performance are a study of how the services provided by ACL grantees influence the social determinants of health and an evaluation of how ACL supports grantee use of evidence-based programs that are required under Title IIID of the Older Americans Act. Results from the social determinants of health study will be posted in FY2023. In 2021 ACL began using the National Standards for Culturally and Linguistically Appropriate Services (CLAS) in Health and Health Care to inform its evaluation framework. Specifically, ACL funded this project to explore the extent to which its grantees employ CLAS standards in their service delivery processes, particularly their responsiveness to cultural practices, language and communication needs, LGBTQ+ needs, and health literacy.  The report from this study will be posted in FY23. It also funded a study to examine the use of and financial value of volunteers to its programs. In addition to a final report, ACL developed an effective practice guide to help grantees use volunteers effectively.

Did the agency collect, analyze, share, and use high-quality administrative and survey data consistent with strong privacy protections to improve (or help other entities improve) outcomes, cost effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY22 (examples: model   data-sharing agreements or data-licensing agreements, data tagging and documentation, data standardization, open data policies, and data use policies)?

5.1 Did the agency have a strategic data plan, including an open data policy [example: Evidence Act 202(c), Strategic Information Resources Plan]?

• As an operating division of a Chief Financial Officers Act agency, HHS, ACL is not required to have its own strategic data plan and utilizes HHS’s data strategy. In 2016, ACL implemented a Public Access Plan as a mechanism for compliance with the White House Office of Science and Technology Policy’s public access policy. The plan focused on making published results of ACL/NIDILRR-funded research more readily accessible to the public, making scientific data collected through ACL/NIDILRR-funded research more readily accessible to the public, and increasing the use of research results and scientific data to further advance scientific endeavors and other tangible applications. In 2019, ACL created a council to improve its data governance, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data collected through its evaluation, grant reporting, and administrative performance measures. In 2020, its first year, the ACL Data Council produced an annotated bibliography to provide essential background information about the topic, a primer to detail best practices in data governance specifically as they apply to ACL, and a Data Quality 101 infographic to guide decision-making processes related to data quality. The next phase of the ACL Data Council is currently under review.

5.2 Did the agency have an updated comprehensive data inventory (example: Evidence Act 3511)?

• The Administration for Community Living provides comprehensive public access to its programmatic data through its Aging, Independence, and Disability Program Data Portal (AGID), which is currently being rebuilt. It also has two data inventories available to the public on the NARIC website: REHABDATA, a database of rehabilitation and disability literature, and the Online Program Directory, which contains NIDILRR’s previously funded, currently funded, and newly funded grants. An ACL/NIDILRR public access plan, first published in February 2016, makes available to the public peer-reviewed publications and scientific data arising from research funded in whole or part by ACL through the NIDILRR to the extent feasible and permitted by law and available resources. The requirements outlined in this plan are being applied prospectively and not retrospectively. In addition, ACL is creating an internal evidence inventory that staff will be able to use to search for relevant program performance and evaluation data by agency priority question.

5.3 Did the agency promote data access or data linkage for evaluation, evidence building, or program improvement [examples: model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; and downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)]?

• The OPE has access to all of ACL’s performance and evaluation data and is able to link those data and advise programs about their availability and usability. In March 2019, ACL completed the ACL Data Restructuring Project to assess the data hosted on AGID and to develop and test a potential restructuring of the data in order to make these data useful and usable for stakeholders. In 2021, ACL published a final report on the Data Restructuring II Project in which ACL awarded a follow-on contract to further integrate its datasets along the lines of conceptual linkages and to better align the measures within its data collections. It funded several grants to promote data linkage. Grants to Enhance State Adult Protective Services (APS) were awarded in FY19 to increase intra- and inter-state sharing of information on APS cases. The 2020 Empowering Communities to Reduce Falls and Falls Risk grant, which was awarded to develop robust partnerships and a result-based comprehensive strategy for reducing falls and fall risks among older adults and adults with disabilities living in the community, directs grantees to consider Centers for Disease Control and Prevention opportunities to broaden and improve the linkage between primary care providers and evidence-based community falls prevention programs supported by ACL.

5.4 Did the agency have policies and procedures to secure data and protect personal confidential information (example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)?

• As an operating division of HHS, ACL follows all departmental guidance regarding data privacy and security. This includes project-specific reviews by ACL’s Office of Information Resource Management, which monitors all of ACL’s data collection activities to ensure the safety and security of ACL’s data assets. In FY19, ACL awarded a contract to establish a data council to enhance the quality, security, and statistical usability of the data collected by ACL through its evaluation, grant reporting, and administrative data collections and to develop effective data governance standards.
• The NIDILRR model systems’ data centers have extensive standard operating procedures that are designed to secure data and protect personal and confidential information. Below are a few illustrative examples from the these data centers:
  • The Burn Model System National Data and Statistical Center has a procedures page that lists all of the standard operating procedures that grantees contributing to this database must follow.
  • The Traumatic Brain Injury Model Systems National Data and Statistical Center has a standard operating procedures page that describes the procedures that all grantees contributing to this database must follow.
  • The Spinal Cord Injury Model Systems National Data and Statistical Center has a page on Using the National Spinal Cord Injury Model Systems Database. Descriptions of what constitutes “de-identified data” can be found on this page.
• In addition to the model systems data centers referenced above, NIDILRR developed Part 2: Preparing Data and Documentation. This page and video are part of the larger training course, NIDILRR Data Archiving and Sharing Training, that NIDILRR grantees must complete. Additional guidance is available on the ICPSR web page Resources for National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) Grantees. Each funding opportunity announcement states that “a data and safety monitoring board (DSMB) is required for all multi-site clinical trials involving interventions”

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• Through the AGID system, which is currently being redeveloped to meet current privacy, security, and technology needs, ACL data sets are made publicly available. While AGID is under construction, ACL is releasing as much data publicly as is feasible given resources and capacity. Technical assistance is provided by ACL staff through presentations and by ACL’s technical assistance resource centers to grantees, including state, tribal, and local governments. The resource centers providing technical assistance include the National Resource Center on Nutrition and Aging, the Alzheimer’s Disease Supportive Services Program, and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service. This technical assistance includes annual workshops and presentations at the Title VI National Training and Technical Assistance Conference and training available through the ACL funded National Ombudsman Resource Center and the Disability and Rehabilitation Research Program, which funds capacity building for minority research entities. In addition, NIDILRR has a number of resources to help the public access its data responsibly: The National Spinal Cord Injury Statistical Center, for example, has a pdf document entitled Using the National Spinal Cord Injury Model Systems Database. This same center also has an online Data Request Form that requestors need to complete before gaining access to data. The National Data and Statistical Center for the Traumatic Brain Injury Model Systems has a web page entitled How to obtain a dataset from the TBIMS. Similarly, the Burn Model Systems’ National Data and Statistical Center has a page with instructions on how to access data.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY22? (Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• The Administration for Community Living defines evidence-based programs on its website. Its NIDILRR has two principal frameworks that guide and inform the generation of new knowledge and products. The stages of research framework is used to guide, inform, and track the creation of new knowledge that in turn becomes part of the larger disability evidence base. The stages of development framework is used to guide, inform, and track the development of new products and technologies. Both of these frameworks are codified in federal regulations and are described on NIDILRR’s Frameworks’ page on the ACL website. The stages of research framework is codified in 45 CFR 1330.4 while the stages of the development framework is codified in 45 CFR 1330.5.

6.2 Did the agency have a common evidence framework for funding decisions?

• The Older Americans Act requires the use of evidence-based programming in Title III-D-funded activities: Disease Prevention and Health Promotion Services. In response, ACL developed a definition of evidence-based programs and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for activities funded by the Older Americans Act. For programs that are not legislatively required to use evidence-based models, ACL, through its funding process, requires all programs to provide clear justification and evidence (where available) that proposed projects will achieve their stated outcomes. In 2018 ACL developed a tool to help a small number of program officers assess grantee progress toward the stated goals of their grants. Using the tool program officers have instituted corrective actions or required underperforming grantees to relinquish grant funds. The agency is developing similar tools for several other grant programs with the intention of rolling out new guidance for program officers in 2023.

6.3 Did the agency have a clearinghouse(s) or a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• The Administration for Community Living does not have a common evidence repository that applies across the entire agency. It publishes intervention summaries of aging and disability evidence-based programs and practices. It funds the Evidence-Based Program Review Council to identify new community programs that meet the criteria established by the Administration for Community Living/Administration on Aging for evidence-based programs funded through the OAA Title III-D. The Model Systems Knowledge Translation Center has worked with NIDILRR’s model systems grantees to develop and publish a variety of evidence-based factsheets about living with spinal cord injury, traumatic brain injury, or burn injury. The ACL Living Well demonstration program requires grantees to use evidence-based and innovative strategies to (1) improve access to and quality of community services; (2) reduce and mitigate abuse and neglect; and (3) support empowerment, self-determination, and self-advocacy.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• To drive improvements in outcomes for older adults and individuals with disabilities, ACL works through its resource centers to help grantees use evidence. For example, with funding from ACL, the National Center on Aging, in collaboration with the Evidence-Based Leadership Council, led an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process resulted in adding six new health promotion programs and three new programs for preventing falls. The Alzheimer’s Disease Supportive Services Program funds competitive grants to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementia and their family caregivers. Extensive evaluation of the National Chronic Disease Self-Management Education and Falls Prevention database helped generate important insights for potential new ACL applicants in preparing their applications using data-driven estimation procedures for participant and completion targets. In addition, ACL funded several grants, such as the Lifespan Respite Care Program: State Program Enhancement Grants and Disability and Rehabilitation Research Projects Program: Chronic Disease Management for People with Traumatic Brain Injury, which are designed in part to develop an evidence base for respite care and related services and to contribute to the evidence base upon which people with traumatic brain injury and their health care providers can use effective chronic disease management practices, respectively. Moreover, NIDILRR provides the Rehabilitation Measures Database, which is a knowledge translation on-line resource with succinct summaries of instruments relevant to rehabilitation populations that provides evidence-based summaries including concise descriptions of each instrument’s psychometric properties, reliability, validity, sensitivity, instructions for administering and scoring, and a representative bibliography with citations.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY22 (examples: prizes and challenges, behavioral science trials, innovation labs/accelerators, performance partnership pilots, and demonstration projects or waivers with rigorous evaluation requirements)?

7.1 Did the agency have staff dedicated to learning innovation efforts to improve the impact of its programs?

• Agency leadership promotes innovation by requiring all program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. The Administration for Community Living also partially funds a Forum on Aging, Disability, and Independence, which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. It funds resource centers, such as the Engagement and Older Adults Resource Center, which provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.

7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?

• In FY21 all ACL Centers were involved in funding innovative work. The agency released several funding opportunity announcements focused on the identification and implementation of innovative approaches to improve programming. These included funding opportunity announcements for the Innovations in Nutrition Programs and Services – Community Research and the ACL/NIDILRR Small Business Innovation Research Program (SBIR) Phase I.  In addition to challenge competitions such as the ACL Social Care Referrals Innovation Challenge, in 2021 ACL funded a small study to identify the innovations and adaptations by grantees in the face of COVID-19. In 2022 and 2023, ACL will more closely examine the most promising innovations to evaluate their effectiveness.
• The Administration for Community Living is a funder of The John A. Hartford Foundation 2022 Business Innovation Award, which recognizes aging and disability community-based organizations for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities.
• There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the agency’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3,500,000 to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• The 2020 reauthorization of the Older Americans Act requires a new research, demonstration, and evaluation center for the Aging Network; new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches toward coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e., Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
• The research and development activities of NIDILRR are guided by the stages of research framework and the stages of development framework. Grantees conducting research projects must identify the current state of research of their projects while grantees conducting development projects must identify the current state of development of their projects.  The stage of a research project depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the intervention development phase. The next phase, known as intervention efficacy, refers to the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as scale-up evaluation, is the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting.
• Similarly, the stage of development for a development project depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage refers to the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants; verifying product requirements; implementing and testing (typically in controlled contexts) key concepts, components, or systems; and resolving technical challenges. The proof of product stage is the stage of development where a fully integrated and working prototype meeting critical technical requirements is created. The proof of adoption stage is the stage of development where a product is substantially adopted by its target population and used for its intended purpose.

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY22 (examples: tiered-evidence frameworks, evidence-based funding set-asides, priority preference points or other preference scoring for evidence, and pay for success provisions)?

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY22, the five largest competitive grant programs are:
  1. Centers for Independent Living: $112,900,000; eligible applicants: nonprofits and public and state controlled institutions of higher education (one of its competitive grants was the Centers for Independent Living Training and Technical Assistance Grant).
  2. National Institute on Disability, Independent Living, and Rehabilitation Research: $113,000,000; eligible applicants: state, local, and tribal governments and nonprofits and public and state controlled institutions of higher education (one of its competitive grants was the Disability and Rehabilitation Research Projects).
  3. State Health Insurance Assistance Program: $52,100,000; eligible applicants: unrestricted.
  4. Medicare Improvements for Patients and Providers Act Programs (MIPPA): $50,000,000; eligible applicants: nonprofits, city or township governments, public and state controlled institutions of higher education, Native American tribal organizations, public housing authorities/Indian housing authorities, private institutions of higher education, special district governments, county governments, state governments, and Independent school districts (one of its competitive grants is the MIPPA Medicare Beneficiary Outreach and Assistance Program: Funding for OAA Title VI Native American Programs).
  5. Alzheimer’s Disease Programs Initiative states and community grants: $24,500,000; eligible applicants: state, local, and tribal governments and nonprofits and public and state controlled institutions of higher education.

8.2 Did the agency use evidence of effectiveness to allocate funds in the five largest competitive grant programs (e.g., were evidence-based interventions/practices required or suggested and was evidence a significant requirement)?

• Independent living notice of funding announcements describe evaluation criteria including plans for technical assistance to enhance grant effectiveness and the provision of information developed about best practices. To continue receiving centers for independent living program funding, eligible centers must provide evidence that they have previously had an impact on the goals and objectives for this funding.
• Based on a strict interpretation of the phrase “evidence of prior effectiveness to make grant awards,” NIDILRR currently does not use evidence of prior effectiveness to make grant awards. Instead, ACL makes these grant awards by largely relying on the expert evaluative judgments of ACL peer reviewers. Making grant awards by using peer review is a standard and widely accepted evidenced-based practice. For example, see pages 7 and 19 of the full DPCP full announcement.
• State health insurance assistance program notices of funding announcements describe evaluation criteria including plans to improve alignment of policies, processes, and procedures to program goals and increased accountability to program expectations at all levels (full announcement, p. 25).
• Funds for Medicare Improvements for Patients and Providers Act Programs are awarded to state grantees and to the National Center for Benefits Outreach and Enrollment. To continue funding without restrictions, state grantees are required to submit state plans that ACL staff review for the specific strategies that grantees will employ to enhance efforts through statewide and local coalition building. National Center applicants must describe the rationale for using the particular intervention, including factors such as evidence of intervention effectiveness. In 2019, the Center was awarded additional funding based on prior performance—specifically, assisting over 7,600,000 individuals to identify over $29,600,000,000 in potential annual benefits.

8.3 Did the agency use its five largest competitive grant programs to build evidence (e.g., requiring grantees to participate in evaluations)?

• Independent living grant recipients are required to show that they are working to “improve performance, outcomes, operations, and governance of CILs.” (Full Announcement, p. 2).  These include reports on “issues, goals, outcome measures, performance and compliance” (p. 6).
• The National Institute on Disability, Independent Living, and Rehabilitation Research and its grantees are in the disability and rehabilitation evidence-building business. Grantees generate new knowledge on particular disability topics or develop new disability products that eventually become part of a larger evidence base. To generate this new knowledge, NIDILRR grantees must conduct a series of research and development activities that produce important outputs. These research and development activities are guided by the NIDILRR stages of research framework (45 CFR 1330.4), and the NIDILRR stages of development framework (45 CFR 1330.5).
• State Health Insurance Assistance Program  grantees are required to build and disseminate evidence of what works through documenting and promoting “knowledge, successes, and lessons learned within the SHIP network. This includes sharing ideas, products, and materials with other SHIP grantees, ACL, and the SHIP Technical Assistance Center” (Full Announcement, p. 5).  They are required to report on specified performance measures, but also encouraged to provide additional evidence and data, such as data related to the cost changes as a result of enrollment in Medicare Part D and Medicare Advantage plans (p. 7).
• Medicare Improvements for Patients and Providers Act Programs grant funds support the identification and dissemination of practices built upon evidence of effectiveness (i.e., practices improving benefits outreach and enrollment).

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs (besides its five largest grant programs)?

• The agency requires that evidence of effectiveness is used in all award decisions. Grant officers attend training regarding ways to include information about evidence building into funding opportunity announcements. This includes information about text that can be included in funding announcements (1) describing requirements for developing measurable outcomes, (2) explaining how the inclusion of evidence and evidence building plans can be used to score grant applications, and (3) instructing grant reviewers regarding rating applicants’ presentation of evidence and evidence building plans. The training was recorded and is available to all staff.
• The Alzheimer’s Disease Programs Initiative translates and implements evidence-based supportive services for persons with Alzhemimer’s disease and related dementia and their caregivers at the community level. Award criteria include the extent to which applicants “describe partnerships, collaborations and innovative activities that will be implemented in support of goal/objective achievement, including the dementia specific evidence-based/evidence informed intervention(s) to be implemented in the project” (Full Announcement, p. 24).
• The review criteria for the Lifespan Respite Care Program: State Program Enhancement Grants includes the applicant’s description of “how the proposed project will build upon the accomplishments made in previous Lifespan Respite Care Program grants” (Full Announcement, p. 23).

8.5 What are the agency’s strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Prior to the development of visual scene displays by the NIDILRR-funded Augmentative and Alternative Communication Rehabilitation Engineering Research Center (AAC RERC), the only augmentative and alternative communication option was traditional grid displays with isolated symbols presented in rows and columns. It was difficult for many adults with acquired conditions resulting in significant language and cognitive limitations to use these traditional grid displays. Visual scene displays offer an easier alternative to traditional grid displays. They go beyond standard pictures and symbols organized in rows and columns by providing information on the situation or context. Put more simply, visual scene displays are photos or pictures that people can use to communicate messages to others. These photos depict familiar scenes, objects or people, and users can touch “hot spots” on the photo to speak messages that relate to the pictured scene or object. For example, a person with aphasia might touch a hotspot on a picture of a sibling and say this is my sister. This additional information on the situation and context makes it easier for persons with complex communication needs to express their wants and needs and therefore enhances their ability to interact and participate with others in the community. Research from the AAC RERC and external researchers demonstrates the effectiveness of visual scene displays for adults with severe chronic aphasia, primary progressive aphasia, and dementia, for example. As a result of the continued efforts of the AAC-RERC and its partners, this visual scene display technology has been successfully transferred to all of the major AAC manufacturers and app developers.
• Grant activities funded by NIDILRR regularly produce publications that use evidence to build knowledge and promote diversity and inclusion. This includes recommendations for reducing barriers to access to health care that face coverings pose, particularly to the deaf and hard of hearing. They also produced a mixed methods study identifying barriers to access to health care faced by individuals with disabilities and also identifying aspects of the Affordable Care Act that have improved enforcement of laws prohibiting discrimination on the basis of disability.

8.6 Did the agency provide guidance that makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity building efforts?

• Funding opportunity announcements and grant reviews stress the need for strong performance measurement and evaluation. The Administration for Community Living’s technical assistance centers—the National Resource Center on Nutrition and Aging, the Alzheimer’s Disease Supportive Services Program, and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service—promote the use and generation of evidence with ACL grantees. Grantee manuals also include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). Staff of ACL’s OPE make presentations regarding the importance of evidence with regional staff who are in frequent contact with state grantees and at grantee conferences (see ACL Track: The ACL Older Americans Act (OAA) Performance System—Crossing the Finish Line and ACL/CMS Track: Raising the Bar in Medicaid HCBS & Community Inclusion—Showcasing Transformation presented at the 2019 HCBS conference and ACL Track: Assuring the Health & Welfare of Medicaid HCBS Beneficiaries and ACL Track: Innovative Housing & Health & Human Services Collaborations presented at the 2018 HCBS conference).

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY22  (examples: evidence-based funding set-asides, requirements to invest funds in evidence-based activities, and pay for success provisions)?

9.1 What were the agency’s five largest noncompetitive programs and their appropriation amounts (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY22, the five largest noncompetitive grant programs are:
  1. Nutrition Services: $951,800,000; eligible applicants: states.
  2. Home and Community Based Supportive Services: $392,600,000; eligible applicants: states.
  3. Caregiver Support Services: $188,900,000; eligible applicants: states.
  4. State Councils on Developmental Disabilities: $79,000,000; eligible applicants: states and nonprofits based in a state.
  5. Developmental Disabilities Protection and Advocacy: $41,800,000; eligible applicants: state and nonprofits based in a state.
• As these are based on formula grants, the funding amount distributed to the states and tribal organizations are not determined using evidence-based application processes. Rather, the states and tribal organizations are responsible for directing the funds to evidence-based programs and organizations.

9.2 Did the agency use evidence of effectiveness to allocate funds in the largest five noncompetitive grant programs (e.g., are evidence-based interventions/practices required or suggested and is evidence a significant requirement)?

• Authorizing legislation for ACL’s largest noncompetitive grant programs requires consideration of evidence-based programming as a requirement of funding. The Developmental Disabilities Assistance and Bill of Rights Act of 2000 allows for the withholding of funding if “(1) the council or agency has failed to comply substantially with any of the provisions required by section 124 to be included in the State plan, particularly provisions required by paragraphs (4)(A) and (5)(B)(vii) of section 124(c), or with any of the provisions required by section 125(b)(3); or (2) the Council or agency has failed to comply substantially with any regulations of the Secretary that are applicable.” As a condition of funding, noncompetitive grantees are required to “determine the extent to which each goal of the Council was achieved for that year” and report that information to ACL.
• States that receive Older Americans Act Home and Community-Based Supportive Services Title III-D funds are required to spend those funds on evidence-based programs to improve health and well-being and reduce disease and injury. In order to receive funding, states must utilize programs that meet ACL’s definition of evidence-based programs or are defined as evidence-based by another HHS operating division. Under the Older American Act, caregiver support programs are required to track and report on their use of evidence-based caregiver support services.

9.3 Did the agency use its five largest noncompetitive grant programs to build evidence (e.g., requiring grantees to participate in evaluations)?

• The Administration for Community Living’s Nutrition Services provides grants for innovations in nutrition programs and services. These research projects must have the potential for broad implementation and demonstrate potential to improve the quality, effectiveness, and outcomes of nutrition service programs by documenting and proving the effectiveness of these interventions and innovations. They must also target services to underserved older adults with greatest social and economic need and individuals at risk for institutional placement to permit such individuals to remain in home and community-based settings. Consistent with its focus on identifying new ways to efficiently improve direct service programs, ACL is using its 1% nutrition authority to fund $3,500,000 in nutrition innovations and to test ways to modernize how meals are provided to a changing senior population. One promising demonstration currently being carried out by the Georgia State University Research Foundation is the Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission. This demonstration has drawn widespread attention for its effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
• Under Home and Community-Based Services, FY12 Congressional appropriations included an evidence-based requirement for the first time. Older Americans Act Title III-D funding may be used only for programs and activities demonstrated to be evidence based. The National Council on Aging maintains a tool to search for evidence-based programs that are approved for funding through OAA Title III-D.
• The agency’s Caregiver Support Services builds evidence in a number of areas. These include a national survey of caregivers of older adult clients, gathering and reporting best practices regarding grandparents raising grandchildren, adapting and scaling evidence-based programs for children and older adults with disabilities through the RESILIENCE Rehabilitation Research and Training Center, and other similar efforts.
• State Councils on Developmental Disabilities design five-year state plans that address new ways of improving service delivery. To implement the state plans, councils work with different groups in many ways, including funding projects to show new ways that people with disabilities can work, play, and learn and seeking information from the public and from state and national sources.
• State Protection and Advocacy Systems encompass multiple avenues of protection and advocacy including specialization in individuals with developmental disabilities, assistive technology, voting accessibility, individuals with traumatic brain injury, and technical assistance. The Developmental Disabilities Assistance and Bill of Rights Act of 2000 requires Administration on Intellectual and Developmental Disabilities grantees to report annually on progress achieved through advocacy, capacity building, and systemic change activities.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other noncompetitive grant programs (besides its five largest grant programs)?

• The 2020 reauthorization of the Older Americans Act requires that assistive technology programs are “aligned with evidence-based practice;” that person-centered, trauma informed programs “incorporate evidence-based practices based on knowledge about the role of trauma in trauma victims’ lives;” and that a newly authorized Research, Demonstration, and Evaluation Center for the Aging Network increases “the repository of information on evidence based programs and interventions available to the aging network, which information shall be applicable to existing programs and interventions, and help in the development of new evidence-based programs and interventions.”

9.5 What are the agency’s  one or two strongest examples of how noncompetitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Since 2017, ACL has awarded Innovations in Nutrition grants to forty-three organizations to develop and expand evidence-based approaches to enhance the quality and effectiveness of nutrition programming. It is currently overseeing ten grantees for innovative projects that will enhance the quality, effectiveness, and outcomes of nutrition services programs provided by the National Aging Services Network. The grants total $2,218,419 for this FY22.  In FY22, ACL awarded a total of $1,448,797 funding for the first year of three 5-year research grants. Through this grant program, ACL aims to identify innovative and promising practices that can be scaled across the country and to increase the use of evidence-informed practices within nutrition programs.

9.6 Did the agency provide guidance that makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• All funding opportunity announcements published by ACL include language about generating and reporting evidence about their progress toward the specific goals set for the funds. Grantee manuals include information about the importance of and requirements for evaluation. The National Ombudsman Resource Center, funded by ACL, provides self-evaluation materials for long-term care ombudsman programs  funded under Title VII of the Older Americans Act.

In FY22, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes (examples: requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; and using rigorous evaluation results to shift funds away from a program)?

10.1 Did the agency have a policy for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• Because much of ACL’s funding is based on noncompetitive formula grants that cannot be reallocated to other programs or grantees, there is not an ACL-wide policy for this purpose. For several programs, such as most under the Older American Act, “entities such as states, U.S. territories, and tribal organizations are allotted funding based on a population-based formula factor (e.g., aged 55 and over, aged 60 and over, or aged 70 and over). Some statutory requirements for program funding allocations include a “hold harmless” provision, which guarantees that state or other entities’ allotments will remain at a certain fiscal year level or amount, provided sufficient funding in a given year.

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• While much of ACL’s funding is based on noncompetitive formula grants, and therefore cannot be reallocated to other programs, evaluation staff work closely with program staff to identify ways to translate evaluation findings into technical assistance and other types of program support. For example, based on early results from an evaluation of the Tribal grant program, ACL has developed new program support materials to improve the delivery of Tribal caregiver programs.
• The agency typically proactively provides technical assistance in order to help programs to be successful, rather than redirecting funding. For example, the State Health Insurance Assistance Program funded a national technical assistance center for this purpose.