2022 Federal Index

Innovation

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY22 (examples: prizes and challenges, behavioral science trials, innovation labs/accelerators, performance partnership pilots, and demonstration projects or waivers with rigorous evaluation requirements)?

Score
7
Millennium Challenge Corporation
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • The Millenium Challenge Corporation supports the creation of multidisciplinary country teams to manage the development and implementation of each compact and threshold program. Teams meet frequently to gather evidence, discuss progress, make project design decisions, and solve problems. Prior to moving forward with a program investment, teams are encouraged to use the lessons from completed evaluations to inform their work.
  • The corporation is implementing proposed innovations to make the compact and threshold program development timeline more rapid, a proposal from the second Millennium Efficiency Challenge (MEC). The MEC was designed to tap into the extensive knowledge of MCC’s staff to identify efficiencies and innovative solutions that can shorten the compact and threshold program development timeline while maintaining MCC’s rigorous quality standards and investment criteria.  The corporation also actively encouraged innovation in the evaluation methods of its programs, including through new data collection techniques.
7.2 Did the agency have programs to promote innovation to improve the impact of its programs?
  • The corporation’s approach to development assistance hinges on its innovative and extensive use of evidence to inform investment decisions, guide program implementation strategies, and assess and learn from its investment experiences. As such, MCC’s Office of Strategic Partnerships offers opportunities within its Annual Program Statement that allow MCC division and country teams to tap into the most innovative solutions to new development issues. These partnerships allow MCC evaluation and economic staff to ensure cutting edge innovation in each new program.
  • These innovations in evidence generation have been even more critical in the past year given the inability to conduct many data collection activities in person. The corporation has utilized local data collection and better technology to maintain evidence generation. For example, MCC partnered with the University of Colorado for use of satellite-connected sensors on water kiosks built by the Sierra Leone Threshold Program’s Water Project. Additionally, MCC partnered with a consortium of the University of Colorado and the SweetSense Inc. technology firm to collect high-frequency monitoring data using emerging and cost-effective technologies to understand the state of water service from the water kiosks constructed by the project. The partnership provided significant flexibility to collaboratively determine how available technology can suit MCC’s monitoring needs, including monitoring in data-challenged environments. It also offered an example of how other MCC water projects can capitalize on the use of similar technology tools to collect more reliable data more frequently.
  • The Millenium Challenge Corporation regularly engages in implementing test projects as part of its overall compact programs. A few examples are (1) an innovative pay-for-results mechanism in Morocco to replicate or expand proven programs that provide integrated support; (2) a “call-for-ideas” in Benin for information regarding potential projects that would expand access to renewable off-grid electrical power; and (3) a regulatory strengthening project in Sierra Leone that includes funding for a results-based financing system.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Although MCC rigorously evaluates all program efforts, it takes special care to ensure that innovative or untested programs are thoroughly evaluated. In addition to producing final program evaluations, MCC continuously monitors and evaluates all programs throughout the program life cycle, including innovation efforts, to determine if mid-program course-correction actions are necessary. These interim data help MCC continuously improve its innovation efforts so that they can be most effective and impactful. Although 36% of MCC’s evaluations use random-assignment methods, all of MCC’s evaluations, both impact and performance, use rigorous methods to achieve the three-part objectives of accountability, learning, and results in the most cost-effective way possible.
Score
6
U.S. Department of Education
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • The Innovation and Engagement Team in the Office of the Chief Data Officer promotes data integration and sharing, making data accessible, understandable, and reusable; engages the public and private sectors on how to improve access to the Department’s data assets; develops solutions that provide tiered access based on public need and privacy protocols; develops consumer information portals and products that meet the needs of external consumers; and partners with OCIO and ED data stewards to identify and evaluate new technology solutions for improving collection, access, and use of data. This team led the ED work on the ESF transparency portal, highlighted above, and also manages and updates College Scorecard. The team is currently developing ED’s first Open Data Plan and a playbook for data quality at ED.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • The Education Innovation and Research (EIR) program is ED’s primary innovation program for K-12 public education. EIR grants are focused on validating and scaling evidence-based practices and encouraging innovative approaches to persistent challenges. The EIR program incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation. Lessons learned from the EIR program have been shared across the agency and have informed policy approaches in other programs.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • ED’s Experimental Sites Initiative is entirely focused on assessing the effects of statutory and regulatory flexibility in one of its most critical programs: Title IV Federal Student Aid programs. FSA collects performance and other data from all participating institutions, while IES conducts rigorous evaluations–including randomized trials–of selected experiments. Recent examples include ongoing work on Federal Work Study and Loan Counseling, as well as recently published studies on short-term Pell grants.
  • The Education Innovation and Research (EIR) program, ED’s primary innovation program for K-12 public education, incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation.
Score
7
U.S. Agency for International Development
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • The agency’s Innovation, Technology, and Research Hub (ITR) serves as a central point for promoting and building capacity for innovation throughout development and national security strategies across USAID, the U.S. government, and the international community. This includes helping to centrally coordinate the agency’s innovation-related work with entrepreneurs, implementing partners, universities, donors, and others to discover, test, and scale innovative solutions and approaches to development problems around the world. Key leaders focused on innovation include the agency’s chief innovation officer and the chief of DIV.  In addition to finding and supporting innovative solutions, the ITR hub also works with other USAID bureaus and independent offices to promote a culture of innovation across the agency to enable it to be a more innovative organization. For example, this includes building internal capacity, skills, and outside-the-box thinking to structure and provide funding in more creative and effective ways (e.g., using fixed amount awards as a grant instrument to pay for outcomes not just inputs).
7.2 Did the agency have programs to promote innovation to improve the impact of its programs?
  • Since 2011, USAID and its partners have launched forty-one innovative programming approaches including prizes, the DIV program, challenges, and Grand Challenges for Development. Across the Grand Challenges portfolio, partners have jointly committed over $619,000,000 ($154,000,000 from USAID) in grants and technical assistance for over 587 innovators in 107 countries. To date, more than $1,000,000,000 in follow-on funding has been catalyzed from external sources, a key measure of success.
  • The agency was honored when the co-founder and scientific director of USAID’s DIV program, Dr. Michael Kremer, received the 2019 Nobel prize for economics, along with Dr. Esther Duflo and Dr. Abhijit Banerjee. Some of the work that led to this honor was connected to USAID’s DIV program, which was launched in 2010. This program values rigorous testing methods such as impact evaluations or robust market tests to measure the impact of USAID innovations. Evidence of clear and measurable outcomes helps demonstrate what is working and what is not. Solutions that demonstrate rigorous evidence of impact can then be scaled to other contexts. Through the DIV program, Dr. Kremer helps USAID use evidence-based approaches to take small risks, identify what works, and scale those approaches to provide greater impact. Since 2010, the DIV program has made 255 grants to find, test, and scale evidence-based innovations directly affecting more than 55,000,000 lives across forty-seven countries. Based on the research by Dr. Kremer and others, announced in October 2020, a subset of grants from DIV’s early portfolio covering 2010-2012 has produced $17 in social benefit for every dollar spent by USAID.
  • As a research and development resource for all of USAID, DIV tests early stage innovations to de-risk and prepare them for adoption by other agency missions and bureaus. By allocating relatively small amounts of money to generate evidence early in an innovation’s development, DIV has enabled the agency to make evidence-driven funding decisions to back proven solutions. Sixteen missions and operating units have contributed funding to awards managed by DIV; some missions later invested significant resources to scale innovations from DIV’s portfolio.
  • In addition, the Exploratory Programs and Innovation Competitions (EPIC) team uses cutting edge program design that prioritizes solutions from local communities. It also supports USAID’s uptake of open innovation competitions, including Grand Challenges, challenges, prizes, and hackathons, to incentivize local and global entrepreneurs, academics, civil society, and other passionate problem solvers to address well-defined problems, pilot and test new solutions, and scale the most promising solutions.
  • Among U.S. Government agencies, USAID is recognized for having dedicated staff, funding, and authorities to encourage innovation and impact. In May 2022, the White House Office of Science and Technology Policy Report on the Implementation of Federal Prize and Citizen Science Authority (for FY19-20), analyzed how federal agencies use prize competitions and challenges to spur innovation, engage nontraditional solvers, address tough problems, and advance their core mission. The report found that among the federal agencies represented, USAID “stood out as exceptional in both how and why it uses prize competitions and challenges”. The findings emphasized that USAID prize competitions and challenges support USAID’s mission by leveraging resources from diverse partners to identify promising solutions to development problems and support them in reaching scale. The report also drew upon the positive return on investment and development impact generated by the Securing Water for Food Grand Challenge to illustrate the strengths of USAID’s Grand Challenge model. By elevating USAID’s strategic application of prize competition and challenge models, this report showcased USAID’s leadership and the expertise of ITR/I/EPIC and the Bureau for Global Health’s Center for Innovation and Impact (GH/CII) open innovation advisors in designing and managing open innovation competitions to the interagency.
  • For innovations specific to a particular sector, agency leadership has supported technical staff in surfacing groundbreaking ideas. For example, the Bureau for Global Health’s CII used open innovation approaches to issue the Saving Lives at Birth Grand Challenge and identify promising life-saving maternal and newborn health innovations. Similarly, the Bureau for Humanitarian Affairs’s Office for Private Sector Engagement, Diaspora, and Innovation supports the Creating Hope in Conflict Humanitarian Grand Challenge.
  • As the Bureau for Global Health’s dedicated innovation office, CII– takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health innovations that address the world’s most important health challenges and assessing and adopting cutting-edge approaches (such as using unmanned aerial vehicles and artificial intelligence).
  • Feed the Future Partnering for Innovation partners with agribusinesses to help them commercialize and scale new agricultural innovations to help improve the livelihoods of smallholder farmers, increasing their productivity and incomes. To date the program has worked with fifty-nine partners in twenty different countries, investing more than $43,000,000 in new technologies and services, and leveraging nearly $100,000,000 in private sector investment. The program has helped commercialize more than 118 innovations, which resulted in an estimated $99,000,000 in sales. It has its own Innovation site that partners can easily see and connect with promising innovations and research.
  • In FY20, USAID released its first digital strategy, moving to a “digital by default” position. Since then USAID’s innovative approaches have helped get digital access to more than 40,000,000 people in the developing world. Its New Partnerships Initiative will allow USAID to work with a more diverse range of partners, strengthen existing partner relationships, and provide more entry points for organizations to work with the agency. The principles behind this initiative are outlined in the agency’s first-ever Acquisition and Assistance Strategy.
  • The agency’s investment in state-of-the-art geo and information intelligence centers mean that any program has the ability to leverage geospatial analysis and critical data sets to drive innovative solutions based on evidence and data. With more than twenty programs experimenting with artificial intelligence and machine learning, and its strong work on digital finance and connectivity, the agency is using technology to drive its programs farther and faster. It has also completed more than 1,500 Global Development Alliances, leveraging private sector in-kind or financial investments.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Monitoring, Evaluation, Research and Learning Innovations Program (MERLIN) is a USAID endeavor designed to support the Bureau for PPPL and is now a part of the LER office within PPL. Formerly, the MERLIN program was part of the U.S. Global Development Lab, which became the ITR hub within the new Bureau for Development, Democracy, and Innovation (DDI). The MERLIN program works to innovate on traditional approaches to monitoring, evaluation, research and learning. While innovative in themselves, these approaches can also be better suited to evaluating an innovation effort. Two examples of MERLIN activities include developmental evaluation, which aims to provide ongoing feedback to managers on implementation through an embedded evaluator, and rapid feedback, which allows implementers to test various methods to reach certain targeted results (more quickly than through traditional midterm or final evaluations). Both of these approaches allow adaptive management during implementation to improve program impacts.
  • Many of the agency’s programs such as Grand Challenges and Development Innovation Ventures have been reviewed by formal audit and other performance and impact evaluations. In 2021, USAID released the findings of a meta-evaluation that encompassed nine Grand Challenges. The findings led to the evaluation’s conclusions about the achievements and effectiveness of USAID’s Grand Challenges: (1) that, overall, they have achieved positive results in varied sectors, many of which are likely to be sustainable, and have supported the scaling of some significant innovations; and (2) that the Grand Challenge model, when implemented well, is a results-driven approach that is both effective at supporting innovations to become scale-ready and at strengthening ecosystems.  The report also includes practical, actionable recommendations to strengthen Grand Challenge programing: strategic recommendations for USAID policy and Grand Challengemanagers and programmatic recommendations for USAID and partner Grand Challenge managers. The report annexes provide additional methodological information, further supporting analysis undertaken for the evaluation on comparators, cost effectiveness, and gender and social inclusion, as well as evidence from the grantee survey conducted for the evaluation.
  • Development Innovation Ventures uses a tiered funding approach to find, test, and scale evidence-based innovations. It’s grant levels include: stage 1 for piloting (up to $200,000), stage 2 for testing and positioning for scale (up to $1,500,000), stage 3 for transitioning to scale (up to $15,000,000), and evidence generation” (up to $1,500,000) for research to determine causal impact of interventions that have already scaled. In particular for stage 2 grants, DIV requires evidence of impact that must be causal and rigorous. The grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period.
  • Evaluation criteria for DIV funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) evidence of Impact, (2) cost effectiveness, and (3) potential for scale and financial sustainability. Expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most Challenge, Grand Challenge, and DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay for results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties.
  • Development Innovation Ventures supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Since 2010, it has provided more than $174,000,000 for 255 grants in forty-seven countries, reaching more than 55,000,000 beneficiaries.
Score
5
AmeriCorps
7.1 Did the agency have staff dedicated to leading innovation efforts to improve the impacts of its programs?
  • AmeriCorps program offices have used American Rescue Plan Act (ARPA) funding as an opportunity to be innovative and to hire term-appointed staff to implement these innovation efforts. Examples of these efforts include the following:
    • ACS Demonstration Authority: ARPA funds will be used to demonstrate and evaluate service to work models for older Americans.
    • ACS Research: ACS and ORE will research COVID impacts on ACS programs, volunteers, and service stations. A third party research firm will be awarded funding in FY22.
    • ASN Planning Grants: ARPA funds will be used to demonstrate and evaluate strategies for increasing access to agency resources in underserved communities including increasing member stipends and waiving match requirements as well as democratizing access to evidence (we will create use cases with ASN grantees utilizing the Impact Genome’s standardized evidence database).
    • AmeriCorps will invest $20,000,000 in nonprofit organizations that seek to pilot a Volunteer Generation Fund in their communities to help address the nation’s learning loss crisis resulting from the pandemic. This investment will be evaluated for its implementation and early outcomes.
    • Public Health AmeriCorps is providing $400,000,000 in Centers for Disease Control and Prevention (CDC) and ARP funding dedicated to building a pipeline of public health workers and strengthening public health systems and the capacity to recover from the pandemic. This initiative will be evaluated (a contract will be awarded to a third party research firm in FY22, and the evaluation design is expected to be quasi-experimental). Findings are expected to inform programmatic improvements in the areas of member and volunteer training/certification, creation of sector-specific career pathways, and the role of cross-agency partnerships in building community capacity to address critical health issues.
    • AmeriCorps VISTA Food Security Initiative: VISTA implemented a second cohort of grantees as part of its food security initiative. The model is being used to establish formal learning communities among sponsor organizations and cohorts of VISTA volunteers to assess whether or not these program enhancements will result in better outcomes for volunteers, organizations, and communities served. Staff from ORE are conducting an internal assessment of this model to determine the feasibility of conducting a more formal evaluation of the initiative in FY23.
7.2: Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • AmeriCorps continued to learn from its National Service and Civic Engagement research grantees, who receive grant funds to engage community residents and leaders in the development of new and innovative national service projects (more information available here). In addition to national service project development, these grants foster civic engagement through community research teams and build community capacity for using research to identify and understand local issues as well as to identify possible solutions. These grantees were given the opportunity to apply for additional funding in FY22 to build on research that is used to innovate and improve national service and volunteer practice and knowledge.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • AmeriCorps uses the research design most appropriate for addressing the research question. When experimental or quasi-experimental designs are warranted, the agency uses them and encourages its grantees to use them, as noted in the agency evaluation policy: “AmeriCorps is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within statutory, budget and other constraints.” As of August 2022, AmeriCorps has received 47 grantee evaluation reports that use experimental design and 144 that use quasi-experimental design.” AmeriCorps has also funded a mixed-methods longitudinal study of NCCC members that includes a matched comparison group. This member development study will conclude in FY23.
Score
6
U.S. Department of Labor
7.1 Did the agency have staff dedicated to leading innovation efforts to improve the impact of its programs?
  • The Department of Labor’s chief innovation officer is responsible for efforts to use innovative technologies, partnerships, and practices to accelerate the department’s mission. The chief innovation officer reports to the deputy secretary and also serves as the senior advisor for delivery for the department.
  • The department’s chief data officer and Chief Evaluation Office Data Analytics team developed a secure data analysis platform accessible to all DOL staff, preloaded with common statistical packages and offering the capability to access and merge various administrative data for analysis. The department supports staff in executing limitless web-based A/B testing and other behaviorally informed trials with the shared service of the advanced Granicus platform’s GovDelivery communications tool, including free technical support. This tool enhances the department’s ability to communicate with the public, such as through targeted email campaigns, and to adjust these communications, informed by testing and data, to increase engagement on relevant topics. The Chief Evaluation Office also has developed toolkits and detailed resources for staff to effectively design behaviorally informed tests, shared on its new Behavioral Interventions website.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • The Chief Evaluation Office uses a variety of communication tools to share rigorous research results, lessons learned, promising practices, and other implications of its research. These include internal briefings from independent contractors and researchers, a brown bag series (Chief Evaluation Office Seminar Series) that features evidence-based promising practices and results shared by DOL staff for DOL staff, and an external expert seminar series featuring new findings or innovations in relevant areas of work. Chief Evaluation Office staff consistently use research findings in the development of new research, and DOL agencies use these findings to design and guide new discretionary grant programs, to refine performance measures for grantees, and to make decisions on compliance and enforcement practices.
  • The department is strongly committed to promoting innovation in its policies and practices. For example, the Employment and Training Administration’s competitive funding routinely supports innovative programming, since grantees typically bundle various program services and components to best meet the needs of the people being served by them in their local contexts. A particularly good example of this innovation is found in the administration’s high-priority area of apprenticeships. In FY19, ETA issued nearly $184,000,000 in Scaling Apprenticeship Through Sector-Based Strategies grants to public-private partnerships for expanding apprenticeships in health care, information technology and other industries. In FY20, ETA awarded nearly $100,000,000 in Apprenticeship: Closing the Skills Gap grants. The Chief Evaluation Office’s Apprenticeship Evidence Building Portfolio project includes implementation evaluations of both of these grant programs. Additionally, the office oversees a contractor-led evaluation of the American Apprenticeship Initiative, which since 2016 has provided $175,000,000 in grants to forty-five grantees across the nation. In July 2022, ETA issued more than $121,000,000 in Apprenticeship Building America grants to strengthen registered apprenticeship programs.
  • In addition, the Chief Evaluation Office’s Behavioral Insights team works with a number of DOL agencies on a continuous basis to identify and assess the feasibility of conducting studies where insights from behavioral science can be used to improve the performance and outcomes of DOL programs. The Wage and Hour Division’s Transformation Team is one such example where continuous improvement efforts are driving innovation. This work has identified potential areas where behavioral interventions and trials may inform program improvement. The Chief Evaluation Office is also working across agencies, including the Wage and Hour Division, Employment and Training Administration, Women’s Bureau, Veterans Employment and Training Service, Office of Federal Contract Compliance Programs, and International Labor Affairs Bureau, to identify and assess the feasibility of other areas where insights from behavioral science can be used to improve the performance and outcomes of DOL programs.
  • The Department of Labor has also built capacity for staff innovation through the Performance Management Center’s CPI Program, an agency-wide opportunity that trains and certifies agency staff on Lean Six Sigma methodologies through real-time execution of DOL process improvement projects. The program includes classroom sessions that prepare participants for Lean Six Sigma Black Belt certification examinations, including the American Society for Quality as well as DOL’s own certification.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Through the annual Learning Agenda process, DOL systematically identifies gaps in the use of evidence. Innovation is about filling known gaps via dissemination, further research, or generation of quick turnaround assessments such as those offered to the Department by the Chief Evaluation Office’s Behavioral Insights Program.
  • The department typically couples innovation with rigorous evaluation to learn from experiments. For example, DOL is participating in the Performance Partnership Pilots (P3) for innovative service delivery for disconnected youth. This program includes not only waivers and blending and braiding of federal funds, but gives bonus points in application reviews for proposing “high tier” evaluations. The Department of Labor  is the lead agency for the evaluation of P3. A final report is available on the Chief Evaluation Office’s completed studies website. In 2021, the Chief Evaluation Office partnered with the Social Security Administration, the Office of Disability Employment Policy, and funded contractor Mathematica to support the ongoing evaluation of the Retaining Employment and Talent After Injury/Illness Network (RETAIN) demonstration projects. The office’s contract supports enrollment data collection and the random assignment of study participants for phase two of the RETAIN demonstration. This impact evaluation aims to assess the effectiveness of intervention strategies from RETAIN demonstration projects operating in Kansas, Kentucky, Minnesota, Ohio, and Vermont.
  • The department routinely uses Job Corps’ demonstration authority to test and evaluate innovative and promising models to improve outcomes for youth. For example, the Chief Evaluation Office is overseeing a rigorous impact evaluation to examine the effectiveness of one of these pilots, the Cascades Job Corps College and Career Academy, in collaboration with ETA.
Score
7
Administration for Children and Families (HHS)
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • In late 2019, ACF implemented a customer experience initiative to enhance its delivery and administration of human services. This initiative focuses on ways to improve the experiences of both grantees and ACF employees. In 2020, ACF named a chief experience officer to lead these efforts. To date, the chief experience officer has led efforts to understand and improve upon the experiences of ACF grantees receiving funding from multiple HHS operating divisions, evaluate and address the challenges that organizations face in applying for competitive grants, and develop an internal tool for ACF teams to assess and improve upon their capability to provide excellent technical assistance to ACF grantees. In 2021, ACF launched an innovation incubator initiative that began with a series of three human-centered design trainings offered to ACF employees to equip them with the skills and resources to identify problems, brainstorm ideas for improvement, and pilot solutions using an empathetic “people-first” mindset. Through this initiative, ACF staff received training in creating, evaluating, awarding, and managing contracts that use human-centered design services. Participating staff also have access to the ACF Innovators community, a shared platform that supports interoffice idea generation and collaboration. While the customer experience initiative is funded on a by-request basis, it received approximately $1,000,000 for FY22 to support this work.
  • ACF is also wrapping up a project, Human-Centered Design for Human Services, to explore the application of human-centered design across its service delivery programs at the federal, state, and local levels. Three state/local agencies received training and technical assistance in using human-centered design and used the approach to address selected challenges. The project developed a theory of change and instruments to assess implementation, supporting future rigorous evaluations of human-centered design in human services programs.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • The Administration for Children and Families’ mission to foster health and well-being by providing federal leadership, partnership and resources for the compassionate and effective delivery of human services is undergirded by six values: dedication, professionalism, integrity, stewardship, respect, and excellence. Its emphasis on excellence, exemplified by innovations and solutions that are anchored in available evidence, build knowledge and transcend boundaries, drives the agency’s support for innovation across programs and practices.
    • ACF’s customer experience initiative is supporting the development of innovative practices for more efficient and responsive agency operations,  improving how ACF understands and meets the needs of grantees and improving its capacity for service delivery. For example, ACF, in partnership with HRSA, convened a gathering of grantees who receive Head Start grants from ACF and federally qualified Health Center grants from HRSA to create opportunities for grantees to learn from one another and share best practices. Additionally, ACF helped a grantee analyze their data across both Head Start and Health Center programs to make operational improvements to their program.
    • A Racial Equity Impact Analysis Tool supports ACF’s commitment to advance equity and helps ACF to have a consistent approach to identifying barriers and gaps in the development of new policies and practices, as well as the review of existing ones. Possible uses of this tool include assessing proposed legislation as part of the annual A-19 budget and legislative development process; evaluating disparity impact statements in Notices of Funding Opportunities; and performing an assessment of the pilot for strategic goal 1 of the ACF Strategic Plan, advancing equity.
    • The 2022 Innovation Challenge identified the best ways to advance ACF’s mission by allowing staff to submit an idea for how ACF could solve an issue it is facing. Staff voted on ideas, which will move forward to a panel of judges. Ten winning teams will then receive support, hands-on training, and coaching for twelve weeks as they develop and pilot their idea.
    • The HHS Equity Research Agenda, one of the tasks supporting Presidential Executive Order 13985 on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, will help guide HHS efforts to (1) identify methods for disparities/equity research that are appropriately community centered, strengths based, and do not pathologize populations that are marginalized and (2) establish ways to appropriately generate new evidence and build on existing evidence to identify and reduce disparities and understand the opportunity and impact of HHS programs on underserved/under resourced communities. The Administration for Children and Families has published its commitment to equity and related research and resources on its website and publicly described its commitment to equity. This includes more than five  projects with an explicit focus on questions and activities related to advancing equity in human services (e.g., Child Welfare Study to Enhance Equity with Data, African American Child and Family Research Center, Race Equity for Fatherhood, Relationship, and Marriage Programs to Empower Black Families, and , Contextual Analysis and Methods of Participant Engagement).
    • The ACFx Customer Experience Survey in 2022 will seek responses from the authorized official for all active ACF discretionary grants with the goal of understanding customer experience. The survey will cover applying for funding, post-award training/orientation, guidance received from the ACF program office, technical assistance received, and experience with reporting and requirements. Specifically, this survey will help give teams working with grant recipients actionable information on where to target improvements, such as identifying whether improvement efforts directed at first-time grant recipients would be more impactful than efforts directed at a particular phase of the grant life cycle.
  • The Administration for Children and Families also administers select grant programs – through innovation projects, demonstration projects, and waivers to existing program requirements – that are designed to both implement and evaluate innovative interventions as a part of an ACF-sponsored evaluation or an individual evaluation to accompany implementation of that innovation. For example:
  • ACF projects that support innovation include the following:
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
4
Substance Abuse and Mental Health Services Administration
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • In FY22, the Center for Substance Use Prevention created an office to focus on innovation called the Office of Prevention Innovation. Using dashboards and data sharing, this office plans to use data to impact its programs and increase capacity for data-driven decision-making. Within the Office of Evaluation, CBHSQ assigned a staff member to focus on aligning SAMHSA to the Evidence Act and moving the goals of the Evidence and Evaluation Board forward.  Although the Office of Prevention Innovation and Office of Evaluation are housed in different centers, there will be regularly scheduled interfaces and other opportunities for engagement.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • Within SAMHSA grant programs, the agency encourages innovation from every state, territory, and community applicant. For example, the FY20-21 Substance Use Prevention and Treatment Block Grant application includes the following language:
  • “There is increased interest in having a better understanding of the evidence that supports the delivery of medical and specialty care including M/SUD services. Over the past several years, SAMHSA has collaborated with CMS, HRSA, SMAs, state M/SUD authorities, legislators, and others regarding the evidence of various mental and substance misuse prevention, treatment, and recovery support services. States and other purchasers are requesting information on evidence-based practices or other procedures that result in better health outcomes for individuals and the general population. While the emphasis on evidence-based practices will continue, there is a need to develop and create new interventions and technologies and in turn, to establish the evidence. SAMHSA supports states’ use of the block grants for this purpose. The National Quality Forum and the Institute of Medicine recommend that evidence play a critical role in designing health benefits for individuals enrolled in commercial insurance, Medicaid, and Medicare. To respond to these inquiries and recommendations, SAMHSA has undertaken several activities. SAMHSA’s Evidence Based Practices Resource Center assesses the research evaluating an intervention’s impact on outcomes and provides information on available 43 resources to facilitate the effective dissemination and implementation of the program. SAMHSA’s EBPRC provides the information & tools needed to incorporate evidence-based practices into communities or clinical settings.”
  • The National Mental Health and Substance Use Policy Laboratory (Policy Lab) leads the equitable coordination, analysis, development, and implementation of national policy to promote mental health, prevent substance misuse and addiction, provide treatment, and support recovery. The Policy Lab supports SAMHSA leadership in national policy efforts through partnership and coordination within SAMHSA; across federal agencies; and with existing, new, and emerging stakeholders and constituent groups. The Policy Lab comprises the following units: (1) Office of the Director; (2) Policy Analysis, Development, and Implementation Team (PADI Team); (3) Evidence-Based Practices Implementation and Dissemination Team (EBP team); (4) Executive Correspondence and Support Branch; and (5) Legislative Affairs Branch.
    • The PADI Team integrates policy and subject matter expertise throughout SAMHSA priority areas (including workgroups, grants and budget review, and information sharing) and collaboration with SAMHSA’s centers and offices, Federal partners, and stakeholders, which ensures a feedback loop, positive adaptation, and collaboration.
    • The EBP Team uses information from evaluations, literature reviews, expert panels, and other sources to inform identification and implementation of policy change; assists in the development of programs, technical assistance, evaluations, and dissemination of best practices/lessons learned; and tracks current EBPs and conducts literature reviews on high-priority topic areas and programs.
    • The Executive Correspondence and Support Branch serves as the agency liaison and action office for policy coordination including management of correspondence, advisory committees, and Freedom of Information Act activities across SAMHSA.
    • The Legislative Affairs Team advises the assistant secretary on legislative matters; provides leadership in the development of legislation; and serves as the primary contact within SAMHSA on all legislative activities.
  • Also in FY22, SAMHSA created the Recovery Innovation Challenge. This competition seeks to identify innovative practices in behavioral health that advance recovery on the ground and in the community. This innovation challenge will support the Office of Recovery, which is designed to advance the agency’s commitment to support recovery for all Americans. The Office of Recovery will serve as a national clearinghouse and resource for recovery services across mental health, substance use, and co-occurring domains to promote recovery by working in partnership with recovery community leaders. The goal of this challenge is to identify innovative practices developed by individuals, groups, and organizations or within state systems that advanced recovery in the decade since SAMHSA established its working definition of recovery. This challenge will allow participants to share details about innovative practices and models being used to promote recovery on the ground and to demonstrate how these innovations have expanded upon SAMHSA’s definition and overcome challenges in incorporating recovery into their services or systems. This challenge competition will offer up to ten awards. The challenge purse is up to $400,000. At the time of this report, SAMHSA has received more than 300 applications.
  • In addition, SAMHSA created the Diversity Inclusion Project Showcase (DIPS), an initiative that aims to connect a more diverse pool of historically marginalized populations to grant funding opportunities by providing the chance to showcase their goals and populations served to federal and state leaders, as well as philanthropic partners. The DIPS model helps to connect community-based organizations to funders and advocates at all levels (e.g., federal and philanthropic organizations, stakeholders, and communities).
  • SAMHSA also sought innovative efforts to build its existing data sets. For example, SAMHSA, in partnership with other federal agencies, submitted a proposal to identify and prioritize research, innovation, and public health and funding gaps in substance use disorders and mental health disorders using machine learning techniques. If funded, this work would be completed in 24 months.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Grantees report innovation and use of evidence-based practices in their reports as required by SAMHSA. Information from these reports is included in rigorous evaluations planned or used for performance management when a formal evaluation has not been scheduled.
  • SAMHSA completed a rigorous evaluation of its innovative SPF-Rx program. Prescription drug misuse continues to be a critical public health problem in the United States, and the Strategic Prevention Framework (SPF) consists of five steps. This framework is a data-driven systemic public health planning approach to substance use prevention that is theory based and involves the implementation of evidence-based strategies. Grantees and subrecipients apply the overarching principles of cultural competence and sustainability throughout the dynamic SPF process.
  • Between 2017 and 2021, twenty-five state and tribal SPF-Rx grantees received funding to focus on supporting high-need community efforts to implement evidence-based practices to prevent and reduce the misuse of prescription opioids. In FY21, findings from a completed evaluation informed SAMHSA of ways to better develop, assess, and manage its prescription drug misuse prevention programs. The cross-site evaluation collected quantitative and qualitative data from three instruments: an annual implementation instrument (130 items collecting data on organization type, funding levels, assessment of capacity building and sustainability, strategic planning, prevention and intervention programming, and ongoing local evaluations); grantee level and community level outcomes modules (two modules including data on prescription drug monitoring programs’ use and prescribing patterns, opioid overdose events, etc.); and grantee level interviews (qualitative data collected at baseline and at the end of the evaluation). Results of this evaluation were used to inform the next generation of SPF Rx grants.
  • The next evaluation of this SPF Rx program has been started with an evaluation plan drafted to include an examination of secondary data on matched communities for comparison of outcome data.
Score
6
U.S. Dept. of Housing & Urban Development
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • HUD has an Office of Innovation led by a deputy assistant secretary that facilitates both routine innovation and improvements to departmental operations, services, and working conditions. Examples of recent work include assessing the feasibility of modernizing HUD headquarters’ energy and information system and creating a standardized Paperwork Reduction Act submission package. In June 2022, HUD hosted the Innovative Housing Showcase, a three-day event that featured new building technologies and housing solutions that are making housing more innovative, resilient, and affordable for families across the country. The Office of Innovation is developing prize competitions to stimulate innovation in housing and HUD policy and programs.
  • The Office of Innovation has hosted learning opportunities designed to leverage data to help stakeholders, such as public housing agencies, city and regional planners, housing developers, schools, state and local governments, and other federal agencies, understand how housing needs have shifted during (and as a result of) the pandemic.
  • The Department of Housing and Urban Development administers five types of juried Secretary’s Awards to encourage excellence in addressing housing and community development challenges: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. An Innovation in Affordable Housing Competition engages multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency.
  • Additionally, the Government National Mortgage Corporation, or Ginnie Mae, has several innovation teams and operates an Innovation Laboratory to advance the application of machine learning to strengthen operations.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • The Department of Housing and Urban Development established the Office of Innovation in 2019 to advance innovation in several domains. The Office of Innovation managed the 2019 Innovative Housing Showcase and prize competitions to stimulate innovation in housing and HUD policy and programs. It also recently worked with The Opportunity Project to improve communication between HUD’s Envision Centers and their local stakeholders and is now working with the Opportunity Project to provide data and visualizations related to the housing industry impacts of the changes to the workplace and society post-COVID.
  • Cooperative agreements for pre-competitive research in homebuilding innovations are funded by FY21 grants, with a similar program for historically Black colleges and universities, for more affordable, energy efficient, resilient, and healthier housing. The department seeks to engage the insights and creativity of these colleges and universities to advance problem solving toward greater diversity and more equitable outcomes.
  • The department promotes evidence-based innovation by using program demonstrations to experimentally test potential policy enhancements. These demonstrations have included eight low-cost, behaviorally informed experiments using interagency data matching and assistance from the GSA Office of Evaluation Sciences. Other innovative research ideas from external stakeholders are supported by the FY21 Research Partnerships program. Competitive awards for Lead and Healthy Homes Technical Studies generate innovation in the evaluation and control of housing-related health and safety hazards.
  • An interagency agreement with the Census Bureau has made datasets from HUD’s RCTs available for linkage with census data and administrative datasets. The RCT datasets are the first intervention data added to Federal Statistical RDCs by any federal agency, and joint support is available to help researchers gain access and learn to use the restricted data successfully for innovative research, with seven projects currently underway.
  • The department’s Rental Assistance Demonstration, which restructures the financing of the nation’s public housing to address capital needs backlogs, has the additional innovative feature of providing tenants with a choice mobility option. Choice mobility supports self-sufficiency by offering priority receipt of a housing choice voucher that provides freedom to move to neighborhoods with greater economic opportunities or better schools and amenities
  • Regulation of manufactured housing production by HUD is guided by a federal advisory committee, the Manufactured Housing Consensus Committee, to provide increased ability for the industry to produce some of the nation’s most innovative, safe, and affordable housing.
  • The department has a Robotics Process Automation initiative devoted to freeing the workforce from low-value, repetitive work through software robotics solutions. Specialized computer programs known as bots automate and standardize repeatable business processes without costly investments in conventional automation. Planned efforts involving payroll, accounts receivable and payable, invoice processing, inventory management, report creation, and data migration have potential to shift over 50,000 hours of employee time from low-value to high-value work.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
4
Administration for Community Living (HHS)
7.1 Did the agency have staff dedicated to learning innovation efforts to improve the impact of its programs?
  • Agency leadership promotes innovation by requiring all program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. The Administration for Community Living also partially funds a Forum on Aging, Disability, and Independence, which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. It funds resource centers, such as the Engagement and Older Adults Resource Center, which provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs?
  • In FY21 all ACL Centers were involved in funding innovative work. The agency released several funding opportunity announcements focused on the identification and implementation of innovative approaches to improve programming. These included funding opportunity announcements for the Innovations in Nutrition Programs and Services – Community Research and the ACL/NIDILRR Small Business Innovation Research Program (SBIR) Phase I.  In addition to challenge competitions such as the ACL Social Care Referrals Innovation Challenge, in 2021 ACL funded a small study to identify the innovations and adaptations by grantees in the face of COVID-19. In 2022 and 2023, ACL will more closely examine the most promising innovations to evaluate their effectiveness.
  • The Administration for Community Living is a funder of The John A. Hartford Foundation 2022 Business Innovation Award, which recognizes aging and disability community-based organizations for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities.
  • There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the agency’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3,500,000 to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • The 2020 reauthorization of the Older Americans Act requires a new research, demonstration, and evaluation center for the Aging Network; new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches toward coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e., Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
  • The research and development activities of NIDILRR are guided by the stages of research framework and the stages of development framework. Grantees conducting research projects must identify the current state of research of their projects while grantees conducting development projects must identify the current state of development of their projects.  The stage of a research project depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the intervention development phase. The next phase, known as intervention efficacy, refers to the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as scale-up evaluation, is the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting.
  • Similarly, the stage of development for a development project depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage refers to the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants; verifying product requirements; implementing and testing (typically in controlled contexts) key concepts, components, or systems; and resolving technical challenges. The proof of product stage is the stage of development where a fully integrated and working prototype meeting critical technical requirements is created. The proof of adoption stage is the stage of development where a product is substantially adopted by its target population and used for its intended purpose.
Back to the Standard

Visit Results4America.org

Copyright 2024 Results for America. All rights reserved. Privacy Policy

Site by Wide Eye Creative