A sixth-grade girl who enjoys taking things apart attends her first day of the afterschool robotics club; a program facilitator gathers the supplies for an egg drop experiment; a parent researches summer program options; a funder convenes a state network of afterschool providers to share best practices in informal STEM education. What unites these people is a desire to benefit from and contribute to afterschool science, engineering, technology, and math (STEM) programs that are engaging, high quality, and have a positive effect on youth.

Afterschool programs were originally conceived as places for youth to participate in a variety of supervised hands-on activities while parents are at work. However, thanks to national recognition that STEM education needs improvement in the United States, many afterschool programs have taken on the additional role of increasing youth access to STEM learning opportunities (Noam & Shah, 2013). The release of the Common Core and Next Generation Science Standards in response to this national shift in educational priority inspired funders to increase the resources invested to support inquiry-based, hands-on STEM learning opportunities in afterschool (Noyce Foundation & Mainspring Consulting, n.d.).

The Charles Stewart Mott Foundation and the Noyce Foundation began a formal collaboration in 2012 to leverage their joint investments and existing infrastructure to expand the availability of quality STEM in afterschool and to impact more students across the country (Noyce Foundation & Mainspring Consulting, n.d.). As of 2016, all 50 states have Mott-funded statewide afterschool network or partnership grants and half have received either STEM system-building or planning grants. System-building states are focused on five interconnected components: partnership and leadership development; evaluation and data collection activities; quality building and professional development opportunities; communication and policy; and financing and sustainability. Each state engages key partners around a vision of quality STEM in afterschool; maps the existing landscape of afterschool and STEM efforts; prioritizes strategies and acts to expand awareness of, supply and quality of STEM in afterschool through communication, policy, and professional development; and measures the effectiveness of these efforts.

Few studies have examined STEM program quality and STEM learning outcomes outside of school and even fewer have looked at the issue on a large, national scale. To this end, the Mott and Noyce Foundations supported a large-scale evaluation to examine the influence of system-building efforts on STEM quality and youth outcomes in afterschool. While the goal of improving quality afterschool STEM was the same for every state, each participating state adopted different approaches and strategies to reach its goal to build a statewide support system for quality afterschool, summer, and expanded learning opportunities.

A Call for Evaluation

A coalition of researchers, funders, and statewide afterschool networks representing 11 state afterschool networks came together with leadership from The PEAR Institute: Partnerships in Education and Resilience at McLean Hospital and Harvard Medical School to design and conduct a national study. The Afterschool & STEM System-Building Evaluation 2016 was conducted by The PEAR Institute and IMMAP: Institute for Measurement, Methodology, Analysis & Policy at Texas Tech University. The selected 11 networks approximate national census data, including rural, urban, and suburban programs, and represent a demographically diverse sample so that one could anticipate finding similar results in states that share the same level of support, professional development, and leadership across the U.S.

Leaders from the 11 state networks worked with The PEAR Institute and IMMAP to select up to 15 informal STEM education programs that best represented the afterschool universe in their state, offered a variety of curriculum options, taught in a range of settings (school-based, community-based, or other), had a range of formality, and represented different demographics, particularly those who are traditionally underserved in STEM. The primary goals of this evaluation were to (1) examine levels of change in STEM-related outcomes among youth in programs receiving resources and training support from system-building states; (2) inform on national trends related to STEM learning, such as gender or grade differences in STEM interest; and (3) link program quality with student outcomes and facilitator attitudes and self-assessments.

Large-Scale Evaluation and Innovative Design

An evaluation of this scale required measurement tools that could gather evidence from the student, facilitator, and program quality perspectives and a high-level of participation across the state networks. The three measurement tools used in this evaluation were developed by The PEAR Institute to measure evidence of STEM learning and program quality: (1) the Common Instrument Suite (CIS), a student self-report survey that assesses STEM-related attitudes and 21st-century skills; (2) the STEM Facilitator Survey, an instructor survey on facilitators’ experience teaching STEM in afterschool and their perceived impact on students’ proficiency and confidence in science and math; (3) the Dimensions of Success (DoS), an observation tool used by certified professionals to establish levels of afterschool program quality. Each of the PEAR Institute’s assessment tools were developed using a translational approach that combines academic research with feedback from practitioners in afterschool and summer settings. The evaluation featured several innovative design components including the use of advanced statistical methods and analysis, and tablet-based technology for participant data collection. The team also employed a retrospective survey design, which has been found to be superior to the traditional pretest-posttest survey design when measuring change in attitudes and beliefs following a given intervention (Howard, 1980; Lam & Bengo, 2003; Little et al., in preparation; Moore & Tananis, 2009; Pratt, McGuigan, & Katzev, 2000). Specifically, in this case, youth were asked at the end of their afterschool STEM program to judge the effect of the program on their skills, beliefs and attitudes toward STEM and 21st-century skills (Allen et al., 2017).

A total of 1,599 students (733 female, 866 male) in Grades 4-12 in 160 programs providing informal STEM instruction participated in the evaluation, with more than half coming from traditionally underserved groups in STEM (female and minority groups). Across the 11 states, 148 facilitators participated in the STEM Facilitator Survey. Over the course of the evaluation, there were 252 DoS program quality observations performed across the participating programs.

Evidence for Afterschool STEM

The study of 11 states found strong evidence in support of the value of afterschool programs. Following is a brief summary of some of the findings.

Participating in afterschool STEM programs improves students’ reported attitudes, knowledge, and skills

An analysis of student self-reported change shows that participation in a STEM afterschool program increases positive attitudes towards STEM. Because of their afterschool experience:

• 78% of students said they are more interested in STEM
• 73% of students said they had a more positive STEM identity
• 80% of students said their STEM career knowledge increased

Not only does participation in STEM afterschool programs influence how students think about STEM, more than 70% of students across all states reported positive gains in 21st-century skills, including perseverance and critical thinking. These findings are important because high science interest levels are associated with improved science literacy (Dabney et al., 2011), greater academic achievement (Hughes, Luo, Kwok, & Loyd, 2008; Schiefele, Krapp, & Winteler, 1992), college readiness and acceptance (Wang & Holcombe, 2010), and STEM course enrollment and career acquisition (Watt et al., 2012). The 21st-century skills also are associated with improved academic performance (Murphy et al., 2015; Oberle, Schonert-Reichl, Hertzman, & Zumbo, 2014).

Facilitators see program impact, desire more professional development

STEM afterschool program participation has benefits that extend beyond the students: More than 88% of facilitators reported improvements in their own interest, confidence, and ability to lead afterschool STEM activities after their programs ended. That confidence extended to their students, with more than 90% of facilitators reporting that they felt their students were more proficient and confident in science, math, and social skills because of their students’ participation in an afterschool STEM program. Despite this high level of confidence, 92% of facilitators desired more professional development in STEM, with the top three priority areas being support in programming ideas, program management, and connecting afterschool programming to the school day. There were also significant, positive correlations between facilitators’ levels of interest, confidence, and ability in STEM facilitation and their perceptions of their students’ proficiency and confidence in math and science. Specifically, facilitators who reported greater interest and ability in STEM facilitation perceived greater gains in their students’ science and math confidence and proficiency.

Afterschool programs with higher quality ratings linked to higher student outcomes

The quality observation data collected using the DoS tool indicated that the greatest strengths among participating programs were organization, materials, space utilization, and relationships, whereas the greatest challenges for programs were STEM content learning, inquiry, reflection, relevance, and youth voice. This finding is consistent with the facilitators’ self-reported need for additional professional development in STEM.

Overall program ratings of STEM knowledge and practices (STEM content learning, inquiry, and reflection) were the lowest across the 11 state networks compared with the other dimensions observed (such as organization, materials, and space utilization). Programs that received the highest ratings in STEM knowledge and practices had students that reported the most positive STEM-related outcomes, particularly for students’ self-reported change in STEM career interest, STEM career knowledge, and STEM identity.

All participating states exhibited superior effects for one or more youth outcomes, with up to 85% of youth reporting positive gains. However, as might be expected, there were differences among states in the quality of programs observed and outcomes for youth. The PEAR Institute will be publishing a detailed analysis of these data, which show a strong association between quality and student outcomes, in a forthcoming article.

Recommendations for future STEM afterschool evaluations

The recent debate on the effectiveness of afterschool programs in the U.S. has made the need for research on these programs’ return on investment more urgent than ever before. This evaluation demonstrates that it is possible to successfully implement a large-scale initiative that measures positive change in youth outcomes in STEM and 21st-century skills across the country. While many high-quality programs were identified over the course of this evaluation, the work to improve afterschool STEM is not done. Many programs would benefit from additional improvement and have a lot of growth potential.

In reviewing the evaluation process, we identified six components of our work that led to its successful implementation and are transferable to future studies.

1. Leverage field leaders’ strengths.
2. Target professional development and quality support.
3. Focus on the linkage between STEM learning and 21st-century skills.
4. Encourage use of data to inform practice.
5. Use innovative out-of-school time evaluation and assessment strategies.
6. Prioritize evaluation in afterschool STEM.

For a more detailed discussion of these recommendations, see the full report: Allen et al., 2017.

Evaluations like this one demonstrate the critical importance of continuing to invest research and resources into afterschool STEM programs. Our results indicate that system-building work makes a difference in creating stronger programs and better outcomes for youth. We recommend that future research should track the specific use of investment in each state to determine strengths and areas for development and to dig deeper into what make states with the strongest outcomes successful. If we can identify specific strategies that are working in these states, we can use this tide of research to lift all STEM afterschool programs by supporting effective, sustainable quality improvements that benefit youth across the country. Future research will need address the connections among investments, network priorities (e.g., professional development, coaching, data systems, recruitment and hiring practices), STEM program quality, student, and facilitator outcomes. In the meantime, it is possible to say there is evidence that STEM afterschool programs show real results.

References

Allen, P. J., Noam, G. G., Little, T. D., Fukuda, E., Gorrall, B. K., & Waggenspack, B. A. (2017). Afterschool & STEM system building evaluation 2016. Belmont, MA: The PEAR Institute: Partnerships in Education and Resilience.

Dabney, K. P., Almarode, J. T., Miller-Friedmann, J. L., Sonnert, G., Sadler, P. M., & Hazari, Z. (2011). Out of school time science activities and their association with career interest in STEM. International Journal of Science Education, Part B: Communication and Public Engagement, 2(1), 63–79.

Dabney, K. P., Tai, R. H., Almarode, J. T., Miller-Friedmann, J. L., Sonnert, G., Sadler, P. M., & Hazari, Z. (2012). Out-of-school time science activities and their association with career interest in STEM. International Journal of Science Education, Part B, 2(1), 63–79. https://doi.org/10.1080/21548455.2011.629455

Watt, H. M. G., Shapka, J. D., Morris, Z. A., Durik, A. M., Keating, D. P., & Eccles, J. S. (2012). Gendered motivational processes affecting high school mathematics participants, educational aspirations, and career plans: A comparison of samples from Australia, Canada, and the United States. Developmental Psychology, 48(6), 1594–1611.

Patricia J. Allen, Ph.D., The PEAR Institute: Partnerships in Education and Resilience at McLean Hospital and Harvard Medical School

Gil G. Noam, Ed.D., Ph.D. (Habil.), The PEAR Institute: Partnerships in Education and Resilience at McLean Hospital and Harvard Medical School

Todd D. Little, Ph.D., IMMAP: Institute for Measurement, Methodology, Analysis & Policy at Texas Tech University