Measuring STEM Engagement in K-12 Educational Initiatives

The nationwide drops in state standardized test scores during the pandemic demonstrated one reason why it is important to use multiple measures, but also that it is important to use measures beyond achievement data. In this month’s blog post we will explore a variety of ways K-12 projects can demonstrate engagement in science, technology, engineering, and math (STEM).

Course Enrollment

To use course enrollment there will need to be a list of courses established. For STEM engagement, SEG often uses a list of STEM courses that would be considered optional or advanced. Examples of potential indicators using this data might include:

• The number or percentage of students enrolled in at least one advanced/optional STEM course

• The number of seats filled in advanced/optional STEM courses

• The average number of advanced/optional STEM courses with student enrollment

Certification Enrollment

The number of students enrolling in STEM-related certificates can also be an indicator of engagement. These might be STEM-related Career and Technical Education Endorsements or professional certifications. Examples of potential indicators might include:

• The number or percentage of students enrolled in a STEM-related certification program

• The number or percentage of students who have earned a STEM-related certificate

Extracurricular Activities

Engagement in extracurricular STEM activities can be used as a measure. Often when people think of extracurricular activities they think about afterschool clubs and organizations, but this could also be clubs that happen during the school day, family events, and enrichment opportunities. Examples of potential indicators might include:

• The number or percentage of students participating in STEM-related extracurricular activities

Student Survey

Student surveys are frequently used for measuring student outcomes, but not always used to measure engagement. Project directors may consider using pulse surveys or exit surveys to measure STEM engagement. Pulse surveys are brief surveys that can be administered with varying frequencies (e.g., monthly, quarterly). Exit feedback forms can be used to measure students’ experiences with an event such as a lab activity, visit from a STEM professional, or field trip. These can be designed to include as few as 2-3 questions to capture engagement at the end of the activity. Examples of potential indicators from surveys and feedback forms might include:

• The percentage of students interested in STEM Careers

• The percentage of students interested in pursuing a STEM degree in the future

• The percentage of students who “agreed” or “strongly agreed” they had fun with an activity

• The percentage of students who “agreed” or “strongly agreed” they enjoy the STEM activities they complete in class

Although STEM examples are used, these types of measures and indicators can be applied to all content areas. During the evaluation planning process, Shaffer Evaluation Group works with clients to design a series of measures that are responsive to your project’s needs and can effectively demonstrate success. If you are interested in partnering with SEG, we would love to talk with you!

Evaluation of NSF LSAMP Awards

The Louis Stokes Alliances for Minority Participation (LSAMP) program is a National Science Foundation (NSF) program that provides funding to alliances of colleges and universities to increase the number of STEM baccalaureate and graduate degrees awarded to populations historically underrepresented in these disciplines.

Evaluation is a key component of the NSF LSAMP program. Evaluation helps to ensure that a LSAMP project is meeting its goals and objectives and achieving outcomes for participating students. Evaluation also helps project teams identify areas where strategies and activities can be improved.

There are a number of different ways to evaluate LSAMP projects. One common approach is to use a mixed-methods approach, which combines quantitative and qualitative data. This approach can provide a more comprehensive understanding of the impact of the project while providing rich process findings to inform project implementation. Quantitative data can be used to measure the project's impact on student outcomes, such as academic performance, retention, and graduation rates. Qualitative data can be used to understand the students' experiences in the program, and the factors that influence their success.

Some common outcome measures used to evaluate LSAMP projects include:

• Sense of belonging: This outcome can be measured using surveys or interviews that ask students about their feelings of connection to the STEM community and their sense of belonging at their institution.

• STEM identity: This can be measured using surveys or interviews that ask students about their identification with STEM fields and their commitment to pursuing a STEM career.

• STEM self-efficacy: This can be measured using surveys or interviews that ask students about their confidence in their ability to succeed in STEM courses and careers.

In addition to these student measures, the impact of LSAMP on faculty and the institution can be measured. It is important to note that the best way to evaluate an LSAMP project will vary depending on the specific goals and objectives of the project. However, the measures described above can provide a good starting point for developing an evaluation plan.

In the Shaffer Evaluation Group's evaluation of the Central Florida STEM Alliance LSAMP project, a mixed methods evaluation design collects and analyzes data on project implementation and outcomes. The outcome evaluation utilizes both quantitative and qualitative data to identify student, faculty, and institutional impacts from this LSAMP project. Student outcome indicators include graduation rate, transfer rate, STEM self-efficacy and identity, and sense of belonging. The implementation evaluation monitors activity levels of students, faculty, and the participating institutions, using these to determine correlations to short-term student outcomes during the program, such as persistence and retention. The implementation evaluation also collects information to identify barriers or challenges that have impacted implementation, track improvement in service delivery, and assess the overall reach of the services provided. It also identifies actions taken by project staff to ensure the sustainability of strategies/activities beyond the grant funding period.

Evaluation of a LSAMP project can help ensure that it is meeting its goals and objectives and that it is making a positive impact on the lives of underrepresented students in STEM. In your LSAMP grant application, including a robust evaluation plan will improve the competitiveness of your application.

Are you applying for a LSAMP grant? Shaffer Evaluation Group welcomes the opportunity to partner with your institution on this grant opportunity. Once you have a draft grant application in hand, reach out to us for assistance with the evaluation plan.

References

• Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago, IL: University of Chicago Press.

• Byars-Winston, A. M., Estrada, M., & Bell, C. (2016). Racial and ethnic identity, STEM self-efficacy, and STEM persistence: A meta-analysis. Educational Psychologist, 51(2), 109-129.

Postsecondary Student Success Program Grantwriting: A Step-by-Step Guide

By: Patricia Moore Shaffer

Principal & CEO

The Postsecondary Student Success Program (PSSP) grant, administered by the U.S. Department of Education, aims to support initiatives that improve student outcomes and success in institutions of higher education (IHE).

The purpose of this program is to equitably improve postsecondary student outcomes, including retention, transfer (including successful transfer of completed credits), credit accumulation, and completion, by leveraging data and implementing, scaling, and rigorously evaluating evidence-based activities to support data-driven decisions and actions by institutional leaders committed to inclusive student success.

To successfully apply for this grant, institutions should follow a step-by-step process that includes careful planning, thorough research, effective proposal writing, and meeting the evaluation requirements.

---

Step 1: Understand the Grant Program

Before starting the application process, it is crucial to thoroughly understand the Postsecondary Student Success Program grant program. Review the program's guidelines, objectives, and eligibility criteria.

Familiarize yourself with the specific requirements and priorities outlined by the U.S. Department of Education, including the evaluation requirements. Visit the Department of Education program page for easy access to these materials.

Step 2: Identify Institutional Needs

Identify the areas within your institution that require improvement and align with the grant program's objectives. Conduct a comprehensive needs assessment to determine the specific challenges faced by your institution and collect data and evidence to substantiate these needs. This may include student enrollment and retention rates, graduation rates, student satisfaction surveys, and any other information that demonstrates the need for the proposed activities.

Step 3: Develop a Project Plan

Based on the identified needs, develop a plan that outlines the project's goals and objectives, and identifies the strategies and activities you propose to implement. Ensure that your plan aligns with the grant program's objectives and priorities.

For instance, if your institution aims to enhance student advising and support services, your project plan may include the implementation of a comprehensive advising program, peer mentoring initiatives, and academic support resources.

At this step, your institution should also decide whether it is best positioned to seek an early-phase or mid-phase/expansion grant. Early-phase grants provide funding to IHEs to develop, implement, and test the feasibility of a program that prior research suggests is likely to improve relevant outcomes, for the purpose of determining whether an initiative improves student retention and completion of postsecondary students.

Mid-phase or expansion grants are supported by moderate or strong evidence and provide funding to IHEs to improve and/or expand initiatives and practices that have been proven to be effective in increasing postsecondary student retention and completion.

Step 4: Evaluation Plan

Develop a comprehensive evaluation plan that outlines how you will measure the success and impact of your project. For this grant application, there are expectations of rigor for the evaluation design.

The evaluation plan will be judged on its ability to produce evidence about the project’s effectiveness that would meet the What Works Clearinghouse standards with or without reservations while also providing performance feedback and permitting periodic assessment of progress toward achieving intended outcomes.

Evaluation plans for mid-phase grant applications also must demonstrate the extent to which the evaluation will provide guidance about effective strategies suitable for replication or testing in other settings.

Step 5: Budget Preparation

Prepare a detailed budget that clearly outlines the costs associated with your proposed activities. Ensure that your budget aligns with the grant program's guidelines and restrictions.

Provide a breakdown of expenses, including personnel, equipment, supplies, and any other necessary resources. Justify each expense and demonstrate how it directly contributes to achieving the project's objectives.

Step 6: Proposal Writing

Write a compelling and well-structured proposal that clearly articulates your institution's needs, project plan, and expected outcomes. Address each component of the grant application, including the abstract, absolute priority, significance, project design, project personnel, management plan/budget, and evaluation plan (see Step 4).

The mid-phase application also requires a description of the program scaling strategy. Use clear and concise language, provide evidence to support your claims, and adhere to the specified word limits.

Step 7: Review and Revise:

Before submitting your application, carefully review and revise your proposal. Ensure that it is free from grammatical errors, typos, and inconsistencies. Seek feedback from colleagues or experts in grant writing to improve the clarity and coherence of your proposal. Make any necessary revisions to strengthen your application.

Step 8: Submit the Application:

Once you have completed all the necessary steps and reviewed your application, it is time to submit it. Follow the instructions provided by the grant program for submitting your application. Ensure that you meet all the deadlines and include all the required documents and attachments.

---

Writing a successful Postsecondary Student Success Program grant application requires careful planning, thorough research, effective proposal writing, and meeting the rigorous evaluation requirements.

By following this step-by-step process, institutions can increase their chances of securing funding to implement innovative strategies that enhance student success in postsecondary education. Remember to align your proposal with the grant program's objectives, provide compelling evidence, and develop a comprehensive evaluation plan that measures the impact of your project. With dedication and attention to detail, institutions can create a strong application that demonstrates their commitment to improving student outcomes and success.

Shaffer Evaluation Group welcomes the opportunity to partner with your institution on this grant opportunity. Once you have a draft project plan in hand, reach out to us for assistance with the evaluation plan.