The Important Role of Evaluation in Research: Exploring, Documenting, and Improving the Process of Scientific Investigation

Everyday, based on our search history, algorithms push snippets of the latest findings of scientific studies about topics that are important to us. What these snippets rarely highlight are the important, sometimes “behind the scenes,” evaluation activities that are central to successful research endeavors. Many of us don’t consider (may not even have known about) the important role of evaluation within (as part of) research studies. So just what role do evaluators play in research? Let’s explore this a bit here - 

We all know that researchers aim to produce knowledge by answering research questions through formal processes of scientific inquiry (experimental designs, quasi-experimental designs, etc.), using structured protocols that engage a specified set of investigators who implement specific interventions. Evaluators, on the other hand, come from outside the set of study investigators, providing independent “feedback” to researchers, and at times their advisory boards, on study implementation, impacts, and outcomes.

Evaluators help researchers learn more about their intervention processes than just the impact on the study subject(s). Evaluators embedded in research projects assist researchers in understanding the fidelity and effectiveness study intervention components using formal evaluation practices including:

• evaluations of fidelity (to what extent have the study interventions been implemented as designed?);

• formative evaluations (to what extent has the study achieved its intended outputs? What successes has it achieved and which component of the study is considered to be most closely associated with this success? What challenges has the study faced and what actions did the investigators take in response?); and

• summative evaluation (to what extent has the study met its intended outcomes?).

Together, these three evaluation practices produce findings that help study investigators know if their interventions are being conducted as originally described in study protocols, whether modifications may be needed, what components may be resulting in the strongest impact, which outcomes are being achieved (or not) and how, among other nuanced findings that could have important implications on scientific findings.

In addition to measuring fidelity and impact, evaluators are skilled in helping researchers understand dissemination opportunities and practices. Funders of studies often require scientists to focus on, and share, outcomes that inform the scientific field and the broader community potentially affected by the research question(s). Evaluators are uniquely positioned, through their technical training, to assist researchers in identifying stakeholders for dissemination of results. Evaluators also are skilled in creating actionable knowledge that is applicable and appropriate for eliciting the strongest impact upon dissemination.

Shaffer Evaluation Group (SEG) supports a variety of research grants, including from the USDA and National Science Foundation, by providing evaluation services in support of scientific inquiry. Feel free to call on us if you have similar needs.

Navigating Educational Policy Change

​As of March 2025, the U.S. K-12 education landscape is experiencing significant shifts due to recent executive orders issued by President Donald Trump. These directives have notably impacted various federal education grant programs within the U.S. Department of Education and other agencies, influencing funding allocations and policy priorities.​

Executive Orders Affecting K-12 Education

• Executive Order 14151: "Ending Radical and Wasteful Government DEI Programs and Preferencing"

  Issued on January 20, 2025, this order terminates all Diversity, Equity, Inclusion, and Accessibility (DEIA) initiatives across federal agencies. It mandates a comprehensive review of grant programs to ensure compliance, leading to the rescission of funding for programs that fail to align with the new policy. 

• Executive Order 14190: "Ending Radical Indoctrination in K-12 Schooling"

  Signed on January 29, 2025, this order aims to eliminate the teaching of materials considered "anti-American" or "subversive," including content related to "gender ideology" and critical race theory. It mandates the cessation of federal funding to schools that promote these ideologies and directs law enforcement to prosecute educators who unlawfully facilitate the social transition of transgender minors. ​

• Executive Order 14191: "Expanding Educational Freedom and Opportunity for Families"

  Signed on January 29, 2025, this order aims to expand school choice initiatives, including publicly funded vouchers for private schooling. It seeks to enhance parental control over education and reduce federal oversight, aligning with the administration's broader educational reform agenda. 

• Executive Order 14201: "Keeping Men Out of Women's Sports"

  Signed on February 5, 2025, this order directs the federal government to interpret Title IX regulations as prohibiting transgender girls and women from participating in female sports. It threatens to revoke federal funding from any educational institution that allows transgender girls to compete on girls' teams, citing violations of Title IX.

• Executive Order 14214: "Keeping Education Accessible and Ending COVID-19 Vaccine Mandates in Schools"

  Issued on February 15, 2025, this order prohibits K-12 schools from enforcing COVID-19 vaccine mandates, asserting the importance of keeping education accessible without medical prerequisites. It threatens to withhold federal funding from institutions that implement such mandates.

Impacted Federal K-12 Grant Programs

The enforcement of these executive orders has led to substantial changes in K-12 grant programs across federal government:​

• US Department of Education (ED)

  The Department of Education terminated $600 million in teacher training grants awarded through the Teacher Quality Partnership and Supporting Effective Educator Development programs, citing concerns over the promotion of "divisive ideologies" such as critical race theory and social justice activism. 

• National Science Foundation (NSF)

  The NSF is in the midst of a comprehensive review of its programs to ensure compliance with the executive orders. This includes scrutinizing grants that involve DEIA principles, potentially affecting funding for educational initiatives aimed at promoting diversity in STEM fields. ​

• Health Resources and Services Administration (HRSA)

  HRSA has directed that grant funds may not be used for activities conflicting with the executive orders, including DEI initiatives. This impacts programs that address health education and services in K-12 schools, particularly those serving diverse populations. ​

Legal Challenges and Responses

The implementation of these executive orders has led to legal challenges:

• Lawsuits Against Funding Cuts

  Eight Democratic-led states have filed lawsuits challenging the Department of Education's decision to cut funding from federal grant programs designed for teacher training. The plaintiffs argue that these grants as essential for addressing teacher shortages, particularly in rural and underserved areas, by funding scholarships and training for aspiring educators. 

• Preliminary Injunctions

  Federal judges have issued preliminary injunctions blocking certain executive orders that attempt to terminate or alter federal grants and contracts related to DEI programs. The courts have cited concerns over vagueness and potential unconstitutionality in penalizing private organizations, including educational institutions. ​

Implications for K-12 Educators and Institutions

The recent executive orders have introduced substantial changes to federal policy and programs affecting K-12 education. The potential dissolution of the Department of Education and the shift toward state-controlled systems may lead to disparities in educational quality and resource allocation. Additionally, the emphasis on school choice and the reduction of DEI programs could impact efforts to promote inclusive and equitable learning environments. Educators and institutions must stay informed and navigate these policy changes carefully to continue providing equitable and quality education. 

Four Ways to Make a STEM Mentoring Program Successful

Effective mentorship plays a crucial role in the success and development of higher education students, particularly in STEM fields. This blog post explores four strategies for training STEM faculty to be effective mentors and discusses the importance of measuring outcomes to ensure the effectiveness of these mentorship programs.

Offer Mentorship Training Programs

To equip STEM faculty with the necessary skills to be effective mentors, institutions of higher education are advised to provide mentorship training programs. These programs can cover topics such as effective communication, building trust, setting goals, and providing constructive feedback. By providing training, institutions ensure that faculty members are equipped with the knowledge and skills to support and guide their mentees effectively.

Establish Mentor-Mentee Guidelines

Clear guidelines and expectations for mentor-mentee relationships help ensure positive mentoring experiences. These guidelines can outline the roles and responsibilities of both mentors and mentees, as well as the frequency and format of meetings. By setting clear expectations, institutions ensure that mentorship relationships are structured and productive. For instance, at the University of Nevada-Las Vegas, guidelines for developing a mentoring program provide guidance to individual faculty or faculty teams.

Create a Supportive Mentorship Network

A supportive mentorship network can provide a space where faculty mentors can connect, share experiences, and seek guidance from their peers. This can be achieved through mentorship circles, faculty mentoring communities, or regular mentorship workshops. For example, at the University of Houston, a monthly mentorship forum is organized where faculty mentors can discuss challenges, share strategies, and learn from each other's experiences. By creating a supportive network, institutions promote continuous learning and improvement among faculty mentors.

Measure Mentoring

Measuring the outcomes of mentorship programs is crucial to assess their effectiveness and make necessary improvements. Some key metrics to consider including:

• Mentee Satisfaction Surveys: Conducting regular surveys to gather feedback from mentees can provide insights into their satisfaction with the mentorship program. These surveys can assess mentees' perceptions of the support received, the quality of guidance provided, and the overall impact of the mentorship relationship.

  
  

• Mentee Academic Performance: Tracking the academic performance of mentees can assess the impact of mentorship on their success. Comparing the performance of mentees with non-mentored students can provide insights into the effectiveness of the mentorship program in improving academic outcomes.

  
  

• Mentee Retention and Graduation Rates: Monitoring the retention and graduation rates of mentees can indicate the impact of mentorship on their persistence and completion of STEM programs. Higher retention and graduation rates among mentees suggest that mentorship has a positive influence on their academic journey.

Training STEM faculty to be effective mentors is crucial for the success and development of students in STEM fields. By providing mentorship training programs, establishing mentor-mentee guidelines, and creating a supportive mentorship network, institutions can enhance the quality of mentorship provided. Measuring outcomes through mentee satisfaction surveys, academic performance, and retention rates allows institutions to assess the effectiveness of their mentorship programs and make necessary improvements. By continuously investing in mentorship training and evaluation, institutions can ensure that STEM faculty mentors are equipped to support and guide their mentees effectively, ultimately contributing to the success and growth of STEM students.

Interested in working with Shaffer Evaluation Group? Contact us today for a free 30-minute consultation: seg@shafferevaluation.com.