How to Read Your Evaluation Plan

So you’ve connected with your evaluator and they have produced an evaluation plan for you, but perhaps evaluation is so new to you that the plan seems to be another language! SEG is here to help break down the components of the evaluation plan to make it more understandable and clearer.

The evaluation plan contains useful information for you, such as the timeline for when to expect deliverables from your evaluator, the logic model that illustrates your program design, and the evaluation matrix or framework (discussed more below).

For a typical SEG evaluation, the evaluation plan is broken into three sections: the fidelity of implementation, formative or process evaluation, and summative evaluation. Each of these sections have their own evaluation questions associated with them. You’ll find each of these questions in the evaluation matrix of your evaluation plan, along with the analytical procedure and data collection procedure.

For example, a question under fidelity of implementation may be ‘to what extent was the project implemented as designed?’ This question helps the evaluator assess if the project is on track with implementation as it was originally laid out in the grant application. Looking at the example below, we see that an implementation task this project needs to complete in its first year is to hire a project director. The evaluator will request project documentation for each implementation task (under data collection procedure) and then review documentation to assess the progress (analytical procedure).

Another example comes from the formative evaluation, with the question: What feedback was offered by campus stakeholders about the implementation and benefits of project activity components? What recommendations for improvement were offered by campus stakeholders? This section focuses predominately on collecting qualitative data on stakeholder feedback about project activities. Questions in the formative evaluation typically address successes, challenges, and sustainability. In the example below, this question focuses on the stakeholder group of students, with evaluators conducting a focus group to collect information (data collection procedure), and then conducting qualitative analysis on the transcript (analytical procedure).

A final example comes from the summative evaluation, with the question: To what extent has the project met its intended objectives? This section typically focuses on project outcomes and, for federal grants, may also contain Government Performance and Results Act (GPRA) performance measures. For this question and example data point from a higher education evaluation, the evaluator would work with an institutional research (IR) office to obtain lists of first-time, full-time students retained from fall-to-fall semester (data collection procedure) and then conduct an independent analysis separate from the IR office to verify the data point (analytical procedure).

The evaluation plan and matrix are particularly important for understanding the data your evaluator needs every year to create an accurate report. Therefore, being able to read and comprehend the plan is helpful when working with your evaluator.

Shaffer Evaluation Group is a trusted partner in evaluation. This partnership starts at pre-award and continues throughout the grant's life cycle. If you're planning to submit a grant application and require an evaluation partner, please contact us at seg@shafferevaluation.com.

Involving Evaluators on Grant Proposals

By Stacy Hayden, SEG Research Associate

Shaffer Evaluation Group recently finished working with several partners to develop and submit grant applications for the Department of Defense Education Activity Broad Agency Announcement for 2023. SEG frequently partners with school districts, institutions of higher education, tribal governments, and community organizations on grant applications. Our partnership starts during the period of grant development and continues into the grant term.

Shaffer Evaluation Group offers fee-based grant writing services as well as post-award support when selected as the external evaluator. We also offer limited complimentary services to support your grant application—we write the evaluation section at no cost in exchange for being named as the evaluator in your grant application. Here are some questions you may have about working with SEG on a complimentary basis to develop a grant proposal.

1. How will you be involved in the grant proposal process? What will your role be?

While every proposal is handled a bit differently, SEG always works as an external partner in the process. We typically ask clients to contact us once a full rough draft is developed as we cannot do our work without understanding the scope and specific activities being proposed for the project. However, sometimes clients need support with developing goals and objectives. If this is the case, SEG can often connect with clients briefly via a virtual meeting to discuss the drafted goals and provide initial feedback.

After the draft is sent to SEG, we work to develop the designated sections. These are sent to the point of contact once developed for feedback. If needed, SEG will make changes and send the final version back to the client.

2. What sections of the grant narrative will you be responsible for preparing?

Each grant solicitation requires different components. Typically, SEG prepares the evaluation section, which may include a narrative section and/or an evaluation framework. In some cases (e.g., Department of Education proposals, National Science Foundation proposals), SEG may also prepare a logic model and/or theory of change for the application. However, the scope varies by type of proposal. Coordinating this with the evaluator at the beginning of the proposal process will ensure everyone is on the same page.

3. Your expertise spreads to other sections of the grant proposal too. Can our organization have your support in those sections as well?

Often there are additional questions/requirements that reference evaluation or the role of the evaluator throughout the grant application beyond the evaluation section (e.g., “plan for monitoring the implementation and effectiveness of professional learning”). SEG is happy to provide support in preparing these sections by providing clarification on components within the evaluation plan, answering questions about typical practices, and providing feedback on client ideas. However, we typically cannot write these sections as part of our complimentary grant support, but feel free to contact us to learn about our fee-based grant writing services.

4. How do we contact you if we have questions throughout the proposal process?

Technology has made collaborating on grants incredibly easy! Many of our partners choose to use tools such as Google Docs to develop their grant proposal. This is a great way to share your grant proposal with us, particularly as it allows us to see changes over time. However, we typically look at this at key points in time as opposed to checking it frequently. When we are providing complimentary grant proposal support, it is helpful if you email us with questions as opposed to tagging us in online documents as we may not be checking these frequently.

5. Are you able to attend all our grant team meetings?

While we find the time we are able to meet with clients who are writing proposals helpful, unfortunately we cannot attend all grant team meetings during the proposal development process. We often have to balance the work we are already doing for our clients with supporting grant proposals. While we are happy to meet to discuss your plan for the grant/answer questions, we try to meet one time toward the beginning of the process and then answer additional questions as needed via email or phone.

We love having the opportunity to partner with clients prior to grant funding. Being able to provide input at that stage helps to ensure projects are set up for success with strong goals, objectives, and plans to measure progress. The complimentary support provided by SEG at no cost in exchange for being named as the evaluator in your grant application varies from proposal to proposal, but the questions above are a great way to discuss expectations with your evaluator prior to beginning the process.

As previously noted, we also offer fee-based grant writing services that position SEG as a member of the grant team and provide more support. Please contact SEG if you are interested in partnering on a grant proposal -- whether using our complimentary support services or fee-based services.

Consulting Stakeholders in the Grant Writing Process

Developing and writing an educational grant can seem like a daunting process of where to begin, especially considering the level of detail that may need to be etched into each goal, objective, or outcome. In addition, it is also important to consider how aligned these goals, objectives, or outcomes are to the needs of the populations you are serving or the populations implementing the project. One tactic that may be useful is consulting stakeholders – students, project staff, faculty, teachers, school / university leadership – during the grant writing process to ensure that their voices and needs are represented early in the process.

Who is a Stakeholder?

Stakeholders are interested individuals or groups with an interest in your project’s implementation or outcomes. Stakeholders can be:

• Implementers of the activities that are part of the project (e.g., teachers, faculty)

• Advocates for sustaining the project based on successful outcomes (e.g., parents, business community)

• Funders or authorizers who will support the continuation or expansion of the project (e.g., school board, regional foundation)

• Beneficiaries of the project (e.g., students)

Tips for Inviting Stakeholder Feedback in the Grant Writing Process

1. Come to stakeholders with specific questions in mind. It may be a good idea to ask your various stakeholders tailored and detailed questions that aim to pull their needs forward (see table).

Stakeholder	Example Questions
Staff implementing the project	We would like to implement this specific project: Is this feasible for us to implement within the timeframe we have specified? What would be the specific action steps needed to implement this program fully? (e.g., are there barriers for getting action steps approved by administrators?) Do staff have the time availability to help fully implement this project?
Project beneficiaries	We are considering implementing this project that you will be receiving: Is this something that you feel is needed? Do you have recommendations on delivering this program? Are there programmatic aspects / components that we are missing?
Funders or authorizers	We are seeking funding for this project: Is this project something that you feel is needed in our community? What type of evidence would you need to fund or authorize the continuation of this project in the future?

2. Make stakeholder groups aware of their roles, responsibilities, and expectations that you have laid out throughout the life of the grant. To help establish that this project is feasible, it is important to fully describe the intended project to stakeholders you are consulting. For example, teachers may be expected to deliver specific components of the program directly to their students, therefore it may be important for teacher feedback on the feasibility of the program to be considered.

3. Demonstrate your commitment to stakeholder participation by including a stakeholder advisory group in your project proposal. A stakeholder advisory group offers benefits across the life of your project, including community support and buy-in, advice and feedback on project implementation, and communication channels to disseminate the results of your project. In your grant application, identify the types of stakeholders you intend to recruit and the types of advice they will contribute across the term of the project.

By inviting the stakeholder groups early in the process, you are helping to enable teamwork, create buy-in, and de-silo the processes the project needs to go through within the grant’s life course. Engaging stakeholders early on helps to ensure project success.