How to Work with an Evaluator during the Pre-award Phase

By Courtney Hagan

Research Associate

Congratulations for deciding to create and submit an application for a grant opportunity! While this is an exciting time, filled with multiple possibilities, it may also be a stressful time, filled with endless inquiries. The evaluator you have invited into the pre-award phase is here to help make this time period a little less stressful. As a project director shared,

"it's like having a thought partner with you through the process."

Below are five tips to work with your evaluator during the submission process.

Tip #1 - Contact the evaluator early in the process.

When you decide to pursue a funding opportunity, reach out to the evaluator to share your intention to submit a proposal. Inquire if the evaluator has time to support your proposal preparation by contributing the evaluation section that would be needed by a particular date. Remember to back that date up from the due date, as often it takes time to route a completed proposal through an organization’s various departments for signatures. Also, let the evaluator know when you anticipate having a solid draft of the goals, objectives, and project activities ready.

Tip #2 - Share your application materials.

To give your evaluator a sense of the scope of your project, the timeline, and the goals/objectives, send your evaluator your grant application materials as you are writing them. A well-developed abstract works quite well in combination with the funder's solicitation, especially the evaluation and reporting requirements. It can be useful to use a shared folder. This helps the evaluator to develop the evaluation approach and logic model (if the application calls for it).

Tip #3 - Check in frequently with your evaluator.

While this may be a hectic time trying to ensure that deadlines are met and that the application is well-written, periodic check-ins with your evaluator ensure communication pathways are open and strong for the dynamic process of the pre-award phase. This is especially vital if a significant change occurs to the planned goals that were previously communicated. As a long-time higher education SEG client reflected, these:

"Interactions are really helpful and valuable to think through what was going to happen and what kind of results we [would] get."

Tip #4 - Make the evaluator aware of deadlines and grant specifications.

Ensure that your evaluator knows your deadlines for their specified pieces of the grant application. Further, if there are grant award requirements or project-specific components that need to be highlighted, communicate that ahead of time to your evaluator as something to include throughout the application.

Tip #5 - Commit to contracting with the evaluator should your organization receive the grant.

If your evaluator is contributing their time and expertise to support your grant application, commit to contracting them should your organization receive a grant award. Request a cost proposal and negotiate their proposed post-award fee prior to finalizing your grant application budget. Always consult your procurement office before engaging an evaluator in the pre-award process.

Shaffer Evaluation Group is a trusted partner in evaluation. This partnership starts at pre-award and continues throughout the grant's life cycle. If you're planning to submit a grant application and require an evaluation partner, please contact us at seg@shafferevaluation.com.

Not Just Another Dusty Shelf Report: How to Use Your Report Beyond Fulfilling a Funding Requirement

By Stacy Hayden, M.A., SEG Research Associate

Evaluator-prepared reports according to one recently interviewed project director have utility beyond meeting a funder’s requirements as they can inform future grant writing and current grant cycle activities. Report recipients can share sections of the report with others (e.g., workgroup, committee, project team) and use specific questions to explore the findings and plan for the future or use the evaluator-prepared report as a source for completing other reports. Six commonly used evaluation report sections are highlighted below and with an explanation of how each may be useful to review with the project team and other key constituents along with list of guiding questions.

1. Executive Summary: The most commonly shared section of longer reports, the executive summary is especially effective for providing leadership with a high-level summary of a project. Many project directors use them as a handout or as bullets in a slide deck. In this application, questions that may be asked include:

• Given the evaluation report findings, what resonates with you?

• What opportunities or connections do you see with the work of this project to other initiatives within our organization (e.g., college, school system)?

2. Fidelity of Implementation Section/Table: Many reports include fidelity of implementation data in a table. Fidelity of Implementation assesses to what extent the project is being implemented in accordance with the proposed plan. When reviewing indicators in a fidelity of implementation section some questions you may wish to ask are:

• Based on the indicators that are met, what are we doing well?

• What strategies helped us to meet this indicator?

• Based on the indicators that are not met, what do we still need to work on?

• Why did we not meet this indicator (if applicable)? What steps would we need to take to meet this indicator in the future?

3. Process Monitoring Section/Table: Some reports will also include a section on process monitoring. Process monitoring provides information to help improve the project over time. Often, this information comes from participants (e.g., students, teachers/faculty, staff). When reviewing indicators or questions in a process monitoring section some questions you may wish to ask are:

• What do interested parties/participants (e.g., students, community members, teachers, administrators) think is going well? How can we continue to improve or expand on what is going well?

• What do interested parties/participants think needs improvement? What action steps should be taken to address these concerns?

• Based on feedback, what aspects of the project might be best/feasible to sustain beyond the period of grant funding? What steps would need to be taken to achieve sustainability?

4. Survey Results: Many evaluations also involve surveys or feedback forms. Reviewing this data can be quite informative. Sometimes, specific data points are selected and presented throughout the report. When this happens, the project director can curate these statistics for review. In other cases, tables or graphs may be presented in the report that can be used.

• What surprises you about that data? In other words, what do you notice as outliers (i.e., higher or lower). Why do you think it is higher or lower?

• What does the data mean for the project? What should be sustained? What may need to be changed?

5. Sustainability: In many reports, the evaluator is asked to discuss what elements of the project are emerging as sustainable. There are seven key domains of sustainability (described in our June 2022 post). As you explore this section, note what areas the evaluator identified as sustainable for your project. Consider:

• What domains of sustainability are not represented in this section?

• What aspects of this project could emerge as sustainable? What action steps are needed to take to accomplish this?

6. Recommendations: Finally, most reports include a recommendations/conclusions section. This section may be written in a bulleted list, which makes it easier for sharing with a work group or committee. As you discuss each recommendation you may wish to explore the following questions:

• Which recommendations are most feasible to address at this time?

• Which recommendations may need a longer timeframe and are worthy of addressing?

• What action steps need to be taken to address this recommendation?

• If a recommendation cannot feasibly be addressed, what is an alternate action that can be taken to partly address the recommendation?

The Hard Numbers: Data Sharing

By Courtney Hagan, Ph.D., SEG Research Associate

Evaluation requires data analysis to ensure that a project or program is making progress on its stated objectives and goals (e.g., increasing retention by 2% every year). Evaluators will request data from project or program teams on a regular basis to help ensure that these data points are included in evaluation reporting. Four tips to help ensure the proper transfer of data when working with your evaluation team are:

1. Make a data sharing plan. As stated in our October 2022 blog, “Maintaining a Project Documentation Archive”, setting up a secure shared folder is important for sharing information with your evaluator; this includes sharing data as well.

2. Involve your data people. Share the evaluation plan and data needs early with the data people in your organization (e.g., institutional research) so that they can provide insight on what they can readily provide, ask questions, and offer refinements. It can be helpful to e-introduce the evaluator and your institutional data point-of-contact.

3. Review your data request (aka "data call"). Your evaluator will request specific data items from your team. These data items align with the evaluation matrix that was created based on your project or program design. Often these data items are accompanied by definitions, since terms such as "student persistence" or "student engagement" can be defined differently on each campus. Only data that is requested needs to be uploaded in the shared folder.

4. Ensure your data is accurate, clean, and interpretable. Sometimes the data calls can come at inopportune and busy times; however, it is important to ensure the data is accurate, clean, and interpretable to third parties to facilitate data analysis by the external evaluator. Further, your evaluation team may request individual-level data (instead of aggregate), to ensure that the team can make the calculations on their own (as an objective third party). Because this data is requested at the individual-level, it is advisable to de-identify the data by using student ID numbers instead of student names.

As always, every evaluation project is unique. Therefore, it is best practice to consult with your evaluation team to make the best determination with how and in what ways you should be sharing your data.

Related post: In our October 2022 blog, we discussed sharing project documentation with your evaluator (Maintaining a Project Documentation Archive).