ABOUT US

By Courtney Hagan, Ph.D., SEG Research Associate


Evaluation requires data analysis to ensure that a project or program is making progress on its stated objectives and goals (e.g., increasing retention by 2% every year). Evaluators will request data from project or program teams on a regular basis to help ensure that these data points are included in evaluation reporting. Four tips to help ensure the proper transfer of data when working with your evaluation team are:


1. Make a data sharing plan. As stated in our October 2022 blog, “Maintaining a Project Documentation Archive”, setting up a secure shared folder is important for sharing information with your evaluator; this includes sharing data as well.


2. Involve your data people. Share the evaluation plan and data needs early with the data people in your organization (e.g., institutional research) so that they can provide insight on what they can readily provide, ask questions, and offer refinements. It can be helpful to e-introduce the evaluator and your institutional data point-of-contact.


3. Review your data request (aka "data call"). Your evaluator will request specific data items from your team. These data items align with the evaluation matrix that was created based on your project or program design. Often these data items are accompanied by definitions, since terms such as "student persistence" or "student engagement" can be defined differently on each campus. Only data that is requested needs to be uploaded in the shared folder.


4. Ensure your data is accurate, clean, and interpretable. Sometimes the data calls can come at inopportune and busy times; however, it is important to ensure the data is accurate, clean, and interpretable to third parties to facilitate data analysis by the external evaluator. Further, your evaluation team may request individual-level data (instead of aggregate), to ensure that the team can make the calculations on their own (as an objective third party). Because this data is requested at the individual-level, it is advisable to de-identify the data by using student ID numbers instead of student names.


As always, every evaluation project is unique. Therefore, it is best practice to consult with your evaluation team to make the best determination with how and in what ways you should be sharing your data.


Related post: In our October 2022 blog, we discussed sharing project documentation with your evaluator (Maintaining a Project Documentation Archive).

In last month’s blog—So Your Project Got Funded…Now What?—we reviewed steps project directors should take to support successful implementation once notice is received that your project has been funded. One of those key steps is to keep an implementation log, which provides a simple way to track grant activities (e.g., meetings, emails, programs), including who was involved and what resulted from the activities. Implementation logs were also discussed in our July 2022 blog post.


Professional development calendars and event sign-in sheets, meeting agendas and minutes, course rosters, event flyers, and other project documentation may be linked by project directors to implementation log entries. These artifacts provide evidence that project activities took place and provide insight on how a project was implemented.

Well organized data enhances the project director’s ease in providing information for the project evaluator. Five tips on how to maintain a project documentation archive appear below using a fictional project called MIST (Motivating Individuals in Science and Technology).


1. Set up a shared folder with your evaluator. Consult with your evaluator about selecting a cloud-based file platform that both parties can securely use. Google Drive is a popular option for many school districts and higher education institutions, but there are plenty of other options, including Box, Dropbox, and OneDrive. If you are transferring sensitive information, such as personally identifiable information, make sure the system you select is secure.


2. Organize sub-folders and files within the shared folder using a structure that you can maintain such as by activity, goal, strategy, reporting timeframe, distinct group (e.g., student, staff, faculty), etc. For MIST, there was an afterschool student club component, grade-level field trip activity, and teacher training. For MIST, an appropriate folder structure included three sub-folders (one for each main area) and a fourth sub-folder with general information such as the budget, grant application, and reporting requirements. Within each sub-folder, add files and additional folders as appropriate.


3. Name files for clarity: consistently name files with the project, what it is, and date. For example, a participant sign-in sheet for the MIST professional development may be “MIST_PD_Sign-in_Fall_2022.”


4. Leave breadcrumbs. Some files are maintained in other places such as an online survey tool, YouTube, or shared from a colleague’s Google Drive. A tip is to have a Word document or Excel spreadsheets with the links to where other documentation sources are found.


5. Annotate your documentation as needed. When something goes great, awry, or perhaps somewhere in between, consider leaving an electronic note on the documentation to remind you later of what occurred. Using the MIST afterschool club as an example, perhaps attendance was down due to the flu, students had great quotes about an activity, or there was tweaking of the initial schedule to better align with student needs and interests that emerged.


Not all project documentation needs to be kept. Your project evaluator is best positioned to provide you guidance on which project documentation you should maintain based on your project’s evaluation plan.

The length of time between when proposals must be submitted for grant programs and when they are funded can often be several months to almost a year. During that time, a lot of things in an organization to change. After you receive notice that your project will be funded, we recommend taking the following steps to support successful project implementation.


1. Reread the Grant Application

When organizations prepare grant applications the information that is included is accurate to the best of their knowledge. However, things can change during the period of time between submitting the proposal and receiving funding. As you reread the funded application, take notes about what your organization has committed to doing. This should also include any timelines or information (e.g., number of participants, number of activities) that were specified. Look for the data that were specified in the application and determine how frequently they were to be measured and reported.


If the application included a logic model or evaluation framework, these may be good places to pull some of the information from. However, it will be important to also add information from the application that is not included. It may be helpful to create a list in excel of each item on your “to do” list.


2. Begin to Keep an Implementation Log

As soon as you begin to work on the project start tracking activities in an implementation log. An implementation log tracks grant activities (e.g., meetings, emails, programs), including who was involved and what resulted from the activities. More information about implementation logs can be found in our August 2022 blog post (https://www.shafferevaluation.com/post/implementation-logs). Particularly in the first year of a project, an implementation log can help to track all of the “behind the scenes” work that happens to implement a project. This is a helpful tool for project staff and evaluators to report on what has occurred with the project.


3. Confirm the Feasibility of the Activities/Data Collection

Hopefully you were able to collaborate with stakeholders or solicit stakeholder feedback while writing the grant proposal. In an ideal world, the time between submitting the application and funding would have been quick as well, allowing for limited changes within the organization. However, this is not always the case.


Once you have created the list of activities or data collection required, think through who will need to be involved. For example, activities may need an administrator to approve it or data may need to be obtained from someone in charge of a certain department. Prepare a list of how each stakeholder will expect to be involved or support the project and then follow up with them and confirm the feasibility of the plan outlined in the grant application. For example, if the grant specified that the middle school would get a new makerspace, create several STEM-focused clubs, and implement a new STEM course, there are several stakeholders project staff should speak to prior to beginning implementation. First, middle school administration needs to be aware of the purpose of the grant, the expected outcomes, and what they are expected to do. It will be important to talk with the administrators to ensure these activities are feasible at their school. If the proposed activities involve teachers working outside of their regular hours (e.g., running STEM clubs), it also may be helpful to reach out to prospective teachers to see if this is something they would be interested in doing. The creation of the STEM course would likely need to follow a district process and confirming the process in advance will help avoid any roadblocks during the process.


In some situations, there may be factors that do not allow a proposed project to be implemented as planned. For example, perhaps there was a space that was planned for the makerspace, but in the time since the application went in the space is now occupied. Working with stakeholders at the onset of the project will allow you to build initial support for the implementation or understand where you will need to pivot or adjust plans.


4. Plan in Advance for Sustainability

Once you hold discussions with stakeholders about planned project activities, you’ll have a better understanding of the logistics involved with each project activity. Although the project is just beginning, this is the best time to start thinking about what can be sustained after grant funding ends. More information about sustainability planning can be found in our June 2022 blog post (https://www.shafferevaluation.com/post/planning-for-sustainability).


5. Contact the Evaluator and Funder to Discuss Any Potential Changes

At this point, if you’ve decided to make any changes to the proposed project, you’ll want to contact the funder and potentially the evaluator. If the evaluator was involved in preparing the initial grant application we would recommend contacting the evaluator first. While the evaluator will be unable to tell you if the funder will approve any changes, the evaluator can help you think through any implications related to the change (e.g., how activities will affect data collection). The evaluator may also have suggestions of questions you should ask the funder when you speak with them.


Next, contacting the funder will be important. Grant programs have different requirements when it comes to changing goals, activities, and expected outcomes. Reviewing any suggested changes with the funder and getting them improved is important. Further, if money is going to be used for a different purpose, there may be procedures to follow to gain approval.


6. Develop a Work Plan

Once you’ve made final decisions about any changes to the program and had these approved, it is time to plan for how to make your plans come to fruition. Although you will have a list of activities to be implemented and data to be collected, there are many steps to ensuring these happen successfully and on time. We recommend developing a work plan. In a work plan, each activity is broken down into the individual steps required for completing that activity. For example, conducting an annual teacher survey might include the following tasks:

· Contact evaluator for survey link

· Send survey link to points of contact

· Confirm points of contact have sent survey link to teachers

· Send follow up email #1

· Send follow up email #2

· Send follow up email #3 (only to schools with low response rate)


Work plans also include the staff responsible for completing the activity and the date of the activity. In some cases, it may also be appropriate to include the amount of time for completing the activity. Developing a work plan helps to break activities down into manageable steps to ensure project activities are completed on time.


Hearing that a project has been funded is an exciting time. Completing the steps above will help project directors ensure smooth and successful project implementation.