Maintaining a Project Documentation Archive

In last month’s blog—So Your Project Got Funded…Now What?—we reviewed steps project directors should take to support successful implementation once notice is received that your project has been funded. One of those key steps is to keep an implementation log, which provides a simple way to track grant activities (e.g., meetings, emails, programs), including who was involved and what resulted from the activities. Implementation logs were also discussed in our July 2022 blog post.

Professional development calendars and event sign-in sheets, meeting agendas and minutes, course rosters, event flyers, and other project documentation may be linked by project directors to implementation log entries. These artifacts provide evidence that project activities took place and provide insight on how a project was implemented.

Well organized data enhances the project director’s ease in providing information for the project evaluator. Five tips on how to maintain a project documentation archive appear below using a fictional project called MIST (Motivating Individuals in Science and Technology).

1. Set up a shared folder with your evaluator. Consult with your evaluator about selecting a cloud-based file platform that both parties can securely use. Google Drive is a popular option for many school districts and higher education institutions, but there are plenty of other options, including Box, Dropbox, and OneDrive. If you are transferring sensitive information, such as personally identifiable information, make sure the system you select is secure.

2. Organize sub-folders and files within the shared folder using a structure that you can maintain such as by activity, goal, strategy, reporting timeframe, distinct group (e.g., student, staff, faculty), etc. For MIST, there was an afterschool student club component, grade-level field trip activity, and teacher training. For MIST, an appropriate folder structure included three sub-folders (one for each main area) and a fourth sub-folder with general information such as the budget, grant application, and reporting requirements. Within each sub-folder, add files and additional folders as appropriate.

3. Name files for clarity: consistently name files with the project, what it is, and date. For example, a participant sign-in sheet for the MIST professional development may be “MIST_PD_Sign-in_Fall_2022.”

4. Leave breadcrumbs. Some files are maintained in other places such as an online survey tool, YouTube, or shared from a colleague’s Google Drive. A tip is to have a Word document or Excel spreadsheets with the links to where other documentation sources are found.

5. Annotate your documentation as needed. When something goes great, awry, or perhaps somewhere in between, consider leaving an electronic note on the documentation to remind you later of what occurred. Using the MIST afterschool club as an example, perhaps attendance was down due to the flu, students had great quotes about an activity, or there was tweaking of the initial schedule to better align with student needs and interests that emerged.

Not all project documentation needs to be kept. Your project evaluator is best positioned to provide you guidance on which project documentation you should maintain based on your project’s evaluation plan.

So Your Project Got Funded... Now What?

The length of time between when proposals must be submitted for grant programs and when they are funded can often be several months to almost a year. During that time, a lot of things in an organization to change. After you receive notice that your project will be funded, we recommend taking the following steps to support successful project implementation.

1. Reread the Grant Application

When organizations prepare grant applications the information that is included is accurate to the best of their knowledge. However, things can change during the period of time between submitting the proposal and receiving funding. As you reread the funded application, take notes about what your organization has committed to doing. This should also include any timelines or information (e.g., number of participants, number of activities) that were specified. Look for the data that were specified in the application and determine how frequently they were to be measured and reported.

If the application included a logic model or evaluation framework, these may be good places to pull some of the information from. However, it will be important to also add information from the application that is not included. It may be helpful to create a list in excel of each item on your “to do” list.

2. Begin to Keep an Implementation Log

As soon as you begin to work on the project start tracking activities in an implementation log. An implementation log tracks grant activities (e.g., meetings, emails, programs), including who was involved and what resulted from the activities. More information about implementation logs can be found in our August 2022 blog post (https://www.shafferevaluation.com/post/implementation-logs). Particularly in the first year of a project, an implementation log can help to track all of the “behind the scenes” work that happens to implement a project. This is a helpful tool for project staff and evaluators to report on what has occurred with the project.

3. Confirm the Feasibility of the Activities/Data Collection

Hopefully you were able to collaborate with stakeholders or solicit stakeholder feedback while writing the grant proposal. In an ideal world, the time between submitting the application and funding would have been quick as well, allowing for limited changes within the organization. However, this is not always the case.

Once you have created the list of activities or data collection required, think through who will need to be involved. For example, activities may need an administrator to approve it or data may need to be obtained from someone in charge of a certain department. Prepare a list of how each stakeholder will expect to be involved or support the project and then follow up with them and confirm the feasibility of the plan outlined in the grant application. For example, if the grant specified that the middle school would get a new makerspace, create several STEM-focused clubs, and implement a new STEM course, there are several stakeholders project staff should speak to prior to beginning implementation. First, middle school administration needs to be aware of the purpose of the grant, the expected outcomes, and what they are expected to do. It will be important to talk with the administrators to ensure these activities are feasible at their school. If the proposed activities involve teachers working outside of their regular hours (e.g., running STEM clubs), it also may be helpful to reach out to prospective teachers to see if this is something they would be interested in doing. The creation of the STEM course would likely need to follow a district process and confirming the process in advance will help avoid any roadblocks during the process.

In some situations, there may be factors that do not allow a proposed project to be implemented as planned. For example, perhaps there was a space that was planned for the makerspace, but in the time since the application went in the space is now occupied. Working with stakeholders at the onset of the project will allow you to build initial support for the implementation or understand where you will need to pivot or adjust plans.

4. Plan in Advance for Sustainability

Once you hold discussions with stakeholders about planned project activities, you’ll have a better understanding of the logistics involved with each project activity. Although the project is just beginning, this is the best time to start thinking about what can be sustained after grant funding ends. More information about sustainability planning can be found in our June 2022 blog post (https://www.shafferevaluation.com/post/planning-for-sustainability).

5. Contact the Evaluator and Funder to Discuss Any Potential Changes

At this point, if you’ve decided to make any changes to the proposed project, you’ll want to contact the funder and potentially the evaluator. If the evaluator was involved in preparing the initial grant application we would recommend contacting the evaluator first. While the evaluator will be unable to tell you if the funder will approve any changes, the evaluator can help you think through any implications related to the change (e.g., how activities will affect data collection). The evaluator may also have suggestions of questions you should ask the funder when you speak with them.

Next, contacting the funder will be important. Grant programs have different requirements when it comes to changing goals, activities, and expected outcomes. Reviewing any suggested changes with the funder and getting them improved is important. Further, if money is going to be used for a different purpose, there may be procedures to follow to gain approval.

6. Develop a Work Plan

Once you’ve made final decisions about any changes to the program and had these approved, it is time to plan for how to make your plans come to fruition. Although you will have a list of activities to be implemented and data to be collected, there are many steps to ensuring these happen successfully and on time. We recommend developing a work plan. In a work plan, each activity is broken down into the individual steps required for completing that activity. For example, conducting an annual teacher survey might include the following tasks:

· Contact evaluator for survey link

· Send survey link to points of contact

· Confirm points of contact have sent survey link to teachers

· Send follow up email #1

· Send follow up email #2

· Send follow up email #3 (only to schools with low response rate)

Work plans also include the staff responsible for completing the activity and the date of the activity. In some cases, it may also be appropriate to include the amount of time for completing the activity. Developing a work plan helps to break activities down into manageable steps to ensure project activities are completed on time.

Hearing that a project has been funded is an exciting time. Completing the steps above will help project directors ensure smooth and successful project implementation.

Quantitative Analysis in Program Evaluation

Quantitative data analysis is an important part of the program evaluation process. Basic statistics help us understand and communicate educational program results.

There are several ways an evaluator can approach statistically analyzing their data, and this approach depends on (1) the research question and (2) the data at hand. For example, the type of statistical test an evaluator may use can depend on if they are interested in testing mean differences or predicting an outcome. Further, the statistical test an evaluator may use will also depend on the variable type (e.g., continuous, categorical, interval, etc.).

For most evaluation and outcome analysis, the most basic and useful analysis is to compare measurements taken at two points in time. A

meaningful measure will be a mean, or average, of the values collected. For example, for a survey administered at the beginning and end of an educational program that collects data on college students’ STEM efficacy, we could use a t-test to see if there is any significant difference between the two average values. An ANOVA will compare two or more means simultaneously and also takes into consideration the variance within a sample and between samples.

Basic statistics that an evaluator may use are:

1. Descriptive statistics: Descriptive statistics are useful for when an evaluator has data that they would like to describe. Tests within this category include means (averages), percentages, or range.

2. Inferential statistics: Inferential statistics allow the evaluator to make inferences (predictions) about the data.

a. Comparison tests: Comparison tests are useful for when an evaluator would like to test the group mean differences between or within participants. Common tests include: t-test, paired t-test (for pre- and post- test data), and ANOVA.

b. Correlational tests: Correlational statistics are useful for when an evaluator would like to examine the relationship between variables or predict an outcome. Common tests include: correlation, regression, and hierarchical linear modeling.