Implementation logs: A small investment that yields powerful evidence

Tracking time and effort as part of federal grant administration is a best practice, as it provides documentary evidence of the amount of time project staff spend on grant activities. Time and effort forms for project personnel are used for tracking. These forms are usually organized by cost objectives that represent a program, function, activity, award, or work unit of which cost data are desired or required for reporting. Employees may be involved in several federally funded projects or locally funded programs that require tracking by cost objective.

Evaluators may ask federally funded project personnel to share documentation from their time and effort forms or to maintain an implementation log. Implementation logs place stronger emphasis on tracking grant-funded activities as opposed to project personnel time. Logs establish a documentation trail that serves both the project and evaluation team. Often created in an Excel spreadsheet or using an online form, project directors make dated entries in an implementation log for activities and include the time spent, participants and note relevant artifacts (see image). Some project directors place Google Drive links in the artifacts cell, while others provide a file name and location. While logs could be updated daily, some are updated on a more periodic basis.

The value of the implementation log becomes apparent when it is time to produce an annual or final grant report. By the time the reporting cycle occurs, so much has transpired that project details may be forgotten and take time to find. For evaluators, the implementation log establishes timeline documentation and provides evidence of project-related activities ranging from acquisition actions and internal meetings to public events. The log is a primary data source for tracking project implementation and assessing fidelity to the project plan. The time required to maintain an implementation log can be as little as 15-30 minutes a week – a small investment that yields powerful evidence of how federal funds were spent.

Planning for Sustainability

Grant proposals often ask applicants to address sustainability. Sustainability is defined as the ability to maintain programming and its benefits over time. Projects that demonstrate sustainability are valuable to funders, because the funding provided during the grant period results in long-term changes, programming, or benefits.

Whether you are planning to submit a grant application or you already received funding, planning for sustainability is a valuable activity. The Program Sustainability Assessment Tool, developed at the Center for Public Health Systems Science at Washington University in St. Louis, identifies seven key domains of sustainability. These include:

• Environmental support: A supportive internal/external climate for the program contributes to sustainability. Examples of environmental support may include a school district fully adopting an instructional strategy that was implemented from a grant across schools.

• Funding stability: Grants provide funding for projects/activities to be implemented. Funding stability is when there are consistent, non-grant funds to support a program. Examples of funding stability include a school district committing to sustain new devices that were purchased by replacing devices as they wear out or a school district adding funds for a position to their annual budget at the end of a grant.

• Partnerships: Programs are more sustainable when they have developed partnerships with stakeholders. Partnerships can help to broaden possible resources and provide/sustain services. An example of partnerships would be partnering with the local military base and other organizations to offer an annual career fair.

• Organizational capacity: Throughout a grant term, organizational capacity may be built. Organizational capacity can encompass many areas. One example would be transitioning from a hired professional learning contractor to an internal staff member in the district.

• Evaluation capacity: Some grant programs require an external evaluator. When an organization builds evaluation capacity, the organization has learned to assess the program, interpret the data, and make data-based decisions on its own.

• Program adaptation: Over time, the most effective components of a program should be sustained, and a program may need to adapt to allow for this change. For example, a program may involve implementing several instructional approaches, but only some are deemed effective. The program may then use the instructional approach that is working and adapt it to be used at a higher level.

• Communications: Regular, strategic communication with stakeholders and the public about a program can help gain visibility and garner support. Examples may include developing websites for programs, starting a newsletter, or having regular meetings with community partners.

Project directors can review the domains of sustainability and have conversations with their team about actions that contribute to sustainability in each domain. It is important to note that a project may not demonstrate sustainability in all domains.

The Program Sustainability Assessment Tool (https://sustaintool.org/psat/) is an excellent resource for those interested in planning for sustainability. The Shaffer Evaluation Group uses this tool as part of our approach to evaluating federal grants. To learn more about our evaluation services, please visit our website.

Keeping Students on Track to College Graduation

You may have encountered recently the inspiring story of Hope Chicago, an innovative college scholarship program committed to investing $1 billion in scholarships to Chicago's students and their parents over the next decade. The surprise announcements this past February in Chicago’s high schools made national news as Hope Chicago awarded 4,000 secondary students and their parents the opportunity for a debt-free college education, as well as the possibility of improved financial stability for themselves and their families for generations to come.

The cost of college is daunting for most American families. The National Center for Education Statistics reported the average tuition and fees at public 4-year institutions in 2019-20 was $9,400. Coupled with the costs of books, supplies, and room and board, the total cost of attending a public university is on average just over $25,000 per year. Given that the median earnings of workers aged 15 and over for the same time period was $41,535, it’s impossible to imagine how the average American family can pay for college. Not surprisingly, the average debt among bachelor’s degree completers is almost $30,000.

Programs like Hope Chicago remove the financial barrier of entering college. But do scholarship programs alone enable students to succeed in college and graduate? One of the few studies that considers the relationship between scholarship and graduation, by Page, Kehoe, Castleman, and Sahadewo (2017), investigated the effectiveness of the Dell Scholars Program. The research team found that there was a higher probability that students receiving scholarships would graduate (from 6 percent to 13 percent). Similarly, an evaluation of the impact of the Florida Student Access Grant (FSAG) on long-term outcomes including college persistence and degree completion rates also found effects: financial aid positively affected student persistence and credit accumulation while increasing the likelihood of students earning a bachelor’s degree (Castleman & Long, 2016).

While financial support is important, keeping students on the path to a college degree is ideally supported by multifaceted support programs, which may combine a proactive, holistic coaching or advising model with financial support, enrollment messaging, and other supports (Ratledge & Wavelet, 2021). Holistic coaching models, such as the evidence-backed

InsideTrack model (What Works Clearinghouse, 2012), are used to assess students’ lives inside and outside of school and help them overcome barriers to academic success. Coaches and mentors proactively and frequently reach out to students, maintaining engagement across the student’s time at college. Universities have also invested in sophisticated CRM (Customer Relationship Management) software to track and engage students from pre-enrollment through graduation, often employ innovative engagement strategies such as digital messaging. Financial support is essential, but it is one of several tools available to support college student success.

Shaffer Evaluation Group works with higher education clients to evaluate the effectiveness of interventions to support students on their pathways through college. We have supported community colleges and universities receiving National Science Foundation and U.S. Department of Education grants to implement student support programs, ranging from CRM to holistic coaching and scholarships. To learn more about how we can help your institution evaluate student support programs, please contact us at seg@shafferevaluation.com.