Evaluating Literacy Interventions

We are evaluating several educational initiatives led by school districts that seek to improve reading

achievement by refining their approach to literacy intervention, specifically targeting students with disabilities. One approach used by two of our district partners is the Leveled Literacy Intervention (LLI) deployed specifically by special education teachers. Districts invest in ongoing teacher professional learning and resources to bolster special education teacher’s access to high quality reading resources and confidence to deliver targeted and intensive reading interventions.

During initial phases of implementation, ensuring the intervention is delivered with fidelity is a priority for project teams. In our role as evaluators, we collaborated closely with school district staff to develop tools, such as classroom observation protocols, and often train district staff to utilize those tools. Training a team of observers builds inter-rater reliability for consistent identification of strategies observed across classrooms. This type of data is useful for identifying program elements that are being implemented successfully, and those that are not. Analysis of fidelity data allows project teams to target continued support and professional learning to close the gaps. Formative data collection, such as professional development feedback, annual staff surveys, and focus groups, provides additional insight on classroom implementation successes and challenges and overall efficacy with the intervention. Triangulation of the data sources enhances data validity and reliability, ensuring our district partners are able to make informed decisions.

Improving mathematics instruction through evidence

We are evaluating several educational initiatives led by school districts that seek to improve student achievement in mathematics by refining their approach to mathematics instruction.

One approach commonly used by our district partners is the Concrete-Representational-Abstract (CRA) model for teaching mathematics, a research-based, three-part, systematic approach designed to build student understanding. At the concrete step, the concept is introduced using manipulatives (base ten blocks, fraction bars, counters) to solve a problem. During the representational stage, students use drawings (pictures, tallies, dots, graph paper) to represent the concrete objects to solve the problem. The abstract step problem solving moves to using symbols, operations, and formulas to solve a problem. Implementing this systematic approach to teaching mathematics has been shown to build student understanding of mathematical concepts.

We frequently support implementation by gathering data on fidelity of implementation through the analysis of teacher lesson plans using a rubric and/or classroom walk-through data with an observation protocol. In both cases, the data collection instruments are tightly aligned with the instructional model. Monitoring the observed frequency of specific instructional indicators provides formative feedback to the project team, which is used to plan for continued professional learning and/or support specific areas of need and monitor implementation progress over time.

Strategic Planning in Charles City, Virginia

When Charles City Public Schools in Virginia was looking for a consultant to assist with strategic planning, they contacted the Shaffer Evaluation Group. We're working closely with school district officials in the development of outcome-focused performance measures that will enable them to assess progress toward accomplishing their strategic goals and objectives. We look forward to continuing to work with them on collecting and/or analyzing performance data across the strategic plan period, and reporting results in a manner that is highly visible and easily accessible to school district staff, students, parents, and the Charles City community.

Strategic planning is more important than ever for school districts who are working to succeed in the face of multiple internal and external challenges. A well-crafted strategic plan clearly defines what an organization intends to achieve. Bryson (2011) defines strategic planning as a “deliberate, disciplined approach to producing fundamental decisions and actions that shape and guide what an organization is, what it goes, and why.” Strategic planning is a process that requires stakeholders to engage with one another, consider where they are, where they want to be, and how to get there (Bryson & Alston, 2004). Evaluators are often asked to support school districts and organizations with their strategic planning process due to their expertise in data collection and performance measurement. Evaluator support may include: data collection from stakeholders and associated analyses during the planning stage; development of SMART performance measures used monitor progress of selected strategies or objectives; and creating data collection guides used by school district staff during implementation. A well-crafted strategic plan should effectively link the organization’s stakeholders to the overarching mission and vision of the school district with clear and measurable performance outcomes of the strategic plan.