The Impact of Evidence of Effectiveness on Local Schools

This article is Part 2 of our ESSA: Evidence of Effectiveness series. Make sure to check out Part 1: Understanding the Difference between Research-based and Evidence-based.

As we shift our focus from research-based to evidence-based, we raise the bar on the level of rigorous support schools need to adopt, support, and scale instructional programming across the classrooms and schools in their district or state.  First came approval of state ESSA plans, and now the next phase of identifying and implementing Evidence of Effectiveness is underway.

But what does that mean? What roles do the State Education Agency (SEA) and the Local Education Agency (LEA) play in establishing and demonstrating evidence? What’s expected of each? And, more importantly, what is at stake in terms of accountability?

Like always, let’s start at the beginning.

The Impact of Evidence of Effectiveness on Schools

If you were an administrator and you were eye-level deep in the bylaws of NCLB, back when it was the law of the land, you probably remember the numerous times you saw the words “scientifically-based research.” In fact, this phrase shows up 110 times in the NCLB law!

In a nutshell, NCLB was narrower in focus, was less rigorous, and indicated that a program had merit if it was “based on scientifically-based research,” which left far too much open for interpretation. NCLB tried to implement a data-driven system, but it fell drastically short.  Because of this, it became exceptionally difficult for schools to define the evidence needed.

In contrast, ESSA seeks data-driven accountability too, but through the use of clearly defined Evidence of Effectiveness criteria. ESSA seeks to hold programs to higher standards by clearly outlining these four-levels of effectiveness. It calls for programs to incorporate systemic evaluation, demonstrating considerable higher levels of program effectiveness, than we have in the past.

ESSA encourages the use of strategies for evaluating Evidence of Effectiveness for programs schools use. Ultimately, ESSA effectively raises the bar on the level of rigorous support schools need to adopt, support, and scale instructional programming across the classrooms and schools in their district and state.

ESSA incentivizes states to work more closely with their schools and districts to use evidence-based programs. The goal is to attain stronger positive student outcomes and reduce spending on programs that lack research and Evidence of Effectiveness.

ESSA: Evidence of Effectiveness Standards

ESSA defines strong, moderate, and promising Evidence of Effectiveness. In addition, a fourth category defines an opportunity for program implementations that use approaches that demonstrate a research-based rationale for effectiveness.  The four categories are defined as follows (in brief):

  1. Tier 1 = Strong: At least one randomized, well-conducted study showing significant positive student outcomes.
  2. Tier 2 = Moderate: At least one quasi-experimental (i.e., matched), well-conducted study showing significant positive student outcomes.
  3. Tier 3 = Promising: At least one correlational, well-conducted study with controls for inputs showing significant positive student outcomes.
  4. Tier 4 = Demonstrates a Rationale: Practices that have a well-defined logic model or theory of action, are supported by research and have some effort underway by an SEA, LEA, or outside research organization to determine their effectiveness.

Based on ESSA’s Evidence of Effectiveness, it is now clear that research-based programs fall under Tier 4 in demonstrating effectiveness.

What Do Evidence Requirements Mean for SEAs, LEAs, and Administrators?

First, and foremost, it’s important to keep in mind that at every level of accountability – school, district, and state – the level of ‘Evidence’ being sought is that of individual student growth. Additionally, keep in mind that all references to evidence eventually filter to effectiveness of use in the classroom.

When ESSA shifted the power from federal to state levels, it essentially allowed states to create their own ESSA accountability plans based on the law. The same is true for how the Department of Education provided guidelines on evidence-based interventions and programs and allowed each State Education Agency (SEA) to interpret the guidance for their specific requirements around rigor and study design. It's also worth noting that states are yet to announce the full role that evidence will have in their funding decisions.

Once the SEA has determined their guidance around evidence-based interventions, it is up to the Local Education Agency (or district) and the schools for executing and establishing the evidence of impact.

Under ESSA, district and school leaders are responsible for selecting effective, sustainable, scalable instructional approaches that will meet the unique needs of their students. Local evidence supplemented by evidence of comparable schools is imperative to achieving positive student outcomes.

Think of it as an inverted triangle.

The broad upper-level, the Department of Education (DOE), is responsible for making sure that states adhere to their ESSA plans. State Education Agencies make sure they maintain adherence to the plans they laid out and sent to the DOE in 2017 and hold Local Education Agencies (LEAs) accountable. These plans submitted to the DOE also included a definition of guidance to districts regarding levels of effectiveness. Schools then report their data (including that of their Evidence of Effectiveness) to the LEA. School administrators are, therefore, responsible for the data and evidence for their classrooms – and ultimately every student.

It Starts at the Classroom & School Level

Take a moment and really think about your school. Think about the classrooms and each of your students.  Think of the demographics, the structure, and your teachers and support staff. Can you think of all the ways that your school could be different from another school? What kind of school would be similar to yours? How are the groups of students similar and different?

Schools vary in so many different ways.  Therefore, seeking evidence that a program can meet needs based on similar demographics, structure, etc., is a step closer to improving outcomes for students. Once the needs have been identified, SEAs, LEAs, schools, and other stakeholders will determine the interventions that best serve their needs, implementing rigorous and relevant Evidence of Effectiveness at the local level.

Implementation planning begins with classroom educators and other school stakeholders understanding their local needs and establishing plans to meet them. We use strategies and programs collectively to put an implementation plan in place and then gather the Evidence of Effectiveness in the form of data.

Strategies, Data, and Implementations

Do you ever feel like part of your job as an administrator is to be part psychometrician? Who would have thought that there could be so much data! At present, ESSA and SEA guidelines are clear that effectiveness results must be determined by a third party that is neutral to any vendors or suppliers involved in an implementation. This means that it is up to the schools to implement and evaluate research efforts towards determining intervention or program effectiveness

Rigorous data must be gathered to identify the strategies and tools that best meet the school’s objectives. After all, how you will be able to decide whether or not a strategy or program is effective if you don’t have data to prove it? The rigor of the data collected determines the eligibility for sustaining and scaling effective solutions. What this means is, the more effective a program is proven to be in your school (Tier 1–3), the more merit it has to be upheld within the school and with stakeholders, as well as implemented throughout the district.

But data is only half the equation.

An effective planning process precedes and defines the data collection process. Like I mentioned, planning starts at the classroom level. First, identify what the needs are for the school, then execute a plan for implementation.

Think of your plan and execution like a formula for success:

                [Implementations] = [Strategies] + [Tools/Programs]

Once you’ve put that process in place (implementation), you’ll be able to collect data and determine Evidence of Effectiveness.

What Are the Stakes of Accountability for Administrators?

Like I mentioned in the previous blog, the bar has been raised by ESSA. In addition to being the law, there are so many reasons why it is important to put a strong plan in place on how you will meet the different levels of Evidence of Effectiveness:

  • The time for Implementing programs that demonstrate strong, moderate, or promising evidence of effectiveness (Tiers 1 -3) are mandatory and is just around the corner (2019).
  • The most important funding that is impacted by these provisions is the 7% Title 1, Part A, funds set aside by each state to assist the lowest-performing schools. If schools do not demonstrate that the top three tiers of evidence support the interventions (programs) they are using, then these funds cannot be made accessible to the school.
  • Programs that you’ve proven to be of the highest tier of effectiveness (Tier 1: Strong Evidence) will ultimately demonstrate the most significant, positive outcomes on your students, and ultimately be considered effective, both as cost-effective and achievement-effective programs.
  • Some ESSA programs – including some competitive grants and Title I, Section 1003 – require the use of “evidence-based” programs that meet higher levels of evidence. It is recommended that you look through the “Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments” to understand best practices and recommendations that your school needs to adhere to for ESSA accountability in using these funds.
  • Implementing programs that have been proven to be of high Evidence of Effectiveness are more defensible and hold more weight with stakeholders. Additionally, these are more scalable and sustainable within the district.

What Are the Next Steps?

First, make sure to read our previous blog about the difference between Research-based and Evidence-based research. Second, I highly recommend exploring your state’s ESSA plans and how your SEA further established outlines for how evidence would apply to local districts and the schools in their state accountability plans.

Join us for our How to Prepare for the New Evidence of Effectiveness Requirements webinar on May 2, 2018 at 1:00 PM EDT. Register today for this 30-minute webinar that explores key questions to the federal and state level requirements around Evidence and how they will affect schools in the 2018/19 school year

Get in Touch!

Odysseyware is committed to partnering with educators in helping every student succeed. District and school leaders are not alone in the battle for attaining evidence. Odysseyware is your partner in establishing compelling local evidence. Engage with one of Odysseyware’s experts today on how we connect educators and strategies all working toward a common goal – student success. We can help demystify the transition to ESSA’s new Evidence of Effectiveness requirements.




Apr 25, 2018 By thyatt