Measurement

Measurement-driven decision-making is also hard

People disagree about data. There can be strong disagreement about what to measure, and equally strong disagreement about how to measure it. Capturing data is also not costless: it consumes dollars, time, talent, and political capital, and for some program models, taking credit for outcomes is genuinely difficult.

Data also has consequences. Data is rational; systems are political. Measuring and reporting data wins friends and makes enemies. It is an act of accountability, which makes it high-stakes.

Data is also a source of power. As Laura Esquivel, the Mexican author and political leader, once observed, “Whoever controls information, whoever controls meaning, acquires power.”

In social initiatives, we measure what we value, so measurement reflects our values. Yet we seldom start by asking all stakeholders, and communities in particular, “What would you value?” and then build measurement out from that stakeholder-centric foundation. Every community, and even every individual, has its own definition of success and of how success is measured. In education reform, when we don’t understand a community’s definition of success and how it is measured, we run a real risk of imposing our own definition and our own measures instead. This risk is not abstract. Systemically marginalized communities have long histories of being denied measurement and data, which denies them power. Those same communities have also experienced data being weaponized against them.

Last, measures are ultimately not numbers but people. At the core of measurement are individuals who want the opportunity to realize their best destinies, and their families and communities who hope that that opportunity is available but fear it will be denied.

Being measurement-driven requires organizations to be strong in all four parts of this cycle:

Many cycles and tools exist for using measures. I think this one is superior to many others because it is explicitly about making decisions. Any data collected should be examined through the lens of how it contributes to a decision, and if it doesn’t, it is probably a distraction. This data cycle originated in the Korean War. A U.S. colonel named John Boyd was tasked with determining why allied planes outperformed communist planes in dogfights. He discovered that opponent planes could accelerate faster, climb more steeply, and turn more sharply than allied planes – which sounded like potent advantages. However, allied planes were superior in two ways. First, pilots literally sat higher in the cockpit, so they had a better field of vision of the battle. Second, they had better controls that allowed them to translate insights into decisions and actions faster. This became known as the Boyd Cycle, or the OODA Loop, for Observe, Orient, Decide, and Act. Sometime in 2006, an education leader in North Carolina taught me the What, So What, Now What, and What Happens.

B. Measures to Measure

There are three critical types of measures (or data) that organizations need to be able to access and use together in this cycle:

The value of putting these three types of data together

• Internal planning and management: Together, these measures enable an organization to: understand what is creating value (and for whom); understand the true resource requirements to achieve that value; identify where there are opportunities to test program variations; and see where options exist to make trade-offs between growth in scale, success, and sustainability.
• External engagement and growth: Together, these measures enable an organization to right-size fundraising and pricing; articulate expectations to partners in proposals and negotiations; communicate its results and what is driving those results to beneficiaries, funders, and other stakeholders; and raise awareness, shift mindsets, and build support about what works and should be supported through policy and funding.

Defining four more measures to employ

Organizations benefit from going one click deeper down the measurement rabbit hole by agreeing on the definitions of another set of measures (some of which have sub-measures). These align roughly with the graphic above. Inputs are financial performance. Outputs are operational performance. Outcomes are beneficiary performance. Program Design sits at the intersection of financial and operational performance – it is the planned deployment of financial resources to operate programs, and is what actual financial and operational performance is then compared with (I decided to spare you yet another graphic depicting this).

When considering Systemic Impact, there is a bonus fifth measure – Infrastructure of Community/Parent Power. For more information on this, please see the parent power page.

What about Impact?

How does all this measurement complexity fit with an organization’s Intended Impact?

Intended Impact is the articulation of who an organization wants to serve, where they are, what it wants to accomplish for them, when (e.g., over what timeframe), and at what cost. Usually Intended Impact is the “top of the pyramid” of beneficiary outputs and outcomes an organization wants to achieve by the end of its strategic plan. But there are many earlier measures an organization will want to manage to because the performance against these measures determines an organization’s ability to achieve its Intended Impact.

For example, while a relatively simple program like ACT preparation (see below) is ultimately focused on test outcomes, there are many measures to manage to in order to achieve an improved ACT score. Even then, there are multiple ways to measure (and set targets) around ACT achievement.

In the second example below of college advising, ACT performance is just one of many outcomes in a more complicated program model. While this example’s ultimate outcome (i.e., its Intended Impact) is the % of 8th grade alumni who graduate from college, there are many measures of interim performance (outputs and outcomes) that determine how successful an organization and the students it serves will be in achieving that ultimate outcome.

Putting measurement into practice: ACT preparation

To see examples of measurement in action, please view this content on a computer.

Putting measurement into practice: College access and success programming

To see examples of measurement in action, please view this content on a computer.

Becoming more measurement-driven takes effort and can be a change-management journey for many organizations

• Very few organizations can simply take another organization’s common language and adopt it. Rather, it usually needs to grow from existing “tribal speak” and a common journey of co-creation.
• Very few organizations can simply adopt another organization’s measures and targets:
  • Measures can vary significantly based on the educational / nonprofit model of programming.
  • Measures may also vary based on population served.
  • Setting targets usually requires first having a baseline, especially for new measures.
• Details really matter. For example: are all people/teams/sites calculating a common measure with a common formula and over a consistent timeframe?
• Humans are humans.
  • People of good conscience can disagree.
  • Different people will care about different measures (based on values, responsibility).
  • People will tend to want to start very simple and then embrace complexity as they grow comfortable.
  • The more people are involved in the journey (in the right role), the more invested they are in the destination.
• Many factors will constrain the speed at which an organization can adopt a new approach to being data-driven. Agreement on what data to collect and how to collect it can take time; selecting and implementing new data/reporting systems can take time; building new staff skills (including accountability systems) to collect data takes time.

C. Setting Targets

Target-setting is both an art and a science

A target (also sometimes called a goal) is NOT the same as a measure of input, program design, output, or outcome (or in the case of Systemic Impact strategies, Infrastructure).

A target is the specific performance an organization wants to achieve for any of these types of measures in a specific period of time.

Not all measures have targets, especially at first. An organization may track some important measures to inform its work but may choose not to set a target for them. Particularly in the case of a new effort (and new measures), organizations will sometimes track initial performance without setting a target to establish a baseline against which to set targets in the future.

Setting targets is about setting expectations and creating a social contract within the organization and externally with partners and funders.

Setting targets can create emotional safety. Targets tell stakeholders what success looks like for a specific period of time, and what they should be managing to. Targets dispel ambiguity.

A target should be what you need to achieve or believe you can achieve, not what you aspire to achieve. Organizations sometimes feel pressure to immediately set very high targets (occasionally called BHAGs or “Big Hairy Audacious Goals”), which, while aspirational, are not actually achievable in the timeframe of a strategic plan. Setting modest but achievable targets isn’t necessarily glamorous to funders and allies. Organizations may feel that they must promise to fly before running or even walking and commit to targets that aren’t achievable.

Explicitly laying out a BHAG as (a) aspirational and (b) achievable over a long period of time can be motivating, and it can be articulated as the organization’s Ultimate Vision. An organization would then be well served to lay out a more specific target or set of targets that are meaningful stepping-stones toward that BHAG and achievable in a shorter timeframe over the course of a strategic plan, and for which the organization can be held responsible.

Additionally, target-setting is based on the resources an organization believes will be available (financial, talent, systems, political capital, and partnerships) to fuel its Theory of Action. Constraints on resources become constraints on ambition and therefore targets of performance.

Setting the wrong expectations on targets can have consequences. Imagine two charter schools get funding from the same foundation. Charter School A sets its target at 50% proficiency in math because that is what it believes it can get in Year 1 of the grant. Charter School B sets its target at 80% proficiency in math in Year 1 because that is what it ultimately aspires to achieve. At the end of Year 1, Charter School A hits 52% proficiency and the funder is happy. Charter School B actually achieves 63% proficiency, but the funder considers pulling its funding because 63% is well below 80%.

An example of setting targets: student retention for a charter school network

In this example, a charter school network I once worked with wanted to improve its student retention (the number of students who stay with a school year-over-year) from a network average of 86% to 92% within three years. It had almost two dozen schools with significant variations in performance. One school, which we will call Fly Trap, had 97% year-over-year retention. In contrast, another school, which we will call Turnaround, had only 70% year-over-year retention.

We set the targets below for each school on the following timeline based on three rules we applied to the entire network.

• Rule 1: If Year 0 performance was >92%, the school’s target was 92%.
• Rule 2: If Year 0 performance was at or above 80%, the Year 1 target was: (92% minus Year 0 performance) divided by 3. The school had three years of the same year-over-year improvement targets to get to 92% by end of Year 3.
• Rule 3: If a school’s retention was below 80%, it had to get to at least 80% in Year 1 and then get to 92% by the end of Year 3.

When I sat down with Fly Trap, the school leader questioned why, at 97%, we were reducing her target to 92%. I explained that 97% is fantastic, but should we then set 98% or 99%? Is that really achievable? Her target was 92%; it didn’t mean her school couldn’t outperform the target (and thus contribute to the network’s overall weighted average retention). But we did not want to set expectations that penalized her team for being merely fantastic rather than perfect.

In contrast, for Turnaround, going from 70% to 92% in one year was not practical. But performance had to change. We set 80% in Year 1, which itself would be a substantial effort. Then the school had to continue to improve over the remaining two years to get to 92% by end of Year 3. This focused the entire staff of Turnaround on a specific target. They looked at other schools nationally and identified a series of strong practices to improve retention: (1) ensuring every student had a 1:1 relationship with a teacher; (2) ensuring every student participated in at least one extracurricular activity; (3) changing their disciplinary policies to focus on being restorative and minimizing out-of-school time; (4) building relationships with every parent or caregiver of a student; and (5) improving instruction (which may seem obvious but could easily be overlooked).

D. Dashboards

Organizations need dashboards to provide a clear visual picture of data that can then be the basis for conducting the “So What” of insight generation.

What makes a good dashboard or set of dashboards:

• Provides data that is important to driving insights and then decisions that will advance impact.
• Is shared at the right frequency to be actionable, balanced against when data becomes available.
• Is shared in a visually accessible way for data to be easily understood.
• Segments data as needed (for example, by program/business lines, beneficiary populations, geographies, partner types, etc.).
• Provides the right level of detail to the right audience (while also enabling stakeholders to double-click into more detail if desired).

To see examples of dashboards, please view this content on a computer.

Different colors were used to clearly differentiate between historic performance, target performance, and final-year actual performance. While some measures had targets, others did not. For example, this charter network cared about how many students it served who were eligible for Free and Reduced-Price Lunch, classified as Limited English Proficient, or in need of special education services. All three were core to the charter’s mission, but there were no specific targets it was managing to. This dashboard existed for the network but could immediately be converted to show each school. Before long, school leaders knew their manager (and the superintendent) would visit their school having looked at data, and they were expected to have reviewed it, analyzed it, and come up with options or decisions on how to respond.

This was one of eight dashboards the charter network used:

1. Enrollment and attendance (above).
2. Retention of students year-over-year.
3. Formative assessments: quarterly pass rates.
4. State assessments, norm-referenced tests, and SAT/ACT.
5. High school graduation, college matriculation, persistence, and completion.
6. Financial performance.
7. Staff performance, satisfaction, retention, and open roles.
8. Fundraising.

E. Contribution and Attribution

Measuring is one thing. Claiming credit for improvement in what you are measuring is something else.

Organizations can more easily demonstrate contribution (I made a difference) or attribution (I made THE difference) when they have more control over their programming, and when their programming controls more variables.

• This is why it is generally easier to demonstrate contribution/attribution for Direct Impact models than for Widespread Impact models, because an organization running a program directly has more control (sometimes significantly more) than it does over programming run by partners.
• This is also why it is easier to demonstrate contribution/attribution for comprehensive models like a charter school, which can control most or all variables in a school (see the graphic in the Theory of Action section), than for a program like professional development (PD) or curriculum, which is just one of many variables in running a school. This doesn’t mean it is impossible to evaluate PD or curriculum for contribution/attribution; it’s just more difficult.

Systemic Impact can be particularly challenging for some organizations seeking to claim contribution/attribution.

• In policy change, as the saying goes, “victory has many parents, and failure is an orphan.” Many different organizations may be working to shift mindsets, relationships, and power in order to change policies, practices, and resource flows, which makes it hard for any single organization to claim credit.
• Even within a single organization’s Systemic Impact campaign, many different actions (e.g., canvassing, online communications, testifying, lobbying, boycotts) can be employed, and it can be challenging to identify which particular action was most effective, especially since these actions are usually meant to reinforce one another.
• A lot can happen between winning an electoral or issue campaign and seeing the resulting program implemented effectively and delivering its intended value. For more on the trials and tribulations of policy implementation, please click here.

When deciding how to demonstrate contribution/attribution for a measure of value, organizations have five options.

1. Common Sense. Nobody is going to defend their doctoral dissertation based on this, but I do think it is a valid approach to consider. For example, just because a program is funded doesn’t mean it will be successful. But if a program isn’t funded in the first place, it won’t exist, so a successful effort to unlock funding should get some credit for the program’s eventual success, if it comes.
2. Perception. Involved stakeholders (system leaders, school leaders, teachers, students, parents) believe that something an organization provided resulted in, or heavily contributed to, the creation of a measure of value. Sometimes perception is reality. If student outcomes improve, and those closest to those students (from teachers to the students themselves) believe it is because of a program, that should carry some weight. Perception does have its limits: people may perceive something that is not true (Learning Heroes does great work demonstrating the gap between parents’ perceptions of student success and reality). However, in that case, hard data showing a lack of performance is available to compare against perception (and dispel it).
3. Evaluations by other organizations. Have other organizations with similar program models conducted an evaluation that demonstrates value? If so, an organization can use that evaluation to make the case that faithful implementation of its model will achieve similar outcomes.
4. One-time (or episodic) evaluation of an organization’s own work to demonstrate value. An organization can use this evaluation to make the case that future faithful implementation of its own specific model will continue to achieve the same outcomes. This is stronger than relying on an evaluation from another organization.
5. Ongoing tracking of outcomes to demonstrate, net of other variables, that an organization is continually creating value. This is the most powerful option but, not surprisingly, can also be the costliest.