Editor’s note: This is the fifth in a six-part series from The Political Center analyzing federal rule changes to Medicaid under H.R. 1, and how Oregon lawmakers can help navigate the shifting regulatory landscape. Read the previous installments below:

• Part I — Implementation Deserves Its Own Conversation
• Part II — Implementation Begins Before the First Notice
• Part III — Where Implementation Becomes Real
• Part IV — No One Implements Alone

---

As Oregon begins implementing federal Medicaid community engagement requirements, the question naturally becomes how leaders will know whether the process is working.

There will be no shortage of things that can be measured, including how many notices were sent, how many exemptions were requested and approved, how long those determinations took, and how many people demonstrated compliance versus how many lost coverage. Those numbers all matter. But none by itself tells us whether implementation succeeded.

Before we can measure success, we have to define it. That may sound obvious, but when it comes to gauging complex systems we naturally gravitate toward measuring what’s easiest to count, but that isn’t always what we most need to know.

What the Numbers Can’t Tell Us

Let’s say Oregon reports 50,000 notices were successfully mailed. How will we know whether members understood what they were reading?

Or imagine we know exactly how many people lost coverage. What does that number really tell us? Some may no longer have qualified for coverage under the law. Others may have remained eligible but lost coverage because they couldn’t get around some administrative roadblock. Those are very different outcomes.

A dashboard may count them, but it cannot necessarily explain them. Data can tell us what happened, but only experience and context can help us understand why it matters.

 

If we want to understand whether implementation is actually working, we need to ask harder questions:

• Did Oregon fully implement what the law requires?
• Could members who met the requirements successfully demonstrate it?
• Did people who qualified for exemptions receive them?
• Were communications understandable and consistent?
• Were determinations timely and accurate?
• Where did administrative friction create an outcome the policy itself did not require?
• What are people throughout the system experiencing that the numbers alone cannot show us?

Measure What Matters

Thoughtful measurement connects outcomes with experiences that help explain them.

We don’t have to agree on the underlying policy to agree that implementation should tell us something useful. If a requirement works as intended, we should know that. If eligible people lose coverage because of avoidable administrative barriers, we should know that too. If something that looked sensible on paper creates an unintended consequence in practice, we should understand why.

The purpose of measurement should be to understand implementation well enough to make the next decision better. 

Now, suppose we define success thoughtfully and measure what matters. Suppose the experience of members, providers, CCOs, community organizations, administrators, and policymakers teach us something we didn’t know when implementation began.

Then what? What are we willing to do with what we learn?