I didn't design the scores. I want to say that plainly up front, because "scoring engine" makes people assume the interesting part is the scoring: what a number means, where a threshold sits, which biological signal counts. That part came from scientists. I designed the system that produced the scores. It takes a sample, runs it through a stack of models and domain logic, and returns a set of personalized health scores a few thousand times a week. It can also still tell you, months later, exactly how it arrived at any single one of them.
That's a different job. It's the one I did for almost four years, and it's the one worth writing about.
Where it started
When I picked it up, "the system" was a research prototype. A toolkit, really. Code that worked because the person running it knew which knobs to turn. It did its job, but it was a black box in the way research code usually is: no metadata tracking, no run orchestration, no config you could read from a database, no way to trace a past result except Jira tickets and whoever remembered. It deployed a couple of times a year. Reasoning about it meant asking someone.
The scores were good. The machinery around them was held together by institutional memory. My job was to turn that into something an organization could actually operate, and to do it without breaking the science, which was still moving.
I was the implementer from day one. That matters to how the rest of this went, so I'll be precise about it: this wasn't a system I inherited and slowly earned the right to change. I was building it. What I didn't have at the start was scope. The people I had to convince were a scientific team I was embedded in, and the whole first stretch of the work was a negotiation with them about what a usable, traceable interface between science and engineering should even look like.
That negotiation is the origin of a contract the engine still enforces today.
Negotiating the interface
The core tension in a system like this never really goes away. Research wants to move fast, change shape, and be easy to experiment with. Production wants things versioned, bounded, and boring at runtime. You don't resolve that by picking a side. You resolve it by drawing a line between the two worlds and making every crossing of that line explicit.
Most of Act One was figuring out where that line went. Which normalizations would actually hold up once real, noisy data flowed through them. Which transformations belonged to the science and which were mine to own. This is where I learned that you can't design this kind of interface from the engineering side alone. You have to sit with the people who understand what the numbers mean and argue it out together.
The clearest example I have of why is a bug I didn't have to go looking for. An output distribution looked wrong to me. Not broken, just off in a way that didn't sit right. I traced it back through the pipeline to a normalization step that was quietly distorting the results. Nobody had flagged it. The scores still came out. You only catch that if you've built enough of a feel for what the biology is supposed to do that a distribution can look wrong before anything actually alarms.
I bring that up not to claim I understand microbiology. I don't, not the way the scientists do. I bring it up because that instinct is what let me be a real counterparty in the negotiation instead of an order-taker. You can't design the contract between two domains if you only speak one of them.
When ownership caught up to reality
At some point a restructure moved me onto full ownership and maintainership of the engine. On paper that's the promotion beat. In practice it mostly formalized something that was already true. The system had been running in production and holding up, so making it the official production schema recognized reality more than it granted new authority. The work didn't change the day the title did. It had already become mine to answer for.
What changed was what I could now go fix.
Cutting the last coupling
The dev/prod split was still incomplete, and it bothered me. Science and engine were handing off through a pickle file — the production system depended directly on the research codebase, which meant the two could bleed into each other in exactly the ways you don't want when one side is trying to iterate and the other is trying to be boring. A change on the research side could ripple into production without anyone deciding it should.
Owning the production side gave me the standing to force that fix. I couldn't have pushed it as a peer inside the science team — it wasn't only their problem to solve. It required negotiating directly with the engineering leadership that owned the development pipeline, and this time the room was bigger. The counterparty was now two organizations with different incentives, each of which had to sign off on the same handshake.
We replaced the pickle handoff with a versioned JSON schema contract. The science package generates JSON; the engine ingests it. Two independent codebases that can't reach into each other's internals. That was the rewrite that made the second version of the system what I actually wanted it to be: a cleaner handoff and, downstream of it, deployments that are easier to trust.
Trust is the real product here, and it isn't a feature you bolt on at the end. It has to be in the shape of the system. A few of the things that shape cashes out to:
- A bad version can't silently corrupt data. The engine verifies model config checksums on startup. Any drift and it short-circuits, refusing to start and alerting the team, rather than running against something that doesn't match what was approved.
- Every result carries its lineage. A score points to the derivation that produced it, which points to the exact config that was deployed. "Logical release" (the version that was approved) and "canonical release" (the code that actually ran) are tracked separately, because those are different questions and someday you'll need to ask both.
- Nothing gets overwritten. Scores live in an append-only ledger. A re-purchase triggers a fresh recompute; the old result stays. What the engine computes and what a customer is shown are deliberately kept as separate concerns, and the business layer decides the second one.
- The output is diffed before it ships. Before a deploy, a batch of samples runs through every transform and post-processing step, and the results are compared against expected output down to ten significant digits, with replicates to measure real biological and technical variation. A release that moves a number it shouldn't doesn't go out.
None of that is exotic. It's just the discipline of treating each version as an immutable artifact and making provenance part of the result instead of something you reconstruct later under pressure. The old system deployed twice a year and you took its word for it. The new one deploys every two weeks, shipping real changes each time, and you don't have to.
What's still unsolved
The honest part, because a retrospective with no unresolved tension is marketing.
Deployment still needs a human. There's a manual trigger and a manual metadata double-check in the path, the most error-prone step left and the one I'd most like to design out. It's the seam where the automation currently stops and attention has to take over, which is exactly the kind of seam this whole project was about removing everywhere else.
And there's no true multi-version serving yet. The system is built so any historical version could theoretically be spun back up. Re-scoring is designed for, even though we've never actually needed it in practice. What isn't there is running multiple versions of the same model side by side in production, a real canary. That's the next shape the platform wants to grow into.
What actually changed
The tidy version of this arc is implementer-to-owner: started taking orders, ended up in charge. It didn't go that way. I was implementing and negotiating the interface from the first week, and the title just caught up to work I was already doing.
What changed was the size of the room. Early on I was arguing about normalizations with a scientific team I sat inside. By the end I was negotiating a schema contract with engineering and product leadership across the org. Same judgment, same instinct for where a distribution looks wrong or a coupling will hurt later, pointed at a wider and wider set of people who had to agree.
The job never changed — only the size of the room I was negotiating in.