The Marginal Cost of Correctness

The marginal cost of correctness is going to zero. You need to internalize what this means for how you practice software engineering. You need to be obsessed with doing things correctly.

A Changing Landscape

In the before times, an engineer could differentiate themselves with raw technical talent: efficient algorithmic thinking, scalable distributed systems, manually grinding through debugger breakpoints. This is becoming less true at an accelerating rate. Agents are empowering anyone to write correct code.

The most important skill you need to develop to differentiate yourself now is a nose: the ability to smell things out that just aren't right and the willpower to make them so.

Sniffing Out A Slow API

We're training a new model at Pangram (don't think I'm leaking too much by acknowledging that) - it will be bigger, smarter, and also *faster *than our previous models. That last superlative would not have been possible without an obsession for doing things correctly just for the sake of it. Because we can. Because we take pride in our craft. Because one day I was testing our api and felt "it should definitely be faster than this".

At the time I had no way of knowing how, exactly, going down that rabbit hole would pay off. I was just armed with the conviction that we should do things right and the belief that with Fable by my side anything was possible.

We came out the other side just a short couple days later, victorious against the latency that had plagued my dogfooding. With Fable I was able to produce an entirely local testing harness for our complex distributed api orchestrator. We generated hypotheses for all the hot spots in our code and tested fixes for each. We pipelined complex Redis ops and validated behavior preservation. We ran simulations of different load patterns based on production traffic and estimated necessary infra.

This would have been an untenable, un-proposable project without agents - taking weeks or months of engineering hours. Instead, it was a side project while my training run finished in the background.

For our efforts, we were rewarded with a ~5x increase in gpu utilization - from 20% (ouch) to 98% and throughput on par with our bare metal nodes. This cost us very little. This earned us everything. It made hosting our new model go from cost prohibitive to a downright steal.

Developing Your Nose

Unfortunately there's no 1 Easy Trick to developing your nose, but it is a skill you can develop. Here's what's worked for me (I think):

Have a low pain tolerance

Anything that frustrates you likely frustrates your users too - don't just gut it out. You need to get in tune with yourself: notice what causes you subtle agitation during the day, the parts of the codebase you might subconsciously avoid, or tasks you get a little too fidgety during. This is where there are wins to be had. This is what kicked off my api optimization deep dive.

I like to say "easy things should be easy". This applies to your codebase (e.g, running a simple training ablation should require no mental effort) and to your product (e.g., a single api call should be barely slower than the forward pass itself). When you find yourself thinking "it shouldn't be this hard", fix it! You have agents now! Spin up a worktree and throw codex /goal at it - what do you have to lose?

Question everything from first principles

Agents make this so easy. "How long should this task take given our batch size, GPU specs, and parameter count". "How much should we be spending on this provider given our data + query volume". When you know what correct looks like, it's much easier to identify what's incorrect.

I do this for every model/node/request-shape topology we support - it's how I tune our vLLM setup to make sure we're near the theoretical limit. It's how I knew there was a gap between our api throughput and what we get on a bare metal node.

This principle applies to your codebase too. If some abstraction looks weird to you, don't just accept "that's how it's always been" as an answer. Rethink the contract from first principles - what should it look like knowing what we know now? Your job is to figure out where to point the agent. The agent can refactor and validate correctness overnight.

Be curious, read a lot

One of my favorite optimizations I shipped during my first week at Pangram was inspired by the DeepSeek V4 hybrid sparse attention mechanism.

My optimization had nothing to do with neural architecture, yet alone attention specifically. It was an improvement to the algorithm we use to detect boundaries between human and ai text, which used to require an unbounded second inference pass.

Hoover up as many interesting concepts from as many smart people as you can. Your job now is to be infinitely curious, to make connections, and to ask "what if". Remember, the marginal cost of finding out is going to zero.

Use your tokens

I have a ritual every sunday where I just set codex on a random /goal to try and use my weekly tokens before the reset. More often than not I end up throwing the code away or it languishes in a forgotten draft PR. Sometimes it pays off. I almost always learn something from the exercise. Either way it costs me nothing. Some people call this playing the slot machines, but I say you can't win if you don't play