The term “vanity metrics” was popularized by Eric Ries in the context of startups to illustrate how founders measure the wrong things to fool themselves into thinking they’re winning, when that couldn’t be further from the truth. Alas, this isn’t limited to startups - it seems to be a human behavior that spans all areas of work and life.
Many people game the way they measure success to maintain appearances and feel better about themselves.
It tends to backfire, especially in the long run, and the most dangerous flavors are a lack of awareness and lying to yourself. You can end up walking towards a fake north star that leads you nowhere.
A few obvious examples, definitely not a comprehensive list:
Startup founders obsessing over signups and clicks instead of metrics like monthly recurring revenue (MRR) and customer retention rate (CRR).
Managers taking pride in the number of people reporting to them.
Product people boasting about how many new features they squeezed in this month.
Those metrics can tell us something about how skilled somebody is at certain things that don’t directly move the needle. They can also be proxies for meaningful outcomes, contingent on other, more important conditions being true. But on their own, they don’t tell us whether we’re actually winning.
For instance - the number of employees might mean something positive if revenue per employee is high and employees are happy and stay with the company for a long time. But it could mean a disaster if revenue per employee is alarmingly low and employees keep churning.
The scary thing is that we can do this without even being aware of it. Especially at the beginning of our journey, we need to deliberately find ways to become aware of the right things to measure and train ourselves to actually measure them. To optimize for signal over noise - what actually drives meaningful outcomes that move the needle.
A New Vanity Metric - AI Agent Count
In software development, there’s a new vanity metric - the number of AI agents running in parallel. The more agents you can spin up at once, the more of a genius you are. Or so the story goes.
As with other vanity metrics, this number alone doesn’t tell us whether we’ve achieved meaningful outcomes, and it’s pretty easy to cheat with it to make yourself sound more impressive than you actually are.
If someone uses 1,000 agents in parallel to write a new feature end-to-end in a few hours, are they a genius? Managing that many agents at once must require extraordinary amounts of intellectual effort.
Naturally, that statement in itself doesn’t tell us much - it’s mostly noise.
To someone who hasn’t used AI agents to code extensively, or at all, that number might suggest they’re managing a different context and set of hard problems for each agent. And developers might not be in a hurry to correct that impression.
To be fair, if you actually are juggling 1,000 different contexts in your head, you probably are a genius. Of course, that’s rarely the case. It’s pretty trivial to launch many agents that all relate to the same problem. The number of agents doesn’t equal the number of distinct contexts and problems managed in your brain.
To illustrate this point - with 1 quick prompt, one could fan out agents to generate 1,000 blog posts in parallel, with each agent’s subagents doing research, crafting the post, and checking its quality. And voilà, they’re using 1,000s of agents.
In practice, it doesn’t mean all of those run concurrently. There are often concurrency limits, so some agents end up queued. But you get the point.
The number of agents running in parallel joins the good old metrics like lines of code, commits, and PRs. In the right environment, they can mean something. On their own, not really.
The actual signal usually (always?) goes back to how something affected the business - either by measuring ROI directly or by measuring things that indirectly translate into sustainably running a good software business (e.g. code maintainability).
Unsupervised Agents
There’s this new hype around developers and tech influencers claiming they’re using 100s or 1,000s of agents to push PRs fully unsupervised. To be fair, those claims are often about producing correct code (as in, it compiles and works), rather than good code (maintainable, scalable, performant…).
Out-of-the-box configurations like Claude Code with Fable on Max & Ultra + gstack seem powerful, yet it seems one still has to keep their agents on track and help them iterate on problems. Or worse, solve the problem themselves and leave the agents to do only the dirty work.
Some people go deep into optimizing their own agents - e.g. by writing a very detailed set of skills. There’s definitely value in that, and you get full control over personalization. The better the agents are tailored to your preferences, the more you can leave them running unsupervised around your codebase, at least in theory.
That said, it seems like you mostly end up with personal flavors of gstack and pstack. Which is fine, but I’m not sure it’s worth the effort, given how quickly the best ideas tend to make their way into out-of-the-box solutions.
There’s probably a good 80/20 here: use existing tools for the general workflow, and focus your customization on your unique development preferences and the needs of each project. Some personal preferences will simply never be common enough to make it into off-the-shelf solutions.
I think there can be a healthy balance between code quality and development speed - more code, faster, with a tolerable percentage of AI slop. But above a certain percentage of slop and below a certain quality threshold, the maintainability tradeoff doesn’t seem worth it, though that’s likely subjective.
For now, the quality of the AI slop is still highly inconsistent, and I personally remain uncomfortable trusting AI agents to roam around unsupervised, pushing PRs for me.
I imagine that over time we’ll be able to let swarms of AI agents do more and more unsupervised - at least, that seems to be the current trend. But it doesn’t feel like they’re good enough to be left on their own just yet.