The Interesting Changes Aren’t Happening in the IDE
What I'm still figuring out about AI and engineering teams
A recent mentoring session with earlier-career engineering leader alumni from my university revealed that the real challenge of AI adoption is not the tools themselves. Instead, the gap between metric-driven adoption and the quality of actual work is widening. Everyone is improvising; the most compelling issues have little to do with the tools.
That conversation kept landing on the same thing: the question we’re all being asked, “how to adopt AI,” is the wrong one. It assumes there’s a destination, and that reaching it means you’re done. The technology is moving faster than the systems that surround it. Faster than your hiring practices, your review processes, your testing infrastructure, your team shapes, and your career ladders. If you wait for a complete answer before you act, the answer will have changed by the time you act on it.
But getting it wrong has asymmetric costs. Over-rotate and you ship slop your customers will eventually feel. Under-rotate, and your team falls behind a market that is figuring this out without you. Which side is more dangerous depends on your business.
The Four Layers of AI Adoption
To help make sense of this complexity, I’ve started breaking the adoption question into four layers. As we move from one layer to the next, the challenges become harder and demand greater leadership. Most companies tend to fixate on layer one while overlooking the rest.
Layer one is tooling. Which tools, which models, which licenses, adoption metrics, and cost measurement? This is the layer that vendors are pitching, both the big players and the new entrants. It’s the visible layer, and it’s the easiest one, which is part of why everyone gravitates toward it.
Layer two is process. What does code review look like when one human and one agent approve, or two agents approve, or no humans approve at all? What does testing mean when pull request volume goes up several times over? What does a sprint look like when ticket sizes grow because agents handle more in a single shot? Every engineering team I know that’s made real progress on AI adoption starts hitting these questions quickly.
Layer three is structure. Team shapes change: smaller teams, different ratios of product to engineering, different ratios of QA to engineering. Designers and product managers start picking up work that used to live elsewhere because dev throughput has outpaced their capacity. Where does QA live when AI is generating test plans and then generating the tests? Career ladders built for human velocity start to creak under human-plus-agent velocity.
Finally, the fourth layer is judgment. When is the output good enough to ship? When is it slop that your customers will eventually feel? When does the work need a deterministic system underneath it, given the real cost of being wrong? This is the layer no one is hiring for, but everyone needs.
Most of the failure modes I’m seeing come from working at the wrong layer. If your code reviews are drowning, buying more tools won’t help. If your team shapes are wrong, fixing your review process won’t help. Energy spent at the wrong layer compounds, because the actual problem keeps growing while you’re busy somewhere else. The slowest part of your team will tell you which layer to work on, and it is rarely the one you wanted to work on.
Change management, not tooling rollout
Once you identify the right layer, the work becomes change management, not tooling rollout. We have decades of experience running change initiatives, and most of those tools apply. The mistake is treating AI adoption as only a tooling decision, not a transformation.
Every transformation runs through people. This insight has surprised me most. These recurring groups appear on every team I’ve observed:
The eager adopters have the most personal upside from looking ahead, or are just genuinely excited about new technology. They produce real value. They also produce slop, because excitement without discernment leads to shipping things that haven’t been fully thought through. That slop gets cleaned up by someone else, who now has a more negative view of the technology than they did when they started. Eager adopters should be rewarded, but the reward needs to be for quality, not for throughput.
The skeptics are usually senior engineers whose craft is being challenged. They’ve spent decades becoming exceptionally skilled at writing software, and now there’s a tool that writes it almost as well as they do, sometimes better. Some of their concerns are right. Their objections about deterministic outputs in financial, legal, or safety-critical contexts are usually correct, and you need to listen. Some are protecting their identity, and you need to tell the difference. I’ve watched this happen more than once: the most thoughtful skeptics, given time, become the most thoughtful adopters. They figure out how to direct the tools to get the quality they expect, and at that point, the technology stops feeling like a threat to their craft. It starts feeling like an extension of it.
The third group is the one I missed for too long: the quietly worried. These folks aren’t against AI. They’re asking what their job is becoming. They won’t raise their hand because the message from leadership is that this is important, and pushing back feels like challenging the company's direction. You have to ask.
Inside that group is a quieter, harder subset I didn’t see coming. Some of these are people who started excited. They evangelized the technology. They learned it. Then they watched the bar move faster than they could keep up. Now they feel like impostors. They were supposed to be ahead, and now they’re not, and many of them are now quietly worried about their jobs, which makes their silence even deeper. Their silence looks identical to the silence of the skeptics, but it means something completely different. If you only listen to who is loud, you’ll miss them entirely.
The hardest engineers to support aren’t the enthusiastic ones. They’re those who used to be.
These people dynamics echo beyond engineering; the same pressures are reshaping how executives behave. CEOs are now generating finished-looking output with AI and handing it to engineering teams, expecting it to be ready to ship. The visionary CEO who used to be an idea factory now has running prototypes, or at least something that looks like one. If it was hard when the CEO had more ideas than the team could implement, imagine how hard it is when the CEO is producing what they consider an MVP. Most engineering leaders I’ve talked to are not yet good at managing this. The conversation about what “ready to ship” actually means is one we’re going to have repeatedly at the executive level.
How my thinking has shifted, and what I’m still figuring out
My thinking has shifted over the last couple of years. A year ago, I thought of AI as a productivity tool. I no longer think that’s the right frame. Six months ago, I thought adoption was the central leadership challenge. I still think it’s a real challenge, but it’s the part we’re most equipped to handle. We have tools for adoption. They’re called change management. The leadership skill we need most right now is judgment about output quality, and we’re not yet training for it.
There’s plenty I’m still figuring out. How to measure the right things, not just usage. Token usage has never been a good metric, and if you measure tokens per engineer, you’ll end up with token-maxing, whether useful or not. How to evaluate engineers when their output includes work that an agent did. How to keep the craft of engineering alive in a world where the generative part is cheap. How to make sure we’re building things customers actually want, not just the things AI made it easy to build.
Where to look
If you’re looking for the AI transformation, don’t look at the IDE. Look at the org chart. Technology adoption is relatively easy. The organizational adoption is where everyone is struggling, and it’s because almost everyone is making it up as they go.
One thing to do this week: ask yourself which of the four layers your organization is actually working on right now. Then ask which of the four is actually failing. If those are not the same layer, that’s your real work.
Companies that succeed won’t just have the best tools. They will have leaders who continue to reflect and adapt, even as others declare challenges solved.
To hear an extended discussion of this topic, please listen to my most recent podcast episode: The Interesting Changes Aren’t Happening in the IDE.
Thanks again for reading! If you find it helpful, please share it with your friends.
Buy on Amazon | Buy on Bookshop.org | Buy on Audible | Other stores
Helpful lay of the land. But I was repeatedly wondering what you're seeing as ways to farm each land's challenges. Opportunity for no end of more posts.
But for one - re the conversation at the executive level about what “ready to ship” actually means - I've found having a Definition of Done extraordinarily useful. Forty-five minutes with each team (engineers, testers, product manager(s), designer) defining what engineering criteria, what "finish" criteria are required for any story or feature or product itself to be production ready. (And in this era, probably a second Definition of Done to be prototype-ready - that is, acceptable enough to feature-flag to just a few customers willing to Beta the thing.)