I’ve been trying to write a follow up to my last LLM post for nearly a year and all the ideas kind of founder because:

a) I think “oh, good another LLM post, just what the world needs”, and

b) At the speed I’ve been finishing things lately, the whole world will have changed three times by the time I finish this.

Still, it’s now been almost exactly a year since I started using LLMs significantly in how I interact with code and I have some thoughts:

They, like, work?

At RubyConf the other week I described the last six months as “On December 15th, everybody in this room stopped writing code.” It’s an exaggeration, but some point in fall 2025, the LLM models hit a tipping point where the generated code got good enough that it was clearly beneficial for many programmers on many tasks to use an LLM to write code. Note that I didn’t say “the best tool for all programmers on all tasks”.

I don’t, like, like this reality, but that is the reality. The LLMs basically work to write code. There’s all sorts of things they aren’t good at, but writing Ruby to a certain level of complexity, that is within their powers.

The tricky question is whether the models are continuing to get better in practice (I know that the metrics say they are getting better in the abstract). It’s hard to measure the problems you don’t have. I will say that although the models do currently struggle with some specific issues, and certainly never met complexity they didn’t like, they are better than they were last fall. I never have dynamic type errors in Ruby, and though I do sometimes have genuine functionality misses, it’s not that often. The models do seem to be getting better at picking up code style from surrounding code. They are also getting better at adjusting to a style or need if you push back.

I will say the harnesses have been getting better, though admittedly I haven’t done much with looping or multiple agents or that path.

The prototype trap

If you’ve ever been a consultant on a project where the client was not particularly technical, you’ve almost certainly shown a demo and had the client want you to ship it as-is. Because it’s done, right?

And you explain that just because the demo UI is pretty doesn’t mean it’s ready. There’s security, and error handling, and edge cases, and performance, and full functionality and all kinds of things. But it can be hard to convince somebody that when they see a polished UI demo, that it’s just a studio set.

In the world of the robots, the cost of a demo has dropped to basically zero. The cost of the other parts of the process – review, performance, security audit, compliance, weird edge cases – they’ve all dropped too , but they haven’t dropped nearly as much. Meaning that the relative cost of a demo to real code is way, way lower than it was.

Presumably, this is leading to increased pressure to ship demos. I see this even in my own work, just dealing with myself. I spend a bunch of time getting the requirements right and doing design, then the code happens in what seems like a blink, and then I have the relatively slow work of validating it, and managing the code logistics, and suddenly that all seems like it takes forever. It’s a genuinely weird and not all that great cognitive phenomenon pushing toward not reviewing the code because that slows things down.

Complexity

Two essays that I wrote about five years ago are interesting right now:

In 20 years, I looked at my first 20 years of professional projects and concluded:

• Software tools like Rails resulted about an order of magnitude improvement on delivery speed for web projects over that time
• That increase had partially been obscured by a corresponding increase in the minimum level of complexity needed to be a viable web app
• Over the same time, hardware had gotten faster by something like two orders of magnitude-ish
• That extra hardware speed had, from a developer’s perspective, been spent on making development more available to more people

In The Road To Legacy, I wrote:

• Code complexity increases exponentially with the number of internal dependencies in the code
• One thing that creates internal dependencies is developers not knowing the full context of a problem when the code is written
• Because the increase is exponential, you often don’t see the problem coming until it’s unavoidable

Put them both together and add robots…

LLMs represent a huge increase in our ability to create and comprehend code, possibly even an increase that scales at hardware speeds. That increase may or may not translate to organizational productivity. But we actually do have options about what we do with that increase. Do we put it into pure speed, do we put it into safer tooling, do we put it into ease of use?

LLMs, almost definitionally, are coding without full context in such a way that creates internal dependencies, and I would expect that without significant human input, they will do so in such a way that increases code complexity super-linearly.

Which suggests one of three potential outcomes:

• LLMs can keep up — the LLM ability to handle context increases exponentially faster than code complexity, forever. This, to me, seems unlikely.
• We figure out how to get LLMs to write code that does not increase in complexity exponentially. I mean, we haven’t so far, but you never know…
• After a brief period of productivity, code winds up back in a state where the totality is too complex to be managed at once, only now the code is two orders of magnitude more complicated.

I mean, I know where I’d bet…

We legitimately don’t know the side effects of letting these tools write code on lots of large codebases for more than a year.

I have my guesses, but I’ve been wrong enough in the last few months to be cautious. I do think that a lot of the ideas that keep cost of change low for human coders (especially encapsulation and DRY) are also going to prove to be useful for LLMs, because the constraints are still the constraints, but honestly, who knows?

Quality, ownership, fun, craftsmanship

After a few months of insisting that I wasn’t going to use LLMs on my personal projects, I gave in and built a couple of prototype level tools for my own use, the most interesting of which is basically a re-skin of Major League Baseball’s daily scoreboard.

Unlike my work projects, I largely haven’t done code review on them (code review is pretty token-intensive…). I’m in kind of a weird place with them. I’m glad they exist – I use the scoreboard basically daily. But I’m not sure I feel like I built them, and I can’t decide if that’s a minor quirk along the lines of “I’ll never use a telephone answering machine” or an existential career crisis. Or both.

In an unrelated story, about six months ago, in response to something or other, I said that I felt more identity threat from LLMs as a technical writer than as a software engineer, because on some level engineering still happens, whereas long-form technical writing seems like it’s being completely subsumed. I think that’s broadly still true, but…

Sometimes, at work or on side projects, using the LLMs does feel like creative work on my part being instantiated by a tool that does the boring part. Sometimes it feels like I’ve sorcerer’s apprenticed a bunch of brooms with buckets. Sometimes it feels like I’m just along for the ride. I can’t yet predict what actions are going to trigger what reactions. It does feel a little bit like when I was a manager briefly and had less contact with the ins and outs of code. And, of course, I stopped being a manager…

Conway’s Law + Robots?

What are the Conway’s Law implications of having a team that is using LLMs for code? Conway’s Law suggests that the structure of a codebase mimics the structure of the organization that produces it, or put another way, if the structure of your codebase does not mimic the structure of the organization, that’s going to lead to problems.

My first reaction was “no effect”, but I’ve definitely had issues trying to “pair program” with an LLM, to the point where it is sometimes easier to have the LLM write a chunk of code for me to review rather than have me edit it directly and have the LLM’s model of the code be out of date. That’s very much the kind of friction that leads to Conway’s Law problems.

One thing that seems true is that most organizations are not prepared for having 10x the code velocity running through their processes. My analogy here – if you ran 10x the water pressure through your plumbing, you would quickly find where the weak joins and leaks are. If you had a small communication problem, congratulations, you now have a big communication problem. A small bottleneck is now – presto chango – a big bottleneck.

A lot of people are coming to the conclusion that LLM use suggests smaller, more focused teams. Which makes a certain amount of sense, if you bound a team by their purview, LLM use at least in theory gives a smaller team the ability to have more purview.

The flip side, of course, is that more teams means more communication paths, and, in general more communication, and the communication is generally the part that is not going at LLM speed. So hopefully those teams have pretty well encapsulated code blocks, because everything is going to get more complicated.

My recommendation here, to the extent that I have one, is basic agile methodology – find your biggest problem, make a directional adjustment until it’s no longer your biggest problem, repeat. Retrospectives and reflections on what’s actually working seem like a good idea. We’re still in early enough days that experimentation still seems like a good idea, but doing so with some idea of what you are trying to do and some way of determining if you’ve done it or not.

Anyway, that’s where I am. 15 months ago I wrote “I don’t yet miss LLM chat when I don’t have it”, and I think the difference is now, I do. And I’m still not sure how I feel about it.