LLMs in Professional Software Engineering

This post is based on my talk at the Waterkant Festival 2026.

The AI hype is as ubiquitous as it is annoying. Some say that AI will beat software engineers at every task in a few months. Some say that human programmers will always be better than AI.

Obviously, both extremes are wildly wrong. But what does it help to say "the truth is more complicated" without actually figuring out what this complicated bit is?

Like most software engineers, I care about solving real problems in the real world. Unsolved problems. Hard problems. The stuff that tickles your mind, and that requires some novel solutions and the right trade-offs. And I think it's fun to do excellent work and build something outstanding.

Can LLMs even help me at all?

I have tried to use LLMs numerous times throughout the years. Every time I found that they produce slop and waste my time. They either forced me to accept bad changes, or they made me fix up their changes manually, which was slower than just doing everything by hand.

In early 2026, this changed. Time to collect some evidence.

Instead of just using the latest hype tool, I wanted to understand this properly. Let's take a more structured approach then:

1. Take a properly hard engineering problem.
2. Get your CTO to pay for infinite tokens.
3. Systematically try out various ways to use LLMs, and write down how it goes.

This is what I did in March 2026, and here is what I found out.

A Hard Problem

At my current employer, we solve large instances of the Vehicle Routing Problem (wiki). One reasonably hard problem from this domain is that we have to compute a distance matrix as input to our smart algorithms. What's more, our distance matrix implementation had long suffered from heaps of legacy code and stability issues, making it a good candidate for replacement.

Now, what is a distance matrix, you may ask?

Let me explain.

Essentially, we want to have a web server with a single endpoint. As input (request body), it accepts an array of locations as lat/lon coordinates.

{
  "coordinates": [
    { "lat": 54.0, "lon": 10.0 },
    { "lat": 54.1, "lon": 10.1 },
    { "lat": 54.2, "lon": 10.4 }
  ]
}

The server then looks at the world's road network and computes the best routes from each point to each other point, and returns only the distances and travel times as a large matrix. We do not return any information about the routes themselves, except for the length and duration.

Conceptually, the output (response body) looks like this:

{
  "distances": [
    [0, 18201, 55879],
    [18204, 0, 32444],
    [61390, 38199, 0]
  ],
  "times": [
    [0, 1319, 3121],
    [1343, 0, 2279],
    [3414, 2670, 0]
  ]
}

Note that from each point to itself, the distance is zero metres (and the time is zero seconds). Due to one-way streets, turn restrictions, etc, going from A to B is only approximately as far as from B to A.

What makes this problem so hard is the extreme performance that we require. For example, for 1,000 locations we have no more than 100 milliseconds.

That's right: one million routes must be computed and measured, and the results must be encoded and transmitted over the network and parsed by the client, and all of this must happen in less than a tenth of a second. Even if we ignore all the networking, we only have 100 nanoseconds to compute each route.

That seems impossible. I guess we can agree that this problem is reasonably hard. Vibe-coding this cannot work.

An Impressive Solution

I don't want to go into the details of the sophisticated algorithms and insane optimisations that were needed to pull this off. After all, this post is about how LLMs helped me, not about how the system works. But here is the data on what it took to build this service:

• 3 weeks of regular full-time work
• by a single person (me)
• with around €1200 in tokens

Around half of the time (and the tokens) was spent on the actual design and the rust implementation. The other half was spent building testing and benchmarking tooling in order to evaluate the solution, as well as to integrate it into our existing infrastructure.

Here is the p90 performance data for our c5a.4xlarge instance on AWS (8 physical AMD Zen 2 cores):

That's pretty solid! Computing 100M distances and travel times in under five seconds on a regular 8-core machine can be counted as a success.

Almost everything is LLM-generated. Out of approximately 15,000 lines of code in total, I think I wrote 4 manually and the other 14,996 or so with an LLM.

Along the way, I wrote a detailed log of what I tried, what worked, and what didn't. I noted down how I tried to work with LLMs, rather than saying anything about which algorithms I tried.

This diary now lets us answer the key question of this entire blog post:

When do LLMs actually help, and when should you avoid them?

The answer is … that it's the wrong question. At least, there is a better question to ask:

What Is an LLM?

I found that it's more helpful to build an understanding of what an LLM is. If you have a good intuition for what an LLM is, it is rather obvious how to characterize the kind of tasks where LLMs can help you.

I mean this in an intuitive sense, not in a technical one. Some people say that LLMs are next-token prediction machines. That's perfectly accurate but not what I mean. This intuition is not very enlightening in day-to-day work.

In other words, “here is a machine to predict the next word for you” does not tell me anything about how I should embed it into my workflow.

Instead, I believe we should understand LLMs as semantic translation machines. They translate an idea or a concept from one representation to another.

The term semantic translation needs some clarification.

Semantic Translation

By semantic translation, I mean that the same underlying concept or idea is translated from one representation to another.

This goes beyond merely translating between two human languages (something that LLMs are obviously very good at). For example, if you have an English text, it can be translated to a matching English summary.

Hypothetically, if you have an English text that describes a program in sufficient detail, such as a line-by-line description of all the operations (corresponding to a given programming language), then LLMs will be extremely good at translating this specification to the actual source code in that language.

Similarly, if you have a lot of source code, an LLM can summarise it for you.

What's common among these examples is that the idea exists, and the LLM rewrites it and lets you move to a different representation of the same idea. It does not have to come up with anything substantial on its own.

But don't we all know that LLMs hallucinate?! Even for simple translations we can't be sure of the output!

Correct. The process is probabilistic.

Probabilistic Semantic Translation

Essentially, when you shoot your shot at a translation, you don't hit your target exactly.

Instead, the LLM will give you output that is very close to what you wanted. The translation has a bit of uncertainty that introduces a slight error.

There is a great deal to be said about reducing this error. For example, going from a lot of info to very little info works well, and the other way around generally does not. (Trying to restore the long English text from its short summary will leave you with tons of hallucinations, and false and inaccurate statements.)

That being said, I will leave the discussion of reducing hallucinations to other people. For now, it is enough to acknowledge that semantic translation is probabilistic.

Another way of looking at this is that every translation incurs a debt to the truth. You not only change the representation of the idea, you also slightly distort the idea itself. This leaves you with a different idea, not quite identical to your original one.

With every hop to another representation, you add a new layer to your stack of adjacent ideas. The more steps you take, the further you will remove yourself from the concept you started with.

Not Semantic Translation

In contrast, here are a few things that are not mere translations.

This may sound obvious. If you want to find out what your customer wants, you should not ask an LLM. You should ask your customer.

If you ask an LLM about a fact, it will apply semantic translation to that question. The LLM effectively tells you:

Great question! People who ask these questions also make these statements about the topic …

… and then proceeds to list “facts” that it may or may not reproduce from its training data.

This is its way of representing the question by an answer that is as similar as possible to the question you asked. However, the facts needed for that answer were not part of the question, so they cannot be part of the translation and have to be made up.¹

Facts, requirements, or novel ideas² cannot be LLM-generated. They can only be LLM-translated.³ (This is especially true for things that did not appear often in the training data.)

LLMs for Software Engineering

Let's get a little more hands-on. We now have a good intuition for LLMs as semantic translation machines. But how exactly does this help programmers?

The thing is, semantic translation can happen in several steps, and combine several data sources. For example, a kind of prompt that works very well might contain

• a source file name
• a problem description
• a brief sketch of a refactoring plan

and then the LLM can perform the following steps of semantic translation:

1. source file name → a Read tool call
2. the problem description + refactoring plan → a set of refactoring steps
3. source code from (1) + refactoring steps → list of Write tool calls

If step (2) is non-trivial, or if the refactoring plan in your prompt does not have enough details, an LLM can fix that for you. Take your initial prompt and let the LLM translate it to a few Read tool calls that give it enough context to write a better prompt (usually called plan mode).

LLMs automate the grunt work.

You understand the problem, and you come up with the solution. The LLM helps you get there faster.⁴

In my case, doing a lot of performance work requires a ton of tasks that LLMs automate easily. Instrumenting code, running benchmarks, generating flamegraphs, sifting through endless amounts of performance metric data, and thereby finding bottlenecks are perfect examples of semantic translation. Those are trivial tasks for LLMs. Once you know what the exact bottleneck is, it's usually straightforward to fix it and repeat the process.

That's how LLMs help.

Addendum: Pattern Recognition and Recombination Machines

A different intuition for LLMs is called pattern recognition and recombination machines (thanks to Marc Heimann for telling me about it). The idea is that LLMs do not translate, but that they detect patterns that they can replicate and recombine. That's arguably a more accurate description when you factor in the underlying technology, but I would not say that it is necessarily more intuitive. If somebody interrupted me during programming and gave me a machine to recombine textual patterns, I would not know how to deal with it.

I intentionally chose a very liberal interpretation of the word translation. It includes things like translating 2 4 6 8 and generate 3 more numbers! to 2 4 6 8 10 12 14. Admittedly, this is clearly more of a pattern recognition task. My stance, however, is that 2 4 6 8 and 2 4 6 8 10 12 14 both are different representations of the same concept (positive even numbers), and continuing the sequence is just picking a different representation of that concept.