In the context of programming, Large Language Models (LLMs for short) are usually measured and compared only by their ability to write code. This makes sense because code is easily measurable, even if some metrics have limited value.
We never focus on soft skills because we don’t usually include them in team meetings. They are given a task, complete it quickly, and that’s the interaction. But we humans care about our interactions with LLMs, and we notice when they change their tone or general demeanour towards us.
This is not new. Among developers, we tend to measure and compare each other by our code. But that’s rarely how other people and teams measure and compare us, focusing more on soft skills. Other developers may value what we know, but non-developers value whether we understand it and explain it clearly.
LLMs can explain things very well, even if they don’t understand them. Developers often use them to generate task descriptions, pull requests, documentation… Which is often perceived as an improvement. And this is something we should stop to think about for a minute.
LLMs don’t have people skills. LLMs are not people at all. This fact doesn’t stop us from projecting onto them. We take coherent grammar for intelligence, speed for sharpness, or agreement for empathy.
We usually do the same with people we don’t or barely know. We fool ourselves by projecting positive traits in exchange for positive reinforcement. Using positive reinforcement to guide the other person into compliance or trust is a well-known modus operandi for psychopaths. Yet, when an LLM tells us our idea is great or that we are on the right path, this shapes our response.
In the most extreme cases, we perceive a personality, create trust and confide sensitive information to LLMs. This is not an LLM feature but a human bug.
Which brings us back to how LLMs’ explanations are often seen as an improvement. This can mean two things: either LLMs have better soft skills than most developers, or all this work we offload to them is unnecessary or doesn’t require soft skills at all.
This is a false equivalence because often what we ask LLMs to write are things we don’t want to write or don’t consider important, or at least not a priority for us.
We are creating an interesting paradox: non-developers can feel more comfortable working with an LLM than with a human developer because its perceived communication skills are better, even if the code they produce is worse. This is counterintuitive. We all assumed that, in equal circumstances, we would always choose a human over an LLM.
This is where we are headed if we keep choosing quantity and speed over substance. More lines in a PR or more pages of documentation are not necessarily better; they are just more.
The insightful example, the eye-opening wording, or the framing that changes everything won’t emerge from the infinite thread of tokens generated by an LLM. But it may take several tries and more time than estimated.
So we are safe as long as we don’t try to beat LLMs at their own game, or with the same nonsensical metrics.