Since the boom in generative AI models, many organisations are relying on these systems to produce content, conduct documentary research, or assist in certain decision-making processes. The responses are fast, structured, and often convincing. However, this quality of formulation can lead to overestimating their level of understanding and their reliability.
Why do these models seem so credible to us? Why do we tend to attribute intelligence to them that they don't possess? What cognitive mechanisms explain this perception?
In this article, find out why generative AIs inspire immediate trust, what their real limitations are and why it remains essential to maintain a critical eye on their answers.
Why do generative AIs give the impression of understanding?
Since the arrival of generative AI models, we seem to be increasingly confusing performance and understanding. We don't see the mechanics, so we imagine a mind. Our brains spontaneously try to attribute intention or logic to behaviours that strike us as coherent.
There's nothing modern about this confusion; it arises as soon as an inert object borrows our codes: it's anthropomorphism. This phenomenon can also be found in military robotics. Some soldiers deployed with robot dogs develop a form of attachment to the robot. The robot moves forward, explores and returns. It repeats a stable, almost familiar behaviour. When it is destroyed, it is not just a tool that is lost: it is an imagined presence. As soon as a system works, reacts or responds, we lend it a form of intention.
The tendency to overestimate the capabilities of a conversational system and attribute to it an understanding or intelligence that it does not possess. Named after ELIZA, a programme created by Joseph Weizenbaum at MIT (1964-1966) which simulated a psychotherapist: although it merely manipulated linguistic patterns, its users confided intimate thoughts to it, convinced that it truly «understood» them. See the full definition
The rise of language models has only amplified this illusion: when an AI nuances, contextualises, or reformulates, our minds naturally lean towards the idea of reasoning. This cognitive mechanism is not neutral: in business, it can lead to granting AI a level of reliability or understanding that it does not possess.
Why chatting with an AI isn't a real conversation.
Another source of confusion lies in the very notion of a «conversation» with LLMs. When a user interacts with a generative AI, they naturally feel as though they are having a continuous dialogue. However, the model does not build a lasting understanding of the exchange as a human interlocutor would. With each response, it simply receives a context composed of the previous messages sent to it and generates the most probable text sequence. It retains neither memories, nor a lasting understanding of the exchange, nor a mental representation of what has been said.
This limitation explains why a model can lose track of a long conversation, forget information mentioned just a few exchanges earlier, or produce an incoherent response when the context becomes incomplete. It is not a lack of attention, but a direct consequence of its functioning. Without context, a language model «knows» nothing of the previous conversation.
This lack of understanding also explains the phenomenon of hallucinations. When information is missing, ambiguous, or out of the available context, the model does not assess the veracity of information: it generates the most probable text sequence given the available context. It can thus invent a reference, attribute a quote to the wrong person, or present an erroneous fact with great confidence. The more fluent and convincing the phrasing, the more difficult it becomes to distinguish an accurate answer from an invented statement.
What place should artificial intelligence be given within an organisation?
In this context, should all of one's projects and strategy be entrusted to generative AI? The question isn't so much whether one should use it, but how one does so. The real risk would be to do so without reflection. The real risk isn't using AI, but using it without a framework, without reflection, and without arbitration.
The term «artificial intelligence» itself influences our perception. It spontaneously evokes capabilities such as understanding, discernment, or intention. Yet, these models do not make decisions in the human sense of the term. They produce responses based on statistical correlations learned during their training. As soon as the word 'intelligence' is used, our minds spontaneously project human attributes: understanding, intention, discernment. However, these systems possess neither consciousness, nor will, nor judgment. They calculate, correlate, predict. Recognising this cognitive mechanism in no way diminishes the usefulness of AI. It simply prevents us from attributing to it what it has never carried: intention, responsibility, a form of spirit.
The challenge is to maintain a clear-headed stance. Use AI for what it truly is: an extremely powerful probabilistic tool, but incapable of discernment.
Comment utiliser une IA générative sans surestimer ses capacités ?
Using generative AI wisely isn't about being systematically distrustful, but about understanding what it's actually doing. These models excel at exploring ideas, reformulating text, synthesising information, or speeding up certain intellectual tasks. On the other hand, they don't distinguish truth from falsehood, they don't verify their claims, and they don't possess a contextual understanding comparable to that of a human.
This distinction invites us to adapt the level of confidence afforded to the generated responses. A proposal may serve as an excellent starting point without necessarily becoming a decision. The more important the consequences of a subject – whether legal, medical, financial, scientific, or strategic – the more indispensable the verification of information and the exercise of human judgment become.
This vigilance is all the more important as model responses are often formulated with confidence. The fluency of the language, the quality of the writing, or the apparent precision of an explanation are not proof of reliability. A convincing answer may be accurate, approximate, or completely wrong. Critical evaluation must therefore focus on the content itself and not on how it is presented.
Finally, using generative AI responsibly means considering it as a tool to aid reflection rather than as an arbiter. It can enrich an analysis, open new avenues, or speed up research work, but it neither replaces business expertise, nor discernment, nor the responsibility for decisions. It is precisely this complementarity that allows its full potential to be exploited without attributing to it capabilities it does not possess.
Clarity consists of not confusing the remarkable performance of generative models with understanding comparable to that of a human. It is a sophisticated probabilistic model, remarkable for what it enables, but devoid of subjectivity. It is only by seeing language models for what they truly are that we can decide what place we want – or do not want – to give them.