AI has shifted from people trying AI to people depending on AI regularly. It is commonplace now for emails and documents to be drafted by AI without adequate human oversight, leading to embarrassing real-life consequences.

According to Parys Gazette, ChatGPT grew from 100 million active weekly users in November 2023 to a staggering 900 million in 2026.

AI use and dependency are likely to increase over time, which necessitates awareness of its inherent risks to allow appropriate safeguards to be developed.

Understanding the risks of hallucination

AI has rapidly become embedded in professional environments and while offering significant efficiency gains, a fabricated AI response (hallucination) or document undermines progress and causes much more harm than good.

Many users mistakenly treat AI like a database when, in fact, it is a probabilistic generator. This misunderstanding leads to unfortunate incidents where legal professionals cite case law that does not exist and, more recently, the government had to withdraw its own policy document on AI governance because it cited fictitious sources, embarrassingly drafted by AI. An analogy comes to mind: ‘Asking the prisoners how they would like to be guarded’.

There are two types of AI: Logic-based and Large Language Models (LLMs). LLMs learn by idea association and operate through intuitive learning. The engine of modern LLMs, such as ChatGPT, acts through pattern completion instead of truth retrieval and is unequipped with a mechanism to distinguish between what is true and what is plausible. The noted objective of the system is to reach a goal: To provide the most helpful and relevant answer. The dichotomy, of course, is that if it cites a false source or makes up a false line of reasoning, it is the exact opposite of helpful or relevant.

A technical flaw and a governance challenge

Incorrect AI-produced output relied on by professionals can lead to incorrect choices, legal exposure, and reputational damage. These incorrect outputs are occasioned by AI hallucination, where systems generate fabricated or misleading information. Hallucinations can produce entirely fabricated yet realistic outputs, including fake references and data. This makes them particularly dangerous in decision-making environments where accuracy is critical. In professional contexts, hallucination becomes not just a technical flaw but a governance challenge.

An appropriate Turkish proverb states: ‘When the axe entered the forest, what did the trees say? Look, the handle is one of us.’ The mind does not question what comes from a trusted voice; it receives it as truth. Often, trust misplaced in someone familiar can lead to destruction as one fails to recognise the threat until it is too late. This is precisely what occurs when AI reaches a tipping point where outputs shift from being accurate and trusted to fabricated. The question then is: Why do they lie and when?

According to a research paper published by D and N Restrepo in March, AI hallucinations are not a random glitch but a foreseeable consequence of the technology’s design. Generative AI works by predicting the most likely next word or phrase, not by understanding what is true. The problem is that it can give several correct answers in a row, building trust, before suddenly producing something completely false.

Navigating accountability and human judgment

Initially, the model provides harmless repetition and then shifts to valid reasoning. At this point, just when the user perceives the system to be reliable, requiring AI to resolve a complex, novel, or unsettled question pushes the model into a region where training data is sparse, leading to a fabricated response. The more complex or uncertain the question, the greater the likelihood of hallucination. Therefore, we know when it will lie.

Users have previously relied on the ‘black box’ defence to avoid accountability. This defence is premised around a misunderstanding of the technology and the belief that users were unaware that AI may hallucinate and create fictitious responses. But as consensus grows that AI-generated falsehoods are a foreseeable engineering risk, rather than an unforeseeable lie, the ‘black box’ defence is unlikely to succeed. This perspective paves the way for a more rigorous standard of technological competence and diligence across all industries.

A user is expected to understand, at least to a reasonable degree, how generative AI works, its limitations, and its propensity for fabrication before relying on it. User competence now not only involves the ability to use the system but also requires a practical understanding of how that software can fail. Moreover, the user must still verify the AI-generated response even if the initial response appears to be accurate.

With that in mind, human judgment remains essential not only for ensuring accuracy and reliability, but also serves as a prerequisite for professional indemnity cover. It is therefore of paramount importance that professionals consider the risks associated with relying on AI-generated responses.

Breaking news at your fingertips… Follow Caxton Network News on Facebook and join our WhatsApp channel.

Nuus wat saakmaak. Volg Caxton Netwerk-nuus op Facebook en sluit aan by ons WhatsApp-kanaal.