ChatGPT's language comprehension unveiled: Misconceived notions exposed
In a groundbreaking study published in PLOS One, psycholinguist Michael Vitevitch from the University of Kansas delved into the intricacies of how AI processes language, using methods typically employed to study human language processing. The focus of the study was on the large language model (LLM) ChatGPT, revealing both its strengths and limitations when faced with complete linguistic nonsense.
The study found that while ChatGPT excels at pattern recognition, its approach differs significantly from humans. When presented with nonsense input, ChatGPT can sometimes produce nonsensical or random errors, or generate vague, irrelevant, or verbose answers as a fallback. This is due to its reliance on patterns learned from coherent language data, as it lacks the cognitive ability to recognise nonsense as such.
One example of ChatGPT's struggle with nonsense is its creation of words like 'rousrage' for anger expressed upon being woken and 'carperpetuation' for a thread that doesn't get sucked up by a vacuum cleaner. In contrast, humans can often identify and disregard gibberish due to semantic and pragmatic awareness.
Another interesting finding was that ChatGPT drew from other languages when responding to Spanish words or inventing new English words for modern concepts. However, it correctly defined 36 out of 52 archaic terms tested, such as 'upknocking', a 19th-century job where people tapped on windows to wake others before alarm clocks.
Despite its limitations, Vitevitch argues that the goal is not to mimic human cognition but to identify where AI can complement our linguistic strengths. The study serves as a reminder that while AI has come a long way, it still falls short in understanding and processing language in the way humans do.
The original publication of this research can be found in Cosmos, providing a fascinating insight into the world of AI and language processing.
The study proves that ChatGPT, a large language model, excels in pattern recognition but produces nonsensical or vague responses when faced with linguistic nonsense, as it lacks the cognitive ability to recognize such inputs as meaningless. In contrast, humans display semantic and pragmatic awareness, enabling them to detect and disregard gibberish effectively.
