ChatGPT has gotten “dumber” since it first launched, but better reasoning abilities will likely return in 2024 updates. Here’s why.
Self-driving cars, robot coworkers, space tourism. The future is coming fast, powered by artificial intelligence. ChatGPT is one of the most exciting AI systems out there. Launched in November 2022, this chatbot wowed us by writing songs, code, and essays.
But lately, some users report ChatGPT has gotten worse at logical reasoning. A 2022 Stanford study found the AI now contradicts itself more often. Even so, ChatGPT’s reasoning skills are poised for a comeback in 2024.
OpenAI CEO Sam Altman just tweeted a roadmap of planned 2024 upgrades. Better reasoning topped the list. With new techniques like “tree of thought” prompting, ChatGPT’s reasoning can improve dramatically. We’ll explore why Altman is prioritizing an upgrade.
We strongly recommend that you check out our guide on how to take advantage of AI in today’s passive income economy.
ChatGPT runs on a version of GPT-3.5, OpenAI’s powerful language model. When GPT-3 debuted in 2020, it showed strong logical reasoning abilities. But between GPT-3 and ChatGPT, something changed.
In July 2022, Stanford researchers tested ChatGPT on tasks requiring logical reasoning. They found “a marked decrease in coherence and common sense.” ChatGPT contradicted itself and made obvious factual errors.
For example, when told “Tom is taller than John” and “John is taller than Bob,” GPT-3 would logically conclude Tom is tallest. But ChatGPT struggled with such syllogisms. This research isn’t peer-reviewed yet, but many users have noticed the reasoning decline.
Altman Acknowledges User Frustration
When ChatGPT launched, people were amazed by its human-like responses. But the luster soon wore off as weird contradictions emerged. Users grew frustrated when ChatGPT made obvious mistakes or refused to engage with logic puzzles.
Sam Altman acknowledged this in his January 2023 Twitter thread. He said better reasoning is “one of the most mentioned things” users want improved. Altman’s team is “taking a look” at ways to upgrade reasoning.
They know the current version of ChatGPT doesn’t meet expectations. But exciting advances are coming that can restore lost logic skills.
“Tree of Thought” Prompting Sharpens Reasoning
Back in 2020, GPT-3 showed an impressive ability to reason about complex topics like philosophy. Its reasoning skills have since declined. But methods exist to restore logical thinking, even within today’s GPT models.
One technique is called “tree of thought prompting.” Here, you walk an AI through reasoning step-by-step to reach a logical conclusion. Explicitly laying out the reasoning chain keeps the AI focused.
For example, suppose you stated: “Liz is taller than Sue. Sue is taller than Jess.” Then asked: “Who is tallest?” Without tree of thought prompting, ChatGPT might give a wrong or nonsensical answer.
But if you prompt:
- Liz is taller than Sue
- Sue is taller than Jess
- So, Liz is taller than Sue
- And, Sue is taller than Jess
- Therefore, Liz is the tallest
Laying it out logically makes it more likely ChatGPT will conclude Liz is tallest. This shows how better prompting can improve reasoning.
“Smart GPT” Video Demonstrates Reasoning Potential
A YouTube channel called Anthropic demonstrated ChatGPT’s latent reasoning skills. In their viral “Smart GPT” video, they use careful prompting to solve logic puzzles far beyond what ChatGPT normally does.
For example, they prompt ChatGPT to figure out a logic grid puzzle. This involves deducing relationships between items based on a set of complex logical constraints. ChatGPT solves it flawlessly with the right prompting.
The Smart GPT creators say, with techniques like tree of thought prompting, “it seems possible to drastically improve GPT’s reasoning capabilities…” They make a strong case that ChatGPT has untapped logic potential.
Reasoning Advances Are Coming in 2024
Sam Altman admitted ChatGPT’s current reasoning capabilities lag behind expectations. But he tweeted “better reasoning” is a top priority for improvements in 2024.
We know from examples like Smart GPT that better prompting can unlock more advanced logic skills from these models. As OpenAI trains new versions of ChatGPT, prompting techniques will likely be incorporated to restore lost reasoning abilities.
OpenAI researcher Jacob Hilton tweeted that “technique improvements” enabled huge reasoning advances between GPT-2 and GPT-3. Similar gains likely occurred between GPT-3.5 and the upcoming GPT-4 model rumored to power future ChatGPT upgrades.
- Users are frustrated with contradictions and lapses in ChatGPT’s logical reasoning
- Careful prompting like “tree of thought” can improve reasoning substantially
- OpenAI CEO Sam Altman named better reasoning a top priority for 2024
- Technique improvements will likely restore ChatGPT’s reasoning to GPT-3 levels and beyond
The reasoning decline between GPT-3 and ChatGPT is unfortunate. But help is on the way in 2024 as OpenAI rolls out major upgrades!