François Chollet: Measures of Intelligence

05-28-26 ▶ 2h 34m 📖 4 min read

Core Takeaways

Francois Chollet sees intelligence as the ability to generalize efficiently to new situations, beyond prior knowledge.

Why it matters This definition challenges AI systems to focus on adaptability rather than rote learning, pushing the field toward more human-like intelligence.

The ARC test, developed by Chollet, benchmarks fluid intelligence by using tasks requiring core knowledge priors without external information. ▶ 1:20:00

Why it matters ARC's design highlights the importance of testing AI on novel problems to truly assess intelligence, influencing AI evaluation standards.

Chollet argues that language is an operating system for the mind, not fundamental to cognition itself. ▶ 10:00

Why it matters This perspective shifts focus to non-verbal cognitive processes, impacting how AI might simulate human-like thought.

Current AI models, like GPT, primarily perform pattern matching rather than true reasoning, limited by data quality. ▶ 2:10:00

Why it matters This highlights the need for better data curation in AI development to improve reasoning capabilities.

Chollet critiques the Turing test as outsourcing intelligence measurement to human judges, limiting its utility. ▶ 2:45:00

Why it matters Chollet's critique suggests the need for more rigorous and objective measures of machine intelligence.

Detailed Insights

Intelligence and Cognition

•

Chollet views intelligence as the ability to generalize beyond prior knowledge.

•

Language is seen as an operating system for the mind, not fundamental to cognition.

•

Cognition involves non-verbal processes like emotions and spatial reasoning.

AI Testing and Evaluation

•

ARC test benchmarks fluid intelligence using core knowledge priors.

•

Psychometrics measures cognitive abilities, focusing on reliability and validity.

•

The Turing test is critiqued for outsourcing intelligence measurement to human judges.

AI Model Limitations

•

GPT models perform pattern matching, not true reasoning.

•

Data quality is the bottleneck for scaling AI models.

•

Semantic web's failure is due to lack of incentive for structured data.

How the conversation moved

The host framed the episode around understanding intelligence and its measurement, inviting Francois Chollet to share his insights on cognitive processes and AI. Chollet began by discussing the influence of Jean Piaget and Geoff Hawking on his understanding of intelligence, particularly emphasizing the role of cognition over language. He argued that language is an operating system for the mind, rather than a fundamental aspect of cognition, challenging traditional views like those of Chomsky.

Chollet's main argument centered on defining intelligence as the ability to efficiently generalize beyond prior knowledge, which he believes is crucial for both human and artificial intelligence. He introduced the ARC test, a benchmark designed to measure fluid intelligence by using tasks that require core knowledge priors without relying on external information. This test aims to evaluate the adaptability and problem-solving skills of AI systems, setting a new standard for intelligence assessment.

Lex did not challenge Chollet's framing of intelligence directly, but there was a notable moment of tension when discussing the limitations of current AI models. Chollet critiqued the Turing test, arguing it outsources intelligence measurement to human judges, which Lex pushed back on by highlighting its historical significance. Chollet maintained that while the Turing test has inspirational value, it lacks the rigor needed for true AI evaluation.

The conversation concluded with a discussion on the limitations of GPT models, highlighting their reliance on pattern matching rather than true reasoning. Chollet emphasized the importance of data quality in AI development and critiqued the semantic web's feasibility due to a lack of incentives for structured data. The episode wrapped up with Chollet's call for more objective measures of intelligence, moving beyond traditional tests like the Turing test to more robust evaluations like the ARC.

Surprising moments

Francois Chollet

Chollet argued that language is an operating system for the mind, not fundamental to cognition, challenging Chomsky's views.

Francois Chollet

Chollet critiqued the Turing test as a cop-out for measuring intelligence, suggesting it relies too heavily on human judges.

Francois Chollet

Chollet pushed back against the notion that neural networks alone could achieve general intelligence, calling out intellectual laziness in the deep learning community.

Topics Covered

Intelligence and Cognition AI Testing and Evaluation AI Model Limitations

Memorable Quotes

"I see language as the operating system of the brain, of the human mind." — Francois Chollet

"Intelligence is how efficiently you're able to generalize far outside of the distribution of things you've seen already." — said_on_episode

"The degree in which the mind can generalize from its evolutionary history, can generalize away from its evolutionary history is much greater than the degree to which a deep learning system today can generalize away from its training data." — Francois Chollet

"The bottleneck in the case of these generative transformer models is absolutely the trained data." — said_on_episode

Still open

Unresolved by the end of the conversation

Chollet questioned whether current AI models can truly generalize beyond their training data, a challenge that remains unresolved.
Lex asked about the potential for neural interfaces to augment human intelligence, but the discussion left the practical implications open.

Jargon glossary

fluid intelligence

The ability to solve new problems, use logic in new situations, and identify patterns.

core knowledge priors

Innate knowledge systems that humans use to interpret the world, such as object permanence and basic geometry.

generalization

The ability to apply learned knowledge to new and varied situations.

References & Resources

On Intelligence by Geoff Hawking book

Goodreads Google Books

The Origins of Intelligence in Children by Jean Piaget book

Goodreads Google Books

Measure of Intelligence by Francois Chollet paper

arXiv Google Scholar

The Imitation Game by Alan Turing paper

arXiv Google Scholar

For the specialist

What a senior practitioner would find new

Chollet's ARC test focuses on fluid intelligence by requiring novel problem-solving without external knowledge, setting a new standard for AI evaluation.
Chollet argues that the Turing test's reliance on human judges limits its utility as an intelligence measure, suggesting a need for more objective benchmarks.

Ask this episode Deep

A preview of how Deep chat answers, grounded in this episode with citations and timestamps:

Cite this episode

For papers, blog posts, anywhere.

Copied!

Related episodes

Where to go next from this conversation.