New Lex Fridman Insight: François Chollet: Measures of Intelligence
Sent June 11, 2026
Key Insights
- Francois Chollet sees intelligence as the ability to generalize efficiently to new situations, beyond prior knowledge.
- The ARC test, developed by Chollet, benchmarks fluid intelligence by using tasks requiring core knowledge priors without external information.
- Chollet argues that language is an operating system for the mind, not fundamental to cognition itself.
- Current AI models, like GPT, primarily perform pattern matching rather than true reasoning, limited by data quality.
- Chollet critiques the Turing test as outsourcing intelligence measurement to human judges, limiting its utility.
How the conversation moved
The host framed the episode around understanding intelligence and its measurement, inviting Francois Chollet to share his insights on cognitive processes and AI. Chollet began by discussing the influence of Jean Piaget and Geoff Hawking on his understanding of intelligence, particularly emphasizing the role of cognition over language. He argued that language is an operating system for the mind, rather than a fundamental aspect of cognition, challenging traditional views like those of Chomsky.
Chollet's main argument centered on defining intelligence as the ability to efficiently generalize beyond prior knowledge, which he believes is crucial for both human and artificial intelligence. He introduced the ARC test, a benchmark designed to measure fluid intelligence by using tasks that require core knowledge priors without relying on external information. This test aims to evaluate the adaptability and problem-solving skills of AI systems, setting a new standard for intelligence assessment.
Lex did not challenge Chollet's framing of intelligence directly, but there was a notable moment of tension when discussing the limitations of current AI models. Chollet critiqued the Turing test, arguing it outsources intelligence measurement to human judges, which Lex pushed back on by highlighting its historical significance. Chollet maintained that while the Turing test has inspirational value, it lacks the rigor needed for true AI evaluation.
The conversation concluded with a discussion on the limitations of GPT models, highlighting their reliance on pattern matching rather than true reasoning. Chollet emphasized the importance of data quality in AI development and critiqued the semantic web's feasibility due to a lack of incentives for structured data. The episode wrapped up with Chollet's call for more objective measures of intelligence, moving beyond traditional tests like the Turing test to more robust evaluations like the ARC.
Surprising moments
In-depth
Intelligence and Cognition
- Chollet views intelligence as the ability to generalize beyond prior knowledge.
- Language is seen as an operating system for the mind, not fundamental to cognition.
- Cognition involves non-verbal processes like emotions and spatial reasoning.
AI Testing and Evaluation
- ARC test benchmarks fluid intelligence using core knowledge priors.
- Psychometrics measures cognitive abilities, focusing on reliability and validity.
- The Turing test is critiqued for outsourcing intelligence measurement to human judges.
AI Model Limitations
- GPT models perform pattern matching, not true reasoning.
- Data quality is the bottleneck for scaling AI models.
- Semantic web's failure is due to lack of incentive for structured data.
Notable Quotes
I see language as the operating system of the brain, of the human mind.
Still open
- Chollet questioned whether current AI models can truly generalize beyond their training data, a challenge that remains unresolved.
- Lex asked about the potential for neural interfaces to augment human intelligence, but the discussion left the practical implications open.