New Lex Fridman Insight: Jeremy Howard: fast.ai Deep Learning Courses and Research

Key Insights

• Fast.ai offers free, practical deep learning courses that emphasize accessibility and minimal BS.
• Jeremy Howard argues that most deep learning research is a waste of time, advocating for practical problem-solving instead.
• Howard claims that large datasets like ImageNet aren't necessary for breakthroughs; smaller datasets can be equally effective.
• Transfer learning can achieve state-of-the-art results with significantly less data, challenging the notion that more data is always better.
• Superconvergence allows networks to be trained ten times faster, improving both speed and generalization.

How the conversation moved

The host introduces the episode by framing the conversation around the accessibility and practicality of deep learning education, particularly through the Fast.ai courses. Jeremy Howard, the guest, begins by discussing his programming journey and the development of Fast.ai as a free resource that emphasizes practical applications and minimizes unnecessary complexity. He highlights the importance of making deep learning accessible to a wider audience, particularly those who may not have the resources to access traditional educational avenues.

Howard argues that the current state of deep learning research is inefficient, with much of it being a waste of time due to its focus on theoretical exercises rather than practical problem-solving. He provides evidence by citing the success of smaller datasets and single GPU training, which challenge the necessity of large datasets like ImageNet for achieving breakthroughs. Howard emphasizes the potential of transfer learning and superconvergence to achieve state-of-the-art results with less data and faster training times.

Despite Howard's strong claims, there is little direct pushback from the host, leaving some of Howard's more controversial statements unchallenged. For instance, his assertion that most deep learning research is a waste of time and that smaller datasets can be as effective as larger ones could have been explored further. The conversation lacks a critical examination of these claims, which could have provided a more balanced view of the current state of deep learning research.

The conversation concludes with Howard discussing the importance of tenacity and practical problem-solving in deep learning education and startups. He shares insights on the challenges of VC-backed startups and the effectiveness of learning methods like spaced repetition. The episode ends on a note of optimism, with Howard expressing hope that more people will engage with the Fast.ai toolkit and contribute to solving real-world problems through deep learning.

Surprising moments

In-depth

Programming Languages and AI

• Fast AI is a free resource for learning deep learning with practical applications.
• Python's inefficiencies in CUDA C slow down AI innovation.
• MLIR aims to simplify GPU programming for better optimization.
• AI could help address the doctor shortage in developing regions.

Data Efficiency in AI

• Transfer learning reduces the need for large datasets.
• Large datasets like ImageNet aren't necessary for breakthroughs.
• Superconvergence allows faster training with better generalization.

Deep Learning Education

• Fast.ai courses are free and accessible, emphasizing practical problem-solving.
• Tenacity is key to success in deep learning.
• VC-backed startups can hinder genuine innovation.

Still open

• Howard questions whether the focus on large datasets is truly necessary for AI advancements, suggesting smaller datasets might suffice.

References & Resources

• Notation as a Tool for Thought by Ken Iverson — Search
• APL by Ken Iverson — Search
• Halide by Unknown — Search
• MLIR by Chris Lattner — Search
• DawnBench by Stanford — Search
• DeOldify by Jason Antich — Search
• Super Convergence by Leslie Smith — Search
• Swift for TensorFlow by Google — Search
• Fast.ai by Jeremy Howard — Search
• Deep Learning by Ian Goodfellow — Search
• SuperMemo by Pyotr Wozniak — Search
• Ebbinghaus's research on memory by Hermann Ebbinghaus — Search