Andrej Karpathy

This week, Andrej Karpathy talked about the impressive capabilities of AI video generation models, which can produce high-fidelity videos by training on a dataset of videos. He also compared the performance of these models to large language models (LLMs), wondering why video generation models are too good while LLMs struggle with text generation.

Karpathy experimented with the o1-pro LLM, generating a custom 'book' on a topic he was interested in. He also discussed the new Gemini 2.0 Flash Thinking model, which shows its reasoning traces, making it interesting to see how the model thinks through different possibilities and ideas.

Karpathy tried out Veo 2 and its 'Automation Wizard' feature, and also mentioned the LLM consortium, which allows multiple LLMs to come to a consensus on a given prompt. He also asked about good prediction markets for AI, mentioning Metaculus as a potential leading one.

Karpathy was impressed by the tldraw computer, which allows users to lay out interactive and visual programs in 2D that incorporate LLM elements. He described it as 'very cool and creative', highlighting the potential for innovative AI applications.

This week, Andrej Karpathy explored various applications of Large Language Models (LLMs), including reading books together and generating discussions. He also highlighted the potential for AI-native reader apps and the creative process of using AI video models for movie-making. Additionally, Karpathy discussed the importance of AI capabilities, not just in solving complex problems, but also in performing everyday tasks like a junior intern.

Karpathy noted the increasing hype around cloud GPUs, with billboards advertising them on the streets of San Francisco. He also mentioned the barrier to movie creation continuing to decrease, with the help of AI video models and tools like ComfyUI.

This week, Andrej Karpathy shared the story of how the Transformer was developed, including the inspiration behind the attention operator. He discussed how attention is a powerful and expressive operation that allows for global pooling and reduction, making it a major unlock in neural network architecture design. Karpathy also touched on the history of the Transformer paper and how it built upon earlier work by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.

Andrej Karpathy shared his thoughts on what it means to 'ask an AI' about something, highlighting that it's more like asking the average data labeler. He emphasized that AI models are trained by imitation on data from human labelers, and that the idea of 'asking an AI' can be misleading. This sparked a discussion on the limitations and potential of AI.

Karpathy also shared an article about the reality of the Turing test, which assesses a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. The article sparked interest and discussion among his followers.