Information Theory for Language Models: Jack Morris

In this episode of Late In Space, Swix interviews Jack Morris, a PhD student at Cornell Tech, about his research in AI, particularly in language models and information theory. They discuss the evolution of AI, the shift in power dynamics, and the impact of models like ChatGPT. Jack shares his experiences in grad school, the challenges of keeping up with industry advancements, and the importance of understanding hardware like GPUs and Mojo. They delve into Jack's research, including Contextual Document Embeddings (CDE), the concept of usable information, and the universal geometry of embeddings. The conversation touches on the limitations of current models, the potential for latent space alignment, and the idea that new datasets drive AI paradigm shifts. Jack also shares his thoughts on language model capacity, the challenges of extracting training data from model weights, and his vision for future research directions.