ICLR 2024 — Best Papers & Talks (ImageGen, Vision, Transformers, State Space Models) ft. Durk Kingma, Christian Szegedy, Ilya Sutskever | Latent Space: The AI Engineer Podcast

This podcast episode explores various topics in deep learning, including variational autoencoders (VAEs), diffusion models, generative models, adversarial machine learning, attention maps, context extension, large language models, and self-pre-training. It discusses the concepts, challenges, applications, and potential future developments in these areas. The episode also highlights the importance of understanding internal representations, unsupervised learning, compression algorithms, and data subselection schemes in improving model performance. Overall, the episode provides valuable insights into different aspects of deep learning and its applications in various domains.

Outlines

Sign in to continue reading, translating and more.

Continue

ICLR 2024 — Best Papers & Talks (ImageGen, Vision, Transformers, State Space Models) ft. Durk Kingma, Christian Szegedy, Ilya Sutskever

Latent Space: The AI Engineer Podcast

Overview of Variational Autoencoders and the Reparameterization Trick

The Reparameterization Trick and Variational Autoencoders

Re-parameterization and the ELBO: Exploring Variational Autoencoders

The Emergence of Diffusion Models in Deep Learning: A Look into the Past and Future

Explaining Generative Models and the Importance of Interpretability

Understanding internal representations and the power of unsupervised learning.

Unsupervised Learning and the Power of Compression Algorithms

The Emergence of Adversarial Learning: Robustness Problems and Research Advances

Understanding Adversarial Examples and Their Generalization

Adversarial Examples, Attention Maps, and the Value of Cross-Dataset Generalization

Analysis of Attention Maps in Dyno v2 and the Role of Outliers

Vision Transformers and the Role of Pause Tokens for Model Performance Improvement

Surprising Findings on Subselection Schemes: Unbiased Subsampling, Weaker Surrogates, and Subsampling vs Full Sample Training

The Importance of Uncertainty-Based Subsampling and Self-Supervised Learning

LongLaura and LongALPACA: Extending Contexts and Designing Long Questions

Context Extension and the Need for High-Quality Data

Synthetic Data and KVCache Compression for Large Language Models

Insights into Attention Sparsity and Compression Techniques in Autoregressive LLM Models

Sharding and Overlapping for Speeding Up Training in DeepSpeed - Interview Highlights

Understanding State Space Models and Mamba-like Models for Language Modeling

Linear Time-Invariant Models and the Solution for Fixing Their Issues

Mamba: A Powerful Model for Byte-level Modeling

Self-Pre-Training for Better Evaluation and Improvement in Performance

ICLR 2024 — Best Papers & Talks (ImageGen, Vision, Transformers, State Space Models) ft. Durk Kingma, Christian Szegedy, Ilya Sutskever

Latent Space: The AI Engineer Podcast

00:12Overview of Variational Autoencoders and the Reparameterization Trick

Overview of Variational Autoencoders and the Reparameterization Trick

08:21The Reparameterization Trick and Variational Autoencoders

The Reparameterization Trick and Variational Autoencoders

18:13Re-parameterization and the ELBO: Exploring Variational Autoencoders

Re-parameterization and the ELBO: Exploring Variational Autoencoders

28:32The Emergence of Diffusion Models in Deep Learning: A Look into the Past and Future

The Emergence of Diffusion Models in Deep Learning: A Look into the Past and Future

39:00Explaining Generative Models and the Importance of Interpretability

Explaining Generative Models and the Importance of Interpretability

48:01Understanding internal representations and the power of unsupervised learning.

Understanding internal representations and the power of unsupervised learning.

57:33Unsupervised Learning and the Power of Compression Algorithms

Unsupervised Learning and the Power of Compression Algorithms

1:04:17The Emergence of Adversarial Learning: Robustness Problems and Research Advances

The Emergence of Adversarial Learning: Robustness Problems and Research Advances

1:13:41Understanding Adversarial Examples and Their Generalization

Understanding Adversarial Examples and Their Generalization

1:23:27Adversarial Examples, Attention Maps, and the Value of Cross-Dataset Generalization

Adversarial Examples, Attention Maps, and the Value of Cross-Dataset Generalization

1:30:09Analysis of Attention Maps in Dyno v2 and the Role of Outliers

Analysis of Attention Maps in Dyno v2 and the Role of Outliers

1:38:03Vision Transformers and the Role of Pause Tokens for Model Performance Improvement

Vision Transformers and the Role of Pause Tokens for Model Performance Improvement

1:48:54Surprising Findings on Subselection Schemes: Unbiased Subsampling, Weaker Surrogates, and Subsampling vs Full Sample Training

Surprising Findings on Subselection Schemes: Unbiased Subsampling, Weaker Surrogates, and Subsampling vs Full Sample Training

2:00:26The Importance of Uncertainty-Based Subsampling and Self-Supervised Learning

The Importance of Uncertainty-Based Subsampling and Self-Supervised Learning

2:10:09LongLaura and LongALPACA: Extending Contexts and Designing Long Questions

LongLaura and LongALPACA: Extending Contexts and Designing Long Questions

2:21:22Context Extension and the Need for High-Quality Data

Context Extension and the Need for High-Quality Data

2:28:23Synthetic Data and KVCache Compression for Large Language Models

Synthetic Data and KVCache Compression for Large Language Models

2:37:33Insights into Attention Sparsity and Compression Techniques in Autoregressive LLM Models

Insights into Attention Sparsity and Compression Techniques in Autoregressive LLM Models

2:49:28Sharding and Overlapping for Speeding Up Training in DeepSpeed - Interview Highlights

Sharding and Overlapping for Speeding Up Training in DeepSpeed - Interview Highlights

2:58:09Understanding State Space Models and Mamba-like Models for Language Modeling

Understanding State Space Models and Mamba-like Models for Language Modeling

3:09:10Linear Time-Invariant Models and the Solution for Fixing Their Issues

Linear Time-Invariant Models and the Solution for Fixing Their Issues

3:18:36Mamba: A Powerful Model for Byte-level Modeling

Mamba: A Powerful Model for Byte-level Modeling

3:27:35Self-Pre-Training for Better Evaluation and Improvement in Performance

Self-Pre-Training for Better Evaluation and Improvement in Performance