Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Thanks to Phillip Christoffersen, Adam Gleave, Anjali Gopal, Soroush Pour, and Fabien Roger for useful discussions and feedback.

TL;DR

This post overviews a research agenda for avoiding unwanted latent capabilities in LLMs. It argues that "deep" forgetting and unlearning may be important, tractable, and neglected. I discuss five things.

The practical problems posed when undesired latent capabilities resurface.
How scoping models down to avoid or deeply remove unwanted capabilities can make them safer.
The shortcomings of standard training methods for scoping.
A variety of methods can be used to better scope models. These can either involve passively forgetting out-of-distribution knowledge or actively unlearning knowledge in some specific undesirable domain.
Desiderata for scoping methods and ways to move forward with research on them.

There has been a lot of recent interest from the AI safety community in [...]

---

Outline:

(00:20) TL;DR

(01:23) The problem: LLMs are sometimes good at things we try to make them bad at

(01:57) Jailbreaks (and other attacks) elicit harmful capabilities

(03:07) Finetuning can rapidly undo safety training

(03:53) Conclusion: the alignment of state-of-the-art safety-finetuned LLMs is brittle

(04:19) A need for safely-scoped models

(04:23) LLMs should only know only what they need to

(05:28) Passive (whitelist) vs. active (blacklist) scoping

(06:23) Standard LLM training methods are not good for scoping

(06:57) Pretraining can introduce harmful artifacts into models.

(07:23) Finetuning is not good at making fundamental mechanistic changes to large pretrained models.

(11:01) Many potential strategies exist for stronger/deeper scoping

(11:06) Curating training data (passive)

(12:47) Plastic learning (passive)

(13:32) Compression/distillation (passive)

(14:26) Meta-learning (active)

(14:47) Model edits and lesions (active)

(16:15) Latent adversarial training (passive or active)

(18:12) Open challenges

(18:16) Improving deep forgetting and unlearning methods

(18:45) Meeting key desiderata

(21:56) Benchmarks

(22:39) What should we scope out of models?

---

First published:
December 5th, 2023

Source:
https://www.lesswrong.com/posts/mFAvspg4sXkrfZ7FA/deep-forgetting-and-unlearning-for-safely-scoped-llms

---

Narrated by TYPE III AUDIO.

“Deep Forgetting & Unlearning for Safely-Scoped LLMs” by scasper

LessWrong (30+ Karma)