Disclaimer: These are negative results and this post is below the normal quality bar I use. These were quick experiments (a couple of days of work) and I don’t have much expertise in RL. They don’t provide strong evidence about the natural emergence of steganography.

People speculate that LLMs might learn to do steganography, and I’ve shown that you can teach LLMs to implement a specific steganography scheme that RLHF could in principle incentivize. But even if incentivized, are current RL methods powerful enough to teach LLMs how to do steganography?

Here are two negative results, which provide some evidence that hidden 1-bit communication is relatively hard for RL to learn. I don’t claim that RL can’t teach LLMs to do steganography - it's very likely that better RL setups (e.g. the ones labs use for instruction tuning or setups that simply have better hyperparameters or more training time) [...]

---

Outline:

(01:30) Experiment 1: PPO on GPT2-large

(03:17) Experiment 2: poor-mans RL on GPT-3.5-Turbo

(04:48) Other experiments

---

First published:
December 9th, 2023

Source:
https://www.lesswrong.com/posts/EEvsL9cpgDAxAhTzt/some-negative-steganography-results

---

Narrated by TYPE III AUDIO.

“Some negative steganography results” by Fabien Roger

LessWrong (30+ Karma)