Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Quintin Pope & Nora Belrose have a new “AI Optimists” website, along with a new essay “AI is easy to control”, arguing that the risk of human extinction due to future AI (“AI x-risk”) is a mere 1% (“a tail risk worth considering, but not the dominant source of risk in the world”). (I’m much more pessimistic.) It makes lots of interesting arguments, and I’m happy that the authors are engaging in substantive and productive discourse, unlike the ad hominem vibes-based drivel which is growing increasingly common on both sides of the AI x-risk issue in recent months.

This is not a comprehensive rebuttal or anything, but rather picking up on a few threads that seem important for where we disagree, or where I have something I want to say.

Summary / table-of-contents:

Note: I think Sections 1 [...]

---

Outline:

(00:54) Summary / table-of-contents:

(02:10) 1. Even if controllable AI has an “easy” technical solution, I’d still be pessimistic about AI takeover

(06:10) 2. Black-and-white (box) thinking

(09:00) 3. What lessons do we learn from “human alignment” (such as it is)?

(13:54) 4. Can we all agree in advance to disavow this whole “AI is easy to control” essay if future powerful AI is trained in a meaningfully different way from current LLMs?

(16:47) 4.1 Examples where the essay is making points that don’t apply to “brain-like AGI” (≈ actor-critic model-based RL)

(19:44) 4.2 No, “brain-like AGI” is not trained similarly to LLMs

---

First published:
December 1st, 2023

Source:
https://www.lesswrong.com/posts/YyosBAutg4bzScaLu/thoughts-on-ai-is-easy-to-control-by-pope-and-belrose

---

Narrated by TYPE III AUDIO.

“Thoughts on ‘AI is easy to control’ by Pope & Belrose” by Steven Byrnes

LessWrong (30+ Karma)