This post is part of a sequence on LLM psychology

TL;DR

We introduce our perspective on a top-down approach for exploring the cognition of LLMs by studying their behavior, which we refer to as LLM psychology. In this post we take the mental stance of treating LLMs as “alien minds,” comparing and contrasting their study with the study of animal cognition. We do this both to learn from past researchers who attempted to understand non-human cognition, as well as to highlight how much the study of LLMs is radically different from the study of biological intelligences. Specifically, we advocate for a symbiotic relationship between field work and experimental psychology, as well as cautioning implicit anthropomorphism in experiment design. The goal is to build models of LLM cognition which help us to both better explain their behavior, as well as to become less confused about how they relate to risks [...]

---

Outline:

(00:12) TL;DR

(03:10) Experiments vs Field Study

(03:40) Experimental psychology

(05:39) Field study

(08:18) Field Study for LLM psychology

(09:34) Studying LLMs is different

(09:45) Anthropomorphism

(10:31) Replicability

(11:50) Volume and diversity of data

(12:34) Ethical considerations

(13:19) Exploring counterfactuals

(15:08) Checkpointing models

(16:15) A two stage model of science

(19:44) Species specific experiments

(22:54) LLM-specific research

(25:36) Chat models are still predictors

(26:35) Conclusion

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
December 5th, 2023

Source:
https://www.lesswrong.com/posts/suSpo6JQqikDYCskw/studying-the-alien-mind-1

---

Narrated by TYPE III AUDIO.

“Studying The Alien Mind” by Quentin FEUILLADE--MONTIXI, NicholasKees

LessWrong (30+ Karma)