I see people use "in-context learning" in different ways.

Take the opening to "In-Context Learning Creates Task Vectors":

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework, where one uses a training set <span>_S_</span> to find a best-fitting function <span>_f(x)_</span> in some hypothesis class.

In one Bayesian sense, training data and prompts are both just evidence. From a given model, prior (random weights), and evidence (training data), you get new model weights. From the new model weights and some more evidence (prompt input), and a distribution of output text. But the "training step" <span>_(text{prior}, text{data}) rightarrow text{weights}_</span> and "inference step" <span>_(text{weights}, text{input}) rightarrow text{output}_</span> could be simplified to a single function:<span>_(text{prior}, text{data}, text{input}) rightarrow text{output}_</span>. An LLM trained on [...]

---

First published:
February 25th, 2024

Source:
https://www.lesswrong.com/posts/vkCCbPzmMhJgvbtfK/in-context-learning-1

---

Narrated by TYPE III AUDIO.

“‘In-Context’ ‘Learning’” by Arjun Panickssery

LessWrong (30+ Karma)