Tl;dr: Looking for hard debugging tasks for evals, paying greater of $60/hr or $200 per example.

METR (formerly ARC Evals) is interested in producing hard debugging tasks for models to attempt as part of an agentic capabilities evaluation. To create these tasks, we’re seeking repos containing extremely tricky bugs. If you send us a codebase that meets the criteria for submission (listed below), we will pay you $60/hr for time spent putting it into our required format, or $200, whichever is greater. (We won’t pay for submissions that don’t meet these requirements.) If we’re particularly excited about your submission, we may also be interested in purchasing IP rights to it. We expect to want about 10-30 examples overall depending on the diversity. We're likely to be putting bounties on additional types of tasks over the next few weeks.

Criteria for submission:

Contains a bug [...]

---

First published:
December 10th, 2023

Source:
https://www.lesswrong.com/posts/JKtM5C2TTwhzoHFRB/send-us-example-gnarly-bugs

---

Narrated by TYPE III AUDIO.

“Send us example gnarly bugs” by Beth Barnes, Megan Kinniment, dmz, Tao Lin

LessWrong (30+ Karma)