"Evaluating LLMs is a minefield"
Arvind Narayanan & Sayash Kapoor
<https:www.cs.princeton.edu/~arvindn/talks/evaluating_llms_minefield/>

"In short, many things can go wrong when we are trying to evaluate LLMs’ performance on a certain task or behavior in a certain scenario.

It has big implications for reproducibility: both for research on LLMs and research that uses LLMs to answer a question in social science or any other field."

Great slidedeck!

No, pair programming does not replace code reviews. That the reviewer(s) of a change is unfamiliar with the code is a *feature*, as that’s how *literally everyone* except for the author(s) will read it later.