LLMs Are Not a “PhD-Level Expert in Your Pocket”, at Least Not a Statistician

External registration required at nyhackr.

This month we have George Perrett casting doubt on LLMs.

Thank you to NYU for hosting us.

Everybody attending must RSVP through the registration form at nyhackr. There is a charge for in-person and virtual tickets are free.

Space is extremely limited and in-person registration closes at 3 PM the day of the talk.

About the Talk:
LLMs are hot right now. Bold claims about LLMs' capacity abound. Sam Altman, the CEO of OpenAI, has described the capacity of their LLM as "a team of PhD-level experts in your pocket". Dario Amodei, the CEO of Anthropic, predicted that by the present, 90% of computer code will be written by LLMs. However, little empirical evidence supports these assertions. Available benchmarking datasets are easily manipulated and do not generalize to novel tasks. In this talk, I will present pilot data that demonstrate the claims about LLMs are vastly overstated, at least within the context of statistics and statistical computing. I leverage a novel research design that enables direct comparison of LLMs and expert statisticians on a data analysis task while controlling for the effects of prompt engineering. Additionally, I apply causal inference methods to challenge the claim that LLMs have improved productivity within the knowledge economy.

About George:
I am a causal inference researcher who develops statistical tools for the social and medical sciences. Currently, I am pursuing my PhD at NYU where I research novel causal inference methods for multiple treatments and causal inference with latent confounding variables. I am also very interested in the political economy of LLMs and their applications within social science.

The venue doors open at 6:30 PM America/New_York where we will continue enjoying pizza together (we encourage the virtual audience to have pizza as well). The talk, and livestream, begins at 7:00 PM America/New_York.

Remember, register at nyhackr.