LLMs won't lead to AGI: The ARC Benchmark and the $1M ARC Prize
Details
Francois Chollet, creator of the Keras Deep Learning framework, believes that LLMs will not get us to AGI. He attributes the success of LLMs to memorization, and believes that LLMs are not very intelligent.
If this was a claim made by Gary Marcus, we probably wouldn't have taken it very seriously. But when such a claim comes from Francois Chollet, creator of one of the most popular Deep Learning frameworks, that's a different story.
Even more remarkable: he published a paper titled On the Measure of Intelligence (2019) where he introduced the Abstraction and Reasoning Corpus (ARC) benchmark for progress towards Artificial General Intelligence (AGI). This was a few years before the first LLMs stormed into the scene. The benchmark was intentionally constructed to resist memorization, and to this date, the SOTA on this benchmark is at 34 % accuracy, whereas an average human easily scores 85%.
LLMs have rapidly saturated other benchmarks like MMLU and GSM 8K, but not ARC. Performance of both symbolic and Deep Learning are similarly poor on this benchmark.
Just yesterday, Chollet and Mark Knoop (founder of Zapier), announced the $1M ARC Prize. They are hosting the competition on Kaggle and anyone who beats the current SOTA can get a portion of that prize money.
In the meetup event, we want to discuss the ARC benchmark, the memorization vs. intelligence paradigm, and ponder about the reasons LLMs perform so poorly on this benchmark. Our discussion may also touch on the following topics:
- Chollet's view on LLMs and whether there is any merit in them
- Whether the ARC benchmark is a good test for AGI
- The current SOTA approaches on this dataset (both Discrete Program Generation and LLM based approaches)
- Other approaches that could be interesting for this dataset
We will meet around 18:30. We will spend the first half an hour socializing with pizza and drinks. Then, from 19:00 onwards, we will start discussing.
The end time is open ended. We have put 21:30 tentatively, but the space allows us to discuss and network longer if we wish to.
There are no speakers in this meetup. We sit around a table as equals and discuss/brainstorm together. Please try to read the paper before joining, so that our discussion can go deep.
Looking forward to seeing you there!
Additional Resources