Testing the Quality of Synthetic Data With a Self-Training Approach by DeepMind

Fine-tuning language models (LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data.

In this paper, our guest speaker, Avi Singh, a research scientist from Google DeepMind, and his collaborators explore whether it's possible to go beyond human data on tasks where there is access to scalar feedback, for example, on math problems where one can verify correctness.

To do so, they investigate a simple self-training method based on expectation-maximization, which they call ReSTEM, where they
(1) generate samples from the model and filter them using binary feedback;
(2) fine-tune the model on these samples;
(3) repeat this process a few times.

Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, they found that ReSTEM scales favorably with model size and significantly surpasses fine-tuning only on human data.

Overall, their findings suggest self-training with feedback can substantially reduce dependence on human-generated data.