Dive into the Transformation of LLM models into Mixture of Experts

Join us at the Canberra Deep Learning Meetup to delve into the transformation of LLM models into Mixture of Experts. This paper discussed the methodology of improving the dense models into sparse Mixture of Experts. This yields improvement in performance without much extra computational requirement.

Paper: https://arxiv.org/pdf/2410.07524