Skip to content

Paper Discussion: XLNet: Generalized Autoregressive Pretraining for Language

Photo of Cheng Yu
Hosted By
Cheng Y. and 2 others
Paper Discussion: XLNet: Generalized Autoregressive Pretraining for Language

Details

From the abstract:
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining ap- proaches based on autoregressive language modeling. However, relying on corrupt- ing the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.

Paper:
https://arxiv.org/pdf/1906.08237.pdf

If we have time, we will also compare XLNet, BERT with RoBERTa (sic)
https://arxiv.org/pdf/1907.11692.pdf

Recommendations:
Briefly read the paper or at least the abstract.
Bring a laptop for reference.

This event is sponsored by Trellis Data (http://www.trellisdata.com.au).

Photo of Canberra Deep Learning Meetup group
Canberra Deep Learning Meetup
See more events
C&Ma House
22 Napier Cl · Deakin, AC