DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

Item #:

075280-3059

Details

DOI: https://doi.org/10.52202/075280-3059
Author(s): Xuanyi Dong, Nan Du, Quoc V Le, Percy Liang, Hanxiao Liu, Yifeng Lu, Tengyu Ma, Hieu Pham, Sang Michael Xie, Adams Wei Yu
Pages: 69798-69818 (21 pages)
Format: PDF Paper Download
Conference: Advances in Neural Information Processing Systems 36
Date/Location: Held 10-16 December 2023, New Orleans, Louisiana, USA.
Series: Advances in Neural Information Processing Systems 36
Publisher: Neural Information Processing Systems Foundation, Inc. (NeurIPS)

DOI: https://doi.org/10.52202/075280-3059
Author(s): Xuanyi Dong, Nan Du, Quoc V Le, Percy Liang, Hanxiao Liu, Yifeng Lu, Tengyu Ma, Hieu Pham, Sang Michael Xie, Adams Wei Yu
Pages: 69798-69818 (21 pages)
Format: PDF Paper Download
Conference: Advances in Neural Information Processing Systems 36
Date/Location: Held 10-16 December 2023, New Orleans, Louisiana, USA.
Series: Advances in Neural Information Processing Systems 36
Publisher: Neural Information Processing Systems Foundation, Inc. (NeurIPS)