The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Hugo Laurençon
(1)
,
Lucile Saulnier
(1)
,
Thomas Wang
(1)
,
Christopher Akiki
(2)
,
Albert Villanova del Moral
(1)
,
Teven Le Scao
(1)
,
Leandro von Werra
(1)
,
Chenghao Mou
(3)
,
Eduardo González Ponferrada
(4)
,
Huu Nguyen
(5)
,
Jörg Frohberg
(6)
,
Mario Šaško
(1)
,
Quentin Lhoest
(1)
,
Angelina Mcmillan-Major
(1, 7)
,
Gérard Dupont
(8)
,
Stella Biderman
(9, 10)
,
Anna Rogers
(11)
,
Loubna Ben Allal
(1)
,
Francesco de Toni
(12)
,
Giada Pistilli
(1)
,
Olivier Nguyen
(13)
,
Somaieh Nikpoor
(14)
,
Maraim Masoud
(3)
,
Pierre Colombo
(15)
,
Javier de la Rosa
(16)
,
Paulo Villegas
(17)
,
Tristan Thrush
(1)
,
Shayne Longpre
(18)
,
Sebastian Nagel
(19)
,
Leon Weber
(20)
,
Manuel Romero Muñoz
(21)
,
Jian Zhu
(22)
,
Daniel van Strien
(23)
,
Zaid Alyafeai
(24)
,
Khalid Almubarak
(25)
,
Vu Minh Chien
(26)
,
Itziar Gonzalez-Dios
(27)
,
Aitor Soroa
(27)
,
Kyle Lo
(28)
,
Manan Dey
(29)
,
Pedro Ortiz Suarez
(30)
,
Aaron Gokaslan
(31)
,
Shamik Bose
(3)
,
David Ifeoluwa Adelani
(32)
,
Long Phan
(33)
,
Hieu Tran
(33)
,
Ian Yu
(34)
,
Suhas Pai
(35)
,
Jenny Chim
(36)
,
Violette Lepercq
(1)
,
Suzana Ilić
(1)
,
Margaret Mitchell
(1)
,
Sasha Luccioni
(1)
,
Yacine Jernite
(1)
1
Hugging Face
2 Leipzig University / Universität Leipzig
3 Independent Researcher
4 Ferrum Health
5 Ontocord.ai
6 Apergo.ai
7 University of Washington [Seattle]
8 Mavenoid
9 Booz Hallen Hamilton Inc
10 EleutherAI
11 UCPH - University of Copenhagen = Københavns Universitet
12 UWA - The University of Western Australia
13 ServiceNow
14 CAIDP - Center for AI and Digital Policy
15 CentraleSupélec
16 National Library of Norway
17 Telefonica I+D - Telefonica Investigación y Desarrollo
18 MIT - Massachusetts Institute of Technology
19 Common Crawl
20 HU Berlin - Humboldt-Universität zu Berlin = Humboldt University of Berlin = Université Humboldt de Berlin
21 Narrativa
22 University of Michigan [Ann Arbor]
23 British Library
24 KFUPM - King Fahd University of Petroleum and Minerals
25 PSAU - Prince Sattam Bin Abdul-Aziz University
26 Detomo Inc.
27 UPV / EHU - Universidad del País Vasco [Espainia] / Euskal Herriko Unibertsitatea [España] = University of the Basque Country [Spain] = Université du pays basque [Espagne]
28 AI2 - Allen Institute for Artificial Intelligence
29 SAP
30 DWS - Data and Web Science Group [Mannheim Univ]
31 Cornell University [New York]
32 Universität des Saarlandes [Saarbrücken] = Saarland University [Saarbrücken]
33 VietAI Research
34 Aggregate Intellect
35 Bedrock AI
36 QMUL - Queen Mary University of London
2 Leipzig University / Universität Leipzig
3 Independent Researcher
4 Ferrum Health
5 Ontocord.ai
6 Apergo.ai
7 University of Washington [Seattle]
8 Mavenoid
9 Booz Hallen Hamilton Inc
10 EleutherAI
11 UCPH - University of Copenhagen = Københavns Universitet
12 UWA - The University of Western Australia
13 ServiceNow
14 CAIDP - Center for AI and Digital Policy
15 CentraleSupélec
16 National Library of Norway
17 Telefonica I+D - Telefonica Investigación y Desarrollo
18 MIT - Massachusetts Institute of Technology
19 Common Crawl
20 HU Berlin - Humboldt-Universität zu Berlin = Humboldt University of Berlin = Université Humboldt de Berlin
21 Narrativa
22 University of Michigan [Ann Arbor]
23 British Library
24 KFUPM - King Fahd University of Petroleum and Minerals
25 PSAU - Prince Sattam Bin Abdul-Aziz University
26 Detomo Inc.
27 UPV / EHU - Universidad del País Vasco [Espainia] / Euskal Herriko Unibertsitatea [España] = University of the Basque Country [Spain] = Université du pays basque [Espagne]
28 AI2 - Allen Institute for Artificial Intelligence
29 SAP
30 DWS - Data and Web Science Group [Mannheim Univ]
31 Cornell University [New York]
32 Universität des Saarlandes [Saarbrücken] = Saarland University [Saarbrücken]
33 VietAI Research
34 Aggregate Intellect
35 Bedrock AI
36 QMUL - Queen Mary University of London
Giada Pistilli
- Function : Author
- PersonId : 1221963
- IdHAL : giada-pistilli
- ORCID : 0000-0003-4941-0505
- IdRef : 279071655
Pierre Colombo
- Function : Author
- PersonId : 743646
- IdHAL : pierre-colombo
- IdRef : 258975059
Pedro Ortiz Suarez
- Function : Author
- PersonId : 178412
- IdHAL : pedro-ortiz-suarez
- ORCID : 0000-0003-0343-8852
- IdRef : 264210743
Abstract
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
Domains
Computation and Language [cs.CL]| Origin | Publisher files allowed on an open archive |
|---|
