Africa-Papers Opens 10-Billion-Token Research Corpus For AI

A new open research corpus has assembled more than 844,000 African and Africa-related academic papers into a dataset containing approximately 10.15 billion tokens for language-model development and analysis.

Africa-Papers was introduced in an arXiv preprint by researcher Mich-Seth Owusu and released through Hugging Face. The project draws from sources including journals indexed by the Directory of Open Access Journals, CGIAR's CGSpace repository, arXiv material and OpenAlex affiliation data.

The release addresses a persistent weakness in large language models: African scholarship is frequently underrepresented in the material used for training and evaluation. The paper estimates that it accounts for less than 0.5 per cent of global pretraining mixtures. That figure is the author's estimate, not an independently audited measure of every major model, but the broader imbalance is well established.

The corpus spans eight subject groupings. Social sciences and economics account for about 24.4 per cent, medicine 23.5 per cent and agriculture 18 per cent. Engineering and earth sciences make up 14.8 per cent, computer science and AI 9.5 per cent, humanities 5.7 per cent, physical and mathematical sciences 3.2 per cent, with smaller multidisciplinary and legal shares.

This composition is valuable because Africa's research record is not limited to language data. Agricultural trials, health studies, economic analysis and environmental work can support search, evidence discovery and domain-specific models. A carefully designed system could help researchers find relevant work that is currently scattered across repositories and poorly indexed by general tools.

Scale alone does not establish quality. The paper is a preprint and has not completed peer review. It also contains a data-version inconsistency that users should note: the abstract and audited table report 844,807 full-text papers, while one contribution statement refers to 789,348 verified papers. The public dataset and documentation need to make clear which count corresponds to the released version.

Provenance and rights require close examination. Open access can permit reading without necessarily permitting every form of model training or redistribution. Dataset users should retain document-level source information, licences and removal mechanisms. Authors and repositories need a practical way to correct metadata or object when material has been included incorrectly.

Representation within Africa is another issue. The project includes journals from 31 African countries, but publication volume, digitisation and repository access vary substantially. English-language and better-funded institutions may dominate even in a continent-focused corpus. Country, language, gender and institutional analyses would help users understand what the dataset cannot represent.

Deduplication is equally important. The same paper may appear in a journal, institutional repository and indexing service. Repetition can distort model behaviour and evaluation. The release is divided into 879 Parquet shards, which makes large-scale use practical, but technical accessibility should be accompanied by transparent duplicate detection and quality flags.

Africa-Papers could become useful infrastructure if it supports reproducible research rather than merely larger training runs. Public benchmarks should test citation retrieval, factual attribution, multilingual access and performance across less represented disciplines. Models trained with the data should disclose the version used and compare results with strong baselines.

The project is an ambitious attempt to make African knowledge more visible in AI development. Its lasting value will depend on documentation, rights management and independent evaluation. A continent-focused corpus should not only contain African research; it should make the origins, gaps and limits of that research easier to see.

Sources: Africa-Papers preprint and dataset on Hugging Face.