research data . Dataset . 2019

Magi Practical Web Article Corpus

Yichao Ji; Xinyang Liu; Kui Ma; Xuezhi Zhao; Qiao Sun;
Open Access Chinese
  • Published: 01 Jun 2019
  • Publisher: Zenodo
Abstract
Magi Practical Web Article Corpus (MPWAC) contains 10 million Chinese web articles consisting of more than 10 billion words. Articles have been shuffled to mitigate the negative influence on mini-batch based training. Each article has been refined such that only the main body of the text is left, without advertisements or other noises. This corpus is extracted in May 2019, by magi.com crawlers located inside Mainland China. The merged file should be processed with Gzip stream. It is encoded in UTF-8, and has the format of '0xFE [URL] 0xFF [TEXT] ...'.
Subjects
free text keywords: Magi, Natural Language Processing, Information Retrieval, Web Crawl, Corpus, Chinese, NLP
Communities
  • Digital Humanities and Cultural Heritage
Download fromView all 2 versions
Open Access
ZENODO
Dataset . 2019
Providers: ZENODO
1 research outcomes, page 1 of 1
Any information missing or wrong?Report an Issue