Actions
  • shareshare
  • link
  • cite
  • add
add
auto_awesome_motion View all 2 versions
Research data . Dataset . 2019

Magi Practical Web Article Corpus

Ji, Yichao; Liu, Xinyang; Ma, Kui; Zhao, Xuezhi; Sun, Qiao;
Open Access
Chinese
Published: 01 Jun 2019
Publisher: Zenodo
Abstract
Magi Practical Web Article Corpus (MPWAC) contains 10 million Chinese web articles consisting of more than 10 billion words. Articles have been shuffled to mitigate the negative influence on mini-batch based training. Each article has been refined such that only the main body of the text is left, without advertisements or other noises. This corpus is extracted in May 2019, by magi.com crawlers located inside Mainland China. The merged file should be processed with Gzip stream. It is encoded in UTF-8, and has the format of '0xFE [URL] 0xFF [TEXT] ...'.
Subjects

Magi, Natural Language Processing, Information Retrieval, Web Crawl, Corpus, Chinese, NLP

Related to Research communities
Digital Humanities and Cultural Heritage
moresidebar