Direct Download Guide for English Wikipedia Database Dumps
Direct Download Guide for English Wikipedia Database Dumps
The official raw English Wikipedia database snapshots and pre-packaged archives are hosted directly on the Wikimedia Foundation servers.
Because the English Wikipedia is significantly larger than other language editions (over 6.8 million articles), file sizes are substantially bigger.
1. Official Wikimedia Directory
All raw database dumps for English Wikipedia are accessible from the main server directory.
- Main Directory Link: https://dumps.wikimedia.org/enwiki/latest/
- This directory automatically points to the most recent completed snapshot folder.
2. Primary XML Files to Download
Inside the directory, files are compressed in .bz2 format. Choose from the following files depending on your project needs:
Complete Articles with Index (Multistream)
- File Name: enwiki-latest-pages-articles-multistream.xml.bz2
- File Size: Approximately 25 GB to 27 GB (compressed) | Decompresses to over 100 GB.
- Best Used For: Most offline readers, search engines, and data processing tools. It includes a companion index file (enwiki-latest-pages-articles-multistream-index.txt.bz2) that allows tools to look up and extract individual articles without uncompressing the entire file.
Complete Articles (Standard Sequential Stream)
- File Name: enwiki-latest-pages-articles.xml.bz2
- File Size: Approximately 20 GB to 22 GB (compressed).
- Best Used For: Sequential data pipelines or custom parsing scripts that read the XML data linearly from start to finish.
Full Current Snapshot (Articles, Talk Pages, and User Pages)
- File Name: enwiki-latest-pages-meta-current.xml.bz2
- File Size: Approximately 30 GB to 35 GB (compressed).
- Best Used For: Researching talk pages, administrative metadata, and project discussions alongside article content.
3. Alternative Pre-Packaged ZIM Archives
For pre-indexed .zim archives designed for offline readers (like Kiwix) rather than raw XML database dumps, download them directly from the Kiwix directory:
- Kiwix English ZIM Directory: https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/
- Full Edition (With Images): Look for files starting with wikipedia_en_all_maxi_ (~115 GB).
- Text-Only Edition (No Images): Look for files starting with wikipedia_en_all_nopic_ (~45 GB to 50 GB).
- Mini Edition (Summaries Only): Look for files starting with wikipedia_en_all_mini_ (~12 GB).
4. Download Recommendations
- Use Torrents for Large Dumps: Due to the massive size of English dumps (25+ GB compressed), direct HTTP browser downloads can easily time out or corrupt. Wikimedia offers official BitTorrent files for raw dumps at https://meta.wikimedia.org/wiki/Data_dump_torrents.
- Flash Drive Storage: Ensure your USB flash drive is formatted as exFAT or NTFS. The older FAT32 format will fail because it cannot hold any single file larger than 4 GB.
Comments
Post a Comment