WikiData Import 2022-03-11
QLever trial
see https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md
see QLever/script as discussed in QLever Issue #562 for the script which makes reproducing this attempt easier.
Environment/prerequisites
>=64 GB RAM and docker environment (e.g. Ubuntu) >1 TB diskspace (SSD preferred for speed)
./qlever -v -e
qlever version : 1.23 $ : 2022/03/12 05:56:20 $
needed software
docker → /usr/bin/docker ✅
top → /usr/bin/top ✅
df → /usr/bin/df ✅
jq → /usr/bin/jq ✅
lsb_release → /usr/bin/lsb_release ✅
free → /usr/bin/free ✅
operating system
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
docker version
Docker version 20.10.13, build a224086
memory
total used free shared buff/cache available
Mem: 125Gi 1,5Gi 30Gi 32Mi 93Gi 123Gi
Swap: 2,0Gi 0B 2,0Gi
diskspace
/dev/sdb5 116G 21G 90G 19% /
tmpfs 63G 0 63G 0% /dev/shm
/dev/sda1 3,6T 1,1T 2,4T 31% /hd/seel
/dev/sdb1 511M 4,0K 511M 1% /boot/efi
soft ulimit for files
1048576
Wikidata RDF Dump latest-all.ttl download (6h)
mkdir -p qlever-indices/wikidata
cd qlever-indices/wikidata
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2;date
Fr 11. Mär 18:50:39 CET 2022
--2022-03-11 18:50:39-- https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93072933618 (87G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’
latest-all.ttl.bz2 100%[============================================>] 86,68G 4,47MB/s in 5h 56m
2022-03-12 00:47:08 (4,15 MB/s) - ‘latest-all.ttl.bz2’ saved [93072933618/93072933618]
Sa 12. Mär 00:47:08 CET 2022
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2;date
Sa 12. Mär 06:35:39 CET 2022
--2022-03-12 06:35:39-- https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 319665811 (305M) [application/octet-stream]
Saving to: ‘latest-lexemes.ttl.bz2’
latest-lexemes.ttl.bz2 100%[============================================>] 304,86M 4,43MB/s in 69s
2022-03-12 06:36:49 (4,42 MB/s) - ‘latest-lexemes.ttl.bz2’ saved [319665811/319665811]
Sa 12. Mär 06:36:49 CET 2022
ls -l
total 91203728
-rw-rw-r-- 1 wf wf 93072933618 Mär 10 04:59 latest-all.ttl.bz2
-rw-rw-r-- 1 wf wf 319665811 Mär 12 00:27 latest-lexemes.ttl.bz2
du -sm *
88762 latest-all.ttl.bz2
305 latest-lexemes.ttl.bz2
With QLever Script
make sure you run this to copy the settings
./qlever --wikidata_download
qlever-indices/wikidata already exists
wikidata.settings.json already copied to qlever-indices/wikidata
wikidata lexemes:latest-lexemes.ttl.bz2 already downloaded
wikidata dump:latest-all.ttl.bz2 already downloaded