Difference between revisions of "WikiData Import 2022-03-11"
Jump to navigation
Jump to search
| Line 4: | Line 4: | ||
see {{Link|target=QLever/script}} as discussed in [https://github.com/ad-freiburg/qlever/issues/562 QLever Issue #562] for the script which makes reproducing this attempt easier. | see {{Link|target=QLever/script}} as discussed in [https://github.com/ad-freiburg/qlever/issues/562 QLever Issue #562] for the script which makes reproducing this attempt easier. | ||
| + | |||
| + | = Environment/prerequisites = | ||
| + | >=64 GB RAM and docker environment (e.g. Ubuntu) | ||
| + | <source lang='bash' highlight='1'> | ||
| + | ./qlever -v -e | ||
| + | qlever version : 1.22 $ : 2022/02/11 16:49:17 $ | ||
| + | needed software | ||
| + | docker → /usr/bin/docker ✅ | ||
| + | top → /usr/bin/top ✅ | ||
| + | df → /usr/bin/df ✅ | ||
| + | jq → /usr/bin/jq ✅ | ||
| + | lsb_release → /usr/bin/lsb_release ✅ | ||
| + | free → /usr/bin/free ✅ | ||
| + | operating system | ||
| + | No LSB modules are available. | ||
| + | Distributor ID: Ubuntu | ||
| + | Description: Ubuntu 20.04.4 LTS | ||
| + | Release: 20.04 | ||
| + | Codename: focal | ||
| + | docker version | ||
| + | Docker version 20.10.13, build a224086 | ||
| + | memory | ||
| + | total used free shared buff/cache available | ||
| + | Mem: 125Gi 1,5Gi 30Gi 32Mi 93Gi 123Gi | ||
| + | Swap: 2,0Gi 0B 2,0Gi | ||
| + | diskspace | ||
| + | /dev/sdb5 116G 21G 90G 19% / | ||
| + | tmpfs 63G 0 63G 0% /dev/shm | ||
| + | /dev/sda1 3,6T 1,1T 2,4T 31% /hd/seel | ||
| + | /dev/sdb1 511M 4,0K 511M 1% /boot/efi | ||
| + | soft ulimit for files | ||
| + | 1048576 | ||
| + | </source> | ||
| + | |||
= Wikidata RDF Dump latest-all.ttl download (6h) = | = Wikidata RDF Dump latest-all.ttl download (6h) = | ||
<source lang='bash' highlight='1-2,12,15-16,26'> | <source lang='bash' highlight='1-2,12,15-16,26'> | ||
Revision as of 06:41, 12 March 2022
QLever trial
see https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md
see QLever/script as discussed in QLever Issue #562 for the script which makes reproducing this attempt easier.
Environment/prerequisites
>=64 GB RAM and docker environment (e.g. Ubuntu)
./qlever -v -e
qlever version : 1.22 $ : 2022/02/11 16:49:17 $
needed software
docker → /usr/bin/docker ✅
top → /usr/bin/top ✅
df → /usr/bin/df ✅
jq → /usr/bin/jq ✅
lsb_release → /usr/bin/lsb_release ✅
free → /usr/bin/free ✅
operating system
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
docker version
Docker version 20.10.13, build a224086
memory
total used free shared buff/cache available
Mem: 125Gi 1,5Gi 30Gi 32Mi 93Gi 123Gi
Swap: 2,0Gi 0B 2,0Gi
diskspace
/dev/sdb5 116G 21G 90G 19% /
tmpfs 63G 0 63G 0% /dev/shm
/dev/sda1 3,6T 1,1T 2,4T 31% /hd/seel
/dev/sdb1 511M 4,0K 511M 1% /boot/efi
soft ulimit for files
1048576
Wikidata RDF Dump latest-all.ttl download (6h)
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2;date
Fr 11. Mär 18:50:39 CET 2022
--2022-03-11 18:50:39-- https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93072933618 (87G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’
latest-all.ttl.bz2 100%[============================================>] 86,68G 4,47MB/s in 5h 56m
2022-03-12 00:47:08 (4,15 MB/s) - ‘latest-all.ttl.bz2’ saved [93072933618/93072933618]
Sa 12. Mär 00:47:08 CET 2022
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2;date
Sa 12. Mär 06:35:39 CET 2022
--2022-03-12 06:35:39-- https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 319665811 (305M) [application/octet-stream]
Saving to: ‘latest-lexemes.ttl.bz2’
latest-lexemes.ttl.bz2 100%[============================================>] 304,86M 4,43MB/s in 69s
2022-03-12 06:36:49 (4,42 MB/s) - ‘latest-lexemes.ttl.bz2’ saved [319665811/319665811]
Sa 12. Mär 06:36:49 CET 2022
ls -l
total 91203728
-rw-rw-r-- 1 wf wf 93072933618 Mär 10 04:59 latest-all.ttl.bz2
-rw-rw-r-- 1 wf wf 319665811 Mär 12 00:27 latest-lexemes.ttl.bz2