Difference between revisions of "WikiData Import 2022-01-29"

From BITPlan Wiki
Jump to navigation Jump to search
(Created page with "<source lang='bash'> wf@merkur:~$ free -h total used free shared buff/cache available Mem: 62Gi 1,3Gi 60Gi 30...")
 
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
<source lang='bash'>
+
{{PageSequence|prev=|category=Wikidata|next=WikiData Import 2022-03-11|categoryIcon=cloud-download}}
 +
 
 +
 
 +
= QLever trial =
 +
{{Import
 +
|target=QLever
 +
|start=2022-02-02
 +
|end=2022-02-06
 +
|url=https://wiki.bitplan.com/index.php/WikiData_Import_2022-01-29
 +
|days=4
 +
|state=✅
 +
|os=Ubuntu 20.04.3 LTS
 +
|cpu=Quad-Core AMD Opteron(tm) Processor 2374 HE
 +
|ram=64
 +
|triples=16.9
 +
|storemode=property
 +
}}
 +
 
 +
This attempt was successful ✅ after copying the index result to a machine with 128 GB for running.
 +
 
 +
see https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md
 +
 
 +
see {{Link|target=QLever/script}} as discussed in [https://github.com/ad-freiburg/qlever/issues/562 QLever Issue #562] for the script which makes reproducing this attempt easier.
 +
 
 +
== Environment ==
 +
=== Operating System ===
 +
<source lang='bash' highlight='1'>
 +
lsb_release -a
 +
No LSB modules are available.
 +
Distributor ID: Ubuntu
 +
Description: Ubuntu 20.04.3 LTS
 +
Release: 20.04
 +
Codename: focal
 +
</source>
 +
=== docker version ===
 +
<source lang='bash'highlight='1'>
 +
docker --version
 +
Docker version 19.03.13, build 4484c46d9d
 +
</source>
 +
=== Memory ===
 +
<source lang='bash' highlight='1'>
 
wf@merkur:~$ free -h
 
wf@merkur:~$ free -h
 
               total        used        free      shared  buff/cache  available
 
               total        used        free      shared  buff/cache  available
Line 6: Line 46:
  
 
</source>
 
</source>
 +
=== diskspace ===
 +
<pre>
 +
vendor              | model              | size      | days  | years |
 +
----------------------+--------------------+-----------+-------+-------+
 +
  ST6000VX0023-2EF110 |                    |    6,00TB |  300 |  0.8 |
 +
              SanDisk |    SSD PLUS 120GB |    120GB |  237 |  0.6 |
 +
                  WDC |  WD20EARS-00MVWB1 |    2,00TB |  2232 |  6.1 |
 +
  ST8000DM004-2U9188 |                    |    8,00TB |    8 |  0.0 |
 +
 +
Model Family:    Seagate Barracuda Compute
 +
Device Model:    ST8000DM004-2U9188
 +
Firmware Version: 0001
 +
User Capacity:    8.001.563.222.016 bytes [8,00 TB]
 +
Sector Sizes:    512 bytes logical, 4096 bytes physical
 +
Rotation Rate:    5400 rpm
 +
Form Factor:      3.5 inches
 +
</pre>
 +
 +
= Steps =
 +
The steps below maybe automated with the {{Link|target=QLever/script}}
 +
== git clone (1 min) ==
 +
<source lang='bash' highlight='1-2'>
 +
export QLEVER_HOME=$(pwd)
 +
date;git clone --recursive https://github.com/ad-freiburg/qlever qlever-code;date
 +
Sa 29. Jan 17:12:45 CET 2022
 +
Cloning into 'qlever-code'...
 +
remote: Enumerating objects: 12917, done.
 +
remote: Counting objects: 100% (622/622), done.
 +
remote: Compressing objects: 100% (514/514), done.
 +
remote: Total 12917 (delta 382), reused 206 (delta 103), pack-reused 12295
 +
Receiving objects: 100% (12917/12917), 111.10 MiB | 6.81 MiB/s, done.
 +
Resolving deltas: 100% (10067/10067), done.
 +
Submodule 'third_party/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'third_party/abseil-cpp'
 +
Submodule 'third_party/antlr4' (https://github.com/antlr/antlr4.git) registered for path 'third_party/antlr4'
 +
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
 +
Submodule 'third_party/re2' (https://github.com/google/re2.git) registered for path 'third_party/re2'
 +
Submodule 'third_party/stxxl' (https://github.com/ad-freiburg/stxxl) registered for path 'third_party/stxxl'
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/abseil-cpp'...
 +
remote: Enumerating objects: 16122, done.       
 +
remote: Counting objects: 100% (248/248), done.       
 +
remote: Compressing objects: 100% (193/193), done.       
 +
remote: Total 16122 (delta 116), reused 106 (delta 55), pack-reused 15874       
 +
Receiving objects: 100% (16122/16122), 10.39 MiB | 6.85 MiB/s, done.
 +
Resolving deltas: 100% (12398/12398), done.
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/antlr4'...
 +
remote: Enumerating objects: 125326, done.       
 +
remote: Counting objects: 100% (2734/2734), done.       
 +
remote: Compressing objects: 100% (1022/1022), done.       
 +
remote: Total 125326 (delta 1488), reused 2292 (delta 1246), pack-reused 122592       
 +
Receiving objects: 100% (125326/125326), 64.11 MiB | 6.72 MiB/s, done.
 +
Resolving deltas: 100% (73647/73647), done.
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/googletest'...
 +
remote: Enumerating objects: 23795, done.       
 +
remote: Counting objects: 100% (259/259), done.       
 +
remote: Compressing objects: 100% (146/146), done.       
 +
remote: Total 23795 (delta 134), reused 200 (delta 105), pack-reused 23536       
 +
Receiving objects: 100% (23795/23795), 9.90 MiB | 6.89 MiB/s, done.
 +
Resolving deltas: 100% (17517/17517), done.
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/re2'...
 +
remote: Enumerating objects: 6996, done.       
 +
remote: Counting objects: 100% (617/617), done.       
 +
remote: Compressing objects: 100% (388/388), done.       
 +
remote: Total 6996 (delta 392), reused 399 (delta 223), pack-reused 6379       
 +
Receiving objects: 100% (6996/6996), 3.49 MiB | 6.94 MiB/s, done.
 +
Resolving deltas: 100% (5287/5287), done.
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl'...
 +
remote: Enumerating objects: 40982, done.       
 +
remote: Counting objects: 100% (45/45), done.       
 +
remote: Compressing objects: 100% (31/31), done.       
 +
remote: Total 40982 (delta 15), reused 28 (delta 7), pack-reused 40937       
 +
Receiving objects: 100% (40982/40982), 14.13 MiB | 6.77 MiB/s, done.
 +
Resolving deltas: 100% (30914/30914), done.
 +
Submodule path 'third_party/abseil-cpp': checked out 'b9b925341f9e90f5e7aa0cf23f036c29c7e454eb'
 +
Submodule path 'third_party/antlr4': checked out 'e4c1a74c66bd5290364ea2b36c97cd724b247357'
 +
Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
 +
Submodule path 'third_party/re2': checked out '13ebb377c6ad763ca61d12dd6f88b1126bd0b911'
 +
Submodule path 'third_party/stxxl': checked out 'a4f884f2a2b4ea078c34c48e1e9a0003f4619f00'
 +
Submodule 'extlib/foxxll' (https://github.com/ad-freiburg/foxxll.git) registered for path 'third_party/stxxl/extlib/foxxll'
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl/extlib/foxxll'...
 +
remote: Enumerating objects: 21414, done.       
 +
remote: Counting objects: 100% (28/28), done.       
 +
remote: Compressing objects: 100% (22/22), done.       
 +
remote: Total 21414 (delta 9), reused 13 (delta 4), pack-reused 21386       
 +
Receiving objects: 100% (21414/21414), 4.60 MiB | 2.48 MiB/s, done.
 +
Resolving deltas: 100% (15789/15789), done.
 +
Submodule path 'third_party/stxxl/extlib/foxxll': checked out '8cbca7bedcdb0b84a6de99e927c5fa27a4bbbfb2'
 +
Submodule 'extlib/tlx' (https://github.com/joka921/tlx.git) registered for path 'third_party/stxxl/extlib/foxxll/extlib/tlx'
 +
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl/extlib/foxxll/extlib/tlx'...
 +
remote: Enumerating objects: 3418, done.       
 +
remote: Counting objects: 100% (53/53), done.       
 +
remote: Compressing objects: 100% (44/44), done.       
 +
remote: Total 3418 (delta 24), reused 22 (delta 9), pack-reused 3365       
 +
Receiving objects: 100% (3418/3418), 1.11 MiB | 2.75 MiB/s, done.
 +
Resolving deltas: 100% (2611/2611), done.
 +
Submodule path 'third_party/stxxl/extlib/foxxll/extlib/tlx': checked out 'ef81a598d9880cc7d242afc47de7328634f07f1d'
 +
Sa 29. Jan 17:13:49 CET 2022
 +
</source>
 +
== docker build (15 mins) ==
 +
<source lang='bash' highlight='1-2'>
 +
cd qlever-code/
 +
wf@merkur:/hd/jurob/qlever/qlever-code$ date;sudo docker build --file Dockerfiles/Dockerfile.Ubuntu20.04 -t qlever .;date
 +
Sa 29. Jan 17:14:51 CET 2022
 +
Sending build context to Docker daemon  126.1MB
 +
Step 1/43 : FROM ubuntu:20.04 as base
 +
...
 +
Successfully tagged qlever:latest
 +
Sa 29. Jan 17:29:27 CET 2022
 +
</source>
 +
 +
=== Wikidata dump Download (6h) ===
 +
This had been done on another machine a few days earlier.
 +
<source lang='bash' highlight='1-2,12,15-16,28'>
 +
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2;date
 +
Thu Jan 27 11:08:10 CET 2022
 +
--2022-01-27 11:08:10--  https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2
 +
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620::861:1:208:80:154:7, 208.80.154.7
 +
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620::861:1:208:80:154:7|:443... connected.
 +
HTTP request sent, awaiting response... 200 OK
 +
Length: 92422961847 (86G) [application/octet-stream]
 +
Saving to: ‘latest-all.ttl.bz2’
 +
 +
latest-all.ttl.bz2  100%[===================>]  86.08G  4.60MB/s    in 5h 46m 
 +
 +
2022-01-27 16:55:06 (4.23 MB/s) - ‘latest-all.ttl.bz2’ saved [92422961847/92422961847]
 +
 +
Thu Jan 27 16:55:06 CET 2022
 +
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2;date
 +
Thu Jan 27 17:36:38 CET 2022
 +
--2022-01-27 17:36:38--  https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2
 +
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620::861:1:208:80:154:7, 208.80.154.7
 +
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620::861:1:208:80:154:7|:443... connected.
 +
HTTP request sent, awaiting response... 200 OK
 +
Length: 315591211 (301M) [application/octet-stream]
 +
Saving to: ‘latest-lexemes.ttl.bz2’
 +
 +
latest-lexemes.ttl. 100%[===================>] 300.97M  4.95MB/s    in 60s   
 +
 +
2022-01-27 17:37:39 (5.00 MB/s) - ‘latest-lexemes.ttl.bz2’ saved [315591211/315591211]
 +
 +
Thu Jan 27 17:37:39 CET 2022
 +
 +
</source>
 +
== Pull QLever ==
 +
<source lang='bash' highlight='1'>
 +
./qlever -p --wikidata_index
 +
pulling qlever docker image started at Sa 12. Mär 06:44:34 CET 2022
 +
Using default tag: latest
 +
latest: Pulling from adfreiburg/qlever
 +
ae6e1b672be6: Already exists
 +
55c87480fb23: Already exists
 +
37363fee5023: Already exists
 +
0aef3ecd8c53: Already exists
 +
d12c5fe208ef: Already exists
 +
0b69b03c02c1: Already exists
 +
5bca5e9bf0ae: Pull complete
 +
532aa782a906: Pull complete
 +
Digest: sha256:387848d3cf1a3eea56c6d21bdcea18189b950740dd5410356932d7f0cbc2c20a
 +
Status: Downloaded newer image for adfreiburg/qlever:latest
 +
docker.io/adfreiburg/qlever:latest
 +
pulling qlever docker image finished at Sa 12. Mär 06:44:51 CET 2022 after 17 seconds
 +
</source>
 +
 +
=== Indexing 4.1 days ===
 +
<source lang='bash' highlight='1'>
 +
./qlever --wikidata_index
 +
2022-02-02 05:53:42.391 - INFO:  �[1mQLever IndexBuilder, compiled on Jan 29 2022 20:19:12�[22m
 +
2022-02-02 05:53:42.418 - INFO:  You specified the input format: TTL
 +
2022-02-02 05:53:42.566 - INFO:  You specified "locale = en_US" and "ignore-punctuation = 1"
 +
2022-02-02 05:53:42.568 - INFO:  You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files (see qlever/docs on GitHub)
 +
2022-02-02 05:53:42.568 - INFO:  You specified "num-triples-per-partial-vocab = 20,000,000", choose a lower value if the index builder runs out of memory
 +
2022-02-02 05:53:42.569 - INFO:  Processing input triples from /dev/stdin ...
 +
2022-02-02 05:59:07.200 - INFO:  Input triples processed: 100,000,000
 +
2022-02-02 06:04:25.068 - INFO:  Input triples processed: 200,000,000
 +
2022-02-02 06:08:49.023 - INFO:  Input triples processed: 300,000,000
 +
2022-02-02 06:13:42.180 - INFO:  Input triples processed: 400,000,000
 +
2022-02-02 06:18:35.785 - INFO:  Input triples processed: 500,000,000
 +
2022-02-02 06:23:33.098 - INFO:  Input triples processed: 600,000,000
 +
2022-02-02 06:28:40.453 - INFO:  Input triples processed: 700,000,000
 +
2022-02-02 06:33:33.224 - INFO:  Input triples processed: 800,000,000
 +
2022-02-02 06:38:29.646 - INFO:  Input triples processed: 900,000,000
 +
2022-02-02 06:43:31.615 - INFO:  Input triples processed: 1,000,000,000
 +
2022-02-02 06:48:50.864 - INFO:  Input triples processed: 1,100,000,000
 +
2022-02-02 06:54:44.529 - INFO:  Input triples processed: 1,200,000,000
 +
2022-02-02 07:01:03.669 - INFO:  Input triples processed: 1,300,000,000
 +
2022-02-02 07:06:57.283 - INFO:  Input triples processed: 1,400,000,000
 +
2022-02-02 07:13:02.002 - INFO:  Input triples processed: 1,500,000,000
 +
2022-02-02 07:19:02.355 - INFO:  Input triples processed: 1,600,000,000
 +
2022-02-02 07:25:32.057 - INFO:  Input triples processed: 1,700,000,000
 +
2022-02-02 07:31:41.109 - INFO:  Input triples processed: 1,800,000,000
 +
2022-02-02 07:37:39.025 - INFO:  Input triples processed: 1,900,000,000
 +
2022-02-02 07:43:24.237 - INFO:  Input triples processed: 2,000,000,000
 +
2022-02-02 07:48:33.112 - INFO:  Input triples processed: 2,100,000,000
 +
2022-02-02 07:54:15.348 - INFO:  Input triples processed: 2,200,000,000
 +
2022-02-02 07:59:55.743 - INFO:  Input triples processed: 2,300,000,000
 +
2022-02-02 08:05:09.401 - INFO:  Input triples processed: 2,400,000,000
 +
2022-02-02 08:10:38.497 - INFO:  Input triples processed: 2,500,000,000
 +
2022-02-02 08:16:27.476 - INFO:  Input triples processed: 2,600,000,000
 +
2022-02-02 08:21:52.338 - INFO:  Input triples processed: 2,700,000,000
 +
2022-02-02 08:27:46.142 - INFO:  Input triples processed: 2,800,000,000
 +
2022-02-02 08:33:28.374 - INFO:  Input triples processed: 2,900,000,000
 +
2022-02-02 08:39:30.528 - INFO:  Input triples processed: 3,000,000,000
 +
2022-02-02 08:45:08.281 - INFO:  Input triples processed: 3,100,000,000
 +
2022-02-02 08:50:52.437 - INFO:  Input triples processed: 3,200,000,000
 +
2022-02-02 09:03:13.800 - INFO:  Input triples processed: 3,300,000,000
 +
2022-02-02 09:08:56.368 - INFO:  Input triples processed: 3,400,000,000
 +
2022-02-02 09:14:37.689 - INFO:  Input triples processed: 3,500,000,000
 +
2022-02-02 09:20:14.816 - INFO:  Input triples processed: 3,600,000,000
 +
2022-02-02 09:25:46.666 - INFO:  Input triples processed: 3,700,000,000
 +
2022-02-02 09:31:28.112 - INFO:  Input triples processed: 3,800,000,000
 +
2022-02-02 09:36:55.886 - INFO:  Input triples processed: 3,900,000,000
 +
2022-02-02 09:42:28.592 - INFO:  Input triples processed: 4,000,000,000
 +
2022-02-02 09:48:02.848 - INFO:  Input triples processed: 4,100,000,000
 +
2022-02-02 09:53:16.606 - INFO:  Input triples processed: 4,200,000,000
 +
2022-02-02 09:58:40.663 - INFO:  Input triples processed: 4,300,000,000
 +
2022-02-02 10:04:17.846 - INFO:  Input triples processed: 4,400,000,000
 +
2022-02-02 10:09:21.045 - INFO:  Input triples processed: 4,500,000,000
 +
2022-02-02 10:14:34.149 - INFO:  Input triples processed: 4,600,000,000
 +
2022-02-02 10:19:51.519 - INFO:  Input triples processed: 4,700,000,000
 +
2022-02-02 10:25:54.075 - INFO:  Input triples processed: 4,800,000,000
 +
2022-02-02 10:31:12.738 - INFO:  Input triples processed: 4,900,000,000
 +
2022-02-02 10:36:28.453 - INFO:  Input triples processed: 5,000,000,000
 +
2022-02-02 10:41:58.489 - INFO:  Input triples processed: 5,100,000,000
 +
2022-02-02 10:47:19.305 - INFO:  Input triples processed: 5,200,000,000
 +
2022-02-02 10:53:03.096 - INFO:  Input triples processed: 5,300,000,000
 +
2022-02-02 10:58:30.772 - INFO:  Input triples processed: 5,400,000,000
 +
2022-02-02 11:03:53.039 - INFO:  Input triples processed: 5,500,000,000
 +
2022-02-02 11:09:07.835 - INFO:  Input triples processed: 5,600,000,000
 +
2022-02-02 11:14:43.148 - INFO:  Input triples processed: 5,700,000,000
 +
2022-02-02 11:20:30.355 - INFO:  Input triples processed: 5,800,000,000
 +
2022-02-02 11:26:12.168 - INFO:  Input triples processed: 5,900,000,000
 +
2022-02-02 11:31:49.249 - INFO:  Input triples processed: 6,000,000,000
 +
2022-02-02 11:37:42.195 - INFO:  Input triples processed: 6,100,000,000
 +
2022-02-02 11:43:14.152 - INFO:  Input triples processed: 6,200,000,000
 +
2022-02-02 11:48:56.857 - INFO:  Input triples processed: 6,300,000,000
 +
2022-02-02 11:54:14.783 - INFO:  Input triples processed: 6,400,000,000
 +
2022-02-02 11:59:44.683 - INFO:  Input triples processed: 6,500,000,000
 +
2022-02-02 12:04:57.135 - INFO:  Input triples processed: 6,600,000,000
 +
2022-02-02 12:10:12.283 - INFO:  Input triples processed: 6,700,000,000
 +
2022-02-02 12:15:35.776 - INFO:  Input triples processed: 6,800,000,000
 +
2022-02-02 12:20:38.895 - INFO:  Input triples processed: 6,900,000,000
 +
2022-02-02 12:26:11.903 - INFO:  Input triples processed: 7,000,000,000
 +
2022-02-02 12:31:24.482 - INFO:  Input triples processed: 7,100,000,000
 +
2022-02-02 12:36:43.304 - INFO:  Input triples processed: 7,200,000,000
 +
2022-02-02 12:42:26.933 - INFO:  Input triples processed: 7,300,000,000
 +
2022-02-02 12:47:32.521 - INFO:  Input triples processed: 7,400,000,000
 +
2022-02-02 12:53:06.737 - INFO:  Input triples processed: 7,500,000,000
 +
2022-02-02 12:58:50.177 - INFO:  Input triples processed: 7,600,000,000
 +
2022-02-02 13:04:13.764 - INFO:  Input triples processed: 7,700,000,000
 +
2022-02-02 13:09:53.474 - INFO:  Input triples processed: 7,800,000,000
 +
2022-02-02 13:15:30.449 - INFO:  Input triples processed: 7,900,000,000
 +
2022-02-02 13:21:18.666 - INFO:  Input triples processed: 8,000,000,000
 +
2022-02-02 13:27:02.409 - INFO:  Input triples processed: 8,100,000,000
 +
2022-02-02 13:32:37.830 - INFO:  Input triples processed: 8,200,000,000
 +
2022-02-02 13:38:08.812 - INFO:  Input triples processed: 8,300,000,000
 +
2022-02-02 13:43:19.219 - INFO:  Input triples processed: 8,400,000,000
 +
2022-02-02 13:48:43.536 - INFO:  Input triples processed: 8,500,000,000
 +
2022-02-02 13:54:49.380 - INFO:  Input triples processed: 8,600,000,000
 +
2022-02-02 14:00:03.529 - INFO:  Input triples processed: 8,700,000,000
 +
2022-02-02 14:05:07.486 - INFO:  Input triples processed: 8,800,000,000
 +
2022-02-02 14:10:24.776 - INFO:  Input triples processed: 8,900,000,000
 +
2022-02-02 14:16:03.699 - INFO:  Input triples processed: 9,000,000,000
 +
2022-02-02 14:21:17.048 - INFO:  Input triples processed: 9,100,000,000
 +
2022-02-02 14:26:38.329 - INFO:  Input triples processed: 9,200,000,000
 +
2022-02-02 14:32:07.014 - INFO:  Input triples processed: 9,300,000,000
 +
2022-02-02 14:37:35.128 - INFO:  Input triples processed: 9,400,000,000
 +
2022-02-02 14:42:42.646 - INFO:  Input triples processed: 9,500,000,000
 +
2022-02-02 14:47:55.735 - INFO:  Input triples processed: 9,600,000,000
 +
2022-02-02 14:53:25.586 - INFO:  Input triples processed: 9,700,000,000
 +
2022-02-02 14:58:45.841 - INFO:  Input triples processed: 9,800,000,000
 +
2022-02-02 15:04:14.244 - INFO:  Input triples processed: 9,900,000,000
 +
2022-02-02 15:09:45.942 - INFO:  Input triples processed: 10,000,000,000
 +
2022-02-02 15:15:29.977 - INFO:  Input triples processed: 10,100,000,000
 +
2022-02-02 15:21:02.689 - INFO:  Input triples processed: 10,200,000,000
 +
2022-02-02 15:26:50.306 - INFO:  Input triples processed: 10,300,000,000
 +
2022-02-02 15:32:30.487 - INFO:  Input triples processed: 10,400,000,000
 +
2022-02-02 15:37:55.973 - INFO:  Input triples processed: 10,500,000,000
 +
2022-02-02 15:43:21.336 - INFO:  Input triples processed: 10,600,000,000
 +
2022-02-02 15:49:07.198 - INFO:  Input triples processed: 10,700,000,000
 +
2022-02-02 15:54:21.597 - INFO:  Input triples processed: 10,800,000,000
 +
2022-02-02 15:59:26.248 - INFO:  Input triples processed: 10,900,000,000
 +
2022-02-02 16:04:47.254 - INFO:  Input triples processed: 11,000,000,000
 +
2022-02-02 16:10:21.939 - INFO:  Input triples processed: 11,100,000,000
 +
2022-02-02 16:16:17.855 - INFO:  Input triples processed: 11,200,000,000
 +
2022-02-02 16:21:46.377 - INFO:  Input triples processed: 11,300,000,000
 +
2022-02-02 16:27:05.461 - INFO:  Input triples processed: 11,400,000,000
 +
2022-02-02 16:32:31.003 - INFO:  Input triples processed: 11,500,000,000
 +
2022-02-02 16:37:43.081 - INFO:  Input triples processed: 11,600,000,000
 +
2022-02-02 16:43:14.589 - INFO:  Input triples processed: 11,700,000,000
 +
2022-02-02 16:48:55.122 - INFO:  Input triples processed: 11,800,000,000
 +
2022-02-02 16:54:17.942 - INFO:  Input triples processed: 11,900,000,000
 +
2022-02-02 16:59:48.271 - INFO:  Input triples processed: 12,000,000,000
 +
2022-02-02 17:05:31.450 - INFO:  Input triples processed: 12,100,000,000
 +
2022-02-02 17:11:21.730 - INFO:  Input triples processed: 12,200,000,000
 +
2022-02-02 17:17:07.859 - INFO:  Input triples processed: 12,300,000,000
 +
2022-02-02 17:22:38.664 - INFO:  Input triples processed: 12,400,000,000
 +
2022-02-02 17:28:27.188 - INFO:  Input triples processed: 12,500,000,000
 +
2022-02-02 17:33:53.612 - INFO:  Input triples processed: 12,600,000,000
 +
2022-02-02 17:39:21.492 - INFO:  Input triples processed: 12,700,000,000
 +
2022-02-02 17:45:10.469 - INFO:  Input triples processed: 12,800,000,000
 +
2022-02-02 17:50:24.948 - INFO:  Input triples processed: 12,900,000,000
 +
2022-02-02 17:55:29.110 - INFO:  Input triples processed: 13,000,000,000
 +
2022-02-02 18:00:57.257 - INFO:  Input triples processed: 13,100,000,000
 +
2022-02-02 18:06:17.378 - INFO:  Input triples processed: 13,200,000,000
 +
2022-02-02 18:11:47.981 - INFO:  Input triples processed: 13,300,000,000
 +
2022-02-02 18:17:10.888 - INFO:  Input triples processed: 13,400,000,000
 +
2022-02-02 18:22:47.533 - INFO:  Input triples processed: 13,500,000,000
 +
2022-02-02 18:28:01.125 - INFO:  Input triples processed: 13,600,000,000
 +
2022-02-02 18:33:20.535 - INFO:  Input triples processed: 13,700,000,000
 +
2022-02-02 18:39:01.344 - INFO:  Input triples processed: 13,800,000,000
 +
2022-02-02 18:44:15.241 - INFO:  Input triples processed: 13,900,000,000
 +
2022-02-02 18:50:00.600 - INFO:  Input triples processed: 14,000,000,000
 +
2022-02-02 18:55:33.580 - INFO:  Input triples processed: 14,100,000,000
 +
2022-02-02 19:01:14.666 - INFO:  Input triples processed: 14,200,000,000
 +
2022-02-02 19:06:46.602 - INFO:  Input triples processed: 14,300,000,000
 +
2022-02-02 19:12:38.371 - INFO:  Input triples processed: 14,400,000,000
 +
2022-02-02 19:18:41.261 - INFO:  Input triples processed: 14,500,000,000
 +
2022-02-02 19:24:17.925 - INFO:  Input triples processed: 14,600,000,000
 +
2022-02-02 19:29:31.631 - INFO:  Input triples processed: 14,700,000,000
 +
2022-02-02 19:35:08.007 - INFO:  Input triples processed: 14,800,000,000
 +
2022-02-02 19:40:52.112 - INFO:  Input triples processed: 14,900,000,000
 +
2022-02-02 19:46:00.751 - INFO:  Input triples processed: 15,000,000,000
 +
2022-02-02 19:51:10.905 - INFO:  Input triples processed: 15,100,000,000
 +
2022-02-02 19:56:46.037 - INFO:  Input triples processed: 15,200,000,000
 +
2022-02-02 20:02:09.632 - INFO:  Input triples processed: 15,300,000,000
 +
2022-02-02 20:07:34.112 - INFO:  Input triples processed: 15,400,000,000
 +
2022-02-02 20:12:41.231 - INFO:  Input triples processed: 15,500,000,000
 +
2022-02-02 20:18:17.512 - INFO:  Input triples processed: 15,600,000,000
 +
2022-02-02 20:23:50.285 - INFO:  Input triples processed: 15,700,000,000
 +
2022-02-02 20:29:25.067 - INFO:  Input triples processed: 15,800,000,000
 +
2022-02-02 20:34:45.926 - INFO:  Input triples processed: 15,900,000,000
 +
2022-02-02 20:40:08.984 - INFO:  Input triples processed: 16,000,000,000
 +
2022-02-02 20:45:35.751 - INFO:  Input triples processed: 16,100,000,000
 +
2022-02-02 20:51:10.034 - INFO:  Input triples processed: 16,200,000,000
 +
2022-02-02 20:57:02.400 - INFO:  Input triples processed: 16,300,000,000
 +
2022-02-02 21:02:45.497 - INFO:  Input triples processed: 16,400,000,000
 +
2022-02-02 21:08:34.531 - INFO:  Input triples processed: 16,500,000,000
 +
2022-02-02 21:14:14.133 - INFO:  Input triples processed: 16,600,000,000
 +
2022-02-02 21:19:58.155 - INFO:  Input triples processed: 16,700,000,000
 +
2022-02-02 21:25:23.936 - INFO:  Input triples processed: 16,800,000,000
 +
2022-02-02 21:30:23.323 - INFO:  Input triples processed: 16,900,000,000
 +
2022-02-02 21:35:25.492 - INFO:  Done, total number of triples read: 16,979,598,468 [may contain duplicates]
 +
2022-02-02 21:35:25.492 - INFO:  Number of QLever-internal triples created: 10,093,281,840 [may contain duplicates]
 +
2022-02-02 21:35:25.492 - INFO:  Merging partial vocabularies in byte order (internal only) ...
 +
2022-02-02 21:40:46.606 - INFO:  Words merged: 100,000,000
 +
2022-02-02 21:48:22.776 - INFO:  Words merged: 200,000,000
 +
2022-02-02 21:55:53.025 - INFO:  Words merged: 300,000,000
 +
2022-02-02 22:06:50.288 - INFO:  Words merged: 400,000,000
 +
2022-02-02 22:15:20.636 - INFO:  Words merged: 500,000,000
 +
2022-02-02 22:21:49.518 - INFO:  Words merged: 600,000,000
 +
2022-02-02 22:43:44.014 - INFO:  Words merged: 700,000,000
 +
2022-02-02 22:50:53.562 - INFO:  Words merged: 800,000,000
 +
2022-02-02 22:56:18.623 - INFO:  Words merged: 900,000,000
 +
2022-02-02 22:57:48.601 - INFO:  Number of words in internal vocabulary: 938,491,990
 +
2022-02-02 22:57:48.602 - INFO:  Merging partial vocabularies in Unicode order (internal and external) ...
 +
2022-02-02 23:06:56.472 - INFO:  Words merged: 100,000,000
 +
2022-02-02 23:17:33.337 - INFO:  Words merged: 200,000,000
 +
2022-02-02 23:28:18.214 - INFO:  Words merged: 300,000,000
 +
2022-02-02 23:42:45.536 - INFO:  Words merged: 400,000,000
 +
2022-02-02 23:52:58.541 - INFO:  Words merged: 500,000,000
 +
2022-02-03 00:01:16.765 - INFO:  Words merged: 600,000,000
 +
2022-02-03 00:11:23.418 - INFO:  Words merged: 700,000,000
 +
2022-02-03 00:19:46.619 - INFO:  Words merged: 800,000,000
 +
2022-02-03 00:45:36.524 - INFO:  Words merged: 900,000,000
 +
2022-02-03 00:58:09.284 - INFO:  Words merged: 1,000,000,000
 +
2022-02-03 01:06:24.768 - INFO:  Words merged: 1,100,000,000
 +
2022-02-03 01:14:50.087 - INFO:  Words merged: 1,200,000,000
 +
2022-02-03 01:23:12.324 - INFO:  Words merged: 1,300,000,000
 +
2022-02-03 01:33:05.021 - INFO:  Words merged: 1,400,000,000
 +
2022-02-03 01:41:01.953 - INFO:  Words merged: 1,500,000,000
 +
2022-02-03 01:50:09.204 - INFO:  Words merged: 1,600,000,000
 +
2022-02-03 02:02:03.331 - INFO:  Words merged: 1,700,000,000
 +
2022-02-03 02:13:53.724 - INFO:  Words merged: 1,800,000,000
 +
2022-02-03 02:22:53.444 - INFO:  Words merged: 1,900,000,000
 +
2022-02-03 02:32:32.792 - INFO:  Words merged: 2,000,000,000
 +
2022-02-03 02:42:29.268 - INFO:  Words merged: 2,100,000,000
 +
2022-02-03 02:52:16.904 - INFO:  Words merged: 2,200,000,000
 +
2022-02-03 03:02:03.858 - INFO:  Words merged: 2,300,000,000
 +
2022-02-03 03:10:43.495 - INFO:  Words merged: 2,400,000,000
 +
2022-02-03 03:20:05.744 - INFO:  Words merged: 2,500,000,000
 +
2022-02-03 03:29:44.044 - INFO:  Words merged: 2,600,000,000
 +
2022-02-03 03:38:34.892 - INFO:  Words merged: 2,700,000,000
 +
2022-02-03 03:47:25.028 - INFO:  Words merged: 2,800,000,000
 +
2022-02-03 03:56:30.070 - INFO:  Words merged: 2,900,000,000
 +
2022-02-03 04:05:56.032 - INFO:  Words merged: 3,000,000,000
 +
2022-02-03 04:16:10.871 - INFO:  Words merged: 3,100,000,000
 +
2022-02-03 04:30:33.889 - INFO:  Words merged: 3,200,000,000
 +
2022-02-03 04:38:45.525 - INFO:  Number of words in external vocabulary: 2,289,883,113
 +
2022-02-03 04:38:45.531 - INFO:  Removing temporary files ...
 +
2022-02-03 04:44:04.818 - INFO:  Converting external vocabulary to binary format ...
 +
2022-02-03 06:04:44.060 - INFO:  Converting triples from local IDs to global IDs ...
 +
2022-02-03 06:07:03.695 - INFO:  Triples converted: 100,000,000
 +
2022-02-03 06:08:47.794 - INFO:  Triples converted: 200,000,000
 +
2022-02-03 06:10:29.184 - INFO:  Triples converted: 300,000,000
 +
2022-02-03 06:12:20.313 - INFO:  Triples converted: 400,000,000
 +
2022-02-03 06:14:02.908 - INFO:  Triples converted: 500,000,000
 +
2022-02-03 06:15:36.523 - INFO:  Triples converted: 600,000,000
 +
2022-02-03 06:17:07.086 - INFO:  Triples converted: 700,000,000
 +
2022-02-03 06:18:31.008 - INFO:  Triples converted: 800,000,000
 +
2022-02-03 06:19:50.168 - INFO:  Triples converted: 900,000,000
 +
2022-02-03 06:21:21.558 - INFO:  Triples converted: 1,000,000,000
 +
2022-02-03 06:22:52.771 - INFO:  Triples converted: 1,100,000,000
 +
2022-02-03 06:24:32.112 - INFO:  Triples converted: 1,200,000,000
 +
2022-02-03 06:25:50.747 - INFO:  Triples converted: 1,300,000,000
 +
2022-02-03 06:27:14.209 - INFO:  Triples converted: 1,400,000,000
 +
2022-02-03 06:28:50.732 - INFO:  Triples converted: 1,500,000,000
 +
2022-02-03 06:30:33.606 - INFO:  Triples converted: 1,600,000,000
 +
2022-02-03 06:32:01.607 - INFO:  Triples converted: 1,700,000,000
 +
2022-02-03 06:33:27.405 - INFO:  Triples converted: 1,800,000,000
 +
2022-02-03 06:35:04.990 - INFO:  Triples converted: 1,900,000,000
 +
2022-02-03 06:36:33.487 - INFO:  Triples converted: 2,000,000,000
 +
2022-02-03 06:38:27.729 - INFO:  Triples converted: 2,100,000,000
 +
2022-02-03 06:40:19.789 - INFO:  Triples converted: 2,200,000,000
 +
2022-02-03 06:42:23.455 - INFO:  Triples converted: 2,300,000,000
 +
2022-02-03 06:44:20.406 - INFO:  Triples converted: 2,400,000,000
 +
2022-02-03 06:46:02.518 - INFO:  Triples converted: 2,500,000,000
 +
2022-02-03 06:47:43.823 - INFO:  Triples converted: 2,600,000,000
 +
2022-02-03 06:49:18.014 - INFO:  Triples converted: 2,700,000,000
 +
2022-02-03 06:50:58.714 - INFO:  Triples converted: 2,800,000,000
 +
2022-02-03 06:52:29.128 - INFO:  Triples converted: 2,900,000,000
 +
2022-02-03 06:54:09.454 - INFO:  Triples converted: 3,000,000,000
 +
2022-02-03 06:55:54.293 - INFO:  Triples converted: 3,100,000,000
 +
2022-02-03 06:57:27.644 - INFO:  Triples converted: 3,200,000,000
 +
2022-02-03 06:59:03.790 - INFO:  Triples converted: 3,300,000,000
 +
2022-02-03 07:00:43.280 - INFO:  Triples converted: 3,400,000,000
 +
2022-02-03 07:02:46.205 - INFO:  Triples converted: 3,500,000,000
 +
2022-02-03 07:04:29.008 - INFO:  Triples converted: 3,600,000,000
 +
2022-02-03 07:06:13.559 - INFO:  Triples converted: 3,700,000,000
 +
2022-02-03 07:07:49.482 - INFO:  Triples converted: 3,800,000,000
 +
2022-02-03 07:09:18.854 - INFO:  Triples converted: 3,900,000,000
 +
2022-02-03 07:11:10.058 - INFO:  Triples converted: 4,000,000,000
 +
2022-02-03 07:12:48.747 - INFO:  Triples converted: 4,100,000,000
 +
2022-02-03 07:14:31.783 - INFO:  Triples converted: 4,200,000,000
 +
2022-02-03 07:16:14.597 - INFO:  Triples converted: 4,300,000,000
 +
2022-02-03 07:17:47.847 - INFO:  Triples converted: 4,400,000,000
 +
2022-02-03 07:19:27.982 - INFO:  Triples converted: 4,500,000,000
 +
2022-02-03 07:21:19.428 - INFO:  Triples converted: 4,600,000,000
 +
2022-02-03 07:23:09.734 - INFO:  Triples converted: 4,700,000,000
 +
2022-02-03 07:25:06.433 - INFO:  Triples converted: 4,800,000,000
 +
2022-02-03 07:26:47.995 - INFO:  Triples converted: 4,900,000,000
 +
2022-02-03 07:28:21.895 - INFO:  Triples converted: 5,000,000,000
 +
2022-02-03 07:30:04.774 - INFO:  Triples converted: 5,100,000,000
 +
2022-02-03 07:31:42.133 - INFO:  Triples converted: 5,200,000,000
 +
2022-02-03 07:33:36.029 - INFO:  Triples converted: 5,300,000,000
 +
2022-02-03 07:35:23.311 - INFO:  Triples converted: 5,400,000,000
 +
2022-02-03 07:37:02.714 - INFO:  Triples converted: 5,500,000,000
 +
2022-02-03 07:38:42.429 - INFO:  Triples converted: 5,600,000,000
 +
2022-02-03 07:40:13.634 - INFO:  Triples converted: 5,700,000,000
 +
2022-02-03 07:41:56.442 - INFO:  Triples converted: 5,800,000,000
 +
2022-02-03 07:43:39.667 - INFO:  Triples converted: 5,900,000,000
 +
2022-02-03 07:45:23.077 - INFO:  Triples converted: 6,000,000,000
 +
2022-02-03 07:46:55.523 - INFO:  Triples converted: 6,100,000,000
 +
2022-02-03 07:48:32.532 - INFO:  Triples converted: 6,200,000,000
 +
2022-02-03 07:50:03.494 - INFO:  Triples converted: 6,300,000,000
 +
2022-02-03 07:51:37.785 - INFO:  Triples converted: 6,400,000,000
 +
2022-02-03 07:53:15.261 - INFO:  Triples converted: 6,500,000,000
 +
2022-02-03 07:55:03.364 - INFO:  Triples converted: 6,600,000,000
 +
2022-02-03 07:56:46.808 - INFO:  Triples converted: 6,700,000,000
 +
2022-02-03 07:58:34.540 - INFO:  Triples converted: 6,800,000,000
 +
2022-02-03 08:00:20.230 - INFO:  Triples converted: 6,900,000,000
 +
2022-02-03 08:02:20.105 - INFO:  Triples converted: 7,000,000,000
 +
2022-02-03 08:03:54.073 - INFO:  Triples converted: 7,100,000,000
 +
2022-02-03 08:05:25.878 - INFO:  Triples converted: 7,200,000,000
 +
2022-02-03 08:07:00.337 - INFO:  Triples converted: 7,300,000,000
 +
2022-02-03 08:08:31.856 - INFO:  Triples converted: 7,400,000,000
 +
2022-02-03 08:10:10.808 - INFO:  Triples converted: 7,500,000,000
 +
2022-02-03 08:11:52.730 - INFO:  Triples converted: 7,600,000,000
 +
2022-02-03 08:13:20.423 - INFO:  Triples converted: 7,700,000,000
 +
2022-02-03 08:14:48.426 - INFO:  Triples converted: 7,800,000,000
 +
2022-02-03 08:16:19.342 - INFO:  Triples converted: 7,900,000,000
 +
2022-02-03 08:17:37.066 - INFO:  Triples converted: 8,000,000,000
 +
2022-02-03 08:19:06.312 - INFO:  Triples converted: 8,100,000,000
 +
2022-02-03 08:20:24.640 - INFO:  Triples converted: 8,200,000,000
 +
2022-02-03 08:21:51.644 - INFO:  Triples converted: 8,300,000,000
 +
2022-02-03 08:23:20.594 - INFO:  Triples converted: 8,400,000,000
 +
2022-02-03 08:25:06.193 - INFO:  Triples converted: 8,500,000,000
 +
2022-02-03 08:26:48.557 - INFO:  Triples converted: 8,600,000,000
 +
2022-02-03 08:28:14.023 - INFO:  Triples converted: 8,700,000,000
 +
2022-02-03 08:29:43.745 - INFO:  Triples converted: 8,800,000,000
 +
2022-02-03 08:31:02.245 - INFO:  Triples converted: 8,900,000,000
 +
2022-02-03 08:32:32.198 - INFO:  Triples converted: 9,000,000,000
 +
2022-02-03 08:34:53.199 - INFO:  Triples converted: 9,100,000,000
 +
2022-02-03 08:37:01.970 - INFO:  Triples converted: 9,200,000,000
 +
2022-02-03 08:38:46.399 - INFO:  Triples converted: 9,300,000,000
 +
2022-02-03 08:40:24.404 - INFO:  Triples converted: 9,400,000,000
 +
2022-02-03 08:42:26.291 - INFO:  Triples converted: 9,500,000,000
 +
2022-02-03 08:45:21.894 - INFO:  Triples converted: 9,600,000,000
 +
2022-02-03 08:48:12.170 - INFO:  Triples converted: 9,700,000,000
 +
2022-02-03 08:52:29.881 - INFO:  Triples converted: 9,800,000,000
 +
2022-02-03 08:55:36.916 - INFO:  Triples converted: 9,900,000,000
 +
2022-02-03 08:58:56.555 - INFO:  Triples converted: 10,000,000,000
 +
2022-02-03 09:00:24.102 - INFO:  Triples converted: 10,100,000,000
 +
2022-02-03 09:01:57.289 - INFO:  Triples converted: 10,200,000,000
 +
2022-02-03 09:03:38.529 - INFO:  Triples converted: 10,300,000,000
 +
2022-02-03 09:05:27.127 - INFO:  Triples converted: 10,400,000,000
 +
2022-02-03 09:07:00.262 - INFO:  Triples converted: 10,500,000,000
 +
2022-02-03 09:08:29.934 - INFO:  Triples converted: 10,600,000,000
 +
2022-02-03 09:09:54.909 - INFO:  Triples converted: 10,700,000,000
 +
2022-02-03 09:11:31.481 - INFO:  Triples converted: 10,800,000,000
 +
2022-02-03 09:13:08.048 - INFO:  Triples converted: 10,900,000,000
 +
2022-02-03 09:14:45.819 - INFO:  Triples converted: 11,000,000,000
 +
2022-02-03 09:16:25.319 - INFO:  Triples converted: 11,100,000,000
 +
2022-02-03 09:17:49.967 - INFO:  Triples converted: 11,200,000,000
 +
2022-02-03 09:19:25.744 - INFO:  Triples converted: 11,300,000,000
 +
2022-02-03 09:21:11.002 - INFO:  Triples converted: 11,400,000,000
 +
2022-02-03 09:23:01.497 - INFO:  Triples converted: 11,500,000,000
 +
2022-02-03 09:24:52.943 - INFO:  Triples converted: 11,600,000,000
 +
2022-02-03 09:26:21.260 - INFO:  Triples converted: 11,700,000,000
 +
2022-02-03 09:27:49.169 - INFO:  Triples converted: 11,800,000,000
 +
2022-02-03 09:29:26.027 - INFO:  Triples converted: 11,900,000,000
 +
2022-02-03 09:30:53.202 - INFO:  Triples converted: 12,000,000,000
 +
2022-02-03 09:32:18.027 - INFO:  Triples converted: 12,100,000,000
 +
2022-02-03 09:33:59.259 - INFO:  Triples converted: 12,200,000,000
 +
2022-02-03 09:35:26.412 - INFO:  Triples converted: 12,300,000,000
 +
2022-02-03 09:37:15.019 - INFO:  Triples converted: 12,400,000,000
 +
2022-02-03 09:39:11.756 - INFO:  Triples converted: 12,500,000,000
 +
2022-02-03 09:40:47.331 - INFO:  Triples converted: 12,600,000,000
 +
2022-02-03 09:42:30.756 - INFO:  Triples converted: 12,700,000,000
 +
2022-02-03 09:44:14.363 - INFO:  Triples converted: 12,800,000,000
 +
2022-02-03 09:46:09.220 - INFO:  Triples converted: 12,900,000,000
 +
2022-02-03 09:47:48.549 - INFO:  Triples converted: 13,000,000,000
 +
2022-02-03 09:49:23.978 - INFO:  Triples converted: 13,100,000,000
 +
2022-02-03 09:51:25.475 - INFO:  Triples converted: 13,200,000,000
 +
2022-02-03 09:53:00.696 - INFO:  Triples converted: 13,300,000,000
 +
2022-02-03 09:54:56.191 - INFO:  Triples converted: 13,400,000,000
 +
2022-02-03 09:56:46.508 - INFO:  Triples converted: 13,500,000,000
 +
2022-02-03 09:58:21.403 - INFO:  Triples converted: 13,600,000,000
 +
2022-02-03 10:00:09.504 - INFO:  Triples converted: 13,700,000,000
 +
2022-02-03 10:02:17.575 - INFO:  Triples converted: 13,800,000,000
 +
2022-02-03 10:04:02.820 - INFO:  Triples converted: 13,900,000,000
 +
2022-02-03 10:05:44.401 - INFO:  Triples converted: 14,000,000,000
 +
2022-02-03 10:07:17.813 - INFO:  Triples converted: 14,100,000,000
 +
2022-02-03 10:09:10.136 - INFO:  Triples converted: 14,200,000,000
 +
2022-02-03 10:10:56.520 - INFO:  Triples converted: 14,300,000,000
 +
2022-02-03 10:13:01.286 - INFO:  Triples converted: 14,400,000,000
 +
2022-02-03 10:14:50.223 - INFO:  Triples converted: 14,500,000,000
 +
2022-02-03 10:16:40.323 - INFO:  Triples converted: 14,600,000,000
 +
2022-02-03 10:18:24.466 - INFO:  Triples converted: 14,700,000,000
 +
2022-02-03 10:20:11.680 - INFO:  Triples converted: 14,800,000,000
 +
2022-02-03 10:22:01.493 - INFO:  Triples converted: 14,900,000,000
 +
2022-02-03 10:24:08.333 - INFO:  Triples converted: 15,000,000,000
 +
2022-02-03 10:25:41.022 - INFO:  Triples converted: 15,100,000,000
 +
2022-02-03 10:27:25.221 - INFO:  Triples converted: 15,200,000,000
 +
2022-02-03 10:29:00.946 - INFO:  Triples converted: 15,300,000,000
 +
2022-02-03 10:30:52.184 - INFO:  Triples converted: 15,400,000,000
 +
2022-02-03 10:32:38.721 - INFO:  Triples converted: 15,500,000,000
 +
2022-02-03 10:34:11.556 - INFO:  Triples converted: 15,600,000,000
 +
2022-02-03 10:35:57.410 - INFO:  Triples converted: 15,700,000,000
 +
2022-02-03 10:37:44.445 - INFO:  Triples converted: 15,800,000,000
 +
2022-02-03 10:39:16.599 - INFO:  Triples converted: 15,900,000,000
 +
2022-02-03 10:41:02.244 - INFO:  Triples converted: 16,000,000,000
 +
2022-02-03 10:42:39.534 - INFO:  Triples converted: 16,100,000,000
 +
2022-02-03 10:44:19.688 - INFO:  Triples converted: 16,200,000,000
 +
2022-02-03 10:46:13.325 - INFO:  Triples converted: 16,300,000,000
 +
2022-02-03 10:47:55.063 - INFO:  Triples converted: 16,400,000,000
 +
2022-02-03 10:49:42.592 - INFO:  Triples converted: 16,500,000,000
 +
2022-02-03 10:51:27.606 - INFO:  Triples converted: 16,600,000,000
 +
2022-02-03 10:53:25.996 - INFO:  Triples converted: 16,700,000,000
 +
2022-02-03 10:55:20.118 - INFO:  Triples converted: 16,800,000,000
 +
2022-02-03 10:57:21.513 - INFO:  Triples converted: 16,900,000,000
 +
2022-02-03 10:59:16.479 - INFO:  Triples converted: 17,000,000,000
 +
2022-02-03 11:01:11.333 - INFO:  Triples converted: 17,100,000,000
 +
2022-02-03 11:03:04.520 - INFO:  Triples converted: 17,200,000,000
 +
2022-02-03 11:04:40.493 - INFO:  Triples converted: 17,300,000,000
 +
2022-02-03 11:06:28.389 - INFO:  Triples converted: 17,400,000,000
 +
2022-02-03 11:08:26.442 - INFO:  Triples converted: 17,500,000,000
 +
2022-02-03 11:10:22.786 - INFO:  Triples converted: 17,600,000,000
 +
2022-02-03 11:12:36.616 - INFO:  Triples converted: 17,700,000,000
 +
2022-02-03 11:14:18.778 - INFO:  Triples converted: 17,800,000,000
 +
2022-02-03 11:16:13.166 - INFO:  Triples converted: 17,900,000,000
 +
2022-02-03 11:18:10.780 - INFO:  Triples converted: 18,000,000,000
 +
2022-02-03 11:19:51.236 - INFO:  Triples converted: 18,100,000,000
 +
2022-02-03 11:21:40.897 - INFO:  Triples converted: 18,200,000,000
 +
2022-02-03 11:23:21.193 - INFO:  Triples converted: 18,300,000,000
 +
2022-02-03 11:25:03.483 - INFO:  Triples converted: 18,400,000,000
 +
2022-02-03 11:26:57.531 - INFO:  Triples converted: 18,500,000,000
 +
2022-02-03 11:28:27.511 - INFO:  Triples converted: 18,600,000,000
 +
2022-02-03 11:30:11.088 - INFO:  Triples converted: 18,700,000,000
 +
2022-02-03 11:31:54.527 - INFO:  Triples converted: 18,800,000,000
 +
2022-02-03 11:33:36.040 - INFO:  Triples converted: 18,900,000,000
 +
2022-02-03 11:35:17.586 - INFO:  Triples converted: 19,000,000,000
 +
2022-02-03 11:37:06.543 - INFO:  Triples converted: 19,100,000,000
 +
2022-02-03 11:38:49.740 - INFO:  Triples converted: 19,200,000,000
 +
2022-02-03 11:40:42.584 - INFO:  Triples converted: 19,300,000,000
 +
2022-02-03 11:42:30.329 - INFO:  Triples converted: 19,400,000,000
 +
2022-02-03 11:44:19.331 - INFO:  Triples converted: 19,500,000,000
 +
2022-02-03 11:46:00.269 - INFO:  Triples converted: 19,600,000,000
 +
2022-02-03 11:47:37.169 - INFO:  Triples converted: 19,700,000,000
 +
2022-02-03 11:49:38.635 - INFO:  Triples converted: 19,800,000,000
 +
2022-02-03 11:51:22.625 - INFO:  Triples converted: 19,900,000,000
 +
2022-02-03 11:53:26.662 - INFO:  Triples converted: 20,000,000,000
 +
2022-02-03 11:55:20.818 - INFO:  Triples converted: 20,100,000,000
 +
2022-02-03 11:56:57.081 - INFO:  Triples converted: 20,200,000,000
 +
2022-02-03 11:58:44.907 - INFO:  Triples converted: 20,300,000,000
 +
2022-02-03 12:00:41.842 - INFO:  Triples converted: 20,400,000,000
 +
2022-02-03 12:02:39.025 - INFO:  Triples converted: 20,500,000,000
 +
2022-02-03 12:04:19.605 - INFO:  Triples converted: 20,600,000,000
 +
2022-02-03 12:06:05.027 - INFO:  Triples converted: 20,700,000,000
 +
2022-02-03 12:07:47.944 - INFO:  Triples converted: 20,800,000,000
 +
2022-02-03 12:09:42.388 - INFO:  Triples converted: 20,900,000,000
 +
2022-02-03 12:11:38.584 - INFO:  Triples converted: 21,000,000,000
 +
2022-02-03 12:13:28.775 - INFO:  Triples converted: 21,100,000,000
 +
2022-02-03 12:15:22.070 - INFO:  Triples converted: 21,200,000,000
 +
2022-02-03 12:17:03.204 - INFO:  Triples converted: 21,300,000,000
 +
2022-02-03 12:18:44.107 - INFO:  Triples converted: 21,400,000,000
 +
2022-02-03 12:20:20.415 - INFO:  Triples converted: 21,500,000,000
 +
2022-02-03 12:22:13.964 - INFO:  Triples converted: 21,600,000,000
 +
2022-02-03 12:24:18.256 - INFO:  Triples converted: 21,700,000,000
 +
2022-02-03 12:26:08.881 - INFO:  Triples converted: 21,800,000,000
 +
2022-02-03 12:27:48.096 - INFO:  Triples converted: 21,900,000,000
 +
2022-02-03 12:29:18.359 - INFO:  Triples converted: 22,000,000,000
 +
2022-02-03 12:31:01.268 - INFO:  Triples converted: 22,100,000,000
 +
2022-02-03 12:32:45.265 - INFO:  Triples converted: 22,200,000,000
 +
2022-02-03 12:34:28.333 - INFO:  Triples converted: 22,300,000,000
 +
2022-02-03 12:36:18.373 - INFO:  Triples converted: 22,400,000,000
 +
2022-02-03 12:38:12.105 - INFO:  Triples converted: 22,500,000,000
 +
2022-02-03 12:40:03.848 - INFO:  Triples converted: 22,600,000,000
 +
2022-02-03 12:41:37.959 - INFO:  Triples converted: 22,700,000,000
 +
2022-02-03 12:43:24.983 - INFO:  Triples converted: 22,800,000,000
 +
2022-02-03 12:45:20.230 - INFO:  Triples converted: 22,900,000,000
 +
2022-02-03 12:47:20.548 - INFO:  Triples converted: 23,000,000,000
 +
2022-02-03 12:49:18.133 - INFO:  Triples converted: 23,100,000,000
 +
2022-02-03 12:51:04.519 - INFO:  Triples converted: 23,200,000,000
 +
2022-02-03 12:52:58.685 - INFO:  Triples converted: 23,300,000,000
 +
2022-02-03 12:54:54.395 - INFO:  Triples converted: 23,400,000,000
 +
2022-02-03 12:56:42.684 - INFO:  Triples converted: 23,500,000,000
 +
2022-02-03 12:58:30.014 - INFO:  Triples converted: 23,600,000,000
 +
2022-02-03 13:00:44.041 - INFO:  Triples converted: 23,700,000,000
 +
2022-02-03 13:02:41.601 - INFO:  Triples converted: 23,800,000,000
 +
2022-02-03 13:04:31.145 - INFO:  Triples converted: 23,900,000,000
 +
2022-02-03 13:06:17.909 - INFO:  Triples converted: 24,000,000,000
 +
2022-02-03 13:07:58.562 - INFO:  Triples converted: 24,100,000,000
 +
2022-02-03 13:09:45.479 - INFO:  Triples converted: 24,200,000,000
 +
2022-02-03 13:11:44.530 - INFO:  Triples converted: 24,300,000,000
 +
2022-02-03 13:13:41.162 - INFO:  Triples converted: 24,400,000,000
 +
2022-02-03 13:15:37.924 - INFO:  Triples converted: 24,500,000,000
 +
2022-02-03 13:17:24.356 - INFO:  Triples converted: 24,600,000,000
 +
2022-02-03 13:19:08.196 - INFO:  Triples converted: 24,700,000,000
 +
2022-02-03 13:21:08.233 - INFO:  Triples converted: 24,800,000,000
 +
2022-02-03 13:23:08.604 - INFO:  Triples converted: 24,900,000,000
 +
2022-02-03 13:25:27.663 - INFO:  Triples converted: 25,000,000,000
 +
2022-02-03 13:27:17.755 - INFO:  Triples converted: 25,100,000,000
 +
2022-02-03 13:29:03.092 - INFO:  Triples converted: 25,200,000,000
 +
2022-02-03 13:30:52.609 - INFO:  Triples converted: 25,300,000,000
 +
2022-02-03 13:33:00.160 - INFO:  Triples converted: 25,400,000,000
 +
2022-02-03 13:34:57.731 - INFO:  Triples converted: 25,500,000,000
 +
2022-02-03 13:36:47.772 - INFO:  Triples converted: 25,600,000,000
 +
2022-02-03 13:38:25.167 - INFO:  Triples converted: 25,700,000,000
 +
2022-02-03 13:40:28.092 - INFO:  Triples converted: 25,800,000,000
 +
2022-02-03 13:42:32.451 - INFO:  Triples converted: 25,900,000,000
 +
2022-02-03 13:44:18.502 - INFO:  Triples converted: 26,000,000,000
 +
2022-02-03 13:46:08.486 - INFO:  Triples converted: 26,100,000,000
 +
2022-02-03 13:47:45.104 - INFO:  Triples converted: 26,200,000,000
 +
2022-02-03 13:49:28.084 - INFO:  Triples converted: 26,300,000,000
 +
2022-02-03 13:51:22.885 - INFO:  Triples converted: 26,400,000,000
 +
2022-02-03 13:53:19.507 - INFO:  Triples converted: 26,500,000,000
 +
2022-02-03 13:55:14.153 - INFO:  Triples converted: 26,600,000,000
 +
2022-02-03 13:57:00.683 - INFO:  Triples converted: 26,700,000,000
 +
2022-02-03 13:58:45.965 - INFO:  Triples converted: 26,800,000,000
 +
2022-02-03 14:00:22.233 - INFO:  Triples converted: 26,900,000,000
 +
2022-02-03 14:02:02.021 - INFO:  Triples converted: 27,000,000,000
 +
2022-02-03 14:03:30.586 - INFO:  Done, total number of triples converted: 27,072,880,308
 +
2022-02-03 14:03:30.791 - INFO:  Building prefix tree from internal vocabulary ...
 +
2022-02-03 14:04:11.957 - INFO:  Words processed: 100,000,000
 +
2022-02-03 14:05:17.470 - INFO:  Words processed: 200,000,000
 +
2022-02-03 14:06:28.451 - INFO:  Words processed: 300,000,000
 +
2022-02-03 14:07:43.608 - INFO:  Words processed: 400,000,000
 +
2022-02-03 14:09:12.898 - INFO:  Words processed: 500,000,000
 +
2022-02-03 14:10:26.170 - INFO:  Words processed: 600,000,000
 +
2022-02-03 14:11:27.474 - INFO:  Words processed: 700,000,000
 +
2022-02-03 14:13:15.041 - INFO:  Words processed: 800,000,000
 +
2022-02-03 14:14:46.773 - INFO:  Words processed: 900,000,000
 +
2022-02-03 14:15:17.044 - INFO:  Computing maximally compressing prefixes (greedy algorithm) ...
 +
2022-02-03 15:16:53.604 - INFO:  Reduction of size of internal vocabulary: 46%
 +
2022-02-03 15:18:21.559 - INFO:  Writing compressed vocabulary to disk ...
 +
2022-02-03 15:40:47.936 - INFO:  Sorting for PSO permutation ...
 +
2022-02-04 01:01:48.418 - INFO:  Removing duplicate triples ...
 +
2022-02-04 11:00:37.645 - INFO:  Done, number of distinct triples is: 22,650,509,456
 +
2022-02-04 11:00:37.646 - INFO:  Creating a pair of index permutations ...
 +
2022-02-04 19:48:26.759 - INFO:  Statistics for PSO: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
 +
2022-02-04 19:48:26.777 - INFO:  Statistics for POS: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
 +
2022-02-04 19:48:26.777 - INFO:  Exchanging multiplicities for PSO and POS ...
 +
2022-02-04 19:48:26.803 - INFO:  Writing meta data for PSO and POS ...
 +
2022-02-04 19:48:26.866 - INFO:  Sorting for SPO permutation ...
 +
2022-02-05 04:18:41.117 - INFO:  Creating a pair of index permutations ...
 +
2022-02-05 10:39:07.231 - INFO:  Statistics for SPO: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
 +
2022-02-05 10:39:07.231 - INFO:  Statistics for SOP: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
 +
2022-02-05 10:39:07.231 - INFO:  Computing predicate patterns ...
 +
2022-02-05 12:44:22.856 - INFO:  Number of patterns: 6,791,592
 +
2022-02-05 12:44:22.856 - INFO:  Number of subjects with pattern: 2,683,434,680 [all]
 +
2022-02-05 12:44:22.856 - INFO:  Total number of distinct subject-predicate pairs: 10,793,505,342
 +
2022-02-05 12:44:22.856 - INFO:  Average number of predicates per subject: 4.0
 +
2022-02-05 12:44:22.874 - INFO:  Average number of subjects per predicate: 243,146
 +
2022-02-05 13:03:47.890 - INFO:  Exchanging multiplicities for SPO and SOP ...
 +
2022-02-05 14:40:43.459 - INFO:  Writing meta data for SPO and SOP ...
 +
2022-02-05 14:40:45.868 - INFO:  Sorting for OSP permutation ...
 +
2022-02-05 23:26:18.899 - INFO:  Creating a pair of index permutations ...
 +
2022-02-06 06:46:43.742 - INFO:  Statistics for OSP: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
 +
2022-02-06 06:46:43.746 - INFO:  Statistics for OPS: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
 +
2022-02-06 06:46:43.746 - INFO:  Exchanging multiplicities for OSP and OPS ...
 +
2022-02-06 08:02:47.855 - INFO:  Writing meta data for OSP and OPS ...
 +
2022-02-06 08:02:50.547 - INFO:  Finished writing permutations
 +
2022-02-06 08:02:50.570 - INFO:  Index build completed
 +
</source>
 +
== Starting Server (3 min)==
 +
Starting the server failed on the 64 GB machine. I had to get myself a 128 GB machine - a used Dell PowerEdge R620 was
 +
the starting point. I had to copy the indexed data from the other machine which took some 2 1/2 h over a 1 GBit network. Then the server would start. During start the memory footprint went up to approx 75 GB and then down to 50 GB.
 +
<source lang='bash'>
 +
wf@sun:/hd/seel/qlever$ ./qlever -p -s
 +
pulling qlever docker image started at Fr 11. Feb 20:46:42 CET 2022
 +
Using default tag: latest
 +
latest: Pulling from adfreiburg/qlever
 +
Digest: sha256:984cf99ac0b2b804fd7035ff15e2362acd19af6c58b653a4aac3e52e7273b2cd
 +
Status: Image is up to date for adfreiburg/qlever:latest
 +
docker.io/adfreiburg/qlever:latest
 +
pulling qlever docker image finished at Fr 11. Feb 20:46:43 CET 2022 after 1 seconds
 +
2022-02-11 19:46:44.074 - INFO:  QLever Server, compiled on Feb  3 2022 19:24:25
 +
2022-02-11 19:46:44.075 - INFO:  Initializing server ...
 +
2022-02-11 19:46:44.096 - INFO:  Reading internal vocabulary from file /index/wikidata.vocabulary ...
 +
2022-02-11 19:47:15.816 - INFO:  Done, number of words: 938,491,990
 +
2022-02-11 19:47:15.819 - INFO:  Number of words in external vocabulary: 2,289,883,113
 +
2022-02-11 19:47:15.849 - INFO:  Registered PSO permutation: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.872 - INFO:  Registered POS permutation: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.881 - INFO:  Registered OPS permutation: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.888 - INFO:  Registered OSP permutation: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.896 - INFO:  Registered SPO permutation: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.903 - INFO:  Registered SOP permutation: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
 +
2022-02-11 19:47:15.903 - INFO:  Reading patterns from file /index/wikidata.index.patterns ...
 +
2022-02-11 19:49:20.619 - INFO:  Sorting random result tables to estimate the sorting performance of this machine ...
 +
2022-02-11 19:50:41.147 - INFO:  The server is ready
 +
</source>
 +
 +
== Issues ==
 +
see
 +
* https://github.com/ad-freiburg/qlever/issues/562
 +
* https://github.com/ad-freiburg/qlever/issues/563
 +
 
[[Category:WikiData]]
 
[[Category:WikiData]]

Latest revision as of 07:03, 15 May 2023


QLever trial

Import
edit
state  ✅
url  https://wiki.bitplan.com/index.php/WikiData_Import_2022-01-29
target  QLever
start  2022-02-02
end  2022-02-06
days  4
os  Ubuntu 20.04.3 LTS
cpu  Quad-Core AMD Opteron(tm) Processor 2374 HE
ram  64
triples  16.9
comment  


This attempt was successful ✅ after copying the index result to a machine with 128 GB for running.

see https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md

see QLever/script as discussed in QLever Issue #562 for the script which makes reproducing this attempt easier.

Environment

Operating System

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

docker version

docker --version
Docker version 19.03.13, build 4484c46d9d

Memory

wf@merkur:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi       1,3Gi        60Gi        30Mi       1,4Gi        60Gi
Swap:          18Gi          0B        18Gi

diskspace

 vendor               | model              | size      | days  | years |
----------------------+--------------------+-----------+-------+-------+
  ST6000VX0023-2EF110 |                    |    6,00TB |   300 |   0.8 |
              SanDisk |     SSD PLUS 120GB |     120GB |   237 |   0.6 |
                  WDC |   WD20EARS-00MVWB1 |    2,00TB |  2232 |   6.1 |
   ST8000DM004-2U9188 |                    |    8,00TB |     8 |   0.0 |

Model Family:     Seagate Barracuda Compute
Device Model:     ST8000DM004-2U9188
Firmware Version: 0001
User Capacity:    8.001.563.222.016 bytes [8,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches

Steps

The steps below maybe automated with the QLever/script

git clone (1 min)

export QLEVER_HOME=$(pwd)
date;git clone --recursive https://github.com/ad-freiburg/qlever qlever-code;date
Sa 29. Jan 17:12:45 CET 2022
Cloning into 'qlever-code'...
remote: Enumerating objects: 12917, done.
remote: Counting objects: 100% (622/622), done.
remote: Compressing objects: 100% (514/514), done.
remote: Total 12917 (delta 382), reused 206 (delta 103), pack-reused 12295
Receiving objects: 100% (12917/12917), 111.10 MiB | 6.81 MiB/s, done.
Resolving deltas: 100% (10067/10067), done.
Submodule 'third_party/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'third_party/abseil-cpp'
Submodule 'third_party/antlr4' (https://github.com/antlr/antlr4.git) registered for path 'third_party/antlr4'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
Submodule 'third_party/re2' (https://github.com/google/re2.git) registered for path 'third_party/re2'
Submodule 'third_party/stxxl' (https://github.com/ad-freiburg/stxxl) registered for path 'third_party/stxxl'
Cloning into '/hd/jurob/qlever/qlever-code/third_party/abseil-cpp'...
remote: Enumerating objects: 16122, done.        
remote: Counting objects: 100% (248/248), done.        
remote: Compressing objects: 100% (193/193), done.        
remote: Total 16122 (delta 116), reused 106 (delta 55), pack-reused 15874        
Receiving objects: 100% (16122/16122), 10.39 MiB | 6.85 MiB/s, done.
Resolving deltas: 100% (12398/12398), done.
Cloning into '/hd/jurob/qlever/qlever-code/third_party/antlr4'...
remote: Enumerating objects: 125326, done.        
remote: Counting objects: 100% (2734/2734), done.        
remote: Compressing objects: 100% (1022/1022), done.        
remote: Total 125326 (delta 1488), reused 2292 (delta 1246), pack-reused 122592        
Receiving objects: 100% (125326/125326), 64.11 MiB | 6.72 MiB/s, done.
Resolving deltas: 100% (73647/73647), done.
Cloning into '/hd/jurob/qlever/qlever-code/third_party/googletest'...
remote: Enumerating objects: 23795, done.        
remote: Counting objects: 100% (259/259), done.        
remote: Compressing objects: 100% (146/146), done.        
remote: Total 23795 (delta 134), reused 200 (delta 105), pack-reused 23536        
Receiving objects: 100% (23795/23795), 9.90 MiB | 6.89 MiB/s, done.
Resolving deltas: 100% (17517/17517), done.
Cloning into '/hd/jurob/qlever/qlever-code/third_party/re2'...
remote: Enumerating objects: 6996, done.        
remote: Counting objects: 100% (617/617), done.        
remote: Compressing objects: 100% (388/388), done.        
remote: Total 6996 (delta 392), reused 399 (delta 223), pack-reused 6379        
Receiving objects: 100% (6996/6996), 3.49 MiB | 6.94 MiB/s, done.
Resolving deltas: 100% (5287/5287), done.
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl'...
remote: Enumerating objects: 40982, done.        
remote: Counting objects: 100% (45/45), done.        
remote: Compressing objects: 100% (31/31), done.        
remote: Total 40982 (delta 15), reused 28 (delta 7), pack-reused 40937        
Receiving objects: 100% (40982/40982), 14.13 MiB | 6.77 MiB/s, done.
Resolving deltas: 100% (30914/30914), done.
Submodule path 'third_party/abseil-cpp': checked out 'b9b925341f9e90f5e7aa0cf23f036c29c7e454eb'
Submodule path 'third_party/antlr4': checked out 'e4c1a74c66bd5290364ea2b36c97cd724b247357'
Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
Submodule path 'third_party/re2': checked out '13ebb377c6ad763ca61d12dd6f88b1126bd0b911'
Submodule path 'third_party/stxxl': checked out 'a4f884f2a2b4ea078c34c48e1e9a0003f4619f00'
Submodule 'extlib/foxxll' (https://github.com/ad-freiburg/foxxll.git) registered for path 'third_party/stxxl/extlib/foxxll'
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl/extlib/foxxll'...
remote: Enumerating objects: 21414, done.        
remote: Counting objects: 100% (28/28), done.        
remote: Compressing objects: 100% (22/22), done.        
remote: Total 21414 (delta 9), reused 13 (delta 4), pack-reused 21386        
Receiving objects: 100% (21414/21414), 4.60 MiB | 2.48 MiB/s, done.
Resolving deltas: 100% (15789/15789), done.
Submodule path 'third_party/stxxl/extlib/foxxll': checked out '8cbca7bedcdb0b84a6de99e927c5fa27a4bbbfb2'
Submodule 'extlib/tlx' (https://github.com/joka921/tlx.git) registered for path 'third_party/stxxl/extlib/foxxll/extlib/tlx'
Cloning into '/hd/jurob/qlever/qlever-code/third_party/stxxl/extlib/foxxll/extlib/tlx'...
remote: Enumerating objects: 3418, done.        
remote: Counting objects: 100% (53/53), done.        
remote: Compressing objects: 100% (44/44), done.        
remote: Total 3418 (delta 24), reused 22 (delta 9), pack-reused 3365        
Receiving objects: 100% (3418/3418), 1.11 MiB | 2.75 MiB/s, done.
Resolving deltas: 100% (2611/2611), done.
Submodule path 'third_party/stxxl/extlib/foxxll/extlib/tlx': checked out 'ef81a598d9880cc7d242afc47de7328634f07f1d'
Sa 29. Jan 17:13:49 CET 2022

docker build (15 mins)

cd qlever-code/
wf@merkur:/hd/jurob/qlever/qlever-code$ date;sudo docker build --file Dockerfiles/Dockerfile.Ubuntu20.04 -t qlever .;date
Sa 29. Jan 17:14:51 CET 2022
Sending build context to Docker daemon  126.1MB
Step 1/43 : FROM ubuntu:20.04 as base
...
Successfully tagged qlever:latest
Sa 29. Jan 17:29:27 CET 2022

Wikidata dump Download (6h)

This had been done on another machine a few days earlier. 

date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2;date
Thu Jan 27 11:08:10 CET 2022
--2022-01-27 11:08:10--  https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620::861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620::861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 92422961847 (86G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’

latest-all.ttl.bz2  100%[===================>]  86.08G  4.60MB/s    in 5h 46m  

2022-01-27 16:55:06 (4.23 MB/s) - ‘latest-all.ttl.bz2’ saved [92422961847/92422961847]

Thu Jan 27 16:55:06 CET 2022
date;wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2;date
Thu Jan 27 17:36:38 CET 2022
--2022-01-27 17:36:38--  https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620::861:1:208:80:154:7, 208.80.154.7
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620::861:1:208:80:154:7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 315591211 (301M) [application/octet-stream]
Saving to: ‘latest-lexemes.ttl.bz2’

latest-lexemes.ttl. 100%[===================>] 300.97M  4.95MB/s    in 60s     

2022-01-27 17:37:39 (5.00 MB/s) - ‘latest-lexemes.ttl.bz2’ saved [315591211/315591211]

Thu Jan 27 17:37:39 CET 2022

Pull QLever

./qlever -p --wikidata_index
pulling qlever docker image started at Sa 12. Mär 06:44:34 CET 2022
Using default tag: latest
latest: Pulling from adfreiburg/qlever
ae6e1b672be6: Already exists 
55c87480fb23: Already exists 
37363fee5023: Already exists 
0aef3ecd8c53: Already exists 
d12c5fe208ef: Already exists 
0b69b03c02c1: Already exists 
5bca5e9bf0ae: Pull complete 
532aa782a906: Pull complete 
Digest: sha256:387848d3cf1a3eea56c6d21bdcea18189b950740dd5410356932d7f0cbc2c20a
Status: Downloaded newer image for adfreiburg/qlever:latest
docker.io/adfreiburg/qlever:latest
pulling qlever docker image finished at Sa 12. Mär 06:44:51 CET 2022 after 17 seconds

Indexing 4.1 days

./qlever --wikidata_index
2022-02-02 05:53:42.391	- INFO:  �[1mQLever IndexBuilder, compiled on Jan 29 2022 20:19:12�[22m
2022-02-02 05:53:42.418	- INFO:  You specified the input format: TTL
2022-02-02 05:53:42.566	- INFO:  You specified "locale = en_US" and "ignore-punctuation = 1"
2022-02-02 05:53:42.568	- INFO:  You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files (see qlever/docs on GitHub)
2022-02-02 05:53:42.568	- INFO:  You specified "num-triples-per-partial-vocab = 20,000,000", choose a lower value if the index builder runs out of memory
2022-02-02 05:53:42.569	- INFO:  Processing input triples from /dev/stdin ...
2022-02-02 05:59:07.200	- INFO:  Input triples processed: 100,000,000
2022-02-02 06:04:25.068	- INFO:  Input triples processed: 200,000,000
2022-02-02 06:08:49.023	- INFO:  Input triples processed: 300,000,000
2022-02-02 06:13:42.180	- INFO:  Input triples processed: 400,000,000
2022-02-02 06:18:35.785	- INFO:  Input triples processed: 500,000,000
2022-02-02 06:23:33.098	- INFO:  Input triples processed: 600,000,000
2022-02-02 06:28:40.453	- INFO:  Input triples processed: 700,000,000
2022-02-02 06:33:33.224	- INFO:  Input triples processed: 800,000,000
2022-02-02 06:38:29.646	- INFO:  Input triples processed: 900,000,000
2022-02-02 06:43:31.615	- INFO:  Input triples processed: 1,000,000,000
2022-02-02 06:48:50.864	- INFO:  Input triples processed: 1,100,000,000
2022-02-02 06:54:44.529	- INFO:  Input triples processed: 1,200,000,000
2022-02-02 07:01:03.669	- INFO:  Input triples processed: 1,300,000,000
2022-02-02 07:06:57.283	- INFO:  Input triples processed: 1,400,000,000
2022-02-02 07:13:02.002	- INFO:  Input triples processed: 1,500,000,000
2022-02-02 07:19:02.355	- INFO:  Input triples processed: 1,600,000,000
2022-02-02 07:25:32.057	- INFO:  Input triples processed: 1,700,000,000
2022-02-02 07:31:41.109	- INFO:  Input triples processed: 1,800,000,000
2022-02-02 07:37:39.025	- INFO:  Input triples processed: 1,900,000,000
2022-02-02 07:43:24.237	- INFO:  Input triples processed: 2,000,000,000
2022-02-02 07:48:33.112	- INFO:  Input triples processed: 2,100,000,000
2022-02-02 07:54:15.348	- INFO:  Input triples processed: 2,200,000,000
2022-02-02 07:59:55.743	- INFO:  Input triples processed: 2,300,000,000
2022-02-02 08:05:09.401	- INFO:  Input triples processed: 2,400,000,000
2022-02-02 08:10:38.497	- INFO:  Input triples processed: 2,500,000,000
2022-02-02 08:16:27.476	- INFO:  Input triples processed: 2,600,000,000
2022-02-02 08:21:52.338	- INFO:  Input triples processed: 2,700,000,000
2022-02-02 08:27:46.142	- INFO:  Input triples processed: 2,800,000,000
2022-02-02 08:33:28.374	- INFO:  Input triples processed: 2,900,000,000
2022-02-02 08:39:30.528	- INFO:  Input triples processed: 3,000,000,000
2022-02-02 08:45:08.281	- INFO:  Input triples processed: 3,100,000,000
2022-02-02 08:50:52.437	- INFO:  Input triples processed: 3,200,000,000
2022-02-02 09:03:13.800	- INFO:  Input triples processed: 3,300,000,000
2022-02-02 09:08:56.368	- INFO:  Input triples processed: 3,400,000,000
2022-02-02 09:14:37.689	- INFO:  Input triples processed: 3,500,000,000
2022-02-02 09:20:14.816	- INFO:  Input triples processed: 3,600,000,000
2022-02-02 09:25:46.666	- INFO:  Input triples processed: 3,700,000,000
2022-02-02 09:31:28.112	- INFO:  Input triples processed: 3,800,000,000
2022-02-02 09:36:55.886	- INFO:  Input triples processed: 3,900,000,000
2022-02-02 09:42:28.592	- INFO:  Input triples processed: 4,000,000,000
2022-02-02 09:48:02.848	- INFO:  Input triples processed: 4,100,000,000
2022-02-02 09:53:16.606	- INFO:  Input triples processed: 4,200,000,000
2022-02-02 09:58:40.663	- INFO:  Input triples processed: 4,300,000,000
2022-02-02 10:04:17.846	- INFO:  Input triples processed: 4,400,000,000
2022-02-02 10:09:21.045	- INFO:  Input triples processed: 4,500,000,000
2022-02-02 10:14:34.149	- INFO:  Input triples processed: 4,600,000,000
2022-02-02 10:19:51.519	- INFO:  Input triples processed: 4,700,000,000
2022-02-02 10:25:54.075	- INFO:  Input triples processed: 4,800,000,000
2022-02-02 10:31:12.738	- INFO:  Input triples processed: 4,900,000,000
2022-02-02 10:36:28.453	- INFO:  Input triples processed: 5,000,000,000
2022-02-02 10:41:58.489	- INFO:  Input triples processed: 5,100,000,000
2022-02-02 10:47:19.305	- INFO:  Input triples processed: 5,200,000,000
2022-02-02 10:53:03.096	- INFO:  Input triples processed: 5,300,000,000
2022-02-02 10:58:30.772	- INFO:  Input triples processed: 5,400,000,000
2022-02-02 11:03:53.039	- INFO:  Input triples processed: 5,500,000,000
2022-02-02 11:09:07.835	- INFO:  Input triples processed: 5,600,000,000
2022-02-02 11:14:43.148	- INFO:  Input triples processed: 5,700,000,000
2022-02-02 11:20:30.355	- INFO:  Input triples processed: 5,800,000,000
2022-02-02 11:26:12.168	- INFO:  Input triples processed: 5,900,000,000
2022-02-02 11:31:49.249	- INFO:  Input triples processed: 6,000,000,000
2022-02-02 11:37:42.195	- INFO:  Input triples processed: 6,100,000,000
2022-02-02 11:43:14.152	- INFO:  Input triples processed: 6,200,000,000
2022-02-02 11:48:56.857	- INFO:  Input triples processed: 6,300,000,000
2022-02-02 11:54:14.783	- INFO:  Input triples processed: 6,400,000,000
2022-02-02 11:59:44.683	- INFO:  Input triples processed: 6,500,000,000
2022-02-02 12:04:57.135	- INFO:  Input triples processed: 6,600,000,000
2022-02-02 12:10:12.283	- INFO:  Input triples processed: 6,700,000,000
2022-02-02 12:15:35.776	- INFO:  Input triples processed: 6,800,000,000
2022-02-02 12:20:38.895	- INFO:  Input triples processed: 6,900,000,000
2022-02-02 12:26:11.903	- INFO:  Input triples processed: 7,000,000,000
2022-02-02 12:31:24.482	- INFO:  Input triples processed: 7,100,000,000
2022-02-02 12:36:43.304	- INFO:  Input triples processed: 7,200,000,000
2022-02-02 12:42:26.933	- INFO:  Input triples processed: 7,300,000,000
2022-02-02 12:47:32.521	- INFO:  Input triples processed: 7,400,000,000
2022-02-02 12:53:06.737	- INFO:  Input triples processed: 7,500,000,000
2022-02-02 12:58:50.177	- INFO:  Input triples processed: 7,600,000,000
2022-02-02 13:04:13.764	- INFO:  Input triples processed: 7,700,000,000
2022-02-02 13:09:53.474	- INFO:  Input triples processed: 7,800,000,000
2022-02-02 13:15:30.449	- INFO:  Input triples processed: 7,900,000,000
2022-02-02 13:21:18.666	- INFO:  Input triples processed: 8,000,000,000
2022-02-02 13:27:02.409	- INFO:  Input triples processed: 8,100,000,000
2022-02-02 13:32:37.830	- INFO:  Input triples processed: 8,200,000,000
2022-02-02 13:38:08.812	- INFO:  Input triples processed: 8,300,000,000
2022-02-02 13:43:19.219	- INFO:  Input triples processed: 8,400,000,000
2022-02-02 13:48:43.536	- INFO:  Input triples processed: 8,500,000,000
2022-02-02 13:54:49.380	- INFO:  Input triples processed: 8,600,000,000
2022-02-02 14:00:03.529	- INFO:  Input triples processed: 8,700,000,000
2022-02-02 14:05:07.486	- INFO:  Input triples processed: 8,800,000,000
2022-02-02 14:10:24.776	- INFO:  Input triples processed: 8,900,000,000
2022-02-02 14:16:03.699	- INFO:  Input triples processed: 9,000,000,000
2022-02-02 14:21:17.048	- INFO:  Input triples processed: 9,100,000,000
2022-02-02 14:26:38.329	- INFO:  Input triples processed: 9,200,000,000
2022-02-02 14:32:07.014	- INFO:  Input triples processed: 9,300,000,000
2022-02-02 14:37:35.128	- INFO:  Input triples processed: 9,400,000,000
2022-02-02 14:42:42.646	- INFO:  Input triples processed: 9,500,000,000
2022-02-02 14:47:55.735	- INFO:  Input triples processed: 9,600,000,000
2022-02-02 14:53:25.586	- INFO:  Input triples processed: 9,700,000,000
2022-02-02 14:58:45.841	- INFO:  Input triples processed: 9,800,000,000
2022-02-02 15:04:14.244	- INFO:  Input triples processed: 9,900,000,000
2022-02-02 15:09:45.942	- INFO:  Input triples processed: 10,000,000,000
2022-02-02 15:15:29.977	- INFO:  Input triples processed: 10,100,000,000
2022-02-02 15:21:02.689	- INFO:  Input triples processed: 10,200,000,000
2022-02-02 15:26:50.306	- INFO:  Input triples processed: 10,300,000,000
2022-02-02 15:32:30.487	- INFO:  Input triples processed: 10,400,000,000
2022-02-02 15:37:55.973	- INFO:  Input triples processed: 10,500,000,000
2022-02-02 15:43:21.336	- INFO:  Input triples processed: 10,600,000,000
2022-02-02 15:49:07.198	- INFO:  Input triples processed: 10,700,000,000
2022-02-02 15:54:21.597	- INFO:  Input triples processed: 10,800,000,000
2022-02-02 15:59:26.248	- INFO:  Input triples processed: 10,900,000,000
2022-02-02 16:04:47.254	- INFO:  Input triples processed: 11,000,000,000
2022-02-02 16:10:21.939	- INFO:  Input triples processed: 11,100,000,000
2022-02-02 16:16:17.855	- INFO:  Input triples processed: 11,200,000,000
2022-02-02 16:21:46.377	- INFO:  Input triples processed: 11,300,000,000
2022-02-02 16:27:05.461	- INFO:  Input triples processed: 11,400,000,000
2022-02-02 16:32:31.003	- INFO:  Input triples processed: 11,500,000,000
2022-02-02 16:37:43.081	- INFO:  Input triples processed: 11,600,000,000
2022-02-02 16:43:14.589	- INFO:  Input triples processed: 11,700,000,000
2022-02-02 16:48:55.122	- INFO:  Input triples processed: 11,800,000,000
2022-02-02 16:54:17.942	- INFO:  Input triples processed: 11,900,000,000
2022-02-02 16:59:48.271	- INFO:  Input triples processed: 12,000,000,000
2022-02-02 17:05:31.450	- INFO:  Input triples processed: 12,100,000,000
2022-02-02 17:11:21.730	- INFO:  Input triples processed: 12,200,000,000
2022-02-02 17:17:07.859	- INFO:  Input triples processed: 12,300,000,000
2022-02-02 17:22:38.664	- INFO:  Input triples processed: 12,400,000,000
2022-02-02 17:28:27.188	- INFO:  Input triples processed: 12,500,000,000
2022-02-02 17:33:53.612	- INFO:  Input triples processed: 12,600,000,000
2022-02-02 17:39:21.492	- INFO:  Input triples processed: 12,700,000,000
2022-02-02 17:45:10.469	- INFO:  Input triples processed: 12,800,000,000
2022-02-02 17:50:24.948	- INFO:  Input triples processed: 12,900,000,000
2022-02-02 17:55:29.110	- INFO:  Input triples processed: 13,000,000,000
2022-02-02 18:00:57.257	- INFO:  Input triples processed: 13,100,000,000
2022-02-02 18:06:17.378	- INFO:  Input triples processed: 13,200,000,000
2022-02-02 18:11:47.981	- INFO:  Input triples processed: 13,300,000,000
2022-02-02 18:17:10.888	- INFO:  Input triples processed: 13,400,000,000
2022-02-02 18:22:47.533	- INFO:  Input triples processed: 13,500,000,000
2022-02-02 18:28:01.125	- INFO:  Input triples processed: 13,600,000,000
2022-02-02 18:33:20.535	- INFO:  Input triples processed: 13,700,000,000
2022-02-02 18:39:01.344	- INFO:  Input triples processed: 13,800,000,000
2022-02-02 18:44:15.241	- INFO:  Input triples processed: 13,900,000,000
2022-02-02 18:50:00.600	- INFO:  Input triples processed: 14,000,000,000
2022-02-02 18:55:33.580	- INFO:  Input triples processed: 14,100,000,000
2022-02-02 19:01:14.666	- INFO:  Input triples processed: 14,200,000,000
2022-02-02 19:06:46.602	- INFO:  Input triples processed: 14,300,000,000
2022-02-02 19:12:38.371	- INFO:  Input triples processed: 14,400,000,000
2022-02-02 19:18:41.261	- INFO:  Input triples processed: 14,500,000,000
2022-02-02 19:24:17.925	- INFO:  Input triples processed: 14,600,000,000
2022-02-02 19:29:31.631	- INFO:  Input triples processed: 14,700,000,000
2022-02-02 19:35:08.007	- INFO:  Input triples processed: 14,800,000,000
2022-02-02 19:40:52.112	- INFO:  Input triples processed: 14,900,000,000
2022-02-02 19:46:00.751	- INFO:  Input triples processed: 15,000,000,000
2022-02-02 19:51:10.905	- INFO:  Input triples processed: 15,100,000,000
2022-02-02 19:56:46.037	- INFO:  Input triples processed: 15,200,000,000
2022-02-02 20:02:09.632	- INFO:  Input triples processed: 15,300,000,000
2022-02-02 20:07:34.112	- INFO:  Input triples processed: 15,400,000,000
2022-02-02 20:12:41.231	- INFO:  Input triples processed: 15,500,000,000
2022-02-02 20:18:17.512	- INFO:  Input triples processed: 15,600,000,000
2022-02-02 20:23:50.285	- INFO:  Input triples processed: 15,700,000,000
2022-02-02 20:29:25.067	- INFO:  Input triples processed: 15,800,000,000
2022-02-02 20:34:45.926	- INFO:  Input triples processed: 15,900,000,000
2022-02-02 20:40:08.984	- INFO:  Input triples processed: 16,000,000,000
2022-02-02 20:45:35.751	- INFO:  Input triples processed: 16,100,000,000
2022-02-02 20:51:10.034	- INFO:  Input triples processed: 16,200,000,000
2022-02-02 20:57:02.400	- INFO:  Input triples processed: 16,300,000,000
2022-02-02 21:02:45.497	- INFO:  Input triples processed: 16,400,000,000
2022-02-02 21:08:34.531	- INFO:  Input triples processed: 16,500,000,000
2022-02-02 21:14:14.133	- INFO:  Input triples processed: 16,600,000,000
2022-02-02 21:19:58.155	- INFO:  Input triples processed: 16,700,000,000
2022-02-02 21:25:23.936	- INFO:  Input triples processed: 16,800,000,000
2022-02-02 21:30:23.323	- INFO:  Input triples processed: 16,900,000,000
2022-02-02 21:35:25.492	- INFO:  Done, total number of triples read: 16,979,598,468 [may contain duplicates]
2022-02-02 21:35:25.492	- INFO:  Number of QLever-internal triples created: 10,093,281,840 [may contain duplicates]
2022-02-02 21:35:25.492	- INFO:  Merging partial vocabularies in byte order (internal only) ...
2022-02-02 21:40:46.606	- INFO:  Words merged: 100,000,000
2022-02-02 21:48:22.776	- INFO:  Words merged: 200,000,000
2022-02-02 21:55:53.025	- INFO:  Words merged: 300,000,000
2022-02-02 22:06:50.288	- INFO:  Words merged: 400,000,000
2022-02-02 22:15:20.636	- INFO:  Words merged: 500,000,000
2022-02-02 22:21:49.518	- INFO:  Words merged: 600,000,000
2022-02-02 22:43:44.014	- INFO:  Words merged: 700,000,000
2022-02-02 22:50:53.562	- INFO:  Words merged: 800,000,000
2022-02-02 22:56:18.623	- INFO:  Words merged: 900,000,000
2022-02-02 22:57:48.601	- INFO:  Number of words in internal vocabulary: 938,491,990
2022-02-02 22:57:48.602	- INFO:  Merging partial vocabularies in Unicode order (internal and external) ...
2022-02-02 23:06:56.472	- INFO:  Words merged: 100,000,000
2022-02-02 23:17:33.337	- INFO:  Words merged: 200,000,000
2022-02-02 23:28:18.214	- INFO:  Words merged: 300,000,000
2022-02-02 23:42:45.536	- INFO:  Words merged: 400,000,000
2022-02-02 23:52:58.541	- INFO:  Words merged: 500,000,000
2022-02-03 00:01:16.765	- INFO:  Words merged: 600,000,000
2022-02-03 00:11:23.418	- INFO:  Words merged: 700,000,000
2022-02-03 00:19:46.619	- INFO:  Words merged: 800,000,000
2022-02-03 00:45:36.524	- INFO:  Words merged: 900,000,000
2022-02-03 00:58:09.284	- INFO:  Words merged: 1,000,000,000
2022-02-03 01:06:24.768	- INFO:  Words merged: 1,100,000,000
2022-02-03 01:14:50.087	- INFO:  Words merged: 1,200,000,000
2022-02-03 01:23:12.324	- INFO:  Words merged: 1,300,000,000
2022-02-03 01:33:05.021	- INFO:  Words merged: 1,400,000,000
2022-02-03 01:41:01.953	- INFO:  Words merged: 1,500,000,000
2022-02-03 01:50:09.204	- INFO:  Words merged: 1,600,000,000
2022-02-03 02:02:03.331	- INFO:  Words merged: 1,700,000,000
2022-02-03 02:13:53.724	- INFO:  Words merged: 1,800,000,000
2022-02-03 02:22:53.444	- INFO:  Words merged: 1,900,000,000
2022-02-03 02:32:32.792	- INFO:  Words merged: 2,000,000,000
2022-02-03 02:42:29.268	- INFO:  Words merged: 2,100,000,000
2022-02-03 02:52:16.904	- INFO:  Words merged: 2,200,000,000
2022-02-03 03:02:03.858	- INFO:  Words merged: 2,300,000,000
2022-02-03 03:10:43.495	- INFO:  Words merged: 2,400,000,000
2022-02-03 03:20:05.744	- INFO:  Words merged: 2,500,000,000
2022-02-03 03:29:44.044	- INFO:  Words merged: 2,600,000,000
2022-02-03 03:38:34.892	- INFO:  Words merged: 2,700,000,000
2022-02-03 03:47:25.028	- INFO:  Words merged: 2,800,000,000
2022-02-03 03:56:30.070	- INFO:  Words merged: 2,900,000,000
2022-02-03 04:05:56.032	- INFO:  Words merged: 3,000,000,000
2022-02-03 04:16:10.871	- INFO:  Words merged: 3,100,000,000
2022-02-03 04:30:33.889	- INFO:  Words merged: 3,200,000,000
2022-02-03 04:38:45.525	- INFO:  Number of words in external vocabulary: 2,289,883,113
2022-02-03 04:38:45.531	- INFO:  Removing temporary files ...
2022-02-03 04:44:04.818	- INFO:  Converting external vocabulary to binary format ...
2022-02-03 06:04:44.060	- INFO:  Converting triples from local IDs to global IDs ...
2022-02-03 06:07:03.695	- INFO:  Triples converted: 100,000,000
2022-02-03 06:08:47.794	- INFO:  Triples converted: 200,000,000
2022-02-03 06:10:29.184	- INFO:  Triples converted: 300,000,000
2022-02-03 06:12:20.313	- INFO:  Triples converted: 400,000,000
2022-02-03 06:14:02.908	- INFO:  Triples converted: 500,000,000
2022-02-03 06:15:36.523	- INFO:  Triples converted: 600,000,000
2022-02-03 06:17:07.086	- INFO:  Triples converted: 700,000,000
2022-02-03 06:18:31.008	- INFO:  Triples converted: 800,000,000
2022-02-03 06:19:50.168	- INFO:  Triples converted: 900,000,000
2022-02-03 06:21:21.558	- INFO:  Triples converted: 1,000,000,000
2022-02-03 06:22:52.771	- INFO:  Triples converted: 1,100,000,000
2022-02-03 06:24:32.112	- INFO:  Triples converted: 1,200,000,000
2022-02-03 06:25:50.747	- INFO:  Triples converted: 1,300,000,000
2022-02-03 06:27:14.209	- INFO:  Triples converted: 1,400,000,000
2022-02-03 06:28:50.732	- INFO:  Triples converted: 1,500,000,000
2022-02-03 06:30:33.606	- INFO:  Triples converted: 1,600,000,000
2022-02-03 06:32:01.607	- INFO:  Triples converted: 1,700,000,000
2022-02-03 06:33:27.405	- INFO:  Triples converted: 1,800,000,000
2022-02-03 06:35:04.990	- INFO:  Triples converted: 1,900,000,000
2022-02-03 06:36:33.487	- INFO:  Triples converted: 2,000,000,000
2022-02-03 06:38:27.729	- INFO:  Triples converted: 2,100,000,000
2022-02-03 06:40:19.789	- INFO:  Triples converted: 2,200,000,000
2022-02-03 06:42:23.455	- INFO:  Triples converted: 2,300,000,000
2022-02-03 06:44:20.406	- INFO:  Triples converted: 2,400,000,000
2022-02-03 06:46:02.518	- INFO:  Triples converted: 2,500,000,000
2022-02-03 06:47:43.823	- INFO:  Triples converted: 2,600,000,000
2022-02-03 06:49:18.014	- INFO:  Triples converted: 2,700,000,000
2022-02-03 06:50:58.714	- INFO:  Triples converted: 2,800,000,000
2022-02-03 06:52:29.128	- INFO:  Triples converted: 2,900,000,000
2022-02-03 06:54:09.454	- INFO:  Triples converted: 3,000,000,000
2022-02-03 06:55:54.293	- INFO:  Triples converted: 3,100,000,000
2022-02-03 06:57:27.644	- INFO:  Triples converted: 3,200,000,000
2022-02-03 06:59:03.790	- INFO:  Triples converted: 3,300,000,000
2022-02-03 07:00:43.280	- INFO:  Triples converted: 3,400,000,000
2022-02-03 07:02:46.205	- INFO:  Triples converted: 3,500,000,000
2022-02-03 07:04:29.008	- INFO:  Triples converted: 3,600,000,000
2022-02-03 07:06:13.559	- INFO:  Triples converted: 3,700,000,000
2022-02-03 07:07:49.482	- INFO:  Triples converted: 3,800,000,000
2022-02-03 07:09:18.854	- INFO:  Triples converted: 3,900,000,000
2022-02-03 07:11:10.058	- INFO:  Triples converted: 4,000,000,000
2022-02-03 07:12:48.747	- INFO:  Triples converted: 4,100,000,000
2022-02-03 07:14:31.783	- INFO:  Triples converted: 4,200,000,000
2022-02-03 07:16:14.597	- INFO:  Triples converted: 4,300,000,000
2022-02-03 07:17:47.847	- INFO:  Triples converted: 4,400,000,000
2022-02-03 07:19:27.982	- INFO:  Triples converted: 4,500,000,000
2022-02-03 07:21:19.428	- INFO:  Triples converted: 4,600,000,000
2022-02-03 07:23:09.734	- INFO:  Triples converted: 4,700,000,000
2022-02-03 07:25:06.433	- INFO:  Triples converted: 4,800,000,000
2022-02-03 07:26:47.995	- INFO:  Triples converted: 4,900,000,000
2022-02-03 07:28:21.895	- INFO:  Triples converted: 5,000,000,000
2022-02-03 07:30:04.774	- INFO:  Triples converted: 5,100,000,000
2022-02-03 07:31:42.133	- INFO:  Triples converted: 5,200,000,000
2022-02-03 07:33:36.029	- INFO:  Triples converted: 5,300,000,000
2022-02-03 07:35:23.311	- INFO:  Triples converted: 5,400,000,000
2022-02-03 07:37:02.714	- INFO:  Triples converted: 5,500,000,000
2022-02-03 07:38:42.429	- INFO:  Triples converted: 5,600,000,000
2022-02-03 07:40:13.634	- INFO:  Triples converted: 5,700,000,000
2022-02-03 07:41:56.442	- INFO:  Triples converted: 5,800,000,000
2022-02-03 07:43:39.667	- INFO:  Triples converted: 5,900,000,000
2022-02-03 07:45:23.077	- INFO:  Triples converted: 6,000,000,000
2022-02-03 07:46:55.523	- INFO:  Triples converted: 6,100,000,000
2022-02-03 07:48:32.532	- INFO:  Triples converted: 6,200,000,000
2022-02-03 07:50:03.494	- INFO:  Triples converted: 6,300,000,000
2022-02-03 07:51:37.785	- INFO:  Triples converted: 6,400,000,000
2022-02-03 07:53:15.261	- INFO:  Triples converted: 6,500,000,000
2022-02-03 07:55:03.364	- INFO:  Triples converted: 6,600,000,000
2022-02-03 07:56:46.808	- INFO:  Triples converted: 6,700,000,000
2022-02-03 07:58:34.540	- INFO:  Triples converted: 6,800,000,000
2022-02-03 08:00:20.230	- INFO:  Triples converted: 6,900,000,000
2022-02-03 08:02:20.105	- INFO:  Triples converted: 7,000,000,000
2022-02-03 08:03:54.073	- INFO:  Triples converted: 7,100,000,000
2022-02-03 08:05:25.878	- INFO:  Triples converted: 7,200,000,000
2022-02-03 08:07:00.337	- INFO:  Triples converted: 7,300,000,000
2022-02-03 08:08:31.856	- INFO:  Triples converted: 7,400,000,000
2022-02-03 08:10:10.808	- INFO:  Triples converted: 7,500,000,000
2022-02-03 08:11:52.730	- INFO:  Triples converted: 7,600,000,000
2022-02-03 08:13:20.423	- INFO:  Triples converted: 7,700,000,000
2022-02-03 08:14:48.426	- INFO:  Triples converted: 7,800,000,000
2022-02-03 08:16:19.342	- INFO:  Triples converted: 7,900,000,000
2022-02-03 08:17:37.066	- INFO:  Triples converted: 8,000,000,000
2022-02-03 08:19:06.312	- INFO:  Triples converted: 8,100,000,000
2022-02-03 08:20:24.640	- INFO:  Triples converted: 8,200,000,000
2022-02-03 08:21:51.644	- INFO:  Triples converted: 8,300,000,000
2022-02-03 08:23:20.594	- INFO:  Triples converted: 8,400,000,000
2022-02-03 08:25:06.193	- INFO:  Triples converted: 8,500,000,000
2022-02-03 08:26:48.557	- INFO:  Triples converted: 8,600,000,000
2022-02-03 08:28:14.023	- INFO:  Triples converted: 8,700,000,000
2022-02-03 08:29:43.745	- INFO:  Triples converted: 8,800,000,000
2022-02-03 08:31:02.245	- INFO:  Triples converted: 8,900,000,000
2022-02-03 08:32:32.198	- INFO:  Triples converted: 9,000,000,000
2022-02-03 08:34:53.199	- INFO:  Triples converted: 9,100,000,000
2022-02-03 08:37:01.970	- INFO:  Triples converted: 9,200,000,000
2022-02-03 08:38:46.399	- INFO:  Triples converted: 9,300,000,000
2022-02-03 08:40:24.404	- INFO:  Triples converted: 9,400,000,000
2022-02-03 08:42:26.291	- INFO:  Triples converted: 9,500,000,000
2022-02-03 08:45:21.894	- INFO:  Triples converted: 9,600,000,000
2022-02-03 08:48:12.170	- INFO:  Triples converted: 9,700,000,000
2022-02-03 08:52:29.881	- INFO:  Triples converted: 9,800,000,000
2022-02-03 08:55:36.916	- INFO:  Triples converted: 9,900,000,000
2022-02-03 08:58:56.555	- INFO:  Triples converted: 10,000,000,000
2022-02-03 09:00:24.102	- INFO:  Triples converted: 10,100,000,000
2022-02-03 09:01:57.289	- INFO:  Triples converted: 10,200,000,000
2022-02-03 09:03:38.529	- INFO:  Triples converted: 10,300,000,000
2022-02-03 09:05:27.127	- INFO:  Triples converted: 10,400,000,000
2022-02-03 09:07:00.262	- INFO:  Triples converted: 10,500,000,000
2022-02-03 09:08:29.934	- INFO:  Triples converted: 10,600,000,000
2022-02-03 09:09:54.909	- INFO:  Triples converted: 10,700,000,000
2022-02-03 09:11:31.481	- INFO:  Triples converted: 10,800,000,000
2022-02-03 09:13:08.048	- INFO:  Triples converted: 10,900,000,000
2022-02-03 09:14:45.819	- INFO:  Triples converted: 11,000,000,000
2022-02-03 09:16:25.319	- INFO:  Triples converted: 11,100,000,000
2022-02-03 09:17:49.967	- INFO:  Triples converted: 11,200,000,000
2022-02-03 09:19:25.744	- INFO:  Triples converted: 11,300,000,000
2022-02-03 09:21:11.002	- INFO:  Triples converted: 11,400,000,000
2022-02-03 09:23:01.497	- INFO:  Triples converted: 11,500,000,000
2022-02-03 09:24:52.943	- INFO:  Triples converted: 11,600,000,000
2022-02-03 09:26:21.260	- INFO:  Triples converted: 11,700,000,000
2022-02-03 09:27:49.169	- INFO:  Triples converted: 11,800,000,000
2022-02-03 09:29:26.027	- INFO:  Triples converted: 11,900,000,000
2022-02-03 09:30:53.202	- INFO:  Triples converted: 12,000,000,000
2022-02-03 09:32:18.027	- INFO:  Triples converted: 12,100,000,000
2022-02-03 09:33:59.259	- INFO:  Triples converted: 12,200,000,000
2022-02-03 09:35:26.412	- INFO:  Triples converted: 12,300,000,000
2022-02-03 09:37:15.019	- INFO:  Triples converted: 12,400,000,000
2022-02-03 09:39:11.756	- INFO:  Triples converted: 12,500,000,000
2022-02-03 09:40:47.331	- INFO:  Triples converted: 12,600,000,000
2022-02-03 09:42:30.756	- INFO:  Triples converted: 12,700,000,000
2022-02-03 09:44:14.363	- INFO:  Triples converted: 12,800,000,000
2022-02-03 09:46:09.220	- INFO:  Triples converted: 12,900,000,000
2022-02-03 09:47:48.549	- INFO:  Triples converted: 13,000,000,000
2022-02-03 09:49:23.978	- INFO:  Triples converted: 13,100,000,000
2022-02-03 09:51:25.475	- INFO:  Triples converted: 13,200,000,000
2022-02-03 09:53:00.696	- INFO:  Triples converted: 13,300,000,000
2022-02-03 09:54:56.191	- INFO:  Triples converted: 13,400,000,000
2022-02-03 09:56:46.508	- INFO:  Triples converted: 13,500,000,000
2022-02-03 09:58:21.403	- INFO:  Triples converted: 13,600,000,000
2022-02-03 10:00:09.504	- INFO:  Triples converted: 13,700,000,000
2022-02-03 10:02:17.575	- INFO:  Triples converted: 13,800,000,000
2022-02-03 10:04:02.820	- INFO:  Triples converted: 13,900,000,000
2022-02-03 10:05:44.401	- INFO:  Triples converted: 14,000,000,000
2022-02-03 10:07:17.813	- INFO:  Triples converted: 14,100,000,000
2022-02-03 10:09:10.136	- INFO:  Triples converted: 14,200,000,000
2022-02-03 10:10:56.520	- INFO:  Triples converted: 14,300,000,000
2022-02-03 10:13:01.286	- INFO:  Triples converted: 14,400,000,000
2022-02-03 10:14:50.223	- INFO:  Triples converted: 14,500,000,000
2022-02-03 10:16:40.323	- INFO:  Triples converted: 14,600,000,000
2022-02-03 10:18:24.466	- INFO:  Triples converted: 14,700,000,000
2022-02-03 10:20:11.680	- INFO:  Triples converted: 14,800,000,000
2022-02-03 10:22:01.493	- INFO:  Triples converted: 14,900,000,000
2022-02-03 10:24:08.333	- INFO:  Triples converted: 15,000,000,000
2022-02-03 10:25:41.022	- INFO:  Triples converted: 15,100,000,000
2022-02-03 10:27:25.221	- INFO:  Triples converted: 15,200,000,000
2022-02-03 10:29:00.946	- INFO:  Triples converted: 15,300,000,000
2022-02-03 10:30:52.184	- INFO:  Triples converted: 15,400,000,000
2022-02-03 10:32:38.721	- INFO:  Triples converted: 15,500,000,000
2022-02-03 10:34:11.556	- INFO:  Triples converted: 15,600,000,000
2022-02-03 10:35:57.410	- INFO:  Triples converted: 15,700,000,000
2022-02-03 10:37:44.445	- INFO:  Triples converted: 15,800,000,000
2022-02-03 10:39:16.599	- INFO:  Triples converted: 15,900,000,000
2022-02-03 10:41:02.244	- INFO:  Triples converted: 16,000,000,000
2022-02-03 10:42:39.534	- INFO:  Triples converted: 16,100,000,000
2022-02-03 10:44:19.688	- INFO:  Triples converted: 16,200,000,000
2022-02-03 10:46:13.325	- INFO:  Triples converted: 16,300,000,000
2022-02-03 10:47:55.063	- INFO:  Triples converted: 16,400,000,000
2022-02-03 10:49:42.592	- INFO:  Triples converted: 16,500,000,000
2022-02-03 10:51:27.606	- INFO:  Triples converted: 16,600,000,000
2022-02-03 10:53:25.996	- INFO:  Triples converted: 16,700,000,000
2022-02-03 10:55:20.118	- INFO:  Triples converted: 16,800,000,000
2022-02-03 10:57:21.513	- INFO:  Triples converted: 16,900,000,000
2022-02-03 10:59:16.479	- INFO:  Triples converted: 17,000,000,000
2022-02-03 11:01:11.333	- INFO:  Triples converted: 17,100,000,000
2022-02-03 11:03:04.520	- INFO:  Triples converted: 17,200,000,000
2022-02-03 11:04:40.493	- INFO:  Triples converted: 17,300,000,000
2022-02-03 11:06:28.389	- INFO:  Triples converted: 17,400,000,000
2022-02-03 11:08:26.442	- INFO:  Triples converted: 17,500,000,000
2022-02-03 11:10:22.786	- INFO:  Triples converted: 17,600,000,000
2022-02-03 11:12:36.616	- INFO:  Triples converted: 17,700,000,000
2022-02-03 11:14:18.778	- INFO:  Triples converted: 17,800,000,000
2022-02-03 11:16:13.166	- INFO:  Triples converted: 17,900,000,000
2022-02-03 11:18:10.780	- INFO:  Triples converted: 18,000,000,000
2022-02-03 11:19:51.236	- INFO:  Triples converted: 18,100,000,000
2022-02-03 11:21:40.897	- INFO:  Triples converted: 18,200,000,000
2022-02-03 11:23:21.193	- INFO:  Triples converted: 18,300,000,000
2022-02-03 11:25:03.483	- INFO:  Triples converted: 18,400,000,000
2022-02-03 11:26:57.531	- INFO:  Triples converted: 18,500,000,000
2022-02-03 11:28:27.511	- INFO:  Triples converted: 18,600,000,000
2022-02-03 11:30:11.088	- INFO:  Triples converted: 18,700,000,000
2022-02-03 11:31:54.527	- INFO:  Triples converted: 18,800,000,000
2022-02-03 11:33:36.040	- INFO:  Triples converted: 18,900,000,000
2022-02-03 11:35:17.586	- INFO:  Triples converted: 19,000,000,000
2022-02-03 11:37:06.543	- INFO:  Triples converted: 19,100,000,000
2022-02-03 11:38:49.740	- INFO:  Triples converted: 19,200,000,000
2022-02-03 11:40:42.584	- INFO:  Triples converted: 19,300,000,000
2022-02-03 11:42:30.329	- INFO:  Triples converted: 19,400,000,000
2022-02-03 11:44:19.331	- INFO:  Triples converted: 19,500,000,000
2022-02-03 11:46:00.269	- INFO:  Triples converted: 19,600,000,000
2022-02-03 11:47:37.169	- INFO:  Triples converted: 19,700,000,000
2022-02-03 11:49:38.635	- INFO:  Triples converted: 19,800,000,000
2022-02-03 11:51:22.625	- INFO:  Triples converted: 19,900,000,000
2022-02-03 11:53:26.662	- INFO:  Triples converted: 20,000,000,000
2022-02-03 11:55:20.818	- INFO:  Triples converted: 20,100,000,000
2022-02-03 11:56:57.081	- INFO:  Triples converted: 20,200,000,000
2022-02-03 11:58:44.907	- INFO:  Triples converted: 20,300,000,000
2022-02-03 12:00:41.842	- INFO:  Triples converted: 20,400,000,000
2022-02-03 12:02:39.025	- INFO:  Triples converted: 20,500,000,000
2022-02-03 12:04:19.605	- INFO:  Triples converted: 20,600,000,000
2022-02-03 12:06:05.027	- INFO:  Triples converted: 20,700,000,000
2022-02-03 12:07:47.944	- INFO:  Triples converted: 20,800,000,000
2022-02-03 12:09:42.388	- INFO:  Triples converted: 20,900,000,000
2022-02-03 12:11:38.584	- INFO:  Triples converted: 21,000,000,000
2022-02-03 12:13:28.775	- INFO:  Triples converted: 21,100,000,000
2022-02-03 12:15:22.070	- INFO:  Triples converted: 21,200,000,000
2022-02-03 12:17:03.204	- INFO:  Triples converted: 21,300,000,000
2022-02-03 12:18:44.107	- INFO:  Triples converted: 21,400,000,000
2022-02-03 12:20:20.415	- INFO:  Triples converted: 21,500,000,000
2022-02-03 12:22:13.964	- INFO:  Triples converted: 21,600,000,000
2022-02-03 12:24:18.256	- INFO:  Triples converted: 21,700,000,000
2022-02-03 12:26:08.881	- INFO:  Triples converted: 21,800,000,000
2022-02-03 12:27:48.096	- INFO:  Triples converted: 21,900,000,000
2022-02-03 12:29:18.359	- INFO:  Triples converted: 22,000,000,000
2022-02-03 12:31:01.268	- INFO:  Triples converted: 22,100,000,000
2022-02-03 12:32:45.265	- INFO:  Triples converted: 22,200,000,000
2022-02-03 12:34:28.333	- INFO:  Triples converted: 22,300,000,000
2022-02-03 12:36:18.373	- INFO:  Triples converted: 22,400,000,000
2022-02-03 12:38:12.105	- INFO:  Triples converted: 22,500,000,000
2022-02-03 12:40:03.848	- INFO:  Triples converted: 22,600,000,000
2022-02-03 12:41:37.959	- INFO:  Triples converted: 22,700,000,000
2022-02-03 12:43:24.983	- INFO:  Triples converted: 22,800,000,000
2022-02-03 12:45:20.230	- INFO:  Triples converted: 22,900,000,000
2022-02-03 12:47:20.548	- INFO:  Triples converted: 23,000,000,000
2022-02-03 12:49:18.133	- INFO:  Triples converted: 23,100,000,000
2022-02-03 12:51:04.519	- INFO:  Triples converted: 23,200,000,000
2022-02-03 12:52:58.685	- INFO:  Triples converted: 23,300,000,000
2022-02-03 12:54:54.395	- INFO:  Triples converted: 23,400,000,000
2022-02-03 12:56:42.684	- INFO:  Triples converted: 23,500,000,000
2022-02-03 12:58:30.014	- INFO:  Triples converted: 23,600,000,000
2022-02-03 13:00:44.041	- INFO:  Triples converted: 23,700,000,000
2022-02-03 13:02:41.601	- INFO:  Triples converted: 23,800,000,000
2022-02-03 13:04:31.145	- INFO:  Triples converted: 23,900,000,000
2022-02-03 13:06:17.909	- INFO:  Triples converted: 24,000,000,000
2022-02-03 13:07:58.562	- INFO:  Triples converted: 24,100,000,000
2022-02-03 13:09:45.479	- INFO:  Triples converted: 24,200,000,000
2022-02-03 13:11:44.530	- INFO:  Triples converted: 24,300,000,000
2022-02-03 13:13:41.162	- INFO:  Triples converted: 24,400,000,000
2022-02-03 13:15:37.924	- INFO:  Triples converted: 24,500,000,000
2022-02-03 13:17:24.356	- INFO:  Triples converted: 24,600,000,000
2022-02-03 13:19:08.196	- INFO:  Triples converted: 24,700,000,000
2022-02-03 13:21:08.233	- INFO:  Triples converted: 24,800,000,000
2022-02-03 13:23:08.604	- INFO:  Triples converted: 24,900,000,000
2022-02-03 13:25:27.663	- INFO:  Triples converted: 25,000,000,000
2022-02-03 13:27:17.755	- INFO:  Triples converted: 25,100,000,000
2022-02-03 13:29:03.092	- INFO:  Triples converted: 25,200,000,000
2022-02-03 13:30:52.609	- INFO:  Triples converted: 25,300,000,000
2022-02-03 13:33:00.160	- INFO:  Triples converted: 25,400,000,000
2022-02-03 13:34:57.731	- INFO:  Triples converted: 25,500,000,000
2022-02-03 13:36:47.772	- INFO:  Triples converted: 25,600,000,000
2022-02-03 13:38:25.167	- INFO:  Triples converted: 25,700,000,000
2022-02-03 13:40:28.092	- INFO:  Triples converted: 25,800,000,000
2022-02-03 13:42:32.451	- INFO:  Triples converted: 25,900,000,000
2022-02-03 13:44:18.502	- INFO:  Triples converted: 26,000,000,000
2022-02-03 13:46:08.486	- INFO:  Triples converted: 26,100,000,000
2022-02-03 13:47:45.104	- INFO:  Triples converted: 26,200,000,000
2022-02-03 13:49:28.084	- INFO:  Triples converted: 26,300,000,000
2022-02-03 13:51:22.885	- INFO:  Triples converted: 26,400,000,000
2022-02-03 13:53:19.507	- INFO:  Triples converted: 26,500,000,000
2022-02-03 13:55:14.153	- INFO:  Triples converted: 26,600,000,000
2022-02-03 13:57:00.683	- INFO:  Triples converted: 26,700,000,000
2022-02-03 13:58:45.965	- INFO:  Triples converted: 26,800,000,000
2022-02-03 14:00:22.233	- INFO:  Triples converted: 26,900,000,000
2022-02-03 14:02:02.021	- INFO:  Triples converted: 27,000,000,000
2022-02-03 14:03:30.586	- INFO:  Done, total number of triples converted: 27,072,880,308
2022-02-03 14:03:30.791	- INFO:  Building prefix tree from internal vocabulary ...
2022-02-03 14:04:11.957	- INFO:  Words processed: 100,000,000
2022-02-03 14:05:17.470	- INFO:  Words processed: 200,000,000
2022-02-03 14:06:28.451	- INFO:  Words processed: 300,000,000
2022-02-03 14:07:43.608	- INFO:  Words processed: 400,000,000
2022-02-03 14:09:12.898	- INFO:  Words processed: 500,000,000
2022-02-03 14:10:26.170	- INFO:  Words processed: 600,000,000
2022-02-03 14:11:27.474	- INFO:  Words processed: 700,000,000
2022-02-03 14:13:15.041	- INFO:  Words processed: 800,000,000
2022-02-03 14:14:46.773	- INFO:  Words processed: 900,000,000
2022-02-03 14:15:17.044	- INFO:  Computing maximally compressing prefixes (greedy algorithm) ...
2022-02-03 15:16:53.604	- INFO:  Reduction of size of internal vocabulary: 46%
2022-02-03 15:18:21.559	- INFO:  Writing compressed vocabulary to disk ...
2022-02-03 15:40:47.936	- INFO:  Sorting for PSO permutation ...
2022-02-04 01:01:48.418	- INFO:  Removing duplicate triples ...
2022-02-04 11:00:37.645	- INFO:  Done, number of distinct triples is: 22,650,509,456
2022-02-04 11:00:37.646	- INFO:  Creating a pair of index permutations ... 
2022-02-04 19:48:26.759	- INFO:  Statistics for PSO: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
2022-02-04 19:48:26.777	- INFO:  Statistics for POS: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
2022-02-04 19:48:26.777	- INFO:  Exchanging multiplicities for PSO and POS ...
2022-02-04 19:48:26.803	- INFO:  Writing meta data for PSO and POS ...
2022-02-04 19:48:26.866	- INFO:  Sorting for SPO permutation ...
2022-02-05 04:18:41.117	- INFO:  Creating a pair of index permutations ... 
2022-02-05 10:39:07.231	- INFO:  Statistics for SPO: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
2022-02-05 10:39:07.231	- INFO:  Statistics for SOP: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
2022-02-05 10:39:07.231	- INFO:  Computing predicate patterns ...
2022-02-05 12:44:22.856	- INFO:  Number of patterns: 6,791,592
2022-02-05 12:44:22.856	- INFO:  Number of subjects with pattern: 2,683,434,680 [all]
2022-02-05 12:44:22.856	- INFO:  Total number of distinct subject-predicate pairs: 10,793,505,342
2022-02-05 12:44:22.856	- INFO:  Average number of predicates per subject: 4.0
2022-02-05 12:44:22.874	- INFO:  Average number of subjects per predicate: 243,146
2022-02-05 13:03:47.890	- INFO:  Exchanging multiplicities for SPO and SOP ...
2022-02-05 14:40:43.459	- INFO:  Writing meta data for SPO and SOP ...
2022-02-05 14:40:45.868	- INFO:  Sorting for OSP permutation ...
2022-02-05 23:26:18.899	- INFO:  Creating a pair of index permutations ... 
2022-02-06 06:46:43.742	- INFO:  Statistics for OSP: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
2022-02-06 06:46:43.746	- INFO:  Statistics for OPS: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
2022-02-06 06:46:43.746	- INFO:  Exchanging multiplicities for OSP and OPS ...
2022-02-06 08:02:47.855	- INFO:  Writing meta data for OSP and OPS ...
2022-02-06 08:02:50.547	- INFO:  Finished writing permutations
2022-02-06 08:02:50.570	- INFO:  Index build completed

Starting Server (3 min)

Starting the server failed on the 64 GB machine. I had to get myself a 128 GB machine - a used Dell PowerEdge R620 was the starting point. I had to copy the indexed data from the other machine which took some 2 1/2 h over a 1 GBit network. Then the server would start. During start the memory footprint went up to approx 75 GB and then down to 50 GB. 

wf@sun:/hd/seel/qlever$ ./qlever -p -s
pulling qlever docker image started at Fr 11. Feb 20:46:42 CET 2022
Using default tag: latest
latest: Pulling from adfreiburg/qlever
Digest: sha256:984cf99ac0b2b804fd7035ff15e2362acd19af6c58b653a4aac3e52e7273b2cd
Status: Image is up to date for adfreiburg/qlever:latest
docker.io/adfreiburg/qlever:latest
pulling qlever docker image finished at Fr 11. Feb 20:46:43 CET 2022 after 1 seconds
2022-02-11 19:46:44.074	- INFO:  QLever Server, compiled on Feb  3 2022 19:24:25
2022-02-11 19:46:44.075	- INFO:  Initializing server ...
2022-02-11 19:46:44.096	- INFO:  Reading internal vocabulary from file /index/wikidata.vocabulary ...
2022-02-11 19:47:15.816	- INFO:  Done, number of words: 938,491,990
2022-02-11 19:47:15.819	- INFO:  Number of words in external vocabulary: 2,289,883,113
2022-02-11 19:47:15.849	- INFO:  Registered PSO permutation: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
2022-02-11 19:47:15.872	- INFO:  Registered POS permutation: #relations = 59,862, #blocks = 44,995, #triples = 22,650,509,456
2022-02-11 19:47:15.881	- INFO:  Registered OPS permutation: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
2022-02-11 19:47:15.888	- INFO:  Registered OSP permutation: #relations = 3,046,826,131, #blocks = 38,121, #triples = 22,650,509,456
2022-02-11 19:47:15.896	- INFO:  Registered SPO permutation: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
2022-02-11 19:47:15.903	- INFO:  Registered SOP permutation: #relations = 2,683,434,680, #blocks = 28,807, #triples = 22,650,509,456
2022-02-11 19:47:15.903	- INFO:  Reading patterns from file /index/wikidata.index.patterns ...
2022-02-11 19:49:20.619	- INFO:  Sorting random result tables to estimate the sorting performance of this machine ...
2022-02-11 19:50:41.147	- INFO:  The server is ready

Issues

see

Retrieved from "https://wiki.bitplan.com/index.php?title=WikiData_Import_2022-01-29&oldid=17619"