Get your own copy of WikiData

From BITPlan Wiki
Revision as of 07:20, 5 February 2024 by Wf (talk | contribs) (→‎Reports)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Why would you want your own Wikidata copy?

The resources behind https://query.wikidata.org/ are scarce and used by a lot of people. You might hit the https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/query_limits quite quickly.

See SPARQL for some examples that work online (mostly) without hitting these limits.

What are alternative endpoints ?

Prerequisites

Getting a copy of Wikidata is not for the faint of heart.

You need quite a bit of patience and some hardware resources to get your own WikiData copy working. The resources you need are a moving target since Wikidata is growing all the time.

On the other hand solutions such as QLever by Hannah Bast are making progress so that you can run your own copy of Wikidata on commodity hardware with an AMD Ryzen 9 16 core processor, 32 GB of RAM a 2 TTB SSD and the indexing will take less than 5 h. Together with the download time of some 6 hours that's less than half a day for getting a current copy so a daily update is feasible these days.

Papers

Getting and hosting your own copy of Wikidata Proceedings of the 3rd Wikidata Workshop 2022 co-located with the 21st International Semantic Web Conference ( ISWC2022 ) CEUR-WS Volume 3262

Successes

Please contact Wolfgang Fahl if you'd like to see your own success report added to the "Reports" table below. The imports table is generated from the documentation of our own Wikidata import trials in this semantic mediawiki.

Reports

# Date Source Target Triples days RAM GB CPU Cores Speed Link
2024-01-29 latest-all.nt (2024-01-29) QLever 19.1 billion 4.5 hours 32 16 AMD Ryzen 9 Hannah Bast - QLever
2023-01 James Hare - Blazegraph & QLever Blazegraph & QLever ? ? ? ? 384 GB James Hare Scatter LLC
2022-07 latest-all.ttl (2022-07-12) stardog 17.2 billion 1 d 19 h 253 Tim Holzheim - BITPlan Wiki
2022-06 latest-all.nt (2022-06-25) QLever 17.2 billion 1 d 2 h 128 8 1.8 GHz Wolfgang Fahl - BITPlan Wiki
2022-05 latest-all.ttl.bz2 (2022-05-29) QLever ~17 billion 14h 128 12/24 4.8 GHz boost Hannah Bast - QLever
2022-02 latest-all.nt (2022-02) stardog 16.7 billion 9h Evren Sirin - stardog
2022-02 latest-all.nt (2022-01-29) QLever 16.9 billion 4 d 2 h 127 8 1.8 GHz Wolfgang Fahl - BITPlan Wiki
2020-08 latest-all.nt (2020-08-15) Apache Jena 13.8 billion 9 d 21 h 64 Wolfgang Fahl BITPlan Wiki
2020-07 latest-truthy.nt (2020-07-15) Apache Jena 5.2 billion 4 d 14 h 64 Wolfgang Fahl BITPlan Wiki
2020-06 latest-all.ttl (2020-04-28) Apache Jena 12.9 billion 6 d 16 h ? Jonas Sourlier - Jena Issue 1909
2020-03 latest-all.nt.bz2 (2020-03-01 Virtuoso ~11.8 billion 10 hours + 1day prep 248 Hugh Williams - Virtuoso
2019-10 blazegraph ~10 billion 5.5 d 104 16 Adam Shoreland Wikimedia Foundation
2019-09 latest-all.ttl (2019-09) Virtuoso 9.5 billion 9.1 hours ? Adam Sanchez - WikiData mailing list
2019-05 wikidata-20190513-all-BETA.ttl Virtuoso ? 43 hours ? -
2019-05 wikidata-20190513-all-BETA.ttl Blazegraph ? 10.2 days Adam Sanchez WikiData mailing list
2019-02 latest-all.ttl.gz Apache Jena ? > 2 days ? corsin - muncca blog
2018-01 wikidata-20180101-all-BETA.ttl Blazegraph 3 billion 4 days 32 4 2.2 GHz Wolfgang Fahl - BITPlan wiki
2017-12 latest-truthy.nt.gz Apache Jena ? 8 hours ? Andy Seaborne Apache Jena Mailinglist

Imports on our own hardware

Importstateurltargetstartenddaysoscpuramtriples
Wikidata Import 2025-08-19❌https://wiki.bitplan.com/index.php/Wikidata Import 2025-08-19QLever19 August 2025Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2025-07-30❌https://wiki.bitplan.com/index.php/Wikidata Import 2025-07-30QLever30 July 20251 August 2025Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2025-07-13❌https://wiki.bitplan.com/index.php/Wikidata Import 2025-07-13QLever13 July 202513 July 2025Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2025-06-07✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-06-07QLever7 June 20258 June 20251Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2025-06-06✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-06-06blazegraph6 June 20257 June 20251Ubuntu 22.04.5 LTSAMD Ryzen 9 5900X 12-Core Processor128
Wikidata Import 2025-06-02✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-06-02QLever2 June 20253 June 2025Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51220,484,725,007
Wikidata Import 2025-05-03✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-05-03QLever3 May 20254 May 20250.6Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51220,096,524,609
Wikidata Import 2025-05-02✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-05-02blazegraph2 May 20255 June 202534Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216,836,266,114
Wikidata Import 2025-03-31✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-03-31QLever31 March 20251 April 20250.6Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2025-02-13✅https://wiki.bitplan.com/index.php/Wikidata Import 2025-02-13QLever13 February 2025Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2024-11-23✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-11-23QLever23 November 202424 November 20240.7Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata Import 2024-11-16❌https://wiki.bitplan.com/index.php/Wikidata Import 2024-11-16QLever16 November 2024Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata Import 2024-11-15❌https://wiki.bitplan.com/index.php/Wikidata Import 2024-11-15QLever15 November 2024Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata import 2024-10-28 Virtuoso✅https://wiki.bitplan.com/index.php?title=Wikidata import 2024-10-28 VirtuosoVirtuoso29 October 20243 November 20244Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata Import 2024-10-26✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-10-26QLever26 October 202427 October 20240.9Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata Import 2024-10-24❌https://wiki.bitplan.com/index.php/Wikidata Import 2024-10-24QLever24 October 2024Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2024-10-17✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-10-17QLever17 October 202418 October 20240.9Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)51216.3
Wikidata Import 2024-04-13✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-04-13QLever13 April 202413 April 2024Ubuntu 22.04.3 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (16 cores)512
Wikidata Import 2024-02-18✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-02-18QLever18 February 202418 February 20240.5Ubuntu 22.04.3 LTSAMD Ryzen 9 5900X 12-Core Processor @ 4.95GHz12815.5
Wikidata Import 2024-01-20✅https://wiki.bitplan.com/index.php/Wikidata Import 2024-01-20QLever20 January 202421 January 20240.5Ubuntu 22.04.2 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz25619.1
Wikidata Import 2023-05-15✅https://wiki.bitplan.com/index.php/Wikidata Import 2023-05-15QLever15 May 2023Ubuntu 22.04.2 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz256
Wikidata Import 2023-05-14❌https://wiki.bitplan.com/index.php/Wikidata Import 2023-05-14blazegraph14 May 2023Ubuntu 22.04.2 LTSIntel(R) Core(TM) i5-3570K CPU @ 3.40GHz32
Wikidata Import 2023-05-10❌https://wiki.bitplan.com/index.php/Wikidata Import 2023-05-10blazegraph10 May 2023Ubuntu 22.04.2 LTSIntel(R) Xeon(R) CPU X5690@3.47GHz6414.7
Wikidata Import 2023-05-05✅https://wiki.bitplan.com/index.php/Wikidata Import 2023-05-05blazegraph5 May 2023Ubuntu 22.04.2 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz25614.7
Wikidata Import 2023-05-03✅https://wiki.bitplan.com/index.php/Wikidata Import 2023-05-03blazegraph3 May 202326 June 202323Ubuntu 20.04.6 LTSIntel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz12814.7
Wikidata Import 2023-04-26❌https://wiki.bitplan.com/index.php/Wikidata Import 2023-04-26blazegraph26 April 2023Ubuntu 22.04.2 LTSIntel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz25614.7
Wikidata Import 2023-04-18❌https://wiki.bitplan.com/index.php/Wikidata Import 2023-04-18blazegraph18 April 202318 April 2023Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz25614.6
Wikidata Import 2023-01-24https://wiki.bitplan.com/index.php/Wikidata Import 2023-01-24QLever24 January 2023
WikiData Import 2022-07-20https://wiki.bitplan.com/index.php/WikiData Import 2022-07-20virtuoso20 July 2022
WikiData Import 2022-07-12✅https://wiki.bitplan.com/index.php/Wikidata On StardogStardog11 July 202214 July 20223256
Wikidata On Stardog✅https://wiki.bitplan.com/index.php/Wikidata On StardogStardog11 July 202214 July 20223256
WikiData Import 2022-06-25✅https://wiki.bitplan.com/index.php/WikiData Import 2022-06-25QLever25 June 202227 June 20221.1Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz128
Wikidata on Allegrograph✅https://wiki.bitplan.com/index.php/Wikidata on AllegrographAllegrograph15 April 20220.3CentOS Linux release 7.9.2009AMD EPYC 7302, 3.0GHz25616.8
WikiData Import 2022-03-11❌https://wiki.bitplan.com/index.php/WikiData Import 2022-03-11QLever12 March 202212 March 2022
WikiData Import 2022-01-29✅https://wiki.bitplan.com/index.php/WikiData Import 2022-01-29QLever2 February 20226 February 20224Ubuntu 20.04.3 LTSQuad-Core AMD Opteron(tm) Processor 2374 HE6416.9
Wikidata Import 2018-01-05✅https://wiki.bitplan.com/index.php/Wikidata Import 2018-01-05blazegraph5 January 2018Quad-Core AMD Opteron(tm) Processor 2374 HE

Links

Questions

Retrieved from "https://wiki.bitplan.com/index.php?title=Get_your_own_copy_of_WikiData&oldid=18362"