Difference between revisions of "Wikidata Import 2023-04-18"

From BITPlan Wiki
Jump to navigation Jump to search
Line 1: Line 1:
 
{{PageSequence|prev=Wikidata Import 2023-01-24|next=Wikidata Import 2023-04-26|category=Wikidata|categoryIcon=cloud-download}}
 
{{PageSequence|prev=Wikidata Import 2023-01-24|next=Wikidata Import 2023-04-26|category=Wikidata|categoryIcon=cloud-download}}
 
❌ This attempt failed - the mirror at your.org is not active any more and should not be used
 
❌ This attempt failed - the mirror at your.org is not active any more and should not be used
 +
=Import=
 +
 +
{{Import
 +
|target=blazegraph
 +
|start=2023-04-18
 +
|triples=14.6
 +
|storemode=property
 +
}}
 
= Download latest wikidata Dump ~2 hours =
 
= Download latest wikidata Dump ~2 hours =
 
== Download Options ==
 
== Download Options ==

Revision as of 07:07, 12 May 2023

❌ This attempt failed - the mirror at your.org is not active any more and should not be used

Import

Import
edit
state  
url  
target  blazegraph
start  2023-04-18
end  
days  
os  
cpu  
ram  
triples  14.6
comment  

Download latest wikidata Dump ~2 hours

Download Options

https://dumps.wikimedia.org/wikidatawiki/entities

latest-all.json.bz2                                12-Apr-2023 17:28         81311587032
latest-all.json.gz                                 12-Apr-2023 08:58        123350716043
latest-all.nt.bz2                                  13-Apr-2023 07:26        157814250243
latest-all.nt.gz                                   12-Apr-2023 13:38        204367313580
latest-all.ttl.bz2                                 12-Apr-2023 18:47        101241734022
latest-all.ttl.gz                                  12-Apr-2023 08:55        123611937673
latest-lexemes.json.bz2                            12-Apr-2023 03:53           292437658
latest-lexemes.json.gz                             12-Apr-2023 03:51           400418207
latest-lexemes.nt.bz2                              14-Apr-2023 23:35           763000897
latest-lexemes.nt.gz                               14-Apr-2023 23:29          1002680227
latest-lexemes.ttl.bz2                             14-Apr-2023 23:30           430016024
latest-lexemes.ttl.gz                              14-Apr-2023 23:26           536802172
latest-truthy.nt.bz2                               14-Apr-2023 18:25         35953038205
latest-truthy.nt.gz                                14-Apr-2023 15:04         59629575771

see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/

latest-all.ttl.gz ~2 hours

nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz&
--2023-04-18 08:27:54--  https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117784845970 (110G) [application/octet-stream]
Saving to: ‘latest-all.ttl.gz’

     0K .......... .......... .......... .......... ..........  0%  417K 3d4h
...
115024200K .......... .......... .......... .......... .......... 99% 29.6M 0s
115024250K .......... ...                                        100%  126M=1h59m

2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970]

latest-all.ttl.bz2

nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2& 
--2023-04-18 12:43:54--  https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96670186177 (90G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’

     0K .......... .......... .......... .......... ..........  0%  384K 2d20h
    50K .......... .......... .......... .......... ..........  0%  193K 4d5h
   100K .......... .......... .......... .......... ..........  0%  385K 3d18h

wikimedia cookbook scripts

https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py that calls bash scripts ( https://github.com/wikimedia/wikidata-query-deploy/blob/master/loadData.sh

Munging