Difference between revisions of "Wikidata Import 2023-04-18"
Jump to navigation
Jump to search
| (9 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | {{PageSequence|prev=Wikidata Import 2023-01-24|next=|category=Wikidata|categoryIcon=cloud-download}} | + | |
| + | =Freitext= | ||
| + | {{PageSequence|prev=Wikidata Import 2023-01-24|next=Wikidata Import 2023-04-26|category=Wikidata|categoryIcon=cloud-download}} | ||
| + | =Import= | ||
| + | {{Import | ||
| + | |state=❌ | ||
| + | |target=blazegraph | ||
| + | |start=2023-04-18 | ||
| + | |end=2023-04-18 | ||
| + | |cpu=Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz | ||
| + | |ram=256 | ||
| + | |triples=14.6 | ||
| + | |storemode=property | ||
| + | |url=https://wiki.bitplan.com/index.php/Wikidata_Import_2023-04-18 | ||
| + | |comment=This attempt failed - the mirror at your.org is not active any more and should not be used | ||
| + | }} | ||
| + | |||
= Download latest wikidata Dump ~2 hours = | = Download latest wikidata Dump ~2 hours = | ||
| + | == Download Options == | ||
https://dumps.wikimedia.org/wikidatawiki/entities | https://dumps.wikimedia.org/wikidatawiki/entities | ||
<pre> | <pre> | ||
| Line 19: | Line 36: | ||
</pre> | </pre> | ||
see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/ | see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/ | ||
| + | == latest-all.ttl.gz ~2 hours == | ||
<source lang='bash' highlight='1'> | <source lang='bash' highlight='1'> | ||
nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz& | nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz& | ||
| Line 35: | Line 53: | ||
2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970] | 2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970] | ||
</source> | </source> | ||
| + | == latest-all.ttl.bz2 == | ||
| + | <source lang='bash' highlight='1'> | ||
| + | nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2& | ||
| + | --2023-04-18 12:43:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2 | ||
| + | Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7 | ||
| + | Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected. | ||
| + | HTTP request sent, awaiting response... 200 OK | ||
| + | Length: 96670186177 (90G) [application/octet-stream] | ||
| + | Saving to: ‘latest-all.ttl.bz2’ | ||
| + | |||
| + | 0K .......... .......... .......... .......... .......... 0% 384K 2d20h | ||
| + | 50K .......... .......... .......... .......... .......... 0% 193K 4d5h | ||
| + | 100K .......... .......... .......... .......... .......... 0% 385K 3d18h | ||
| + | </source> | ||
| + | |||
| + | = wikimedia cookbook scripts = | ||
| + | https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py | ||
| + | that calls bash scripts ( https://github.com/wikimedia/wikidata-query-deploy/blob/master/loadData.sh | ||
| + | |||
= Munging = | = Munging = | ||
* https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py | * https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py | ||
Latest revision as of 07:00, 15 May 2023
Freitext
Import
| Import | |
|---|---|
 edit
| |
| state | ❌ |
| url | https://wiki.bitplan.com/index.php/Wikidata_Import_2023-04-18 |
| target | blazegraph |
| start | 2023-04-18 |
| end | 2023-04-18 |
| days | |
| os | |
| cpu | Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz |
| ram | 256 |
| triples | 14.6 |
| comment | This attempt failed - the mirror at your.org is not active any more and should not be used |
Download latest wikidata Dump ~2 hours
Download Options
https://dumps.wikimedia.org/wikidatawiki/entities
latest-all.json.bz2 12-Apr-2023 17:28 81311587032 latest-all.json.gz 12-Apr-2023 08:58 123350716043 latest-all.nt.bz2 13-Apr-2023 07:26 157814250243 latest-all.nt.gz 12-Apr-2023 13:38 204367313580 latest-all.ttl.bz2 12-Apr-2023 18:47 101241734022 latest-all.ttl.gz 12-Apr-2023 08:55 123611937673 latest-lexemes.json.bz2 12-Apr-2023 03:53 292437658 latest-lexemes.json.gz 12-Apr-2023 03:51 400418207 latest-lexemes.nt.bz2 14-Apr-2023 23:35 763000897 latest-lexemes.nt.gz 14-Apr-2023 23:29 1002680227 latest-lexemes.ttl.bz2 14-Apr-2023 23:30 430016024 latest-lexemes.ttl.gz 14-Apr-2023 23:26 536802172 latest-truthy.nt.bz2 14-Apr-2023 18:25 35953038205 latest-truthy.nt.gz 14-Apr-2023 15:04 59629575771
see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/
latest-all.ttl.gz ~2 hours
nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz&
--2023-04-18 08:27:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117784845970 (110G) [application/octet-stream]
Saving to: ‘latest-all.ttl.gz’
0K .......... .......... .......... .......... .......... 0% 417K 3d4h
...
115024200K .......... .......... .......... .......... .......... 99% 29.6M 0s
115024250K .......... ... 100% 126M=1h59m
2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970]
latest-all.ttl.bz2
nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2&
--2023-04-18 12:43:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96670186177 (90G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’
0K .......... .......... .......... .......... .......... 0% 384K 2d20h
50K .......... .......... .......... .......... .......... 0% 193K 4d5h
100K .......... .......... .......... .......... .......... 0% 385K 3d18h
wikimedia cookbook scripts
https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py that calls bash scripts ( https://github.com/wikimedia/wikidata-query-deploy/blob/master/loadData.sh