Difference between revisions of "Wikidata Import 2023-04-18"
Jump to navigation
Jump to search
(18 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | {{PageSequence|prev= | + | |
− | = Download latest wikidata Dump ~ | + | =Freitext= |
+ | {{PageSequence|prev=Wikidata Import 2023-01-24|next=Wikidata Import 2023-04-26|category=Wikidata|categoryIcon=cloud-download}} | ||
+ | =Import= | ||
+ | {{Import | ||
+ | |state=❌ | ||
+ | |target=blazegraph | ||
+ | |start=2023-04-18 | ||
+ | |end=2023-04-18 | ||
+ | |cpu=Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz | ||
+ | |ram=256 | ||
+ | |triples=14.6 | ||
+ | |storemode=property | ||
+ | |url=https://wiki.bitplan.com/index.php/Wikidata_Import_2023-04-18 | ||
+ | |comment=This attempt failed - the mirror at your.org is not active any more and should not be used | ||
+ | }} | ||
+ | |||
+ | = Download latest wikidata Dump ~2 hours = | ||
+ | == Download Options == | ||
https://dumps.wikimedia.org/wikidatawiki/entities | https://dumps.wikimedia.org/wikidatawiki/entities | ||
+ | <pre> | ||
+ | latest-all.json.bz2 12-Apr-2023 17:28 81311587032 | ||
+ | latest-all.json.gz 12-Apr-2023 08:58 123350716043 | ||
+ | latest-all.nt.bz2 13-Apr-2023 07:26 157814250243 | ||
+ | latest-all.nt.gz 12-Apr-2023 13:38 204367313580 | ||
+ | latest-all.ttl.bz2 12-Apr-2023 18:47 101241734022 | ||
+ | latest-all.ttl.gz 12-Apr-2023 08:55 123611937673 | ||
+ | latest-lexemes.json.bz2 12-Apr-2023 03:53 292437658 | ||
+ | latest-lexemes.json.gz 12-Apr-2023 03:51 400418207 | ||
+ | latest-lexemes.nt.bz2 14-Apr-2023 23:35 763000897 | ||
+ | latest-lexemes.nt.gz 14-Apr-2023 23:29 1002680227 | ||
+ | latest-lexemes.ttl.bz2 14-Apr-2023 23:30 430016024 | ||
+ | latest-lexemes.ttl.gz 14-Apr-2023 23:26 536802172 | ||
+ | latest-truthy.nt.bz2 14-Apr-2023 18:25 35953038205 | ||
+ | latest-truthy.nt.gz 14-Apr-2023 15:04 59629575771 | ||
+ | </pre> | ||
+ | see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/ | ||
+ | == latest-all.ttl.gz ~2 hours == | ||
+ | <source lang='bash' highlight='1'> | ||
+ | nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz& | ||
+ | --2023-04-18 08:27:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz | ||
+ | Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7 | ||
+ | Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected. | ||
+ | HTTP request sent, awaiting response... 200 OK | ||
+ | Length: 117784845970 (110G) [application/octet-stream] | ||
+ | Saving to: ‘latest-all.ttl.gz’ | ||
+ | |||
+ | 0K .......... .......... .......... .......... .......... 0% 417K 3d4h | ||
+ | ... | ||
+ | 115024200K .......... .......... .......... .......... .......... 99% 29.6M 0s | ||
+ | 115024250K .......... ... 100% 126M=1h59m | ||
+ | |||
+ | 2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970] | ||
+ | </source> | ||
+ | == latest-all.ttl.bz2 == | ||
+ | <source lang='bash' highlight='1'> | ||
+ | nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2& | ||
+ | --2023-04-18 12:43:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2 | ||
+ | Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7 | ||
+ | Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected. | ||
+ | HTTP request sent, awaiting response... 200 OK | ||
+ | Length: 96670186177 (90G) [application/octet-stream] | ||
+ | Saving to: ‘latest-all.ttl.bz2’ | ||
+ | |||
+ | 0K .......... .......... .......... .......... .......... 0% 384K 2d20h | ||
+ | 50K .......... .......... .......... .......... .......... 0% 193K 4d5h | ||
+ | 100K .......... .......... .......... .......... .......... 0% 385K 3d18h | ||
+ | </source> | ||
+ | |||
+ | = wikimedia cookbook scripts = | ||
+ | https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py | ||
+ | that calls bash scripts ( https://github.com/wikimedia/wikidata-query-deploy/blob/master/loadData.sh | ||
+ | |||
+ | = Munging = | ||
+ | * https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py |
Latest revision as of 07:00, 15 May 2023
Freitext
Import
Import | |
---|---|
edit | |
state | ❌ |
url | https://wiki.bitplan.com/index.php/Wikidata_Import_2023-04-18 |
target | blazegraph |
start | 2023-04-18 |
end | 2023-04-18 |
days | |
os | |
cpu | Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz |
ram | 256 |
triples | 14.6 |
comment | This attempt failed - the mirror at your.org is not active any more and should not be used |
Download latest wikidata Dump ~2 hours
Download Options
https://dumps.wikimedia.org/wikidatawiki/entities
latest-all.json.bz2 12-Apr-2023 17:28 81311587032 latest-all.json.gz 12-Apr-2023 08:58 123350716043 latest-all.nt.bz2 13-Apr-2023 07:26 157814250243 latest-all.nt.gz 12-Apr-2023 13:38 204367313580 latest-all.ttl.bz2 12-Apr-2023 18:47 101241734022 latest-all.ttl.gz 12-Apr-2023 08:55 123611937673 latest-lexemes.json.bz2 12-Apr-2023 03:53 292437658 latest-lexemes.json.gz 12-Apr-2023 03:51 400418207 latest-lexemes.nt.bz2 14-Apr-2023 23:35 763000897 latest-lexemes.nt.gz 14-Apr-2023 23:29 1002680227 latest-lexemes.ttl.bz2 14-Apr-2023 23:30 430016024 latest-lexemes.ttl.gz 14-Apr-2023 23:26 536802172 latest-truthy.nt.bz2 14-Apr-2023 18:25 35953038205 latest-truthy.nt.gz 14-Apr-2023 15:04 59629575771
see https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits/
latest-all.ttl.gz ~2 hours
nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz&
--2023-04-18 08:27:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.gz
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117784845970 (110G) [application/octet-stream]
Saving to: ‘latest-all.ttl.gz’
0K .......... .......... .......... .......... .......... 0% 417K 3d4h
...
115024200K .......... .......... .......... .......... .......... 99% 29.6M 0s
115024250K .......... ... 100% 126M=1h59m
2023-04-18 10:27:37 (15.6 MB/s) - ‘latest-all.ttl.gz’ saved [117784845970/117784845970]
latest-all.ttl.bz2
nohup wget https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2&
--2023-04-18 12:43:54-- https://dumps.wikimedia.your.org/wikidatawiki/entities/latest-all.ttl.bz2
Resolving dumps.wikimedia.your.org (dumps.wikimedia.your.org)... 163.237.219.7, 2001:4978:1000:1499::7
Connecting to dumps.wikimedia.your.org (dumps.wikimedia.your.org)|163.237.219.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96670186177 (90G) [application/octet-stream]
Saving to: ‘latest-all.ttl.bz2’
0K .......... .......... .......... .......... .......... 0% 384K 2d20h
50K .......... .......... .......... .......... .......... 0% 193K 4d5h
100K .......... .......... .......... .......... .......... 0% 385K 3d18h
wikimedia cookbook scripts
https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py that calls bash scripts ( https://github.com/wikimedia/wikidata-query-deploy/blob/master/loadData.sh