Difference between revisions of "Snapquery"

From BITPlan Wiki
Jump to navigation Jump to search
 
(16 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
|title=snapquery
 
|title=snapquery
 
|url=https://github.com/WolfgangFahl/snapquery
 
|url=https://github.com/WolfgangFahl/snapquery
−
|version=0.0.3
+
|version=0.0.7
−
|description=Frontend to Introduce Named Queries and Named Query Middleware to wikidata
+
|description=Just Query wikidata and other SPARQL endpoints by qury name - a frontend to Introduce Named Queries and Named Query Middleware to wikidata and other SPARQL endpoints
−
|date=2024-05-04
+
|date=2024-05-07
 
|since=2024-05-03
 
|since=2024-05-03
 
}}
 
}}
 +
= Motivation =
 +
Querying Knowledge Graphs such as wikidata that are based on {{Link|target=SPARQL}} is too complex for mere mortals.
 +
A simple
 +
<pre>snapquery cats</pre>
 +
should work.
 +
See https://snapquery.bitplan.com/query/wikidata-examples/cats for an example
 +
 +
= Demos =
 +
* [https://snapquery.bitplan.com/ BITPlan]
 +
* [https://snapquery.wikidata.dbis.rwth-aachen.de/ RWTH Aachen i5]
  
 
= Usage =
 
= Usage =
 +
== RESTFul API ==
 +
http://snapquery.bitplan.com/docs
 +
=== query ===
 +
<source lang=bash' highlight="1,2">
 +
curl 'https://snapquery.bitplan.com/api/query/wikidata-examples/cats.mediawiki?limit=5'
 +
</source>
 +
==== cats ====
 +
 +
===== query =====
 +
<source lang='sparql'>
 +
 +
SELECT ?item ?itemLabel
 +
WHERE {
 +
  ?item wdt:P31 wd:Q146. # Must be a cat
 +
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
 +
}
 +
 +
LIMIT 5
 +
</source>
 +
 +
[https://query.wikidata.org/#%0ASELECT%20%3Fitem%20%3FitemLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP31%20wd%3AQ146.%20%23%20Must%20be%20a%20cat%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0A/#%0ASELECT%20%3Fitem%20%3FitemLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP31%20wd%3AQ146.%20%23%20Must%20be%20a%20cat%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0A%0ALIMIT%205 try it!]
 +
===== result =====
 +
{| class="wikitable" style="text-align: left;"
 +
|+ <!-- caption -->
 +
|-
 +
! item                                  !! itemLabel
 +
|-
 +
| http://www.wikidata.org/entity/Q378619 || CC
 +
|-
 +
| http://www.wikidata.org/entity/Q498787 || Muezza
 +
|-
 +
| http://www.wikidata.org/entity/Q677525 || Orangey
 +
|-
 +
| http://www.wikidata.org/entity/Q851190 || Mrs. Chippy
 +
|-
 +
| http://www.wikidata.org/entity/Q893453 || Unsinkable Sam
 +
|}
 +
=== sparql ===
 +
<source lang='bash' highlight='1'>
 +
curl 'https://snapquery.bitplan.com/api/sparql/wikidata-examples/cats'
 +
 +
SELECT ?item ?itemLabel
 +
WHERE {
 +
  ?item wdt:P31 wd:Q146. # Must be a cat
 +
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
 +
}
 +
</source>
 +
 
== Commandline ==
 
== Commandline ==
 
=== Help ===
 
=== Help ===
Line 68: Line 126:
 
</source>
 
</source>
 
=== Run Query ===
 
=== Run Query ===
 +
 +
 +
==== cats ====
 
<source lang='bash' highlight='1'>
 
<source lang='bash' highlight='1'>
 
snapquery -qn cats --limit 10 -f mediawiki
 
snapquery -qn cats --limit 10 -f mediawiki
 
</source>
 
</source>
−
==== cats =====
 
  
 
===== query =====
 
===== query =====
Line 111: Line 171:
 
| http://www.wikidata.org/entity/Q1371145 || Socks
 
| http://www.wikidata.org/entity/Q1371145 || Socks
 
|}
 
|}
 +
==== scholia  author_list-of-publications with parameter Q80 (papers by Tim-Berners-Lee) ====
 +
<source lang='bash' highlight='1'>
 +
snapquery --namespace scholia -qn author_list-of-publications --params q=Q80 --limit 2 -f mediawiki
 +
</source>
 +
 +
===== query =====
 +
<source lang='sparql'>
 +
#defaultView:Table
 +
PREFIX target: <http://www.wikidata.org/entity/Q80>
 +
 +
SELECT
 +
  (MIN(?dates) AS ?date)
 +
  ?work ?workLabel
 +
  (GROUP_CONCAT(DISTINCT ?type_label; separator=", ") AS ?type)
 +
  (SAMPLE(?pages_) AS ?pages)
 +
  ?venue ?venueLabel
 +
  (GROUP_CONCAT(DISTINCT ?author_label; separator=", ") AS ?authors)
 +
  (CONCAT("../authors/", GROUP_CONCAT(DISTINCT SUBSTR(STR(?author), 32); separator=",")) AS ?authorsUrl)
 +
WHERE {
 +
  ?work wdt:P50 target: .
 +
  ?work wdt:P50 ?author .
 +
  OPTIONAL {
 +
    ?author rdfs:label ?author_label_ . FILTER (LANG(?author_label_) = 'en')
 +
  }
 +
  BIND(COALESCE(?author_label_, SUBSTR(STR(?author), 32)) AS ?author_label)
 +
  OPTIONAL { ?work wdt:P31 ?type_ . ?type_ rdfs:label ?type_label . FILTER (LANG(?type_label) = 'en') }
 +
  ?work wdt:P577 ?datetimes .
 +
  BIND(xsd:date(?datetimes) AS ?dates)
 +
  OPTIONAL { ?work wdt:P1104 ?pages_ }
 +
  OPTIONAL { ?work wdt:P1433 ?venue }
 +
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,da,de,es,fr,jp,no,ru,sv,zh". } 
 +
}
 +
GROUP BY ?work ?workLabel ?venue ?venueLabel
 +
ORDER BY DESC(?date) 
 +
 +
LIMIT 2
 +
</source>
 +
 +
[https://query.wikidata.org/#%23defaultView%3ATable%0APREFIX%20target%3A%20%3Chttp%3A//www.wikidata.org/entity/%7B%7B%20q%20%7D%7D%3E%0A%0ASELECT%0A%20%20%28MIN%28%3Fdates%29%20AS%20%3Fdate%29%0A%20%20%3Fwork%20%3FworkLabel%0A%20%20%28GROUP_CONCAT%28DISTINCT%20%3Ftype_label%3B%20separator%3D%22%2C%20%22%29%20AS%20%3Ftype%29%0A%20%20%28SAMPLE%28%3Fpages_%29%20AS%20%3Fpages%29%0A%20%20%3Fvenue%20%3FvenueLabel%0A%20%20%28GROUP_CONCAT%28DISTINCT%20%3Fauthor_label%3B%20separator%3D%22%2C%20%22%29%20AS%20%3Fauthors%29%0A%20%20%28CONCAT%28%22../authors/%22%2C%20GROUP_CONCAT%28DISTINCT%20SUBSTR%28STR%28%3Fauthor%29%2C%2032%29%3B%20separator%3D%22%2C%22%29%29%20AS%20%3FauthorsUrl%29%0AWHERE%20%7B%0A%20%20%3Fwork%20wdt%3AP50%20target%3A%20.%0A%20%20%3Fwork%20wdt%3AP50%20%3Fauthor%20.%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20%3Fauthor%20rdfs%3Alabel%20%3Fauthor_label_%20.%20FILTER%20%28LANG%28%3Fauthor_label_%29%20%3D%20%27en%27%29%0A%20%20%7D%0A%20%20BIND%28COALESCE%28%3Fauthor_label_%2C%20SUBSTR%28STR%28%3Fauthor%29%2C%2032%29%29%20AS%20%3Fauthor_label%29%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP31%20%3Ftype_%20.%20%3Ftype_%20rdfs%3Alabel%20%3Ftype_label%20.%20FILTER%20%28LANG%28%3Ftype_label%29%20%3D%20%27en%27%29%20%7D%0A%20%20%3Fwork%20wdt%3AP577%20%3Fdatetimes%20.%0A%20%20BIND%28xsd%3Adate%28%3Fdatetimes%29%20AS%20%3Fdates%29%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP1104%20%3Fpages_%20%7D%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP1433%20%3Fvenue%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%2Cda%2Cde%2Ces%2Cfr%2Cjp%2Cno%2Cru%2Csv%2Czh%22.%20%7D%20%20%0A%7D%0AGROUP%20BY%20%3Fwork%20%3FworkLabel%20%3Fvenue%20%3FvenueLabel%0AORDER%20BY%20DESC%28%3Fdate%29%20%20%0A/#%23defaultView%3ATable%0APREFIX%20target%3A%20%3Chttp%3A//www.wikidata.org/entity/Q80%3E%0A%0ASELECT%0A%20%20%28MIN%28%3Fdates%29%20AS%20%3Fdate%29%0A%20%20%3Fwork%20%3FworkLabel%0A%20%20%28GROUP_CONCAT%28DISTINCT%20%3Ftype_label%3B%20separator%3D%22%2C%20%22%29%20AS%20%3Ftype%29%0A%20%20%28SAMPLE%28%3Fpages_%29%20AS%20%3Fpages%29%0A%20%20%3Fvenue%20%3FvenueLabel%0A%20%20%28GROUP_CONCAT%28DISTINCT%20%3Fauthor_label%3B%20separator%3D%22%2C%20%22%29%20AS%20%3Fauthors%29%0A%20%20%28CONCAT%28%22../authors/%22%2C%20GROUP_CONCAT%28DISTINCT%20SUBSTR%28STR%28%3Fauthor%29%2C%2032%29%3B%20separator%3D%22%2C%22%29%29%20AS%20%3FauthorsUrl%29%0AWHERE%20%7B%0A%20%20%3Fwork%20wdt%3AP50%20target%3A%20.%0A%20%20%3Fwork%20wdt%3AP50%20%3Fauthor%20.%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20%3Fauthor%20rdfs%3Alabel%20%3Fauthor_label_%20.%20FILTER%20%28LANG%28%3Fauthor_label_%29%20%3D%20%27en%27%29%0A%20%20%7D%0A%20%20BIND%28COALESCE%28%3Fauthor_label_%2C%20SUBSTR%28STR%28%3Fauthor%29%2C%2032%29%29%20AS%20%3Fauthor_label%29%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP31%20%3Ftype_%20.%20%3Ftype_%20rdfs%3Alabel%20%3Ftype_label%20.%20FILTER%20%28LANG%28%3Ftype_label%29%20%3D%20%27en%27%29%20%7D%0A%20%20%3Fwork%20wdt%3AP577%20%3Fdatetimes%20.%0A%20%20BIND%28xsd%3Adate%28%3Fdatetimes%29%20AS%20%3Fdates%29%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP1104%20%3Fpages_%20%7D%0A%20%20OPTIONAL%20%7B%20%3Fwork%20wdt%3AP1433%20%3Fvenue%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%2Cda%2Cde%2Ces%2Cfr%2Cjp%2Cno%2Cru%2Csv%2Czh%22.%20%7D%20%20%0A%7D%0AGROUP%20BY%20%3Fwork%20%3FworkLabel%20%3Fvenue%20%3FvenueLabel%0AORDER%20BY%20DESC%28%3Fdate%29%20%20%0A%0ALIMIT%202 try it!]
 +
===== result =====
 +
{| class="wikitable" style="text-align: left;"
 +
|+ <!-- caption -->
 +
|-
 +
! date      !! work                                      !! workLabel                                                            !! type              !! venue                                    !! venueLabel                                                                                                                !! authors                                                                !! authorsUrl                                            !! align="right"|  pages
 +
|-
 +
| 2023-07-07 || http://www.wikidata.org/entity/Q125750037 || Linked Data - The Story So Far                                      || chapter          || http://www.wikidata.org/entity/Q125750044 || Linking the World’s Information                                                                                            || Christian Bizer, Tom Heath, Tim Berners-Lee                            || ../authors/Q17744291,Q17744272,Q80                    || align="right"|
 +
|-
 +
| 2017-01-01 || http://www.wikidata.org/entity/Q30096408  || Linked Data Notifications: A Resource-Centric Communication Protocol || scholarly article || http://www.wikidata.org/entity/Q30092087  || The Semantic Web: 14th International Conference, ESWC 2017, Portorož, Slovenia, May 28 – June 1, 2017, Proceedings, Part I || Christoph Lange, Amy Guy, Sarven Capadisli, Sören Auer, Tim Berners-Lee || ../authors/Q30276490,Q30086138,Q30078591,Q27453085,Q80 || align="right"|      17
 +
|}
 +
 +
=== Mass Import Queries ===
 +
<source lang='bash' highlight='1'>
 +
snapquery --import samples/scholia.json
 +
Importing Named Queries: 100%|████████████| 372/372 [00:00<00:00, 226127.69it/s]
 +
Imported 372 named queries from samples/scholia.json.
 +
</source>
 
{{pip|snapquery}}
 
{{pip|snapquery}}
 +
== Initial database import ==
 +
<source lang='bash'>
 +
scripts/restore_db
 +
Importing Named Queries: 100%|██████████████████| 28/28 [00:25<00:00,  1.09it/s]
 +
Imported 28 named queries from samples/ceur-ws.json.
 +
Importing Named Queries: 100%|████████████| 372/372 [00:00<00:00, 210308.81it/s]
 +
Imported 372 named queries from samples/scholia.json.
 +
Importing Named Queries: 100%|████████████| 298/298 [00:00<00:00, 252149.00it/s]
 +
Imported 298 named queries from samples/wikidata-examples.json.
 +
</source>
 +
= Links =
 +
* https://etherpad.wikimedia.org/p/wmh2024-Wikimedia_Hackathon_2024_Closing

Latest revision as of 06:52, 7 May 2024

OsProject

OsProject
id  snapquery
state  active
owner  WolfgangFahl
title  snapquery
url  https://github.com/WolfgangFahl/snapquery
version  0.0.7
description  Just Query wikidata and other SPARQL endpoints by qury name - a frontend to Introduce Named Queries and Named Query Middleware to wikidata and other SPARQL endpoints
date  2024-05-07
since  2024-05-03
until  

Motivation

Querying Knowledge Graphs such as wikidata that are based on SPARQL is too complex for mere mortals. A simple 

snapquery cats

should work. See https://snapquery.bitplan.com/query/wikidata-examples/cats for an example

Demos

Usage

RESTFul API

http://snapquery.bitplan.com/docs

query

curl 'https://snapquery.bitplan.com/api/query/wikidata-examples/cats.mediawiki?limit=5'

cats

query
SELECT ?item ?itemLabel
WHERE {
  ?item wdt:P31 wd:Q146. # Must be a cat
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}

LIMIT 5

try it!

result
item itemLabel
http://www.wikidata.org/entity/Q378619 CC
http://www.wikidata.org/entity/Q498787 Muezza
http://www.wikidata.org/entity/Q677525 Orangey
http://www.wikidata.org/entity/Q851190 Mrs. Chippy
http://www.wikidata.org/entity/Q893453 Unsinkable Sam

sparql

curl 'https://snapquery.bitplan.com/api/sparql/wikidata-examples/cats'

SELECT ?item ?itemLabel
WHERE {
  ?item wdt:P31 wd:Q146. # Must be a cat
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}

Commandline

Help

snapquery -h
usage: snapquery [-h] [-a] [--apache APACHE] [-c] [-d]
                 [--debugServer DEBUGSERVER] [--debugPort DEBUGPORT]
                 [--debugRemotePath DEBUGREMOTEPATH]
                 [--debugLocalPath DEBUGLOCALPATH] [-l] [-i INPUT] [-rol]
                 [--host HOST] [--port PORT] [-s] [-V] [-ep ENDPOINTPATH]
                 [-en ENDPOINTNAME] [-le]

Introduce Named Queries and Named Query Middleware to wikidata

options:
  -h, --help            show this help message and exit
  -a, --about           show about info [default: False]
  --apache APACHE       create an apache configuration file for the given
                        domain
  -c, --client          start client [default: False]
  -d, --debug           show debug info [default: False]
  --debugServer DEBUGSERVER
                        remote debug Server
  --debugPort DEBUGPORT
                        remote debug Port
  --debugRemotePath DEBUGREMOTEPATH
                        remote debug Server path mapping - remotePath - path
                        on debug server
  --debugLocalPath DEBUGLOCALPATH
                        remote debug Server path mapping - localPath - path on
                        machine where python runs
  -l, --local           run with local file system access [default: False]
  -i INPUT, --input INPUT
                        input file
  -rol, --render_on_load
                        render on load [default: False]
  --host HOST           the host to serve / listen from [default: localhost]
  --port PORT           the port to serve from [default: 9862]
  -s, --serve           start webserver [default: False]
  -V, --version         show program's version number and exit
  -ep ENDPOINTPATH, --endpointPath ENDPOINTPATH
                        path to yaml file to configure endpoints to use for
                        queries
  -en ENDPOINTNAME, --endpointName ENDPOINTNAME
                        Name of the endpoint to use for queries - use
                        --listEndpoints to list available endpoints
  -le, --listEndpoints  show the list of available endpoints

List Endpoints

snapquery --listEndpoints
wikidata:https://query.wikidata.org:https://query.wikidata.org/sparql(POST)
qlever-wikidata:https://qlever.cs.uni-freiburg.de/wikidata:https://qlever.cs.uni-freiburg.de/api/wikidata(POST)

Run Query

cats

snapquery -qn cats --limit 10 -f mediawiki
query
SELECT ?item ?itemLabel
WHERE {
  ?item wdt:P31 wd:Q146. # Must be a cat
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}

try it!

result
item itemLabel
http://www.wikidata.org/entity/Q378619 CC
http://www.wikidata.org/entity/Q498787 Muezza
http://www.wikidata.org/entity/Q677525 Orangey
http://www.wikidata.org/entity/Q851190 Mrs. Chippy
http://www.wikidata.org/entity/Q893453 Unsinkable Sam
http://www.wikidata.org/entity/Q1050083 Catmando
http://www.wikidata.org/entity/Q1185550 Oscar
http://www.wikidata.org/entity/Q1201902 Tama
http://www.wikidata.org/entity/Q1207136 Dewey Readmore Books
http://www.wikidata.org/entity/Q1371145 Socks

scholia author_list-of-publications with parameter Q80 (papers by Tim-Berners-Lee)

snapquery --namespace scholia -qn author_list-of-publications --params q=Q80 --limit 2 -f mediawiki
query
#defaultView:Table
PREFIX target: <http://www.wikidata.org/entity/Q80>

SELECT
  (MIN(?dates) AS ?date)
  ?work ?workLabel
  (GROUP_CONCAT(DISTINCT ?type_label; separator=", ") AS ?type)
  (SAMPLE(?pages_) AS ?pages)
  ?venue ?venueLabel
  (GROUP_CONCAT(DISTINCT ?author_label; separator=", ") AS ?authors)
  (CONCAT("../authors/", GROUP_CONCAT(DISTINCT SUBSTR(STR(?author), 32); separator=",")) AS ?authorsUrl)
WHERE {
  ?work wdt:P50 target: .
  ?work wdt:P50 ?author .
  OPTIONAL {
    ?author rdfs:label ?author_label_ . FILTER (LANG(?author_label_) = 'en')
  }
  BIND(COALESCE(?author_label_, SUBSTR(STR(?author), 32)) AS ?author_label)
  OPTIONAL { ?work wdt:P31 ?type_ . ?type_ rdfs:label ?type_label . FILTER (LANG(?type_label) = 'en') }
  ?work wdt:P577 ?datetimes .
  BIND(xsd:date(?datetimes) AS ?dates)
  OPTIONAL { ?work wdt:P1104 ?pages_ }
  OPTIONAL { ?work wdt:P1433 ?venue }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,da,de,es,fr,jp,no,ru,sv,zh". }  
}
GROUP BY ?work ?workLabel ?venue ?venueLabel
ORDER BY DESC(?date)  

LIMIT 2

try it!

result
date work workLabel type venue venueLabel authors authorsUrl pages
2023-07-07 http://www.wikidata.org/entity/Q125750037 Linked Data - The Story So Far chapter http://www.wikidata.org/entity/Q125750044 Linking the World’s Information Christian Bizer, Tom Heath, Tim Berners-Lee ../authors/Q17744291,Q17744272,Q80
2017-01-01 http://www.wikidata.org/entity/Q30096408 Linked Data Notifications: A Resource-Centric Communication Protocol scholarly article http://www.wikidata.org/entity/Q30092087 The Semantic Web: 14th International Conference, ESWC 2017, Portorož, Slovenia, May 28 – June 1, 2017, Proceedings, Part I Christoph Lange, Amy Guy, Sarven Capadisli, Sören Auer, Tim Berners-Lee ../authors/Q30276490,Q30086138,Q30078591,Q27453085,Q80 17

Mass Import Queries

snapquery --import samples/scholia.json 
Importing Named Queries: 100%|████████████| 372/372 [00:00<00:00, 226127.69it/s]
Imported 372 named queries from samples/scholia.json.


Installation

pip install snapquery
# alternatively if your pip is not a python3 pip
pip3 install snapquery 
# local install from source directory of snapquery 
pip install .

upgrade

pip install snapquery  -U
# alternatively if your pip is not a python3 pip
pip3 install snapquery -U


Initial database import

scripts/restore_db 
Importing Named Queries: 100%|██████████████████| 28/28 [00:25<00:00,  1.09it/s]
Imported 28 named queries from samples/ceur-ws.json.
Importing Named Queries: 100%|████████████| 372/372 [00:00<00:00, 210308.81it/s]
Imported 372 named queries from samples/scholia.json.
Importing Named Queries: 100%|████████████| 298/298 [00:00<00:00, 252149.00it/s]
Imported 298 named queries from samples/wikidata-examples.json.

Links

Retrieved from "https://wiki.bitplan.com/index.php?title=Snapquery&oldid=18595"