Difference between revisions of "Py-3rdparty-mediawiki"

From BITPlan Wiki
Jump to navigation Jump to search
(Created page with "{{OsProject |storemode=property }}")
 
 
(48 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
=OsProject=
 +
 
{{OsProject
 
{{OsProject
 +
|id=Py-3rdparty-mediawiki
 +
|state=active
 +
|owner=WolfgangFahl
 +
|title=Py-3rdparty-mediawiki
 +
|description=python 3rd party Mediawiki bot support enhancements e.g. encrypted credential handling
 +
|url=https://github.com/WolfgangFahl/py-3rdparty-mediawiki
 +
|version=0.14.1
 +
|date=2024-11-09
 +
|since=2020-03-26
 
|storemode=property
 
|storemode=property
 
}}
 
}}
 +
 +
{{pip|py-3rdparty-mediawiki}}
 +
__TOC__
 +
= What is it =
 +
Extended functionality for
 +
# [https://github.com/mwclient/mwclient mwclient]
 +
Originally we intended to use also support:
 +
* [https://www.mediawiki.org/wiki/Manual:Pywikibot/de pywikibot]
 +
the results where disappointing - pywikibot kept being a source of trouble more than a solution to our needs
 +
 +
= Github =
 +
* https://github.com/WolfgangFahl/py-3rdparty-mediawiki
 +
 +
 +
== Via Source code ==
 +
<source lang='bash'>
 +
git clone https://github.com/WolfgangFahl/py-3rdparty-mediawiki
 +
./install
 +
</source>
 +
 +
= wikipush / wikibackup / wikiedit / wiknuke / wikirestore/ wikiquery / wikiupload / wikiuser scripts =
 +
== Setup method ==
 +
If you installed with the method above console_script will have been added to your environment. You can e.g. check
 +
<source lang='bash' highlight='1'>
 +
which wikipush
 +
/Users/wf/Library/Python/3.8/bin/wikipush
 +
</source>
 +
and there should be a wikipush script in your path.
 +
= WikiPush =
 +
WikiPush allows to copy pages from one wiki to another including the images on the page.
 +
To identify yourself you use the credential property files created with the wikiuser script (using python) or the Mediawiki-Japi {{Link|target=CommandLine}}
 +
== usage ==
 +
<source lang='bash'>
 +
wikipush -h
 +
family and mylang are not set.
 +
Defaulting to family='test' and mylang='test'.
 +
usage: wikipush.py [-h] [-d] [-V] [-l] [-f] [-i] [-q QUERY] -s SOURCE -t
 +
                  TARGET [-p PAGES [PAGES ...]]
 +
 +
Created on 2020-10-29
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
USAGE
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: None]
 +
  -V, --version        show program's version number and exit
 +
  -l, --login          login to source wiki for access permission
 +
  -f, --force          force to overwrite existing pages
 +
  -i, --ignore          ignore upload warnings e.g. duplicate images
 +
  -q QUERY, --query QUERY
 +
                        select pages with given SMW ask query
 +
  -s SOURCE, --source SOURCE
 +
                        source wiki id
 +
  -t TARGET, --target TARGET
 +
                        target wiki id
 +
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
 +
                        list of page Titles to be pushed
 +
</source>
 +
== Example ==
 +
<source lang='bash' highlight='1'>
 +
wikipush -s smw -t test2 -q "[[Category:City]]|limit=5"
 +
family and mylang are not set.
 +
Defaulting to family='test' and mylang='test'.
 +
copying 4 pages from smw to test2
 +
copying Demo:Tokyo ...✅
 +
copying image File:SMW-Info-button.png ...✅
 +
copying image File:Tokyo-Tsukishima-0011.jpg ...✅
 +
copying Vienna ...✅
 +
copying Warsaw ...✅
 +
copying image File:6140285934 02e81b845f z.jpg ...✅
 +
copying Demo:Würzburg ...✅
 +
</source>
 +
= wikiquery =
 +
wikiquery allows to send SMW ask-query via commandline and get the results in json or csv format. With the query division parameter the limits of SMW for the maximum  amount of displayed
 +
results can be overcome. E.g. if you set
 +
<pre>
 +
$smwgQMaxInlineLimit=1500;
 +
$smwgQMaxInlineLimitSets=1500;
 +
$smwgQMaxLimit = 5000;
 +
</pre>
 +
You'll be able to get more than 1500/5000 results.
 +
== usage ==
 +
<source lang='bash' highlight='1'>
 +
wikiquery -h
 +
usage: wikiquery [-h] [-d] [-V] [-l] -s SOURCE [--format FORMAT]
 +
                [--entityName ENTITYNAME] [--limit LIMIT] [--progress]
 +
                [-q QUERY] [--queryFile QUERYFILE] [-qf QUERYFIELD]
 +
                [-p PAGES [PAGES ...]] [-ui] [-qd QUERYDIVISION]
 +
 +
wikipush
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: False]
 +
  -V, --version        show program's version number and exit
 +
  -l, --login          login to source wiki for access permission
 +
  -s SOURCE, --source SOURCE
 +
                        source wiki id
 +
  --format FORMAT      format to use for query result csv,json,xml,ttl or
 +
                        wiki
 +
  --entityName ENTITYNAME
 +
                        name of the entites that are queried - only needed for
 +
                        some output formats - default is 'data'
 +
  --limit LIMIT        limit for query
 +
  --progress            shows progress for query
 +
  -q QUERY, --query QUERY
 +
                        select pages with given SMW ask query
 +
  --queryFile QUERYFILE
 +
                        file the query should be read from
 +
  -qf QUERYFIELD, --queryField QUERYFIELD
 +
                        query result field which contains page
 +
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
 +
                        list of page Titles to be pushed
 +
  -ui, --withGUI        Pop up GUI for selection
 +
  -qd QUERYDIVISION, --queryDivision QUERYDIVISION
 +
                        divide query into equidistant subintervals to limit
 +
                        the result size of the individual queries
 +
</source>
 +
== Examples ==
 +
=== query1.ask ===
 +
<source lang='bash'>
 +
{{#ask: [[IsA::Event]][[Acronym::~ES*]][[start date::>2018]][[start date::<2019]]
 +
| mainlabel=pageTitle
 +
| ?Title = title
 +
| ?Event in series = series
 +
| ?ordinal=ordinal
 +
| ?Homepage = homepage
 +
| format=table
 +
}}
 +
</source>
 +
 +
=== csv ===
 +
<source lang='bash' highlight='1'>
 +
wikiquery -s or --queryFile query1.ask --format csv
 +
pageTitle;title;series;ordinal;homepage
 +
ESA 2018;26th Annual European Symposium on Algorithms;ESA;None;http://algo2018.hiit.fi/esa/
 +
ESEC/FSE 2018;26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE);ESEC/FSE;None;https://2018.fseconference.org/
 +
ESOP 2018;27th European Symposium on Programming;ESOP;None;https://etaps.org/2018/esop
 +
ESORICS 2018;23rd European Symposium on Research in Computer Security,;ESORICS;None;None
 +
ESSCIRC 2018;44th European Solid-State Circuits Conference;ESSCIRC;None;None
 +
ESWC 2018;15th European Semantic Web Symposium (ESWS);ESWC;None;http://2018.eswc-conferences.org/
 +
</source>
 +
 +
=== json ===
 +
<source lang='bash'>
 +
wikiquery -s or --queryFile query1.ask --format json
 +
</source>
 +
<source lang='json'>
 +
{
 +
  "data": [
 +
      {
 +
        "pageTitle": "ESA 2018",
 +
        "title": "26th Annual European Symposium on Algorithms",
 +
        "series": "ESA",
 +
        "ordinal": null,
 +
        "homepage": "http://algo2018.hiit.fi/esa/"
 +
      },
 +
      {
 +
        "pageTitle": "ESEC/FSE 2018",
 +
        "title": "26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)",
 +
        "series": "ESEC/FSE",
 +
        "ordinal": null,
 +
        "homepage": "https://2018.fseconference.org/"
 +
      },
 +
      {
 +
        "pageTitle": "ESOP 2018",
 +
        "title": "27th European Symposium on Programming",
 +
        "series": "ESOP",
 +
        "ordinal": null,
 +
        "homepage": "https://etaps.org/2018/esop"
 +
      },
 +
      {
 +
        "pageTitle": "ESORICS 2018",
 +
        "title": "23rd European Symposium on Research in Computer Security,",
 +
        "series": "ESORICS",
 +
        "ordinal": null,
 +
        "homepage": null
 +
      },
 +
      {
 +
        "pageTitle": "ESSCIRC 2018",
 +
        "title": "44th European Solid-State Circuits Conference",
 +
        "series": "ESSCIRC",
 +
        "ordinal": null,
 +
        "homepage": null
 +
      },
 +
      {
 +
        "pageTitle": "ESWC 2018",
 +
        "title": "15th European Semantic Web Symposium (ESWS)",
 +
        "series": "ESWC",
 +
        "ordinal": null,
 +
        "homepage": "http://2018.eswc-conferences.org/"
 +
      }
 +
  ]
 +
}
 +
</source>
 +
 +
= wikibackup =
 +
== usage ==
 +
<source lang='bash'>
 +
wikibackup -h
 +
usage: wikibackup [-h] [-d] [-V] [-g] [-l] -s SOURCE [-wi]
 +
                  [--backupPath BACKUPPATH] [--limit LIMIT] [--progress]
 +
                  [-q QUERY] [--queryFile QUERYFILE] [-qf QUERYFIELD]
 +
                  [-p PAGES [PAGES ...]] [-ui] [-qd QUERYDIVISION]
 +
 +
wikipush
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: False]
 +
  -V, --version        show program's version number and exit
 +
  -g, --git            use git for version control
 +
  -l, --login          login to source wiki for access permission
 +
  -s SOURCE, --source SOURCE
 +
                        source wiki id
 +
  -wi, --withImages    copy images on the given pages
 +
  --backupPath BACKUPPATH
 +
                        path where the backup should be stored
 +
  --limit LIMIT        limit for query
 +
  --progress            shows progress for query
 +
  -q QUERY, --query QUERY
 +
                        select pages with given SMW ask query
 +
  --queryFile QUERYFILE
 +
                        file the query should be read from
 +
  -qf QUERYFIELD, --queryField QUERYFIELD
 +
                        query result field which contains page
 +
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
 +
                        list of page Titles to be pushed
 +
  -ui, --withGUI        Pop up GUI for selection
 +
  -qd QUERYDIVISION, --queryDivision QUERYDIVISION
 +
                        divide query into equidistant subintervals to limit
 +
                        the result size of the individual queries
 +
</source>
 +
 +
= WikiNuke =
 +
wikinukes.py allows mass deletion of pages
 +
== usage ==
 +
<source lang='bash'>
 +
usage: wikinuke.py [-h] [-d] [-V] [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET [-p PAGES [PAGES ...]]
 +
 +
Created on 2020-11-12
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
USAGE
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: None]
 +
  -V, --version        show program's version number and exit
 +
  -f, --force          force to delete pages - default is 'dry' run only listing pages
 +
  -q QUERY, --query QUERY
 +
                        select pages with given SMW ask query
 +
  -qf QUERYFIELD, --queryField QUERYFIELD
 +
                        query result field which contains page
 +
  -t TARGET, --target TARGET
 +
                        target wiki id
 +
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
 +
                        list of page Titles to be pushed
 +
 +
</source>
 +
== Example ==
 +
The default behavior is a dry run only listing whether the pages exist
 +
<source lang='bash' highlight='1'>
 +
wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3
 +
deleting 3 pages in test (dry run)
 +
1/3 (  33%): deleting deleteMe1 ...👍
 +
2/3 (  67%): deleting deleteMe2 ...👍
 +
3/3 ( 100%): deleting deleteMe3 ...👍
 +
</source>
 +
After checking you might want to (carefully) use the "-f" option to actually force the deletion:
 +
<source lang='bash' highlight='1'>
 +
wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3 -f
 +
deleting 3 pages in test (forced)
 +
1/3 (  33%): deleting deleteMe1 ...✅
 +
2/3 (  67%): deleting deleteMe2 ...✅
 +
3/3 ( 100%): deleting deleteMe3 ...✅
 +
</source>
 +
 +
= WikiEdit =
 +
wikiedit.py  allows mass editing of pages using python regular expressions
 +
== usage ==
 +
<source lang='bash'>
 +
wikiedit -h
 +
usage: wikiedit.py [-h] [-d] [-V] --search SEARCH --replace REPLACE [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET
 +
                  [-p PAGES [PAGES ...]]
 +
 +
Created on 2020-11-12
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
USAGE
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: None]
 +
  -V, --version        show program's version number and exit
 +
  --search SEARCH      search pattern
 +
  --replace REPLACE    replace pattern
 +
  -f, --force          force to edit pages - default is 'dry' run only listing pages
 +
  -q QUERY, --query QUERY
 +
                        select pages with given SMW ask query
 +
  -qf QUERYFIELD, --queryField QUERYFIELD
 +
                        query result field which contains page
 +
  -t TARGET, --target TARGET
 +
                        target wiki id
 +
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
 +
                        list of page Titles to be pushed
 +
</source>
 +
 +
== example search/replace ==
 +
<source lang='bash' highlight='1'>
 +
wikiedit -t test -q "[[isA::CFP]]"  --search "CALL FOR PAPER" --replace "CFP"
 +
editing 1 pages in test (dry run)
 +
1/1 ( 100%): editing CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) ...👍 |isA=CFP
 +
-|Acronym=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ)
 +
-|Title=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ)
 +
+|Acronym=CFP Journal: Advances in Multimedia - An International Journal (AMIJ)
 +
+|Title=CFP Journal: Advances in Multimedia - An International Journal (AMIJ)
 +
|Start date=2010/11/01
 +
}}
 +
-CALL FOR PAPER
 +
+CFP
 +
Journal: Advances in Multimedia - An International Journal (AMIJ)
 +
</source>
 +
== example with query script ==
 +
=== qscholars ===
 +
<source lang='bash'>
 +
#!/bin/zsh
 +
# WF 2023-02-27
 +
 +
#ansi colors
 +
#http://www.csc.uvic.ca/~sae/seng265/fall04/tips/s265s047-tips/bash-using-colors.html
 +
blue='\033[0,34m'
 +
red='\033[0,31m'
 +
green='\033[0,32m' # '\e[1,32m' is too bright for white bg.
 +
endColor='\033[0m'
 +
 +
#
 +
# a colored message
 +
#  params:
 +
#    1: l_color - the color of the message
 +
#    2: l_msg - the message to display
 +
#
 +
color_msg() {
 +
  local l_color="$1"
 +
  local l_msg="$2"
 +
  echo -e "${l_color}$l_msg${endColor}"
 +
}
 +
 +
#
 +
# error
 +
#
 +
# show the given error message on stderr and exit
 +
#
 +
#  params:
 +
#    1: l_msg - the error message to display
 +
#
 +
error() {
 +
  local l_msg="$1"
 +
  # use ansi red for error
 +
  color_msg $red "Error:" 1>&2
 +
  color_msg $red "\t$l_msg" 1>&2
 +
  exit 1
 +
}
 +
 +
#
 +
# list of scholar properties
 +
#
 +
props() {
 +
  cat << EOF
 +
dblp,P2456,dblpId,DBLP author ID
 +
gnd,P227,gndId,GND ID
 +
google,P1960,googleScholarUser,Google Scholar author ID
 +
homepage,P856,homepage,official website
 +
linkedin,P6634,linkedInId,LinkedIn personal profile ID
 +
orcid,P496,orcid,ORCID ID
 +
research,P2038,researchGate,ResearchGate profile ID
 +
EOF
 +
}
 +
 +
#
 +
# get the ask query
 +
#
 +
ask_query() {
 +
  cat << EOF
 +
  {{#ask: [[Concept:Scholar]]
 +
  | mainLabel=scholar
 +
  | ?Scholar wikiDataId = item
 +
  | ?Scholar description = description
 +
  | ?Scholar name = name
 +
  | ?Scholar firstName = firstName
 +
EOF
 +
props | while IFS=, read -r option prop field prop_label
 +
do
 +
  echo "| ?Scholar $field=$field"
 +
done
 +
cat << EOF
 +
  | ?Scholar smartCRMId = smartCRMId
 +
  | ?Creation date=creation Date
 +
  |sort=Scholar name,Scholar firstName
 +
  |order=ascending,ascending
 +
  }}
 +
EOF
 +
}
 +
 +
#
 +
# query scholars from the given wikiId in the given FlorianMatthes
 +
#
 +
# params
 +
#  1: wikiId
 +
#  2: format e.g. -j/--json or -c/--csv
 +
#
 +
scholars() {
 +
  local l_wikiId="$1"
 +
  local l_option="$2"
 +
 +
  qf=/tmp/scholars$$
 +
  json=/tmp/scholar.json$$
 +
  csv=/tmp/scholar.csv$$
 +
  ask_query>$qf
 +
  #https://stackoverflow.com/questions/32960857/how-to-convert-arbitrary-simple-json-to-csv-using-jq
 +
  wikiquery -l -s ${l_wikiId} --queryFile $qf > $json
 +
  cat $json | jq '.data | (map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | join(";")' > $csv
 +
  case $l_option in
 +
    -j|--json) cat $json;;
 +
    -c|--csv) cat $csv | sed -e 's/"//g';;
 +
  esac
 +
  rm $json
 +
  rm $csv
 +
  rm $qf
 +
}
 +
 +
# show usage
 +
#
 +
usage() {
 +
  echo "$0 [-h|--help|--verbose]"
 +
  echo -n "["
 +
  local delim=""
 +
  props | while IFS=, read -r option prop field prop_label
 +
  do
 +
    echo -n "$delim$option"
 +
    delim="|"
 +
  done
 +
  echo "]"
 +
  echo "[Q*]*"
 +
  echo "-h  |--help:  show this usage"
 +
  echo "-u  |--update:  update the scholars json cache"
 +
  echo "-w  |--wikiId wikiId: set the wikiId to query"
 +
  echo "desc: wikidata description"
 +
  props | while IFS=, read -r option prop field prop_label
 +
  do
 +
    echo "$option: $prop_label"
 +
  done
 +
  echo "Q*: any list of wikidata identifiers to be filtered"
 +
  exit 1
 +
}
 +
 +
#
 +
# update the scholars list from the wiki
 +
#
 +
update_scholars() {
 +
  local l_wikiId="$1"
 +
  if [ ! -f "$jscholars" ]
 +
  then
 +
    color_msg $blue "getting scholars from wiki $l_wikiId"
 +
    scholars "$l_wikiId" -j > $jscholars
 +
  else
 +
    color_msg $green "$jscholars exists"
 +
  fi
 +
  scount=$(jq ".data[]| [.item] | @csv" $jscholars | wc -l)
 +
  color_msg $green "$scount scholars available"
 +
}
 +
 +
#
 +
# filter the scholars
 +
#
 +
# params:
 +
#    jscholars: the input json file to operate on
 +
#    field: the field to read and modify
 +
#    qlist: the list of q identifiers to filter for
 +
#    limit: the number of scholars to maximally read
 +
#
 +
filter_scholars() {
 +
  local jscholars="$1"
 +
  local field="$2"
 +
  local qlist="$3"
 +
  local limit="$4"
 +
  local csv="/tmp/scholars_$$.csv"
 +
  jq -r ".data[]| select (.item!=null) | [ .scholar, .item, .$field  ] | @csv" $jscholars \
 +
  | head -$limit > $csv
 +
  if [ "$qlist" = "" ]
 +
  then
 +
    pattern=".*"
 +
  else
 +
    pattern="$qlist"
 +
  fi
 +
  cat $csv | grep -E $pattern
 +
}
 +
 +
#
 +
# update the wiki
 +
#
 +
update_wiki() {
 +
  local l_wikiId="$1"
 +
  local prop="$2"
 +
  local field="$3"
 +
  local prop_label="$4"
 +
  local qlist="$5"
 +
  color_msg "updating wiki $l_wikiId for $field $prop $prop_label"
 +
  case $prop in
 +
    desc)
 +
    ;;
 +
    *)
 +
    wd props | grep "'"$prop"'"
 +
    ;;
 +
  esac
 +
  filter_scholars $jscholars $field "$qlist" 10000 | while IFS=, read -r page qid value
 +
  do
 +
    qid=$(echo $qid | sed 's#"##g')
 +
    page=$(echo $page | sed 's#"##g')
 +
    value=$(echo $value | sed 's#"##g')
 +
    if [ "$verbose" = "true" ]
 +
    then
 +
      echo "$page ($qid) $field=$value"
 +
    fi
 +
    if [ "$value" = "" ]
 +
    then
 +
      case $prop in
 +
        desc)
 +
          #color_msg $blue "getting $field for $qid"
 +
          wdValue=$(wd desc $qid 2>&1)
 +
          #echo $wdValue
 +
        ;;
 +
        *)
 +
        wdValue=$(wb query -s "$qid" -p $prop 2>&1)
 +
        ;;
 +
      esac
 +
      if [ $? -eq 0 ]
 +
      then
 +
        case $wdValue in
 +
          "*error*"|"no result found")
 +
          ;;
 +
          *)
 +
            color_msg $blue "updating $page $field to $wdValue from wikidata $qid"
 +
            wikiedit -t $l_wikiId -p "$page" --template Scholar --property $field --value "$wdValue" -f
 +
        esac
 +
      fi
 +
    fi
 +
  done
 +
}
 +
verbose="false"
 +
qlist=""
 +
delim=""
 +
wikiId="ceur-ws"
 +
jscholars=/tmp/${wikiId}_scholars.json
 +
if [ $# -lt 1 ]
 +
then
 +
  usage
 +
else
 +
  while [  "$1" != ""  ]
 +
  do
 +
    option="$1"
 +
    case $option in
 +
      -d|--debug)
 +
        set -x
 +
        ;;
 +
      -h|--help)
 +
        usage
 +
        ;;
 +
      desc)
 +
        prop="desc"
 +
        field="description"
 +
        prop_label="description"
 +
        update_wiki $wikiId $prop $field $prop_label $qlist
 +
        ;;
 +
      Q*)
 +
        qlist="$qlist$delim$option"
 +
        delim="|"
 +
        shift
 +
        continue
 +
        ;;
 +
      --props)
 +
        props | while IFS=, read -r option prop field prop_label
 +
        do
 +
          echo "$option:$prop:$field:$prop_label"
 +
        done
 +
        shift
 +
        continue
 +
        ;;
 +
      -u|--update)
 +
        update_scholars $wikiId
 +
        ;;
 +
      --verbose)
 +
          verbose="true"
 +
          ;;
 +
      -w|--wikiId)
 +
        shift
 +
        if [ $# -lt 1 ]
 +
        then
 +
          usage
 +
        fi
 +
        wikiId=$1;
 +
        jscholars=/tmp/${wikiId}_scholars.json
 +
        ;;
 +
      *)
 +
        found="false"
 +
        props | while IFS=, read -r l_option l_prop l_field l_prop_label
 +
        do
 +
          if [ "$option" = "$l_option" ]
 +
          then
 +
            shift
 +
            prop=$l_prop
 +
            field=$l_field
 +
            update_wiki $wikiId $prop $field $l_prop_label $qlist
 +
            found="true"
 +
            break
 +
          fi
 +
        done
 +
        if [ "$found" = "true" ]
 +
        then
 +
          continue
 +
        fi
 +
        echo "unknown field $1"
 +
        exit 1
 +
      ;;
 +
    esac
 +
    shift
 +
  done
 +
fi
 +
</source>
 +
==== updating description field from wikidata ===
 +
<source lang='bash' highlight='1'>
 +
qscholars desc
 +
updating wiki ceur-ws for description desc description
 +
updating Sören Auer description to founder of OntoWiki from wikidata Q27453085
 +
editing 1 pages in ceur-ws (forced)
 +
1/1 (100.00%): editing Sören Auer ...:weißes_häkchen:
 +
updating Tim Berners-Lee description to English computer scientist, inventor of the World Wide Web (born 1955) from wikidata Q80
 +
editing 1 pages in ceur-ws (forced)
 +
1/1 (100.00%): editing Tim Berners-Lee ...:weißes_häkchen:
 +
updating Christian Bizer description to researcher from wikidata Q17744291
 +
editing 1 pages in ceur-ws (forced)
 +
...
 +
</source>
 +
 +
= wikirestore =
 +
Tool to restore wiki pages from an local backup, created with wikibackup, to an destination wiki.
 +
 +
== Arguments ==
 +
{| class="wikitable"
 +
|-
 +
!| Argument
 +
!| Description
 +
|-
 +
| -s
 +
| Source wiki - Only used to query page names. The queried page names will then be looked up in the backup.
 +
|-
 +
| -t
 +
| Target wiki - The backup is restored in this wiki
 +
|-
 +
| -q
 +
| SMW query to select the pages to be restored. Note that the query is only used to select the page names the actual backup is then restored from the local backup.
 +
|-
 +
| -p
 +
| Names of the pages to be restored
 +
|-
 +
| --backupPath
 +
| define location of the backup. Default is the default backup location of the target wiki.
 +
|-
 +
 +
|}
 +
If argument '''-s''' is used a page query is executed therefore all arguments related to an page query can be used such as '''-ui''' and '''--limit'''.
 +
 +
== Examples ==
 +
=== --backupPath ===
 +
Use this argument to define a different backup folder
 +
====wikibackup====
 +
<syntaxhighlight lang="shell" line='line'>
 +
$ wikibackup -s orth --backupPath "/home/user/wikibackup/orth_copy" -q "[[isA::Event]]" --limit 10
 +
 +
downloading 10 pages from orth to /home/user/wikibackup/orth_copy
 +
1/10 (  10%): downloading " DBKDA 2021" ...✅
 +
2/10 (  20%): downloading "ENERGY 2021" ...✅
 +
3/10 (  30%): downloading "ICAS 2021" ...✅
 +
4/10 (  40%): downloading "ICNS 2021" ...✅
 +
5/10 (  50%): downloading 2021 ICIMP ...✅
 +
6/10 (  60%): downloading 3DUI 2020 ...✅
 +
7/10 (  70%): downloading 3IA 2009 ...✅
 +
8/10 (  80%): downloading 3PGIC 2010 ...✅
 +
9/10 (  90%): downloading 4S4D 2017 ...✅
 +
10/10 ( 100%): downloading 5GU 2017 ...✅
 +
</syntaxhighlight>
 +
 +
====wikirestore====
 +
<syntaxhighlight lang="shell">
 +
$ wikirestore -t orth --backupPath "/home/user/wikibackup/orth_copy"
 +
 +
restoring 10 pages from /home/user/wikibackup/orth_copy to orth
 +
1/10 (  10%): restore 2021 ICIMP ...✅
 +
2/10 (  20%): restore "ICNS 2021" ...✅
 +
3/10 (  30%): restore 3PGIC 2010 ...✅
 +
4/10 (  40%): restore 4S4D 2017 ...✅
 +
5/10 (  50%): restore "ENERGY 2021" ...✅
 +
6/10 (  60%): restore 3DUI 2020 ...✅
 +
7/10 (  70%): restore " DBKDA 2021" ...✅
 +
8/10 (  80%): restore 3IA 2009 ...✅
 +
9/10 (  90%): restore "ICAS 2021" ...✅
 +
10/10 ( 100%): restore 5GU 2017 ...✅
 +
</syntaxhighlight>
 +
 +
=== Scenario: Restore triangle ===
 +
<syntaxhighlight lang="shell">
 +
$ wikirestore -s or -q "[[isA:Event]]" -t orth --backupPath "/home/user/wikibackup/orth_copy"
 +
</syntaxhighlight>
 +
With this command we query all page names that are an Event from the wiki '''or''' and restore them in the wiki '''orth''' with the version of the page that is stored in '''/home/user/wikibackup/orth_copy'''.
 +
 +
= wikiupload =
 +
wikiupload.py allows to mass upload files
 +
== usage ==
 +
<source lang='bash'>
 +
wikiupload -h
 +
usage: wikiupload.py [-h] [-d] [-V] --files FILES [FILES ...] [-f] -t TARGET
 +
 +
Created on 2020-11-12
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
USAGE
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: None]
 +
  -V, --version        show program's version number and exit
 +
  --files FILES [FILES ...]
 +
                        list of files to be uploaded
 +
  -f, --force          force to (re)upload existing files - default is false
 +
  -t TARGET, --target TARGET
 +
                        target wiki id
 +
</source>
 +
 +
== example ==
 +
<source lang='bash' highlight='1'>
 +
wikiupload -t test --files car.png
 +
uploading 1 files to test
 +
1/1 ( 100%): uploading car.png ...✅
 +
</source>
 +
 +
= wikiuser =
 +
wikiuser.py  creates credential files and assigns a WikiId under which you can now operate. This simplifies access to your wiki.
 +
The credential file is compatible to the Java Mediawiki-Japi see {{Link|target=CommandLine#Credential_mode}}
 +
 +
== usage ==
 +
<source lang='bash'>
 +
wikiuser -h
 +
usage: wikiuser.py [-h] [-d] [-V] [-e EMAIL] [-f FILEPATH] [-l URL]
 +
                  [-s SCRIPTPATH] [-p PASSWORD] [-u USER] [-v VERSION]
 +
                  [-w WIKIID] [-y]
 +
 +
WikiUser credential handling
 +
 +
  Created by Wolfgang Fahl on 2020-10-31.
 +
  Copyright 2020 Wolfgang Fahl. All rights reserved.
 +
 +
  Licensed under the Apache License 2.0
 +
  http://www.apache.org/licenses/LICENSE-2.0
 +
 +
  Distributed on an "AS IS" basis without warranties
 +
  or conditions of any kind, either express or implied.
 +
 +
USAGE
 +
 +
optional arguments:
 +
  -h, --help            show this help message and exit
 +
  -d, --debug          set debug level [default: None]
 +
  -V, --version        show program's version number and exit
 +
  -e EMAIL, --email EMAIL
 +
                        email of the user
 +
  -f FILEPATH, --file FILEPATH
 +
                        ini-file path
 +
  -l URL, --url URL    url of the wiki
 +
  -s SCRIPTPATH, --scriptPath SCRIPTPATH
 +
                        script path
 +
  -p PASSWORD, --password PASSWORD
 +
                        password
 +
  -u USER, --user USER  os user id
 +
  -v VERSION, --wikiVersion VERSION
 +
                        version of the wiki
 +
  -w WIKIID, --wikiId WIKIID
 +
                        wiki Id
 +
  -y, --yes            immediately store without asking
 +
</source>
 +
=== Example ===
 +
E.g. if you have an account on www.semantic-mediawiki.org you can start wikiuser in interactive mode.
 +
 +
<source lang='bash' highlight='1'>
 +
wikiuser
 +
email: john@doe.com
 +
scriptPath: /w
 +
user: jd
 +
url: http://www.semantic-mediawiki.org
 +
version: Mediawiki 1.33
 +
wikiId: smw
 +
password: *****
 +
shall i store jd smw? yes/no y/ny
 +
</source>
 +
 +
Now you can e.g. use "smw" as the wikiid for this wiki when using wikipush
 +
 +
= Prerequisites =
 +
You might want to prepare some credential ini files with the wikiuser script or Mediawiki-Japi [[CommandLine]].
 +
 +
== user-config.py ==
 +
pywikibot expects a user-config.py file. The minimum recommended file for intranet usecases is:
 +
<source lang='python'>
 +
# https://stackoverflow.com/a/60885381/1497139
 +
# Slow down the robot such that it never makes a second page edit within
 +
# 'put_throttle' seconds.
 +
put_throttle = 0
 +
# avoid warnings ...
 +
family='bitplan'
 +
mylang='en'
 +
</source>
 +
The easiest way is to put it at $HOME/.pywikibot/user-config.py
 +
 +
= Features =
 +
 +
== Encrypted credential handling ==
 +
Py-3rdparty-mediawiki allows using pywikibot by simply giving each wiki an id and using the credential information created by MediaWiki-Japi. The needed family file is automatically created and registered. If you'd like to get a pure python solution for credential handling please file an issue on github - it's no big deal but i personally don't need it yet since i'm fine with the new CommandLine feature added recently.
 +
 +
== Semantic MediaWiki API support ==
 +
see https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/1
 +
 +
== Example ==
 +
<source lang='python'>
 +
from wikibot.wikibot import WikiBot
 +
wikibot=WikiBot.ofWikiId("test2")
 +
wikibot.site ...
 +
</source>

Latest revision as of 09:20, 9 November 2024

OsProject

OsProject
edit
id  Py-3rdparty-mediawiki
state  active
owner  WolfgangFahl
title  Py-3rdparty-mediawiki
url  https://github.com/WolfgangFahl/py-3rdparty-mediawiki
version  0.14.1
description  python 3rd party Mediawiki bot support enhancements e.g. encrypted credential handling
date  2024-11-09
since  2020-03-26
until  


Installation

pip install py-3rdparty-mediawiki
# alternatively if your pip is not a python3 pip
pip3 install py-3rdparty-mediawiki 
# local install from source directory of py-3rdparty-mediawiki 
pip install .

upgrade

pip install py-3rdparty-mediawiki  -U
# alternatively if your pip is not a python3 pip
pip3 install py-3rdparty-mediawiki -U


What is it

Extended functionality for

  1. mwclient

Originally we intended to use also support:

the results where disappointing - pywikibot kept being a source of trouble more than a solution to our needs

Github


Via Source code

git clone https://github.com/WolfgangFahl/py-3rdparty-mediawiki
./install

wikipush / wikibackup / wikiedit / wiknuke / wikirestore/ wikiquery / wikiupload / wikiuser scripts

Setup method

If you installed with the method above console_script will have been added to your environment. You can e.g. check 

which wikipush
/Users/wf/Library/Python/3.8/bin/wikipush

and there should be a wikipush script in your path.

WikiPush

WikiPush allows to copy pages from one wiki to another including the images on the page. To identify yourself you use the credential property files created with the wikiuser script (using python) or the Mediawiki-Japi CommandLine

usage

wikipush -h
family and mylang are not set.
Defaulting to family='test' and mylang='test'.
usage: wikipush.py [-h] [-d] [-V] [-l] [-f] [-i] [-q QUERY] -s SOURCE -t
                   TARGET [-p PAGES [PAGES ...]]

Created on 2020-10-29

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -l, --login           login to source wiki for access permission
  -f, --force           force to overwrite existing pages
  -i, --ignore          ignore upload warnings e.g. duplicate images
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -s SOURCE, --source SOURCE
                        source wiki id
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

Example

wikipush -s smw -t test2 -q "[[Category:City]]|limit=5"
family and mylang are not set.
Defaulting to family='test' and mylang='test'.
copying 4 pages from smw to test2
copying Demo:Tokyo ...✅
copying image File:SMW-Info-button.png ...✅
copying image File:Tokyo-Tsukishima-0011.jpg ...✅
copying Vienna ...✅
copying Warsaw ...✅
copying image File:6140285934 02e81b845f z.jpg ...✅
copying Demo:Würzburg ...✅

wikiquery

wikiquery allows to send SMW ask-query via commandline and get the results in json or csv format. With the query division parameter the limits of SMW for the maximum amount of displayed results can be overcome. E.g. if you set 

$smwgQMaxInlineLimit=1500;
$smwgQMaxInlineLimitSets=1500;
$smwgQMaxLimit = 5000;

You'll be able to get more than 1500/5000 results.

usage

wikiquery -h
usage: wikiquery [-h] [-d] [-V] [-l] -s SOURCE [--format FORMAT]
                 [--entityName ENTITYNAME] [--limit LIMIT] [--progress]
                 [-q QUERY] [--queryFile QUERYFILE] [-qf QUERYFIELD]
                 [-p PAGES [PAGES ...]] [-ui] [-qd QUERYDIVISION]

wikipush

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: False]
  -V, --version         show program's version number and exit
  -l, --login           login to source wiki for access permission
  -s SOURCE, --source SOURCE
                        source wiki id
  --format FORMAT       format to use for query result csv,json,xml,ttl or
                        wiki
  --entityName ENTITYNAME
                        name of the entites that are queried - only needed for
                        some output formats - default is 'data'
  --limit LIMIT         limit for query
  --progress            shows progress for query
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  --queryFile QUERYFILE
                        file the query should be read from
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed
  -ui, --withGUI        Pop up GUI for selection
  -qd QUERYDIVISION, --queryDivision QUERYDIVISION
                        divide query into equidistant subintervals to limit
                        the result size of the individual queries

Examples

query1.ask

{{#ask: [[IsA::Event]][[Acronym::~ES*]][[start date::>2018]][[start date::<2019]] 
| mainlabel=pageTitle
| ?Title = title 
| ?Event in series = series 
| ?ordinal=ordinal 
| ?Homepage = homepage 
| format=table 
}}

csv

wikiquery -s or --queryFile query1.ask --format csv
pageTitle;title;series;ordinal;homepage
ESA 2018;26th Annual European Symposium on Algorithms;ESA;None;http://algo2018.hiit.fi/esa/
ESEC/FSE 2018;26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE);ESEC/FSE;None;https://2018.fseconference.org/
ESOP 2018;27th European Symposium on Programming;ESOP;None;https://etaps.org/2018/esop
ESORICS 2018;23rd European Symposium on Research in Computer Security,;ESORICS;None;None
ESSCIRC 2018;44th European Solid-State Circuits Conference;ESSCIRC;None;None
ESWC 2018;15th European Semantic Web Symposium (ESWS);ESWC;None;http://2018.eswc-conferences.org/

json

wikiquery -s or --queryFile query1.ask --format json

{
   "data": [
      {
         "pageTitle": "ESA 2018",
         "title": "26th Annual European Symposium on Algorithms",
         "series": "ESA",
         "ordinal": null,
         "homepage": "http://algo2018.hiit.fi/esa/"
      },
      {
         "pageTitle": "ESEC/FSE 2018",
         "title": "26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)",
         "series": "ESEC/FSE",
         "ordinal": null,
         "homepage": "https://2018.fseconference.org/"
      },
      {
         "pageTitle": "ESOP 2018",
         "title": "27th European Symposium on Programming",
         "series": "ESOP",
         "ordinal": null,
         "homepage": "https://etaps.org/2018/esop"
      },
      {
         "pageTitle": "ESORICS 2018",
         "title": "23rd European Symposium on Research in Computer Security,",
         "series": "ESORICS",
         "ordinal": null,
         "homepage": null
      },
      {
         "pageTitle": "ESSCIRC 2018",
         "title": "44th European Solid-State Circuits Conference",
         "series": "ESSCIRC",
         "ordinal": null,
         "homepage": null
      },
      {
         "pageTitle": "ESWC 2018",
         "title": "15th European Semantic Web Symposium (ESWS)",
         "series": "ESWC",
         "ordinal": null,
         "homepage": "http://2018.eswc-conferences.org/"
      }
   ]
}

wikibackup

usage

wikibackup -h
usage: wikibackup [-h] [-d] [-V] [-g] [-l] -s SOURCE [-wi]
                  [--backupPath BACKUPPATH] [--limit LIMIT] [--progress]
                  [-q QUERY] [--queryFile QUERYFILE] [-qf QUERYFIELD]
                  [-p PAGES [PAGES ...]] [-ui] [-qd QUERYDIVISION]

wikipush

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: False]
  -V, --version         show program's version number and exit
  -g, --git             use git for version control
  -l, --login           login to source wiki for access permission
  -s SOURCE, --source SOURCE
                        source wiki id
  -wi, --withImages     copy images on the given pages
  --backupPath BACKUPPATH
                        path where the backup should be stored
  --limit LIMIT         limit for query
  --progress            shows progress for query
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  --queryFile QUERYFILE
                        file the query should be read from
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed
  -ui, --withGUI        Pop up GUI for selection
  -qd QUERYDIVISION, --queryDivision QUERYDIVISION
                        divide query into equidistant subintervals to limit
                        the result size of the individual queries

WikiNuke

wikinukes.py allows mass deletion of pages

usage

usage: wikinuke.py [-h] [-d] [-V] [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET [-p PAGES [PAGES ...]]

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -f, --force           force to delete pages - default is 'dry' run only listing pages
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

Example

The default behavior is a dry run only listing whether the pages exist 

wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3
deleting 3 pages in test (dry run)
1/3 (  33%): deleting deleteMe1 ...👍
2/3 (  67%): deleting deleteMe2 ...👍
3/3 ( 100%): deleting deleteMe3 ...👍

After checking you might want to (carefully) use the "-f" option to actually force the deletion: 

wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3 -f
deleting 3 pages in test (forced)
1/3 (  33%): deleting deleteMe1 ...✅
2/3 (  67%): deleting deleteMe2 ...✅
3/3 ( 100%): deleting deleteMe3 ...✅

WikiEdit

wikiedit.py allows mass editing of pages using python regular expressions

usage

wikiedit -h
usage: wikiedit.py [-h] [-d] [-V] --search SEARCH --replace REPLACE [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET
                   [-p PAGES [PAGES ...]]

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  --search SEARCH       search pattern
  --replace REPLACE     replace pattern
  -f, --force           force to edit pages - default is 'dry' run only listing pages
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

example search/replace

wikiedit -t test -q "[[isA::CFP]]"  --search "CALL FOR PAPER" --replace "CFP"
editing 1 pages in test (dry run)
1/1 ( 100%): editing CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) ...👍 |isA=CFP
-|Acronym=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) 
-|Title=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) 
+|Acronym=CFP Journal: Advances in Multimedia - An International Journal (AMIJ) 
+|Title=CFP Journal: Advances in Multimedia - An International Journal (AMIJ) 
 |Start date=2010/11/01
 }}
-CALL FOR PAPER
+CFP
 Journal: Advances in Multimedia - An International Journal (AMIJ)

example with query script

qscholars

#!/bin/zsh
# WF 2023-02-27

#ansi colors
#http://www.csc.uvic.ca/~sae/seng265/fall04/tips/s265s047-tips/bash-using-colors.html
blue='\033[0,34m'
red='\033[0,31m'
green='\033[0,32m' # '\e[1,32m' is too bright for white bg.
endColor='\033[0m'

#
# a colored message
#   params:
#     1: l_color - the color of the message
#     2: l_msg - the message to display
#
color_msg() {
  local l_color="$1"
  local l_msg="$2"
  echo -e "${l_color}$l_msg${endColor}"
}

#
# error
#
# show the given error message on stderr and exit
#
#   params:
#     1: l_msg - the error message to display
#
error() {
  local l_msg="$1"
  # use ansi red for error
  color_msg $red "Error:" 1>&2
  color_msg $red "\t$l_msg" 1>&2
  exit 1
}

#
# list of scholar properties
#
props() {
  cat << EOF
dblp,P2456,dblpId,DBLP author ID
gnd,P227,gndId,GND ID
google,P1960,googleScholarUser,Google Scholar author ID
homepage,P856,homepage,official website
linkedin,P6634,linkedInId,LinkedIn personal profile ID
orcid,P496,orcid,ORCID ID
research,P2038,researchGate,ResearchGate profile ID
EOF
}

#
# get the ask query
#
ask_query() {
  cat << EOF
  {{#ask: [[Concept:Scholar]]
  | mainLabel=scholar
  | ?Scholar wikiDataId = item
  | ?Scholar description = description
  | ?Scholar name = name
  | ?Scholar firstName = firstName
EOF
props | while IFS=, read -r option prop field prop_label
do
  echo "| ?Scholar $field=$field"
done
cat << EOF
  | ?Scholar smartCRMId = smartCRMId
  | ?Creation date=creation Date
  |sort=Scholar name,Scholar firstName
  |order=ascending,ascending
  }}
EOF
}

#
# query scholars from the given wikiId in the given FlorianMatthes
#
# params
#  1: wikiId
#  2: format e.g. -j/--json or -c/--csv
#
scholars() {
  local l_wikiId="$1"
  local l_option="$2"

  qf=/tmp/scholars$$
  json=/tmp/scholar.json$$
  csv=/tmp/scholar.csv$$
  ask_query>$qf
  #https://stackoverflow.com/questions/32960857/how-to-convert-arbitrary-simple-json-to-csv-using-jq
  wikiquery -l -s ${l_wikiId} --queryFile $qf > $json
  cat $json | jq '.data | (map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | join(";")' > $csv
  case $l_option in
    -j|--json) cat $json;;
    -c|--csv) cat $csv | sed -e 's/"//g';;
  esac
  rm $json
  rm $csv
  rm $qf
}

# show usage
#
usage() {
  echo "$0 [-h|--help|--verbose]"
  echo -n "["
  local delim=""
  props | while IFS=, read -r option prop field prop_label
  do
    echo -n "$delim$option"
    delim="|"
  done
  echo "]"
  echo "[Q*]*"
  echo "-h  |--help:  show this usage"
  echo "-u  |--update:  update the scholars json cache"
  echo "-w  |--wikiId wikiId: set the wikiId to query"
  echo "desc: wikidata description"
  props | while IFS=, read -r option prop field prop_label
  do
    echo "$option: $prop_label"
  done
  echo "Q*: any list of wikidata identifiers to be filtered"
  exit 1
}

#
# update the scholars list from the wiki
#
update_scholars() {
  local l_wikiId="$1"
  if [ ! -f "$jscholars" ]
  then
    color_msg $blue "getting scholars from wiki $l_wikiId"
    scholars "$l_wikiId" -j > $jscholars
  else
    color_msg $green "$jscholars exists"
  fi
  scount=$(jq ".data[]| [.item] | @csv" $jscholars | wc -l)
  color_msg $green "$scount scholars available"
}

#
# filter the scholars
#
# params:
#    jscholars: the input json file to operate on
#    field: the field to read and modify
#    qlist: the list of q identifiers to filter for
#    limit: the number of scholars to maximally read
#
filter_scholars() {
  local jscholars="$1"
  local field="$2"
  local qlist="$3"
  local limit="$4"
  local csv="/tmp/scholars_$$.csv"
  jq -r ".data[]| select (.item!=null) | [ .scholar, .item, .$field  ] | @csv" $jscholars \
  | head -$limit > $csv
  if [ "$qlist" = "" ]
  then
    pattern=".*"
  else
    pattern="$qlist"
  fi
  cat $csv | grep -E $pattern
}

#
# update the wiki
#
update_wiki() {
  local l_wikiId="$1"
  local prop="$2"
  local field="$3"
  local prop_label="$4"
  local qlist="$5"
  color_msg "updating wiki $l_wikiId for $field $prop $prop_label"
  case $prop in
    desc)
    ;;
    *)
    wd props | grep "'"$prop"'"
    ;;
  esac
  filter_scholars $jscholars $field "$qlist" 10000 | while IFS=, read -r page qid value
  do
    qid=$(echo $qid | sed 's#"##g')
    page=$(echo $page | sed 's#"##g')
    value=$(echo $value | sed 's#"##g')
    if [ "$verbose" = "true" ]
    then
      echo "$page ($qid) $field=$value"
    fi
    if [ "$value" = "" ]
    then
      case $prop in
        desc)
          #color_msg $blue "getting $field for $qid"
          wdValue=$(wd desc $qid 2>&1)
          #echo $wdValue
        ;;
        *)
        wdValue=$(wb query -s "$qid" -p $prop 2>&1)
        ;;
      esac
      if [ $? -eq 0 ]
      then
        case $wdValue in
          "*error*"|"no result found")
          ;;
          *)
            color_msg $blue "updating $page $field to $wdValue from wikidata $qid"
            wikiedit -t $l_wikiId -p "$page" --template Scholar --property $field --value "$wdValue" -f
        esac
      fi
    fi
  done
}
verbose="false"
qlist=""
delim=""
wikiId="ceur-ws"
jscholars=/tmp/${wikiId}_scholars.json
if [ $# -lt 1 ]
then
  usage
else
  while [  "$1" != ""  ]
  do
    option="$1"
    case $option in
      -d|--debug)
        set -x
        ;;
      -h|--help)
        usage
        ;;
      desc)
        prop="desc"
        field="description"
        prop_label="description"
        update_wiki $wikiId $prop $field $prop_label $qlist
        ;;
      Q*)
        qlist="$qlist$delim$option"
        delim="|"
        shift
        continue
        ;;
      --props)
        props | while IFS=, read -r option prop field prop_label
        do
          echo "$option:$prop:$field:$prop_label"
        done
        shift
        continue
        ;;
      -u|--update)
        update_scholars $wikiId
        ;;
      --verbose)
          verbose="true"
          ;;
      -w|--wikiId)
        shift
        if [ $# -lt 1 ]
        then
          usage
        fi
        wikiId=$1;
        jscholars=/tmp/${wikiId}_scholars.json
        ;;
      *)
        found="false"
        props | while IFS=, read -r l_option l_prop l_field l_prop_label
        do
          if [ "$option" = "$l_option" ]
          then
            shift
            prop=$l_prop
            field=$l_field
            update_wiki $wikiId $prop $field $l_prop_label $qlist
            found="true"
            break
          fi
        done
        if [ "$found" = "true" ]
        then
          continue
        fi
        echo "unknown field $1"
        exit 1
      ;;
    esac
    shift
  done
fi

= updating description field from wikidata

qscholars desc
updating wiki ceur-ws for description desc description
updating Sören Auer description to founder of OntoWiki from wikidata Q27453085
editing 1 pages in ceur-ws (forced)
1/1 (100.00%): editing Sören Auer ...:weißes_häkchen:
updating Tim Berners-Lee description to English computer scientist, inventor of the World Wide Web (born 1955) from wikidata Q80
editing 1 pages in ceur-ws (forced)
1/1 (100.00%): editing Tim Berners-Lee ...:weißes_häkchen:
updating Christian Bizer description to researcher from wikidata Q17744291
editing 1 pages in ceur-ws (forced)
...

wikirestore

Tool to restore wiki pages from an local backup, created with wikibackup, to an destination wiki.

Arguments

Argument Description
-s Source wiki - Only used to query page names. The queried page names will then be looked up in the backup.
-t Target wiki - The backup is restored in this wiki
-q SMW query to select the pages to be restored. Note that the query is only used to select the page names the actual backup is then restored from the local backup.
-p Names of the pages to be restored
--backupPath define location of the backup. Default is the default backup location of the target wiki.

If argument -s is used a page query is executed therefore all arguments related to an page query can be used such as -ui and --limit.

Examples

--backupPath

Use this argument to define a different backup folder

wikibackup

 1$ wikibackup -s orth --backupPath "/home/user/wikibackup/orth_copy" -q "[[isA::Event]]" --limit 10
 2
 3downloading 10 pages from orth to /home/user/wikibackup/orth_copy
 41/10 (  10%): downloading " DBKDA 2021" ...✅
 52/10 (  20%): downloading "ENERGY 2021" ...✅
 63/10 (  30%): downloading "ICAS 2021" ...✅
 74/10 (  40%): downloading "ICNS 2021" ...✅
 85/10 (  50%): downloading 2021 ICIMP ...✅
 96/10 (  60%): downloading 3DUI 2020 ...✅
107/10 (  70%): downloading 3IA 2009 ...✅
118/10 (  80%): downloading 3PGIC 2010 ...✅
129/10 (  90%): downloading 4S4D 2017 ...✅
1310/10 ( 100%): downloading 5GU 2017 ...✅

wikirestore

$ wikirestore -t orth --backupPath "/home/user/wikibackup/orth_copy"

restoring 10 pages from /home/user/wikibackup/orth_copy to orth
1/10 (  10%): restore 2021 ICIMP ...✅
2/10 (  20%): restore "ICNS 2021" ...✅
3/10 (  30%): restore 3PGIC 2010 ...✅
4/10 (  40%): restore 4S4D 2017 ...✅
5/10 (  50%): restore "ENERGY 2021" ...✅
6/10 (  60%): restore 3DUI 2020 ...✅
7/10 (  70%): restore " DBKDA 2021" ...✅
8/10 (  80%): restore 3IA 2009 ...✅
9/10 (  90%): restore "ICAS 2021" ...✅
10/10 ( 100%): restore 5GU 2017 ...✅

Scenario: Restore triangle

$ wikirestore -s or -q "[[isA:Event]]" -t orth --backupPath "/home/user/wikibackup/orth_copy"

With this command we query all page names that are an Event from the wiki or and restore them in the wiki orth with the version of the page that is stored in /home/user/wikibackup/orth_copy.

wikiupload

wikiupload.py allows to mass upload files

usage

wikiupload -h
usage: wikiupload.py [-h] [-d] [-V] --files FILES [FILES ...] [-f] -t TARGET

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  --files FILES [FILES ...]
                        list of files to be uploaded
  -f, --force           force to (re)upload existing files - default is false
  -t TARGET, --target TARGET
                        target wiki id

example

wikiupload -t test --files car.png
uploading 1 files to test
1/1 ( 100%): uploading car.png ...✅

wikiuser

wikiuser.py creates credential files and assigns a WikiId under which you can now operate. This simplifies access to your wiki. The credential file is compatible to the Java Mediawiki-Japi see CommandLine#Credential_mode

usage

wikiuser -h
usage: wikiuser.py [-h] [-d] [-V] [-e EMAIL] [-f FILEPATH] [-l URL]
                   [-s SCRIPTPATH] [-p PASSWORD] [-u USER] [-v VERSION]
                   [-w WIKIID] [-y]

WikiUser credential handling

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -e EMAIL, --email EMAIL
                        email of the user
  -f FILEPATH, --file FILEPATH
                        ini-file path
  -l URL, --url URL     url of the wiki
  -s SCRIPTPATH, --scriptPath SCRIPTPATH
                        script path
  -p PASSWORD, --password PASSWORD
                        password
  -u USER, --user USER  os user id
  -v VERSION, --wikiVersion VERSION
                        version of the wiki
  -w WIKIID, --wikiId WIKIID
                        wiki Id
  -y, --yes             immediately store without asking

Example

E.g. if you have an account on www.semantic-mediawiki.org you can start wikiuser in interactive mode. 

wikiuser
email: john@doe.com
scriptPath: /w
user: jd
url: http://www.semantic-mediawiki.org
version: Mediawiki 1.33
wikiId: smw
password: *****
shall i store jd smw? yes/no y/ny

Now you can e.g. use "smw" as the wikiid for this wiki when using wikipush

Prerequisites

You might want to prepare some credential ini files with the wikiuser script or Mediawiki-Japi CommandLine.

user-config.py

pywikibot expects a user-config.py file. The minimum recommended file for intranet usecases is: 

# https://stackoverflow.com/a/60885381/1497139
# Slow down the robot such that it never makes a second page edit within
# 'put_throttle' seconds.
put_throttle = 0
# avoid warnings ...
family='bitplan'
mylang='en'

The easiest way is to put it at $HOME/.pywikibot/user-config.py

Features

Encrypted credential handling

Py-3rdparty-mediawiki allows using pywikibot by simply giving each wiki an id and using the credential information created by MediaWiki-Japi. The needed family file is automatically created and registered. If you'd like to get a pure python solution for credential handling please file an issue on github - it's no big deal but i personally don't need it yet since i'm fine with the new CommandLine feature added recently.

Semantic MediaWiki API support

see https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/1

Example

from wikibot.wikibot import WikiBot
wikibot=WikiBot.ofWikiId("test2")
wikibot.site ...
Retrieved from "https://wiki.bitplan.com/index.php?title=Py-3rdparty-mediawiki&oldid=19203"