OsProject

OsProject
edit
id	pymediawikidocker
state	active
owner	WolfgangFahl
title	pymediawikidocker
url	https://github.com/WolfgangFahl/pymediawikidocker
version	0.8.3
description	Python controlled mediawiki docker image installation
date	2023-04-06
since	2021-06-08
until

tickets

Motivation

Official Mediawiki docker images are available for the different Mediawiki versions.

These images need additional infrastructure to create useable Mediawiki environments. The goal of this project is to semi-automatically create such environments and test the functionality. Python was choosen as a development environment for the libraries that are needed and available to perform the tasks:

https://github.com/gabrieldemarmiesse/python-on-whales - to automate the docker configuration and start

python on whales was chosen over:

https://pypi.org/project/docker/

Since it works on the command line interface and not via the SDK and supports docker compose.

How it works

The core concept is a "MediawikiCluster" that is a set of docker applications that run MediaWiki instances with varying versions of MediaWiki and the underlying database. The graphic belows shows the docker containers which are created for different versions of Mediawiki

Installation

Prerequisites

Docker on Windows, Mac or Linux suppporting "docker compose up" using the Docker composer V2 CLI

Ubuntu 22.04 LTS installation

see https://docs.docker.com/engine/install/ubuntu/ on how to setup the docker repositorysudo

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
docker --version
Docker version 23.0.1, build a5ee5b1
docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

...

Compatibility test

# test on a Mac OS machine 2022-07-02
docker 2>&1 | grep compose
  compose*    Docker Compose (Docker Inc., v2.4.1)
# test on an Ubuntu machine 2023-02-22
docker 2>&1 | grep compose
 compose*    Docker Compose (Docker Inc., v2.16.0)

Installation of Docker Compose V2 CLI

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

docker compose version
Docker Compose version v2.12.0

sudo apt  install docker-compose
docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
which docker-compose
/usr/bin/docker-compose

pymediawikidocker installation via pip

pip install pymediawikidocker
# alternatively if your pip is not a python3 pip
pip3 install pymediawikidocker

upgrade

pip install pymediawikidocker -U
# alternatively if your pip is not a python3 pip
pip3 install pymediawikidocker -U

check version

mwcluster -V
mwcluster v0.7.0 (2023-01-25)

Usage

mwcluster -h
usage: mwCluster [-h] [-bp BASEPORT] [-c] [-cn CONTAINER_NAME] [-d]
                 [-el [EXTENSIONNAMELIST ...]] [-ej EXTENSIONJSONFILE] [-f]
                 [--logo LOGO] [-mv MARIADBVERSION] [-p PASSWORD]
                 [-sp SQLPORT] [-smw SMWVERSION] [-u USER] [-V]
                 [-vl [VERSIONS ...]] [-wl [WIKIIDLIST ...]]

mwcluster Created by Wolfgang Fahl on 2021-06-21. Copyright 2021-2022 Wolfgang
Fahl. All rights reserved. Licensed under the Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0 Distributed on an "AS IS" basis
without warranties or conditions of any kind, either express or implied.

options:
  -h, --help            show this help message and exit
  -bp BASEPORT, --basePort BASEPORT
                        set how base html port 80 to be exposed - incrementing
                        by one for each version [default: 9080]
  -c, --check           check the wikis [default: False]
  -cn CONTAINER_NAME, --container_name CONTAINER_NAME
                        set container name (only valid and recommended for
                        single version call) (default: None)
  -d, --debug           set debug level [default: False]
  -el [EXTENSIONNAMELIST ...], --extensionList [EXTENSIONNAMELIST ...]
                        list of extensions to be installed [default: ['Admin
                        Links', 'Header Tabs', 'SyntaxHighlight',
                        'Variables']]
  -ej EXTENSIONJSONFILE, --extensionJson EXTENSIONJSONFILE
                        additional extension descriptions default: None
                        (default: None)
  -f, --forceRebuild    shall the applications rebuild be forced (with stop
                        and remove of existing containers) (default: False)
  --logo LOGO           set Logo [default:
                        $wgResourceBasePath/resources/assets/wiki.png]
  -mv MARIADBVERSION, --mariaDBVersion MARIADBVERSION
                        mariaDB Version to be installed [default: 10.9]
  -p PASSWORD, --password PASSWORD
                        set password for initial user [default: sysop-1234!]
  -sp SQLPORT, --sqlBasePort SQLPORT
                        set base mySql port 3306 to be exposed - incrementing
                        by one for each version [default: 9306]
  -smw SMWVERSION, --smwVersion SMWVERSION
                        set SemanticMediaWiki Version to be installed default
                        is None - no installation of SMW (default: None)
  -u USER, --user USER  set username of initial user with sysop rights
                        [default: Sysop]
  -V, --version         show program's version number and exit
  -vl [VERSIONS ...], --versionList [VERSIONS ...]
                        mediawiki versions to create docker applications for
                        [default: ['1.27.7', '1.31.16', '1.35.8', '1.37.6',
                        '1.38.4']]
  -wl [WIKIIDLIST ...], --wikiIdList [WIKIIDLIST ...]
                        list of wikiIDs to be used for for py-3rdparty-
                        mediawiki wikiuser quick access (default: None)

Examples

Freitext

Pymediawikidocker_example_2022-10-26

Default settings

Starts a cluster with versions 1.27.7, 1.31.16, 1.35.9, 1.37.6,1.38.4 and 1.39.1 using MariaDB 10.9

mwcluster -f
creating docker compose applications for mediawiki versions ['1.27.7', '1.31.16', '1.35.9', '1.37.6', '1.38.4', '1.39.1']
...
starting mediawiki 1.27.7 docker application ...
..
Initializing MediaWiki SQL tables
Trying DB-Connection to wiki on localhost port 9308 with user wikiuser with max 6 tries and 10s timeout per try - initial sleep 2.5s
Connection to wiki on localhost with user wikiuser established database returns: ('wiki',)

Mediawiki applications will be available at

http://localhost:9080 http://localhost:9080/index.php/Special:Version

http://localhost:9081 http://localhost:9081/index.php/Special:Version

http://localhost:9082 http://localhost:9082/index.php/Special:Version

http://localhost:9083 http://localhost:9083/index.php/Special:Version

http://localhost:9084 http://localhost:9084/index.php/Special:Version

http://localhost:9085 http://localhost:9085/index.php/Special:Version

check

mwcluster -c
mwcluster -c
checking docker access for mediawiki versions ['1.27.7', '1.31.16', '1.35.9', '1.37.6', '1.38.4', '1.39.1']
0:checking 1.27.7 ...
mediawiki webserver container mw1_27_7-mw:✅
mediawiki database container mw1_27_7-db:✅
port binding 9080= expected  port 9080?:✅
Checking http://arche.bitplan.com:9080/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.27.7= expected  1.27.7?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
1:checking 1.31.16 ...
mediawiki webserver container mw1_31_16-mw:✅
mediawiki database container mw1_31_16-db:✅
port binding 9081= expected  port 9081?:✅
Checking http://arche.bitplan.com:9081/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.31.16= expected  1.31.16?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
2:checking 1.35.9 ...
mediawiki webserver container mw1_35_9-mw:✅
mediawiki database container mw1_35_9-db:✅
port binding 9082= expected  port 9082?:✅
Checking http://arche.bitplan.com:9082/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.35.9= expected  1.35.9?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
3:checking 1.37.6 ...
mediawiki webserver container mw1_37_6-mw:✅
mediawiki database container mw1_37_6-db:✅
port binding 9083= expected  port 9083?:✅
Checking http://arche.bitplan.com:9083/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.37.6= expected  1.37.6?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
4:checking 1.38.4 ...
mediawiki webserver container mw1_38_4-mw:✅
mediawiki database container mw1_38_4-db:✅
port binding 9084= expected  port 9084?:✅
Checking http://arche.bitplan.com:9084/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.38.4= expected  1.38.4?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
5:checking 1.39.1 ...
mediawiki webserver container mw1_39_1-mw:✅
mediawiki database container mw1_39_1-db:✅
port binding 9085= expected  port 9085?:✅
Checking http://arche.bitplan.com:9085/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.39.1= expected  1.39.1?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅

Single 1.35.8

# python mwdocker/mwcluster.py  can be used instead of mwcluster if you test from your locally cloned environment
mwcluster \
  --forceRebuild\
  --versionList 1.35.8\
  --basePort 8480\
  --sqlBasePort 10306\
  --mariaDBVersion 10.9\
  --container_name mw135lts\
  --logo https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Bigsmile_smiley_yellow_simple.svg/128px-Bigsmile_smiley_yellow_simple.svg.png

Mediawiki application will be available at

http://localhost:8480 http://localhost:8480/index.php/Special:Version

Single 1.35 with Semantic Mediawiki

There might be more current versions of SMW available see https://www.semantic-mediawiki.org/wiki/Semantic_MediaWiki_Version_History and adapt the example below if you see the need to do so.

Version 4.0.2

mwcluster \
  --container_name mw135smw4\
  --forceRebuild\
  --versionList 1.35.8\
  --basePort 8481\
  --sqlBasePort 10307\
  --mariaDBVersion 10.9\
  --wikiIdList mw135smw4_test\
  -smw 4.0.2

debug docker compose results - ports and other configuration parts

cd ~/.pymediawikidocker/mw135smw4
docker-compose up
# in another terminal session:
docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS        PORTS                     NAMES
3ab643a8e647   mw135smw4_mw   "docker-php-entrypoi…"   18 hours ago   Up 18 hours   0.0.0.0:8481->80/tcp      mw135smw4-mw
a456c87a83a6   mariadb:10.9   "docker-entrypoint.s…"   18 hours ago   Up 18 hours   0.0.0.0:10307->3306/tcp   mw135smw4-db

open bash in docker container

docker exec -it mw135smw4-mw /bin/bash

check installExtensions.sh script

cat /tmp/installExtensions.sh 
#!/bin/bash
# install the required extensions
# WF 2021-06-23
cd /var/www/html/extensions

# Admin Links
# https://www.mediawiki.org/wiki/Extension:Admin_Links
git clone https://github.com/wikimedia/mediawiki-extensions-AdminLinks --single-branch --branch REL1_35 AdminLinks

# Header Tabs
# https://www.mediawiki.org/wiki/Extension:Header_Tabs
git clone https://github.com/wikimedia/mediawiki-extensions-HeaderTabs --single-branch --branch REL1_35 HeaderTabs

# SyntaxHighlight
# https://www.mediawiki.org/wiki/Extension:SyntaxHighlight
# no installation script command specified

# Variables
# https://www.mediawiki.org/wiki/Extension:Variables
git clone https://github.com/wikimedia/mediawiki-extensions-Variables --single-branch --branch REL1_35 Variables

check Semantic MediaWiki settings in LocalSettings.php

grep -i semantic LocalSettings.php 
# enable Support for Semantic MediaWiki
# see https://www.semantic-mediawiki.org/wiki/Help:EnableSemantics
# Version of SemanticMediaWiki at install time: 4.0.2
wfLoadExtension( 'SemanticMediaWiki' );
enableSemantics();

SMW Version 3.2.3

Installation

mwcluster \
  --container_name mw135smw3\
  --forceRebuild\
  --versionList 1.35.8\
  --basePort 8482\
  --sqlBasePort 10308\
  --mariaDBVersion 10.9\
  --wikiIdList mw135smw3_test\
  --logo https://www.semantic-mediawiki.org/w/images/4/4e/SMW_logo_180.png\
  -smw 3.2.3

Check

mwcluster \
  --container_name mw135smw3\
  --forceRebuild\
  --versionList 1.35.8\
  --basePort 8482\
  --sqlBasePort 10308\
  --mariaDBVersion 10.9\
  --wikiIdList mw135smw3_test\
  -smw 3.2.3 --check
checking docker access for mediawiki versions ['1.35.8']
0:checking 1.35.8 ...
mediawiki webserver container mw135smw3-mw:✅
mediawiki database container mw135smw3-db:✅
port binding 8482= expected  port 8482?:✅
Checking http://localhost:8482/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.35.8= expected  1.35.8?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅

Extensions

see Pymediawikidocker/Extensions

Debugging

Docker files

You'll find the files created by pymediawikidocker in ".pymediawikidocker" in your homedirectory.

pwd
/Users/***/.pymediawikidocker/mw1_35_2
ls -l
total 176
-rw-r--r--  1 wf  staff   1057 22 Jun 07:03 Dockerfile
-rw-r--r--  1 wf  staff   4580 22 Jun 07:03 LocalSettings.php
-rw-r--r--  1 wf  staff   1161 22 Jun 07:03 docker-compose.yml
-rw-r--r--  1 wf  staff    317 22 Jun 07:03 initdb.sh
-rw-r--r--  1 wf  staff    125 22 Jun 07:03 phpinfo.php
-rw-r--r--  1 wf  staff  64515 22 Jun 07:03 wiki.sql

Check installed docker images

docker images | grep mw
mw1_38_4_mw    latest    c647ad44a344   About a minute ago   996MB
mw1_37_6_mw    latest    af9de508138c   About a minute ago   980MB
mw1_35_8_mw    latest    f41c84ab16c3   2 minutes ago        988MB
mw1_31_16_mw   latest    9a8a26949b58   2 minutes ago        897MB
mw1_27_7_mw    latest    1fa159362ec9   2 minutes ago        898MB

Bash into docker containers

# Mediawiki container
docker exec -it mw1_38_4-mw /bin/bash
docker exec -it mw1_37_6-mw /bin/bash
docker exec -it mw1_35_8-mw /bin/bash
docker exec -it mw1_31_16-mw /bin/bash
docker exec -it mw1_27_7-mw /bin/bash

# MariaDB container
docker exec -it mw1_38_4-db /bin/bash
docker exec -it mw1_37_6-db /bin/bash
docker exec -it mw1_35_8-db /bin/bash
docker exec -it mw1_31_16-db /bin/bash
docker exec -it mw1_27_7-db /bin/bash

Monkey patching

# Enable raw html in for the wiki
docker exec  mw1_35_8-mw sh -c "echo '\$wgRawHtml = true;\n' >> /var/www/html/LocalSettings.php"

No space left on device issue

e.g. when starting mariadb container see https://stackoverflow.com/questions/37645879/how-can-i-fix-docker-mac-no-space-left-on-device-error

Wiki Backup

The passwords are available from the docker files e.g.

grep PASS docker-compose.yml 
      MYSQL_PASSWORD: "TnbKf2zGRmZPjUvz7B"
      MYSQL_ROOT_PASSWORD: "QWehnR9yLMY7YAb9pw"

In a docker bash a dump can be created

mysqldump wiki -u wikiuser --password=TnbKf2zGRmZPjUvz7B > /tmp/wiki.sql

and then copied to your local machine

docker cp mw1_36_0_db_1:/tmp/wiki.sql /tmp

Docker cleanup procedures

Stop and Remove all running containers

⚠️ These commands will stop and remove all running docker containers (not only the ones created pymediawikidocker/the mwcluster command)

# kill/stop all running containers
docker kill $(docker ps -q)
#docker stop $(docker ps -aq)
# remove all docker containers
docker rm $(docker ps -aq)

Recreate Clean docker state

remove all images

⚠️ This command will remove all your images not only the ones created by pymediawikidocker (docker will hopefully refuse to do this without the -f option)

# remove all images
docker rmi $(docker images -q)

system prune

#⚠️ WARNING! This will remove:
#  - all stopped containers
#  - all networks not used by at least one container
#  - all dangling images
#  - all dangling build cache
docker system prune

docker volumes

⚠️ these commands will remove all your volumes - you might loose valuable data if you don't check what your volumes contain before starting this command

# check volumes
docker volume ls
# remove unused volumes
docker volume prune
# remove all volumes
# see https://stackoverflow.com/a/46091568/1497139
docker volume rm $(docker volume ls -qf dangling=true | xargs)

Links

Wikidata entry for Mediawiki software

Pymediawikidocker

OsProject

tickets

Motivation

How it works

Installation

Prerequisites

Ubuntu 22.04 LTS installation

Compatibility test

Installation of Docker Compose V2 CLI

pymediawikidocker installation via pip

upgrade

check version

Usage

Examples

Freitext

Default settings

check

Single 1.35.8

Single 1.35 with Semantic Mediawiki

Version 4.0.2

debug docker compose results - ports and other configuration parts

open bash in docker container

SMW Version 3.2.3

Installation

Check

Extensions

Debugging

Docker files

Check installed docker images

Bash into docker containers

Monkey patching

No space left on device issue

Wiki Backup

Docker cleanup procedures

Stop and Remove all running containers

Recreate Clean docker state

remove all images

system prune

docker volumes

Links

Navigation menu

Search