Pymediawikidocker
Revision as of 11:14, 28 October 2022 by Wf (talk | contribs) (→debug docker compose results - ports and other configuration parts)
OsProject
| OsProject | |
|---|---|
 edit
| |
| id | pymediawikidocker |
| state | |
| owner | WolfgangFahl |
| title | Python controlled mediawiki docker image installation |
| url | https://github.com/WolfgangFahl/pymediawikidocker |
| version | 0.5.0 |
| description | |
| date | 2022-10-27 |
| since | |
| until | |
Motivation
Official Mediawiki docker images are available for the different Mediawiki versions.
These images need additional infrastructure to create useable Mediawiki environments. The goal of this project is to semi-automatically create such environments and test the functionality. Python was choosen as a development environment for the libraries that are needed and available to perform the tasks:
- https://github.com/gabrieldemarmiesse/python-on-whales - to automate the docker configuration and start
python on whales was chosen over:
Since it works on the command line interface and not via the SDK and supports docker compose.
How it works
The core concept is a "MediawikiCluster" that is a set of docker applications that run MediaWiki instances with varying versions of MediaWiki and the underlying database.
Installation
Prerequisites
Docker on Windows, Mac or Linux suppporting "docker compose up" using the Docker composer V2 CLI
Ubuntu 22.04 LTS installation
see https://docs.docker.com/engine/install/ubuntu/ on how to setup the docker repositorysudo
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
docker --version
docker run hello-world
Docker version 20.10.20, build 9fdeb9c
Compatibility test
# test on a Mac OS machine 2022-07-02
docker 2>&1 | grep compose
compose* Docker Compose (Docker Inc., v2.4.1)
# test on an Ubuntu machine 2022-10-23
docker 2>&1 | grep compose
compose* Docker Compose (Docker Inc., v2.12.0)
Installation of Docker Compose V2 CLI
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
docker compose version
Docker Compose version v2.12.0
sudo apt install docker-compose
docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022
which docker-compose
/usr/bin/docker-compose
pymediawikidocker installation via pip
pip install pymediawikidocker
# alternatively if your pip is not a python3 pip
pip3 install pymediawikidocker
upgrade
pip install pymediawikidocker -U
# alternatively if your pip is not a python3 pip
pip3 install pymediawikidocker -U
check version
mwcluster -V
mwCluster v0.5.0 (2022-10-27)
Usage
mwcluster -h
usage: mwCluster [-h] [-bp BASEPORT] [-c] [-cn CONTAINER_NAME] [-d]
[-el [EXTENSIONNAMELIST ...]] [-ej EXTENSIONJSONFILE] [-f]
[--logo LOGO] [-mv MARIADBVERSION] [-p PASSWORD]
[-sp SQLPORT] [-smw SMWVERSION] [-u USER] [-V]
[-vl [VERSIONS ...]] [-wl [WIKIIDLIST ...]]
mwcluster Created by Wolfgang Fahl on 2021-06-21. Copyright 2021-2022 Wolfgang
Fahl. All rights reserved. Licensed under the Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0 Distributed on an "AS IS" basis
without warranties or conditions of any kind, either express or implied.
options:
-h, --help show this help message and exit
-bp BASEPORT, --basePort BASEPORT
set how base html port 80 to be exposed - incrementing
by one for each version [default: 9080]
-c, --check check the wikis [default: False]
-cn CONTAINER_NAME, --container_name CONTAINER_NAME
set container name (only valid and recommended for
single version call) (default: None)
-d, --debug set debug level [default: False]
-el [EXTENSIONNAMELIST ...], --extensionList [EXTENSIONNAMELIST ...]
list of extensions to be installed [default: ['Admin
Links', 'Header Tabs', 'SyntaxHighlight',
'Variables']]
-ej EXTENSIONJSONFILE, --extensionJson EXTENSIONJSONFILE
additional extension descriptions default: None
(default: None)
-f, --forceRebuild shall the applications rebuild be forced (with stop
and remove of existing containers) (default: False)
--logo LOGO set Logo [default:
$wgResourceBasePath/resources/assets/wiki.png]
-mv MARIADBVERSION, --mariaDBVersion MARIADBVERSION
mariaDB Version to be installed [default: 10.9]
-p PASSWORD, --password PASSWORD
set password for initial user [default: sysop-1234!]
-sp SQLPORT, --sqlBasePort SQLPORT
set base mySql port 3306 to be exposed - incrementing
by one for each version [default: 9306]
-smw SMWVERSION, --smwVersion SMWVERSION
set SemanticMediaWiki Version to be installed default
is None - no installation of SMW (default: None)
-u USER, --user USER set username of initial user with sysop rights
[default: Sysop]
-V, --version show program's version number and exit
-vl [VERSIONS ...], --versionList [VERSIONS ...]
mediawiki versions to create docker applications for
[default: ['1.27.7', '1.31.16', '1.35.8', '1.37.6',
'1.38.4']]
-wl [WIKIIDLIST ...], --wikiIdList [WIKIIDLIST ...]
list of wikiIDs to be used for for py-3rdparty-
mediawiki wikiuser quick access (default: None)
Examples
see also Pymediawikidocker_example_2022-10-26
Default settings
Starts a cluster with versions 1.27.7, 1.31.16, 1.35.8, 1.37.6 and 1.38.4 using MariaDB 10.9
mwcluster -f
creating docker applications for mediawiki versions ['1.27.7', '1.31.16', '1.35.8', '1.37.6', '1.38.4']
...
starting mediawiki 1.27.7 docker application ...
..
Initializing MediaWiki SQL tables
Trying DB-Connection to wiki on localhost port 9308 with user wikiuser with max 6 tries and 10s timeout per try - initial sleep 2.5s
Connection to wiki on localhost with user wikiuser established database returns: ('wiki',)
Mediawiki applications will be available at
http://localhost:9080 http://localhost:9080/index.php/Special:Version
http://localhost:9081 http://localhost:9081/index.php/Special:Version
http://localhost:9082 http://localhost:9082/index.php/Special:Version
http://localhost:9083 http://localhost:9083/index.php/Special:Version
http://localhost:9084 http://localhost:9084/index.php/Special:Version
check
mwcluster -c
mwcluster -c
checking docker access for mediawiki versions ['1.27.7', '1.31.16', '1.35.8', '1.37.6', '1.38.4']
0:checking 1.27.7 ...
mediawiki webserver container mw1_27_7-mw:✅
mediawiki database container mw1_27_7-db:✅
port binding 9080= expected port 9080?:✅
Checking http://localhost:9080/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.27.7= expected 1.27.7?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
1:checking 1.31.16 ...
mediawiki webserver container mw1_31_16-mw:✅
mediawiki database container mw1_31_16-db:✅
port binding 9081= expected port 9081?:✅
Checking http://localhost:9081/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.31.16= expected 1.31.16?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
2:checking 1.35.8 ...
mediawiki webserver container mw1_35_8-mw:✅
mediawiki database container mw1_35_8-db:✅
port binding 9082= expected port 9082?:✅
Checking http://localhost:9082/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.35.8= expected 1.35.8?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
3:checking 1.37.6 ...
mediawiki webserver container mw1_37_6-mw:✅
mediawiki database container mw1_37_6-db:✅
port binding 9083= expected port 9083?:✅
Checking http://localhost:9083/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.37.6= expected 1.37.6?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
4:checking 1.38.4 ...
mediawiki webserver container mw1_38_4-mw:✅
mediawiki database container mw1_38_4-db:✅
port binding 9084= expected port 9084?:✅
Checking http://localhost:9084/index.php/Special:Version ...
Special Version accessible ...:✅
Mediawiki Version 1.38.4= expected 1.38.4?:✅
Maria DB Version 10.9 fitting expected 10.9?:✅
Single 1.35.8
# python mwdocker/mwcluster.py can be used instead of mwcluster if you test from your locally cloned environment
mwcluster \
--forceRebuild\
--versionList 1.35.8\
--basePort 8480\
--sqlBasePort 10306\
--mariaDBVersion 10.9\
--container_name mw135lts\
--logo https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Bigsmile_smiley_yellow_simple.svg/128px-Bigsmile_smiley_yellow_simple.svg.png
Mediawiki application will be available at
http://localhost:8480 http://localhost:8480/index.php/Special:Version
Single 1.35 with Semantic Mediawiki
mwcluster \
--container_name mw135smw4\
--forceRebuild\
--versionList 1.35.8\
--basePort 8481\
--sqlBasePort 10307\
--mariaDBVersion 10.9\
--wikiIdList mw135smw4_test\
-smw 4.0.2
debug docker compose results - ports and other configuration parts
cd ~/.pymediawikidocker/mw135smw4
docker-compose up
# in another terminal session:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3ab643a8e647 mw135smw4_mw "docker-php-entrypoi…" 18 hours ago Up 18 hours 0.0.0.0:8481->80/tcp mw135smw4-mw
a456c87a83a6 mariadb:10.9 "docker-entrypoint.s…" 18 hours ago Up 18 hours 0.0.0.0:10307->3306/tcp mw135smw4-db
open bash in docker container
docker exec -it mw135smw-mw4 /bin/bash
cat /tmp/installExtensions.sh
#!/bin/bash
# install the required extensions
# WF 2021-06-23
cd /var/www/html/extensions
# Admin Links
# https://www.mediawiki.org/wiki/Extension:Admin_Links
git clone https://github.com/wikimedia/mediawiki-extensions-AdminLinks --single-branch --branch REL1_35 AdminLinks
# Header Tabs
# https://www.mediawiki.org/wiki/Extension:Header_Tabs
git clone https://github.com/wikimedia/mediawiki-extensions-HeaderTabs --single-branch --branch REL1_35 HeaderTabs
# SyntaxHighlight
# https://www.mediawiki.org/wiki/Extension:SyntaxHighlight
# no installation script command specified
# Variables
# https://www.mediawiki.org/wiki/Extension:Variables
git clone https://github.com/wikimedia/mediawiki-extensions-Variables --single-branch --branch REL1_35 Variables
Extensions
see Pymediawikidocker/Extensions
Debugging
Docker files
You'll find the files created by pymediawikidocker in ".pymediawikidocker" in your homedirectory.
pwd
/Users/***/.pymediawikidocker/mw1_35_2
ls -l
total 176
-rw-r--r-- 1 wf staff 1057 22 Jun 07:03 Dockerfile
-rw-r--r-- 1 wf staff 4580 22 Jun 07:03 LocalSettings.php
-rw-r--r-- 1 wf staff 1161 22 Jun 07:03 docker-compose.yml
-rw-r--r-- 1 wf staff 317 22 Jun 07:03 initdb.sh
-rw-r--r-- 1 wf staff 125 22 Jun 07:03 phpinfo.php
-rw-r--r-- 1 wf staff 64515 22 Jun 07:03 wiki.sql
Check installed docker images
docker images | grep mw
mw1_38_4_mw latest c647ad44a344 About a minute ago 996MB
mw1_37_6_mw latest af9de508138c About a minute ago 980MB
mw1_35_8_mw latest f41c84ab16c3 2 minutes ago 988MB
mw1_31_16_mw latest 9a8a26949b58 2 minutes ago 897MB
mw1_27_7_mw latest 1fa159362ec9 2 minutes ago 898MB
Bash into docker containers
# Mediawiki container
docker exec -it mw1_38_4-mw /bin/bash
docker exec -it mw1_37_6-mw /bin/bash
docker exec -it mw1_35_8-mw /bin/bash
docker exec -it mw1_31_16-mw /bin/bash
docker exec -it mw1_27_7-mw /bin/bash
# MariaDB container
docker exec -it mw1_38_4-db /bin/bash
docker exec -it mw1_37_6-db /bin/bash
docker exec -it mw1_35_8-db /bin/bash
docker exec -it mw1_31_16-db /bin/bash
docker exec -it mw1_27_7-db /bin/bash
Monkey patching
# Enable raw html in for the wiki
docker exec mw1_35_3-mw-1 sh -c "echo '\$wgRawHtml = true;\n' >> /var/www/html/LocalSettings.php"
No space left on device issue
e.g. when starting mariadb container see https://stackoverflow.com/questions/37645879/how-can-i-fix-docker-mac-no-space-left-on-device-error
Wiki Backup
The passwords are available from the docker files e.g.
grep PASS docker-compose.yml
MYSQL_PASSWORD: "TnbKf2zGRmZPjUvz7B"
MYSQL_ROOT_PASSWORD: "QWehnR9yLMY7YAb9pw"
In a docker bash a dump can be created
mysqldump wiki -u wikiuser --password=TnbKf2zGRmZPjUvz7B > /tmp/wiki.sql
and then copied to your local machine
docker cp mw1_36_0_db_1:/tmp/wiki.sql /tmp
Stop and Remove all running containers
These commands will stop and remove all running docker containers (not only the ones created pymediawikidocker/the mwcluster command)
# kill/stop all running containers
docker kill $(docker ps -q)
#docker stop $(docker ps -aq)
# remove all docker containers
docker rm $(docker ps -aq)
Recreate Clean docker state
# remove all images
docker rmi $(docker images -q)