Difference between revisions of "Pynomina"

From BITPlan Wiki
Jump to navigation Jump to search
 
(23 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
|title=pynomina
 
|title=pynomina
 
|url=https://github.com/WolfgangFahl/pynomina
 
|url=https://github.com/WolfgangFahl/pynomina
−
|version=0.0.1
+
|version=0.0.5
 
|description=personal finance tool
 
|description=personal finance tool
−
|date=2024-10-06
+
|date=2024-10-10
 
|since=2024-10-06
 
|since=2024-10-06
 
}}
 
}}
 +
= Demo =
 +
https://nomina.bitplan.com
 +
= Introduction =
 +
pynomina is a personal finance tool designed to provide a flexible and enduring solution for managing financial records from diverse personal accounting tool sources.
 +
It aims to address the challenges of data conversion between different accounting software and ensure long-term readability of financial data.
 +
There is only limited support for the actual accounting tasks - this can be done much better by existing commercial and open source solutions. pynomina aims to give you the long-term freedom of
 +
choice between tools.
 +
 +
== Motivation ==
 +
In the past decades the author used different personal accounting tools:
 +
* [https://en.wikipedia.org/wiki/Quicken Quicken]
 +
* [https://en.wikipedia.org/wiki/Microsoft_Money Microsoft Money]
 +
* [https://www.wikidata.org/wiki/Q1822341 Lexware Finanzmanager]
 +
* [https://www.wikidata.org/wiki/Q130438296 BankingZV]
 +
 +
The [https://wiki.bitplan.com/index.php/IT_Pain_Scale pain] the conversion between those tools created was finally big enough to do something about it.
 +
 +
== Goals ==
 +
* use a computer and human-readable ledger format that is ready to survive decades
 +
* convert from and to the formats of the tool of choice
 +
* allow for simple sanity checks and reports
 +
* allow for systematic tidy up
 +
* allow for integration into a larger organizational knowledge graph
 +
 +
 +
== Hub & Spoke Conversion ==
 +
The pyNomina tool follows a '''Hub and Spoke''' model for conversion between different personal accounting file formats. The '''Ledger Book (YAML/JSON)''' format acts as the hub, with each supported format serving as a spoke. This setup simplifies conversions by allowing data to be transformed from any spoke to the hub and then to any other spoke format.
 +
 +
https://diagrams.bitplan.com/render/png/0xa0be5aae.png
 +
 +
=== Supported Formats ===
 +
{| class="wikitable"
 +
! Format !! Type !! Description !! Wikidata Entry
 +
|-
 +
| '''Ledger Book YAML/JSON''' || '''Hub''' || Main format of pyNomina for converting between formats. || [https://www.wikidata.org/wiki/Q281876 Ledger Book]
 +
|-
 +
| Beancount || Spoke || A plaintext accounting format. || [https://www.wikidata.org/wiki/Q130456404 Beancount]
 +
|-
 +
| GnuCash XML || Spoke || An XML-based format used by GnuCash. || [https://www.wikidata.org/wiki/Q130445392 GnuCash]
 +
|-
 +
| Microsoft Money || Spoke || Zip File exported with mny_export script using mdb-tools || [https://www.wikidata.org/wiki/Q117428 Microsoft Money]
 +
|-
 +
| Finanzmanager Deluxe (QIF) || Spoke || A variant of QIF used by Finanzmanager Deluxe. || [https://www.wikidata.org/wiki/Q1822341 Finanzmanager Deluxe]
 +
|-
 +
| Quicken Interchange Format || Spoke || Quicken Interchange Format (QIF) || [https://www.wikidata.org/wiki/Q750657 Quicken]
 +
|-
 +
| pyNomina Banking ZV YAML || Spoke || A format for exporting banking data in YAML or JSON. || [https://www.wikidata.org/wiki/Q130438296 Banking ZV]
 +
|}
 +
 +
{{pip|pynomina}}
 +
 +
= Usage =
 +
== command line ==
 +
<source lang='bash' highlight='1'>
 +
nomina -h
 +
usage: nomina [-h] [-a] [--apache APACHE] [-c] [-d]
 +
              [--debugServer DEBUGSERVER] [--debugPort DEBUGPORT]
 +
              [--debugRemotePath DEBUGREMOTEPATH]
 +
              [--debugLocalPath DEBUGLOCALPATH] [-l] [-i INPUT] [-rol]
 +
              [--host HOST] [--port PORT] [-s] [-V] [-v] [-rp ROOT_PATH]
 +
              [--convert CONVERT] [--format {LB-YAML,GC-XML,BEAN}] [-o OUTPUT]
 +
 +
Personal finance tool
 +
 +
options:
 +
  -h, --help            show this help message and exit
 +
  -a, --about          show about info [default: False]
 +
  --apache APACHE      create an apache configuration file for the given
 +
                        domain
 +
  -c, --client          start client [default: False]
 +
  -d, --debug          show debug info [default: False]
 +
  --debugServer DEBUGSERVER
 +
                        remote debug Server
 +
  --debugPort DEBUGPORT
 +
                        remote debug Port
 +
  --debugRemotePath DEBUGREMOTEPATH
 +
                        remote debug Server path mapping - remotePath - path
 +
                        on debug server
 +
  --debugLocalPath DEBUGLOCALPATH
 +
                        remote debug Server path mapping - localPath - path on
 +
                        machine where python runs
 +
  -l, --local          run with local file system access [default: False]
 +
  -i INPUT, --input INPUT
 +
                        input file
 +
  -rol, --render_on_load
 +
                        render on load [default: False]
 +
  --host HOST          the host to serve / listen from [default: localhost]
 +
  --port PORT          the port to serve from [default: 9849]
 +
  -s, --serve          start webserver [default: False]
 +
  -V, --version        show program's version number and exit
 +
  -v, --verbose        show verbose output [default: False]
 +
  -rp ROOT_PATH, --root_path ROOT_PATH
 +
                        path to nomina files [default:
 +
                        /Users/wf/Library/Python/3.12/lib/python/site-
 +
                        packages/nomina_examples]
 +
  --convert CONVERT    Convert the specified file to the desired format
 +
  --format {LB-YAML,GC-XML,BEAN}
 +
                        Output format for conversion [default: LB-YAML]
 +
  -o OUTPUT, --output OUTPUT
 +
                        Output file
 +
</source>
 +
== command line conversion ==
 +
There is a script test_cmdline that tries out some combinations of input and output file formats.
 +
see also [https://github.com/WolfgangFahl/pynomina/issues/7 Issue 7:command line conversion support]
 +
<source lang='bash' highlight='1'>
 +
scripts/test_cmdline
 +
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.yaml: LB-YAML✅
 +
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.gnucash: GC-XML✅
 +
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.beancount: BEAN✅
 +
...
 +
</source>
 +
== starting as local webapp ==
 +
<source lang='bash' highlight='1'>
 +
nomina -s -l
 +
NiceGUI ready to go on http://localhost:9849
 +
</source>
 +
 +
= Tutorials =
 +
== Quicken ==
 +
{{Link|target=Pynomina/Quicken}}
 +
== GnuCash ==
 +
{{Link|target=Pynomina/GnuCash}}
 +
== Beancount ==
 +
{{Link|target=Pynomina/Beancount}}
 +
== Microsoft Money ==
 +
{{Link|target=Pynomina/MsMoney}}
 +
 
=tickets=
 
=tickets=
 +
# {{Ticket
 +
|number=19
 +
|title=add account view
 +
|project=pynomina
 +
|createdAt=2024-10-13 13:22:07+00:00
 +
|closedAt=2024-10-13 13:22:27+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=18
 +
|title=add a book view
 +
|project=pynomina
 +
|createdAt=2024-10-12 16:52:44+00:00
 +
|closedAt=2024-10-12 16:53:06+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=16
 +
|title=handle VAT Codes
 +
|project=pynomina
 +
|createdAt=2024-10-12 11:49:11+00:00
 +
|closedAt=
 +
|state=open
 +
}}
 +
# {{Ticket
 +
|number=15
 +
|title=enforce beancount account naming
 +
|project=pynomina
 +
|createdAt=2024-10-11 13:07:50+00:00
 +
|closedAt=
 +
|state=open
 +
}}
 +
# {{Ticket
 +
|number=14
 +
|title=add calc_balances and remove_unused_accounts function
 +
|project=pynomina
 +
|createdAt=2024-10-10 12:04:44+00:00
 +
|closedAt=2024-10-10 12:05:49+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=12
 +
|title=add Microsoft Money support
 +
|project=pynomina
 +
|createdAt=2024-10-09 11:43:54+00:00
 +
|closedAt=2024-10-10 03:49:15+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=10
 +
|title=Beancount conversion problem with expenses2024_bzv example
 +
|project=pynomina
 +
|createdAt=2024-10-09 09:32:22+00:00
 +
|closedAt=2024-10-11 08:26:00+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=9
 +
|title=file format detection
 +
|project=pynomina
 +
|createdAt=2024-10-08 09:25:56+00:00
 +
|closedAt=2024-10-08 10:07:27+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=8
 +
|title=Beancount conversion support
 +
|project=pynomina
 +
|createdAt=2024-10-07 06:50:25+00:00
 +
|closedAt=2024-10-09 09:32:36+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=7
 +
|title=command line conversion support
 +
|project=pynomina
 +
|createdAt=2024-10-06 19:30:37+00:00
 +
|closedAt=2024-10-09 09:30:53+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=6
 +
|title=Subsembly JSON (BankingZV) export support
 +
|project=pynomina
 +
|createdAt=2024-10-06 19:26:20+00:00
 +
|closedAt=2024-10-09 09:12:14+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=5
 +
|title=Quicken QIF format support (very simple or Finanzmanager Deluxe Export format for a start ...)
 +
|project=pynomina
 +
|createdAt=2024-10-06 16:50:15+00:00
 +
|closedAt=2024-10-06 19:25:04+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=4
 +
|title=GnuCash XML support
 +
|project=pynomina
 +
|createdAt=2024-10-06 12:07:54+00:00
 +
|closedAt=2024-10-06 19:25:23+00:00
 +
|state=closed
 +
}}
 +
# {{Ticket
 +
|number=3
 +
|title=Abstract ledger
 +
|project=pynomina
 +
|createdAt=2024-10-06 08:58:56+00:00
 +
|closedAt=2024-10-06 12:06:15+00:00
 +
|state=closed
 +
}}
 
# {{Ticket
 
# {{Ticket
 
|number=2
 
|number=2
Line 21: Line 260:
 
|state=closed
 
|state=closed
 
}}
 
}}
 +
 +
= Testing =
 +
The testcases are python unittest modules see https://github.com/WolfgangFahl/pynomina/tree/main/tests.
 +
These tests are run as part of the [https://github.com/WolfgangFahl/pynomina/blob/main/.github/workflows/build.yml continuous integration github actions]
 +
To run the tests manually there is a "test" script in the scripts directory:
 +
<source lang='bash' highlight='1'>
 +
scripts/test
 +
Starting test test_read_bzv, debug=True ...
 +
# Accounts: 3
 +
# Transactions: 2
 +
Date Range: 2024-10-06 to 2024-10-06
 +
# Categories: 1
 +
# Currencies: EUR: 2
 +
Other Details:
 +
  name: expenses2024
 +
  owner: John Doe
 +
test test_read_bzv, debug=True took  0.0 s
 +
.Starting test test_conversions, debug=True ...
 +
Converting Ledger Book None to GC-XML
 +
 +
...
 +
Ran 17 tests in 4.518s
 +
 +
OK
 +
</source>
 +
{{:Pynomina/ledger}}

Latest revision as of 08:59, 15 October 2024

OsProject

OsProject
id  pynomina
state  active
owner  WolfgangFahl
title  pynomina
url  https://github.com/WolfgangFahl/pynomina
version  0.0.5
description  personal finance tool
date  2024-10-10
since  2024-10-06
until  

Demo

https://nomina.bitplan.com

Introduction

pynomina is a personal finance tool designed to provide a flexible and enduring solution for managing financial records from diverse personal accounting tool sources. It aims to address the challenges of data conversion between different accounting software and ensure long-term readability of financial data. There is only limited support for the actual accounting tasks - this can be done much better by existing commercial and open source solutions. pynomina aims to give you the long-term freedom of choice between tools.

Motivation

In the past decades the author used different personal accounting tools:

The pain the conversion between those tools created was finally big enough to do something about it.

Goals

  • use a computer and human-readable ledger format that is ready to survive decades
  • convert from and to the formats of the tool of choice
  • allow for simple sanity checks and reports
  • allow for systematic tidy up
  • allow for integration into a larger organizational knowledge graph


Hub & Spoke Conversion

The pyNomina tool follows a Hub and Spoke model for conversion between different personal accounting file formats. The Ledger Book (YAML/JSON) format acts as the hub, with each supported format serving as a spoke. This setup simplifies conversions by allowing data to be transformed from any spoke to the hub and then to any other spoke format.

0xa0be5aae.png

Supported Formats

Format Type Description Wikidata Entry
Ledger Book YAML/JSON Hub Main format of pyNomina for converting between formats. Ledger Book
Beancount Spoke A plaintext accounting format. Beancount
GnuCash XML Spoke An XML-based format used by GnuCash. GnuCash
Microsoft Money Spoke Zip File exported with mny_export script using mdb-tools Microsoft Money
Finanzmanager Deluxe (QIF) Spoke A variant of QIF used by Finanzmanager Deluxe. Finanzmanager Deluxe
Quicken Interchange Format Spoke Quicken Interchange Format (QIF) Quicken
pyNomina Banking ZV YAML Spoke A format for exporting banking data in YAML or JSON. Banking ZV


Installation

pip install pynomina
# alternatively if your pip is not a python3 pip
pip3 install pynomina 
# local install from source directory of pynomina 
pip install .

upgrade

pip install pynomina  -U
# alternatively if your pip is not a python3 pip
pip3 install pynomina -U


Usage

command line

nomina -h
usage: nomina [-h] [-a] [--apache APACHE] [-c] [-d]
              [--debugServer DEBUGSERVER] [--debugPort DEBUGPORT]
              [--debugRemotePath DEBUGREMOTEPATH]
              [--debugLocalPath DEBUGLOCALPATH] [-l] [-i INPUT] [-rol]
              [--host HOST] [--port PORT] [-s] [-V] [-v] [-rp ROOT_PATH]
              [--convert CONVERT] [--format {LB-YAML,GC-XML,BEAN}] [-o OUTPUT]

Personal finance tool

options:
  -h, --help            show this help message and exit
  -a, --about           show about info [default: False]
  --apache APACHE       create an apache configuration file for the given
                        domain
  -c, --client          start client [default: False]
  -d, --debug           show debug info [default: False]
  --debugServer DEBUGSERVER
                        remote debug Server
  --debugPort DEBUGPORT
                        remote debug Port
  --debugRemotePath DEBUGREMOTEPATH
                        remote debug Server path mapping - remotePath - path
                        on debug server
  --debugLocalPath DEBUGLOCALPATH
                        remote debug Server path mapping - localPath - path on
                        machine where python runs
  -l, --local           run with local file system access [default: False]
  -i INPUT, --input INPUT
                        input file
  -rol, --render_on_load
                        render on load [default: False]
  --host HOST           the host to serve / listen from [default: localhost]
  --port PORT           the port to serve from [default: 9849]
  -s, --serve           start webserver [default: False]
  -V, --version         show program's version number and exit
  -v, --verbose         show verbose output [default: False]
  -rp ROOT_PATH, --root_path ROOT_PATH
                        path to nomina files [default:
                        /Users/wf/Library/Python/3.12/lib/python/site-
                        packages/nomina_examples]
  --convert CONVERT     Convert the specified file to the desired format
  --format {LB-YAML,GC-XML,BEAN}
                        Output format for conversion [default: LB-YAML]
  -o OUTPUT, --output OUTPUT
                        Output file

command line conversion

There is a script test_cmdline that tries out some combinations of input and output file formats. see also Issue 7:command line conversion support 

scripts/test_cmdline 
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.yaml: LB-YAML✅
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.gnucash: GC-XML✅
nomina_examples/empty.yaml →  /tmp/nomina/empty_converted.beancount: BEAN✅
...

starting as local webapp

nomina -s -l 
NiceGUI ready to go on http://localhost:9849

Tutorials

Quicken

Pynomina/Quicken

GnuCash

Pynomina/GnuCash

Beancount

Pynomina/Beancount

Microsoft Money

Pynomina/MsMoney

tickets

  1. Issue 19 - add account view✓
  2. Issue 18 - add a book view✓
  3. Issue 16 - handle VAT Codes
  4. Issue 15 - enforce beancount account naming
  5. Issue 14 - add calc_balances and remove_unused_accounts function✓
  6. Issue 12 - add Microsoft Money support✓
  7. Issue 10 - Beancount conversion problem with expenses2024_bzv example✓
  8. Issue 9 - file format detection✓
  9. Issue 8 - Beancount conversion support✓
  10. Issue 7 - command line conversion support✓
  11. Issue 6 - Subsembly JSON (BankingZV) export support✓
  12. Issue 5 - Quicken QIF format support (very simple or Finanzmanager Deluxe Export format for a start ...)✓
  13. Issue 4 - GnuCash XML support✓
  14. Issue 3 - Abstract ledger✓
  15. Issue 2 - setup open checkos compatible opensource project✓

Testing

The testcases are python unittest modules see https://github.com/WolfgangFahl/pynomina/tree/main/tests. These tests are run as part of the continuous integration github actions To run the tests manually there is a "test" script in the scripts directory: 

scripts/test 
Starting test test_read_bzv, debug=True ...
# Accounts: 3
# Transactions: 2
Date Range: 2024-10-06 to 2024-10-06
# Categories: 1
# Currencies: EUR: 2
Other Details:
  name: expenses2024
  owner: John Doe
test test_read_bzv, debug=True took   0.0 s
.Starting test test_conversions, debug=True ...
Converting Ledger Book None to GC-XML

...
Ran 17 tests in 4.518s

OK

see also https://github.com/WolfgangFahl/pynomina/issues/3


Ledger Book

The pynomina Ledger Book model consists of four main classes:

  • Book
  • Account
  • Transaction
  • Split

These are the necessary classes which work together to represent a comprehensive financial ledger records.

Class Structure

0xfec2cab6.png

Book

The Book class represents the main container for all financial data. It includes:

  • Basic information: name, owner, creation date, and source URL
  • Collections of accounts and transactions
  • Methods for managing accounts and transactions, including:
    • get_stats(): Retrieves statistics about the book
    • filter(): Filters transactions by date range
    • create_account(): Creates a new account
    • add_account(): Adds an account to the book
    • lookup_account(): Finds an account by ID

Account

The Account class represents a hierarchy of individual financial accounts within the ledger. It includes:

  • account_id: Unique identifier for the account
  • name: Human-readable account name
  • account_type: Type of account (e.g., EXPENSE, INCOME)
  • description: Optional account description
  • currency: Account currency (default: EUR)
  • parent_account_id: Optional parent account for hierarchical structure

Transaction

The Transaction class represents individual financial transactions. It includes:

  • isodate: Date of the transaction in International Standards Organization date format yyyy-mm-dd
  • description: Description of the transaction
  • splits: List of Split objects representing the movement of money
  • payee: Optional payee information
  • memo: Optional additional notes
  • total_amount(): Method to calculate the total transaction amount

Split

The Split class represents the individual components of a transaction, showing how money moves between accounts. It includes:

  • amount: The amount of money involved in the split
  • account_id: The account associated with this part of the transaction
  • memo: Optional notes for this split
  • reconciled: Boolean indicating if the split has been reconciled

Example Data

Here's an example of how the ledger model is used in practice:

Yaml Format

owner: Wolfgang Fahl
url: https://github.com/WolfgangFahl/pynomina/blob/main/nomina_examples/expenses2024.yaml
since: 2024-10-06
accounts:
  Expenses:
    account_id: Expenses
    name: Expenses
    account_type: EXPENSE
    description: 'General Expenses'
    currency: EUR
  Expenses:Food:
    account_id: Expenses:Food
    name: Dining
    account_type: EXPENSE
    description: 'Expenses for Food'
    currency: EUR
    parent_account_id: Expenses
  Cash in wallet:
    account_id: Wallet
    name: Cash in Wallet
    account_type: EXPENSE
    description: ''
    currency: EUR
transactions:
  Bakery2024-10-06_0900_1:
    isodate: '2024-10-06'
    description: Bread
    splits:
      - amount: -3.50
        account_id: Cash in Wallet
      - amount: 3.50
        account_id: Expenses:Dining
        memo: Fresh sourdough bread
  Bakery2024-10-06_0900_2:
    isodate: '2024-10-06'
    description: Buns for Breakfast
    splits:
      - amount: -2.40
        account_id: Cash in Wallet
      - amount: 2.40
        account_id: Expenses:Dining
        memo: 4 whole grain buns

Implementation

The ledger model is implemented in Python, utilizing dataclasses and type hinting for clear and maintainable code. The `@lod_storable` decorator is used to enable easy serialization and deserialization of the data in YAML and/or JSON and other formats. The core idea is that the records should be readily available an tabular "list of dicts (lod)" format.

Retrieved from "https://wiki.bitplan.com/index.php?title=Pynomina&oldid=19093"