DgraphAndWeaviateTest: Difference between revisions

From BITPlan Wiki
Jump to navigation Jump to search
 
(28 intermediate revisions by the same user not shown)
Line 11: Line 11:
}}
}}


This is sample project to test Python based access to {{Link|target=Dgraph}} and {{Link|target=Weaviate}}
This is sample project to test Python based storage with
= Installation =
* {{Link|target=Dgraph}}
* {{Link|target=Weaviate}}
* {{Link|target=Apache Jena}}
* [https://www.sqlite.org/index.html sqlite]
 
The motivation for this project was the [[ProceedingsTitleParser#Choice_of_Database.2FStorage_system|Choice of a Database storage system]] for the {{Link|target=ProceedingsTitleParser}}
 
= Installation and test =
== Prerequisites ==
* [https://www.python.org/ python] > version 3.6 - tested with version 3.6/3.7/3.8
* [https://www.docker.com/ docker] e.g. docker desktop community - tested with e.g. docker desktop 2.3.0.4 Docker version 19.03.12
* [https://openjdk.java.net/ java] e.g. openjdk - tested with Java 1.8 and Java 11
* Operating system that can run bash scripts e.g. macports, linux - tested on Mac OS 10.13.6 Macports 2.6.2, Ubuntu 18.04 LTS
 
== Installation ==
<source lang='bash'>
<source lang='bash'>
https://github.com/WolfgangFahl/DgraphAndWeaviateTest
https://github.com/WolfgangFahl/DgraphAndWeaviateTest
Line 18: Line 32:
scripts/install
scripts/install
</source>
</source>
== Starting servers ==
see also DgraphAndWeaviateTest
<source lang='bash'>
# command to run Dgraph
# pull dgraph
scripts/dgraph -p
# run dgraph
scripts/dgraph
# pull and run weaviate
scripts/weaviate
# install apache jena and load example data
scripts/jena -l sampledata/example.ttl
# run apache jena fuseki server
scripts/jena -f example
</source>
== Test ==
<source lang='bash'>
scripts/test
</source>
= Sample Data =
== UML ==
<uml>
package Royals {
  entity Person {
    name : TEXT <<PK>>
    born : DATE
    numberInLine : INTEGER
    wikidataurl : TEXT
    age : FLOAT
    ofAge : BOOLEAN
    lastmodified : TIMESTAMP
  }
}
</uml>
== Python ==
<source lang='python'>
@staticmethod
    def getRoyals():
        listOfDicts=[
            {'name': 'Elizabeth Alexandra Mary Windsor', 'born': Sample.dob('1926-04-21'), 'numberInLine': 0, 'wikidataurl': 'https://www.wikidata.org/wiki/Q9682' },
            {'name': 'Charles, Prince of Wales',        'born': Sample.dob('1948-11-14'), 'numberInLine': 1, 'wikidataurl': 'https://www.wikidata.org/wiki/Q43274' },
            {'name': 'George of Cambridge',              'born': Sample.dob('2013-07-22'), 'numberInLine': 3, 'wikidataurl': 'https://www.wikidata.org/wiki/Q1359041'},
            {'name': 'Harry Duke of Sussex',            'born': Sample.dob('1984-09-15'), 'numberInLine': 6, 'wikidataurl': 'https://www.wikidata.org/wiki/Q152316'}
        ]
        today=date.today()
        for person in listOfDicts:
            born=person['born']
            age=(today - born).days / 365.2425
            person['age']=age
            person['ofAge']=age>=18
            person['lastmodified']=datetime.now()
        return listOfDicts
</source>
= Weaviate =
== Weaviate start ==
<source lang='bash'>
scripts/weaviate
</source>
= Dgraph =
== Dgraph Installation ==
== Dgraph Installation ==
docker based pull:
docker based pull:
Line 35: Line 110:
scripts/dgraph -k
scripts/dgraph -k
</source>
</source>
== Example unit test ==
== scripts/dgraph usage ==
<source lang='python'>
<source lang='bash'>
def testDgraph(self):
scripts/dgraph -h
        '''
scripts/dgraph [-b|--bash|-c|--clean|-h|--help|-k|--kill|-p|--pull]
        test basic Dgraph operation
 
        '''
-b | --bash: start a bash terminal shell within the currently running container
        dgraph=Dgraph(debug=True)
-h | --help: show this usage
        # drop all data and schemas
-k | --kill: stop the docker image
        dgraph.drop_all()
-p | --pull: pull the docker image
        # create a schema for Pokemons
-c | --clean: clean start with kill and purge of all data
        schema='''
        name: string @index(exact) .
        weight: float .
        height: float .
type Pokemon {
  name
  weight
  height
}'''
        dgraph.addSchema(schema)
        # prepare a list of Pokemons to be added
        pokemonList=[{'name':'Pikachu', 'weight':  6, 'height': 0.4 },
                  {'name':'Arbok',  'weight': 65, 'height': 3.5 },
                  {'name':'Raichu',  'weight': 30, 'height': 0.8 },
                  {'name':'Sandan',  'weight': 12, 'height': 0.6 }]
        # add the list in a single transaction
        dgraph.addData(obj=pokemonList)
        # retrieve the data via GraphQL+ query
        graphQuery='''{
# list of pokemons
  pokemons(func: has(name), orderasc: name) {
    name
    weight
    height
  }
}'''
        queryResult=dgraph.query(graphQuery)
        # check the result
        self.assertTrue('pokemons' in queryResult)
        pokemons=queryResult['pokemons']
        self.assertEqual(len(pokemonList),len(pokemons))
        sortindex=[1,0,2,3]
        for index,pokemon in enumerate(pokemons):
            expected=pokemonList[sortindex[index]]
            self.assertEquals(expected,pokemon)
        # close the database connection
        dgraph.close()
</source>
</source>
= Issues, Questions and Answers =
== Issues ==
* https://github.com/semi-technologies/weaviate/issues/1215
* https://discuss.dgraph.io/t/dgraph-v20-07-0-v20-03-0-unreliability-in-mac-os-environment/9376/14
* https://discuss.dgraph.io/t/input-for-predicate-location-of-type-scalar-is-uid/9381
== Stackoverflow questions ==
* https://stackoverflow.com/questions/63486767/how-can-i-get-the-fuseki-api-via-sparqlwrapper-to-properly-report-a-detailed-err
* https://stackoverflow.com/questions/63435157/listofdict-to-rdf-conversion-in-python-targeting-apache-jena-fuseki
* https://stackoverflow.com/questions/63358495/how-to-delete-all-nodes-with-a-given-type
* https://stackoverflow.com/questions/63260073/starting-zero-alpha-and-ratel-in-a-single-command-e-g-in-macosx-and-other-envir
* https://stackoverflow.com/questions/63098344/weaviate-error-code-400-parsing-body-from-failed-invalid-character-g-looki
* https://stackoverflow.com/questions/63075787/translating-sidif-to-weaviate
== Stackoverflow answers ==
* https://stackoverflow.com/questions/63435157/listofdict-to-rdf-conversion-in-python-targeting-apache-jena-fuseki/63440396#63440396#
* https://stackoverflow.com/questions/63358495/how-to-delete-all-nodes-with-a-given-type/63358827#63358827
* https://stackoverflow.com/questions/63260073/starting-zero-alpha-and-ratel-in-a-single-command-e-g-in-macosx-and-other-envir/63265154#63265154
= Example unit tests =


== Example test session ==
== Example test session ==
see https://travis-ci.org/github/WolfgangFahl/DgraphAndWeaviateTest/jobs/715131236
see https://travis-ci.org/github/WolfgangFahl/DgraphAndWeaviateTest/jobs/715131236

Latest revision as of 07:05, 22 September 2020

OsProject

OsProject
edit
id  DgraphAndWeaviateTest
state  
owner  Wolfgang Fahl
title  DgraphAndWeaviateTest
url  https://github.com/WolfgangFahl/DgraphAndWeaviateTest
version  0.0.1
description  
date  2020/08/05
since  
until  

This is sample project to test Python based storage with

The motivation for this project was the Choice of a Database storage system for the ProceedingsTitleParser

Installation and test

Prerequisites

  • python > version 3.6 - tested with version 3.6/3.7/3.8
  • docker e.g. docker desktop community - tested with e.g. docker desktop 2.3.0.4 Docker version 19.03.12
  • java e.g. openjdk - tested with Java 1.8 and Java 11
  • Operating system that can run bash scripts e.g. macports, linux - tested on Mac OS 10.13.6 Macports 2.6.2, Ubuntu 18.04 LTS

Installation

https://github.com/WolfgangFahl/DgraphAndWeaviateTest
cd DgraphAndWeaviateTest
scripts/install

Starting servers

see also DgraphAndWeaviateTest 

# command to run Dgraph
# pull dgraph
scripts/dgraph -p
# run dgraph
scripts/dgraph
# pull and run weaviate
scripts/weaviate
# install apache jena and load example data
scripts/jena -l sampledata/example.ttl
# run apache jena fuseki server
scripts/jena -f example

Test

scripts/test

Sample Data

UML

Python

@staticmethod
    def getRoyals():
        listOfDicts=[
            {'name': 'Elizabeth Alexandra Mary Windsor', 'born': Sample.dob('1926-04-21'), 'numberInLine': 0, 'wikidataurl': 'https://www.wikidata.org/wiki/Q9682' },
            {'name': 'Charles, Prince of Wales',         'born': Sample.dob('1948-11-14'), 'numberInLine': 1, 'wikidataurl': 'https://www.wikidata.org/wiki/Q43274' },
            {'name': 'George of Cambridge',              'born': Sample.dob('2013-07-22'), 'numberInLine': 3, 'wikidataurl': 'https://www.wikidata.org/wiki/Q1359041'},
            {'name': 'Harry Duke of Sussex',             'born': Sample.dob('1984-09-15'), 'numberInLine': 6, 'wikidataurl': 'https://www.wikidata.org/wiki/Q152316'}
        ]
        today=date.today()
        for person in listOfDicts:
            born=person['born']
            age=(today - born).days / 365.2425
            person['age']=age
            person['ofAge']=age>=18
            person['lastmodified']=datetime.now()
        return listOfDicts

Weaviate

Weaviate start

scripts/weaviate

Dgraph

Dgraph Installation

docker based pull: 

scripts/dgraph -p

Dgraph start

docker based start of

  • alpha
  • ratel
  • zero
scripts/dgraph

Dgraph stop

scripts/dgraph -k

scripts/dgraph usage

scripts/dgraph -h
scripts/dgraph [-b|--bash|-c|--clean|-h|--help|-k|--kill|-p|--pull]

-b | --bash: start a bash terminal shell within the currently running container
-h | --help: show this usage
-k | --kill: stop the docker image
-p | --pull: pull the docker image
-c | --clean: clean start with kill and purge of all data

Issues, Questions and Answers

Issues

Stackoverflow questions

Stackoverflow answers

Example unit tests

Example test session

see https://travis-ci.org/github/WolfgangFahl/DgraphAndWeaviateTest/jobs/715131236

Retrieved from "https://wiki.bitplan.com/index.php?title=DgraphAndWeaviateTest&oldid=12980"