Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge develop onto main for 0.3.0 release #75

Open
wants to merge 188 commits into
base: main
Choose a base branch
from
Open

Merge develop onto main for 0.3.0 release #75

wants to merge 188 commits into from

Conversation

amoeba
Copy link
Contributor

@amoeba amoeba commented Aug 10, 2022 •
edited
Loading

This probably doesn't need a code review but I'd recommend testing it out by installing to the dev cluster and reviewing the results from there.

Changes:

  • Upgraded from Python 2 to 3
  • Complete rewrite using a much cleaner codebase and new patterns
  • Restructured code to follow recommended Python packaging guidelines
  • Added a first pass of a CLI for interacting with the service (not done yet)
  • Refactored tests into unit and integration sub-suites. pytest defaults to running just unit and pytest --integration runs all tests.
  • Primary deployment strategy switched from Docker Compose to Kuerbernetes (via Helm). Docker Compose still supported.
  • Rewrote triplification pattern to match Schema.org/Science-on-Schema pattern
  • Added Sphinx documentation
  • GitHub Actions Workflows for
    • Unit tests (pytest)
    • Sphinx documentation build and push to GitHub Actions
  • Container images use ghcr.io instead of Docker Hub

Remaining TODOs before merge:

  • Remove old deploy YAML files
  • Confirm the persistent volume configuration

TODO after merge:

  • Turn on GitHub Pages on this repo to enable the sphinx docs build
  • Tag and create a 0.3.0 Release
  • Decomission slinkythegraph DockerHub account (@ThomasThelen )

mbjones and others added 30 commits March 19, 2021 15:01
@mbjones
Start a modified document describing the Dataset graph pattern.
10fa806
@ThomasThelen
Port the essential services from Docker to Kubernetes
456bb45
@ThomasThelen
Add the Dockerfiles for building the K8 requirements and add a Readme…
683d625 
… on using them.
@amoeba @ThomasThelen
Run 2to3 -w on all .py files
7e7e402
@amoeba @ThomasThelen
Add a basic Dockerfile to run tests
fc45196
@amoeba @ThomasThelen
Upgrade for Python3
d85e04a 
Test suite now passes on Python 3.9.1
@amoeba @ThomasThelen
Add py.bak files to gitignore
70e63dc
@ThomasThelen
Fix dockerfile merge conflicts
ff27fdf
@amoeba @ThomasThelen
Add Pipfile for worker service
b414825
@ThomasThelen
Add the docker folder to the helm ignore
75fd8e5
@amoeba
Merge pull request #22 from DataONEorg/feature_kubernetes_deployment
9ef251e 
Port the essential services from Docker to Kubernetes
@amoeba
Add docs for how to install d1lod on macOS+pyenv
2fd8b20
@ThomasThelen
Port K8 to Python 3
e385b02
@ThomasThelen
Merge branch 'develop' into feature_1_python3
7142084
@ThomasThelen
Add instructions for adding authentication to the SPARQL endpoint
ff7a02b
@amoeba
Merge pull request #25 from DataONEorg/feature_1_python3
0077585 
Python 3 Upgrade
@amoeba
Merge pull request #26 from DataONEorg/feature_auth_sparql
d598ad8 
Add instructions for an authenticated SPARQL endpoint
@amoeba
WIP: Partially update mappings document
b7ef1bc 
(Just needed to run out for the day)
@amoeba
Finish first draft of mappings documents
8e6cda2
@amoeba
Remove 2to3.txt from Python 2 to 3 migration
e018a08
@amoeba
Remove unneeded void.ttl file from d1lod folder
d9e67c1
@amoeba
Update d1lod package readme and setup.py
05741d9
@amoeba
Tweak mappings doc and add full example
8ca22e0
@amoeba
Fix URL in mappings doc
da57b25
@amoeba
Create initial prototype of web app
4cfebac
@amoeba
WIP Begin overhaul of classes for Slinky
109bc54
@amoeba
Merge branch 'feature_web_app' into feature_update_graph_pattern
8378a34
@amoeba
Completely refactor d1lod package
478b62f 
The old Interface and Graph classes had a lot of cruft and were really confusing to keep straight. I've done a refactor that uses a different class structure that's a lot easier for me to understand. Hopefully it's easier for you too.

See the README for more info, including a fancy picture, but the Interface class is now wrapped into a top-level SlinkyClient class and the old Graph class is now tied into a SparqlTripleStore class. With the old setup, you had to instantiate an Interface and a Graph. Now you just instantiate a SlinkyClient and you're good to go.

Here's some more detail, copied from the README:

- `SlinkyClient`: Entrypoint class that manages a connection to DataONE, a triple store, and Redis for short-term persistence and delayed jobs
- `FilteredCoordinatingNodeClient`: A view into a Coordinating Node that can limit what content appears to be available based on a Solr query. e.g., a CN client that can only see datasets that are part of a specific EML project or in a particular region
- `SparqlTripleStore`: Handles inserting into and querying a generic SPARQL-compliant RDF triplestore via SPARQL queries. Designed to be used with multiple triple stores.
- `Processor`: Set of classes that convert documents of various formats (e.g., XML, JSON-LD) into a set of RDF statements

The old package code is left in the legacy submodule (to be deleted in the future) and its tests are still alive and working via pytest.
@amoeba
Remove top-level makefile
46009a7
@amoeba
Convert web front-end to use a SlinkyClient
062c562 
This is pretty basic but you can run this container and hit /get?id=foo to get the Slinky RDF for given DataONE PID.
amoeba added 22 commits August 2, 2022 18:57
@amoeba
Fix issue in Helm chart where sparql queries don't work
8844e3d
@amoeba
Make Slinky web proxy actually proxy queries
c244c3f
@amoeba
Update root docker-compose file to match repo
50fcd59
@amoeba
Update redlands install docs
1a06a10
@amoeba
Add full support for SOSO award structure
7aa9d94 
Ref #64
@amoeba
Fix predicate in EML220 processor for funding
d8fbec1
@amoeba
Update readme an arch diagram
102941f
@amoeba
Tweak identifier logic in processor.py
77309b3
@amoeba
Add units support to variableMeasured
8f62b43
@amoeba
Fix bug in handling of userId
931d266
@amoeba
Refactor slinky to use rdflib.Namespace
c5e2979
@amoeba
Update all SDO references to http://
17718d0 
We decided we needed to change this all back because schema.org gave up on switching to https
@amoeba
Add missing str.strip() call in eml220_processor
6aefcec
@amoeba
Add support for seriesIds
f2faac5
@amoeba
Add pytest GHA workflow
4600081
@amoeba
Make CLI's get command output more compact
da665c5
@amoeba
Make GHA Workflow use :latest tag
ac31bd5 
Sets us up for later when we automatically build and push a latest tag from the tip of develop
@amoeba
Add sphinx documentation to the python package
9ced01e
@amoeba
Rename python package to slinky
a47c4b4
@amoeba
Add GHA workflow for sphinx build
32decbf
@amoeba
Switch sphinx GHA workflow to only trigger on main
dc93656
@amoeba
v0.3.0
f878c5a
@amoeba amoeba added this to the 0.3.0 milestone Aug 10, 2022
@amoeba amoeba requested review from mbjones and ThomasThelen August 10, 2022 03:10
@amoeba amoeba self-assigned this Aug 10, 2022
@amoeba amoeba changed the title [WIP] Merge develop onto main for 0.3.0 release Merge develop onto main for 0.3.0 release Aug 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mbjones mbjones Awaiting requested review from mbjones

@ThomasThelen ThomasThelen Awaiting requested review from ThomasThelen

At least 1 approving review is required to merge this pull request.

Assignees

@amoeba amoeba

Labels
None yet
Projects
None yet
Milestone
0.3.0
Development

Successfully merging this pull request may close these issues.
3 participants
@amoeba @mbjones @ThomasThelen