[TST] Add integration test against local test graph (#357)

* Added `recipes` and `neurobagel_examples` as submodules

* Set up test graph

Co-authored-by: Sebastian Urchs <[email protected]>
Co-authored-by: Alyssa Dai <[email protected]>

* [ENH] Added integration test

works only with the test graph enabled

* [ENH] Default disable integration tests

With a special flag. Otherwise the tests fail
if we do not also launch the test graph

* Add note in GH workflow to enable integration test

* [FIX] Remove unused ENV variable

* Mock fewer internal VARs to get the SPARQL query to run

* register & set default behaviour for integration marker in pytest.ini

* update instructions to run integration tests

* update dependabot config

---------

Co-authored-by: rmanaem <[email protected]>
Co-authored-by: Alyssa Dai <[email protected]>

.github/dependabot.yml

@@ -10,3 +10,12 @@ updates:
       - "_bot"
       - "maint:dependency"
       - "type:maintenance"
+
+  - package-ecosystem: 'gitsubmodule'
+    directory: '/'
+    schedule:
+      interval: 'weekly'
+    labels:
+      - "_bot"
+      - "maint:dependency"
+      - "type:maintenance"

.github/workflows/test.yaml

@@ -15,6 +15,8 @@ jobs:
     steps:
 
     - uses: actions/checkout@v4
+      # TODO: Check out the submodules as well and then launch a test graph with docker compose to run the
+      # integration tests
 
     - name: Set up Python 3.10
       uses: actions/setup-python@v5

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "recipes"]
+	path = recipes
+	url = https://github.com/neurobagel/recipes
+[submodule "neurobagel_examples"]
+	path = neurobagel_examples
+	url = https://github.com/neurobagel/neurobagel_examples

.test_env

@@ -0,0 +1,18 @@
+# ---- PROJECT NAME ----
+COMPOSE_PROJECT_NAME=neurobagel_test_node
+
+# ---- CONFIGURATION FOR graph ----
+# Replace DBUSER with the username you want to set for your graph database user
+# NB_GRAPH_USERNAME=DBUSER
+# Replace my_db with the name you want to give your graph database
+# NB_GRAPH_DB=repositories/my_db
+# Replace ./data with the path to your JSONLD files
+LOCAL_GRAPH_DATA=./recipes/data
+
+# Additional configurable parameters - uncomment to change the defaults
+# Change NB_GRAPH_PORT_HOST if port 7200 is already in use on the machine
+# NB_GRAPH_PORT_HOST=7200
+# Replace ./secrets with the directory path containing the text files with your desired 
+# secure passwords for GraphDB (NB_GRAPH_ADMIN_PASSWORD.txt and NB_GRAPH_PASSWORD.txt)
+# NB_GRAPH_SECRETS_PATH=./secrets
+# ---------------------------------

README.md

@@ -143,14 +143,41 @@ If you get a 401 response to your API request with an `"Unauthorized: "` error m
 
 Neurobagel API utilizes [Pytest](https://docs.pytest.org/en/7.2.x/) framework for testing.
 
-To run the tests first make sure you're in repository's main directory and in your environment where the dependencies are installed and environment variables are set.
+To run the tests, first ensure you're in the repository's root directory and in the environment where the dependencies are installed.
+
+Install the submodules used by the tests:
+```bash
+git submodule init
+git submodule update
+```
 
 You can then run the tests by executing the following command in your terminal:
 
 ```bash
 pytest tests
 ```
 
+To run the integration tests of SPARQL queries (skipped by default), also launch the test graph store:
+
+```bash
+docker compose up -d test_graph
+```
+
+_NOTE: Don't have a `.env` file in the root directory as it will conflict with the environment variable configuration of the docker compose file, 
+since docker compose will try to use `.env` by default._
+
+Then, run all tests using:
+```bash
+pytest -m "integration or not integration"
+# OR
+pytest -m ""
+```
+
+Or, to run only the integration tests:
+```bash
+pytest -m "integration"
+```
+
 ## The default Neurobagel SPARQL query
 
 [`docs/default_neurobagel_query.rq`](docs/default_neurobagel_query.rq) contains a sample default SPARQL query sent by the Neurobagel API to a graph database to retrieve all available phenotypic and imaging data.

docker-compose.yml

@@ -0,0 +1,26 @@
+services:
+  test_graph:
+    extends:
+      file: recipes/docker-compose.yml
+      service: graph
+    volumes:
+      - "graphdb_home:/opt/graphdb/home"
+      - "./recipes/scripts:/usr/src/neurobagel/scripts"
+      - ".test_env:/usr/src/neurobagel/.env"
+      - "./recipes/vocab:/usr/src/neurobagel/vocab"
+      - "./neurobagel_examples/data-upload/pheno-bids-derivatives:/data"
+    environment:
+      NB_GRAPH_USERNAME: "DBUSER"
+      NB_GRAPH_DB: "repositories/my_db"
+    secrets:
+      - db_admin_password
+      - db_user_password
+
+secrets:
+  db_admin_password:
+    file: ./recipes/secrets/NB_GRAPH_ADMIN_PASSWORD.txt
+  db_user_password:
+    file: ./recipes/secrets/NB_GRAPH_PASSWORD.txt
+
+volumes:
+  graphdb_home:

pytest.ini

@@ -0,0 +1,5 @@
+[pytest]
+markers =
+    integration: mark integration tests that need the test graph to run
+; Default to not running tests with the integration marker
+addopts = -m "not integration"

tests/conftest.py

@@ -29,8 +29,8 @@ def disable_auth(monkeypatch):
 @pytest.fixture(scope="function")
 def set_test_credentials(monkeypatch):
     """Set random username and password to avoid error from startup check for set credentials."""
-    monkeypatch.setenv(util.GRAPH_USERNAME.name, "SomeUser")
-    monkeypatch.setenv(util.GRAPH_PASSWORD.name, "SomePassword")
+    monkeypatch.setenv(util.GRAPH_USERNAME.name, "DBUSER")
+    monkeypatch.setenv(util.GRAPH_PASSWORD.name, "DBPASSWORD")
 
 
 @pytest.fixture()

tests/test_query.py

@@ -619,10 +619,29 @@ def test_query_without_token_succeeds_when_auth_disabled(
     mock_successful_get,
     monkeypatch,
     disable_auth,
+    set_test_credentials,
 ):
     """
     Test that when authentication is disabled, a request to the /query route without a token succeeds.
     """
     monkeypatch.setattr(crud, "get", mock_successful_get)
     response = test_app.get(ROUTE)
     assert response.status_code == 200
+
+
+@pytest.mark.integration
+def test_integration_query_without_auth_succeeds(
+    test_app, monkeypatch, disable_auth, set_test_credentials
+):
+    """
+    Running a test against a real local test graph
+    should succeed when authentication is disabled.
+    """
+    # Patching the QUERY_URL directly means we don't need to worry about the constituent
+    # graph environment variables
+    monkeypatch.setattr(
+        util, "QUERY_URL", "http://localhost:7200/repositories/my_db"
+    )
+
+    response = test_app.get(ROUTE)
+    assert response.status_code == 200