Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I167 upgrade python #168

Merged
merged 32 commits into from
May 29, 2024
Merged

I167 upgrade python #168

merged 32 commits into from
May 29, 2024

Conversation

eyvorchuk
Copy link
Contributor

@eyvorchuk eyvorchuk commented May 28, 2024

This PR resolves #167. It makes the following changes to the package:

  • Changes Python installation process from pip to poetry and modifies Dockerfile, Makefile, CI actions, and user/dev docs accordingly
  • Upgrades supported Python versions in CI and changes version in Dockerfile from3.8 to 3.10
  • Re-runs notebooks and updates formatted demos to ensure the new setup works
  • Sets an environment variable for the THREDDS host used in notebooks rather than just docker-dev03
  • Formats .py files with the newer version of black

Using the make command, I have verified that the installation process, automated tests in tests/, and notebooks all work correctly.

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@eyvorchuk
Copy link
Contributor Author

Ended up having to make an adjustment to wps_update_metadata.py to accommodate the new version of pywps, but I reran the tests/notebooks, and everything's good.

@eyvorchuk eyvorchuk requested a review from rod-glover May 29, 2024 18:21
Copy link

@rod-glover rod-glover left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great! Just a few comments, mainly about avoiding hardcoding the host name.

@bash -c '${PYTHON} -m pytest -v tests/'
@bash -c 'poetry run pytest -v tests/'

.PHONY: test-not-online

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This target should also be documented in the 'help' target

Makefile Outdated
@-python3 setup.py sdist
@-python3 setup.py bdist_wheel
@bash -c 'poetry build -f sdist'
@bash -c 'poetry build -f wheel'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

poetry build defaults to both sdist and wheel. No need to specify them.

@@ -34,7 +34,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Using thunderbird on https://docker-dev03.pcic.uvic.ca/twitcher/ows/proxy/thunderbird/wps\n"
"Using thunderbird on https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thunderbird/wps\n"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host is subject to change (you just did it). Is there some way for this to be set programmatically, or from an env var (THREDDS_HOST)?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also possible this file is the output of some automated process, if so, ignore my comment.

url = "https://pypi.pacificclimate.org/simple"
priority = "supplemental"

[tool.poetry.dependencies]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see some exact version specs here (e.g., cftime = "1.6.3"), which are not desirable unless they are necessary to prevent installation errors. Please check whether you can loosen them.

It looks as if these are all or mostly top-level deps, yay. However, if there are any subsidiary deps, please see if you can remove them. Minimal dependencies are a good goal.

cdo = "1.6.0"
poetry-dotenv-plugin = "^0.2.0"

[tool.poetry.group.dev.dependencies]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same remarks as above.

requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.pytest.ini_options]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@@ -52,4 +53,4 @@ def get_url():
if url:
return url

return "https://docker-dev03.pcic.uvic.ca/twitcher/ows/proxy/thunderbird/wps"
return "https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thunderbird/wps"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it would be best if this were obtained from an env var or other external source.

Makefile Outdated
DEV_PORT ?= $(shell bash -c 'read -ep "Target port: " port; echo $$port')

# Used in target refresh-notebooks to make it looks like the notebooks have
# been refreshed from the production server below instead of from the local dev
# instance so the notebooks can also be used as tutorial notebooks.
OUTPUT_URL = https://docker-dev03.pcic.uvic.ca/wpsoutputs
OUTPUT_URL = https://marble-dev01.pcic.uvic.ca/wpsoutputs

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get from env var THREDDS_HOST?

.cruft.json Outdated

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest finding out what this suspiciously named file is for, and remove it if it truly is cruft.

.env Outdated
@@ -0,0 +1 @@
THREDDS_HOST="marble-dev01.pcic.uvic.ca"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get host name from this env var wherever you can in other code, to avoid hardcoding the value. I've added a few comments to that effect below, but there may be other places where it could be done.

@rod-glover
Copy link

And one final remark, as we discussed a couple of minutes ago: If Black is going to reformat a lot of the project, not code you are actually modifying the logic/function of, do a separate Black PR (before your coding PR). The Black PR won't need reviewing.

@eyvorchuk eyvorchuk merged commit 3658bce into master May 29, 2024
4 checks passed
@eyvorchuk eyvorchuk deleted the i167-upgrade-python branch May 29, 2024 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade Python versions and use poetry instead of pip
2 participants