Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate PyPi publusing using CI #45

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Publish Package to PyPI

on:
push:
tags:
- 'v*.*.*' # Matches tags like v1.0.0

jobs:
publish:
runs-on: ubuntu-latest

steps:
- name: Check out code
uses: actions/checkout@v3
with:
fetch-depth: 0 # Fetch all history for all branches and tags

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build twine

- name: Build package
run: python -m build

- name: Publish package
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload dist/*

143 changes: 133 additions & 10 deletions docs/contribute.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,44 @@
# Contribute to Spinner

## Setting up the environment as a developer
Welcome to the Spinner project! This guide will help you set up your development environment, understand our linting policies, learn how to use Git tags for versioning, and understand how our build automation works.

## Setting Up the Development Environment

To set up your development environment for Spinner, follow these steps:

```sh
python3 -m ensurepip
python3 -m pip3 install virtualenv
python3 -m pip install virtualenv
python3 -m virtualenv .venv
source .venv/bin/activate
python3 -m pip install pip --upgrade
python -m pip install -e ".[dev]"
```

## Mandatory lint policy
This will:

- **Ensure `pip` is installed**.
- **Install `virtualenv`** if it's not already installed.
- **Create a virtual environment** in the `.venv` directory.
- **Activate the virtual environment**.
- **Upgrade `pip`** to the latest version.
- **Install the package** in editable mode along with development dependencies specified in `pyproject.toml`.

## Mandatory Lint Policy

We enforce a mandatory linting policy to maintain code quality and consistency. We recommend adding the following pre-commit hook to automatically format your code before each commit.

### Setting Up the Pre-Commit Hook

Create a file named `pre-commit` in the `.git/hooks/` directory and make it executable:

```sh
touch .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
```

Add the following content to the `pre-commit` file:

We recomend the following commit hook (`.git/hooks/pre-commit`)
```sh
#!/bin/bash

Expand All @@ -36,25 +61,29 @@ format_c_cpp_files() {
fi
}

# Function to format all Python files using black
# Function to format all Python files using black and isort
format_python_files() {
if command -v black >/dev/null 2>&1; then
if command -v black >/dev/null 2>&1 && command -v isort >/dev/null 2>&1; then
for file in $(git diff --cached --name-only --diff-filter=ACM "$against" | grep -E '\.py$'); do
black "$file"
isort "$file"
git add "$file"
done
else
echo "black not found, skipping Python file formatting."
echo "black or isort not found, skipping Python file formatting."
fi
if command -v black >/dev/null 2>&1; then
}

# Function to format Jupyter notebooks
format_notebooks() {
if command -v black >/dev/null 2>&1 && command -v isort >/dev/null 2>&1; then
for file in $(git diff --cached --name-only --diff-filter=ACM "$against" | grep -E '\.ipynb$'); do
black "$file"
isort "$file"
git add "$file"
done
else
echo "black for notebook not found, skipping Python file formatting."
echo "black or isort not found, skipping notebook formatting."
fi
}

Expand All @@ -73,8 +102,102 @@ clear_notebook_output() {
# Run the formatting functions
format_c_cpp_files
format_python_files
format_notebooks
clear_notebook_output

# Check for changes
exec git diff-index --check --cached "$against" --
```
```

This script will:

- **Format C/C++ files** using `clang-format`.
- **Format Python files and notebooks** using `black` and `isort`.
- **Clear outputs** from Jupyter notebooks to avoid committing unnecessary data.
- **Automatically add the formatted files** back to the commit.

## Versioning with Git Tags

We use Git tags to manage our release versions, which are crucial for packaging and distributing the Spinner project via PyPI.

### Using Git Tags

To create a new Git tag for a release, follow these steps:

1. **Ensure all changes are committed**:

```sh
git add .
git commit -m "Prepare for release v0.0.2"
```

2. **Create a new tag**:

```sh
git tag v0.0.2
```

Replace `v0.0.2` with the appropriate version number.

3. **Push the tag to GitHub**:

```sh
git push origin v0.0.2
```

### Best Practices for Tagging

- **Semantic Versioning**: We follow [Semantic Versioning](https://semver.org/) with the format `vMAJOR.MINOR.PATCH` (e.g., `v1.2.3`).
- **MAJOR** version when you make incompatible API changes.
- **MINOR** version when you add functionality in a backwards-compatible manner.
- **PATCH** version when you make backwards-compatible bug fixes.

- **Annotated Tags**: Use annotated tags to include additional metadata such as the tagger name, email, date, and a message.

```sh
git tag -a v0.0.2 -m "Release version 0.0.2"
```

- **Consistency**: Always start tags with a `v` to denote version (e.g., `v0.0.2`).

- **Changelog Updates**: Before tagging, update the `CHANGELOG.md` to reflect the changes in the new version.

- **Testing Before Release**: Ensure all tests pass before creating a tag.

## Build Automation with GitHub Actions

Our project uses GitHub Actions to automate the build and deployment process. When a new tag is pushed to the repository, the build bot (GitHub Actions workflow) is triggered to:

- **Build the Package**: Create source and wheel distributions of the package.
- **Publish to PyPI**: Upload the distributions to PyPI using Twine.

### How the Build Bot Works

1. **Trigger**: The workflow is triggered on pushing tags that match the pattern `v*.*.*` (e.g., `v0.0.2`).

2. **Workflow File**: The workflow is defined in `.github/workflows/publish.yml`.

3. **Steps**:
- **Checkout Code**: Fetches the repository code and tags.
- **Set Up Python**: Configures the Python environment.
- **Install Dependencies**: Installs build tools like `build` and `twine`.
- **Build Package**: Uses `python -m build` to create distributions.
- **Publish Package**: Uploads the package to PyPI using Twine and the stored PyPI API token.

### Configuring PyPI Credentials

- **PyPI API Token**: The API token for PyPI is stored securely in GitHub Secrets under the name `PYPI_API_TOKEN`.
- **Security**: Never commit API tokens or passwords to the repository.

### Important Notes

- **Ensure Tags are Pushed**: The build bot relies on the Git tags to determine the version. Always push your tags to GitHub.
- **Check Workflow Status**: After pushing a tag, monitor the Actions tab in GitHub to ensure the workflow runs successfully.
- **Test Locally**: Before pushing a tag, you can test the build process locally:

```sh
python -m build
```
---

Happy coding!
3 changes: 2 additions & 1 deletion docs/examples/extra_args.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ metadata:
timeout: 10
retry: True
retry_limit: 1
os_envs: [PATH, USER]

sleep_test:
command:
Expand All @@ -14,4 +15,4 @@ metadata:
sleep_test:
sleep_ammount:
- 1
- 2
- 2
12 changes: 11 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ authors = [
{name = "Rodrigo Ceccato de Freitas", email = "[email protected]"},
{name = "Jhonatan Cléto", email="[email protected]"}
]
version = "0.0.1"

dynamic = ["version"]

requires-python = ">=3.9.12, <4"
dependencies = [
"click==8.1.7",
Expand Down Expand Up @@ -49,3 +51,11 @@ line_length = 79
[tool.taskipy.tasks]
lint = "black --check --diff . && isort --check --diff ."
format = "black . && isort ."

[build-system]
requires = ["setuptools>=42", "wheel", "setuptools_scm"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
version_scheme = "release-branch-semver"
local_scheme = "no-local-version"
20 changes: 18 additions & 2 deletions spinner/runner/utilities.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,15 @@ def run_benchmarks(config, output, extra_args):
bench_metadata["start_timestamp"] = str(pd.Timestamp.now())
bench_metadata.update(extra_args)
bench_metadata["runner_hostname"] = str(os.uname()[1])
bench_metadata["start_env"] = str(os.environ.copy())

# Capture specified environment variables
if "os_envs" in bench_metadata:
env_vars_to_capture = bench_metadata["os_envs"]
bench_metadata["start_env"] = {
var: os.environ.get(var) for var in env_vars_to_capture
}
else:
bench_metadata["start_env"] = {}

# Iterate over settings in bench_config, excluding metadata
bench_names = [
Expand Down Expand Up @@ -92,7 +100,15 @@ def _update_progress(
progress_callback()

bench_metadata["end_timestamp"] = str(pd.Timestamp.now())
bench_metadata["end_env"] = str(os.environ.copy())

# Capture specified environment variables at the end
if "os_envs" in bench_metadata:
env_vars_to_capture = bench_metadata["os_envs"]
bench_metadata["end_env"] = {
var: os.environ.get(var) for var in env_vars_to_capture
}
else:
bench_metadata["end_env"] = {}

rprint(execution_df)
with open(output, "wb") as f:
Expand Down
Loading