GH-20: Add release scripts #496

kou · 2025-01-09T08:34:34Z

Fixes GH-20.

This is based on apache/arrow-go#122 .

Workflow:

Cut a RC:

Bump version by dev/release/bump_version.sh 19.0.0
Run dev/release/release_rc.sh
1. dev/release/release_rc.sh pushes v${version}-rc${rc} tag
2. .github/workflows/rc.yml creates apache-arrow-java-${version}.tar.gz{,.sha256,.sha512}
3. .github/workflows/rc.yml uploads apache-arrow-java-${version}.tar.gz{,.sha256,.sha512} to GitHub Releases
4. dev/release/release_rc.sh downloads apache-arrow-java-${version}.tar.gz from GitHub Releases
5. dev/release/release_rc.sh signs apache-arrow-java-${version}.tar.gz as apache-arrow-go-${version}.tar.gz.asc
6. dev/release/release_rc.sh uploads apache-arrow-java-${version}.tar.gz.asc to GitHub Releases
Start a vote

(GitHub Actions instead of
https://dist.apache.org/repos/dist/dev/arrow/ is used like ADBC.)

Verify a RC:

Run dev/release/verify_rc.sh

Release an approved RC:

Run dev/release/release.sh
1. dev/release/release.sh pushes v${version} tag that refers that same commit ID as v${version}-rc${rc}
2. .github/workflows/release.yml create a GitHub Releases for v${version} tag by copying the RC GitHub Releases
3. dev/release/release.sh downloads apache-arrow-java-${version}.tar.gz{,.asc,.sha256,.sha512} from GitHub Releases
4. dev/release/release.sh uploads apache-arrow-java-${version}.tar.gz{,.asc,.sha256,.sha512} to https://dist.apache.org/repos/dist/release/arrow
5. dev/release/release.sh removes old releases from https://dist.apache.org/repos/dist/release/arrow
Add this release to ASF's report database: https://reporter.apache.org/addrelease.html?arrow

Follow-up tasks:

Add support for building binary packages (.jar)
Add support for releasing binary packages (.jar)
Add support for running JNI test in the verification script
Add support for running integration test in the verification script
Add support for verifying binary packages (.jar) in the verification script

kou · 2025-01-09T08:35:13Z

I'll test this on my fork.
If you have any concerns for this workflow, please share them.

lidavidm · 2025-01-10T00:47:58Z

dev/release/README.md

+## Overview
+
+  1. Test the revision to be released
+  2. Bump version by `dev/release/bump_version.sh X.Y.Z`


The script appears to create a PR, let's be explicit that the PR should be merged and then the release script should be run from that commit?

It makes sense.
I'll do it.

lidavidm · 2025-01-10T00:49:08Z

dev/release/release_rc.sh

+if [ "${RELEASE_PULL}" -gt 0 ]; then
+  echo "Ensure using the latest commit"
+  git checkout main
+  git pull --rebase --prune


Hmm, wouldn't --ff-only be better in case there's an unpushed local commit for some reason?

Good catch! We should use --ff-only here. I'll fix this.

lidavidm · 2025-01-10T00:50:35Z

.github/workflows/release.yml

+        run: |
+          version=${GITHUB_REF_NAME#v}
+          gh release create ${GITHUB_REF_NAME} \
+            --notes "TODO" \


I suppose we should consider something like conventional commits for this repo too? Or else manually curate a CHANGELOG.md?

Let's discuss this as a separated task: #499

lidavidm · 2025-01-10T00:51:33Z

.github/workflows/rc.yml

I suppose at some point we should refactor the JNI build so we can include binary artifacts here too.

Ah, it's a good idea!
Can I merge the existing test_jni.yml to here?
Or should we keep these workflows separated?

I don't think we need to tackle this right away. GitHub lets you run actions as a substep, so maybe we can run test_jni.yml as a substep, then pull the artifacts back into the release pipeline.

Are you thinking about a composite action (e.g. we're using it in apache/arrow: https://github.com/apache/arrow/tree/main/.github/actions/sync-nightlies ) or reusable workflow?

Anyway, lets' defer this: GH-500

I was thinking reusable workflow, but either way works for me - I just mean there's no need to merge/duplicate the pipelines. I'm OK with deferring it, yes.

.github/workflows/rc.yml

wgtmac · 2025-01-10T02:22:54Z

dev/release/bump_version.sh

+  tag=v${version}
+  ;;
+esac
+mvn versions:set-scm-tag "-DnewTag=${tag}" -DgenerateBackupPoms=false -pl :arrow-java-root


Why do we need set-scm-tag for SNAPSHOT version?

Because we don't create a branch for release in the proposed workflow.

My understanding is that this updates project.scm.tag:

arrow-java/pom.xml

Line 84 in 4a535af

<tag>main</tag>

pom.xml in the main branch uses vX.Y.Z while we're releasing. We need to back it to main after we released X.Y.Z.

Do we want to create a release branch? If we have a release branch, we don't need to back it to main because pom.xml in the main branch always uses main for project.scm.tag. pom.xml in release branches use vX.Y.Z for project.scm.tag.

(If we create a release branch, release process will be a bit complex.)

If we want to follow up with a 19.0.1 release to fix issues in 19.0.0, does it mean that we have to create the RC from main branch too?

Yes.

FYI: apache/arrow-julia and apache/arrow-go use this style.

If we have enough resource to maintain multiple major releases (e.g. 18.Y.Z and 19.Y.Z), we can create a release branch for each major release. But I think that we don't have enough resource for now...

Agreed. Let's not over-complicate the script for now.

wgtmac · 2025-01-10T02:26:53Z

dev/release/release_rc.sh

+if [ "${RELEASE_PULL}" -gt 0 ] || [ "${RELEASE_PUSH_TAG}" -gt 0 ]; then
+  git_origin_url="$(git remote get-url origin)"
+  if [ "${git_origin_url}" != "[email protected]:apache/arrow-java.git" ]; then
+    echo "This script must be ran with working copy of apache/arrow-java."


Should we also add that it cannot accept https mode?

Are you (and other commiters/PMC members who may become a release manager) using https:// for writable clone?
If so, let's accept https:// too.

FYI: I'm using https:// for read-only clones.

Usually I use https:// to git clone a large repo in the beginning which becomes the origin remote by default. I can manually switch origin to use ssh to avoid complication here.

OK. Let's accept https:// too.
It will not be too complex.

lidavidm · 2025-01-10T04:47:13Z

.github/workflows/rc.yml

I was thinking reusable workflow, but either way works for me - I just mean there's no need to merge/duplicate the pipelines. I'm OK with deferring it, yes.

Fixes apacheGH-20. This is based on apache/arrow-go#122 . Workflow: Cut a RC: 1. Bump version by `dev/release/bump_version.sh 19.0.0` 2. Run `dev/release/release_rc.sh` 1. `dev/release/release_rc.sh` pushes `v${version}-rc${rc}` tag 2. `.github/workflows/rc.yml` creates `apache-arrow-java-${version}.tar.gz{,.sha256,.sha512}` 3. `.github/workflows/rc.yml` uploads `apache-arrow-java-${version}.tar.gz{,.sha256,.sha512}` to GitHub Releases 4. `dev/release/release_rc.sh` downloads `apache-arrow-java-${version}.tar.gz` from GitHub Releases 5. `dev/release/release_rc.sh` signs `apache-arrow-java-${version}.tar.gz` as `apache-arrow-go-${version}.tar.gz.asc` 6. `dev/release/release_rc.sh` uploads `apache-arrow-java-${version}.tar.gz.asc` to GitHub Releases 3. Start a vote (GitHub Actions instead of https://dist.apache.org/repos/dist/dev/arrow/ is used like ADBC.) Verify a RC: 1. Run `dev/release/verify_rc.sh` Release an approved RC: 1. Run `dev/release/release.sh` 1. `dev/release/release.sh` pushes `v${version}` tag that refers that same commit ID as `v${version}-rc${rc}` 2. `dev/release/release.sh` downloads `apache-arrow-java-${version}.tar.gz{,.asc,.sha256,.sha512}` from GitHub Actions 3. `dev/release/release.sh` uploads `apache-arrow-java-${version}.tar.gz{,.asc,.sha256,.sha512}` to https://dist.apache.org/repos/dist/release/arrow 4. `dev/release/release.sh` removes old releases from https://dist.apache.org/repos/dist/release/arrow 2. Add this release to ASF's report database: https://reporter.apache.org/addrelease.html?arrow Follow-up tasks: * Add support for building binary packages (.jar) * Add support for releasing binary packages (.jar) * Add support for running JNI test in the verification script * Add support for running integration test in the verification script * Add support for verifying binary packages (.jar) in the verification script * Add support for releasing a release note

kou · 2025-01-10T08:28:24Z

I tried the workflow on my fork. It worked with some fixes.

18.2.0 RC0 GitHub Release: https://github.com/kou/arrow-java/releases/tag/v18.2.0-rc0
18.2.0 GitHub Release: https://github.com/kou/arrow-java/releases/tag/v18.2.0

raulcd

Thanks @kou ! Looks great!

dev/release/README.md

Co-authored-by: Raúl Cumplido <[email protected]>

raulcd

👍

kou requested review from lidavidm, laurentgo and wgtmac as code owners January 9, 2025 08:34

lidavidm approved these changes Jan 10, 2025

View reviewed changes

kou mentioned this pull request Jan 10, 2025

[Release] How to create release note? #499

Open

wgtmac reviewed Jan 10, 2025

View reviewed changes

kou mentioned this pull request Jan 10, 2025

[Release] Add support for releasing .jar including JNI binaries #500

Closed

kou force-pushed the release-script branch from 3c5be3d to cb2f1b2 Compare January 10, 2025 04:10

kou mentioned this pull request Jan 10, 2025

MINOR: Bump version to 19.0.0 #501

Closed

lidavidm approved these changes Jan 10, 2025

View reviewed changes

kou added 18 commits January 10, 2025 14:47

Don't run release workflow for PR

88a3456

Fix style

b417364

Add missing arguments

c2a91fa

Fix lint errors

fed3ea5

Add missing prefix

29aa210

Remove needless GITHUB_TOKEN

ed52615

Don't accept addition commit in main

9a11639

Accept https://github.com/apache/arrow-java.git

191e584

Describe dev/release/bump_version.sh

ca473de

Use --generate-notes

7c51faa

Nightly

f08525b

Update end tag

ef414bf

Fix style

cf645a6

Use main as the base branch

405073e

Specify repository explicitly

9ff0ff2

Push explicitly

b736ee2

Specify branch explicitly

e89edb1

Set upstream explicitly

eb3c429

kou force-pushed the release-script branch from d875b80 to eb3c429 Compare January 10, 2025 05:47

kou added 3 commits January 10, 2025 14:49

Add missing GH_TOKEN

f996be6

Accept https://

6402ef4

Fix release path

d23d600

Fix name

13df24d

kou mentioned this pull request Jan 10, 2025

[Release] dev/release/release.sh uploads artifacts to wrong path apache/arrow-go#242

Closed

lidavidm approved these changes Jan 10, 2025

View reviewed changes

raulcd reviewed Jan 10, 2025

View reviewed changes

dev/release/README.md Outdated Show resolved Hide resolved

dev/release/README.md Show resolved Hide resolved

dev/release/README.md Show resolved Hide resolved

dev/release/README.md Show resolved Hide resolved

dev/release/README.md Show resolved Hide resolved

kou and others added 5 commits January 11, 2025 05:21

Fix a typo

711bf2e

Co-authored-by: Raúl Cumplido <[email protected]>

Fix how to update KEYS

e59a1b1

Add a note about merging PR

ab28353

Describe -SNAPSHOT update

784fc63

Use release/arrow/KEYS for verification

4156c80

raulcd approved these changes Jan 13, 2025

View reviewed changes

kou merged commit 14dcd50 into apache:main Jan 13, 2025
21 checks passed

kou deleted the release-script branch January 13, 2025 09:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-20: Add release scripts #496

GH-20: Add release scripts #496

kou commented Jan 9, 2025 •

edited

Loading

kou commented Jan 9, 2025

lidavidm Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

wgtmac Jan 10, 2025

kou Jan 10, 2025

wgtmac Jan 10, 2025

kou Jan 10, 2025

wgtmac Jan 10, 2025

wgtmac Jan 10, 2025

kou Jan 10, 2025

wgtmac Jan 10, 2025

kou Jan 10, 2025

lidavidm Jan 10, 2025

kou commented Jan 10, 2025

raulcd left a comment

raulcd left a comment

GH-20: Add release scripts #496

GH-20: Add release scripts #496

Conversation

kou commented Jan 9, 2025 • edited Loading

kou commented Jan 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Jan 10, 2025

raulcd left a comment

Choose a reason for hiding this comment

raulcd left a comment

Choose a reason for hiding this comment

kou commented Jan 9, 2025 •

edited

Loading