Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pisi/eopkg: Add versioned FilesDB and LazyDB instances #4063

Merged
merged 3 commits into from
Nov 4, 2024

Conversation

ermo
Copy link
Contributor

@ermo ermo commented Oct 15, 2024

Summary

  • eopkg.py2 now writes versioned (v3), gdbm format FilesDB shelves, and versioned LazyDB caches; both using Pickle Protocol Version 2.
  • eopkg.py3/bin now writes versioned (v4), gdbm format FilesDB shelves, and versioned LazyDB caches; both using Pickle Protocol Version 2.
  • FilesDB and LazyDBs are autoupgraded on version mismatch.
  • The output of sudo eopkg.py2/py3/bin rdb operations has been cleaned up significantly.
  • Taskfile now uses eopkg.bin for indexing operations due to the improved output it provides.
  • ypkg-install-deps supports an optional -e/--eopkg-cmd argument, which defaults to eopkg.py3
    (look at Add installCommand to ypkg-install-deps invocation solbuild#117 for how this will be leveraged in solbuild).

This is intended to help make the transition between using eopkg.py2 and eopkg.py3/bin much smoother.

Changelog

Test Plan

  • Check out this branch
  • Clean out the local repo with go-task clean-local
  • Build pisi w/ go-task localcp and install it
    • Pay attention to any mentions of FilesDB rebuilds in subsequent build logs (e.g. eopkg, ypkg and whatever else you decide to build for testing purposes). In particular, pay attention to the FilesDB (version: part.
  • Build eopkg w/ go-task localcp and install it and its python-eopkg subpackage
    • Notice how after installing the updated eopkg, subsequent builds will now show a list of which packages are indexed in the local repo prior to the build.
  • Build ypkg w/ go-task localcp and install it
  • Follow the Test Plan in solbuild: Drop fakeroot use, use eopkg.bin #3850

Checklist

  • Package was built and tested against unstable

@ermo ermo force-pushed the eopkg-and-pisi-FilesDB-and-LazyDB-update branch 2 times, most recently from bcac2e3 to 7c36e5b Compare October 15, 2024 19:29
@ermo ermo changed the title pisi: Update to v3.12.5 (FilesDB and LazyDB update) pisi/eopkg: Add versioned FilesDB and LazyDB instances Oct 15, 2024
@ermo ermo force-pushed the eopkg-and-pisi-FilesDB-and-LazyDB-update branch 3 times, most recently from fbd2427 to 20c9b58 Compare October 16, 2024 17:10
@ermo ermo requested a review from sheepman4267 October 16, 2024 17:29
@EbonJaeger EbonJaeger added the hacktoberfest-accepted This PR is accepted for Hacktoberfest label Oct 18, 2024
Smokolak pushed a commit to Smokolak/packages that referenced this pull request Oct 19, 2024
**Summary**
 * General:
   * Change: Give experience when squishing a scavenger (commit:736fae8e874744a7156de54ad4330cb22ab9ec01, getsolus#4046)
   * Fix: Various potential crashes (too many commits, getsolus#4050, getsolus#4051, getsolus#4053, getsolus#4063)
 * Graphics:
   * Fix: Reset object animation state in a few places (commit:84b93d9c0b12e924a2aabaf35ce02d5124485df1, getsolus#4049)
   * Fix: Additional graphics backend fallback options for macOS and Linux (commit:347b7d07835afe1ecf91fd4750890c67e6e3c1b4, getsolus#4052)
 * Campaign:
   * Fix: Visibility of units restored during limbo missions (commit:22fa9439c8515b52fd816323f9f0a2a0fe985b7e, getsolus#4045)
   * Fix: Reset structure functionality references to objects when going offworld (commit:1de0822cdc1c3f1f650b1c531ebce2fe03918bf0, getsolus#4047)
   * Fix: Fill VTOLs when going offworld (commit:87ae96c79799ebe066b5213ea3d49891b3a8a1eb, getsolus#4048)
   * Fix: Update Remastered balance (too many commits, getsolus#4056)
   * Fix: Update Reclamation addon campaign (commit:9f2ca2e418ed8dfdb9488cb0e11d052fdd3aa1b0, getsolus#4059)
   * Fix: Update Fractured Kingdom addon campaign (commit:c980864ed7a920f1419d4027633438276da85d6f, getsolus#4059)
 * Translations:
   * Change: Numerous translation updates (commit:c0b1156087bc8654cc3d1f90fa588ff79769c7c7, getsolus#3996)
@ermo ermo force-pushed the eopkg-and-pisi-FilesDB-and-LazyDB-update branch from 20c9b58 to 7f31179 Compare October 20, 2024 14:17
@ermo ermo added Topic: Platform Integration Integration of various components within Solus Topic: Plumbing Core components Topic: Tooling Tools and scripts labels Oct 20, 2024
@ermo ermo added this to the Solus 4.7 milestone Oct 20, 2024
- With this update, eopkg.py2 now writes versioned (v3), gdbm format
  FilesDB shelves, and versioned LazyDB caches; both using Pickle
  Protocol Version 2.
- FilesDB and LazyDBs are autoupgraded on version mismatch.
- The output of `sudo eopkg.py2 rdb` operations has been cleaned up
  significantly.

This is intended to help make the transition between using `eopkg.py2`
and `eopkg.py3` much smoother.

Changelog [here](https://github.com/getsolus/eopkg/releases/tag/v3.12.5)

Signed-off-by: Rune Morling <[email protected]>
@ermo ermo force-pushed the eopkg-and-pisi-FilesDB-and-LazyDB-update branch from 7f31179 to d9fdb37 Compare October 23, 2024 16:54
ermo added 2 commits October 23, 2024 19:17
- With this update, eopkg.py3/bin now writes versioned (v4), gdbm format
  FilesDB shelves, and versioned LazyDB caches; both using Pickle
  Protocol Version 2.
- FilesDB and LazyDBs are autoupgraded on version mismatch.
- The output of `sudo eopkg.py3 rdb` operations has been cleaned up
  significantly and is identical to the output of `eopkg.py2`.
- The output of `sudo eopkg index` operations has been improved. It now
  lists each package added to the index.

This should help make the transition between using `eopkg.py2` and
`eopkg.py3` much smoother.

Changelog:
- [4.1.5](https://github.com/getsolus/eopkg/releases/tag/v4.1.5)
- [4.1.6](https://github.com/getsolus/eopkg/releases/tag/v4.1.6)

Signed-off-by: Rune Morling <[email protected]>
This build of ypkg has a version of ypkg-install-deps which supports the
--eopkg-cmd argument.

With no argument supplied, ypkg-install-deps defaults to calling
`eopkg.py3`.

Signed-off-by: Rune Morling <[email protected]>
@ermo ermo force-pushed the eopkg-and-pisi-FilesDB-and-LazyDB-update branch from d9fdb37 to 321e2f4 Compare October 23, 2024 17:19
@ermo
Copy link
Contributor Author

ermo commented Oct 23, 2024

Note: I have double bumped this past the ypkg release that staff have been using for local tests, hence the failing tests.

@ermo ermo marked this pull request as ready for review October 23, 2024 17:22
@ermo ermo mentioned this pull request Nov 1, 2024
Copy link
Member

@silkeh silkeh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ermo ermo merged commit 01be8e6 into main Nov 4, 2024
1 check failed
@ermo ermo deleted the eopkg-and-pisi-FilesDB-and-LazyDB-update branch November 4, 2024 22:41
ermo added a commit that referenced this pull request Nov 4, 2024
**Summary**

Adds the new solbuild with no-fakeroot functionality.

Important note: Landing this PR to -unstable needs to be followed up by
updating the solbuild image, otherwise we will get some really annoying
failures related to libglib-json moving certain files to a -devel
subpackage IIUC.

Depends on #4063 and should be
landed after it.

**Test Plan**

- Complete the Test Plan for
#4063 first
- Check out the branch associated with the present PR
- Build solbuild with `go-task localcp` and install it
- Run `solbuild version` and check that it shows 1.7.0
- Check out the branch from
#4063 again
- Demonstrate to yourself that the two PRs taken together are
self-hosting:
- Re-build pisi, eopkg, ypkg from the 4063 branch w/ `go-task localcp`
and re-install each of them after each build completes
- Switch back to the present solbuild branch, and rebuild solbuild with
`go-task localcp` and re-install it after the build completes
- It is important that you do _not_ clean the Local repo cache before
doing this!
- Once again, pay attention to the `FilesDB (version:` output. It should
shift from being version 3 to version 4 from `eopkg` and onwards, which
is a sign that the new solbuild is now asking `ypkg-install-deps` to use
`eopkg.bin` to install deps (it was hardcoded to `eopkg -> eopkg.py2` in
ypkg before).
- Build a bunch of packages of your choice (using `go-task local` or
`go-task localcp`) to test the new tooling versions.
- Again, pay special attention to the `FilesDB (version:` output in the
build logs.
- If you do, you will realise that the FilesDB is getting rebuilt during
each build. This is because the solbuild backing image has not yet been
generated with the new version of `eopkg.bin`. Once the #4063 PR and the
present PR has been landed, we can then update the solbuild images to
also use eopkg.bin and `FilesDB (version: 4)`.

**Checklist**

- [x] Package was built and tested against unstable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hacktoberfest-accepted This PR is accepted for Hacktoberfest Topic: Platform Integration Integration of various components within Solus Topic: Plumbing Core components Topic: Tooling Tools and scripts
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants