Mechanism for triggering template build #3935

marmarek · 2018-05-27T14:50:14Z

Right now template build require direct access to build machine. This is bad for various reasons, including:

lack of transparency
inability to delegate it

We've using a better mechanism for individual packages for some time. It consists of:

a script that build and upload to current-testing a package that got new signed version tag
a script that process signed comment on github to migrate package from current-testing to current (Transparent mechanism for migrating packages from current-testing to current #2573)

This works well for normal packages, lets do something similar for templates.
The problem here is there is no natural place for template version tag - template consists of more than just linux-template-builder - in many cases we need to build new template after just updating individual packages (which doesn't involve any change in linux-template-builder).

I propose using something similar to #2573 for triggering the build too. The work flow would be:

template maintainer open new issue (in updates-status repository? or here?) with inline-signed command to build the template
appropriate script verify the signature and proceed to build the template (using make build-template-in-dispvm)
when build succeed, result is reported to https://github.com/QubesOS/updates-status/issues repository and template is uploaded to online repository
build failure would be also reported, in https://github.com/QubesOS/build-issues/issues

Open questions:

How build trigger command should look like (what we need there)? My current idea:
```
 build-template QUBES_VERSION TEMPLATE TIMESTAMP
```
For example: build-template r4.0 fc28+minimal 201805171636
Do we need templates-itl-testing and templates-community-testing repositories and upload templates there first? And the use similar workflow to migrate them later to final repositories?
Replay protection - in the example above I've added TIMESTAMP (which will be used as %release part of template version) exactly for this reason. The script would verify if it isn't too far in the past. Any better idea?
Template package version is pretty meaningless right now (it's template builder version (but not builder plugin - containing most actual building scripts) + timestamp). Maybe we could use this occasion to change this too? For example specify template version manually too? Or build it based on qubes version (3.2.x for templates for Qubes 3.2 etc)?

This all would for example allow @adrelanos to trigger new Whonix template build or @fepitre for Fedora and CentOS, or @ptitdoc for Archlinux, or @unman for Debian.

cc @andrewdavidwong

The text was updated successfully, but these errors were encountered:

andrewdavidwong · 2018-05-27T18:44:01Z

Do we need templates-itl-testing and templates-community-testing repositories and upload templates there first? And the use similar workflow to migrate them later to final repositories?

This would probably be a good idea.

Preparation for auto-build-template script, avoid code duplication. QubesOS/qubes-issues#3935

This require little changes to update-repo: - not stripping dist flavors for linux-template-builder component - handling simplified repository layout (no dom0/fc25 subdir) - setting TEMPLATE_NAME variable - calculating it is done in various makefiles and need to be done before parsing templates.spec; unfortunately variables set in makefile aren't available to $(shell ...) in the very same makefile, so this needs to be set earlier QubesOS/qubes-issues#3935

Allow easily get calculated template name. While the command looks simple, it needs to be called with right environment variables (especially - with settings in builder.conf). QubesOS/qubes-issues#3935

If builder.conf is generated by setup script, it is set, but otherwise it isn't. And because of this, various scripts guess release version based on repository dir name. Lets have it in one place. QubesOS/qubes-issues#3935

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

QubesOS/qubes-issues#3935

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

This require little changes to update-repo: - not stripping dist flavors for linux-template-builder component - handling simplified repository layout (no dom0/fc25 subdir) - setting TEMPLATE_NAME variable - calculating it is done in various makefiles and need to be done before parsing templates.spec; unfortunately variables set in makefile aren't available to $(shell ...) in the very same makefile, so this needs to be set earlier. See https://savannah.gnu.org/bugs/?10593 QubesOS/qubes-issues#3935

Allow easily get calculated template name. While the command looks simple, it needs to be called with right environment variables (especially - with settings in builder.conf). QubesOS/qubes-issues#3935

If builder.conf is generated by setup script, it is set, but otherwise it isn't. And because of this, various scripts guess release version based on repository dir name. Lets have it in one place. QubesOS/qubes-issues#3935

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

QubesOS/qubes-issues#3935

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

This allows to reuse standard update-repo-* and sign-* make targets. QubesOS/qubes-issues#3935

QubesOS/qubes-issues#3935

Don't assume UPDATE_REPO contains trailing slash, there may be no more subdirectories (templates-* repos). QubesOS/qubes-issues#3935

This require little changes to update-repo: - not stripping dist flavors for linux-template-builder component - handling simplified repository layout (no dom0/fc25 subdir) - setting TEMPLATE_NAME variable - calculating it is done in various makefiles and need to be done before parsing templates.spec; unfortunately variables set in makefile aren't available to $(shell ...) in the very same makefile, so this needs to be set earlier. See https://savannah.gnu.org/bugs/?10593 QubesOS/qubes-issues#3935

Allow easily get calculated template name. While the command looks simple, it needs to be called with right environment variables (especially - with settings in builder.conf). QubesOS/qubes-issues#3935

If builder.conf is generated by setup script, it is set, but otherwise it isn't. And because of this, various scripts guess release version based on repository dir name. Lets have it in one place. QubesOS/qubes-issues#3935

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

QubesOS/qubes-issues#3935

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

It already had custom recipe, but since now it got Makefile.builder file, really bypass the default one. QubesOS/qubes-issues#3935

adrelanos · 2018-07-21T08:58:44Z

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

qubes-template-whonix-gw-14 4.0.1-201807190624 is being downloaded.

Upload failure not detected?

Out of disk space?

https://yum.qubes-os.org/r4.0/templates-community-testing/rpm/ has:

qubes-template-whonix-gw-14-4.0.1-2018 07130946.noarch.rpm
qubes-template-whonix-gw-14-4.0.1-2018 07171801.noarch.rpm
qubes-template-whonix-ws-14-4.0.1-2018 07131452.noarch.rpm
qubes-template-whonix-ws-14-4.0.1-2018 07171801.noarch.rpm

By the way, all older templates can all be purged to save disk space.

marmarek · 2018-07-21T12:47:23Z

qubes-template-whonix-gw-14 4.0.1-201807210556 even though QubesOS/updates-status#609 says it has been uploaded, is not actually available in templates-community-testing.

It is, for 3.2 (not 4.0), here: https://yum.qubes-os.org/r3.2/templates-community-testing/rpm/

adrelanos · 2018-07-24T13:46:29Z

QubesOS/updates-status#572 (comment)

Upload core-agent-linux ec251da5d8e6ed5544e91d92695a41f4d86d43d5 r4.0 current repo

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

I guess such a notification if upload actually happened could save a some time in future in case it fails for some reason.

marmarek · 2018-07-24T16:24:26Z

There's was no notification Package for fc28 was built (build log) and uploaded to current repository. Expected?

Sort of. This is because it wasn't uploaded. Because there was already newer package: QubesOS/updates-status#593

ptitdoc · 2018-07-28T09:29:26Z

Hello,

Sorry, I did not tested yet but the key D85EE12F967851CCF433515A2043E7ACC1833B9C is good for me for building templates.

ptitdoc · 2018-08-20T08:00:43Z

Hello,

I'm only starting to test this. I'm not sure where I should create specific archlinux configuration. Should I put it there:
https://github.com/QubesOS/qubes-builder/tree/master/release-configs
or there:
https://github.com/QubesOS/qubes-template-configs/tree/master/R4.0/templates-community

marmarek · 2018-09-02T00:16:49Z

The latter.

QubesOS/qubes-issues#3935

This enables use of different mirrors. Specifify fastestmirror=1 explicitly for templates repository, as it has the biggest impact there. QubesOS/qubes-issues#3935 (cherry picked from commit 217d223)

marmarek · 2019-01-03T05:26:48Z

@unman what key do you want to use for signing build requests? I have two of them:

C6E2533D976E4B29A82B79CB0C6D15D5E0A9816E
4B1F400DF25651B53C4141B38B3F30F9C8C0C2EF

You tried the second one already, but I want explicit confirmation.

unman · 2019-01-03T13:57:41Z

@marmarek The second is my signing key for Qubes:
4B1F400DF25651B53C4141B38B3F30F9C8C0C2EF

Confirmed here

marmarek · 2019-01-03T13:59:54Z

Ok, I read this as "use 4B1F400DF25651B53C4141B38B3F30F9C8C0C2EF for build commands access list"

unman · 2019-01-03T14:20:00Z

Yes, sorry. You've understood my somewhat gnomic comment aright.
Please use key 4B1F400DF25651B53C4141B38B3F30F9C8C0C2EF for access list.

QubesOS/qubes-issues#3935

marmarek · 2019-01-03T14:29:42Z

Done.

adrelanos · 2021-11-25T10:39:07Z

Using...

REPO_PROXY="" make DISTS_VM="whonix-gateway-16 whonix-workstation-16" template-github

Builds templates for Qubes R4.0.

What's the command for Qubes R4.1?

adrelanos · 2021-11-25T10:44:07Z

Comment out from builder.conf : RELEASE = 4.0 and

RELEASE=4.1 REPO_PROXY="" make DISTS_VM="whonix-gateway-16 whonix-workstation-16" template-github

marmarek · 2023-03-04T01:53:26Z

This is all done for quite some time already.

andrewdavidwong added enhancement C: builder Qubes Builder C: templates P: major Priority: major. Between "default" and "critical" in severity. labels May 27, 2018

andrewdavidwong added this to the Release 4.1 milestone May 27, 2018

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

Move utility functions from auto-build script to separate file

f4a857e

Preparation for auto-build-template script, avoid code duplication. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

Fix make sign-all to sign templates with flavors

7d976fd

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

WIP Make check-release-status script work for templates too

f219aad

QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

Simplify sign-all target for templates

aafef12

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

Fix make sign-all to sign templates with flavors

295c827

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

WIP Make check-release-status script work for templates too

f9a265a

QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 4, 2018

Simplify sign-all target for templates

8b1e92f

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-linux-template-builder that referenced this issue Jun 4, 2018

Pretend the template rpm is build the standard way

a78fc2d

This allows to reuse standard update-repo-* and sign-* make targets. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Jun 13, 2018

Allow adding RPM_QUERY_DEFINES in builder.conf or Makefile.builder

8509e67

QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder-rpm that referenced this issue Jun 13, 2018

Fix update-repo target for simplified repositories

32d80c1

Don't assume UPDATE_REPO contains trailing slash, there may be no more subdirectories (templates-* repos). QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 17, 2018

Fix make sign-all to sign templates with flavors

4debdbf

Yet another place where template flavors needs not to be stripped when handling template. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 17, 2018

WIP Make check-release-status script work for templates too

3db9872

QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 17, 2018

Simplify sign-all target for templates

eba4116

Since templates can be now handled as other rpm packages, there is no need for separate section. QubesOS/qubes-issues#3935

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jun 17, 2018

Exclude linux-template-builder from default build recipe

cc1f34c

It already had custom recipe, but since now it got Makefile.builder file, really bypass the default one. QubesOS/qubes-issues#3935

adrelanos mentioned this issue Jul 21, 2018

/usr/local fails to persist in Qubes-Whonix 14 / mount-dirs.sh[295]: head: error writing 'standard output': Broken pipe #4095

Closed

fepitre added a commit to fepitre/qubes-builder that referenced this issue Sep 3, 2018

Allows to trig template build on Github inside the builder

9910450

QubesOS/qubes-issues#3935

fepitre mentioned this issue Sep 3, 2018

Allows to trig template build on Github inside the builder QubesOS/qubes-builder#61

Merged

qubesos-bot mentioned this issue Sep 17, 2018

installer-qubes-os v23.19.10-5-anaconda (r3.2) QubesOS/updates-status#698

Closed

marmarek mentioned this issue Oct 22, 2018

qubes-core-agent-networking deb dependency issue #4411

Closed

andrewdavidwong mentioned this issue Oct 23, 2018

Add debian-minimal to official template repo #4422

Closed

marmarek mentioned this issue Dec 31, 2018

Place for build template commands QubesOS/updates-status#566

Open

marmarek added a commit to marmarek/qubes-builder that referenced this issue Jan 3, 2019

Add @unman key to debian-template build access list

6f73c6f

QubesOS/qubes-issues#3935

adrelanos mentioned this issue Jun 24, 2021

qubes-builder make template-github curl broken - wrong environment variable handling #6737

Open

andrewdavidwong modified the milestones: Release 4.1, Release 4.1 updates, Release 4.2 Feb 5, 2022

marmarek closed this as completed Mar 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mechanism for triggering template build #3935

Mechanism for triggering template build #3935

marmarek commented May 27, 2018

andrewdavidwong commented May 27, 2018

adrelanos commented Jul 21, 2018

marmarek commented Jul 21, 2018

adrelanos commented Jul 24, 2018

marmarek commented Jul 24, 2018

ptitdoc commented Jul 28, 2018

ptitdoc commented Aug 20, 2018

marmarek commented Sep 2, 2018

marmarek commented Jan 3, 2019

unman commented Jan 3, 2019

marmarek commented Jan 3, 2019

unman commented Jan 3, 2019

marmarek commented Jan 3, 2019

adrelanos commented Nov 25, 2021

adrelanos commented Nov 25, 2021

marmarek commented Mar 4, 2023

Mechanism for triggering template build #3935

Mechanism for triggering template build #3935

Comments

marmarek commented May 27, 2018

andrewdavidwong commented May 27, 2018

adrelanos commented Jul 21, 2018

marmarek commented Jul 21, 2018

adrelanos commented Jul 24, 2018

marmarek commented Jul 24, 2018

ptitdoc commented Jul 28, 2018

ptitdoc commented Aug 20, 2018

marmarek commented Sep 2, 2018

marmarek commented Jan 3, 2019

unman commented Jan 3, 2019

marmarek commented Jan 3, 2019

unman commented Jan 3, 2019

marmarek commented Jan 3, 2019

adrelanos commented Nov 25, 2021

adrelanos commented Nov 25, 2021

marmarek commented Mar 4, 2023