From c7465317f7531d8b26275307e08a09ce837ba913 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Fri, 20 Jan 2023 09:34:42 +0100 Subject: [PATCH 1/4] Enhance documentation of checksums EC parameter Add more examples of which formats are possible. Also contains not-yet implemented or broken features. --- docs/writing-easyconfig-files.md | 58 ++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 2 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 7238a928c..0f63de40b 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -232,8 +232,7 @@ highly recommended. If checksums are provided, the checksum of the corresponding source files and patches is verified to match. -The `checksums` easyconfig parameter is usually defined as a list of -strings. +The `checksums` easyconfig parameter is a list usually containing strings. Until EasyBuild v3.3.0, only MD5 checksums could be provided through a list of strings. Since EasyBuild v3.3.0, the checksum type is determined @@ -257,6 +256,61 @@ example: checksums = [('sha512', 'f962008105639f58e9a4455c8057933ab0a5e2f43db8340ae1e1afe6dc2d24105bfca3b2e1f79cb242495ca4eb363c9820d8cea6084df9d62c4c3e5211d99266')] ``` +It is also possible to specify alternative checksums using a tuple of +checksum elements where any match is sufficient (logical OR). +This is helpful when the release was updated but can still be used +(e.g. only doc changes). + +``` python +checksums = [('mainchecksum', 'alternativechecksum')] # Placeholders used +``` + +The opposite is also possible: +A list instead of a tuple denotes that **all** checksums must match (logical AND). +In both cases each element can also be a type-value-tuple: + +``` python +checksums = [[('size', 42), 'sha256checksum')] # Placeholder used +``` + +Finally a checksum can be specified as a dictionary mapping filenames to checksums. +This is useful when the source file is specified using e.g. the `%(arch)s` template. +Again elements (values) can be strings or type-value-tuples. +For example: +``` python +checksums = [{ + 'src_x86_64.tgz': 'f962008105639f58e9a4455c8057933ab0a5e2f43db8340ae1e1afe6dc2d2410', + 'src_aarch64.tgz': ('size', 42), +}] +``` + +Of course this can be combined with the logical AND/OR semantics using lists or tuples: +``` python +checksums = [{ + 'src_x86_64.tgz': ('oldchecksum', 'newchecksum'), + 'src_aarch64.tgz': [('size', 42), 'checksumthatmustmatchtoo'], +}] +``` + +When the checksum cannot be specified for a file +(e.g. when using a git clone instead of an archive) +a value of `None` can be used to skip the checksum check. +This is possible in the list of checksums as well as as a value in a dictionary, e.g.: +``` python +checksums = [ + None, # No checksum for first source file + 'checksumfor2ndfile', + { + 'third_file_x86_64.tgz': 'checksum', + 'third_file_aarch64.tgz': None, + }, +] +``` + +The difference between having an entry in the dict with the value of `None` and not having an entry +only matters when using the `--enforce_checksums` option which will raise an error in the latter case. + + ##### Adding or replacing checksums using `--inject-checksums` {: #inject_checksums } Using the `--inject-checksums` command line option, you can let From 0937747e0cefd22c982b0d1c4271bc4eb4f5b148 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Tue, 28 Feb 2023 17:40:37 +0100 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Sam Moors --- docs/writing-easyconfig-files.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 0f63de40b..88c84cfb5 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -258,7 +258,7 @@ checksums = [('sha512', 'f962008105639f58e9a4455c8057933ab0a5e2f43db8340ae1e1afe It is also possible to specify alternative checksums using a tuple of checksum elements where any match is sufficient (logical OR). -This is helpful when the release was updated but can still be used +This is helpful when the release was updated with changes that don't affect the behavior of the software in any way (e.g. only doc changes). ``` python @@ -270,12 +270,13 @@ A list instead of a tuple denotes that **all** checksums must match (logical AND In both cases each element can also be a type-value-tuple: ``` python -checksums = [[('size', 42), 'sha256checksum')] # Placeholder used +checksums = [[('size', 42), 'sha256checksum']] # Placeholder used ``` -Finally a checksum can be specified as a dictionary mapping filenames to checksums. -This is useful when the source file is specified using e.g. the `%(arch)s` template. -Again elements (values) can be strings or type-value-tuples. +Finally, a checksum can be specified as a dictionary mapping filenames to checksums, removing any ambiguity. +This style is used by Easybuild with `eb --inject-checksums` when 2 or more source files are specified. +This style is also useful when the source file is specified using e.g. the `%(arch)s` template. +Again, elements (values) can be strings or type-value-tuples. For example: ``` python checksums = [{ @@ -293,7 +294,7 @@ checksums = [{ ``` When the checksum cannot be specified for a file -(e.g. when using a git clone instead of an archive) +(e.g. when using a git clone instead of an archive), a value of `None` can be used to skip the checksum check. This is possible in the list of checksums as well as as a value in a dictionary, e.g.: ``` python From 42582ee39545cae729c67b2b9ae72af24a12d94f Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 16 Nov 2023 09:26:34 +0100 Subject: [PATCH 3/4] Add review feedback (mostly better placeholders) --- docs/writing-easyconfig-files.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 88c84cfb5..89161935a 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -262,7 +262,7 @@ This is helpful when the release was updated with changes that don't affect the (e.g. only doc changes). ``` python -checksums = [('mainchecksum', 'alternativechecksum')] # Placeholders used +checksums = [('0123456789...abcdef', 'fedcba...9876543210')] ``` The opposite is also possible: @@ -270,39 +270,43 @@ A list instead of a tuple denotes that **all** checksums must match (logical AND In both cases each element can also be a type-value-tuple: ``` python -checksums = [[('size', 42), 'sha256checksum']] # Placeholder used +checksums = [[('sha256', '0123456789...abcdef'), 'fedcba...9876543210']] ``` Finally, a checksum can be specified as a dictionary mapping filenames to checksums, removing any ambiguity. -This style is used by Easybuild with `eb --inject-checksums` when 2 or more source files are specified. -This style is also useful when the source file is specified using e.g. the `%(arch)s` template. +This style is used by EasyBuild with `eb --inject-checksums` when 2 or more source files are specified, +and is particularly useful when the source file is specified using a template value like `%(arch)s`. +Especially when many source files and patches are used this also directly documents the file each checksum is for. Again, elements (values) can be strings or type-value-tuples. For example: + ``` python checksums = [{ - 'src_x86_64.tgz': 'f962008105639f58e9a4455c8057933ab0a5e2f43db8340ae1e1afe6dc2d2410', - 'src_aarch64.tgz': ('size', 42), + 'src_x86_64.tgz': '0123456789...abcdef', + 'src_aarch64.tgz': ('sha256', 'fedcba...9876543210'), }] ``` Of course this can be combined with the logical AND/OR semantics using lists or tuples: + ``` python checksums = [{ - 'src_x86_64.tgz': ('oldchecksum', 'newchecksum'), - 'src_aarch64.tgz': [('size', 42), 'checksumthatmustmatchtoo'], + 'src_x86_64.tgz': ('0123456789...abcdef', 'fedcba...9876543210'), # Match either one + 'src_aarch64.tgz': [('sha256', '9876543210...fedcba'), 'abcdef...0123456789'], # Match both }] ``` When the checksum cannot be specified for a file (e.g. when using a git clone instead of an archive), a value of `None` can be used to skip the checksum check. -This is possible in the list of checksums as well as as a value in a dictionary, e.g.: +This is possible in the list of checksums as well as a value in a dictionary, e.g.: + ``` python checksums = [ None, # No checksum for first source file - 'checksumfor2ndfile', + '0123456789...abcdef', # checksum for 2nd file { - 'third_file_x86_64.tgz': 'checksum', + 'third_file_x86_64.tgz': 'fedcba...9876543210', 'third_file_aarch64.tgz': None, }, ] @@ -538,7 +542,7 @@ cannot be automated. Reasons for this include: You can use the `download_instructions` parameter to specify steps for the user to do. This parameter takes string value and prints it whenever build fails because any file needed was not found. If -`download_instructions` is not specified, Easybuild prints the default +`download_instructions` is not specified, EasyBuild prints the default message stating the paths that were tested. ``` python From d2c645d102c728d008cff148584ee70baab90557 Mon Sep 17 00:00:00 2001 From: Alexander Grund Date: Thu, 16 Nov 2023 09:36:08 +0100 Subject: [PATCH 4/4] Document the importance of `None` as a dict value --- docs/writing-easyconfig-files.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/writing-easyconfig-files.md b/docs/writing-easyconfig-files.md index 89161935a..82c59c9e5 100644 --- a/docs/writing-easyconfig-files.md +++ b/docs/writing-easyconfig-files.md @@ -312,9 +312,9 @@ checksums = [ ] ``` -The difference between having an entry in the dict with the value of `None` and not having an entry -only matters when using the `--enforce_checksums` option which will raise an error in the latter case. - +Note that not having an entry in the dict for a file will raise an error +while a value of `None` will skip the checksum verification for that file. +But even in the latter case the `--enforce_checksums` option will raise an error. ##### Adding or replacing checksums using `--inject-checksums` {: #inject_checksums }