Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to the file method to checksum #4415

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jordeu
Copy link
Collaborator

@jordeu jordeu commented Oct 16, 2023

Description

Adds the checksum option to the file method that allows to verify that a downloaded file at the stage directory is valid.

Notes:

  • this is only verified the first time that the file is downloaded into the local stage folder
  • this is only supported by XPath filesystem (HTTP, HTTPS and FTP protocols).

Guidelines for testing

  1. Checkout this branch
  2. make clean compile
  3. Run ./launch.sh run jordeu/nf-tests -r checksum (pipeline should success)
  4. Run ./launch.sh run jordeu/nf-tests -r checksum --md5 8ddd8be4b179a529afa5f2ffae4b98 (pipeline should fail).

Example fail output:

$ ./launch.sh run jordeu/nf-tests -r checksum --md5 8ddd8be4b179a529afa5f2ffae4b98
N E X T F L O W  ~  version 23.10.0
Launching `https://github.com/jordeu/nf-tests` [jolly_pare] DSL2 - revision: 81ac09fbc7 [checksum]
[-        ] process > MD5      -
[-        ] process > SHA1 (1) -
[-        ] process > SHA256   [  0%] 0 of 1
ERROR ~ Error executing process > 'MD5 (1)'

Caused by:
  Invalid checksum at /jordeu/nf-tests/checksum/hello.md5


 -- Check '.nextflow.log' file for details

When it fails you can check the computed checksums at the logs:

$ cat .nextflow.log | grep "Expected checksum"
Oct-17 15:31:45.429 [FileTransfer-2] DEBUG nextflow.file.FileHelper - Expected checksum 'sha1::a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b' matches computed one
Oct-17 15:31:45.497 [FileTransfer-3] DEBUG nextflow.file.FileHelper - Expected checksum '8ddd8be4b179a529afa5f2ffae4b98' found '8ddd8be4b179a529afa5f2ffae4b9858'
Oct-17 15:31:45.548 [FileTransfer-1] DEBUG nextflow.file.FileHelper - Expected checksum 'sha256::03ba204e50d126e4674c005e04d82e84c21366780af1f43bd54a37816b6ab340' matches computed one

Closes #2491

@netlify
Copy link

netlify bot commented Oct 16, 2023

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 30163c7
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/652d5c790494390008c88580

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add an option to the file method to check for md5sum
1 participant