Resolve Fusion symlinks when publishing files

[bentsherman]

Close #4309

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	eb24f72
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6552821b7268cb000860cbf8

[bentsherman]

I updated Jordi's example script as follows:

nextflow.enable.dsl=2

process CREATE {

    output:
    path "data.txt"

    script:
    """
    echo HELLO > data.txt
    """
}

process FORWARD {

    input:
    path "data.txt"

    output:
    path "data.txt"

    script:
    """
    echo AND >> data.txt
    """
}

process PUBLISH {
    publishDir "${params.outdir}"

    input:
    path "data.txt"

    output:
    path "data.txt"

    script:
    '''
    echo BYE >> data.txt
    '''
}

workflow {
    CREATE | FORWARD | PUBLISH
}

And I found that the output file in PUBLISH links to the output from FORWARD rather than CREATE. I will try to fix that in Nextflow.

[bentsherman]

Actually, it might not be so simple. When an input is forwarded as an output multiple times, there are conflicting needs:

• linking to the original file regardless of the degrees of separation is nice to avoid multiple levels of links
• however, these links are needed to determine task provenance (and the automatic work directory cleanup that is WIP)

If we can find a way to resolve the actual symlinks but preserve Nextflow's metadata, then maybe we can have it both ways. For example, we could add a field to FileHolder e.g. originalStorePath that points to the original file regardless of intermediate links.

[bentsherman]

I worked around the previous problem by using the .fusion.symlinks files. However there is one more issue:

nextflow/modules/nextflow/src/main/groovy/nextflow/processor/PublishDir.groovy

Line 329 in 945c01f

def target = sourceDir ? sourceDir.relativize(source) : source.getFileName()

This line does not work with the resolved Fusion symlink, because now the resolved file is not in the current task directory. So instead of publishing to results/data.txt it goes to something like results/dc/14a14e8ef9bc91cbf92e14e33715da/data.txt.

I think this line is meant to deal with files that are in a subdirectory of the task directory, I should be able to fix it by resolving the Fusion symlink after this line.

[bentsherman]

PublishOpTest is failing, but since PublishOp is not used and we decided not to make a publish operator, maybe it's better to delete

[jordeu]

We have some open tickets related to this problem. I'd be good to make it progress to master.

[pditommaso]

Unit tests should be added for key logic

[bentsherman]

Is there a simple way to mock the S3 interactions here? Or do you want to use real S3 objects?

[marcodelapierre]

note: also backporting to latest stable

[pditommaso]

Regex "compilation" can be slow. Moved into a separate const, so it's parsed only the very first time 👉 0d853ea

[pditommaso]

The test is failing

[pditommaso]

Still failing

[bentsherman]

I think the integration tests are not set up to use Fusion, seems like AWS credentials are not provided

[pditommaso]

Umm, this it should be added exportStorageCredentials (good opportunity to update here as well)

https://github.com/nextflow-io/nextflow/blob/816216614dd90b83a0261192729fccf85fc55e76/validation/wave-tests/example6/nextflow.config#L1-L21