Note that I'm aware of publishDir, but that creates a symbolic link in the publishDir to the real file within workDir. I don't want a symlink to workDir because that link will be broken if workDir gets cleaned.

Note that I'm aware of publishDir, but that creates a symbolic link in the publishDir to the real file within workDir. I don't want a symlink to workDir because that link will be broken if workDir gets cleaned.

Hello, @mcallaway. You can set publishDir to move or copy files, instead of symlinking them. You can read more about it here: https://www.nextflow.io/docs/latest/process.html#publishdir

Snippet below:

process foo {
    publishDir '/data/chunks', mode: 'copy', overwrite: false

    output:
    path 'chunk_*'

    '''
    printf 'Hola' | split -b 1 - chunk_
    '''
}

@bentsherman and @pditommaso - storeDir is a great feature yet it took me a long time to figure out why it was skipping a process and eventually figure out it was the optional output. If this cannot be fixed (which I hope it can!), is it possible to add a warning that storeDir does not work for optional outputs? Thanks for your consideration!

I think it is a fundamental limitation of storeDir, because there is no cache metadata to verify whether the optional output should be there from a previous run. The same is true for an output with a variable number of files, as there is no way to verify the number of files produced by a previous run.

For now I think it's worth documenting these limitations. We are investigating some ideas that I think could replace storeDir in the long-term.

The best workaround that I can think of is that a process that uses storeDir should always have at least one non-optional file output. This will guarantee that the process is executed at least once. You might have to store a dummy output file to make it work.

I believe there are also cases where a process has multiple optional outputs, but in practice at least one of those outputs is expected to be present. That is more of a modeling problem that we need to solve in the language. For example, instead of having two optional outputs for BAM and CRAM, there should be one required output that is somehow modeled as "either BAM or CRAM". Invalid states should be unrepresentable