Merge branch 'dev' into fix-de-tutorial

.github/workflows/ci.yml

@@ -51,7 +51,6 @@ jobs:
       matrix:
         NXF_VER:
           - "24.04.2"
-          - "latest-everything"
         nf_test_files: ["${{ fromJson(needs.nf-test-changes.outputs.nf_test_files) }}"]
         profile:
           - "docker"

CHANGELOG.md

@@ -3,19 +3,52 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-# 3.18.0dev - xxxx-xx-xx
+# 3.19.0dev - xxxx-xx-xx
+
+### Credits
+
+### Enhancements & fixes
+
+- [PR #1480](https://github.com/nf-core/rnaseq/pull/1480) - Bump version after release 3.18.0
+- [PR #1482](https://github.com/nf-core/rnaseq/pull/1482) - Update trimgalore module for save_unpaired fix
+- [PR #1486](https://github.com/nf-core/rnaseq/pull/1486) - Bump STAR build for multiprocessing fix
+- [PR #1490](https://github.com/nf-core/rnaseq/pull/1490) - Make genomic FASTA input optional
+
+# 3.18.0 - 2024-12-19
 
 ### Credits
 
 Special thanks to the following for their contributions to the release:
 
 - [Caitlin Winkler](https://github.com/oligomyeggo)
+- [Jonathan Manning](https://github.com/pinin4fjords)
+- [Lorenzo Sola](https://github.com/LorenzoS96)
+- [Maxime Garcia](https://github.com/maxulysse)
 - [Siddhartha Bagaria](https://github.com/siddharthab)
 
 ### Enhancements & fixes
 
 - [PR #1369](https://github.com/nf-core/rnaseq/pull/1369) - Add umicollapse as an alternative to umi-tools
 - [PR #1461](https://github.com/nf-core/rnaseq/pull/1461) - Add FASTQ linting during preprocessing
+- [PR #1463](https://github.com/nf-core/rnaseq/pull/1463) - Move channel operations outside of the onComplete() block
+- [PR #1467](https://github.com/nf-core/rnaseq/pull/1467) - Add test suite for UMI handling functionality
+- [PR #1466](https://github.com/nf-core/rnaseq/pull/1466) - Factor out UMI handling
+- [PR #1470](https://github.com/nf-core/rnaseq/pull/1470) - Update subworkflow to account for fix to bad argument handling
+- [PR #1469](https://github.com/nf-core/rnaseq/pull/1469) - Minor docs fix
+- [PR #1459](https://github.com/nf-core/rnaseq/pull/1466) - Remove reference to unused "skip_sample_count" value in email templates
+- [PR #1471](https://github.com/nf-core/rnaseq/pull/1471) - Fix prepare_genome subworkflow for sortmerna
+- [PR #1473](https://github.com/nf-core/rnaseq/pull/1473) - Bump STAR modules
+- [PR #1474](https://github.com/nf-core/rnaseq/pull/1474) - Bump versions to 3.18.0
+- [PR #1475](https://github.com/nf-core/rnaseq/pull/1475) - Fix log publishing around umitools/ umicollapse
+- [PR #1447](https://github.com/nf-core/rnaseq/pull/1447) - Add tutorial series for analysing count data
+
+## Parameters
+
+| Old parameter | New parameter         |
+| ------------- | --------------------- |
+|               | `--skip_linting`      |
+|               | `--extra_fqlint_args` |
+|               | `--umi_dedup_tool`    |
 
 ### Software dependencies
 

assets/email_template.html

@@ -34,25 +34,6 @@ <h4 style="margin-top: 0; color: inherit">nf-core/rnaseq execution completed uns
         <p>The full error message was:</p>
         <pre style="white-space: pre-wrap; overflow: visible; margin-bottom: 0">${errorReport}</pre>
       </div>
-      """ } else if(skip_sample_count > 0) { out << """
-      <div
-        style="
-          color: #856404;
-          background-color: #fff3cd;
-          border-color: #ffeeba;
-          padding: 15px;
-          margin-bottom: 20px;
-          border: 1px solid transparent;
-          border-radius: 4px;
-        "
-      >
-        <h4 style="margin-top: 0; color: inherit">nf-core/rnaseq execution completed with warnings!</h4>
-        <p>
-          The pipeline finished successfully, but samples were skipped. Please check warnings at the top of the MultiQC report.
-        </p>
-        <p></p>
-      </div>
-
       """ } else { out << """
       <div
         style="

assets/email_template.txt

@@ -17,13 +17,6 @@ The full error message was:
 
 ${errorReport}
 """
-} else if (skip_sample_count > 0) {
-    out << """##################################################
-## nf-core/rnaseq execution completed with warnings ##
-##################################################
-The pipeline finished successfully, but samples were skipped.
-Please check warnings at the top of the MultiQC report.
-"""
 } else {
     out << "## nf-core/rnaseq execution completed successfully! ##"
 }

bin/filter_gtf.py

@@ -6,7 +6,7 @@
 import argparse
 import re
 import statistics
-from typing import Set
+from typing import Optional, Set
 
 # Create a logger
 logging.basicConfig(format="%(name)s - %(asctime)s %(levelname)s: %(message)s")
@@ -27,14 +27,15 @@ def tab_delimited(file: str) -> float:
         return statistics.median(line.count("\t") for line in data.split("\n"))
 
 
-def filter_gtf(fasta: str, gtf_in: str, filtered_gtf_out: str, skip_transcript_id_check: bool) -> None:
+def filter_gtf(fasta: Optional[str], gtf_in: str, filtered_gtf_out: str, skip_transcript_id_check: bool) -> None:
     """Filter GTF file based on FASTA sequence names."""
     if tab_delimited(gtf_in) != 8:
         raise ValueError("Invalid GTF file: Expected 9 tab-separated columns.")
 
-    seq_names_in_genome = extract_fasta_seq_names(fasta)
-    logger.info(f"Extracted chromosome sequence names from {fasta}")
-    logger.debug("All sequence IDs from FASTA: " + ", ".join(sorted(seq_names_in_genome)))
+    if (fasta is not None):
+        seq_names_in_genome = extract_fasta_seq_names(fasta)
+        logger.info(f"Extracted chromosome sequence names from {fasta}")
+        logger.debug("All sequence IDs from FASTA: " + ", ".join(sorted(seq_names_in_genome)))
 
     seq_names_in_gtf = set()
     try:
@@ -44,7 +45,7 @@ def filter_gtf(fasta: str, gtf_in: str, filtered_gtf_out: str, skip_transcript_i
                 seq_name = line.split("\t")[0]
                 seq_names_in_gtf.add(seq_name)  # Add sequence name to the set
 
-                if seq_name in seq_names_in_genome:
+                if fasta is None or seq_name in seq_names_in_genome:
                     if skip_transcript_id_check or re.search(r'transcript_id "([^"]+)"', line):
                         out.write(line)
                         line_count += 1
@@ -63,7 +64,7 @@ def filter_gtf(fasta: str, gtf_in: str, filtered_gtf_out: str, skip_transcript_i
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Filters a GTF file based on sequence names in a FASTA file.")
     parser.add_argument("--gtf", type=str, required=True, help="GTF file")
-    parser.add_argument("--fasta", type=str, required=True, help="Genome fasta file")
+    parser.add_argument("--fasta", type=str, required=False, help="Genome fasta file")
     parser.add_argument("--prefix", dest="prefix", default="genes", type=str, help="Prefix for output GTF files")
     parser.add_argument(
         "--skip_transcript_id_check", action="store_true", help="Skip checking for transcript IDs in the GTF file"

conf/arm.config

@@ -121,11 +121,11 @@ process {
     }
 
     withName: 'STAR_GENOMEGENERATE' {
-        container = { workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/a2/a2d5226e4ce3dee8b29154c16a87d282d96c76e75b6678d032643902591586e2/data' : 'community.wave.seqera.io/library/htslib_samtools_star_gawk:1d1b7da208684cac' }
+        container = { workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/40/40d803371e50330de0773c7cc50315e2c3b4b41dcf123823adeb0a07d71654c1/data' : 'community.wave.seqera.io/library/htslib_samtools_star_gawk:ae438e9a604351a4' }
     }
 
     withName: 'STAR_ALIGN' {
-        container = { workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/a2/a2d5226e4ce3dee8b29154c16a87d282d96c76e75b6678d032643902591586e2/data' : 'community.wave.seqera.io/library/htslib_samtools_star_gawk:1d1b7da208684cac' }
+        container = { workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/40/40d803371e50330de0773c7cc50315e2c3b4b41dcf123823adeb0a07d71654c1/data' : 'community.wave.seqera.io/library/htslib_samtools_star_gawk:ae438e9a604351a4' }
     }
 
     withName: 'TXIMETA_TXIMPORT' {

docs/output.md

@@ -74,7 +74,7 @@ The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes d
 
 If multiple libraries/runs have been provided for the same sample in the input samplesheet (e.g. to increase sequencing depth) then these will be merged at the very beginning of the pipeline in order to have consistent sample naming throughout the pipeline. Please refer to the [usage documentation](https://nf-co.re/rnaseq/usage#samplesheet-input) to see how to specify these samples in the input samplesheet.
 
-# fq lint
+### fq lint
 
 <details markdown="1">
 <summary>Output files</summary>
@@ -120,7 +120,7 @@ If multiple libraries/runs have been provided for the same sample in the input s
 
 </details>
 
-[UMI-tools](https://github.com/CGATOxford/UMI-tools) deduplicates reads based on unique molecular identifiers (UMIs) to address PCR-bias. Firstly, the UMI-tools `extract` command removes the UMI barcode information from the read sequence and adds it to the read name. Secondly, reads are deduplicated based on UMI identifier after mapping as highlighted in the [UMI-tools dedup](#umi-tools-dedup) section.
+[UMI-tools](https://github.com/CGATOxford/UMI-tools) and [UMICollapse](https://github.com/Daniel-Liu-c0deb0t/UMICollapse) deduplicate reads based on unique molecular identifiers (UMIs) to address PCR-bias. Firstly, the UMI-tools `extract` command removes the UMI barcode information from the read sequence and adds it to the read name. Secondly, reads are deduplicated based on UMI identifier after mapping as highlighted in the [UMI dedup](#umi-dedup) section.
 
 To facilitate processing of input data which has the UMI barcode already embedded in the read name from the start, `--skip_umi_extract` can be specified in conjunction with `--with_umi`.
 
@@ -305,7 +305,7 @@ The original BAM files generated by the selected alignment algorithm are further
 
 ![MultiQC - SAMtools mapped reads per contig plot](images/mqc_samtools_idxstats.png)
 
-### UMI-tools dedup
+### UMI dedup
 
 <details markdown="1">
 <summary>Output files</summary>
@@ -314,7 +314,7 @@ The original BAM files generated by the selected alignment algorithm are further
   - `<SAMPLE>.umi_dedup.sorted.bam`: If `--save_umi_intermeds` is specified the UMI deduplicated, coordinate sorted BAM file containing read alignments will be placed in this directory.
   - `<SAMPLE>.umi_dedup.sorted.bam.bai`: If `--save_umi_intermeds` is specified the BAI index file for the UMI deduplicated, coordinate sorted BAM file will be placed in this directory.
   - `<SAMPLE>.umi_dedup.sorted.bam.csi`: If `--save_umi_intermeds --bam_csi_index` is specified the CSI index file for the UMI deduplicated, coordinate sorted BAM file will be placed in this directory.
-- `<ALIGNER>/umitools/`
+- `<ALIGNER>/umitools/` (UMI-tools only)
   - `*_edit_distance.tsv`: Reports the (binned) average edit distance between the UMIs at each position.
   - `*_per_umi.tsv`: UMI-level summary statistics.
   - `*_per_umi_per_position.tsv`: Tabulates the counts for unique combinations of UMI and position.
@@ -323,7 +323,7 @@ The content of the files above is explained in more detail in the [UMI-tools doc
 
 </details>
 
-After extracting the UMI information from the read sequence (see [UMI-tools extract](#umi-tools-extract)), the second step in the removal of UMI barcodes involves deduplicating the reads based on both mapping and UMI barcode information using the UMI-tools `dedup` command. This will generate a filtered BAM file after the removal of PCR duplicates.
+After extracting the UMI information from the read sequence (see [UMI-tools extract](#umi-tools-extract)), the second step in the removal of UMI barcodes involves deduplicating the reads based on both mapping and UMI barcode information. UMI deduplication can be carried out either with [UMI-tools](https://github.com/CGATOxford/UMI-tools) or [UMICollapse](https://github.com/Daniel-Liu-c0deb0t/UMICollapse), set via the `umi_dedup_tool` parameter. The output BAM files are the same, though UMI-tools has some additional outputs, as described above. Either method will generate a filtered BAM file after the removal of PCR duplicates.
 
 ### picard MarkDuplicates
 

docs/usage.md

@@ -132,7 +132,7 @@ You also have the option to pseudoalign and quantify your data directly with [Sa
 
 The library preparation protocol (library type) used by Salmon quantification is inferred by the pipeline based on the information provided in the samplesheet, however, you can override it using the `--salmon_quant_libtype` parameter. You can find the available options in the [Salmon documentation](https://salmon.readthedocs.io/en/latest/library_type.html). Similarly, strandedness is taken from the sample sheet or calculated automatically, and passed to Kallisto on a per-library basis, but you can apply a global override by setting the Kallisto strandedness parameters in `--extra_kallisto_quant_args` like `--extra_kallisto_quant_args '--fr-stranded'` see the [Kallisto documentation](https://pachterlab.github.io/kallisto/manual).
 
-When running Salmon in mapping-based mode via `--pseudo_aligner salmon` the entire genome of the organism is used by default for the decoy-aware transcriptome when creating the indices (see second bulleted option in [Salmon documentation](https://salmon.readthedocs.io/en/latest/salmon.html#preparing-transcriptome-indices-mapping-based-mode)).
+When running Salmon in mapping-based mode via `--pseudo_aligner salmon`, supplying a genome fasta via `--fasta` and not supplying a Salmon index, the entire genome of the organism is used by default for the decoy-aware transcriptome when creating the indices, as is recommended (see second bulleted option in [Salmon documentation](https://salmon.readthedocs.io/en/latest/salmon.html#preparing-transcriptome-indices-mapping-based-mode)). If you do not supply a FASTA file or an index, Salmon will index without those decoys, using only transcript sequences in the index. This second option is not usually recommended, but may be useful in limited circumstances. Note that Kallisto does not index with genomic sequences.
 
 Two additional parameters `--extra_star_align_args` and `--extra_salmon_quant_args` were added in v3.10 of the pipeline that allow you to append any custom parameters to the STAR align and Salmon quant commands, respectively. Note, the `--seqBias` and `--gcBias` are not provided to Salmon quant by default so you can provide these via `--extra_salmon_quant_args '--seqBias --gcBias'` if required. You can now also supply additional arguments to Kallisto via `--extra_kallisto_quant_args`.
 
@@ -209,7 +209,7 @@ When supplying reference files as discussed below, it is important to be consist
 
 ### Explicit reference file specification (recommended)
 
-The minimum reference genome requirements for this pipeline are a FASTA and GTF file, all other files required to run the pipeline can be generated from these files. For example, the latest reference files for human can be derived from Ensembl like:
+The minimum reference genome requirements for this pipeline are a FASTA file (genome and/ or transcriptome) and GTF file, all other files required to run the pipeline can be generated from these files. For example, the latest reference files for human can be derived from Ensembl like:
 
 ```
 latest_release=$(curl -s 'http://rest.ensembl.org/info/software?content-type=application/json' | grep -o '"release":[0-9]*' | cut -d: -f2)
@@ -227,6 +227,7 @@ Notes:
 - If `--gene_bed` is not provided then it will be generated from the GTF file.
 - If `--additional_fasta` is provided then the features in this file (e.g. ERCC spike-ins) will be automatically concatenated onto both the reference FASTA file as well as the GTF annotation before building the appropriate indices.
 - When using `--aligner star_rsem`, both the STAR and RSEM indices should be present in the path specified by `--rsem_index` (see [#568](https://github.com/nf-core/rnaseq/issues/568)).
+- If the `--skip_alignment` option is used along with `--transcript_fasta`, the pipeline can technically run without providing the genomic FASTA (`--fasta`). However, this approach is **not recommended** with `--pseudo_aligner salmon`, as any dynamically generated Salmon index will lack decoys. To ensure optimal indexing with decoys, it is **highly recommended** to include the genomic FASTA (`--fasta`) with Salmon, unless a pre-existing decoy-aware Salmon index is supplied. For more details on the benefits of decoy-aware indexing, refer to the [Salmon documentation](https://salmon.readthedocs.io/en/latest/salmon.html#preparing-transcriptome-indices-mapping-based-mode).
 
 #### Reference genome
 
@@ -304,7 +305,7 @@ Notes:
 
 ### GTF filtering
 
-By default, the input GTF file will be filtered to ensure that sequence names correspond to those in the genome fasta file, and to remove rows with empty transcript identifiers. Filtering can be bypassed completely where you are confident it is not necessary, using the `--skip_gtf_filter` parameter. If you just want to skip the 'transcript_id' checking component of the GTF filtering script used in the pipeline this can be disabled specifically using the `--skip_gtf_transcript_filter` parameter.
+By default, the input GTF file will be filtered to ensure that sequence names correspond to those in the genome fasta file (where supplied), and to remove rows with empty transcript identifiers. Filtering can be bypassed completely where you are confident it is not necessary, using the `--skip_gtf_filter` parameter. If you just want to skip the 'transcript_id' checking component of the GTF filtering script used in the pipeline this can be disabled specifically using the `--skip_gtf_transcript_filter` parameter.
 
 ## Contamination screening options
 
@@ -332,6 +333,21 @@ nextflow run \
     -profile docker
 ```
 
+You can also run without a genomic FASTA file, provided you skip the alignment step and provide a transcriptome FASTA directly:
+
+```bash
+nextflow run \
+    nf-core/rnaseq \
+    --input <SAMPLESHEET> \
+    --outdir <OUTDIR> \
+    --gtf <GTF> \
+    --transcript_fasta <TRANSCRIPTOME FASTA> \
+    --skip_alignment \
+    -profile docker
+```
+
+This is not usually recommended with Salmon unless you also supply a previously generated decoy-aware Salmon transcriptome index.
+
 > **NB:** Loading iGenomes configuration remains the default for reasons of consistency with other workflows, but should be disabled when not using iGenomes, applying the recommended usage above.
 
 This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles.