Skip to content

Commit

Permalink
Merge pull request #82 from rabix/feature/wrabbit
Browse files Browse the repository at this point in the history 
Feature/wrabbit
  • Loading branch information
@pavlemarinkovic
pavlemarinkovic authored May 14, 2024
2 parents a016df2 + 55ab5c2 commit a47bd01
Show file tree
Hide file tree
Showing 13 changed files with 569 additions and 1,275 deletions.
1 change: 1 addition & 0 deletions requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
ruamel.yaml >= 0.16
sevenbridges-python >= 2.0
nf-core==2.1
wrabbit==0.1.1
cwlformat
packaging
90 changes: 83 additions & 7 deletions sbpack/noncwl/Readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,21 @@ $ sbpack_nf -h
usage: sbpack_nf [-h]
[--profile PROFILE]
--appid APPID
[--appid APPID]
--workflow-path WORKFLOW_PATH
[--entrypoint ENTRYPOINT]
[--sb-package-id SB_PACKAGE_ID]
[--sb-doc SB_DOC]
[--dump-sb-app]
[--no-package]
[--executor-version EXECUTOR_VERSION]
[--execution-mode {single-instance,multi-instance}]
[--json]
[--sb-schema SB_SCHEMA]
[--revision-note REVISION_NOTE]
[--app-name APP_NAME]
[--exclude EXCLUDE [EXCLUDE ...]]
[--sample-sheet-schema SAMPLE_SHEET_SCHEMA]
[--auto]
optional arguments:
-h, --help show this help message and exit
Expand All @@ -79,24 +82,41 @@ optional arguments:
than one '*.nf' script is detected, an error is
raised.
--sb-package-id SB_PACKAGE_ID
Id of an already uploaded package
Id of an already uploaded package.
--sb-doc SB_DOC Path to a doc file for sb app. If not provided,
README.md will be used if available
--dump-sb-app Dump created sb app to file if true and exit
--no-package Only provide a sb app schema and a git URL for
entrypoint
--executor-version EXECUTOR_VERSION
Version of the Nextflow executor to be used with the
app.
--execution-mode {single-instance,multi-instance}
Execution mode for your application.
Execution mode for your application. Can be multi-
instance or single-instance
--json Dump sb app schema in JSON format (YAML by default)
--sb-schema SB_SCHEMA
Do not create new schema, use this schema file. It is
sb_nextflow_schema in JSON or YAML format.
--revision-note REVISION_NOTE
Revision note to be placed in the CWL schema if the
app is uploaded to the sbg platform.
--app-name APP_NAME Name of the app to be shown on the platform.
--exclude EXCLUDE [EXCLUDE ...]
Glob patterns you want to exclude from the code
package. By default the following patterns are
excluded: ['*.git', '*.git*', '.git', '.git*', 'work',
'.nextflow.log', '.DS_Store', '.devcontainer',
'.editorconfig', '.gitattributes', '.nextflow', '.pre-
commit-config.yaml', '.prettierignore',
'.prettierrc.yml', '.idea', '.pytest_cache', '*.egg-
info']
--sample-sheet-schema SAMPLE_SHEET_SCHEMA
Path to the sample sheet schema yaml. The sample sheet
schema should contain the following keys:
'sample_sheet_input', 'sample_sheet_name', 'header',
'rows', 'defaults', 'group_by', 'format_'
--auto Automatically detect all possible inputs directly from
the --workflow-path location
```
### Example

Expand Down Expand Up @@ -147,4 +167,60 @@ sbcopy --appid division-name/project-name/app-name --projectid division-name/des
Copying an app from one division to another.
```
sbcopy --profile source_division target_division --appid source-division-name/project-name/app-name --projectid target-division-name/destination-project-name
```
```

# sbmanifest

Developed to help remap file paths contained within a manifest file to platform
file locations. This is useful when utilizing sample sheets to run nextflow
pipelines on SB powered platforms.

### Usage

```
usage: sbmanifest [-h] [--profile PROFILE] --projectid PROJECTID
--sample-sheet SAMPLE_SHEET --columns string [string ...]
[--output OUTPUT] [--upload] [--tags string [string ...]]
[--validate]
optional arguments:
-h, --help show this help message and exit
--profile PROFILE SB platform profile as set in the SB API credentials
file.
--projectid PROJECTID
Takes the form {user or division}/{project}.
--sample-sheet SAMPLE_SHEET
Path to the sample sheet.
--columns string [string ...]
Specify columns that contain paths to files on the
platformas a list of strings separated by spaces.
--output OUTPUT, -o OUTPUT
Name of the output file.
--upload, -u Upload the file to the project after making it.
--tags string [string ...]
Specify tags that you want the sample sheet to have on
the platform, after it is uploaded.
--validate Validate if each file exists on target project
location.
```

### Examples

Remap a sample sheet and upload it to the platform with the tag "SampleSheet"
```
sbmanifest --projectid user/project_id --sample-sheet /path/to/rnaseq_samplesheet.csv --columns fastq_1 fastq_2 -o rnaseq_samplesheet.csv -u --tags SampleSheet
```
Given the contents of this sample sheet is:

| sample | fastq_1 | fastq_2 | strandedness |
|:--------|:---------------------------------|:---------------------------------|:-------------|
| SAMPLE1 | RNAseq_inputs/SAMPLE1_1.fastq.gz | RNAseq_inputs/SAMPLE1_2.fastq.gz | reverse |
| SAMPLE2 | RNAseq_inputs/SAMPLE2_1.fastq.gz | RNAseq_inputs/SAMPLE2_2.fastq.gz | reverse |


Remapped file will be:

| sample | fastq_1 | fastq_2 | strandedness |
|:--------|:------------------------------------------------------------------|:------------------------------------------------------------------|:-------------|
| SAMPLE1 | vs:///Projects/project-root-uuid/RNAseq_inputs/SAMPLE1_1.fastq.gz | vs:///Projects/project-root-uuid/RNAseq_inputs/SAMPLE1_2.fastq.gz | reverse |
| SAMPLE2 | vs:///Projects/project-root-uuid/RNAseq_inputs/SAMPLE2_1.fastq.gz | vs:///Projects/project-root-uuid/RNAseq_inputs/SAMPLE2_2.fastq.gz | reverse |
9 changes: 0 additions & 9 deletions sbpack/noncwl/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,9 +0,0 @@
import os

PACKAGE_PATH = os.path.abspath(os.path.dirname(__file__))
JS_PATH = os.path.join(PACKAGE_PATH, 'js_templates')


def read_js_template(file_name):
with open(os.path.join(JS_PATH, file_name), 'r') as f:
return f.read()
99 changes: 23 additions & 76 deletions sbpack/noncwl/constants.py
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
from enum import Enum
from sbpack.noncwl import read_js_template


# ############################## Generic Bits ############################### #
PACKAGE_SIZE_LIMIT = 100 * 1024 * 1024 # 100 MB
REMOVE_INPUT_KEY = "REMOVE_THIS_KEY"
Expand All @@ -22,8 +18,6 @@ class EXTENSIONS:
# ############################ CWL Standard Bits ############################ #
# A generic SB input array of files that should be available on the
# instance but are not explicitly provided to the execution as wdl params.
SAMPLE_SHEET_FUNCTION = read_js_template("sample_sheet_generator.js")
SAMPLE_SHEET_SWITCH = read_js_template("sample_sheet_switch.js")

GENERIC_FILE_ARRAY_INPUT = {
"id": "auxiliary_files",
Expand All @@ -41,18 +35,6 @@ class EXTENSIONS:
"that is required for workflow execution."
}

GENERIC_NF_OUTPUT_DIRECTORY = {
"id": "nf_workdir",
"type": "Directory?",
"label": "Work Directory",
"doc": "This is a template output. "
"Please change glob to directories specified in "
"publishDir in the workflow.",
"outputBinding": {
"glob": "work"
}
}

GENERIC_WDL_OUTPUT_DIRECTORY = {
"id": "output_txt",
"doc": "This is a template output. "
Expand Down Expand Up @@ -84,62 +66,27 @@ class EXTENSIONS:
AUX_FILES_REQUIREMENT
]


def sample_sheet(
file_name, sample_sheet_input, format_, input_source, header, rows,
defaults, group_by):
basename = ".".join(file_name.split(".")[:-1])
ext = file_name.split(".")[-1]
new_name = f"{basename}.new.{ext}"

return {
"class": "InitialWorkDirRequirement",
"listing": [
{
"entryname": f"${{ return {sample_sheet_input} ? {sample_sheet_input}.nameroot + '.new' + {sample_sheet_input}.nameext : '{file_name}' }}",
"entry": SAMPLE_SHEET_FUNCTION.format_map(locals()),
"writable": False
}
]
}


# ############################## Nextflow Bits ############################## #
# Keys that should be skipped when parsing nextflow tower yaml file

NF_SCHEMA_DEFAULT_NAME = 'nextflow_schema.json'
SB_SCHEMA_DEFAULT_NAME = 'sb_nextflow_schema'

# Mappings of nextflow input fields to SB input fields
# nextflow_key: cwl_key mapping
NF_TO_CWL_PORT_MAP = {
'default': 'sbg:toolDefaultValue',
'description': 'label',
'help_text': 'doc',
'mimetype': 'format',
'fa_icon': 'sbg:icon',
'pattern': 'sbg:pattern',
'hidden': 'sbg:hidden',
}

# Mappings of nextflow definition fields to SB category fields
# nextflow_key: cwl_key mapping
NF_TO_CWL_CATEGORY_MAP = {
'title': 'sbg:title',
'description': 'sbg:doc',
'fa_icon': 'sbg:icon',
}

# What keys to skip from the tower.yml file
SKIP_NEXTFLOW_TOWER_KEYS = [
'tower',
'mail',
# Nextflow
DEFAULT_EXCLUDE_PATTERNS = [
"*.git",
"*.git*",
".git",
".git*",
# ".github",
# ".gitignore",
# ".gitpod.yml",
"work",
".nextflow.log",
".DS_Store",
".devcontainer",
".editorconfig",
".gitattributes",
".nextflow",
# ".nf-core.yml",
".pre-commit-config.yaml",
".prettierignore",
".prettierrc.yml",
".idea",
".pytest_cache",
"*.egg-info",
]


class ExecMode(Enum):
single = 'single-instance'
multi = 'multi-instance'

def __str__(self):
return self.value
Loading

0 comments on commit a47bd01

Please sign in to comment.