Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cactus-pangenome error in clip_vg step (gfaffix?) #1573

Open
dirkjanvw opened this issue Dec 30, 2024 · 2 comments
Open

cactus-pangenome error in clip_vg step (gfaffix?) #1573

dirkjanvw opened this issue Dec 30, 2024 · 2 comments

Comments

@dirkjanvw
Copy link

I'm running cactus-pangenome (using the docker image quay.io/comparative-genomics-toolkit/cactus:v2.9.3-gpu pulled as singularity container) as follows:

cactus-pangenome results/2.graph_construction/minigraph-cactus/Lsativa/mc-pangenome/js results/2.graph_construction/minigraph-cactus/Lsativa/mc-pangenome.list --reference Salinas_V11 --vcfReference Salinas_V11 --outDir results/2.graph_construction/minigraph-cactus/Lsativa/mc-pangenome --outName mc-pangenome --vcf --giraffe --gbz --xg --gfa --odgi --chrom-vg --chrom-og --mgCores 50 --mapCores 50 --consCores 50 --indexCores 50

But it fails at the clip_vg stage. The following is what it logs (it also spits out some thousands of GFA lines which I leave out):

[2024-12-27T16:43:55+0100] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker clip_vg file:/path/to/results/2.graph_construction/minigraph-cactus/Lsativa/mc-pangenome/js kind-clip_vg/instance-pfdwoyly.
[2024-12-27T16:43:55+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'clip_vg' kind-clip_vg/instance-pfdwoyly v1
Exit reason: None
[2024-12-27T16:43:55+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'clip_vg' kind-clip_vg/instance-pfdwoyly v2
[2024-12-27T16:43:55+0100] [MainThread] [W] [toil.leader] Log from job "kind-clip_vg/instance-pfdwoyly" follows:
=========>
	[2024-12-27T16:14:04+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2024-12-27T16:14:04+0100] [MainThread] [I] [toil] Running Toil version 7.0.0-d569ea5711eb310ffd5703803f7250ebf7c19576 on host handelsman.
	[2024-12-27T16:14:04+0100] [MainThread] [I] [toil.worker] Working on job 'clip_vg' kind-clip_vg/instance-pfdwoyly v1
	[2024-12-27T16:14:04+0100] [MainThread] [I] [toil.worker] Loaded body Job('clip_vg' kind-clip_vg/instance-pfdwoyly v1) from description 'clip_vg' kind-clip_vg/instance-pfdwoyly v1
	[2024-12-27T16:14:06+0100] [MainThread] [I] [cactus.shared.common] Running the command ['vg', 'convert', '-W', '-f', '/tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/e126/job/tmpxvz_89p1/NC_056626.2.vg']
	[2024-12-27T16:14:06+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:14:06.928707: Running the command: "vg convert -W -f /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/e126/job/tmpxvz_89p1/NC_056626.2.vg"
	[2024-12-27T16:14:53+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:14:53+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:14:53+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:14:53.442515: Successfully ran: "vg convert -W -f /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/e126/job/tmpxvz_89p1/NC_056626.2.vg" in 46.5086 seconds and 2.1 Gi memory with job-memory 35.6 Gi. Percent utilization: 6.032
	[2024-12-27T16:14:53+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:14:53.442820: Running the command: "gfaffix /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/e126/job/tmpxvz_89p1/NC_056626.2.vg.gfa --output_refined /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/e126/job/tmpxvz_89p1/NC_056626.2.vg.gfaffixed.gfa --check_transformation --dont_collapse Salinas_V11#[.]*"
	oriented_parent_nodes	oriented_child_nodes	prefix_length	prefix



	[2024-12-27T16:52:12+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:52:12+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:52:12+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:12.501974: Successfully ran: "gfaffix /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfa --output_refined /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed.gfa --check_transformation --dont_collapse 'Salinas_V11#[.]*'" in 443.9222 seconds and 22.1 Gi memory with job-memory 35.6 Gi. Percent utilization: 62.21
	[2024-12-27T16:52:12+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:12.502428: Running the command: "head -1 /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfa"
	[2024-12-27T16:52:12+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:52:12+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:12.513398: Successfully ran: "head -1 /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfa" in 0.0051 seconds and 1.6 Mi memory
	[2024-12-27T16:52:12+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:12.513613: Running the command: "sed -i /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed.gfa -e 1s/.*/H	VN:Z:1.1	RS:Z:Salinas_V11 _MINIGRAPH_/"
	[2024-12-27T16:52:32+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T16:52:32+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:32.545763: Successfully ran: "sed -i /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed.gfa -e '1s/.*/H	VN:Z:1.1	RS:Z:Salinas_V11 _MINIGRAPH_/'" in 20.0267 seconds and 146.2 Mi memory
	[2024-12-27T16:52:32+0100] [MainThread] [I] [toil-rt] 2024-12-27 16:52:32.546042: Running the command: "bash -c set -eo pipefail && vg convert -g -p /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed.gfa | vg mod -u -"
	[2024-12-27T17:01:52+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T17:01:52+0100] [MainThread] [I] [toil-rt] 2024-12-27 17:01:52.926522: Successfully ran: "bash -c 'set -eo pipefail && vg convert -g -p /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed.gfa | vg mod -u -'" in 560.3755 seconds and 10.3 Gi memory
	[2024-12-27T17:01:52+0100] [MainThread] [I] [toil-rt] 2024-12-27 17:01:52.927089: Running the command: "bash -c set -eo pipefail && clip-vg /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed -f -e Salinas_V11 -d _MINIGRAPH_ -L | vg mod -X 1024 - | vg paths -x - -n -Q Salinas_V11 -t 1 | vg clip -d 1 - -P Salinas_V11 | vg ids -s -"
	[2024-12-27T17:12:10+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T17:12:10+0100] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.bytes2human()" instead."
	[2024-12-27T17:12:10+0100] [MainThread] [I] [toil-rt] 2024-12-27 17:12:10.981788: Successfully ran: "bash -c 'set -eo pipefail && clip-vg /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.gfaffixed -f -e Salinas_V11 -d _MINIGRAPH_ -L | vg mod -X 1024 - | vg paths -x - -n -Q Salinas_V11 -t 1 | vg clip -d 1 - -P Salinas_V11 | vg ids -s -'" in 618.0497 seconds and 13.1 Gi memory with job-memory 35.6 Gi. Percent utilization: 36.7
	[2024-12-27T17:12:10+0100] [MainThread] [I] [toil-rt] 2024-12-27 17:12:10.982001: Running the command: "vg validate /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.clip"
	[2024-12-27T17:13:37+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
	[2024-12-27T17:13:37+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-export_vg/instance-w17wk5h9/file-bedf44b92b8949e786268571222c9a3c/out.vg' to path '/tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg'
	[2024-12-27T17:13:37+0100] [MainThread] [C] [toil.worker] Worker crashed with traceback:
	Traceback (most recent call last):
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 438, in workerScript
	    job._runner(jobGraph=None, jobStore=job_store, fileStore=fileStore, defer=defer)
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2984, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2895, in _run
	    return self.run(fileStore)
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 3158, in run
	    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap_join.py", line 752, in clip_vg
	    cactus_call(parameters=['vg', 'validate', clipped_path])
	  File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 914, in cactus_call
	    raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out))
	RuntimeError: Command /usr/bin/time -f "CACTUS-LOGGED-MEMORY-IN-KB: %M" vg validate /tmp/toilwf-90579744a5f4551fb664cc1b833dcce5/7434/job/tmplno7x13n/NC_056626.2.vg.clip exited 1: stderr=graph invalid: missing edge between 631th step (4258332:0) and 632th step (4258332:1) of path Stem_lettuce#0#HiC_scaffold_48#0
	graph invalid: missing edge between 632th step (4258332:0) and 631th step (4258332:0) of path Stem_lettuce#0#HiC_scaffold_48#0
	graph: invalid
	Command exited with non-zero status 1
	CACTUS-LOGGED-MEMORY-IN-KB: 2791436
	
	
	[2024-12-27T17:13:37+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host handelsman
<=========

I found some other similar issues on this github page linking to marschall-lab/GFAffix#11, but it seems that that issue is resolved. Also, the version of gfaffix in the docker container is the newest (gfaffix 0.1.5b).
@glennhickey
Copy link
Collaborator

Hmm, this does indeed seem like a gfaffix issue. There's a new version of GFAffix (that will be added to cactus shortly). Are you able to try installing that yourself and rerunning Cactus? Hopefully it will have a fix for your issue.

@dirkjanvw
Copy link
Author

Curious to try out that new version! There is not that much haste involved, so I will wait for the new release to test it using the docker image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@glennhickey @dirkjanvw