Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Fargate resource type for AWS Batch #3474

Merged
merged 22 commits into from
Dec 2, 2023
Merged

Conversation

pditommaso
Copy link
Member

This PR adds support Fargate resource type for AWS Batch executor.

Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso force-pushed the master branch 2 times, most recently from e2b4a93 to f32ea0b Compare December 8, 2022 15:16
@pditommaso pditommaso force-pushed the master branch 2 times, most recently from cefb067 to e523afd Compare December 22, 2022 20:43
@pditommaso pditommaso marked this pull request as draft February 18, 2023 21:20
Signed-off-by: Paolo Di Tommaso <[email protected]>
@ShanWeera
Copy link

ShanWeera commented Sep 16, 2023

@pditommaso hey there Paolo!. Just stumbled upon this after hours of trying to get Nextflow working with AWS Batch + Fargate. Fargate support would be really great to have so really appreciate the work you have put in here.

Going from what you mentioned over at: https://groups.google.com/g/nextflow/c/JFneg8d3x2w/m/EjYGXUubCQAJ

The problem with Fargate is that it does not allow host mounts which NF needs to access the aws cli tool without installing it in the container.

Have you been able to find a workaround for this limitation?

As an aside, I feel the documentation does not make it very clear that Fargate is in fact not supported by Nextflow. Fargate is actually the default selection when creating AWS Batch job definitions and queues, and it is arguably the most convenient way to get started with AWS Batch (among other advantages).

All this to suggest that it might be worth it to explicitly mention this limitation somewhere in the documentation, as I imagine many people would go with Fargate just to realise later that this is not supported (at least for the moment).

@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@nextflow-io nextflow-io deleted a comment from sonatype-lift bot Sep 18, 2023
@bentsherman
Copy link
Member

Don't think we've found a workaround yet. We either need to be able to mount the AWS CLI or use Fusion, but there also we are waiting on this proposal to make that possible.

Copy link

netlify bot commented Dec 1, 2023

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 5700dfd
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/656b4ff96fd1c30008020fae
😎 Deploy Preview https://deploy-preview-3474--nextflow-docs-staging.netlify.app/aws
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso marked this pull request as ready for review December 2, 2023 02:29
Signed-off-by: Paolo Di Tommaso <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso added this to the 23.12.0-edge milestone Dec 2, 2023
docs/aws.md Outdated Show resolved Hide resolved
Co-authored-by: Ben Sherman <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso merged commit 47cf335 into master Dec 2, 2023
4 checks passed
@pditommaso pditommaso deleted the aws-fargate branch December 2, 2023 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants