Docker Stack Compose Directories Contain All Files Stored in Git Repository

[Adagis-MC]

Is your feature request related to a problem? Please describe.
AS an Administrator
I WANT to Portainer to only pull down the files necessary from Git to support the Compose Stack
SO THAT only necessary files are within the Compose folder and the full repository is not downloaded for each Compose Stack

Describe the solution you'd like
GIVEN THAT a Compose Stack has been pulled from Git
WHEN I navigate to the Portainer Compose folder
AND I view a Compose Stack subfolder
THEN only the files required by the Compose Stack are stored

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
For each Compose Stack retrieved from Git, the entire repository is retrieved and stored in the Compose folder for that Stack. If you store multiple compose files (and other files, documentation etc) in that repository, these files are also pulled. This results in unnecessary replication of unused data, increasing diskspace and potentially opening security risks for data that should not be generally available.

[jamescarppe]

The issue here becomes how you determine what files are "necessary" in order to deploy your compose stack. If your compose file references a .env file, you'd need to pull that as well. If it contains an image build, it would need to retrieve the Dockerfile, then potentially parse that Dockerfile to see what files that it requires.

[Adagis-MC]

I think that if it needed to retrieve the Dockerfile and parse it, then that would be done when the compose file is run. Pulling an entire repository which might contain multiple files (potentially multiple images) impacts network, disk space and timing. If you are developing, having it pull the entire repository every time you just want it to update a small compose file isn't an intelligent use of resource.

[jamescarppe]

We only pull the repo when the stack is created (ie, the compose file is run), so this would happen at the same time. Are you proposing that we instead pull once to get just the compose file, then once again to pull dependency files?

Bear in mind the docker compose command which we leverage to create the stacks requires that all the necessary files already be available or it will fail. We'd essentially need to write (and maintain) a parser for the compose file format that would detect the need to pull additional files. There's a myriad of ways that additional files could be included, the examples I gave are only two possibilities, so it would be quite a complex implementation.

A better bet might be to instead think about the structure of your remote repositories in terms of what is actually needed for deployments and tune them to suit.

It's worth noting also that we did originally just pull the compose files, but we ran into issues with missing dependencies and the desires of our users for us to pull the whole repo instead, so we switched to pulling the whole thing.

[Adagis-MC]

It's interesting because my use case is primarily for running stacks that pull images from the Docker hub, rather than a private repository. Consequently, all my Compose files are in one branch of one repository, since it doesn't make any sense to create separate repos to hold one YAML file of a few kb.
What then happens is that the entire repo is pulled rather than just the branch containing the specific stack. So I then have ten different copies of the same repository pulled each time when all I want is a small file.

So at the least, it should really only pull the branch that has been requested, not the entire repo. At the most, it could allow a user to select which files from a branch to pull (or parse the compose file to identify which files are needed).

The difficulty is of course that everyone uses repositories in different ways, so there is no one size fits all solution to this!