Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup of joblogs #46

Open
wvengen opened this issue Jan 2, 2025 · 0 comments
Open

Cleanup of joblogs #46

wvengen opened this issue Jan 2, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@wvengen
Copy link
Member

wvengen commented Jan 2, 2025

After #45, joblogs can be stored on container storage. As jobs slowly expire, their logs remain on container storage (as desired).
But over time, the logs can add up in storage size.

Allow configuring a joblogs expiration time.

There is a tradeoff to make between working out-of-the-box (integrate in the scrapyd-k8s core, incl. scheduling it regularly) and simplicify (e.g. a cronjob, perhaps as a separate part of the code). I would tend to use a cronjob (CronJob for k8s, a system-based cronjob to run a specific command for Docker), and have a separate script to clean container storage (e.g. python3 -m scrapyd_k8s.joblogs.cleanup or so). Or, perhaps there is already an existing tool to remove files on container storage that are older than X - though ideally it works with the scrapyd-k8s config, so maybe writing our own on top of the integration is simpler).

@wvengen wvengen added the enhancement New feature or request label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant