You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After #45, joblogs can be stored on container storage. As jobs slowly expire, their logs remain on container storage (as desired).
But over time, the logs can add up in storage size.
Allow configuring a joblogs expiration time.
There is a tradeoff to make between working out-of-the-box (integrate in the scrapyd-k8s core, incl. scheduling it regularly) and simplicify (e.g. a cronjob, perhaps as a separate part of the code). I would tend to use a cronjob (CronJob for k8s, a system-based cronjob to run a specific command for Docker), and have a separate script to clean container storage (e.g. python3 -m scrapyd_k8s.joblogs.cleanup or so). Or, perhaps there is already an existing tool to remove files on container storage that are older than X - though ideally it works with the scrapyd-k8s config, so maybe writing our own on top of the integration is simpler).
The text was updated successfully, but these errors were encountered:
After #45, joblogs can be stored on container storage. As jobs slowly expire, their logs remain on container storage (as desired).
But over time, the logs can add up in storage size.
Allow configuring a joblogs expiration time.
There is a tradeoff to make between working out-of-the-box (integrate in the scrapyd-k8s core, incl. scheduling it regularly) and simplicify (e.g. a cronjob, perhaps as a separate part of the code). I would tend to use a cronjob (CronJob for k8s, a system-based cronjob to run a specific command for Docker), and have a separate script to clean container storage (e.g.
python3 -m scrapyd_k8s.joblogs.cleanup
or so). Or, perhaps there is already an existing tool to remove files on container storage that are older than X - though ideally it works with the scrapyd-k8s config, so maybe writing our own on top of the integration is simpler).The text was updated successfully, but these errors were encountered: