Code

6 scheduling#36

Code

Open

vlerkin wants to merge 23 commits intoq-m:mainq-m/scrapyd-k8s:mainfrom vlerkin:6-schedulingvlerkin/scrapyd-k8s:6-schedulingCopy head branch name to clipboard

+978-35

Commits

Commits on Nov 1, 2024

• extracted resource watching logic into a separate class; implemented logic for observer in a RecourceWatcher class; added method to stop a thread gracefully

  vlerkin

  committed
  4c0a82d

  Copy full SHA for 4c0a82d

Commits on Nov 6, 2024

• add kubernetes scheduler class that subscribes to the event observer that handles the logic to unsuspend jobs and get the next in order according to the creation timestamp; modify schedule endpoint…

  Show description for da8254f
  vlerkin

  committed
  da8254f

  Copy full SHA for da8254f

• Merge branch 'main' into 6-scheduling

  vlerkin

  committed
  9dae80d

  Copy full SHA for 9dae80d

Commits on Nov 7, 2024

• finish the observer logic extraction; merge changes from main; add resource watcher instance to enable_joblogs to subscribe to the event watcher if the log feature is configured; delete logic about…

  Show description for 7f66895
  vlerkin

  committed
  7f66895

  Copy full SHA for 7f66895

• change enable_joblogs method signature in docker

  vlerkin

  committed
  963f279

  Copy full SHA for 963f279

Commits on Nov 11, 2024

• refactor api.py and launcher/k8s.py to keep api.py launcher agnostic

  vlerkin

  committed
  7dc24bc

  Copy full SHA for 7dc24bc

Commits on Nov 12, 2024

• add implementation for docker to check the number of running containers and run more from the queue of created jobs when capacity is available; add backgroung thread that sleeps for 5 sec and trigg…

  Show description for 98f5559
  vlerkin

  committed
  98f5559

  Copy full SHA for 98f5559

• remove update from RBAC because we perform patching of the resource and patch is sufficient

  vlerkin

  committed
  01b120b

  Copy full SHA for 01b120b

Commits on Nov 14, 2024

• add number of retry attempts and exponential backoff time to the reconnect loop for event watcher; make number of reconnect attempts, backoff time and a coefficient for exponential growth configura…

  Show description for 08f1785
  vlerkin

  committed
  08f1785

  Copy full SHA for 08f1785

Commits on Nov 20, 2024

• create a helper finction _filter_jobs that accepts a filter function and a label selector to make the code in listjobs, get_running_jobs and list_suspended_jobs DRY; refactor listjobs to use the he…

  Show description for 2d5e6bf
  vlerkin

  committed
  2d5e6bf

  Copy full SHA for 2d5e6bf

• change the signature of the filtering method to also accept a parse function because list jobs uses a different logic

  vlerkin

  committed
  36217d9

  Copy full SHA for 36217d9

Commits on Nov 21, 2024

• add number of retry attempts and exponential backoff time to the reconnect loop for event watcher; make number of reconnect attempts, backoff time and a coefficient for exponential growth configura…

  Show description for 2d843c1
  vlerkin

  committed
  2d843c1

  Copy full SHA for 2d843c1

• add logic to reset number of reconnect attempts and backoff time when connection to the k8s was achieved so only sequential failures detected; add exception handling to watch_pods to handle failure…

  Show description for 923915f
  vlerkin

  committed
  923915f

  Copy full SHA for 923915f

• added a CONFIG.md file with detailed explanations about parameters used to re-establish connection to the Kubernetes wather

  vlerkin

  committed
  6a64485

  Copy full SHA for 6a64485

• move section about config file from README.md to CONFIG.md; add a link to the CONFIG.md in the README.md; remove variables for reconnection_attempts, backoff_time and backoff_coefficient fron the s…

  Show description for 3604267
  vlerkin

  committed
  3604267

  Copy full SHA for 3604267

• Merge branch 'main' into 6-scheduling

  vlerkin

  committed
  6394633

  Copy full SHA for 6394633

Commits on Nov 22, 2024

• make number of job limitation optional by separating k8s_scheduler in a package that has an enable function in launcher/k8s.py which also part of resource watcher initialization; initialize the sch…

  Show description for e3197f2
  vlerkin

  committed
  e3197f2

  Copy full SHA for e3197f2

• implement the feature to limit running jobs to be optional for Docker; remove max_proc from config file since by default we want to runn all scheduled jobs in parallel; add a section about max_proc…

  Show description for e23bfce
  vlerkin

  committed
  e23bfce

  Copy full SHA for e23bfce

Commits on Nov 25, 2024

• add exceptions to the KubernetesScheduler class, for the init method, propagate the exception

  vlerkin

  committed
  17d14b6

  Copy full SHA for 17d14b6

• Merge remote-tracking branch 'upstream/main' into 6-scheduling

  vlerkin

  committed
  a30c531

  Copy full SHA for a30c531

• add custom exceptions to the class for log handling; add exceptions to handle and propagate the critical ones

  vlerkin

  committed
  8d8032c

  Copy full SHA for 8d8032c

Commits on Nov 27, 2024

• modify method that fetches nez job id to be unsuspended to add creation_timestamps to the jobs that do not have them, so they are proccessed at the end of the queue; add unit tests for k8s_schedule…

  Show description for 331090f
  vlerkin

  committed
  331090f

  Copy full SHA for 331090f

• make the proper test structure according to the standards; add tests for docker implementation of the scheduler

  vlerkin

  committed
  16a8276

  Copy full SHA for 16a8276