Job Retry

When creating jobs objects you have the option of setting the retryMax and retryDelay values. Both of these values will determine what will happen if a job fails either by a worker node not responding or the job process failing.

Every job has a property called dateRetry which is used to determine if the job is ready for processing after a failure has occurred. The dateRetry value is set when the job is retrieved from the database for processing. The retrieval query will not return jobs where the dateRetry value is greater than the current date time value.

Currently the formula used to set the dateRetry value during the job retrieval process is:

now() + job.timeout + ( job.retryDelay * job.retryCount )

The plan in the future is to move this to an exponential formula once RethinDB has a power function.

As you can see, to disable the retry process and make jobs retry as soon as possible, simply set the retryDelay to zero.

If we take the job default properties and the Master Queue default masterReviewPeriod which is 310 seconds, then the following sequence of events will occur:

1. The job has never been processed and has default properties.

   • status = 'waiting'
   • timeout = 300
   • retryCount = 0
   • retryMax = 3
   • retryDelay = 600

2. The job is retrieved from the database setting the dateRetry value.

-   `status = 'active'`
-   `dateRetry = now() + timeout`

3. The job fails for some reason.

-   `status = 'retry'`
-   `retryCount = 1`

4. The job remains inactive within the database until after 300 seconds.

5. The Master Queue database review is initiated and the job is retrieved from the database for first retry.

-   `status = 'active'`
-   `dateRetry = now + timeout + (retryDelay * retryCount)` _900 seconds_

6. The job fails again for some reason.

-   `status = 'retry'`
-   `retryCount = 2`

7. The job remains inactive within the database until after 900 seconds.

8. The Master Queue database review is initiated and the job is retrieved from the database for second retry.

-   `status = 'active'`
-   `dateRetry = now + timeout + (retryDelay * retryCount)`

9. The job fails again. What is wrong with this job?

-   `status = 'retry'`
-   `retryCount = 3`

10. The job remains inactive within the database until after 1500 seconds.

11. The Master Queue database review is initiated and the job is retrieved from the database for third and final retry.

-   `status = 'active'`
-   `dateRetry = now + timeout + (retryDelay * retryCount)` _this is redundant however still set_

12. The job fails for the last time.

-   `status = 'failed'`

13. Because the job status is set to failed it will no longer be retrieved from the database.