Retry Logic — Aquilia Documentation
Comprehensive guide and documentation for Retry Logic in the Aquilia framework. View API reference, examples, and implementation patterns.
Background Tasks / Retry Logic Retry Logic & Error Handling Understand how Aquilia processes background task failures, computes exponential backoffs with jitter, and routes jobs to the dead-letter queue. How Retries are Processed When a task handler raises an unhandled exception, the worker catches the error, increments the retry counter, and determines if it can run again based on the task policy: Exception Intercepted: The worker catches exceptions raised inside the task coroutine. Retry Evaluation: The worker compares job.retry_count to job.max_retries. Backoff Math: If retries remain, the next schedule delay is computed with exponential backoff and ±25% random jitter. State Transition: The job state is updated to JobState.RETRYING . Queue Re-push: The job is pushed back into the heap queue with its scheduled_at timestamp set to the cooldown expiration. Dead-Letter Route: If all retries are exhausted, the job state transitions to JobState.DEAD and is sent to the dead-letter queue. State Transitions RUNNING FAILED RETRYING PENDING DEAD raised error tries remaining backoff delay limit reached scheduled worker poll Exponential Backoff & Jitter To prevent the "thundering herd" problem when external services fail, Aquilia adds a ±25% random jitter to the calculated exponential backoff delay. # Source code from aquilia/tasks/job.py @property def next_retry_delay(self) -> float: """Calculate next retry delay with exponential backoff + jitter.""" import random delay = self.retry_delay * (self.retry_backoff ** self.retry_count) delay = min(delay, self.retry_max_delay) # Add random jitter (±25%) jitter = delay * 0.25 * (2 * random.random() - 1) return max(0.1, delay + jitter) Retry Backoff Intervals (Default Settings) Formula: delay = retry_delay * (retry_backoff ^ attempt) capped at retry_max_delay. Attempt Formula Base Delay Actual Interval (with ±25% jitter) ))} Configuring Retry Policies Aggressive (Short Wait, High Attempts) @task( max_retries=10, retry_delay=0.5, # Start at 500ms retry_backoff=1.5, # Slower exponential curve retry_max_delay=60.0, # Max cap of 1 minute ) async def aggressive_retry_task(): ... Conservative (Long Wait, Low Attempts) @task( max_retries=3, retry_delay=30.0, # Start at 30 seconds retry_backoff=3.0, # Slower exponential curve retry_max_delay=1800.0, # Max cap of 30 minutes ) async def conservative_retry_task(): ... Disable Retries (One-Shot Task) @task(max_retries=0) async def non_retryable_task(): # Any failure routes directly to DEAD state ... Dead-Letter Queues (DLQ) Jobs that exceed `max_retries` transition to the DEAD state. You can monitor, list, and manually trigger retries for dead jobs. from aquilia.tasks import JobState # Query dead jobs from the manager dead_jobs = await manager.list_jobs(state=JobState.DEAD) for job in dead_jobs: print(f"Dead Job ID: | Task: ") print(f"Error: ") # Manually re-enqueue/retry the job await manager.retry_job(job.id) )
Go to Homepage