Run jobs, queues, and schedules.
Nothing to operate.
Push a worker and Darwa gives it a queue, a scheduler, retry policy, a dead-letter queue, concurrency limits, and per-job logs. When the backlog grows it adds workers — and tells you when the queue will be clear.
Bring the worker framework you already use.
Your existing BullMQ, Celery, or Sidekiq worker deploys as-is. Darwa supplies the broker, the scheduler, and the runtime around it.
worker: image-processing runtime: node22 start: node worker.js queue: name: images type: priority concurrency: 20 scale: min: 1 max: 40 on: queue_depth retries: attempts: 5 backoff: exponential dead_letter: true rate_limit: stripe: 100/minute
Every queue pattern, without assembling one.
FIFO for ordering, priority for urgency, delayed for later, scheduled for recurring — declared in config rather than built out of primitives.
Strict ordering per queue or per key, so jobs for one customer never overtake each other.
Ten priority levels. A password reset jumps ahead of a nightly export without a second queue.
Run in ten minutes, at 02:00, or on a timestamp you compute — held by the broker, not a sleeping process.
Existing BullMQ or Sidekiq code connects unchanged; the connection string is injected for you.
Enqueue the same job id twice and only one runs — useful for webhook storms.
Push thousands of jobs in one call and track them as a group with a single progress figure.
| Trigger | What starts the job |
|---|---|
| HTTP request | POST a payload to a worker endpoint and get a job id back |
| Queue message | Anything your app pushes to a Redis-compatible queue |
| Schedule | Cron expressions, or plain-language intervals |
| Database event | Row inserted, updated, or deleted in your managed Postgres |
| Storage upload | A file landing in a bucket starts the job that processes it |
| Webhook | Signed third-party callbacks, verified before your code runs |
Scheduling that does not need its own container.
Cron expressions or plain intervals, attached to the worker that runs them. Missed windows are recorded rather than silently skipped.
Failures retry themselves, then land somewhere you can see.
Exponential backoff by default, a dead-letter queue with a real interface, and a diagnosis instead of a stack trace.
Attempts, backoff curve, and ceiling are per queue. A rate-limited third party gets patient retries; a malformed payload fails fast and goes straight to the dead-letter queue.
Different work needs different limits.
Image processing wants twenty workers. A rate-limited API wants five. An expensive model call wants two. Set it per queue and let the platform hold the line.
| Queue | Concurrency | Scale range | Rate limit | Why |
|---|---|---|---|---|
| image-processing | 20 | 1 → 40 | — | CPU bound, safe to widen |
| email-sending | 5 | 1 → 5 | 300/min | Provider throttles above this |
| stripe-billing | 4 | 1 → 4 | 100/min | API key quota |
| ai-summaries | 2 | 1 → 3 | 20/min | Model cost per call |
| nightly-export | 1 | 1 → 1 | — | Must not run twice |
Rate limits are enforced across the whole pool, not per worker — so scaling to forty workers cannot accidentally quadruple the requests you send a third party.
Per-job logs, not one giant stream to grep.
Open a job and see only its lines, its payload, its attempts, and its timings. Metrics answer the question you actually have: is the backlog growing or shrinking?
Cost is attributed per job, so an expensive queue is obvious before the invoice explains it.
Workers idle at 3 a.m. and you still pay for them.
Darwa watches the shape of your backlog over the week and proposes floors and ceilings that match it — with the saving stated before you agree.
What is queued next.
Not shipped yet. Everything above works without it — this is where the product is heading.
Orchestration without orchestration code — each node keeps its own retries, concurrency, and logs.
Pay for worker time, not per message.
Per-message pricing punishes exactly the workloads that suit a queue. You pay for the worker hours you actually consume, and the queue itself is included.
Move one queue over and watch the board.
Your worker code does not change. The broker, scheduler, retries, and dead-letter queue stop being your problem.