Two of the most-viewed threads in the Hangfire forum history are about job retention: "How to configure the retention time of job?" and "Failed jobs never expire/delete." Between them, they've been viewed well over 60,000 times. If you've hit either wall — succeeded jobs vanishing overnight, or 60,000 failed jobs piling up across a year — here's what's actually happening, and what you can do about it.
Hangfire runs a background sweep that removes job records from storage once they age past a retention window. The exact default depends on your storage provider and version, but on most setups the behavior is the same: succeeded jobs are pruned after roughly a day, and the sweep runs continuously in the background. It's a sensible default — without it, the Job and State tables grow without bound, and on SQL Server that turns into lock contention and dashboard slowdowns fast.
The twist: the same sweep doesn't reliably remove failed jobs. A failed job sits in storage until you either delete it manually from the dashboard or write your own cleanup. Over months, that means a steady accumulation — we've seen real production databases where the Hangfire.State table had grown to several gigabytes purely from failed jobs nobody had gotten around to clearing.
For succeeded jobs, your storage provider may expose a configuration option (sometimes via JobStorageOptions.JobExpirationTimeout, sometimes via the storage's own settings) that lets you raise the retention window. Set it to 30 days and you get a month of history instead of a day. So far, so good.
The catch is what happens to the storage tables. The Hangfire schema wasn't designed for long-term archival — the State table in particular is write-heavy and index-heavy, and on SQL Server the dashboard's SELECT TOP queries against a multi-million-row State table start taking seconds rather than milliseconds. A community-reported GitHub issue (#1962) describes the dashboard becoming unusably slow after a few months of raised retention. You're trading history retention for dashboard responsiveness, and there's no middle ground within Hangfire itself.
For failed jobs, the situation is worse — there's no built-in "auto-delete failed jobs older than X" option at all. You can write a recurring job that does the cleanup yourself, but that's a self-built feature, not configuration.
The instinct is to raise retention as high as it'll go and leave it. The problem is that Hangfire's storage is operational — it exists to drive the queue, the workers, and the dashboard, not to act as a permanent audit log. The moment you treat it as your history-of-record, three things break:
InvoiceService.SendInvoice job that failed in Q1").The clean answer is to stop asking Hangfire's storage to be both the live queue and the long-term archive. Hangfire does what it's good at — orchestrating jobs — and a separate system streams state changes out as they happen and keeps them somewhere that doesn't share Hangfire's lifecycle.
That's the specific gap QueueHawk closes: QueueHawk.Agent hooks into Hangfire's OnStateElection / OnStateApplied filters and streams every state change out over HTTPS as it happens. History lives in its own database, with 7 to 365 days of retention depending on plan, and survives redeploys, storage migrations, and full Hangfire restarts — because it was never sitting in the same database Hangfire sweeps. The Hangfire schema stays lean and the dashboard stays fast; the audit trail lives somewhere else.
See also: why job history also disappears after every deploy (a related but distinct problem — that one's about schema resets, this one's about retention), and how QueueHawk compares to the built-in Hangfire dashboard more broadly.