Akhet Queue Dashboard

A queue is the part of an application you cannot see. Jobs go in, workers take them out, and the only evidence either happened is that something else changed later. Akhet is the horizon you watch that from: what is waiting, what is running, what has been failing, and how long any of it takes.

Akhet is a Vizier extension, not a separate dashboard. One panel, one login, one layout — the queue sits next to the rest of the admin rather than on its own port behind its own password.

Akhet watches; it does not supervise

Laravel’s Horizon starts, stops and scales worker processes. Akhet does none of that, deliberately — your process supervisor already runs the application, and a framework package that also spawns processes is a second, invisible supervisor to reason about. See Coming From Laravel.

Akhet needs a queue to watch

Akhet reports on Khepri. It does not provide a queue, and it will render an empty dashboard — correctly, and with no error — in an application that has not got one. Before installing it, check that all of this is true:

khepri is a dependency and QueueServiceProvider is registered;
...khepriMigrations has run, so jobs and failed_jobs exist;
some Job classes exist and something dispatches them;
a worker is running — queue:work, or queue:work --stop-when-empty for a one-off — because Akhet records finished attempts and an undispatched or unworked job has not finished anything.

An application that has never run a queue sees three empty pages and no indication of why. That is not a fault; it is the honest report of a queue with no history, and it is the single most likely reason a fresh install looks broken.

Installing

Akhet needs Khepri for the queue and Vizier for the panel.

dependencies:
  akhet: ^0.1.0

maat akhet:install

That publishes the two views the panel renders. Then wire the three pieces:

khepriMigrations comes first, and the order matters: Akhet reads Khepri’s jobs and failed_jobs live rather than copying them, so those tables have to exist before Akhet’s own.

final migrations = <Migration>[...khepriMigrations, ...akhetMigrations];

// bootstrap/app.dart — the recording half
Application.configure(...).withExtensions([akhetExtension]).create();

// bootstrap/vizier_app.dart — the panel half
Panel(..., extensions: [akhetVizier()]);

Two opt-ins rather than one: a worker process wants the recording and has no panel, and the panel is the only half that needs Vizier. An application that runs its API and its workers as separate deployments registers the extension in both and the panel in neither.

Configuration

final akhet = {
  'enabled': env('AKHET_ENABLED') == 'true',
  'prune_hours': 168,
  'ignore_jobs': ['Heartbeat'],
  'record_completed': true,
};

Key	Default	What it does
`enabled`	`false`	Records nothing at all until this is `true`.
`prune_hours`	`168`	The default window for `akhet:prune`.
`ignore_jobs`	`[]`	Registered job names never recorded.
`record_completed`	`true`	When `false`, only failures and releases are stored.

An empty Job runs table with a populated Failed jobs table

That exact pair means enabled is still false. Failed jobs are read straight out of Khepri’s table and appear whatever Akhet is set to; job runs come from Akhet’s own recorder and appear only when it is on. It is the fastest way to tell “the dashboard is off” from “the wiring is broken”.

enabled is off in every environment unless you say otherwise. A dashboard that turns itself on writes a row per job on the day it is installed, which is how observability tools earn their reputation for being the thing that filled the disk.

Every value falls back to a working default when it is missing or the wrong type. A malformed config leaves Akhet quiet; it never stops the application booting.

A busy queue

A queue running thousands of jobs an hour does not need a row per success.

'record_completed': false,

That turns the history from one row per job into one row per problem. You lose throughput and average-runtime figures — both are computed from completed runs — and keep every failure and retry. ignore_jobs is the finer instrument: name the one high-frequency job that would otherwise dominate the table and keep full history for everything else.

What the panel shows

Queues

The overview page, one row per queue:

Column	What it means
Pending	Available now and unclaimed — the backlog.
Reserved	Claimed by a worker.
Delayed	Pushed with a delay, or released after a failed attempt.
Oldest wait	How long the oldest available job has been waiting.
Completed	Completions inside the window.
Average runtime	Mean runtime of those completions.
Failures	Jobs that exhausted their attempts inside the window.

Oldest wait is the number to watch. Depth alone cannot tell a busy queue from a stuck one: a queue holding 500 jobs and clearing them in seconds is healthy, and a queue holding 3 jobs that have been waiting an hour is not.

Delayed is counted apart from pending for the same reason. A large delayed count is normal — that is retries and scheduled work waiting their turn. A large pending count is a worker that is not keeping up.

Reserved counts every row with a claim on it, not only the ones a worker is actively running: a job whose worker died, or whose handle() has run longer than the queue’s retry_after, still shows as reserved until another worker reclaims it. A reserved count that never drains is as much a signal as a climbing pending count — it just points at a stuck or crashed worker instead of a slow one.

The figures are read at the moment you open the page, with no caching. A cached dashboard that shows a queue as healthy after it has stalled is worse than a slow one.

Job runs

Every finished attempt, newest first, searchable by job, queue and status.

Status	Meaning
`completed`	Handled successfully and left the queue.
`released`	Threw, and went back for another attempt.
`failed`	Threw, with no attempts left.

A job that fails after two retries produces three rows: released, released, failed. They share a job_id and have their own id each, because the table records attempts rather than jobs.

Failed jobs

Khepri’s failed list, with the two things an operator opens a queue dashboard to do:

Retry pushes the stored payload back onto its queue and forgets the record. The replayed job starts at attempt one.
Forget deletes the record without replaying it.

The Exception column shows the first line only — a Dart exception carries its message and its whole stack trace in one string, and rendering that verbatim makes every row about twenty lines tall. Open a record for the full trace and the payload Retry would replay.

Both are bulk actions and both ask for confirmation. Both go through Khepri’s own FailedJobStore and QueueConnection rather than writing failed_jobs here. Retry behaves exactly like queue:retry on the command line — same push-then-forget order, one implementation of the verb. There is no queue:forget command; Forget calls FailedJobStore.forget directly, the same method queue:retry uses internally to drop the record once its replay is queued.

Retry pushes before it forgets, deliberately. If the process dies between the two, the job is queued and the record is still listed, so an operator may retry it twice. Forgetting first would lose the payload outright if the push then failed. A duplicate run beats lost work.

Total backlog and total failures, plus a line per queue. The two numbers that mean “something is wrong right now”; everything else is on the overview page, where there is room to read it.

Keeping the table small

maat akhet:prune              # uses prune_hours, a week by default
maat akhet:prune --hours=48

Schedule it — see Task Scheduling:

.withSchedule((schedule) {
  schedule.command('akhet:prune').daily();
})

A command, never a timer the package starts on its own. The application already has a way to run scheduled work, and a package that quietly starts a timer inside a long-lived process is one nobody can reason about.

akhet:prune refuses a window that is zero, negative, non-numeric, absurdly large, or an explicit --hours= with nothing after it. Each of those would otherwise delete the whole table: a zero window puts the cutoff at now, and an unbounded one overflows into a negative duration that puts the cutoff in the future, where every row matches.

What it costs

One insert per finished job attempt, on the worker, after the job has already resolved. Nothing runs on the request path.

The recorder subscribes to Khepri’s JobObservers as an observer, never a wrapper: Akhet reports what a job did and has nothing to contribute while it runs. (Udjat is the package that wraps, because it needs a batch to hang the job’s queries and log lines off. Both can subscribe at once.)

A write that cannot happen — a missing table, a dropped connection — is logged and dropped. A dashboard must never be able to fail a job by failing to describe it.

When enabled is false nothing is subscribed at all, rather than subscribed and asked to keep quiet. In a long-lived process that is what “disabled” has to mean.

Reading it from code

The panel is one consumer of the metrics; your own health endpoint can be another.

final metrics = app<AkhetMetrics>();

for (final queue in await metrics.snapshot()) {
  if ((queue.oldestWait ?? Duration.zero) > const Duration(minutes: 10)) {
    await app<Notifier>().alert('${queue.queue} is backing up');
  }
}

AkhetMetrics takes its window and its clock, so a test can drive it to a fixed moment rather than waiting for one:

final metrics = AkhetMetrics(
  window: const Duration(hours: 1),
  clock: () => DateTime.utc(2026, 9, 6, 12),
);
expect((await metrics.snapshotOf('mail')).pending, 2);

Storage

One table, akhet_job_runs: id, job_id, queue, job, status, attempts, duration_ms, finished_at, exception.

Everything else comes from Khepri and is read live — pending, reserved and oldest wait from jobs, the failed list from failed_jobs. A dashboard that kept its own copy of the queue would be a second source of truth, and it would drift the first time a worker was killed mid-job.

id is the run’s own uuid and job_id is the queue’s job id. They are separate because a job released twice before it fails produces three rows that all share one job id — keying on that would make every attempt after the first collide and silently vanish.

Localisation

Every string the panel renders comes from akhetVizier()’s translations map, in both en and ar. Nothing is hardcoded in the published views: a stub with an English word baked into it renders an Arabic panel half-translated, and the stub is the last place a reviewer looks. If you edit a published view, keep new strings going through translations[...] too.