Durable Retry And Session Cleanup

Purpose

Aether uses two durable recovery paths when realtime work cannot finish synchronously:

• SQL-backed persistence retry jobs for recoverable character, gameplay, combat reward, and transfer activation writes.
• Redis-backed session lifecycle journal entries for cleanup side effects after realtime streams close.

Both paths make the durable write happen before relying on an in-memory worker signal. If the signal is dropped, queued work remains recoverable.

Persistence Retry Jobs

PersistenceRetryJob wraps a PersistenceRetryPayload, optional session token, session impact, source label, and shared authoritative idempotency envelope. Payload kinds are:

• character_location
• character_appearance
• character_equipment
• gameplay_state
• combat_rewards
• transfer_activation

queue_persistence_retry_job_durable_with_metrics first records the durable intent in Postgres, then signals the in-memory retry worker. If the mpsc queue is full or closed, the function returns a durable-only signal outcome and the SQL row remains for polling.

The worker handles two paths:

• signaled jobs are claimed by retry job id;
• periodic polling loads due rows every 5 seconds, up to 128 jobs per poll.

SQL Schema And Leases

0011_persistence_retry_jobs.sql creates persistence_retry_jobs with retry job id, job kind, character id, realm id, zone id, serialized payload, last error, attempt count, next attempt time, lease token, lease owner, lease expiry, and timestamps.

The due loader claims rows with a single SQL statement:

• select due rows whose next_attempt_at <= NOW();
• ignore rows whose lease has not expired;
• order by next_attempt_at and creation time;
• use FOR UPDATE SKIP LOCKED;
• stamp lease_token, lease_owner, and lease_expires_at.

The current SQL lease duration is 30 seconds. SKIP LOCKED lets multiple game-server instances poll without processing the same row. If a worker dies, the lease expiry makes the row claimable again.

Attempts And Dead Letters

persist_retry_job_best_effort retries a claimed job up to the configured persistence retry limit. On each failure it upserts the retry row with the latest error, attempt count, next attempt time, and cleared lease fields.

Backoff uses character_location_retry_backoff, an exponential millisecond delay based on PERSISTENCE_RETRY_BACKOFF_MILLIS. The name is historical; the helper is used for all persistence retry payload kinds.

On success, the retry row is deleted. When attempts are exhausted, the worker writes a row to persistence_job_dead_letters and then deletes the retry row.

Active persistence dead letters are rows whose requeued_at is still null. The lifecycle migration adds:

• requeued_at, stamped when an operator requeues the dead letter.
• requeued_retry_job_id, the new retry job id restored from the dead-letter payload.

Do not edit or delete the dead-letter row manually. Use the protected requeue endpoint after confirming the current character/session state and fixing the underlying persistence failure.

Player And Session Impact

PersistenceSessionImpact records how much a queued persistence write affects the active session:

• None: background or cleanup work can retry without degrading the player session.
• SessionDegraded: the session may continue, but replay/repair attention is required if the write remains pending.
• DisconnectRequired: reserved for failures that should force disconnect; the current policy maps known origins to None or SessionDegraded.

persistence_session_policy.rs maps origins such as background location, session cleanup, session takeover, client appearance/equipment, gameplay state, combat rewards, and transfer activation into impact. New persistence origins should be added there, not by ad hoc impact values at call sites.

Session Lifecycle Journal

The session lifecycle journal uses the reusable Redis durable work queue. Its queue spec has pending, processing, and dead-letter lists under the session lifecycle keyspace.

The current work item is RealtimeStreamCleanup. It records:

• session token;
• player snapshot;
• realtime runtime profile;
• optional pending resume-ledger record;
• whether transfer activation was confirmed;
• a SessionLifecycleEffectPlan.

Realtime stream cleanup appends the journal work item before removing the active session. If append fails, the active session is left in place and the cleanup outcome carries the enqueue error. If append succeeds but the wake signal is lost, the worker still drains on its interval.

Cleanup Effect Plans

SessionLifecycleEffectPlan owns cleanup side effects:

• reconnectable cleanup persists the resume ledger, persists location, and refreshes presence;
• permanent removal cleanup deletes the resume ledger and persists location;
• confirmed transfer cleanup deletes the resume ledger and skips location/presence refresh.

When processing a cleanup item, the worker deletes the resume ledger for confirmed transfers or plans that request deletion. If the plan does not delete the ledger and the session token is active again, cleanup side effects are skipped so reconnects are not undone by stale work.

Redis Queue Mechanics

The generic durable queue stores DurableWorkQueueEntry<T> payloads. It uses:

• LPUSH to append pending work;
• RPOPLPUSH to move pending work into processing when leasing an entry;
• LREM to ack a processed payload;
• a Lua LREM plus LPUSH move to atomically retry or dead-letter processing entries.

Redis processing entries are not time-leased. Instead, spawn_session_lifecycle_journal_worker calls requeue_processing_entries on startup, moving any leftover processing entries back to pending.

Retrying a processing entry increments its attempts and moves it back to pending. After 8 failed attempts, the journal writes a DurableDeadLetterEntry with dead-letter id, original entry id, failed attempts, error message, timestamp, and item.

Inspect And Requeue

Operators can inspect and requeue dead letters through protected game-server internal routes. All routes are behind signed internal service auth.

Persistence retry dead letters use:

• GET /v1/internal/persistence-retry/dead-letters
• POST /v1/internal/persistence-retry/dead-letters/requeue

Inspection accepts an optional limit query parameter clamped to the server-side range. It returns the count of pending retry jobs, the count of active dead letters, and summaries with dead-letter id, retry job id, job kind, character id, realm id, zone id, failed attempts, error, timestamp, session token, source, and session impact.

Requeue requires the current internal RPC version and a non-empty UUID dead_letter_id. The server loads only active dead letters, decodes the saved PersistenceRetryJob, restores it to persistence_retry_jobs as a fresh retry job, stamps requeued_at plus requeued_retry_job_id, and signals the retry worker. If the id is missing or was already requeued, the response returns requeued: false.

Session lifecycle dead letters use:

• GET /v1/internal/session-lifecycle-journal/dead-letters
• POST /v1/internal/session-lifecycle-journal/dead-letters/requeue

Inspection returns queue depths plus summarized dead letters: dead-letter id, entry id, attempts, error, timestamp, session token, character id, and character name.

Requeue moves a matching dead letter back to pending with a new pending entry id and reset attempts, then wakes the worker. Requeue is safe only after the underlying Redis/Postgres/service issue has been fixed. Before requeueing, check whether the session token is active again, whether the work would delete a resume ledger, and whether transfer activation was already confirmed. The processing code skips some stale reconnect cleanup, but operators should still avoid requeueing blindly.

Operational Signals

Monitor:

• persistence retry rows growing or repeatedly leasing;
• persistence_job_dead_letters inserts;
• session lifecycle pending/processing/dead-letter depths in game metrics;
• durable-only queue-full or queue-closed log messages;
• cleanup outcomes with lifecycle enqueue errors.

Common recovery patterns:

1. Fix the underlying database, Redis, catalog, or zone-runtime issue.
2. Let leased SQL retry rows expire or restart the worker if necessary.
3. Requeue session lifecycle dead letters only after reading the summary and confirming the session impact.
4. For persistence dead letters, use the protected requeue endpoint only after confirming the saved payload is still the recovery action you want.

Related Pages

• Control Plane And Service Ops
• Session Handoff And Transfer