Ryuu/AetherOnline
Watch
1
0
Fork
You've already forked AetherOnline
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Page: service observability and metrics contract
Pages
Authoring Debugging Home Operations Protocol Reference Runtime Systems abilities loadouts and progression ability rotation and combo planner active realtime session actor directory admin actions and internal auth animation workbench operator guide aoi snapshot replication app mode bootstrap and lifecycle architecture guardrails and module tree tests asset loading routing and vfs authored ui surface renderer authoring command bus and transaction history authoring reference registry contributor extension contract authoring reference registry authoritative audit event schema and projections authoritative world time calendar and environmental scheduler bake product registry build strategy and loadout optimizer campaign journey and progression simulator catalog source map and record routing character gameplay state json store character paperdoll assets character stats and equipment modifiers chat presence and social client local offline save and bootstrap cache client local user preferences and account sync client multiplayer connection flow and session cleanup client realtime authority store and frame projection client scene presentation and player preview client snapshot staging and repair receipts client ui focus and text input planning combat action resolution and mob ai semantics combat presentation cues and transient effects combat rewards and progression persistence content dependency graph and scenario playtests content migration and catalog refactors control agent api and remote service runner control panel config and secret persistence control panel service build invalidation and log normalization control panel service lifecycle and quality gates control plane allocation and zone transfer routing control plane and service ops control plane route policy and coordination api control service supervisor manifest and stale process recovery crafting recipes and service integration dependency aware content health deterministic combat balance lab developer workspace doctor and readiness report diagnostics provider input lanes and failure isolation dialogue quest localization handoff dialogue runtime state and choice authority duplicate realtime claim takeover durable retry and session cleanup durable work queue consumer contract dynamic world state and scene phasing editor authored content workspace manifest and cache editor event console and asset debug dashboards editor extension and contribution registries editor scene snapshot lane demand editor settings registry and workspace presets editor viewport and world tools editor workbench and dock encounter difficulty mutators and reward scaling encounter instance and wave director encounter lobby ready check and entry encounter mechanics and boss timelines encounter memory persistence encounter reward eligibility lockouts and pity enemy archetypes combat kits and encounters engine report log and recovery routing environment configuration reference external engine patterns and boundaries farming footstep surface audio materials forgejo wiki publishing and docs check game data catalog game data generated item family adapters game database schema readiness game server catalog load health and fallback policy game server reconciliation and transfer marker cleanup game server startup readiness and registration game transport endpoints and quic exposure gameplay action authority and command audit gameplay action replay ledger and idempotency gameplay authority replay fixtures and drift gates gameplay conditions effects and events gameplay system slice scaffold gameplay ui runtime and bevy sync generated artifact drift and repair generated artifact registry and quality gate integration glossary hotbar hud runtime projection and input in process zone transfer phase journal input profiles and keybinds interactive world map minimap and cartography internal auth and admin delegation chain inventory items and stack limit safety item authoring lifecycle and reference repairs json catalog hot reload batch and candidate revalidation legacy compatibility and migration register local player input control and reconciliation local playtest and standalone preview localization extraction and runtime catalog login account admin repository and cascades login database migrations and schema contracts login server auth sessions and character entry login session cache throttle and revocation loot table authoring and drop audits mob animation presentation events and projectile cues mob encounters and scene combat movement authority networking tick health and zone pressure npc ai schedules and deterministic behavior npc definition catalog npc schedules routines and contextual behavior npc services npc state memory and interactions pack export content gate party composition roles and synergy simulator passive traits and procs persistent interactive world objects placed light runtime budget and preview heatmap player appearance equipment persistence player economy authority player homesteads housing and property auctions player journal and codex player navigation tracking and world guidance player notification inbox authority and delivery player preview animation profile player preview debug gizmos and collision overlays player preview render cache and keyed entity lifecycle player preview weather masks and effect budget player snapshot assembly and durable row normalization post process volume blending and preview overlays project files import profiles and asset change impact public endpoint detection and upnp exposure quality gate jsonl events and panel failure summaries quest runtime projection and turn in authority quic reliable stream topology and negotiation realtime admission throttling and load shedding realtime bootstrap manifest and snapshot service realtime bootstrap wire sections and compression realtime client reconnect journal and command replay realtime datagram codecs and ingress policy realtime egress mailboxes and lane scheduling realtime ingress validation and failure policy realtime packet debug capture and triage realtime protocol registry realtime registry and wire contract realtime resume ledger record format realtime rollout policy and compatibility negotiation realtime runtime profile and transport adaptation realtime session activation hydration boundary realtime timebases tick ordering and clock ownership redis keyspace ttls and cleanup targets regions points of interest and exploration discovery relationship reputation gates and service deltas remote avatar replica binding and interpolation render surface guard and window safety runtime actor identity and authority contract runtime camera arrival and cutscene overrides runtime debug overlays and developer options runtime environment registry and startup config validation runtime path roots and packaged asset resolution runtime profiling tracy captures and budget gates runtime scene manifests and zone authoring runtime ui fonts and localized text presentation scenario sandbox and timeline inspector scene audio runtime routing scene command center and visual graph tools scene cutscene authoring scene documents scene layer schema depth and bands scene lighting post process and placed lights scene navigation and collision scene preview scenario replay cli scene runtime media weather and camera phases scene runtime schedule and frame budget scene trigger authority and reconciliation scene triggers scene validation and repairs service json rpc contracts service observability and metrics contract service registry heartbeats drain and shutdown session handoff and transfer shared content workspace shared test fixtures and fixture workspaces shared validation report schema source inventory and engine health standalone account realm and character flow standalone asset pack and preflight standalone loading pause death and recovery screens standalone shell flow and ui lifecycle standalone ui preview root and transient lifecycle starter character economy bootstrap status effects and combat modifiers story content loop text asset migration journals and atomic rollback tiled prefab import and placement tilemap asset import contracts tileset atlas preview surfaces trigger permissions targeting and action pipelines typed disconnect codes and transport advisories ui layout binding graph and editor workflow ui layout ui skins validation and bake validation errors validation provider registry and strict preflight vendor stock and shop economy authoring viewport spatial planning overlays vscode dev tasks and generated launch profiles workspace package roles and scoped quality profiles world entry smoke harness and soak telemetry world events and scenario director world gameplay assemblies world interactions and activity boards world mechanism and puzzle graphs world overlay gizmos and culling world travel waypoints and zone transitions xtask content gates and plans
1 service observability and metrics contract
forgejo-actions edited this page 2026-07-11 12:38:13 +00:00
Table of contents

Service Observability And Metrics Contract

Service observability covers health, readiness, info, metrics, tracing, and the control-panel telemetry history that derives rates from raw counters. The contract is shared through common-proto service status and metrics types.

Purpose

Use this page when adding a service metric, debugging a missing control-panel graph, or deciding which service should expose /metrics.

Endpoint Capability Matrix

Service descriptors declare which status endpoints a service supports. Health and readiness are baseline operational endpoints. Metrics support is service-specific; the control panel should consult endpoint capabilities instead of assuming every service exposes every endpoint.

The control plane currently has a metrics response type in the shared protocol, but not the same live metric surface as login and game.

Login Metrics

Login server metrics include request rate, auth failure rate, token issuance rate, database latency, and Redis latency. The login support layer measures DB/Redis calls and records auth failures through audit contexts.

If login metrics are zero during traffic, inspect middleware and measurement wrappers before blaming the panel.

Game Metric Families

Game metrics include active and pending realtime sessions, AOI enters/leaves, visible entity counts, full and delta snapshots, rate-limit decisions, packet-validation decisions, move-input transport decisions, movement authority decisions/backlog, trigger authority decisions, QUIC transport decisions/samples, queue depths, tick health, and session lifecycle journal depths.

Dynamic metric maps must use descriptor-generated keys so panel deserialization remains forward-compatible.

Session Lifecycle Journal Metrics

The game /metrics snapshot also reads session lifecycle journal pending, processing, and dead-letter depths from Redis. If those fields are missing or stale, inspect Redis access from the metrics endpoint rather than only the worker.

Control-Panel Derived Rates

The control panel derives per-second series from monotonically increasing counters using CounterRateTracker. It also keeps rolling telemetry for health, active sessions, AOI, snapshot rates, trigger authority, and packet validation.

Do not change counter semantics to gauges without updating panel derivation.

Tracing And RUST_LOG

Services initialize tracing through aether_observability. Use RUST_LOG or the service's logging config to raise detail during incident debugging. Metrics should answer "what is happening"; traces should answer "which path did it take".

Adding Metrics Safely

Add a descriptor or typed field, update the service snapshot, update control-panel deserialization/visualization if needed, and add tests for dynamic-key compatibility. Keep labels stable; dashboards and alert rules depend on them.

Related Pages