Ryuu/AetherOnline
Watch
1
0
Fork
You've already forked AetherOnline
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Page: control plane and service ops
Pages
Authoring Debugging Home Operations Protocol Reference Runtime Systems abilities loadouts and progression ability rotation and combo planner active realtime session actor directory admin actions and internal auth animation workbench operator guide aoi snapshot replication app mode bootstrap and lifecycle architecture guardrails and module tree tests asset loading routing and vfs authored ui surface renderer authoring command bus and transaction history authoring reference registry contributor extension contract authoring reference registry authoritative audit event schema and projections authoritative world time calendar and environmental scheduler bake product registry build strategy and loadout optimizer campaign journey and progression simulator catalog source map and record routing character gameplay state json store character paperdoll assets character stats and equipment modifiers chat presence and social client local offline save and bootstrap cache client local user preferences and account sync client multiplayer connection flow and session cleanup client realtime authority store and frame projection client scene presentation and player preview client snapshot staging and repair receipts client ui focus and text input planning combat action resolution and mob ai semantics combat presentation cues and transient effects combat rewards and progression persistence content dependency graph and scenario playtests content migration and catalog refactors control agent api and remote service runner control panel config and secret persistence control panel service build invalidation and log normalization control panel service lifecycle and quality gates control plane allocation and zone transfer routing control plane and service ops control plane route policy and coordination api control service supervisor manifest and stale process recovery crafting recipes and service integration dependency aware content health deterministic combat balance lab developer workspace doctor and readiness report diagnostics provider input lanes and failure isolation dialogue quest localization handoff dialogue runtime state and choice authority duplicate realtime claim takeover durable retry and session cleanup durable work queue consumer contract dynamic world state and scene phasing editor authored content workspace manifest and cache editor event console and asset debug dashboards editor extension and contribution registries editor scene snapshot lane demand editor settings registry and workspace presets editor viewport and world tools editor workbench and dock encounter difficulty mutators and reward scaling encounter instance and wave director encounter lobby ready check and entry encounter mechanics and boss timelines encounter memory persistence encounter reward eligibility lockouts and pity enemy archetypes combat kits and encounters engine report log and recovery routing environment configuration reference external engine patterns and boundaries farming footstep surface audio materials forgejo wiki publishing and docs check game data catalog game data generated item family adapters game database schema readiness game server catalog load health and fallback policy game server reconciliation and transfer marker cleanup game server startup readiness and registration game transport endpoints and quic exposure gameplay action authority and command audit gameplay action replay ledger and idempotency gameplay authority replay fixtures and drift gates gameplay conditions effects and events gameplay system slice scaffold gameplay ui runtime and bevy sync generated artifact drift and repair generated artifact registry and quality gate integration glossary hotbar hud runtime projection and input in process zone transfer phase journal input profiles and keybinds interactive world map minimap and cartography internal auth and admin delegation chain inventory items and stack limit safety item authoring lifecycle and reference repairs json catalog hot reload batch and candidate revalidation legacy compatibility and migration register local player input control and reconciliation local playtest and standalone preview localization extraction and runtime catalog login account admin repository and cascades login database migrations and schema contracts login server auth sessions and character entry login session cache throttle and revocation loot table authoring and drop audits mob animation presentation events and projectile cues mob encounters and scene combat movement authority networking tick health and zone pressure npc ai schedules and deterministic behavior npc definition catalog npc schedules routines and contextual behavior npc services npc state memory and interactions pack export content gate party composition roles and synergy simulator passive traits and procs persistent interactive world objects placed light runtime budget and preview heatmap player appearance equipment persistence player economy authority player homesteads housing and property auctions player journal and codex player navigation tracking and world guidance player notification inbox authority and delivery player preview animation profile player preview debug gizmos and collision overlays player preview render cache and keyed entity lifecycle player preview weather masks and effect budget player snapshot assembly and durable row normalization post process volume blending and preview overlays project files import profiles and asset change impact public endpoint detection and upnp exposure quality gate jsonl events and panel failure summaries quest runtime projection and turn in authority quic reliable stream topology and negotiation realtime admission throttling and load shedding realtime bootstrap manifest and snapshot service realtime bootstrap wire sections and compression realtime client reconnect journal and command replay realtime datagram codecs and ingress policy realtime egress mailboxes and lane scheduling realtime ingress validation and failure policy realtime packet debug capture and triage realtime protocol registry realtime registry and wire contract realtime resume ledger record format realtime rollout policy and compatibility negotiation realtime runtime profile and transport adaptation realtime session activation hydration boundary realtime timebases tick ordering and clock ownership redis keyspace ttls and cleanup targets regions points of interest and exploration discovery relationship reputation gates and service deltas remote avatar replica binding and interpolation render surface guard and window safety runtime actor identity and authority contract runtime camera arrival and cutscene overrides runtime debug overlays and developer options runtime environment registry and startup config validation runtime path roots and packaged asset resolution runtime profiling tracy captures and budget gates runtime scene manifests and zone authoring runtime ui fonts and localized text presentation scenario sandbox and timeline inspector scene audio runtime routing scene command center and visual graph tools scene cutscene authoring scene documents scene layer schema depth and bands scene lighting post process and placed lights scene navigation and collision scene preview scenario replay cli scene runtime media weather and camera phases scene runtime schedule and frame budget scene trigger authority and reconciliation scene triggers scene validation and repairs service json rpc contracts service observability and metrics contract service registry heartbeats drain and shutdown session handoff and transfer shared content workspace shared test fixtures and fixture workspaces shared validation report schema source inventory and engine health standalone account realm and character flow standalone asset pack and preflight standalone loading pause death and recovery screens standalone shell flow and ui lifecycle standalone ui preview root and transient lifecycle starter character economy bootstrap status effects and combat modifiers story content loop text asset migration journals and atomic rollback tiled prefab import and placement tilemap asset import contracts tileset atlas preview surfaces trigger permissions targeting and action pipelines typed disconnect codes and transport advisories ui layout binding graph and editor workflow ui layout ui skins validation and bake validation errors validation provider registry and strict preflight vendor stock and shop economy authoring viewport spatial planning overlays vscode dev tasks and generated launch profiles workspace package roles and scoped quality profiles world entry smoke harness and soak telemetry world events and scenario director world gameplay assemblies world interactions and activity boards world mechanism and puzzle graphs world overlay gizmos and culling world travel waypoints and zone transitions xtask content gates and plans
1 control plane and service ops
forgejo-actions edited this page 2026-07-06 16:45:10 +00:00
Table of contents

Control Plane And Service Ops

Purpose

The control-plane, control-agent, and control-panel tools give local and LAN operators a shared way to start services, inspect health, register live instances, allocate game servers, and debug handoff or zone-transfer routing.

The ownership split is simple:

  • The control-plane owns fleet records, allocation decisions, handoff records, and transfer activation state.
  • The control-agent owns process supervision on one host.
  • The control-panel polls services and agents, starts or stops local and remote managed services, and signs internal operator requests.

Startup Order

Start infrastructure first. The control-plane requires Redis for readiness. Login and game services require their configured database, Redis, and control-plane connection.

For managed local startup, use this order:

  1. Control-plane
  2. Login server
  3. Game server

Login and game both depend on the control-plane. The local panel process launcher waits for each dependency's /ready endpoint before starting the next service and stops services in the reverse order.

The control-agent may be started before or after the control-plane because it is only a host supervisor. When using an agent to start remote services, start control-plane before login or game; the agent API starts the requested service directly and does not apply the panel's dependency order for you.

The control-panel can start at any time. Until services are reachable, it will show polling errors rather than owning recovery.

Allocation Ownership

The control-plane exposes protected registry and allocator routes for instances, presence, game assignment, zone-transfer assignment, handoff prepare/preflight/consume/redirect, and transfer activation confirmation/status.

Services publish and delete InstanceRecord data through the control-plane. Redis stores instance registry entries with a short TTL, so stale instances naturally fall out when services stop heartbeating or deregister.

Game allocation loads current game instance records, keeps the latest incarnation per placement, rejects records that do not match realm or zone, rejects draining or full targets, rejects unavailable zone pressure, and requires healthy allocation endpoints. When more than one target is valid, the allocator prefers region match, healthier endpoints, healthier zone pressure, already populated zones, tighter available capacity, and stable endpoint ordering.

Zone-transfer allocation requires a canonical scenes/*.scene target, excludes the current instance, resolves hosted zone scene data from workspace assets, checks the requested entry point, and only returns a target that passes admission. Invalid scene references return a bad request; no healthy target returns service unavailable.

Do not make the agent or panel the authority for allocation. They can start processes and send requests, but the control-plane is the allocator.

Remote Process Management

The control-agent serves:

  • GET /health
  • GET /ready
  • GET /v1/services
  • GET /v1/services/{service}/logs
  • POST /v1/services/{service}/start
  • POST /v1/services/{service}/stop
  • POST /v1/services/{service}/restart

Set AETHER_AGENT_WORKSPACE_ROOT to the workspace the agent should build and run from. AETHER_AGENT_BIND defaults to 0.0.0.0:7300; AETHER_AGENT_NAME defaults to the host name; AETHER_AGENT_TOKEN enables bearer-token auth. Do not expose a LAN agent without a token and firewall rules.

On start, the agent runs cargo build -p <service package> in the workspace, launches the debug service binary, records child pid and start time, and streams stdout/stderr into an in-memory log buffer capped at 500 lines. On stop, it tries the shared process-tree terminator first, then falls back to killing the child process. Status refresh records process exits and process status errors in the service snapshot.

Configure environment on the host that runs the service. Remote agent start requests select the service; they do not carry a full per-service environment payload from the panel.

Internal Request Signing

All protected control-plane APIs require signed internal-service headers. The accepted identities are login-server, game-server, and control-panel.

The signature covers method, path and query, internal service identity, timestamp, nonce, and body SHA-256. The timestamp skew window is 300 seconds. Nonces are registered in Redis under the control-plane nonce scope with SET ... EX ... NX; replayed nonces are rejected.

Common failure modes:

  • Missing, stale, disallowed, or bad signatures return unauthorized.
  • Body read failures return bad request.
  • Redis failure during nonce registration returns internal server error.
  • Client helpers reject non-HTTPS control-plane URLs unless insecure HTTP is explicitly allowed.
  • Public or LAN deployments must not use the default development AETHER_CONTROL_AUTH_SECRET.

The panel uses the shared signing helpers for control-plane reads and internal JSON posts, so panel requests should fail the same way login and game server requests fail.

Panel Polling

The control-panel default poll interval is 3 seconds. Polling runs with a blocking HTTP client and a 2 second timeout.

Each poll:

  • Probes service health, readiness, info, and metrics according to the service descriptor.
  • Reads control-plane instances and presence through signed GET requests.
  • Polls each configured control-agent at /v1/services, using bearer auth when the agent config has a token.
  • Attempts external endpoint discovery for the network status panel.

A service reachability error is shown only when that service is expected to support health or readiness and both probes fail. A signed control-plane read failure is reported separately as a monitor error for instances or presence.

Health And Readiness

Control-plane /health is an unconditional service liveness response. Control-plane /ready pings Redis and fails if Redis cannot be reached or does not return PONG.

Control-agent /health is an unconditional liveness response. Control-agent /ready fails if the configured workspace root no longer exists. Agent startup also fails immediately when the workspace root is missing.

Service start failures usually show in the agent action response and logs. Build failures return internal server error from the agent and keep the service stopped. Runtime exits appear in /v1/services as last_exit_status and in the log buffer as process-exit lines.

Allocation failures should be debugged from the control-plane side. A 503 from game allocation means no healthy registered game instance currently satisfies realm, zone, endpoint, draining, pressure, and capacity checks.

Local And LAN Workflows

Local defaults are:

  • Login: 127.0.0.1:7000
  • Game: 127.0.0.1:7100
  • Control-plane: 127.0.0.1:7200
  • Local agent: 127.0.0.1:7300

Local HTTP is allowed by default for control-plane clients. The shared development control secret is acceptable only for local development.

For LAN workflows:

  1. Set a non-default AETHER_CONTROL_AUTH_SECRET on control-plane, login, game, and panel environments.
  2. Use HTTPS for control-plane traffic, or set AETHER_CONTROL_ALLOW_INSECURE_HTTP=true only for trusted development LANs.
  3. Start a control-agent on each host with AETHER_AGENT_WORKSPACE_ROOT and AETHER_AGENT_TOKEN.
  4. Add each agent base URL and token to the panel config.
  5. Open only the required service, realtime, QUIC, control-plane, and agent ports.
  6. Keep host clocks in sync so signed request timestamps stay inside the 300 second skew window.

When a LAN panel cannot see a service, check in this order: panel config URL, firewall, agent token, service /health, service /ready, control-plane signed reads, and then allocator state.

Related Pages