Fleet Observability

Read what your fleet is actually doing: the live workspace view, the fleet-state and matchmaking-stats APIs, and how to judge autoscaler health.

The fleet page in your workspace shows the fleet's configuration next to its live state as reported by the Discovery service, and the same live data is available over the API for your own tooling.

The fleet page

Open Hosting > Fleets in the sidebar and click Manage on a fleet (requires fleets.view). The page refreshes silently every 15 seconds while visible.

Live State panel

  • Live Servers: everything Discovery currently sees for this fleet, across all lifecycle states.

  • Ready: idle, allocatable servers, shown against the target: the summed Ready Buffer of all member deployments. Ready at or above the target means the warm buffer is healthy. Ready persistently below it means the autoscaler is still catching up or a location is pinned at its Max.

  • In Session: servers currently running matches.

  • Draining: servers that are out of rotation (booting, unhealthy, retiring for a release, or shutting down). They are not allocatable, and players do not see them on the public server list.

  • Desired Ready: the matchmaker's TTL'd demand hint, or "no active hint" when it has expired or was never written.

Each member deployment row also carries its own live lifecycle counts next to its scaling policy, so you can spot one location lagging while the others are healthy. Under the counts, the row shows:

  • The players capacity its game servers advertise and where it comes from, for example 16 players (deployment player count) or 32 players (agent settings). A warning appears when a game server already holds more players than that capacity, because every backfill onto it is refused.

  • Scale-up blocked by ... when adding game servers failed on a workspace resource limit, with the limit and its usage. The autoscaler retries after about 10 minutes.

  • An alert when at least half of the deployment's live game servers are missing from Discovery, for example 3 of 4 live game servers are absent from Discovery, with the likely causes. Scale-up beyond Min is held while it shows. See How the autoscaler decides.

Matchmaking panel

  • Queued Tickets: tickets waiting in the first-party matchmaker.

  • Pending Allocations: allocations claimed but not yet confirmed by a session start.

  • Recent allocations: the newest allocations (at most 25), each with its allocation id, server, and age. The list is diagnostic only; entries expire on their own.

The most common under-capacity signal is rising Queued Tickets with no Ready servers anywhere: raise Ready Buffer or Max on the relevant locations, or have your matchmaker write a higher desired-capacity hint.

Unmapped servers

The Live State panel warns when servers are live in Discovery that no platform game server explains. These are usually deleted game servers still draining out of the TTL, or external servers sharing the fleet's Discovery app (anything heartbeating with the same app's tokens). A persistent entry means something outside this fleet is using the linked app.

When live data is unavailable

The panel can show "Live data unavailable" while the rest of the page keeps working. That means the live layer could not be read this cycle: the Discovery service is unreachable, the linked app is disabled, or the app has no active ingest token. The fleet itself is fine; its configuration is intact and the panel retries automatically. Do not diagnose the fleet from a live-data outage.

The same applies to the API: the platform's fleet live-state endpoint answers 503 in that situation, and the fleet's configuration endpoints keep answering normally.

Session logs

Every time a session ends on a fleet server, the server writes a compressed slice of its game and application logs for that session and uploads it to PingCore. The slices are also drained before a server is removed, so the logs of a server the autoscaler scaled down or a release replaced are still readable afterwards.

  • Find them in the session log list on the server's own page, and in a collapsed list on the fleet page. Both require brand.servers.view.

  • Each entry carries the time the session ended, its allocation id and session id when the server reported them, and a download link for the gzip file.

  • A slice holds the tail of the logs, up to 4 MiB uncompressed, and a single uploaded archive may not exceed 8 MiB. A server keeps at most 10 slices locally before the oldest is dropped, and at most 40 MiB of stored archives on the platform.

  • Archives are deleted after 7 days. Your brand can set a different retention between 1 and 30 days.

  • When a server's pod crashed rather than exiting cleanly, PingCore stores the previous container's log tail instead, so the crash is still visible.

Slices are best effort. A server that could not upload before it was removed logs one line and the removal continues, so a missing slice never blocks a scale-down or a release.

Reading live state over the API

For dashboards and alerting you have two options.

Through the platform

GET /api/fleets/{fleetId}/live on the platform API (requires fleets.view) returns the same aggregation the fleet page renders: per-member-deployment lifecycle counts with per-server detail, unmapped servers, app-wide lifecycle state counts, the desired-ready hint, and the matchmaking stats. Discovery credentials never leave the platform; this is the recommended path for workspace tooling.

Each member deployment entry also carries the signals the rows show:

  • absent and absentServerIds: how many of the deployment's live game servers Discovery does not report, and which.

  • alert: the absence alert text when at least half of them are absent (scale-up is held), otherwise null.

  • scaleUpBlocked: null, or the active resource-limit block as reason (brand_memory_limit, brand_cpu_limit or brand_server_limit), detail, message, and recordedAt and retryAt in Unix seconds.

Directly from Discovery

Your own backend can read the raw projections from the Discovery service. On a private app any active token of the app works. On an open-registration app you need an Allocate (or Both) token: the in-game Heartbeat token ships inside the game binary, so it gets a 403 with reason: "scope" on both reads.

GET /v1/apps/{publicId}/fleet-state
{
  "error": false,
  "servers": [
    {
      "serverId": "4211",
      "tier": "agent",
      "state": "ready",
      "counters": { "sessions": { "count": 0, "capacity": 1 } },
      "pendingClaims": null,
      "updatedAt": 1756500000000,
      "name": "Fleet 4211",
      "ip": "203.0.113.9",
      "port": 27015,
      "players": 3,
      "maxPlayers": 16,
      "version": "1.0.0",
      "verified": "unverified",
      "lastProbeError": "No UDP echo response before timeout",
      "reason": null
    }
  ],
  "counts": { "total": 1, "ready": 1, "allocated": 0, "inSession": 0, "draining": 0, "external": 0 },
  "desiredReadyServers": 12
}

One entry per live server. tier distinguishes fleet servers (agent) from plain heartbeat servers; state is the lifecycle state (heartbeat-tier servers read external); pendingClaims shows capacity claimed by allocations not yet confirmed; updatedAt is the last state assertion in epoch milliseconds; reason says why a draining server is out of rotation (booting, unhealthy, retiring or shutdown). Fields are only ever added, never removed or renamed, so a tolerant reader stays compatible across service updates.

verified and lastProbeError report the reachability probe on both tiers. They are null ("not probed") when the app's verification mode is No verification. Otherwise verified is pending until the first probe, then verified or unverified, with the failure in lastProbeError. For a hosted (agent) game server the result is diagnostic only: the example above is a hosted game server that is still listed for players although its game port did not answer the probe. See Reachability Verification.

GET /v1/apps/{publicId}/matchmaking-stats
{
  "error": false,
  "queuedTickets": 4,
  "pendingAllocations": 1,
  "desiredReadyServers": 12,
  "desiredReadyServersByLocation": { "eu-west-ams": 8, "us-east-nyc": 3 },
  "recentAllocations": [
    { "allocationId": "…", "serverId": "4211", "createdAt": 1756500000000 }
  ]
}

recentAllocations is newest first, capped at 25, and purely diagnostic. desiredReadyServers is null in both endpoints when no hint is currently active.

desiredReadyServersByLocation splits the hint by location: how many ready game servers are wanted in each location id from GET /v1/locations, including a location where no game server runs yet. The first-party matchmaker fills it whenever it writes the hint. Matches with no latency map and no pinned location count only in desiredReadyServers, so the values add up to at most that number. Locations with no demand are left out. It is null exactly when desiredReadyServers is null, and a hint you write yourself with PUT .../desired-capacity has no split and clears it.

Discovery app live view

Outside of fleets, every Discovery app page shows a live server table for the app (states, verification results, player counts), refreshed on the same 15 second cadence. For heartbeat-only integrations that is usually all the observability you need; the heartbeat response's verified and lastProbeError fields cover per-server diagnosis, as described in Reachability Verification.

Matchmaking card on the Discovery app page

Under Live Servers on the app page sits a Matchmaking card (requires discovery.view), refreshed every 15 seconds while visible. Its header badge counts the tickets currently queued. It has three tables and, when any exist, a fourth.

  • Queues: one row per queue with queued tickets. Queued is the ticket count, By location shows how many of them accept each location as chips (eu-west-ams: 3), Join-in-progress counts tickets that opted into backfill, Oldest wait is the longest wait in the queue, and Joinable sessions and Open seats summarise the joinable records published for that queue.

  • Joinable sessions (shown when any record is live): Server, Queue, Open seats as effective / published (the effective figure is the published openSeats minus joiners already matched but unconfirmed minus seats held by reservations on the same server; the matcher sells against the effective figure), Pending joiners, Publisher (backend for the HTTP route, agent for a hosted server's own record), Allocatable, Delivered to agent, and Expires.

  • Tickets: the queued tickets, oldest first, at most 100. Party and Session (partySize and minSessionSize..sessionSize), Owner (Backend, or Player followed by a 12-character hash of the player id; the raw id never reaches the workspace and a ticket's context is never shown), Waiting, Flags (relaxed once the ticket's relaxation delay has elapsed, join-in-progress), Attributes as key=value chips, and Best latency (the lowest entry in the ticket's latency map with its location).

  • Recent matches: matches formed in the last 24 hours, newest first, at most 50. Players as players / sessionSize, Tickets, Location, Server, Allocation, Flags (relaxed, backfill), and Wait (how long the oldest ticket in the match waited). Matches older than 24 hours drop off on their own; there is no longer history.

By location is the counter to read when constrained tickets sit queued. A ticket with a maxLatencyMs ceiling counts once under every location it accepts; a ticket with no ceiling counts under the Unconstrained chip, because it can be placed anywhere. A location with queued tickets but no ready servers is a fleet capacity gap: the desired-capacity hint the matchmaker writes is one number for the whole app and cannot say which zone is short, so this table is where you see it. The fix is on the fleet page: raise Min (or Ready Buffer) on the member deployment in that location.

Delivered to agent has three readings:

  • yes: backfills pushed to this server are being acknowledged by its PingCore agent.

  • A dash: no backfill has been pushed to this server since the record was published, so nothing can be said yet.

  • no: a backfill was pushed and has not been acknowledged. Either the server has no PingCore agent (an HTTP-published record on a server you host elsewhere: poll GET .../joinable-session or GET .../allocations/{id} to learn the joiners), or it runs an older game supervisor that does not deliver backfills to the game (move it to a current image; see Supervisor Base Images). On a current hosted game server the game reads its joiners from GET /v1/backfills on the local SDK; see Making Your Game Fleet-Ready. When any row reads no, a warning under the table repeats this and links to the joinable-sessions section of Matchmaking Integration.

When nothing is queued and no match has formed in 24 hours, the card shows one pre-filled ticket curl for your app (masked token on screen, the real one on copy, when an Allocate-scope token was issued in this page session) that demonstrates a relaxed match: run it twice with different ticket ids and the match forms on the next tick once a fleet server is ready.

The card reads Live data unavailable when the platform could not reach the Discovery service this cycle, and retries automatically; the app's configuration is unaffected. On an open-registration app it can also read Matchmaking cannot be read with this app's tokens: an open-registration app needs an Allocate-scope token for the inspector. Issue one in the Heartbeat Tokens section. The heartbeat token of an open app ships inside the game binary, so it cannot read the queue; the platform uses the app's strongest token (Allocate or Both first) to read the card. With no active token at all the card reads Matchmaking cannot be read: this app has no active token. Issue one in the Heartbeat Tokens section and this card starts working automatically.

The card reads three Discovery endpoints your own tooling may call too, each answering generatedAt in epoch milliseconds:

  • GET /v1/apps/{publicId}/matchmaking/queues: the queue rows (queue, queuedTickets, joinInProgressTickets, oldestWaitSeconds, joinableSessions, openSeats, queuedByLocation keyed by location with "*" for tickets with no ceiling), totalQueuedTickets, truncated, and the joinableSessions rows (serverId, queue, openSeats, effectiveOpenSeats, pendingJoiners, sessionSize, publisher, allocatable, deliveredToAgent as true, false, or null, location, publishedAt, expiresAt, and lastBackfillRefusal). lastBackfillRefusal is null, or { reason, at } naming why the last backfill onto that record did not go through (counter_full, server_not_in_session, seats_unavailable, no_capacity, or agent_push_unavailable); a republish clears it. The reasons are explained under Troubleshooting on Matchmaking Integration.

  • GET /v1/apps/{publicId}/matchmaking/tickets?queue=ranked&limit=100: the redacted ticket rows (ticketId, queue, sessionSize, minSessionSize, relaxAfterSeconds, partySize, joinInProgress, ownerKind, playerIdHash, waitSeconds, relaxed, attributes, rules, latency, maxLatencyMs, filters, createdAt), oldest first. limit is 1 to 100; queue is an optional exact filter. Ticket context is never included.

  • GET /v1/apps/{publicId}/matchmaking/matches?limit=50: the recent matches (matchedAt, queue, sessionSize, players, tickets, relaxed, backfill, location, serverId, allocationId, anchorWaitSeconds, maxWaitSeconds), newest first, with no ticket ids or player data.

On a private app any active token of the app reads them; on an open-registration app only an Allocate (or Both) scope token does, and a heartbeat token answers 403 with reason: "scope". The three routes share their own per-app budget of 120 requests per minute by default, separate from the allocation budget, so polling the inspector never competes with your matchmaker's allocation traffic.