Shipping an Update

Roll a new build across a fleet without interrupting a match: picking a build version, the release panel, additive and clean installs, rollback, and what to do when a release fails.

A release moves every server in a fleet from one build to another, one server per location at a time, waiting for each server's match to end first. You start one from the release panel on the fleet page: open Fleets in the sidebar, click Manage on the fleet, and use the Build version picker above the member deployments. Starting a release requires fleets.manage; reading the panel requires fleets.view.

Before a fleet has ever shipped a release, the panel reads No release is running. Every server in this fleet is on build {buildVersion}., or Servers in this fleet are on more than one build: {list}. when the servers have drifted apart.

What a release does

A new release starts as pending and is picked up within about a minute. From there it runs in four phases and the panel shows which one it is in:

  • surging: the target build is pinned on every member deployment, then one extra server per deployment is started on the new build. Nothing is taken out of rotation yet.

  • rolling: old servers are retired one at a time per deployment. A retiring server stops accepting new matches and reports draining to Discovery, finishes the match it is running, then gets updated and checked.

  • completing: every old server is on the target build, so the spare servers from the surge are scaled down.

  • completed: the fleet is on one build.

A release that stops early sits in failed until you dismiss it, and the panel says which server stopped it.

A release also writes a pin, and the pin outlives the release. Until a fleet's first release, a deployment follows whatever the branch's image tag or the CDN's latest snapshot happens to be. On a CDN or Steam branch an unpinned server re-syncs to the newest build at the end of a match. On an image branch an unpinned deployment picks up a new tag only when a pod is recreated, so a scale-up can land on a different build from its neighbours. Once the pin is written, every restart and every new server produces the same build.

Before you start

  • The fleet must be active, its Discovery app enabled, and the app must have a live ingest token. A release is refused with fleet_not_connectable otherwise, and fails if any of the three disappears while it runs.

  • Every server in the fleet must be running a supervisor that reports its build. An older container is refused at creation with supervisor_too_old. Restart the server once on a current image and try again. A server PingCore cannot reach at all is refused with server_unreachable, which is a different problem: check the server is up first.

  • For image branches, push a new tag for every build. Tags in your brand's registry namespace are immutable, so a push that reuses an existing tag is refused. A release that targets a tag the fleet is already running also has nothing to do.

Shipping a release

  1. Open the fleet page and check the Build column on the member deployment rows. Every row shows the build its servers report. You should see: one build version repeated across the rows, or a short list if the fleet has drifted.

  2. Pick the target in Build version. The list is the builds PingCore can see for the branch behind this fleet (image tags, CDN snapshots, or Steam manifests). You should see: the picker showing the target you just pushed. If it is missing, see Troubleshooting.

  3. Open Advanced if you want to change the rollout shape, the grace period, or the time window. The defaults are one spare server and one retiring server per deployment. You should see: maxSurge 1 and maxUnavailable 1, with no time window set.

  4. Click Start release. PingCore checks the target exists before it writes anything: the image tag is resolved in your registry namespace, a CDN snapshot is checked against the content server, and a Steam target is checked against the branch's depot ids. You should see: the panel switch to the release progress rows, with the phase reading pending, then surging within about a minute.

  5. Watch the progress rows. Each member deployment shows how many servers are on the old build, how many are on the new one, and how many are retiring. You should see: one extra server per deployment reach ready on the new build, then the phase change to rolling.

  6. Leave it running. The release retires one server per deployment at a time, waits for its match to end, updates it, and verifies the new build is running before moving to the next. You should see: the old count fall and the new count rise, one server at a time, with waiting for server {id} to finish its session shown while a match is still running.

  7. Wait for the phase to reach completing. The spare servers from the surge are removed once every other server is on target. You should see: the release finish as completed, the spare servers gone, and the Build column showing the new build on every row.

The Build column on Discovery > Live Servers reports the same value per server, so you can watch the rollout there as well.

Picking a build version

What a build version is depends on the branch's data source:

  • Baked into container image: the image tag, without the registry or repository. A release patches the deployment to the new tag and recreates the pod, keeping the volume. See Baked-In Game Files and Pushing Images to the Registry.

  • CDN Source: the snapshot directory name, for example snapshot_20260917_120000. The snapshot is pinned, which also protects it from the content server's normal retention, on the master and on every child node. A source holds at most 4 pinned snapshots at once; a fifth pin is refused with too_many_pins. Finish or roll forward an older release to free one.

  • Depot Downloader (Steam): the manifest ids of the depots the branch installs, rendered as depotId=manifestId pairs. The branch must have its depot ids set, and you must pick a manifest for every depot it installs.

  • Branches with no data source have no build version, and cannot be released.

For CDN and Steam, the release does not recreate the pod. It asks the server to sync once its match has ended, which is why these releases keep the same container and the same volume.

What happens to a match in progress

A match in progress is never interrupted. A server is only updated after it reports that it has no active session:

  • The server is told to retire. It withdraws its joinable record so no backfill lands on it, then reports draining to Discovery so no matchmaker allocates it.

  • It finishes the match it is running. Recycling is suspended while it is retiring, so it does not restart itself back into rotation.

  • Only then is the update applied and verified.

If a release is cancelled or fails, every retiring server is released back into rotation.

Additive and clean installs

Every release is additive by default, in both directions. Files the new build contains are overwritten, and anything else on the volume is left alone. Worlds, saves, player databases and edited config survive a release, and they survive a rollback to an older build too.

A clean install is an explicit choice per release, and it requires a list of the paths to keep, relative to the game root. The list cannot be empty: a release with installMode set to clean and no paths is refused with clean_requires_preserve_paths. A clean install deletes everything under the game root that the target build does not contain and that your list does not name, so list your world and save directories before you confirm. The confirm dialog names how many servers would be reinstalled and the paths that are preserved.

Each preserved path must be a plain relative path with no spaces and no wildcard characters, so it means exactly the folder you typed. saves/session is valid. saves/* is refused with preserve_path_pattern. An absolute path such as /opt/game/saves, or one containing .., is refused as not being a relative path under the game root.

If you build your own image, keep world and save data out of the image tree. The supervisor copies the image tree onto the volume, so anything the image ships at the same path is overwritten on every build that changes it, and a clean install removes what the image does not carry. Write saves to a directory the image does not contain, and name that directory in the preserve list on any clean install.

Advanced settings

  • maxSurge: extra servers started per member deployment before anything is retired. Default 1, allowed 0 to 5. Use 0 when a location's Max is 1 and there is no room for a spare. With no spare, that location has no server while its one server updates, so expect a short outage there.

  • maxUnavailable: how many servers per deployment may be retiring at once. Default 1, allowed 1 to 5. Higher finishes sooner and takes more capacity out of rotation.

  • Grace period: how long a server in a session may keep running after it is asked to retire. Default 300 seconds, allowed 0 to 3600.

  • Time window: an optional UTC window (HH:MM to HH:MM, may wrap past midnight) during which the release is allowed to retire the next server. Outside it the release waits and says so.

A clean install and its preserve list are set on the release itself, described above.

The release also pauses the autoscaler's scale-down for the deployments it is updating, so the spare servers it created are not the first thing removed. Scale-up keeps running, so a crashed server is still replaced and the ready buffer is still refilled.

Persistent worlds and the grace period

A world server never ends a session, so it never reports that it is free. The grace period covers that case. Once the retire request has been in place for gracePeriodSeconds, the server counts as ready to retire whether or not players are connected. Set a grace period long enough for your players to notice, and set a time window if you want the update to land at a fixed hour.

PingCore does not announce the restart in game. There is no RCON announcement step in a release, so send your own warning from the game if you need one.

Rolling back

Roll back to {previousBuildVersion} on the panel starts a new release targeting the build the fleet was on before. A rollback is an ordinary release and behaves exactly like one, additive unless you choose a clean install.

Rolling back works because the previous build is still there. For image branches that is the old tag in your registry, so do not delete tags a fleet might need. For CDN branches it is the pinned snapshot, which the content server keeps past its retention count for as long as the pin exists.

When a release fails

The release stops at the first server that fails, and that one server is reverted to the build it was running. Other servers keep the build they already have, so the fleet ends on a mix, which the Build column shows. The deployment pins are left at the target, so a retried release does not have to pin again. No volume is deleted.

The fleet page carries a banner naming the failed server and the reason until you dismiss it. The button reads Dismiss and reclaim spare servers, and its text says how many spare servers that removes and that they are billed until then. If nobody dismisses it, the release is dismissed automatically after 6 hours and the spare servers are reclaimed by the normal autoscaler.

Cancelling a running release does the same thing: it stops, releases every retiring server back into rotation, and the fleet is left on a mix of builds.

Troubleshooting

The release failed with image_pull

  • Symptom: a server that was updating never becomes ready, and the release fails with image_pull naming that server.

  • Cause: the tag is not reachable from the cluster. It was never pushed to your brand's registry namespace, or the repository sits outside it.

  • Fix: confirm the repository is in your namespace, push the build under a new tag, then start a new release. The failed server has already been reverted to its previous image, so the fleet keeps serving throughout.

The release failed with verification_failed

  • Symptom: the updated server comes up healthy, then the release fails with verification_failed about two minutes later.

  • Cause: the server is not running the build that was asked for. On image branches the pod came up on a different reference from the target. On CDN and Steam branches the sync did not install the pinned version.

  • Fix: check the server's own logs for the pull or the sync, confirm the tag, snapshot or manifest is still published, then start a new release against a build you have just verified is there.

Start release is refused with snapshot_not_found

  • Symptom: picking a CDN snapshot returns That snapshot is no longer on the content server.

  • Cause: the content server deleted that snapshot under its retention policy. Only pinned snapshots survive past the retention count, and a snapshot is pinned only while a deployment or an active release points at it.

  • Fix: pick a snapshot the CDN still holds, or publish the build again and release the new snapshot.

Start release is refused with steam_depot_not_pinned

  • Symptom: picking a Steam build returns This branch has no depot pinned, so a specific build cannot be selected.

  • Cause: the branch does not list the depot ids it installs, so there is nothing to attach a manifest to. A related refusal, steam_manifest_map_incomplete, means the manifest map missed a depot the branch installs.

  • Fix: set the depot ids under Workspace > Games, open the game, Branches, edit the branch, Data Configuration, then pick a manifest for each of those depots and start the release again.

The release sits on one server

  • Symptom: the progress row reads waiting for server {id} to finish its session and stays there for minutes at a time.

  • Cause: that server's session has not ended. This is normal for a persistent world, and it also happens for a server with no SDK integration, which counts as in a session until it restarts.

  • Fix: the release waits up to the grace period (300 seconds by default) before it retires a server that is still in a session, so a wait of a few minutes per server is expected. Raise or lower the grace period under Advanced on the next release, or set a time window there so the interruption lands at an hour you choose. A release started with a grace period of 0 waits for the session itself, which on a persistent world never ends.