Skip to main content

Admin backend operations runbook

A practical reference for calling admin backend operations directly — from an API client, a test, or a script. It assumes you're already authenticated as an admin or super-admin.

This is engineer-facing. If you're looking for how to do the job through the admin screens, the operator handbook is written for that, in plain language.

The five checks before any mutating call​

Run through these every time. They take seconds and they prevent the failures that are painful to unpick.

  1. Check the role is sufficient — admin, super-admin, or the cash-out-approval permission. Getting this wrong gives you a 403, which is the harmless outcome. The point is knowing what you're invoking.
  2. Refresh the record immediately before acting. Many commands require an expected state or expected balances, and they compare against what you send. Stale input gets rejected, which is the system working correctly.
  3. Send an Idempotency-Key for anything touching money or durable state. Reuse the same key only when retrying the exact same request. A new intention needs a new key.
  4. Write a real reason wherever the schema asks for one. These become audit evidence that somebody reads months later, possibly during an investigation. "test" is not a reason.
  5. Upload and finalize proofs first, before any settlement command that requires finalized proof.

Admin accounts​

ActionRoleNotes
Create adminSuper-adminEmail must be unique, role must be valid
Update admin role/statusSuper-adminThe backend rejects unsafe admin-account states
Reset admin passwordSuper-adminNeeds Idempotency-Key; the target must be an admin account

Source: http_admin_accounts.go.

Deposits​

ActionRoleWhat to know
List / detailAdminRead-only
Create / update off-app draftAdminNeeds idempotency; update requires expected requested state
Settle off-app depositSuper-adminCredits the wallet and records events atomically. This is the money step
Void off-app draftSuper-adminEnds a draft with no credit
Reversal previewSuper-adminChecks whether reversal is safe, before you commit to it
Reverse settled depositSuper-adminNeeds expected succeeded, a reason code, idempotency, and a clean preview
Proof upload / finalize / previewAdminFinalized proof is audit evidence; storage is S3 or filesystem by config

If reversal comes back with a conflict and blockers, don't retry. Read the blockers — they'll tell you the principal is already spent, or that reward, referral, or mission effects are entangled, or that the state drifted. Each of those is a business decision, not a technical retry. The blocker is doing its job.

Cash-outs​

ActionRoleWhat to know
List / detailAdminOpen/Paid/Closed queues, with backend-owned counts and operator status
Acknowledge reviewAdminRecords the first reviewer once. Changes no financial state
Approvewithdrawals.approveCurrently super-admin. Queues the payout flow
Rejectwithdrawals.approveNeeds category and reason; releases reserved value
Approve (hand-paid)withdrawals.approveManual payouts only, through the mark-paid endpoint (ADR 0019, 2026-11-16). The admin answers Yes to "Did you pay the player already?" (player_paid_confirmed); approving records the cash-out paid in one step. The tag name (paid via) and payment tag default to the player's own request; the receipt screenshot is required only when the payout proof setting requires it; the fee defaults to zero and the net is computed. An approval reason is required while the player has open risk signals. The plain approve refuses hand-paid cash-outs
Confirm failedwithdrawals.approveUse after review, when the payout can't complete
Proof upload / finalize / previewwithdrawals.approveA proof given to mark-paid must belong to this cash-out and be finalized
Provenance readsAdmin or super-admin, per endpointExplains where the funds came from, before a payout decision

Two things that trip people up:

Queue grouping is not the lifecycle state. Everything unknown or non-final sits in Open. Paid is succeeded and nothing else. Closed is the rejected, canceled, expired, and failed final outcomes. Operator status is derived by the backend — don't recompute it from the lifecycle state, because the mapping is deliberately not one-to-one.

Acknowledging is not approving. Opening a row records that somebody looked. It doesn't approve, pay, fail, or transition anything. That's what makes it safe for the UI to do automatically when a row is opened.

On unknown_requires_reconcile: it is never auto-resolved, on purpose. Inspect the payment events, the provider evidence, and the provenance before applying any manual command. An unknown outcome resolved by assumption is worse than an unknown outcome left open.

Wallet adjustments​

Super-admin corrections. Always preview, then execute:

  1. Preview the target player, any source payment, and the resource deltas.
  2. Read the blockers and the exact_deposit_reversal flag.
  3. Execute using the expected balances the preview returned.
  4. Treat a balance_changed conflict as a hard stop — refresh and start again. It means the world moved while you were deciding, which is exactly the situation compare-and-set exists to catch.

Adjustment records are immutable and may reference ledger transaction, reward event, and payment reversal IDs.

Never reproduce an adjustment with direct SQL. It skips the ledger, the events, and the idempotency record, and it produces a state the application can't reason about afterwards.

Linked game accounts​

Verified links use normal redemption with system acceptance of old credits; no new first-redemption review is created (ADR 0027). The review actions below finish historical rows only.

ActionRoleNotes
Link an existing game account for a playerAdminReason required; read-only provider verification; see Linked game accounts
Approve a historical first redemption reviewAdminassignment_confirmed + snapshot must match a fresh re-read; payout 0..reviewed value; wallet credited only after verified zero
Reject a historical first redemption reviewAdminNo provider call; historical review closes, normal linked-account rules apply
Resolve an unproven reviewed debitSuper-adminOnly in reconcile_required; provider evidence reference required; debit_confirmed returns 200 once attested and credits only after a verified-zero read (now or later by the reconciler); debit_not_applied moves no money
Enable Add existing for a providerAdminPATCH .../capabilities/player_link, only after a same-tenant read/link/redemption smoke; disabled everywhere by default
Unlink a game accountAdminRefused while a transfer/review/provisioning is open; deletes no history

Save missing logins our own create proved​

playgo admin provider-accounts backfill-created-logins [--dry-run] runs inside the backend container (it needs the credential key). It finds active game accounts with no saved login whose create our system sealed with its own password (the created identity matches and nothing changed the account since), and saves that password as native_creation_confirmed. It makes no game provider call. An account adopted by name only is never saved; it stays staff work. Run --dry-run first: each line shows the account id, game, username and would_save, skipped or failed, never a password. A non-zero exit means at least one failed; read the error and do not rerun until it is understood.

Release stuck password changes and game-account setups​

playgo admin provider-accounts release-stuck [--apply --reason TEXT] runs inside the backend container. It makes no game provider call. Without --apply it only lists:

  • key lines: password-change commands still "processing" an hour past their expiry. fail means no credential hold names the command, so it never reached the game provider; --apply stores a failed result ("did not finish, nothing changed, try again"). skip-held means a hold names it and a reset may have been sent; it stays with native reset recovery.
  • setup lines: unknown game-account setups whose read-only reconcile gave up. --apply queues four more lookups and clears the staff escalation; a setup the lookups still cannot prove stays unknown and escalates again.

Run it without --apply first and keep the list; each write rechecks its row, so a second run changes nothing.

DollarPay expired unpaid checkouts​

Authenticated DollarPay order queries with pay_status 3 (number or string) mean the provider's two-hour unpaid checkout expiry. Reconciliation records checkout_expired evidence and closes the deposit in the existing abandoned state, without wallet credit. The fresh intent lock, durable-credit recovery and retained paid evidence prevent an unpaid observation from discarding earlier payment proof. Expiry requires no vendor money command.

A later verified success retains the closed state and opens a financial investigation case; it never automatically credits the wallet. Open that exact deposit case and its Verify procedure to obtain the merchant/provider order IDs, provider panel link and neutral evidence steps. Record provider evidence and escalate for an operator; no settlement outcome is recommended. Do not resend the checkout or edit balances.

For existing status-3 cases, a server-side deposit refresh/reconciliation after this fix uses the authenticated expiry response to close unpaid. The affected player's POST /api/v1/payment-deposits/{payment_id}/refresh follows the same path. A super admin can use the normal financial-case preview/resolve flow with independently verified not-applied evidence; that existing manual action marks the case failed rather than abandoned. Inspect the preview and expected state before confirming, one case at a time. No production resolution was executed as part of this change.

Provider health and provider controls​

ActionRoleNotes
List / detail provider healthAdminStatus, recent runs, canaries, redacted evidence and durable unresolved legacy probe verification items
Run read / launch probesAdminRead reuses recent successful observations; neither moves credits
Quarantine canaryAdminCanary inventory is operational state
Set up test accountSuper-adminCreates one provider test account and moves no credits
Acknowledge health statusAdminAnnotation only. Does not make a provider healthy
Patch provider capabilityAdminNeeds expected status, next status, and reason. Persisted, and survives restart
Script fake outcomeAdmin, local onlyLocal synthetic testing only
Put game fallback overrideSuper-adminNeeds a valid fallback state and reason

Historical health-probe transfers in unknown_requires_reconcile or failed_internal remain in verify_items independently of the five recent diagnostic runs. Use the provider panel link, exact player and external order ID to inspect operation history and retain an evidence reference. Current balance and missing history alone do not establish the historic outcome. Escalate with transfer/run IDs for owner-reviewed reconciliation; the health view offers no settlement action or recommended outcome and makes no provider calls. Do not resend loads or redemptions, or edit balances. The normal financial-case resolver rejects health-probe transfers. Mutating health probes remain removed under ADR 0025.

Reporting and analytics​

  • Business and player reporting are super-admin reads, using America/New_York reporting windows. If a number looks off by a day, check the timezone before anything else.
  • Reporting excludes tester and provider-health system accounts by default. include_internal is deliberately exceptional — reach for it knowingly, not habitually.
  • Business report caveats come back in the API response. Don't strip them in a downstream dashboard. The caveat is part of the number's meaning; a figure shown without it is misleading even when it's arithmetically correct.
  • Growth, product, and ops-event endpoints are read models for investigation. They are not a source of truth for money reconciliation.

Support​

Admin support actions are append and update operations on conversations: list and get conversations, list and post messages, mark read, claim, resolve and reopen.

Use these for the player-visible conversation only. Payment, wallet, and provider decisions belong in their own domain commands — a decision recorded only in a support message has no audit trail, no ledger effect, and no way to be found later.

Rewards​

  • Admin grants need the admin role, a target user, a positive amount (freeplay in whole USD cents), a balance kind, a reason code, and an Idempotency-Key.
  • Reward history reads return the target user's full reward event feed.
  • The reward sandbox is local/dev only and never touches production state. It's for deterministic scenario walkthroughs.

What each error code means here​

StatusIn admin flows
400Something in the body is invalid — enum, expected state, amount, proof, or business input
401No authenticated browser session
403Authenticated, but the role, permission, CSRF token, or gate is insufficient
404The record doesn't exist — or a local/dev-only sandbox is disabled
409State changed, idempotency conflict or in-progress, balance mismatch, or command conflict
503An optional dependency is down, such as proof storage or the payment service

Two worth calling out:

404 doesn't always mean "missing". Sandbox endpoints return 404 to everyone when disabled, admins included. Check whether the feature is switched on before you go hunting for the record.

409 is usually correct behaviour, not a bug. It means the system protected you from acting on stale information. Refresh and re-read before you retry — and if the same 409 keeps coming back, that's a signal to investigate rather than to loop.

Review a redemption​

Player redemptions debit game credits immediately. With Redemptions require review on (the default), a proven debit enters pending_review and the full quoted cash value stays in nonspendable review holding. The wallet stays unchanged. Open the redemption review in the admin Inbox or redemption review queue, inspect its immutable quote and the player's recent story, then approve or reject with a reason. Any admin can decide; each command needs CSRF and an idempotency key.

Approval moves the quoted cash-out balance amount to the wallet and the quoted forfeit, including consumed freeplay, to the house. Rejection first enters reject_restoring: a worker restores the original credits to the same game. Only proven restoration releases holding and records rejected. A timeout is an unknown outcome and reconciles without sending another credit. Missing/inactive accounts or refused credits open a financial case with holding retained. Escalate or record evidence there; an applied decision requires proof of restored game credits. not_applied cannot turn this holding into wallet cash.

Superadmins can change the audited setting using its current revision and a reason. The change affects subsequently issued player quotes. Existing reviews keep their original setting and amounts. Admin overrides and reviewed first redemptions already have an admin decision and settle directly.

See also​

Player-related email and Telegram alerts show the player's display name, falling back to the email local part. Support alerts contain no message body or raw conversation ID. A single immediate Telegram alert has one title and the recipient-local time, for example: 💬 New support message from Jordan · Oct 6, 3:03 AM CDT. Digests retain their summary and bounded item list.

Telegram setup checks and relinking​

Default-on payout and deposit notifications​

Payouts → Notifications shows the latest 50 immutable payout events. Open the cash-out or player for the current status; an approval event is not proof of payment. After approval, a deduplicated system message appears in the player's Support conversation with the frozen payout quote and existing-detail steps. It does not ask the player to submit a second house-account request.

Notification settings on this tab has four switches, all on by default: player payout instructions, the payout feed, ordinary deposit alerts, and Telegram support previews. Superadmins change them with a reason; other staff can read them. Save conflicts or an uncertain result require Reload notification settings before another edit. Delivery failures never alter money.

Ordinary deposits distinguish a submitted checkout/claim from a confirmed wallet credit. Important and All recipients receive the new event topics through their configured alert channels; Custom subscriptions remain unchanged. Telegram still needs an enabled recipient, configured bot and completed private-chat /start handshake. Turning a switch back on starts fresh, without resending skipped history. Support previews are Telegram-only: at most 140 scrubbed characters for a recipient mapped to active staff. Credentials and sensitive context are withheld; email and persisted alert bodies keep the summary.

Super admins can use Test bot before any recipient is linked. It checks the stored sealed token with getMe and reports the authenticated bot identity. It sends no message and consumes no updates. Missing configuration is a conflict; a rejected token needs replacement; an unavailable response can be tried again.

Remove a recipient's Telegram setup by clearing their Telegram username, even while waiting for Start. This preserves the email recipient and subscriptions. Saving the same username explicitly resets the chat link; that user must send a fresh /start. Queued Telegram messages for the old link are skipped. Archiving the whole external-notification recipient is a separate email-recipient action.

Inbox situations are newest first with stable key ties. The Payouts filter and badge include pending redemption reviews. The dedicated redemption review list is newest first by creation time, with transfer ID as the tie-break, so the latest pending review stays visible on its first page through a backlog.

Risk policy​

Risk policy requires schema migration 224 (000224_admin_risk_policy.sql). Release 6.9.2 uses 222, and the deposit-bonus I2 migration uses 223.

Settings → Risk holds the audited, versioned policy. Ordinary admins can read it; active super admins edit a trigger in a centered popup, choosing enabled, information/review handling and an integer threshold. Every save requires a coded reason. History names the actor and keeps each complete policy revision. A stale save returns 409; close and reopen to inspect the latest policy before retrying.

Defaults are 25 credited deposits/24h, 50 loads or redemptions/24h, 10 failures/7d per source (provider-operations information), and five completed cash-outs/24h as information. Shared destinations are information at two accounts and review at three. Device review at three additionally needs a review-level player signal. Unknown/internal-failure money is review at one record.

Information is a quiet line on Player details, never a Risk review waiting row, notification arrival or manual payout shortcut barrier. Review-only evidence is projected from current policy for payout eligibility. Routine reminders are 24h by default; count/source changes do not reset a pending item's arrival, and new uncertain money/new review causes escalate immediately. Opens and transitions have typed reasons and historical uncoded decisions say legacy_unspecified.

Risk-policy owner tests cover information-only projection, manual payout eligibility and replay with credited deposits and completed cash-outs. An operator records authorized Penguin sanity results without changing global policy to satisfy a test. Script acknowledgements are not human decisions.

Player actions in Inbox​

GET /api/v1/admin/users/{user_id}/pending-actions is an active-admin, player-scoped read of existing owner records. It makes no provider calls. The default page is three ranked items; Player details requests the full paged list. Approved money waiting to be sent comes first, then held cash-outs and uncertain money leaving games, missing deposit/load credit, game login/setup, and remaining risk reviews. Resolving the chat does not resolve these records. Read-only staff can inspect money; the owner commands still enforce their roles and current state.

POST /api/v1/admin/provider-account-provisionings/{provisioning_id}/retry requires an active admin, CSRF, a stable Idempotency-Key and a closed reason_code (player_requested, provider_configuration_fixed, temporary_failure_resolved, other). Other requires a 3–500-character note. It locks and rechecks the original definitive failure, refuses active jobs or an unconfirmed creation attempt, and atomically queues the provisioning owner's existing job with an append-only audit event. Replay returns the original receipt; changing the target or reason with the same key is a conflict. Unknown creation must be investigated rather than retried.

Money confirmations show the owner's recorded amount and require a reason. Draft settlement accepts the existing settlement reason family as optional audit metadata in its API; the admin confirmation requires it and records it in the existing settlement event. This does not change deposit credit calculations. Commands and uncertain requests keep their existing idempotence identity. Active visible chat polls every 15 seconds and central effects refresh the bar and conversation tags; a successful approval may become another pending send-payment item.

Support and provisioning owner tests cover pending-action projection, definitive refusal and original-key replay. Record authorized Penguin sanity results separately; a Mini test is not a live run.

Global Games: store players​

Open Games in the work sidebar, select the configured store, then find a registered account. GameRoom, Cash Machine and Mafia have full cached tables. The other stores show Find a store player and a separately labelled list of linked site players. Recorded inventories, balances and timestamps remain dated; an unavailable observation is never zero. Refresh is an explicit audited request. API Off keeps local records visible and prevents native requests.

Super admins may leave API Off until I turn it on, or choose an optional 15-minute, 30-minute, 1-hour, 2-hour or custom timer (1 minute to 7 days). The stored deadline appears with a countdown. Turn on now cancels it; Extend saves a new audited deadline after reviewing the current revision. Auto-on runs through the same audited service after expiry, including after a server restart, and retains the staff member who scheduled it. Until that On revision is committed, the hard switch remains Off; an elapsed countdown shows auto-on pending. The emergency Off control always remains available and an invalid custom timer can be bypassed with Turn Off without a timer.

A non-site store player can use Load from agent credit or Redeem to agent credit when the backend offers the action. Review the exact agent credit ↔ store game credits before confirming. The site wallet is unchanged. Keep the original request after a timeout or lost response. Check original request reads that receipt; Check original provider order accepts only matching native order, player, direction and whole-credit evidence. A current balance cannot settle a historical request. When native evidence is unavailable, enter an audited panel evidence reference and explicitly confirm the original order, player, amount and final outcome. Resolution never issues another money movement.

The Password section has one provider-appropriate primary action. More contains audited reveal/copy and additional supported actions. Recorded panel passwords are encrypted and retain staff-attested provenance; saving does not claim a successful native login. Check a password through the offered native check when supported. Unknown reset recovery retains the original command and candidate. A lost-process dispatch must be older than two minutes and its exact hold expired before a staff-confirmed finished-request panel resolution can be admitted. Fresh/active work stays blocked; expiry alone never releases it.

View player opens a linked account's existing player Games management. Linking an unlinked store identity requires a real site-player/provider slot, current identity/credential witnesses, a reason and no unresolved store work. Promotion preserves credential and opening-credit provenance. It creates no site wallet credit or paid-load entitlement. Local app history remains immutable and paged; provider history unavailable is distinct from empty local history.

Source: api/providerstore, internal/providerstore/httpapi, ADR0046.

Owner tests cover exact password bytes, encrypted storage, public receipt privacy, stale witnesses and same-key replay. Native reset tests retain an unknown command and its hold until the existing resolution owner confirms the result. Linking tests preserve credential provenance and financial history.

Golden replay tests (internal/providers/providergolden) use native adapters with sanitized fixtures. Authorized Penguin sanity checks use synthetic store players and bounded provider work. Record original command IDs and unresolved outcomes. Never retry an unknown write with a new identity. See ADR 0054.

Per-brand game API connections​

Active super admins open Admin > Platform > the game > Connect API on a site whose games.store_management_enabled setting is on (ADR 0061). Check credentials uses the current unsaved fields and appears when the adapter has a simple read-only agent check. Test everything runs the read phase in S1. The full capability list always appears; steps not built yet on our side remain FAIL, with that reason stated separately from provider failures. A required read failure stops dependent reads. Player creation, credit moves and password changes are not dispatched in S1. Both tests leave saved settings and API access unchanged.

A finished run shows outcome counts and Download issue .txt / Copy issue .txt. Reports retain masked wire exchanges, adapter signature recipes and result rows for 30 days. Agent identity is masked. Egress metadata comes from the deployment's CFG_PROVIDERS_DIAGNOSTIC_EGRESS_IP literal-IP setting; missing metadata is shown as unavailable and no discovery call is made. An interrupted read is UNKNOWN; Check original request retrieves its receipt without repeating provider IO.

Save connection submits the current draft independently of any test. Turn API Off and drain site/store work first. Saving does not enable API access. Keep an unknown save's original request and use Check original request. Credentials stay write-only and encrypted in immutable revisions. Tests never persist drafts. Activation failure blocks dispatch without file fallback. Historical accounts prevent changing the agent identity; drained same-agent secret rotation is allowed. A site with the setting off refuses these routes and hides the screens.

Connection owner tests cover encrypted drafts and revisions, receipt replay, stale revisions and player-role refusal. Golden replay tests (internal/providers/providergolden) check the native read protocol. Record authorized Penguin sanity results separately. See ADR 0054.

Converged release: shipping rules​

The first converged release (ADR 0061) reaches Penguin first as the canary, then SlotsFam in a planned maintenance window. Rehearse the SlotsFam steps on the Mac mini against a restored copy of its database before the real window, and record the outputs.

  • The neoplay-infra commit that drops CFG_GAMES_CONNECT_ENABLED, CFG_PAYMENTS_CONNECT_ENABLED and CFG_GAMES_CUSTOM_PROVIDERS_ENABLED from Penguin's compose (env-layout then refuses them on every site) and moves the Penguin runner to go1.26.10 ships in the same release manifest as the converged backend, applied in the same deploy-release --infra run. Applied alone first, it turns Connect off on Penguin's current backend. The converged backend without it refuses to start, because config refuses unknown keys; CFG_GAMES_STORE_MANAGEMENT_ENABLED replaces all three.
  • Penguin takes the converged release without Fire Kirin in its provider host file (neoplay-infra 7bb93708 keeps the provider files Penguin ran before b121d44). Boot refuses a host-file provider without a stored endpoint, and a maintenance deploy has no step between migrations and start where an import could run, so a host file listing Fire Kirin would keep Penguin down. infra-apply and deploy-release never write the decrypted host file /opt/neoplay/state/provider-integrations.json, so the release restores it on the host instead of re-encrypting it: after the infra roll and before the maintenance deploy, it saves the live file as provider-integrations.json.firekirin-on-<UTC timestamp> and installs the 25-entry backup without Fire Kirin with install -m 600 -o penguin -g penguin, checking the entry count and hash. env.live and proof-storage.env are not touched. After the release is verified, the operator runs qa/orch/run-logs/converge/FIREKIRIN-PENGUIN.sh from the laptop's neoplay workspace (it is an operator script, not part of this repository or the host). Its roll rolls infra with Fire Kirin back in (blue-green keeps the serving backend) and reinstalls the saved firekirin-on copy the same way. Its import runs playgo admin game-providers import-definitions --provider firekirin --reason "..." with the new image. The operator then turns API Off, saves the credentials, runs the credential check (it is refused while API is On), turns API On and restarts the backend. --provider limits the import to Fire Kirin, so riverpay's http Connect revision is not imported and its stored https endpoint stays. Migration 292 (the next release, a normal blue-green deploy) then makes Fire Kirin's catalog entry public, so Penguin serves it like its other games without CFG_PROVIDERS_CATALOG_PROVIDER_CODES. If the import refuses, the script's back-out rolls the host file back without restarting the backend.

SlotsFam convergence window​

Before the window​

  1. Penguin has run the converged release as the canary.

  2. The SlotsFam compose keeps CFG_GAMES_STORE_MANAGEMENT_ENABLED: "false", still carries the DollarPay and NOWPayments keys and the provider host file (they are the import source), and keeps the secret key the server uses for sealed settings.

  3. Choose an active super admin email as the payment import actor.

  4. Confirm the URLs the server derives from CFG_APP_PUBLIC_BASE_URL (internal/payments/connection/factory.go) equal SlotsFam's deployed ones: https://slotsfam.com/webhooks/payments/{dollarpay,nowpayments}/deposit and /withdraw, NOWPayments https://slotsfam.com/payment/success and /payment/cancel, vendor hosts https://mh.dollarpaywallet.com and https://api.nowpayments.io. They matched on 2026-11-10.

  5. Open-payment stop condition (ADR 0055). A DollarPay or NOWPayments deposit or withdrawal created before the switch has no saved revision, and the converged server cannot verify, look up, dispatch or close it. Stop new ones: turn the maintenance page on (./bin/deploy-release --env fam --maintenance on), switch the DollarPay and NOWPayments deposit routing rules off in admin payment settings, and approve no DollarPay or NOWPayments withdrawal. The maintenance page does not block API calls or webhooks, so the routing rules are what stops deposits. Then count, read-only, on the fam host:

    SELECT 'deposit' AS kind, provider_code, state, count(*) FROM payment_intents
    WHERE provider_code IN ('dollarpay','nowpayments')
    AND state NOT IN ('succeeded','failed','rejected','canceled','expired','voided','reversed','superseded','abandoned')
    GROUP BY 1,2,3
    UNION ALL
    SELECT 'withdrawal', provider_code, state, count(*) FROM payment_withdrawal_intents
    WHERE provider_code IN ('dollarpay','nowpayments')
    AND state NOT IN ('succeeded','failed','rejected','canceled','expired')
    GROUP BY 1,2,3;

    (docker exec neoplay-postgres psql -U playgo -d playgo -c "<query>".) The terminal sets are the ones migration 275 uses. Open means everything else, including checkout_created past its stale time, stale_pending, requested or approved withdrawals, unknown_requires_reconcile and failed_internal. Every row on SlotsFam is unpinned before the window, so the query has no integration_id filter. It must return no rows. Otherwise wait for settlement or resolve each one in admin (deposit closeout, reject or settle the withdrawal) while old main still serves them, and count again. Recount immediately before deploy-release; a new row is a stop.

In the window​

SlotsFam deploys blue-green. deploy-release runs Goose with the new image while the old main (blue) keeps serving, then starts the new server (green). Green refuses to boot until the provider import has run, so the first deploy-release reports [cutover] blue retained: old main keeps serving on the migrated database. That is expected. All schema changes in the window are additive, and old main ignores the new tables and columns.

  1. Out-of-order apply. deploy-release migrates with Goose -allow-missing. SlotsFam applies Penguin's 262, 272, 274, 276–279 and 281–284 once, then 291 and 292, the converged release's new files. 292 writes nothing on SlotsFam: it runs before the provider import, so Fire Kirin has no endpoint yet and keeps its allowlist_only entry, which SlotsFam's providers.catalog_provider_codes decides anyway. Check goose status: every file is applied and 264 stays ignored. Any other missing or changed file is a stop. goose version then prints 292, the version the server checks. These files set lock_timeout = '500ms' (276, 277, 283, 291 and 292; 291 also builds a small partial index on payment_events, among them) and each runs in its own transaction, so a lock timeout rolls back that file only and rerunning deploy-release is safe. The maintenance page keeps lock contention low.

  2. Provider import. Run the command with the new image against SlotsFam's runtime env on the fam host:

    docker run --rm --network neoplay-internal \
    --env-file /opt/neoplay/state/env.live --env-file /opt/neoplay/state/proof-storage.env \
    -e CFG_GAMES_STORE_MANAGEMENT_ENABLED=false \
    -v /opt/neoplay/state/provider-integrations.json:/opt/neoplay/state/provider-integrations.json:ro \
    neoplay-backend:<converged-sha> admin game-providers import-definitions --reason "SlotsFam convergence window"

    Each provider prints written or unchanged. A non-zero exit is a stop: read the refusal, fix its cause, and rerun until the command exits 0 with every provider written or unchanged. A stored https endpoint is never replaced by an http one: the command refuses and names the provider. Pass --allow-endpoint-downgrade CODE only with hman's approval for that provider. A live_mutations_enabled warning needs hman's decision before you continue.

  3. Run deploy-release again with the same manifest. Green must boot and take over; a boot refusal is a stop.

  4. Payment import, with the same docker run prefix: admin payments import-config --actor-email EMAIL --reason "SlotsFam convergence window" --activate. Each vendor prints its saved revision and its activation. A vendor that does not activate keeps deposits off for that vendor; do not retry with changed settings, read the printed reason first. A vendor the server cannot reach also prints The provider did not accept this check., so check the server's outbound access before you suspect the keys.

  5. Count again; this query must still return no rows:

    SELECT 'deposit' AS kind, provider_code, state, count(*) FROM payment_intents
    WHERE provider_code IN ('dollarpay','nowpayments')
    AND integration_id IS NULL
    AND state NOT IN ('succeeded','failed','rejected','canceled','expired','voided','reversed','superseded','abandoned')
    GROUP BY 1,2,3
    UNION ALL
    SELECT 'withdrawal', provider_code, state, count(*) FROM payment_withdrawal_intents
    WHERE provider_code IN ('dollarpay','nowpayments')
    AND integration_id IS NULL
    AND state NOT IN ('succeeded','failed','rejected','canceled','expired')
    GROUP BY 1,2,3;

Verify before reopening​

  • GET /api/v1/payment-deposits/options (signed-in tester) offers DollarPay and NOWPayments, and a synthetic tester deposit opens a checkout.
  • The game catalog lists every SlotsFam provider, including Fire Kirin, and each runs on SlotsFam's own endpoint; a tester balance read succeeds.
  • GET /api/v1/admin/providers reports game_store_management_enabled: false, and /api/v1/admin/integrations refuses.
  • Switch the deposit routing rules back on, then ./bin/deploy-release --env fam --maintenance off.

Rollback​

If a vendor does not activate and SlotsFam cannot wait, redeploy the previous main release with deploy-release and its manifest. The window's schema changes are additive, so old main runs on the migrated database and serves DollarPay and NOWPayments from deployment config again. Saved revisions and imported definitions stay in the database unused until the next attempt. Deposits made on the converged release carry a revision pin that old main ignores; they settle with the same deployment keys.

After the window​

The server takes DollarPay and NOWPayments keys only from activated revisions; the deployment keeps their panel URLs, timeouts and checkout TTLs. payments.dollarpay.enabled and payments.nowpayments.enabled only make the server require the credential vault; they do not turn a vendor on or off. To turn a vendor off, deactivate its saved revision. A late callback for a payment created before the switch is kept once (a resend of the same body adds nothing) on its payment as an unverified webhook_mismatch or withdrawal_webhook_mismatch event, credits nothing, and parks a still-open payment in unknown_requires_reconcile for staff. On a deposit it also opens a staff financial case, even when the deposit is already failed, because money may have arrived. A parked withdrawal closes with the case tooling's confirm paid or confirm not paid actions, which do not contact the vendor.

Delete, in a later release, once both sites have run the converged release:

  • the DollarPay and NOWPayments key fields in internal/platform/config and their CFG_PAYMENTS_* keys in neoplay-infra;
  • playgo admin payments import-config;
  • the ADR 0040 convergence policy entry in build/schema-compatibility-policy.json, once origin/penguin is retired.