Incident runbooks

Step-by-step responses to the failures that are actually likely. Each starts with the fastest way to confirm the diagnosis.

Severity guide

Level	Meaning	Examples
P0	Funds at risk, or the platform is down	Key compromise, backend down, contract exploit
P1	A product is unusable	Automation stopped, no routes quoting, chat broken
P2	Degraded	One provider down, worse prices, stale SEO pages
P3	Cosmetic or single-user	One stuck bridge, a rendering bug

P0 — `TX_SIGNER` suspected compromised

Confirm: unexpected MetaTransactionExecuted events, or the key appearing anywhere it should not.

Rotate first, investigate second

setSigner(newSignerAddress)     # on EVERY chain — 22 transactions

Then update TX_SIGNER (and POL_TX_SIGNER for chain 137) in the VM .env and pm2 restart bz-backend --update-env.

On-chain first, .env second. The reverse leaves the backend signing with a key the contract no longer accepts, so every swap fails.

Impact assessment: an attacker with this key can make the relayer .call an arbitrary target — but funds are pulled from msg.sender, so they spend their own money. Damage is griefing and reputational, not theft of user balances. That is why rotating immediately costs nothing.

□ setSigner on all 22 chains
□ .env updated, backend restarted
□ node scripts/check-contract-owner.js  — confirms the new signer
□ A test swap succeeds
□ Review MetaTransactionExecuted events for the exposure window

P0 — Executor key compromised

Confirm: TradeExecuted or AutoClaimed events you did not expect.

Immediate containment:

setRouters([...allCurrentRouters], false)     # Autopilot — halts ALL routing
pause()                                       # DCA vault

Revoking routers is the strongest Autopilot lever

The vault is fail-safe: with no allow-listed routers, no trade can execute at all. That stops every agent instantly without touching any user's funds.

Then:

setExecutor(newExecutorAddress)    # on BOTH vaults

This key is also the DCA vault owner

So a compromise includes UUPS upgrade authority over the DCA vault. Check getImplementation (EIP-1967 slot) on the DCA proxy against the expected 0x930a2b7F1B123B11d402bEfb6E1100f465082aD9 before assuming state is intact.

□ Autopilot routers revoked
□ DCA vault paused
□ New executor funded with gas on each automation chain
□ setExecutor on both vaults
□ .env updated, backend restarted
□ DCA implementation address verified unchanged
□ Re-seed Autopilot routers, unpause DCA
□ Reconcile: replay events against expected cron activity

P0 — Owner key compromised

There is no clean lever, and that is the point

The owner holds _authorizeUpgrade. If an attacker has it, they can replace the implementation and bypass everything.

If you still control the key: pause() all three contracts immediately, then transferOwnership to a secure address.

If you do not: there is nothing to do on-chain. This is exactly why moving owners to multisigs is a launch blocker. See secrets.

P1 — Backend down

Confirm:

bash

./scripts/deploy.gcp.sh status

Diagnose in order:

bash

./scripts/deploy.gcp.sh logs bz-backend 200
./scripts/deploy.gcp.sh ssh
# on the VM:
pm2 status
pm2 logs bz-backend --lines 100
df -h              # disk full?
free -m            # memory?

Cause	Fix
Crash loop	Read the logs; usually a missing env var after a deploy
OOM	PM2 restarts at 1 GB; a repeated restart means a leak
MongoDB unreachable	Check Atlas status and Network Access
Port already bound	`pm2 delete bz-backend && pm2 start ecosystem.config.cjs`
Bad deploy	Roll back: `git checkout <previous>` locally, redeploy

bash

./scripts/deploy.gcp.sh restart bz-backend

A bound port is not health

The deploy health check only confirms the port is listening. A process can bind and then throw on every request. Always read the log lines too.

P1 — No routes quoting

Confirm:

bash

curl -N 'https://api.blazpay.com/api/defi/quotes?fromChainId=42161&toChainId=42161&fromToken=…&toToken=…&amount=1&type=swap' | grep -c '^data:'

A liquid pair on Arbitrum should return roughly a dozen. Zero or two is the incident.

Diagnose:

Check	Command / place
Are the tokens and chain seeded?	A `404 Chain not found` / `token not found` answers this immediately
Is Redis alive?	Quotes stream but cannot be built without it
Which providers failed?	`logs bz-backend 300`, look around the quote request
Was `swap-sdk` just bumped?	Check the installed version the deploy printed
Is the whole upstream internet fine?	Try one provider's API directly

Per-aggregator errors are swallowed by design

So "no routes" with a healthy process means either every provider failed at once (rare — usually a shared dependency like the price enrichment or a network egress problem) or the token/chain lookup failed before the fan-out began.

P1 — Automation stopped (CSIP and/or Autopilot)

Confirm first, because one cause dominates:

bash

node scripts/checkDeployerBalance.js

Executor gas is the most common cause, and it fails silently

No error, no alert. Roughly 0.00002 ETH per Arbitrum batch. Top up and both products resume on the next tick.

If gas is fine, work down:

□ CRON=false accidentally set?         grep CRON ~/bz-backend/.env
□ Crons actually started?              logs | grep 'cron-ables'
□ Backend up at all?                   status
□ DCA: lastError on the position documents?
□ Autopilot: routers allow-listed?     ← if never seeded, NO agent has ever traded
□ Autopilot: policy hash matches on-chain?  a mismatch skips the agent silently
□ Autopilot: AUTOPILOT_ADDRESSES set for the chain? empty = skipped
□ Autopilot: agent `active`, before `expiry`, within daily caps?
□ Route availability: logs | grep 'Route via'

Activity feed for a specific agent:

GET /api/defi/autopilot/{chainId}/{agentId}/activity

P1 — Chat broken

Confirm: send a message; observe whether SSE events arrive.

Cause	Check	Fix
`bz-agent` down	`./scripts/deploy.gcp.sh status` — port 4000	`restart bz-agent`
`AGENT_URL` wrong	`.env` on the VM	Correct and restart
Extraction-provider outage	Every extraction node fails, so all intents break	Wait; there is no fallback for extraction
Redis down	Multi-turn context lost; classification still works	Restore Redis
Mongo down	Messages cannot persist	Restore Mongo
One intent broken	The rest work	See add an AI intent common mistakes

bz-agent is safe to restart aggressively

No keys that move funds, no chain access, and nothing depends on it except chat.

P2 — One provider down

Confirm: route count dropped on a pair that previously had more.

bash

./scripts/deploy.gcp.sh logs bz-backend 300 | grep -i '<provider>'

If it is transient: do nothing. The fan-out already tolerates it.

If it is permanent:

// swap-sdk/src/index.ts — comment out WITH the reason inline
// MyProvider: api.myprovider.xyz returns 404 since 2026-08-01.
// Re-enable when a current URL is published.
// this.aggregatorFactory.register(AGGREGATORS.MY_PROVIDER, new MyProviderAggregator())

Then tag, release, and bump both consumers including lockfiles. See swap-sdk releases.

This is currently discovered by humans, not monitoring

Three providers are already disabled for exactly this reason. See monitoring.

P2 — All SEO pages returning the SPA shell

Confirm:

bash

curl -s https://defi.blazpay.com/swap/hemi/ | grep -c 'Hemi is'

Zero means the prerendered file is not being served.

Cause	Fix
`force = true` on the SPA rewrite in `netlify.toml`	Remove it. Non-forced rewrites let Netlify serve prerendered files first
`npm run build:app` used instead of `npm run build`	Rebuild with the full command
`prerender-seo.mjs` failed	Read the Netlify build log
Netlify build failed entirely	Publish the previous deploy from the dashboard

P2 — AI site serving defi canonicals

Confirm:

bash

curl -s https://ai.blazpay.com/ | grep -o 'canonical[^>]*'

Cause: scripts/rewrite-ai-shell.mjs did not run — the wrong build command was used. It is wired into build:ai and fails the build if any defi.blazpay.com string survives, so this means build or build:app ran instead.

Fix: rebuild with npm run build:ai.

This is worse than it looks

Every AI URL telling Google canonical=defi.blazpay.com says "this whole domain is a duplicate". Fix and re-request indexing.

P3 — A single stuck trade

Full decision tree: debugging a stuck trade.

Fast triage:

Observation	Meaning
`txHash` is a UUID	Never signed. Nothing moved
Reverted	Atomic. Nothing moved. Read the revert reason
Succeeded, no output	Likely an `approve()`-step route. Nothing moved but gas
Bridge, source confirmed	Two-phase. Poll `GET /defi/txn/{wallet}`

P3 — Funds stranded in a vault by accounting

There is precedent, and it worked.

On 2026-07-03, V1-era phantom credits left two positions unclaimable. Recovery via the owner key:

1. Position 1: stranded 0.000134 ETH sent directly to the user
2. Position 2: 0.0011 WETH withdrawn, swapped to 1.903 USDC via
   Uniswap SwapRouter02 (0x68b3…Fc45, fee tier 500),
   1.29397 sent BACK INTO the DCA contract to fund the user's
   credited remainder so it became claimable again, rest to the user

Why fund the contract rather than pay the user directly

The user's balance was credited in the vault's accounting. Paying them off-contract would leave the credit outstanding, so they could claim twice. Funding the contract makes the accounting true again.

A production database cleanup the same day (user-approved) reclassified 13 dead pending rows with no positionId and 38 stale/dust/malformed success rows to failed, keeping 4 legitimate actives. The cron went from scanning 42+ positions per minute to 4.

Get explicit approval before any production data mutation

And enumerate exactly what will change first. Several scripts at the bz-backend root delete rows and are not idempotent.

Autopilot agent never trades

The single most likely cause, stated separately because it is so common on a new deployment:

Check the router allow-list before anything else

bash

cd bz-backend/contract-deployment
node scripts/probe-autopilot-routers.cjs     # which routers deliver in-tx output?
node scripts/seed-autopilot-routers.cjs      # setRouters([...], true)

The vault is fail-safe: with an empty allowedRouter map, every trade reverts RouterNotAllowed. Robinhood Chain (4663) is deployed and still has none seeded.

Many aggregators on newer chains are solver/intent based and deliver no in-transaction output — the probe exists to find the ones that do.

Post-incident

□ Root cause written down
□ Was it detectable? If not, add the check — see monitoring's build-first list
□ Is a runbook missing? Add it here
□ Did a doc page mislead? Fix it
□ If a key rotated: is the old one revoked everywhere?
□ If data was mutated: is there a record of what changed?

Escalation contacts

Situation	Path
Bridge stuck past the expected window	The provider's support, with the source hash. Blazpay has no lever
Orderly outage	Orderly. Perpetuals have no fallback venue
Fiat ramp / KYC stuck	The provider. Blazpay holds no identity documents
Atlas or Netlify outage	Their status pages
Contract exploit	Pause, then assess. Do not upgrade under pressure without layout validation

Incident runbooks ​

Severity guide ​

P0 — TX_SIGNER suspected compromised ​

P0 — Executor key compromised ​

P0 — Owner key compromised ​

P1 — Backend down ​

P1 — No routes quoting ​

P1 — Automation stopped (CSIP and/or Autopilot) ​

P1 — Chat broken ​

P2 — One provider down ​

P2 — All SEO pages returning the SPA shell ​

P2 — AI site serving defi canonicals ​

P3 — A single stuck trade ​

P3 — Funds stranded in a vault by accounting ​

Autopilot agent never trades ​

Post-incident ​

Escalation contacts ​

Incident runbooks

Severity guide

P0 — `TX_SIGNER` suspected compromised

P0 — Executor key compromised

P0 — Owner key compromised

P1 — Backend down

P1 — No routes quoting

P1 — Automation stopped (CSIP and/or Autopilot)

P1 — Chat broken

P2 — One provider down

P2 — All SEO pages returning the SPA shell

P2 — AI site serving defi canonicals

P3 — A single stuck trade

P3 — Funds stranded in a vault by accounting

Autopilot agent never trades

Post-incident

Escalation contacts