Monitoring & health

An honest account: monitoring is thin. This page covers what exists, then the gaps — because the gaps are the operationally interesting part.

What exists

Mechanism	Covers
deploy.gcp.sh status	PM2 process table + all four ports bound
deploy.gcp.sh logs <app> <n>	PM2 logs
Deploy health check	Port bound + last 12 log lines, exits non-zero on failure
PM2 autorestart + max_memory_restart: 1G	Process-level recovery
Request logger	Every request: timestamp, method, URL, forwarded IP, origin
SENTRY_DSN	Error reporting, if configured
ENABLE_MONITORING, LOG_TARGET, LOG_TARGETS	Flags whose wiring should be verified before relying on them
TELEGRAM_ALERTS_CHAT_ID	An alert channel exists
models/networkStatusModel.js	Per-chain health records
models/monitoredTransactionModel.js	Transactions under watch
GET /api/debug-stats	Unauthenticated, registered before all middleware

Health checks you can run now

cd bz-backend

./scripts/deploy.gcp.sh status              # processes + ports

# read-only chain and contract checks
node scripts/checkRelayerDeployed.js        # relayer presence per chain
node scripts/check-contract-owner.js        # owner / signer verification
node scripts/check-contract-state.js        # fee config, pause state
node scripts/checkDeployerBalance.js        # EXECUTOR GAS — check this often
node scripts/fetchAllChainBalances.js
node scripts/blazai-analytics.js            # chat usage figures

HTTP checks:

# is the API alive and can it quote?
curl 'https://api.blazpay.com/api/defi/quotes?fromChainId=42161&toChainId=42161&fromToken=0x…&toToken=0x…&amount=1&type=swap'

# accumulated fees across all chains, bypassing cache
curl 'https://api.blazpay.com/api/defi/fees/summary?force=1'

Executor gas

This is the single most important thing to watch

PRIVATE_KEY_EXECUTOR (0x5935745431D1385ae1a9CE2644fb5f5d1f140459) pays gas for every CSIP buy, every auto-claim and every Autopilot trade.

When it empties, both automation products stop silently. No error surfaces, no alert fires, and users simply see their scheduled buys stop happening.

Arbitrum consumption is roughly 0.00002 ETH per batch.

node scripts/checkDeployerBalance.js

There is no automated top-up and no threshold alert. Until there is, this belongs on a human checklist.

Silent provider failure

The biggest observability gap in the platform

AggregatorFactory.getQuotes wraps every aggregator in a try/catch so one failing API cannot kill an SSE stream. That resilience is correct — but it means a permanently broken provider disappears with no signal. The user sees fewer routes and worse prices; nothing alerts.

Three providers are already commented out for exactly this reason (nitro, icecream_swap, kima), and each was discovered by someone noticing, not by monitoring.

The cheap fix: count per-provider quote successes and failures per hour, and alert when a previously-healthy provider drops to zero. Nothing like this exists today.

Manual check — quote a common pair and count the routes:

curl -N 'https://api.blazpay.com/api/defi/quotes?…' | grep -c '^data:'

Compare against the 21 registered providers. A sudden drop from ~12 routes to ~4 on a liquid pair is the symptom.

What each failure looks like

Failure	Symptom	Detected by
Backend down	Everything 502s	Port check, user reports
bz-agent down	Chat returns errors; trading unaffected	status, user reports
MongoDB unreachable	Everything fails	Logs
Redis unreachable	Quotes cannot be built (5-min cache gone), Socket.io adapter off, Autopilot dip_buy stops firing	Logs, gradually
One aggregator down	Fewer routes, worse prices	Nothing
All aggregators down	No routes at all	User reports
Executor out of gas	Automation stops	Nothing
Autopilot routers not allow-listed	No agent ever trades	Nothing — see below
Policy hash mismatch	That agent silently skipped	autopilotEventModel
Orderly down	Perpetuals unusable	User reports
Zerion down	Empty portfolios, wallet cards fail	User reports
Etherscan down	Auditor, token safety, approvals, wallet card fail	User reports
Extraction provider down	Audit and token safety fail; all extraction nodes fail → chat broken	User reports
Conversation provider partially configured	Answers quietly get worse	Nothing
Netlify build failure	Frontend stale	Netlify notification
IndexNow 403	New pages not submitted to Bing	Plugin log

Three failures produce no signal at all

Executor gas, single-provider outage, and an unseeded Autopilot router allow-list. All three degrade a product to zero while every process reports healthy.

The conversation-tier silent downgrade

A partially configured environment quietly costs answer quality

getConversationLLM() needs all three of LLM_GOOGLE_PROJECT_ID, LLM_GOOGLE_CLIENT_EMAIL and LLM_GOOGLE_PRIVATE_KEY. With any one missing it falls back to the cheaper extraction model instead of the conversation tier. No error, no log line, just worse conversation.

Check all three together after any .env edit.

Reading the logs

./scripts/deploy.gcp.sh logs bz-backend 500

On the VM: /home/bisht/.pm2/logs/bz-backend-{error,out}.log, merged, YYYY-MM-DD HH:mm:ss Z.

Grep	Finds
cron-ables	Startup banner — crons started
cron job is disabled	CRON=false
/defi/quotes	Quote requests, with per-aggregator failures nearby
Route via	Execution-engine route rejections
Not swap calldata	Calldata-guard hits
delivers no in-tx output	Probe rejections (solver providers)
error creating record	DCA creation failures
Not allowed by CORS	An origin missing from the allowlist
A wallet address	That user's entire session
🔌 [Socket.io]	Socket startup and adapter state

Log volume is high

Every request logs a line, and entryNode in bz-agent logs the classified intent on every chat message. Useful for debugging, noisy for storage. There is no log rotation policy documented beyond PM2's defaults.

Usage analytics

node scripts/blazai-analytics.js

Produces the figures used in PROOF_POINTS — messages, sessions, unique wallets, non-fallback rate.

The non-fallback rate is not an accuracy measure

It counts messages that reached a real intent rather than the fallback path. A message classified as the wrong intent counts as a success. The headline "99.9%+ accuracy" should be read with that in mind, and re-verified before any public claim.

Frontend analytics: GA4 via src/utils/analytics.ts, a no-op unless VITE_GA_MEASUREMENT_ID is set. It also detects AI referrers (chatgpt, perplexity, gemini, copilot, claude, grok, meta-ai) and fires ai_referral once per session — the only measurement of AEO/GEO performance.

Revenue monitoring

curl 'https://api.blazpay.com/api/defi/fees/summary?force=1'

Walks the relayer deployment record, reads native and known-token balances per chain over RPC, prices via CoinGecko, returns a USD total. Cached five minutes.

Only tokens with a coingeckoId are counted

Long-tail and spam tokens never accumulate as relayer fees, and skipping them keeps the RPC fan-out inside the 8-second per-chain timeout on big-catalogue chains. So the number is a floor, not an exact total.

Autopilot and DCA earnings accrue inside their vaults (platformEarnings) and are not in this summary. Read them on-chain or sweep with withdrawEarnings.

A practical daily check

Until there is real monitoring, this is the substitute:

□ ./scripts/deploy.gcp.sh status                  — four ports bound
□ node scripts/checkDeployerBalance.js            — executor has gas
□ Quote a liquid pair; count the routes            — provider health
□ ./scripts/deploy.gcp.sh logs bz-backend 200     — no repeated errors
□ curl /defi/fees/summary?force=1                 — fees still accruing
□ Spot-check a CSIP position executed today
□ If Autopilot is live: an agent traded, or a documented reason it did not

What to build first

Ranked by outage-minutes prevented per hour of work:

1. Executor gas alert. A balance threshold with a Telegram message. TELEGRAM_ALERTS_CHAT_ID already exists.
2. Per-provider quote success rate. Count and alert on a healthy provider dropping to zero. Closes the largest blind spot.
3. Cron heartbeat. Write a lastRunAt per job and alert when one goes stale. A dead cron is currently invisible.
4. Uptime check on /api plus a synthetic quote. Catches the "process is up but every request fails" case that the port check misses.
5. Verify Sentry is actually wired. SENTRY_DSN is read; confirm errors arrive before trusting it.
6. Autopilot router-allow-list assertion at startup. Log loudly when a configured chain has zero allow-listed routers.
7. An intent-classification eval set. 50 labelled messages run on every prompt change — the only way to detect mis-classification.

Related

• Cron jobs — what runs and how often
• Incident runbooks — what to do when something breaks
• Errors & failure modes — symptom → cause → fix