# Midtrans Payment Sweeper — production setup and operator runbook Operational guide for the background Snap sweeper and the stranded-payment reconciliation report that goes with it. - `GET /api/v1/sales-orders/payment-reconciliation` — the report. **Owned by this repository now**, documented and tested in `internal/sales/reconciliation.go`. - The sweep endpoint itself is owned by the sweeper change. Its name, shape and behaviour are referenced below because an operator has to configure and call it; it is NOT part of the report lane and is deliberately not duplicated in `docs/openapi.yaml` (the sweeper's own change owns that). --- ## 1. Why this exists Midtrans retries the capture webhook to the configured Payment Notification URL for roughly **5.7 hours** after a capture. After that it stops. A buyer who closes the tab before the webhook lands leaves an invoice that looks unpaid forever. Nothing retries, nothing reconciles, and the only thing that ever settled those was the buyer's own device polling `GET /api/v1/sales-orders/{id}/payment-status` — which only works while the buyer still has the page open. Two different things fix two different problems, and it is worth being precise about which is which: | | Scope | Fixes | |---|---|---| | **Sweeper** (background) | every future invoice | polls Midtrans for invoices whose webhook is late, before the retry window closes | | **Report** (this lane) | invoices already stranded | nothing automated can ever settle these. They need a human and the Midtrans dashboard. | The sweeper will **not** help an invoice that is already past the retry window. That is what makes the report a first-class operational tool rather than a stopgap. --- ## 2. `MIDTRANS_SWEEP_TOKEN` The sweeper is driven over HTTP by cron, not by an in-process goroutine. A goroutine would die with every deploy, drift with every replica, and could not be paused during an incident without a redeploy. A cron + bearer token is inspectable, rate-limitable and killable with `crontab -e`. Generate a long random value and put it in the deployment environment next to `MIDTRANS_SERVER_KEY`: ```bash openssl rand -hex 32 ``` ```bash # deploy/.env (production VPS, alongside docker-compose.yml) MIDTRANS_SERVER_KEY= MIDTRANS_CLIENT_KEY= MIDTRANS_IS_PRODUCTION=true MIDTRANS_SWEEP_TOKEN= ``` `.env` is already the backend's `env_file` (`docker-compose.yml`), so no compose change is needed for the variable to reach the container. `deploy/.env.prod.example` carries the Midtrans block; add the sweep token there when the sweeper lands so a fresh deploy does not silently come up without it. Then restart the backend so it picks the value up: ```bash docker compose up -d backend ``` ### Why an unset token answers 404, not 503 This is a **deliberate deviation** from the `ErrGatewayDisabled` convention used everywhere else in this codebase. Everywhere else, "the gateway is not configured" is a misconfiguration of *this* service, so the honest answer is `503 Service Unavailable` and saying so helps the caller fix it. The sweep endpoint is different: the interesting question is not "is our gateway configured" but **"is a sweeper deployed here at all"**. A 503 on that route is an advertisement. A scanner that finds a 503 learns three things for free — that the deployment runs a payment sweeper, that it is currently disabled, and exactly which URL to keep probing until it comes up. A 404 leaks none of that: it is indistinguishable from a route this build never had. So the route 404s when `MIDTRANS_SWEEP_TOKEN` is unset, and the code says so at the branch. The trade-off is a real one and it is the right way round here: the cost of the wrong answer here is a stranger learning about a money-moving endpoint, while the cost of the right answer is that a misconfigured cron job gets a 404 in its logs instead of a 503. The crontab in §3 is the thing that tells an operator their job is not running, and a 404 in the crontab's own output is just as legible for that purpose. **Do not "fix" this to 503 to match the rest of the codebase.** --- ## 3. The crontab entry ```cron */15 * * * * curl -fsS -X POST https://api.keluargamart.local/api/v1/midtrans/sweep \ -H "Authorization: Bearer ${MIDTRANS_SWEEP_TOKEN}" \ -H "Idempotency-Key: sweep-$(date -u +\%Y\%m\%dT\%H\%M\%SZ)" \ -H 'Content-Type: application/json' \ -d '{"limit":50}' \ >> /var/log/keluargamart/sweep.log 2>&1 ``` **The `Idempotency-Key` is required, not optional.** The route carries `idem.Handler`, so a POST without the header is a `422` and the sweep never runs. It must be **unique per run** — the idempotency store replays the recorded response for 24h, so a fixed key would return the first run's report forever and the sweeper would appear to work while settling nothing after the first pass. `$(date -u +...)` gives a new key every 15 minutes. Install as the deploying user, not root: ```bash crontab -e ``` The `env` line must be present in the crontab itself, or the shell will not see the variable: ```cron SHELL=/bin/bash MIDTRANS_SWEEP_TOKEN= */15 * * * * curl -fsS ... ``` ### Why every 15 minutes The constraint is the ~5.7 hour Midtrans webhook retry horizon, and the goal is to poll **while that window is still open** so the sweeper settles the invoice from the provider directly rather than leaving it for a human. `*/15` gives **92 attempts per day**, and inside a 5.7-hour window that is **~23 separate chances** to catch a late webhook. The reasoning behind the number: - **At least ~20 attempts** is the floor worth having. A buyer closing the tab is not a rare event and Midtrans flakiness is not correlated, so a single poll landing during a provider hiccup should not be the end of it. - **Every 15 minutes keeps the whole horizon covered** for an invoice captured at any time of day. An invoice captured at 23:59 still gets ~22 attempts before the window shuts. - **Going much faster buys nothing.** Each run makes one Midtrans `GET /v2/{order_id}/status` call per candidate invoice. Sweeping more often mostly re-polls invoices that are genuinely unpaid, and the risk of being rate-limited goes up without improving coverage. - **Going much slower is the actual failure mode.** At hourly, an invoice gets only ~5 attempts and the tail of the window is thin. At `0 */6 * * *` an invoice could have exactly one attempt in the whole horizon, and missing it strands the invoice permanently — which is the bug this whole mechanism exists to prevent. `*/15` is the largest interval that still gives comfortable margin on the attempt count while keeping request volume modest. --- ## 4. The Midtrans rate-limit position **Midtrans publishes no documented rate limit for `GET /v2/{order_id}/status`.** There is no published quota to design against, which means the numbers have to be chosen defensively rather than to a spec. The chosen posture: **stay far enough below any plausible limit that being flagged is not a failure mode.** The four numbers, and why each is what it is: | Number | Value | Reasoning | |---|---|---| | Cron cadence | `*/15` | ~23 attempts per invoice inside the 5.7h retry window. See §3. | | Batch cap | **50** per pass | ~100 status calls worst case (two refs per invoice: the stored one and the canonical one). Caps a pass at roughly a minute against a healthy provider, and bounds the damage from a backlog. | | Per-invoice throttle | **10 minutes** | Enforced in SQL by `invoices.snap_last_checked_at`, deliberately *under* the 15-minute cadence so the throttle never becomes the thing standing between an invoice and its next attempt. A hard provider error is not stamped, so a failing invoice is retried on the very next pass. | | Concurrency | **1** | Sequential. At ~100 calls a pass, a worker pool buys nothing and a burst from one IP is the easiest way to be flagged. | Concretely: - Worst case, the sweeper makes one status call per candidate invoice per run, capped at 50 candidates and at most two refs each. 96 runs/day × 50 = 4,800 status calls a day at the cap, or fewer, spread evenly across 24 hours rather than bursty. That is a rounding error against any plausible limit. - Status reads are the cheapest class of call Midtrans offers; they are not transaction creation, which is the operation that carries the real constraints. - The interval in §3 exists partly for this reason. `*/15` is frequent enough for correctness and slow enough that no run looks like a scrape. **Because no limit is published, being rate-limited is a hypothesis, not a known condition.** If Midtrans ever starts returning 429 or throttling, the response is a failed sweep, not lost money: a swept invoice that fails to settle is simply still open, the report in §6 still lists it, and the next run 15 minutes later tries again. The design degrades toward "needs a human", which is where it already was. Do not add a much shorter interval to "get ahead" of the webhook. The window is the only thing that matters, and the window is generous. --- ## 5. Endpoint shapes ### 5.1 The sweep (owned by the sweeper change, called by cron) ``` POST /api/v1/midtrans/sweep Authorization: Bearer Idempotency-Key: Content-Type: application/json ``` ```json { "limit": 50 } ``` `limit` is optional and is a **batch size only**. Absent, zero, negative and anything above the cap of 50 all take 50. There is no invoice id, no order id and no ref in the request and there never will be: the candidate set comes from a database predicate, so the only thing a caller can ask for is "sweep the candidate set". Do not add a way to name an invoice — it is the whole security property of the route. Response shape (the shared success envelope from `docs/api-design.md` §3): ```json { "data": { "candidates": 42, "settled": 3, "already_settled": 5, "written_off": 0, "not_attempted": 1, "pending": 30, "unknown_ref": 3, "hard_errors": 0, "amount_mismatch": 0, "skipped": 0, "stranded_oldest": "INV/2026/09/0007", "duration_ms": 8123 }, "meta": { "request_id": "..." } } ``` The counters are mutually exclusive: every candidate the pass reached lands in exactly one. A pass that hits its whole-run deadline (2 minutes) leaves the remaining candidates **uncounted** — nothing was written for them, their throttle stamps rolled back, and the next pass retries them. Reading them: | Field | Means | Action | |---|---|---| | `settled` | the sweeper did its job | none | | `written_off` | money captured against an invoice whose debt was already forgiven | **act**: refund at Midtrans or re-book. The most consequential number here. | | `hard_errors` | provider or database failure, transient | watch; a rising count means Midtrans is unwell. Stamps rolled back, so it self-heals. | | `amount_mismatch` | provider reported a gross we cannot account for | **act**: fails closed, nothing written. Never transient. | | `pending` | provider holds a transaction that has not settled | none; the buyer is mid-payment | | `unknown_ref` | provider has never heard of the ref | none; the buyer abandoned Snap | | `stranded_oldest` | oldest invoice the pass could not close | where to start looking | Outcomes that matter when reading the crontab log: | Status | Meaning | Action | |---|---|---| | `200` | sweep ran | nothing | | `404` | `MIDTRANS_SWEEP_TOKEN` unset on the server | fix §2, the job is a no-op | | `401` | token wrong on either side | check the crontab `env` line matches `.env` | | `422` | missing/invalid `Idempotency-Key` | see the cron line in §3 | | `429` | rate-limited by Midtrans | back the interval off to hourly; see §4 | ### 5.2 The reconciliation report (this lane) ``` GET /api/v1/sales-orders/payment-reconciliation?limit=200 Authorization: Bearer ``` Read-only. No request body, no side effects. `limit` is optional. Absent, unparseable and negative values all take the default of 50; anything above the cap of 200 is cut to 200. A caller cannot request an unbounded scan. ```json { "data": [ { "invoice_number": "INV/2026/09/0007", "so_number": "SO/2026/09/0007", "order_status": "invoiced", "customer": "Budi Santoso", "total_amount": 51872.5, "snap_order_ref": "KM-INV/2026/09/0007-42-1757904000", "canonical_ref": "KM-INV/2026/09/0007-42", "last_attempt_at": "2026-09-26T21:14:03+07:00" } ], "meta": { "request_id": "..." } } ``` An empty report serialises as `"data": []`, not `null`, so "nothing is stranded" is unambiguous. | Field | Meaning | |---|---| | `invoice_number` | the invoice to act on | | `so_number` | its sales order, for the operator's own cross-check | | `order_status` | sales-order lifecycle status | | `customer` | customer name | | `total_amount` | the invoice figure as billed — the amount an operator verifies against Midtrans | | `snap_order_ref` | `invoices.snap_order_ref`, the ref stored by the **last** Snap attempt | | `canonical_ref` | the **first-issue** ref, re-derived as `KM--` | | `last_attempt_at` | `invoices.updated_at`, i.e. the last write to the invoice — the last gateway touch | **Both refs are always present and they are usually different. See §6.** ### Why the report has no write path, no force-recheck and no UI This is a deliberate scope decision, not an oversight. The operator's job here is *"show me invoices that look stranded, with their refs, so I can check the Midtrans dashboard."* That is a query. The resolution already exists and is already shipped: a human looks at the Midtrans dashboard and either refunds there or books the payment through `POST /api/v1/invoices/{id}/pay`. Adding a "force recheck" button would put a money-moving action behind a UI control whose failure modes — acting on a misidentified row, double-settling a captured invoice, refunding a payment the buyer did make — are exactly the ones a report should not be able to cause. There is no UI because the audience is whoever is already holding the Midtrans dashboard open during an incident. --- ## 6. What an operator does with a row For each row in the report: 1. **Open the Midtrans dashboard** and search for the invoice. **Search for `canonical_ref` first, then `snap_order_ref`** — the order matters because they are not the same value. **Why both, and why this is not redundant.** A SnapToken retry **overwrites** `invoices.snap_order_ref` with `-`, because Midtrans rejects a duplicate `order_id` and the buyer clicking pay again needs a fresh transaction. So a buyer who paid on attempt 1 and then retried on attempt 2 has left money captured at Midtrans under the **canonical** ref while the row points at the **retry** ref. The superseded ref is not recoverable by editing the stored string: invoice numbers carry their own trailing digits, so stripping a trailing `-` is ambiguous and `canonicalSnapRef` explicitly forbids it. The canonical ref is re-derived from the invoice; the retry ref is the only record the retry exists at all. Neither tells you about the other, and this report is the only surface that shows both at once. **If one search returns nothing, search the other before concluding the buyer never paid.** 2. **If Midtrans shows no transaction for either ref** — the buyer abandoned the payment. Nothing to reconcile. Leave the invoice open; it ages out of the report's 12-hour horizon concern and normal credit handling takes it. 3. **If Midtrans shows a capture/settlement** — money was taken and the invoice is unpaid. Pick one: - **Refund at Midtrans**, if the order should not have been paid (buyer cancelled, fraud, duplicate). Refund at the provider; the invoice stays unpaid and the order can be cancelled. Refunding at Midtrans is the provider's action and is not mirrored in this ledger, so follow it with the order/invoice state change in the ERP. - **Book it here** with `POST /api/v1/invoices/{id}/pay` — already shipped, already permissioned behind `fakturisasi.invoices.pay`, and the same route finance uses today. Use this when the payment is good and the invoice simply never settled. Pass the amount Midtrans actually captured, not the invoice total, and put the Midtrans `transaction_id` in the notes so the two sides are traceable. The idempotency middleware is already on that route; do not re-post it after a timeout without checking whether the first post landed. 4. **Then** confirm the invoice is `paid` and the order has moved on. The row leaves the report on its own — the predicate only reports invoices still in `unpaid`/`partial`. ### Permission The report is gated on **`fakturisasi.invoices.view`**, the same key `GET /invoices` uses — confirmed against `internal/auth/catalog.go` (the "Sales & Finance" group) and against the `role_permissions` grants in migrations `000058` and `000081`, which seed it for `finance` and `finance_head`. No new permission key and no new migration are needed. It is deliberately **not** gated on `sales.sales-orders.view`. This is finance looking at invoices, and a sales-order reader must not inherit the ledger view. Booking a payment additionally needs `fakturisasi.invoices.pay`, which the same roles already hold. Both keys ship in the same migrations, so anyone who can read the report can also resolve it. --- ## 7. Verifying a deployment ```bash # 1. Is the report reachable with an invoice-reading token? curl -fsS -H "Authorization: Bearer $USER_TOKEN" \ https://api.keluargamart.local/api/v1/sales-orders/payment-reconciliation?limit=5 | jq # expect: {"data":[...],"meta":{...}} — [] is a valid, healthy answer # 401 -> token lacks fakturisasi.invoices.view # 403 -> role lacks the grant # 2. Is the sweeper actually enabled? (404 = token unset, see §2) curl -s -o /dev/null -w '%{http_code}\n' -X POST \ -H "Authorization: Bearer $MIDTRANS_SWEEP_TOKEN" \ -H "Idempotency-Key: verify-$(date -u +%Y%m%dT%H%M%SZ)" \ https://api.keluargamart.local/api/v1/midtrans/sweep # 404 -> MIDTRANS_SWEEP_TOKEN unset on the server: fix .env, restart backend # 401 -> token mismatch between .env and the crontab # 422 -> Idempotency-Key missing; see the cron line in §3 # 200 -> enabled (this DOES run a pass and moves money — it is not a dry run) # 3. Is cron actually running it? tail -n 20 /var/log/keluargamart/sweep.log grep -i payment-sweep /var/log/syslog | tail -n 20 ``` ## 8. Incident notes **Production had 7 stranded invoices totalling 363,132 at the time this report was written.** They are past the webhook retry window and the sweeper cannot help them. Work them through §6, one row at a time, and remember both refs. Two things this report deliberately will not tell you: - **Whether a payment exists.** It reads the local ledger only. "Stranded" means *looks stranded* — the answer is at Midtrans, which is why the row carries the refs instead of a verdict. - **Whether a row is being worked on.** It has no state, no claim, no assignee. Two operators working the same list will both act on the same row. The Midtrans dashboard and `POST /invoices/{id}/pay` are the coordination points: pay is idempotent, and a Midtrans refund is visible immediately.