Backtest and run a shadow period

Backtest on your history

A backtest replays a few months of your real history through Runledger in the sandbox, on a test clock, and compares the invoices it makes with the ones you sent. It shows what settlement would have changed before any customer is involved.

1. Export your history: every outcome with its reference and time, every later signal (reopens, escalations, satisfaction scores) with its reference and time, and the complaints your team decided.
2. In the sandbox, create the same credit types, outcome definitions and plans you’ll use live, with the scripts you’ll run against live.
3. Create a test clock at the start of the history, and your customers on it.
4. Replay in steps shorter than your shortest confirmation window, such as an hour. Advance the clock to the end of each step, wait until it’s ready, then send the step’s events in time order, up to 100 to a batch. Events keep their original timestamps; on a test clock, the 35-day limit counts back from the clock’s time.
5. Open each complaint as a dispute in the step it arrived, and resolve it the way your team did.
6. Compare each finalized invoice with what you billed for the same period.

import { Runledger, RunledgerClient } from "@runledger/sdk";
// What your current billing system knows about one customer.
interface ExistingAccount {
  orgId: string;
  name: string;
  planKey: string;
}
const sandbox = new RunledgerClient({ apiKey: process.env.RUNLEDGER_SANDBOX_KEY }); // rl_sk_test_...
// Your history in time order, cut into steps shorter than your shortest confirmation window.
interface ReplayStep {
  end: string; // RFC 3339
  events: Runledger.TrackRequest[]; // outcomes and signals, with their original timestamps
}
export async function backtest(accounts: ExistingAccount[], historyStart: string, steps: ReplayStep[]) {
  const clock = await sandbox.testClocks.create({ frozen_time: historyStart, name: "backtest" });
  const customerIds = new Map<string, string>(); // your org ID to Runledger's cus_ ID
  for (const account of accounts) {
    const customer = await sandbox.customers.create({
      external_id: account.orgId,
      name: account.name,
      currency: "usd",
      test_clock: clock.id,
    });
    customerIds.set(account.orgId, customer.id);
    await sandbox.subscriptions.create({
      external_customer_id: account.orgId,
      plan: account.planKey,
      collection_method: "send_invoice",
    });
  }
  for (const step of steps) {
    // Move the clock first, so no event is dated in the clock's future.
    await sandbox.testClocks.advance({ test_clock_id: clock.id, frozen_time: step.end });
    while ((await sandbox.testClocks.retrieve({ test_clock_id: clock.id })).status === "advancing") {
      await new Promise((resolve) => setTimeout(resolve, 1000));
    }
    for (let i = 0; i < step.events.length; i += 100) {
      const { results } = await sandbox.events.batch({ events: step.events.slice(i, i + 100) });
      for (const result of results) {
        if (result.status === "rejected") console.warn(result.idempotency_key, result.error?.code);
      }
    }
  }
  // Runledger's finalized invoices, to set beside the ones you sent for the same periods.
  const invoices: { orgId: string; periodStart: string; total: number }[] = [];
  for (const [orgId, customerId] of customerIds) {
    for await (const invoice of await sandbox.invoices.list({ customer_id: customerId })) {
      if (invoice.status !== "draft") invoices.push({ orgId, periodStart: invoice.period_start, total: invoice.total });
    }
  }
  return invoices;
}

Plan around two limits: a clock moves at most 90 days per call, and the sandbox deletes a clock and its customers 30 days after you create it (deletes_at), so record the results before then.

Expect differences, and read the reversals and disputes behind each one:

• A charge still pending when a period closes is billed in the next period, so totals can move between months.
• Customers who came back and were never credited show up as reversals your team didn’t make.
• Outcomes your old system counted differently show up as extra charges, or missing ones.

Run a shadow period

A shadow run sends your real production traffic to the sandbox for one full billing period while your current billing keeps sending the invoices. It tests the integration at your real volume and timing.

• Create sandbox customers that mirror your live ones with a sandbox key, a customer and a subscription for each account, as the backtest above does. Leave the billing email out, so nothing from the sandbox reaches a customer.
• Point production at the sandbox with a restricted sandbox key (rl_rk_test_...), and turn enforcement off: record what check() would have denied, and let the work run.
• Register a sandbox webhook endpoint. Your handler should skip side effects, such as emailing an invoice, for events with livemode: false.
• Sandbox rate limits are lower than live ones. If your volume needs more, ask us to raise yours for the shadow run.

Call gateWork() wherever you called checkOrFailOpen():

import { Runledger } from "@runledger/sdk";
import { checkOrFailOpen, type Gate } from "./runledger.js";
const enforce = process.env.RUNLEDGER_ENFORCE === "true"; // false for the shadow run
export async function gateWork(request: Runledger.CheckRequest): Promise<Gate> {
  const gate = await checkOrFailOpen(request);
  if (!gate.allowed && !enforce) {
    metrics.increment("runledger.check.would_deny", { reason: gate.reason ?? "unknown" });
    return { ...gate, allowed: true };
  }
  return gate;
}

At the end of the period, compare each customer’s Runledger invoice with the one you sent, as in the backtest, and review the denials enforcement would have caused. The shadow run is done when every difference is explained and your team has decided a real dispute from its queue.

Next steps

• Cut over to live switches production to the live key.