Make your integration production-ready

The steps, in order:

Make checks safe to depend on

check() is the call your users wait for, because it runs before the work starts. Runledger is hosted in Australia, so from North America or Europe each call makes a long network round trip, and an outage at Runledger shouldn’t become an outage in your product.

Check once per run

Call check() when a conversation or run starts, and hold its whole budget: the price of one outcome, or an estimate for usage. Capture the hold with track() when the work finishes, extend it (in whole days) if a long run outlives it, and release it if the work fails. A check before every step adds a round trip each time and reserves nothing the first hold didn’t. Checks and holds covers sizing and expiry.

Fail open when Runledger doesn’t answer

Decide what your product does when check() times out or Runledger is down:

Fail open only when Runledger didn’t answer, or answered with an error marked retryable. A 400 or 401 means your request or key is wrong, and failing open on it would hide the bug until the invoices are wrong. Pick a timeout your product can afford: when the check runs alongside the first step of the work, as in the next section, a second or two costs your users nothing.

// runledger.ts
import { Runledger, RunledgerClient, RunledgerError, RunledgerTimeoutError } from "@runledger/sdk";
export const runledger = new RunledgerClient(); // reads RUNLEDGER_API_KEY
// True when Runledger didn't answer, or answered that the request can be retried.
export function isRetryable(error: unknown): boolean {
  if (error instanceof RunledgerTimeoutError) return true;
  if (!(error instanceof RunledgerError)) return false;
  if (error.statusCode === undefined) return true; // no response: a network error
  const body = error.body as { retryable?: boolean } | undefined;
  return body?.retryable ?? (error.statusCode === 429 || error.statusCode >= 500);
}
export interface Gate {
  allowed: boolean;
  failedOpen: boolean;
  holdId?: string;
  reason?: Runledger.CheckDenialReason;
}
export async function checkOrFailOpen(request: Runledger.CheckRequest): Promise<Gate> {
  try {
    // Two seconds and no retries: after that, letting the work run beats waiting.
    const check = await runledger.check(request, { timeoutInSeconds: 2, maxRetries: 0 });
    return { allowed: check.allowed, failedOpen: false, holdId: check.hold?.id, reason: check.reason };
  } catch (error) {
    if (!isRetryable(error)) throw error; // a bad request or key is a bug: don't hide it
    metrics.increment("runledger.check.failed_open");
    return { allowed: true, failedOpen: true };
  }
}

The synchronous Python client, Runledger, takes the same request_options.

Start the work and the check together

A caller shouldn’t wait in silence while a check crosses an ocean. Start the check and the greeting together, and act on the result once both are done. A chat agent can do the same with its first model call.

import { checkOrFailOpen } from "./runledger.js";
export async function answerCall(call: Call) {
  // The caller hears the greeting while the check runs.
  const [gate] = await Promise.all([
    checkOrFailOpen({
      external_customer_id: call.orgId,
      outcome_definition: "call_resolution",
      hold: true,
      hold_expires_in: "PT1H",
      run_id: call.id,
    }),
    greet(call),
  ]);
  if (!gate.allowed) return transferToHuman(call, gate.reason);
  return runVoiceAgent(call, gate.holdId);
}

Send events through your queue

track() records work that already happened, so an event should never be lost and never count twice. Put the call on the job queue you already run, such as SQS, Sidekiq, Celery or BullMQ, and enqueue the event when it happens:

• Build idempotency_key from your own IDs, such as conv_4821-resolved, so every retry of the job sends the same key. Runledger keeps keys for 35 days. A new random key on each attempt would bill a retried event twice.
• Set timestamp to when the event happened. Events can be dated up to 35 days back, so a job that waits out an outage is billed in the right period, and one that arrives after its period’s invoice is final goes on the next invoice.
• Let the queue retry errors marked retryable, and set the rest aside for a person: they won’t succeed on retry.

import { Runledger } from "@runledger/sdk";
import { isRetryable, runledger } from "./runledger.js";
// When the agent resolves a conversation: enqueue the event with your own key and the real time.
export async function onResolved(conversation: Conversation) {
  const event: Runledger.TrackRequest = {
    idempotency_key: `${conversation.id}-resolved`,
    name: "resolution.chat",
    external_customer_id: conversation.orgId,
    reference: conversation.id,
    timestamp: new Date().toISOString(),
    hold: conversation.holdId,
  };
  await queue.add("runledger.track", event);
}
// The queue's worker. Throwing makes the queue retry the job later, with the same key.
export async function trackJob(event: Runledger.TrackRequest) {
  let result: Runledger.Event;
  try {
    result = await runledger.track(event);
  } catch (error) {
    if (isRetryable(error)) throw error;
    return deadLetter(event, error); // won't succeed on retry: a person looks at it
  }
  for (const warning of result.warnings) await reportWarning(event, warning);
}

Act on warnings

track() succeeds even when an event doesn’t do what you expected, and says so in warnings:

Catch up after an outage

If your queue backs up, drain it with POST /v1/events/batch. It takes up to 100 events and reports each one as created, duplicate or rejected, so a batch that was partly sent is safe to send again. An event dated in a period whose invoice is already final goes on the customer’s current draft invoice, with no warning.

Use one reference per piece of work

Signals find their charge by customer and reference, so every event about the same conversation must carry the same reference. Use the ID of the thing you charge for. When a customer comes back through another channel, such as a call about yesterday’s chat, your code makes the link and sends the chat’s ID as the reference; Runledger can’t make that link for you. One charge per reference explains what happens when an outcome repeats.

Wire up webhooks and disputes

Handle webhooks the way the webhooks page describes: verify the signature on the raw body, drop duplicates on webhook-id, and fetch the object before acting. An endpoint that fails for five days is disabled, and missed events aren’t replayed for you, so alert on failing_since and redeliver by hand after an outage. Besides the events your product already uses, subscribe to these:

Each outcome definition has a decision_window, which you set or take from a preset (seven days in support_chat), and a default_decision, which is reverse unless you change it. A dispute nobody decides in time is reversed. Before you go live:

• Send dispute.decision_due to the people who decide disputes, such as a support lead’s queue, or add a notification rule to email them without code.
• Remember that a dispute still open when a charge’s reversal horizon passes keeps the charge from locking until it’s decided, so a slow queue delays when revenue becomes final.
• Give them the open disputes, soonest deadline first, with the evidence. List disputes filters by status and decide_before.
• Set default_decision to uphold if your contracts say a charge stands unless it’s reversed.

Next steps

• Going live is the one-page plan.
• Idempotency and errors lists every error code and which are safe to retry.