Decision Replay or Impact Simulation: What to Compare a Rule Change Against

LexQ's Impact Simulation checks a rule change by rerunning past traffic through both the new version (the target version) and a baseline version, then comparing the two. Decision Replay reruns only the new version (the candidate) and compares it with the decision each call actually got.

Which of the two you use changes the number you report to the risk team. On one day of 2,000 card transactions, lowering the review threshold from $500 to $300 shows up as 67 more reviews in Impact Simulation's per-rule counts. Decision Replay shows it as 45 changed decisions. Both 67 and 45 are correct. They differ because a hotfix that lowered the threshold to $200 decided half of that day's calls. With the usual $500 version picked as the baseline, Impact Simulation counts changes as if that version had decided every call, while Decision Replay measures them against the decisions that actually went out.

One change, two counts

A card issuer sends transactions from new accounts, those younger than 30 days, to manual review when the amount is above $500. Everything else is approved. The card system calls one LexQ policy group for each transaction, and the group holds two rules. The review rule has the highest priority. A catch-all with an empty condition has the lowest and approves whatever the review rule leaves. Both rules share the mutex group review-route, so exactly one of them decides each transaction.

The risk team wants to lower the threshold to $300. The hotfix mentioned above is v2. Version v3 is the $300 change, and only the review rule differs from the live version, v1:

{
  "name": "Manual review: account under 30 days, over $300",
  "condition": {
    "type": "GROUP",
    "operator": "AND",
    "children": [
      {
        "type": "SINGLE",
        "field": "accountAgeDays",
        "operator": "LESS_THAN",
        "value": 30,
        "valueType": "NUMBER"
      },
      {
        "type": "SINGLE",
        "field": "transactionAmountUsd",
        "operator": "GREATER_THAN",
        "value": 300,
        "valueType": "NUMBER"
      }
    ]
  },
  "actions": [
    {
      "type": "SET_FACT",
      "parameters": { "targetVar": "reviewRoute", "value": "MANUAL_REVIEW" }
    }
  ],
  "mutexGroup": "review-route",
  "mutexMode": "EXCLUSIVE",
  "mutexStrategy": "HIGHEST_PRIORITY",
  "mutexLimit": 1,
  "isEnabled": true
}

The rule reads two facts that the card system sends with each call, accountAgeDays and transactionAmountUsd. Its action writes the decision into a third fact, reviewRoute. LexQ uses the word fact for both kinds of named value: the ones a call carries in and the ones rules write.

The hotfix v2, which lowered the threshold to $200 after a fraud spike, was already live when the day chosen for the check began. The team went back to v1 later that day. Of the 2,000 transactions that day, v2 decided the first 1,000 and v1 the last 1,000.

All three are versions of the same policy group and differ only in that amount. Each check below covers the same 2,000 calls:

The Impact Simulation numbers come from the Rule Statistics on the Simulation Detail screen. Those statistics show counts only for the target version, so 204 came from a run with v3 as the target and v1 as the baseline, and 137 from a second run with the two swapped. When you start a run through the API, these counts appear only if you set includeRuleStats; the console's Include Rule Statistics box is checked by default. The difference between the two counts is a net figure. Here it equals the number of calls v1 and v3 decide differently, because they differ in one direction only. Any amount over $500 is also over $300, so every call v1 sends to review goes to review under v3 too.

The default metric in the same screen's Metric Summary counts inputs where any rule matched. The catch-all approves every transaction the review rule does not match, so every input matches some rule and the metric is 2,000 for both versions. The metric therefore shows this change as 0, even though the two versions decide 67 calls differently. When you start a run, you can also pick a numeric fact (the Metric Target Fact) to sum or average, but that does not help here. The SUM and AVG aggregations read only numeric values, and reviewRoute holds a text value such as MANUAL_REVIEW or APPROVE.

Both checks also read only the newest 1,000 stored calls unless you set a larger maxRecords. Here that would have left out the 1,000 calls the hotfix decided, so every run was set to read at least 2,000.

Why 67 and 45 differ

The three versions disagree only on new-account transactions above $200 and up to $500. All three approve new-account transactions of $200 or less. They also approve every transaction from accounts 30 days or older. All three send new-account transactions above $500 to review. Split that day's new-account transactions above $200 and up to $500 by amount and by the version that decided them, and you get four groups:

Impact Simulation compares the "Under v1" and "Under v3" columns. What actually went out that day never enters the comparison. The columns differ in the second and fourth rows, so 37 plus 30, or 67 transactions, move from approval to review.

Decision Replay compares the "Actual decision that day" column with "Under v3", and now the first and fourth rows differ. v1 approved the 30 in the fourth row, and v3 would send them to review. The 15 in the first row go the other way: the hotfix sent them to review, and v3 would approve them. The 37 in the second row were already in review under v2 and stay there under v3. That leaves 30 plus 15, or 45 changed decisions.

For the risk team, the 67 answers how many more reviews $300 brings per day compared with $500, and the 45 answers how many of the decisions that customers got that day would change.

Besides rerunning a date range (a window replay), Decision Replay can rerun one stored call by its traceId (a single replay). The traceId is the ID LexQ returns with every decision. A single replay of one of the 15 in the first row, a $290.09 transaction from a 14-day-old account, shows the rule behind the stored decision:

{
  "traceId": "e16ac25f-d8fe-404e-8741-b69d729455c3",
  "decisionChanged": true,
  "baselineFired": [
    { "ruleName": "Manual review: account under 30 days, over $200", ... }
  ],
  "candidateFired": [
    { "ruleName": "Approve: everything else", ... }
  ],
  ...
}

baselineFired, shown as Baseline (stored) in the console, is the rule that decided this transaction that day: the $200 rule that exists only in v2. candidateFired, shown as Candidate, is the catch-all that approves it under v3.

Choosing a check

Impact Simulation and window replays count usage in records, one per input they re-evaluate. Each feature has its own monthly allowance, and the two are the same size: 100 records on Free, 10,000 on Growth, and 100,000 on Pro. A single replay of one trace does not count toward usage. Every Impact Simulation run and window replay in this post used 2,000 records.

When Decision Replay is the wrong tool

• The policy group has never served a call. Decision Replay reads stored calls, and a new policy group has none. Run Impact Simulation on a file instead, as in testing a rule change before deploy.
• The candidate reads a new fact. No stored call sent it. In a window replay, a condition on that fact cannot be evaluated, so its rule may not fire, and differences caused by the missing value can end up in the changed count. A single replay stops with an error instead, which makes it a cheap first check.
• The window is older than your retention period. Decision Replay reads calls that are still kept for querying: 30 days on Free, 180 on Growth, and 365 on Pro.
• The window holds more calls than the job reads. One job replays at most 50,000 calls, newest first, and only 1,000 unless you set --max-records in the CLI (maxRecords in the API). A window with more than 50,000 calls has to be split into shorter ones, and their changed counts added up. On the day above, the newest 1,000 are the calls v1 decided, so a v3 replay left at the default would count only the 30 new-account transactions above $300 and up to $500 among them. When the window holds more calls than the job reads, the result sets capped to true.

Window replays skip composite calls, which run several policy groups in one request, and the result does not say how many were skipped. The Decision Replay pattern, which uses a refund-limit rule as its example, works through this and the other edge cases.

When Impact Simulation is the wrong tool

• You need to compare with the decisions customers actually got. Impact Simulation reruns the baseline version you pick instead of reading the stored decisions. On the day above, it counted as if v1 had decided all 2,000 calls, so nothing in its results shows that the hotfix decided the first 1,000.
• You need to open the calls that changed. Impact Simulation's results carry no trace IDs, so you cannot open the calls behind the counts. Decision Replay returns the traceId of up to 100 changed calls, and a single replay shows the rule behind one stored decision.

Impact Simulation was still the right tool for asking how many more reviews $300 brings compared with $500.

When you don't need either

Suppose your logs already keep each transaction's amount, the account's age at the time, and the decision that went out. As long as your rule looks only at amount and account age, as this one does, you need neither check, because one query answers both questions. Counting new-account transactions above $300 and up to $500 gives the 67. Counting every transaction whose logged decision differs from what v3 would decide gives the 45.

→ When your rules outgrow a single query, test the next threshold change on your own transactions before it ships. Start free at lexq.io