Isolating a Slow Rule on a Hot Path with Per-Rule Profiling

The problem

An API quota check runs on the hottest path a SaaS product has. Every request asks the same question before doing any work: is this tenant still inside its plan's monthly allowance. The answer has to arrive in the time budget of a middleware, not a background job.

That check starts as three comparisons and does not stay there. A plan tier arrives. A segment routing rule joins to tag requests for a rollout. A fail-closed rule lands after an incident. Each addition is small and each is justified, and none of them is the one anyone points at when the latency graph moves.

Then p99 moves. The APM span says policy evaluation 12ms and stops there, because the rules are data, not stack frames — a CPU profiler samples the evaluator, not the policy. Averages hide it further: a rule that is fast on most requests and slow on the ones carrying a long tag list disappears into a mean.

The question this pattern answers: how does a team find out which rule in a hot-path decision got slower, separately from the request that happened to carry it, and decide on that evidence whether the rule needs changing.

The naive approach

The first version keeps the rules in code and wraps them in timing.

public class QuotaGate {

    private static final Map<String, Long> QUOTAS =
            Map.of("free", 10_000L, "growth", 100_000L, "pro", 1_000_000L);

    public Decision check(Request req) {
        long t0 = System.nanoTime();
        Decision d = evaluate(req);
        long elapsed = System.nanoTime() - t0;

        // One number for the whole gate. Which branch spent it is not recorded.
        log.info("quota_gate_ns={}", elapsed);
        return d;
    }

    private Decision evaluate(Request req) {
        Long quota = QUOTAS.get(req.plan());
        if (quota == null) {
            return Decision.deny("plan outside known plans");
        }
        if (req.callsThisMonth() > quota) {
            return Decision.deny("monthly quota exceeded");
        }
        // Added later, for a rollout. Walks the tag list on every request.
        if (matchesAnySegment(req.tags())) {
            return Decision.allowWithRoute(routeFor(req.tags()));
        }
        return Decision.allow();
    }
}

It works at small scale, and the timing line was worth adding. What breaks is not the logic but the measurement, and it breaks in three ways.

• The instrumentation is one number wide. quota_gate_ns covers every branch at once. To split it, a timer goes around each if, and then the timers have to be maintained alongside the rules they wrap. The measurement drifts from the logic the moment someone reorders a branch.
• A log line is not a distribution. What ships is a mean, or a count of slow requests over a threshold somebody guessed. The question that matters — how much slower is this branch than its siblings — needs percentiles per branch. A log aggregator computes those only if every branch emits its own field.
• The first call and the thousandth are averaged together. The first request after a deploy pays for loading and compiling the ruleset; the rest do not. Mixed into one series, that cost looks like sporadic tail latency, and the tail is exactly what the team is trying to explain.

Defining the pattern

The fix is to stop instrumenting the decision and let the engine account for it. When the rules are data in a policy version, every evaluation already splits into rule and phase. The engine records where the time went without anyone wrapping anything.

The accounting names three things in LexQ terms: what the engine reads, what it evaluates, and the two phases each rule is measured in.

• Fact: what the middleware hands the engine on every request. tenantPlan, apiCallsThisMonth, userTags.
• Rule: one rule per quota decision, plus the ones that accumulated — a segment router, a fail-closed catch-all.
• Phase: each rule is measured in two parts. CONDITION is the time to evaluate the condition tree; ACTION is the time to run the actions of a rule that matched. They are reported separately because they answer different questions.

The quota rules partition requests by plan tier, and the segment router writes its own fact, so no mutex group applies.

{
  "name": "Block: free plan over monthly quota",
  "condition": {
    "type": "GROUP",
    "operator": "AND",
    "children": [
      {
        "type": "SINGLE",
        "field": "tenantPlan",
        "operator": "EQUALS",
        "value": "free",
        "valueType": "STRING"
      },
      {
        "type": "SINGLE",
        "field": "apiCallsThisMonth",
        "operator": "GREATER_THAN",
        "value": 10000,
        "valueType": "NUMBER"
      }
    ]
  },
  "actions": [
    {
      "type": "BLOCK",
      "parameters": {
        "reason": "apiCallsThisMonth exceeds the free plan quota 10000"
      }
    }
  ],
  "isEnabled": true
}

The rule that accumulated looks different. Its condition is an OR of eight AND-groups, each opening with a list operation over userTags. It runs on every request the same way the quota rules do.

What the engine will not do is tell you a rule is slow in absolute terms. A rule is flagged when its p50 reaches ten times the median of the per-rule p50s in the same group — a moving comparison against its own siblings. The engine supports no absolute threshold, by design. What counts as slow depends on the host, the ruleset, and the traffic, and a threshold written into a document outlives all three.

Impact Simulation strategy

Profiling says which rule costs the most. It does not say the rule is safe to change. Splitting an eight-branch condition into narrower rules, or reordering it so the cheap comparison runs first, is a policy edit like any other. What has to hold before it ships is that the decisions do not move.

Duplicate the live version, restructure the expensive rule, and run the candidate against historical execution data with that live version as the baseline. The pass condition here is inverted from most simulations: the interesting number is zero.

lexq analytics simulation start --json '{
  "policyVersionId": "<candidate-version-id>",
  "dataset": {
    "type": "HISTORICAL",
    "source": "EXECUTION_LOGS",
    "from": "2026-07-01",
    "to": "2026-07-31"
  },
  "options": {
    "baselinePolicyVersionId": "<baseline-version-id>",
    "includeRuleStats": true,
    "maxRecords": 50000
  }
}'

Two comparisons decide whether the restructured rule ships. First, every rule's match rate must be unchanged against the baseline. A restructured condition that matches a different set of requests is a policy change wearing a performance change's clothes, and the rule statistics are what separate the two. Second, the decision mix must be identical — same blocks, same passes, same reasons. Testing a rule change before deploy works through that comparison as a pattern of its own; here it is used to prove a change is inert rather than to size one.

The latency question is worth asking once the simulation reports no behavioral difference, and the profile of the live candidate settles it.

Decision Trace output

The engine traces this decision the way it traces any other — the quota rules carry NO_MATCH or SELECTED, and a block arrives in generatedVariables as isBlocked with its blockReason. What this pattern adds is a second output over the same executions: the profile.

{
  "policyGroupId": "<group-id>",
  "policyVersionId": "<version-id>",
  "ruleCacheState": "HIT",
  "droppedRows": 0,
  "summary": [
    {
      "cacheState": "HIT",
      "total": { "n": 16199, "p50Nanos": 1490943, "p95Nanos": 3571711, "p99Nanos": 8519679 }
    },
    {
      "cacheState": "MISS",
      "total": { "n": 2, "p50Nanos": null, "p95Nanos": null, "p99Nanos": null,
                 "minNanos": 48234496, "maxNanos": 81264639 }
    }
  ],
  "baselines": [
    { "phase": "CONDITION", "baselineP50Nanos": 1535, "cohortSize": 6, "status": "OK" },
    { "phase": "ACTION", "baselineP50Nanos": null, "cohortSize": 1, "status": "INSUFFICIENT_COHORT" }
  ],
  "rules": [
    {
      "ruleId": "<segment-routing>",
      "phases": [
        { "phase": "CONDITION", "stats": { "n": 177, "p50Nanos": 4095, "p95Nanos": 9727, "p99Nanos": null },
          "baselineMultiple": 2.67, "flagged": false },
        { "phase": "ACTION", "stats": { "n": 168, "p50Nanos": 24063, "p95Nanos": 99839, "p99Nanos": null },
          "baselineMultiple": null, "flagged": false }
      ]
    }
  ]
}

summary is the group total, recorded for every call and split by cache state. baselines is the comparison the flag is computed against, derived per phase. Each rule reports n beside every percentile, and baselineMultiple places it against its siblings. Here the accumulated router evaluates its condition at 2.67 times the group's median per-rule p50 — the highest of the six, and still unflagged. The number to act on is the ranking, not a threshold.

Edge cases

The measurement method carries over to any hot-path ruleset, and the quota rules here are only the case at hand. A few properties of the data limit what these numbers can prove.

• Rule detail is sampled; the group total is not. Every call contributes to summary, while the per-rule table comes from a deterministic 1% sample. A group serving thousands of requests still yields a per-rule n in the low hundreds, so the rule table rests on two orders of magnitude fewer observations than the total. Size the window accordingly rather than reading a fresh deploy's first minutes as a rule-level signal.
• The display gate and the judgment gate are different. A percentile is withheld — null, never estimated — unless n × (1 − q) ≥ 3, which puts p50 at n ≥ 6, p95 at n ≥ 60, and p99 at n ≥ 300. Judgment has its own gate at n ≥ 100, and baselines reports INSUFFICIENT_COHORT when fewer than three rules qualify. A table can therefore show percentiles with no flag computed against them, which is a statement about sample size and not about the rules.
• CONDITION and ACTION do not share a sample size. Conditions are evaluated on every call, so every rule's CONDITION carries the same n. Actions run only when a rule matches, so a rule that rarely fires may never reach the judgment gate in ACTION while its CONDITION clears that gate comfortably. That asymmetry is why the two phases are reported separately.
• HIT and MISS are separate populations. A MISS is a deep load plus a compile, and it is paid by the first calls against a version. Mixed into one distribution it reads as tail latency; kept separate it is a startup cost with a known cause. Read HIT for steady state and MISS for what a deploy costs the first requests through the new version.
• The series is not smoothed. Per-window points carry that window's own values with no display gate applied, so a window holding a single call reports the same number as its p50, p95, and p99. Missing windows are genuine gaps in traffic and are never interpolated. Read the series for shape and the merged distribution for magnitude.

Production rollout

A candidate that clears both comparisons moves to production with Deploy. What goes out is a hashed, integrity-verified snapshot of the rules, and the deployment record names the version, the person, and the time.

A restructured rule can ship to all traffic at once, because the simulation already established that no decision changes. What deploy adds is the measurement the simulation could not produce: the profile of the new version under real traffic. Take the profile from before the change as the comparison, and read a window of the same length after the deploy.

Roll back at once on either of these readings in the post-deploy profile:

• The restructured rule's baselineMultiple has not moved, or has moved the wrong way. The cost does not sit where the profile placed it, so the rewrite buys nothing.
• The group's MISS distribution stays populated well past the rollout. New versions are being loaded more often than the deploy accounts for, and the cost being measured is compilation rather than evaluation.

A rollback returns the policy group to the previous version and leaves a deployment record, so the rollback itself stays in the audit trail. Once the candidate serves all traffic, the per-rule table is the standing answer to which rule in a hot-path decision spends the time — a query scoped to a group, a version, and a window, rather than a timer somebody remembered to add.

---

See how LexQ works for yourself in the playground.