Engineering Practitioner Brief / 18 May 2026

Parallel Run Refactor Cost

The parallel-run pattern is the highest-confidence refactor verification technique available. Both the old and new implementations execute on every real production request; a comparator records every divergence. The pattern catches behavioural regressions that test suites miss because production traffic contains the edge cases that synthetic tests do not. The cost is real and the benefit is concrete; this page works through both.

The Pattern

The pattern was articulated most clearly by GitHub in 2015 when they published the ScientistRuby library and the supporting blog post. The Scientist API is built around the experiment metaphor:

# Pseudocode for the Scientist pattern
result = Scientist.run("user-permissions") do |experiment|
  experiment.use { current_implementation(args) }   # control
  experiment.try { new_implementation(args) }       # candidate
end
# 'result' returns the 'use' value
# Scientist records divergences asynchronously

Both implementations run on every call. The control's result is what the caller sees. The candidate's result is compared to the control and any divergence is recorded for later analysis. The candidate runs asynchronously so its slowness does not affect user-facing latency.

The Scientist pattern has been ported to many languages: JavaScript (scientist.js), Python (laboratory), Java (multiple ports), .NET (Scientist.net), Go (scientist-go). The underlying concept transfers without changing.

Service-Level Parallel Run with Diffy

Twitter Diffy generalises the Scientist pattern to whole HTTP services. Diffy sits as a proxy in front of three running services: the current production (primary), the new candidate, and a control instance of the current production. Diffy forwards every request to all three, then compares the candidate's response to the primary, using the differences between primary and control to subtract out non-deterministic noise.

The control-subtraction step is the clever part. Many services have legitimate non-determinism (timestamps, random IDs, ordering of result lists). Without the control, every response would diverge on these fields and the real signal (actual logic differences) would be drowned in noise. By comparing two production instances against each other in addition to candidate-vs-production, Diffy can identify which divergences are noise and which are signal.

Diffy is open source and still works in 2026, though its commit cadence is low. Several similar tools exist (diffit, replayed, the various traffic-replay frameworks). For most teams the choice is between Diffy and building a service-specific comparator using the same pattern with custom rules for known noise sources.

The Cost Breakdown

For a typical service-level parallel run lasting four weeks, the total cost is 200 to 800 engineer-hours plus marginal infrastructure. At $85 per hour, $17,000 to $68,000 per parallel-run campaign. For high-stakes services (payments, pricing) this is straightforwardly worth doing. For low-stakes services, a feature-flag rollout is usually sufficient.

When Parallel Run Pays Back

Three categories of refactor justify the parallel-run cost:

Pricing and billing logic. Customer-visible dollar amounts that differ between old and new implementations are catastrophic. A single billing-side regression can cost more than the entire parallel-run infrastructure for the year. The conservative pattern is to run pricing changes in shadow mode for 4 to 8 weeks before cutover, comparing every line item.

Regulatory calculations. Tax computations, financial reporting, healthcare claims processing. A divergence between old and new might be invisible to users but visible to auditors. Parallel-run with full divergence logging provides the evidence trail that the new system matches the old before being granted regulatory cutover.

Machine-learning feature pipelines. Subtle differences in feature engineering between old and new pipelines silently degrade model performance. Parallel-run comparison of feature values for the same input is the only reliable way to verify pipeline parity. The cost is high (feature pipelines have wide outputs) but the cost of silent ML degradation is higher because it surfaces as gradual revenue erosion that is hard to root-cause.

The Non-Determinism Problem

The harder part of any parallel-run setup is handling non-determinism. Most production code has incidental non-determinism that does not reflect logic differences:

• Timestamps in responses (always different between calls).
• Generated IDs (UUIDs, sequence numbers).
• Floating-point arithmetic that may round differently between implementations.
• Ordering of items in result lists (set semantics vs list semantics).
• Cache state (the candidate has an empty cache; the production instance has a warm one).
• External-service dependencies that respond differently on retry.

Each source of non-determinism either needs to be subtracted out by the comparator (Diffy's control-vs-control approach) or accepted as background noise and the divergence threshold raised. Either path requires investment in the comparator setup; this is why the one-time setup cost is the largest single line in the parallel-run budget.

Exit Criteria

A parallel-run that exits too early misses meaningful divergences. A parallel-run that runs forever consumes engineer attention indefinitely. The standard exit criteria, applied in combination:

1. Divergence rate has stabilised at or below the agreed threshold for at least 2 consecutive weeks.
2. All categories of divergence are explained: each one is either accepted (known non-determinism) or fixed (a real bug in the candidate).
3. The run has covered the major seasonal and event-driven patterns of the service's traffic. A payment service that has not yet seen a Black Friday should not exit parallel-run before November.
4. The engineers most familiar with the candidate sign off on the comparison results.
5. The rollback plan if cutover-time issues emerge is documented and rehearsed.

A typical service meets these criteria within 4 to 8 weeks of starting parallel-run. Some services (regulatory, payment-critical, ML pipelines with long evaluation cycles) need 12 to 24 weeks. The length is dictated by the traffic pattern and the divergence rate, not by a fixed calendar.

Related Reading

• Strangler fig migration cost
• Feature flag refactor cost
• Blue-green codebase cost
• Big-bang rewrite cost
• Legacy code refactoring cost

Frequently Asked Questions