# Is Provider B Really Better? The Sample-Size Math Behind Every PSP Comparison

Canonical URL: https://y.uno/en/blog/is-provider-b-really-better-the-sample-size-math-behind-every-psp-comparison

> This is the markdown rendition for AI agents. The canonical page is served as HTML at the URL above.

By Yuno · Published 2026-10-30 · Payment orchestration

Most head-to-head PSP comparisons are decided on too little data. This piece teaches the sample-size math that separates a real approval-rate difference from random noise, and shows how to stage a valid test.

What you will learn: How to calculate the number of transactions you need before a PSP comparison sample size is large enough to trust, why splitting results by BIN group, country, or card type multiplies that requirement by the number of segments, and how to move through four test stages without betting your volume on noise.
The thesis here is direct: most head-to-head provider comparisons are noise. A two-point approval gap measured on a week of traffic in one segment usually proves nothing, according to standard hypothesis-testing principles. Yet switching decisions get made on exactly that evidence, every week, at companies of every size.

## Key takeaways

- Detecting a real 2-point approval-rate gap at an 85% baseline requires roughly 4,720 transactions per provider, not a week of mixed traffic.
- Every segment you add (BIN group, country, card type) multiplies the volume requirement by the number of segments.
- An 18-cell segmented test needs about 84,960 transactions per provider, versus 4,720 for the topline.
- Shadow and canary stages prove integration health. Only a properly sized statistical split can prove an approval-rate difference is real.
- When comparing providers, debate results at adequate volume, not concepts, narratives, or a single week of data.

## Why the number that looks real usually is not
Any two groups will produce a different number when measured over a short window. That is variance, not insight.
According to Deloitte's guide to hypothesis testing, when comparing two groups, there will almost always be a difference greater than zero. The question is whether that difference is larger than what random chance alone would produce. A two-point gap on a small sample at an 85% approval baseline is often well within the range of noise, and the gap needs to clear a statistical threshold before it is actionable.
Most payment comparisons skip this step entirely. A new provider runs for a week. One number comes in higher. Someone in a meeting says "Provider B is better." The router gets updated. The real cause of the gap (a different card mix that week, a public holiday in one market, a BIN skew in the sample) never gets examined.

## The formula every head of payments should know
The pooled two-proportion z-test gives you the minimum transactions per provider before a measured gap means anything. You do not need a statistics degree to use it.
The formula is:
n = ((z_alpha/2 + z_beta) / delta)² × 2 × p_pooled × (1 − p_pooled)
Where delta is the gap you want to detect as a proportion, p_pooled is the average of the two approval rates, z_alpha/2 is the critical value for 5% significance, and z_beta is the critical value for 80% power. These significance and power levels are standard conventions in controlled testing.
At an 85% baseline, detecting a 2-point gap gives a p_pooled of (0.85 + 0.87) / 2 = 0.86. Plugging in the standard critical values:
n = ((1.96 + 0.84) / 0.02)² × 2 × 0.86 × 0.14 ≈ 4,720 per provider
That is roughly 9,440 transactions total before a 2-point gap clears the statistical bar. A 1-point gap needs about 19,440 per provider. A half-point gap needs about 77,760 per provider. The relationship is not linear: as the gap you want to detect shrinks, the volume requirement grows steeply. That is the core lesson of this piece.
Read the curve carefully. The x-axis is the gap you are trying to detect. At 5 points, you need relatively modest volume. At 1 point, you need roughly four times what a 2-point test requires. At 0.5 points, the requirement climbs to around 77,760 per provider. Most providers compete in the 1-to-3-point range. That range is expensive to prove.
For more on how approval rates behave across different routing strategies, see best practices for improving payment authorization rates globally.

## Worked example: one merchant, two providers, three questions
This company is hypothetical. Every number below is stated explicitly as an assumption or derived from the formula above.
Assume a mid-size subscription merchant processing cards globally. Baseline approval rate: 85%. They want to know whether Provider B is 2 points better than Provider A. They also want to break results down by three BIN groups, three countries, and two card types.
Assumptions:

- Baseline approval rate (Provider A): 85%
- Minimum detectable effect: 2 percentage points
- Statistical power: 80%
- Significance level: 5% (two-tailed)
- Segments: 3 BIN groups × 3 countries × 2 card types = 18 cells
The topline test (no segmentation) requires 4,720 per provider, or 9,440 total. That is the floor. Without segmentation, the merchant cannot tell whether Provider B is better in Germany on Visa debit but worse in Mexico on Mastercard credit. They see only one blended number.
The segmented test requires 4,720 per provider per cell. With 18 cells, that is 84,960 per provider, or 169,920 transactions total across both providers. That is 18 times the topline requirement.

Image: Segmenting by 18 cells multiplies required volume 18-fold. Transactions per provider. Hypothetical: 85% baseline, 2-point gap, 80% power, 5% significance. Segments: 3 BIN groups × 3 countries × 2 card types = 18 cells. Topline test (no segments) vs 18-cell segmented test: 4,720 vs 84,960 transactions per provider.
Segmenting by 18 cells multiplies required volume 18-fold. Source: Pooled two-proportion formula; 80% power and 5% significance are standard conventions
At 10,000 transactions per day split across both providers, the topline test clears in about one day. The 18-cell segmented test takes about 17 days. Most merchants running a "one-week test" are nowhere near the volume needed to validate segment-level results. They are comparing noise to noise.

### What drives the multiplication
Each segment cell is an independent comparison. Adding a third country does not share sample size with the first two countries. Each cell needs its own 4,720 per provider. If traffic is unevenly distributed across countries, some cells fill slowly and others fill fast. The test is only complete when the slowest cell reaches its target.
This is why the question "which payment platform handles high-volume cross-border transactions better?" has no single honest answer. Better on which corridor, with which card type, at which ticket size? Each sub-question needs its own sample. Blending them into one number produces a result that reflects your traffic mix, not provider quality.

## The four stages of a valid provider test
A valid test moves through four stages, each with a specific purpose and a hard limit on what it can prove.

Image: Only a sized statistical split can prove an approval-rate difference is real. Illustrative staging framework. What each stage can and cannot prove. 1. Mirror traffic, no charges. 2. Send 5-10% of live traffic. 3. 50/50 at required sample size. 4. Route by segment performance.
Only a sized statistical split can prove an approval-rate difference is real. Source: Illustrative

### Stage 1: Shadow
Mirror all traffic to the new provider without charging the customer. Every transaction goes to your existing provider and simultaneously to Provider B in read-only mode. This stage proves the integration is healthy. It surfaces technical errors, timeout patterns, and response-code mismatches. It cannot prove an approval-rate difference because no real authorisations occur on the shadow side.

### Stage 2: Canary
Send 5 to 10 percent of live traffic to Provider B. This catches real-world errors that shadow testing misses: authentication failures, currency mismatches, 3DS edge cases. A canary stage proves the provider can handle live volume. It cannot prove the approval-rate gap is real, because the volume reaching Provider B is a fraction of what a valid split requires.

### Stage 3: Statistical split
This is the first stage where a valid claim about approval rates becomes possible. Randomly assign 50% of eligible traffic to each provider. Run until every cell in your segment plan reaches its sample-size target. Do not stop early because one number looks good. Early stopping inflates false-positive rates, and this is the stage most merchants skip, moving straight from canary to ramp on a number that has not cleared the statistical bar.

### Stage 4: Ramp
Once the split test confirms a real difference, shift volume toward the winning provider. "Winning" is not permanent, however. Card mix, BIN distribution, and acquiring rules change. A result validated on last quarter's traffic may not hold on next quarter's. Set a re-test cadence. Payment routing that is set and not revisited degrades silently.
For a deeper look at routing mechanics that affect approval rates at each stage, see payment routing: seven ways to boost your approval rates.

## Where Yuno fits in this process
Running a statistically valid PSP comparison requires routing the same transaction types to two providers simultaneously, tracking outcomes at the cell level, and switching the winner on without rebuilding your integration. That is what a neutral orchestration layer is designed to do.
Yuno has processed over one billion payments across 150 or more currencies since January 2025. Its Smart Routing delivers an average 8% approval uplift by choosing the best route per transaction across providers, rather than committing all volume to a single answer. The routing logic is informed by observed outcomes per BIN, country, card type, and time of day, which is exactly the segmentation a valid test requires.
Yuno does not touch the funds or own acquiring. Its routing recommendations carry no conflict of interest. Merchants asking which payment platform has the best smart routing for higher authorization rates get an answer grounded in actual segment-level performance on their own traffic, not a vendor's aggregate benchmark.
McDonald's across 21 countries in Latin America (Arcos Dorados) and ride-hailing platforms like inDrive have used this architecture to manage provider performance across markets where card mix and local acquiring rules vary significantly by corridor. The mechanics are the same whether you operate in two countries or twenty: each corridor needs its own sample before the result is actionable.
For more on how merchants with strong approval rates approach routing, see the merchants with the highest approval rates all use smart routing.

## How to act on this now
The immediate action is not to run a new test. It is to audit what your last test actually proved.
Pull the transaction count from your most recent provider comparison. Identify the minimum detectable effect it could have supported at 80% power and 5% significance. If the gap you concluded was real falls inside the noise band for that sample size, the conclusion was premature. That does not mean the provider is bad. It means you do not yet know.
From there, use the formula in this piece to set volume targets before your next split test begins. Define your segments upfront. Run shadow and canary stages to confirm integration health. Do not call the winner until every cell reaches its target. Then ramp, and schedule a re-test. As Deloitte's hypothesis-testing guidance makes clear, observed differences between groups must be evaluated against statistical thresholds before they are actionable. A two-point gap on a week of traffic is a hypothesis. Run the math before you route the money.