Average — the New Quartile

By Jedd Fowers, Senior Benchmarking Consultant, EDS

It’s a common human trait: No one wants to be an average. Customers of outsourcing services are also no different. They want better than average services for less than average rates. Yet when it comes to pricing for IT services, making that better-than-average distinction is more challenging than it may seem for firms who measure it. 

As it turns out, outsourcing customers, service providers and third-party experts have changed the way price benchmarking is conducted in the outsourcing deals. These changes involve the realization that some former benchmarking practices, including the limited use of “top quartile” or “lowest quartile” pricing standards (i.e. best 25 percent), are not feasible in practice.  Today’s improved methods allow for better, more accurate measures of an outsourcing deal’s price competitiveness.

First, it’s helpful to understand how we reached the current situation.

Benchmarking History
Price benchmarking of outsourced IT services began in the late 1990s.  Customers were signing long-term deals but were concerned that prices they agreed to today might later become decoupled from market rates, especially as technology advanced and drove down costs.  To ensure long-term, competitive rates, customers began requesting the right to conduct periodic market-price assessments to ensure the outsourcer continued to deliver value for money.  These contractual rights became known as a contract’s benchmarking clause, a provision giving the customer the right to engage a third party to compare the provider’s current prices against those in the prevailing market. If discrepancies were found, the provider might modify its rates to better align with the market.

Often, a benchmarking clause will stipulate the level at which the price comparison should occur.  As an example, a benchmarker will typically select four similar contracts, or peers, as a basis for the comparison.  After “normalizing” the peers’ prices to adjust for inherent differences, the benchmarker calculates the average of the peers’ adjusted prices to produce a market-competitive level.  This simple average calculation is the basis for the comparison and is the industry-standard approach.

The Problem
Early on, a few customers requested better than average pricing in their benchmarking clauses.  Statistical concepts such as “top quartile” or even “top decile” made their way into contracts’ benchmarking wording. Some outsourcers and customers agreed to these more stringent comparisons without a good understanding of the inherent limitations imposed by such measures. In some cases, actual benchmarking results using quartile-type comparisons showed radical and unexplainable discrepancies between an outsourcer’s price and the market; so different, in fact, that both the outsourcer and the customer agreed that something must be wrong. This caused frustration — even lawsuits — as the parties debated over
confusing and unworkable benchmarking outcomes.

What Went Wrong
Since it is mathematically impossible for “all” customers to receive better than average rates, benchmarkers and industry thought leaders realized that a misapplication of statistical norms was the main cause of confusion.

Statisticians agree that for calculations such as quartiles to have statistical significance, the number of underlying observations (or peers) must meet some minimum. Most statisticians, according to the study titled Formal Benchmarking in Outsourcing Contracts, by Technology Partners International (TPI), the industry’s preeminent outsourcing advisor, in Mar. ’07, conclude that 60 or more observations are necessary to obtain a solid, robust result.

Herein lies the problem: Most benchmarkers only use four to six peers in their benchmarking studies. These relatively small peer groups are a function of most benchmarkers’ modest database sizes, as well as the complexity and variation inherent in modern outsourcing deals. Indeed, it is rare for a benchmarker to find six contracts in its database that share enough similarities with the target contract for a reasonable comparison to take place.

Since benchmarkers could not produce a statistically significant number of peers, a top quartile benchmarking result from a small group (say, four peers) often dictated that the outsourcer simply match the lowest-priced peer in the group. This statistically flawed approach often led to disagreements.

Not only that, according to the said study, because the benchmarking process involves sampling — an attempt to derive the qualities of a population through a subset of that population — there is always some margin of error implicit in the result (a reasonable margin of error is plus or minus 10 to 15 percent). That margin of error is compounded when the results are further sliced through quartiling attempts from a small group of observations.

In short, if statistical calculations are used, statistical norms must apply.  That wasn’t occurring in the past — a problem that caused more than one sticky situation for customers and outsourcers. These challenges would continue without some course correction.

The Solution
Modern price benchmarking practices have coalesced around a workable comparison standard, one which is equitable to both parties. Leading benchmarkers’ price comparison products no longer include quartiling (or other percentiling) approaches.  Instead, benchmarkers use the normalized average (or mean) price of a group of peers as the optimal comparison level.

TPI supports this approach and also points to the need for recognizing a margin of error in the outcome.  Effectively, this process produces a reasonable price range within which the outsourcer’s price should fall.
Does this mean that the post-benchmark customers are bound to get mediocre pricing and services? No! In fact, quite the opposite can occur. If done right, a benchmark can provide a good measure of the “right” price for the specific service in question, especially when the benchmarker uses peer data from outstanding outsourcing contracts. Also, pricing is just one of the many factors that determine the overall health and competitiveness of an IT outsourcing deal. Customers could also consider broader questions such as “do we have the best operational solution for our needs?” and “how does our outsourcing governance practice compare to others?” 

Viewing and benchmarking a deal holistically — considering the “overall” value from a price, performance and relationship perspective — is the best path to achieving “best practice” outsourcing success.

Jedd is a senior benchmarking consultant with EDS and a veteran of the IT-services industry for 13 years. He has consulted over 300 clients on benchmarking and pricing matters.