Looking for a UX / CRO agency to optimize my online store: audit, retainer or performance-based billing – which should I choose?

Optimizing your online store is one of the fundamental activities that belongs in your e-commerce strategy.

For your store to earn more from every person who visits it, investing in CRO or UX is essential.

As an owner or manager in e-commerce, you can build an in-house store optimization team or hire an outside agency.

In this article I’ll focus on agencies. I’ll tell you what services CRO / UX agencies offer and which of them I’d recommend.

This article is for you if:

1. you want to hire a UX / CRO agency 
2. or you’re considering changing your current agency or freelancer.

A UX / CRO agency – 4 possible engagement models

The CRO / UX agencies you start talking to will probably propose one (or several) of these services:

1. a one-off UX audit
2. individual consultations or studies (usability testing, for instance)
3. a retainer / billing by the hour
4. performance-based billing

Let’s take a closer look at those 4 options.

1. An online store UX audit

There are plenty of different usability audits on the market. I’ll split them into 2 types: basic and extended. 

A basic online store UX audit

What does a standard UX agency offer as part of an audit like that? 

Heuristic (expert) review, checklist-based analysis, or purchase path analysis. 

Those clever-sounding terms can basically be boiled down to this: a basic UX audit involves going through the site and writing a list of things that aren’t done according to some set of guidelines. 

The problem with audits of this kind is that they’re often done without even the slightest attempt to understand the business, without digging into the analytics, and without running any research beyond assessing the store against a list of “universal UX/UI practices”.

An agency employee simply clicks through the site and builds a report on that basis.

And while that doesn’t have to be a bad thing for smaller stores, where traffic and investment budget are modest, for larger stores this approach can lead to serious inefficiency. 

Why? Here’s the answer:

You have to remember that an audit like that is only an opinion. 

Which means not all the conclusions from a purchase path analysis will be right. It’s a subjective assessment, nothing more. 

As the owner of a CRO agency, I often get asked: “You’re the expert, so look at my site and tell me what I should change to raise conversion.” 

And yes, I’ll often spot potential problems someone else might miss. But that doesn’t mean I’m right with 100% certainty. It’s only my opinion. 

And my opinion doesn’t actually matter until I’ve verified it through further research and testing.

I often say my work is among the most honest and measurable there is, because I try to verify every conclusion I reach with another study and an A/B test.

I’d recommend that approach to you too. If your store has already won its first hundreds or thousands of customers and the design isn’t completely amateurish, don’t stop at a basic UX audit.

To sum up: a UX audit shouldn’t be the only, final form of store optimization.

I recommend a basic UX audit for: 

• new stores with modest traffic, few transactions and a small investment budget; 
• stores with very amateurish design.

I don’t recommend it for: 

• stores with over 2,000 transactions a month; 
• stores that don’t have very serious conversion problems;
• stores that want to grow quickly using more advanced, more precise research methods.

An extended online store UX audit

Beyond the purchase path analysis, an audit like that may include business workshops, Google Analytics analysis, Hotjar analysis, competitor analysis, surveys and user testing.

The advantage of an extended audit is that it contains more objective research methods than an agency employee’s assessment of the store alone. 

Which means certain hypotheses the agency formed during the purchase path analysis can also be confirmed by, say, the Hotjar analysis. 

And that means the probability of that hypothesis being right goes up. But still – without running an A/B test – that probability is unknown).

Extended UX audits have the following drawbacks:

• Preparing an extended audit can take as long as 2-3 months. You usually get all the recommendations at once, in one large document. Which means the wait for any recommendations at all is very long.
  
• Often a sizeable share of the recommendations isn’t matched to the specific technology and template your store runs on. For example: WooCommerce has plenty of limitations around building a checkout. Which means not every recommendation can be implemented on it.
  
• Audits are usually presented in text form only, with the occasional screenshot from the store being analyzed or from other stores. The people who commissioned the audit often miss having a visual representation of how a given recommendation should look in their store. What exactly it should look like and where it should sit.
  
• The UX agency running the audit focuses on delivering the client the most extensive file of analysis possible, rather than on delivering value as fast as possible. By delivering value fast I mean: identifying a problem and solving it in the shortest possible time. And that, after all, is what you care about most as someone working in e-commerce. You optimize the store to increase profits. Ideally as fast as possible, with strong confidence that the changes will actually work.
  
• And since we’re on the subject of confidence – the recommendations from an audit (even an extended one) are still a set of hypotheses that may not hold true in your e-commerce business. So to raise your confidence that a given recommendation will lift your sales, you should run A/B tests. Unfortunately, most UX audits have no planned strategy or prioritization for A/B testing.

Summary – the extended UX audit

Although it’s a more advanced way of improving a store, it still has plenty of downsides. Most of the problems above are answered by working on a retainer or billing by the hour. More on that in a moment.

I recommend an extended UX audit for:

• stores that want an outside pair of eyes on their purchase path, want very extensive research at the same time, and then intend to run the A/B tests themselves.

I don’t recommend it for: 

• stores whose main goal in working with an agency is growing sales
• stores that want to start solving problems and turning more visitors into buyers as fast as possible
• stores that want to launch an A/B testing program 1-2 weeks into the engagement rather than after 1-3 months

2. Retainer / billing by the hour (time & materials)

Working with a UX / CRO agency on a retainer solves most of the problems that come up in a one-off audit. 

By retainer I mean a number of working hours per month that you agree with the agency at a set rate. Plus any additional costs for analytics tools or A/B testing software. 

For example: you could agree that the agency will devote 70 specialist hours a month to optimizing your store, at a rate of 200 PLN (~$50) an hour. 

That number of hours can of course be adjusted as you go. 

Depending on the specific case, those hours are spent on:

• strategists, CRO / UX specialists, researchers, analysts (identifying problems, finding solutions, planning A/B tests, analyzing results, setting priorities and so on)
• designers, UI specialists (building wireframes, redesigns)
• developers (shipping A/B tests, implementing changes)

If you’re wondering how long you’ll need a retainer like that, good CRO agencies will offer you open-ended contracts, so you can terminate whenever you want. 

Which means you can continue working with a UX / CRO agency for exactly as long as it delivers value to your business. 

Be wary of agencies that want to tie you into long contracts.

At Digispot, many of our clients have worked with us for over 12 months. Which in practice means that even after a year, the additional revenue we bring to our clients’ online stores is higher than what they pay for the retainer.

But of course, we give them the option to walk away, even in the first week of the engagement.

Back to costs – depending on your traffic, you may also cover the cost of testing and analytics tools, 1,500 PLN (~$375) a month, say. 

If the numbers I’ve given seem large, I’ll say this: the amount you’ll earn or save thanks to those specialists and those tools will often be several times higher.

All right, you know how the billing works, so let’s get to how a retainer differs from an audit in practice.

What are the advantages of working on a retainer?

Above all, working on a retainer is geared toward delivering value to the client as fast as possible. Not toward building the most extensive audit file possible.

In practice that means that instead of spending a whole month writing an extensive report and only then running further research and A/B tests, the agency shortens the time from finding a problem to solving it.

A specific example will make this clearer. 

Imagine this situation:

You start working with a UX / CRO agency. In the first days of work the agency runs a Google Analytics analysis and finds a major problem that’s probably costing you money – a great many visitors are going back from checkout to the cart.

Now the agency looks for the answer to why that’s happening. It runs an expert review of the checkout. It turns out the checkout summary has no product images and no shipping time information. Both of those are in the cart. 

The agency also looks at Hotjar and analyzes recordings of users who were on the checkout page. It turns out a large share of them fill in the form, choose payment and delivery methods, and then desperately hunt for some piece of information – scrolling up and down the page, then going back to the cart. 

In all likelihood the absence of product images and shipping time information in the checkout is lowering conversion in your store, and you’re losing money because of it. 

The best thing now is for the agency to run an A/B test and verify whether that hypothesis holds. If it does, they ship the change to the store.

Now let’s look at how the agency’s work differs depending on whether you commissioned an audit or you’re working with them on a retainer.

If you bought an audit:

You might not learn about the checkout problem until 1-3 months after commissioning the audit (depending on how extensive it is and how fast the agency works). 

Which means you won’t be able to run the A/B test until after that time.

If you bought a retainer:

The agency will try to bring additional revenue into your business as fast as possible, rather than focusing on producing an extensive document. 

So if it spots a serious problem, it focuses on that from the very start – finds solutions for it, tests them and ships them.

And only then turns to less urgent research and optimization. (For instance: why run a thorough analysis of the blog when a great many customers are dropping out at the checkout)

Working on a retainer, you’ll learn about the most serious problems within the first two weeks of the engagement. 

A good UX / CRO agency will aim to A/B test solutions to the most serious ones as early as the second week of work.

You have to remember that store optimization is a process. Every store has something that can be improved. Every store has something that can be tested. 

The best example is Amazon, which has an extraordinarily high conversion rate and still runs research and tests constantly. Store optimization is a cycle.

The store optimization cycle:

I don’t want to describe the whole process in detail here, because that’s a topic for a separate book. I’ll briefly outline what the store optimization cycle looks like, so you can better understand how working on a retainer functions.

Step 1. Research and analysis.

To know what specifically to improve in your store, a UX / CRO agency has to run research and find out WHERE and WHY your sales are leaking. 

Research is the heart of the whole optimization process and every agency should give it serious weight.

The research agencies use:

• Google Analytics analysis
• expert review of the purchase path 
• form analysis, 
• heatmap and scrollmap analysis, 
• session recording analysis,
• chat message analysis,
• customer support interviews,
• user testing,
• competitor store analysis,
• customer interviews,
• tree testing,
• card sorting,
• customer surveys.

As you can see, there’s a genuinely large number of studies you could run. Each has its own character and is useful in different situations, depending on the problems identified in earlier research.

Imagine this situation:

During the expert review, the UX / CRO agency noticed that the store’s category menu may be confusing and unintuitive. The Hotjar analysis confirms that users get lost while using the menu. So the agency forms a hypothesis that rebuilding the menu will lift conversion. It would be good now to run further research to develop a new menu that’s comprehensible and intuitive. Card sorting and tree testing are excellent for researching information architecture. Those are the studies that should be run now.

And here another advantage of the retainer over the audit emerges:

A retainer is a more agile way of optimizing a store. 

It’s obvious, after all, that until you’ve identified the problem (during an expert review and Hotjar analysis, say), you won’t know that solving it will require further research (card sorting, tree testing).

Working on a retainer, you do NOT agree on the specific studies the agency will run before the engagement starts. 

The goal of an agency working on a retainer is matching the research appropriately. So that the most pressing problems get solved as fast as possible and additional revenue comes into your store. 

Before an engagement starts it’s very hard to determine which studies will be best for your store and how many hours you should devote to them.

The truth is that no agency, before starting work, will be able to determine how much time it should spend on Google Analytics analysis and how much on expert review for that to be effective. 

Of course, you can buy an audit that includes 30 hours of Google Analytics analysis. 

But what if it turns out the agency identifies the biggest problems in 10 hours? 

You’ll waste the other 20, because they could have gone to other research or even A/B tests – so that the biggest problems get solved first, and only then the other, less urgent things get analyzed in Google Analytics.

Things look  different if the most pressing problem is in the information architecture, and different again if most conversion is leaking at the checkout.

So only after running the first studies (expert review, GA analysis, a conversation with customer support) can we have the right data and conclusions to decide what next – whether we do a deep expert review of the checkout, dig harder into Hotjar, or perhaps run tree testing.

To sum up: the goal of a retainer, unlike an audit, isn’t running defined studies in a pre-agreed number of hours. The goal of a retainer is finding and solving conversion problems as fast as possible. 

Step 2. Growth opportunities, problems and priorities

Once the most important problems and growth opportunities have been found, it’s time to prioritize them and decide whether they need immediate implementation or A/B testing.

Organizing the research findings, problems, solutions and priorities well is enormously important for keeping everything clear. 

At Digispot we use our own reports in Notion, which our clients have access to from day one of the engagement. Notion holds:

• a report with the problems and solutions, 
• a list of tasks and studies, 
• the A/B testing plan, 
• the implementation plan 
• and much more. 

Our clients can see them 24/7, so they know at any moment what work we’re doing and what value we’re giving them.

Step 3. Shipping quick wins (immediate implementations)

Not every problem needs A/B testing. An obvious functional bug (a button that doesn’t work, say) should be fixed right away.

On top of that, if your store doesn’t have enough traffic and transactions, you won’t be able to test everything, because you won’t gather enough data.

Another case where immediate implementation may be essential is when the UX / CRO agency identifies a great many serious problems in your store but you don’t have enough store traffic to test it all. Then a more radical redesign may be advisable, but you have to be careful here. 

You can read more about radical vs evolutionary redesign here: An online store redesign – is it a good idea?

Step 4. Validation, A/B testing

A/B tests tell you what genuinely grows your sales and what doesn’t. A test verifies whether the hypotheses formed during research are sound.

From a test you can learn the probability that the changed version of the page sells better than the original.

You don’t have to guess whether changing the menu architecture is a good idea. You can measure it.

If your e-commerce business has over 2,000 transactions a month, one of the main activities of the agency you work with should be running A/B tests and drawing conclusions from them.

In stores with under 2,000 transactions a month, most A/B tests won’t gather enough data in a short time, so for stores like that it’s worth investing more heavily in user testing.

If any agency wants to ship universal “best practices” in your store without running research and tests, run as far from them as you can. 

You can read more about why there’s no such thing as best practices that work everywhere here.

Want to learn more about A/B testing? Read: A/B testing in e-commerce: what it is and what you can achieve with it

Step 5. Conclusions and implementation

Once the test finishes, the UX / CRO agency analyzes the results and decides (together with you) which of the tested changes to ship to the store and which not to.

Step 6. Repeat

First round of research, tests and implementations behind you? Don’t stop there. 

As I mentioned earlier, store optimization is a process. Every store has something that can be improved. 

So if the UX / CRO agency found and eliminated the most serious problems in the first weeks of work, it’s time to take on the next ones. 

You certainly still have plenty of room to lift conversion. Which means more research, tests and implementations lie ahead.

Summary – retainer / billing by the hour

Working on a retainer reflects the heart of store optimization far better – the endless cycle of research and analysis -> recommendations and prioritization -> shipping quick wins -> A/B testing -> analyzing test results and implementing.

The classic UX audit is built on a somewhat dated project management model (waterfall), where first we run as much research as possible and find the potential problems, and only then (once all the research is finished) do we get to solving them.

Store optimization on a retainer is built on a modern management system (agile). It’s a sprint cycle, where we can quickly find the biggest problems and start solving them right away, instead of hunting for further, less significant problems as happens in an audit.

I recommend a retainer for: 

• stores with over 2,000 transactions a month; 
• stores with fewer than 2,000 transactions but a high average order value (around 800 PLN / ~$200, say); 
• stores whose main goal in working with an agency is growing sales;
• stores that want to start solving problems and turning more visitors into buyers as fast as possible;
• stores that want to launch an A/B testing program 1-2 weeks into the engagement rather than after 1-3 months;
• stores that prefer working on the model: find the most serious conversion killers as fast as possible -> quickly test the changes that could bring the most sales in the shortest time, rather than the model: find every potential sales killer -> set a plan for what happens next.

I don’t recommend it for: 

• new stores with modest traffic and few transactions;
• e-commerce businesses whose only need is an outside pair of eyes on their store and a list of recommendations (in other words: they want steps 1 and 2 of the CRO cycle done once, rather than all the stages run cyclically),
• stores that want to run their A/B tests themselves.

3. Individual studies / consultations – UX / CRO agency

If you feel the need, you can also order just one specific study. User testing, customer interviews, A/B tests or a consultation, for instance. But I’d only recommend that in very narrow circumstances.

I recommend it for: 

• stores that have a precisely defined problem and are convinced they know how to solve it through a single study,
• stores that have an in-house store optimization team but want to consult on a specific area or run a specific study with an outside firm.

4. Performance-based billing – CRO

Let’s look at one more engagement model that, on common-sense grounds, might seem to make sense. 

Once you know how a CRO agency works, you might think: “if they’re that effective, let’s pay them on performance”. 

After all: “the more money they bring us, the more they earn themselves”. And by the same logic – if they produce no results, they get no payment.

Most often the system works by the agency earning a defined percentage from each winning test.

At first glance it looks good.

Nothing could be further from the truth.

1) Performance-based billing is unpredictable and often costs far more than a retainer.

Let me tell you the story of Invesp, a CRO agency in the US:

“When we started working on CRO in 2006, we experimented with a performance-based billing model.

We were at the beginning of our journey and we thought it would be a mutually beneficial arrangement. It made sense for the client because they could lower their upfront costs. It made sense for us because we earned from every winning test.

Everything was fine until we ran an A/B test where we achieved a 35% conversion lift at 99% probability. The client wasn’t convinced. 

So we ran the same test twice more to make sure the results were consistent. They were. 

Then the client asked us to run the test using different software. We gave in. And again, the experiment showed we had a clear winner. 

Under the contract, we were owed a payment of $45,000.

The client disappeared for several weeks. Then we got a long letter from the CEO trying to discredit A/B testing and statistical analysis generally. After long disputes we made two decisions:

• We walked away from that engagement.
• We will never again take on a project based on performance billing.”

The biggest challenge in performance-based engagements is agreeing on a clear methodology for determining CRO’s actual effect on revenue. 

Remember that an A/B test showing a 15% improvement in conversion rate does NOT mean you’ll generate 15% sales growth next month.

An A/B test showing a 15% improvement in conversion rate means that version B during the period the experiment ran sold 15% better than version A at a defined probability of error.

But that doesn’t have to mean your conversion rate will be 15% higher next month than the month before. It might be, it might not.

Too many factors affect conversion for it to be that simple. Among them: 

• seasonality (a grill store will inevitably sell better in summer than in winter); 
• competitor activity (imagine your direct competitor, selling an identical product, launches a 50% promotion. There’s no way your conversion doesn’t fall); 
• price;
• the economy (in a crisis or recession, most people won’t buy as readily as in good times); 
• the weather (when the sun finally comes out after a long winter, some people will go outside and won’t care about shopping online); 
• marketing and brand recognition (if your business is just starting out with no recognition at all, it won’t sell like crazy even if you have the best-optimized store in the world);
• every other aspect of your business;

Which of course doesn’t mean CRO is pointless. A 15% lift documented through a trustworthy A/B test is enormous growth for your business. It can be worth vast sums over the course of a year.

You simply can’t count on your conversion rate rising month over month with every winning test. It might, it might not.

I’d recommend an excellent post by Johny Longden on this: link to the LinkedIn post.

2) Performance-based billing is a trap for agencies.

Agencies that bill this way focus on running as many tests as possible at the lowest possible cost and chase “winning” tests without running solid research (which contradicts good CRO methodology — you’ll find a description of good methodology here). Which means they often test weak hypotheses.

Agencies like that may also drop quality assurance (QA) to lower the cost of shipping tests. And that carries a serious risk of errors on the site.

What’s more, an agency billing on performance has a very strong temptation to distort or manipulate the data to show more winning tests and funnel more revenue into its own pocket. 

I probably don’t need to say much more about the pathological situations that can create 😉

3) CRO isn’t only about hunting for winning variants, it’s also about saving money by not shipping changes that lower conversion. 

Hunting for a winning variant isn’t the only goal of testing at all.

The heart of testing is drawing conclusions, verifying hypotheses and learning what raises and what lowers sales in your store.

Variant B losing isn’t bad news at all. In fact, you could argue it’s good news, because it means you won’t ship a change that lowers your sales. 

If you hadn’t run the test, you’d probably have shipped the change straight away and  never known it caused a conversion drop.

So variant B losing is equally valuable information. Because it saves you money. 

So if A/B testing isn’t only about hunting for winning variants – how do you bill for it on performance?

Summary 

I hope I’ve given you enough knowledge about the models for working with UX / CRO agencies. 

Time for a short summary.

Choose a retainer if:

• you record over 2,000 transactions and/or over 100,000 PLN (~$25K) in revenue a month; 
• your main goal in working with an agency is growing sales;
• you want to start solving problems and turning more visitors into buyers as fast as possible;
• you want to launch an A/B testing program 1-2 weeks into the engagement rather than after 1-3 months;

Go for an audit if:

• you have modest traffic and a small investment budget; 
• your store design is very amateurish;
• you want an outside pair of eyes on your store’s purchase path, want very extensive research at the same time, and then intend to run the A/B tests yourself.

Choose individual studies (user testing, for instance) if:

• you have a precisely defined problem and you’re convinced you know how to solve it through a single study,
• you have an in-house store optimization team but want to consult on a specific area or run a specific study with an outside firm.

Don’t go into performance-based billing, because:

• it’s unpredictable and often costs far more than a retainer,
• it’s a trap for agencies, because they simplify their process and may manipulate the data,
• it contradicts the whole idea of experimentation, because it focuses on hunting for winning variants, and A/B testing isn’t only about that.

That’s all!

And if you’re looking for a CRO agency, get in touch.

If you want to optimize your store and start turning more visitors into buyers, book a call with us. 

At Digispot we grow online store sales every day without spending more on ads. 

One of the e-commerce businesses we’ve brought additional millions to is Ph. Doctor. Here’s the case study: 3,000,000 PLN (~$750K) in additional annual revenue for Ph.Doctor

Get in touch and let’s talk, with no obligation, about what you need to improve in your store and how we can do it.