Developing an Experimentation Culture

It is widely agreed by CRO experts that for optimisation to be successful it requires a cultural change within an organisation. But how do you develop and encourage a culture of experimentation. If you bottle those ingredients it would be worth millions of dollars.

To understand if my views of what it takes to create a culture of optimisation are consistent with what other experts in CRO thought I used Slack to conduct a small survey in Conversion World. Overall I think we all agreed that there is no single formulae or way to go about it. If life were that easy everyone would follow the same approach.

Developing a culture of experimentation is all about one thing: contextualisation. No ‘single’ approach exists. Tips, tactics and techniques can definitely help but ultimately, your approach needs to be contextualised around the people in the business, your stakeholders, their ego, the business legacy, the values of the business and it’s purpose. etc. etc. It’s bloody hard. – David Mannheim (Owner and Head of Optimisation, User Conversion)

However, we can learn from what works elsewhere and provided it fits with your business goals and values it is worth trying these ideas out. From my experience in CRO I strongly agree with David that “it’s bloody hard”, but the potential rewards are well worth the effort.

Agree a single unified vision:

Like any journey, if you want to get to your destination you need to know where it is. Having a clear, single unified vision helps because it avoids different teams creating conflicts because they don’t agree on the overall vision of the organisation. Amazon for example has a clear vision:

“Our vision is to be earth’s most customer-centric company; to build a place where people can come to find and discover anything they might want to buy online.” – Source: The

Set a common goal:

If individual business units are left to their own devices they will often create tactical goals and objectives that may conflict with what other teams are trying to achieve. By setting a common goal that everyone can agree on this allows people to understand how they can contribute to achieving the goal and it eliminates the problem of teams setting their own goals. It also makes it much easier to identify relevant success metrics for experiments.

For example at Amazon this goal has been set to ensure everyone has the same focus:

“It is the company’s goal to make it irresponsible to not be a Prime member.” – Jeff Bezos

Being customer focused:

Actions speak louder than words. Having customer centricity in your brand values doesn’t mean anything unless you act upon it and consider how decisions you make will affect the customer. In simple terms this means understanding what customers want and making sure you deliver it to them.

Sounds easy, but as we know this can be a challenge when you have so many competing priorities. However, you can’t exceed expectations unless you understand what customers want in the first place.

This means getting feedback from customers and potential customers, conducting usability testing, using browser recordings, getting mouse and scroll heat maps and using many other forms of user feedback. For CRO experimentation to be sustainable the customer needs to be at the heart of the programme.

Process is more important than analysis:

Having a set process for experimentation and innovation is essential. Indeed, Dan Lovallo and Olivier Sibony studied major business decisions over 5 years and found that “process mattered more than analysis—by a factor of six.” That’s not to say analysis isn’t important because you still need insights to help give your investigations some direction, but if your process of experimentation is not fit for purpose this will be even more damaging for your success.

Ensure you agree a systematic and transparent process for experimentation. Set some basic rules to guide your experiments. At Conversion Uplift we use an eight step process for optimisation which provides a consistent framework for generating insights and developing hypothesis for experiments.


6 Types of Tests for optimising a website page Final

Some of the guidelines you can also set include issues such as:

  • All tests must have a hypothesis based upon some insight rather than just gut feeling.
  • Tests must be evaluated based upon agreed criteria to ensure good use of resources
  • The number of experiments is not important in itself.
  • Don’t just test things that we think will work.
  • Test concepts as well as ideas
  • No experiments are a failure provided we learn from them.
  • Share test results widely to encourage buy-in and participation across the business.

Invest in analytics and insights:

Despite being in the age of big data and KPIs much of the data that is processed for online businesses is not used to understand what it tells us about customers and how they feel. We need to look beyond company performance and get better at discovering insights about what it’s like to be a customer.

This means investing in web analytics, data science, people and processes to generate real insights. Data on its own is meaningless and all too often even basic web analytics have not been properly implemented and fully integrated with other relevant systems.   Unless your organisation has confidence in the data and insights it generates you will struggle to get traction with experimentation because either the wrong insights will be made or people won’t trust the data enough to make decisions based upon it.


“For me this boils down to setting the right targets for people and making the part of the process of making these goals. And to make these goals you need experimentation and by having set and measuring the right metrics and making everyone internally familiar with these metrics really helps. And making all efforts and success and failures visible helps this hugely.” – Arnout Hellemans, Head of Marketing @ Sitly

Use small, cross-functional teams to deliver experiments:

Silos are the killer of many experimentation programmes. To allow collaboration across individual departments and business units it is important to establish small teams with a range of expertise to take ideas forward. Give these teams the power to make decisions, gather data directly and rapidly prototype and test ideas with customers.

This can be achieved by creating teams with experts in UX design, developers, marketing specialists, experimenters and product managers. This set up allows them to develop hypothesis, build tests and quickly implement successful concepts and ideas.

“Even in the most experiment-centric organizations, it’s easy for testing to enter product silos. Not only must there be a culture of running experiments, but also one of sharing, discussing, and building new hypothesis across teams”. – Chad Sanderson, Program Manager- Experimentation and Personalization at Subway

Check out the corporate model which uses small teams to take ideas forward and gives them the autonomy to publish, access data directly and make decisions without the normal interference line management. This encourages a strong culture of experimentation which now means all IT related projects are now delivered on time.

Test concepts as well as variations:

With any form of experimentation it is important to be bold and look to the long term. There will always be opportunities to test small ideas, but it is also essential to sometimes test concepts that may take years to reach their full potential. Amazon tested Amazon Prime without knowing whether it would work and it now achieves a conversion rate of around 74 per cent.

“Start by framing experiments as a method to encourage innovation. Most business leaders are horrendously opposed to major change; if a bad decision goes awry it could cost them their jobs. Experimentation helps prove the viability of new ideas without the need for complete buy-in or funding”. – Chad Sanderson, Program Manager- Experimentation and Personalization at Subway

Create two or more pipelines:

It’s great to have a strong pipeline of tactical tests to learn from. However, the danger is that the more strategic and complex tests will never get the time or resource because they don’t get prioritised sufficiently highly compared to more simple experiments. That’s why you should have a separate pipeline for more strategic tests and maybe a further one for concepts.

Make it into a competition:

Competition can provide a strong motivation for people to undertake a behaviour and it can also make it fun. Some organisations use competitions to promote a culture of experimentation by rewarding successful or innovate test ideas. You could develop ‘leader boards’ in your CRO team for those who generate the most test ideas or the number of successful experiments.

However, don’t just focus on the CRO team. Use competitions and suggestion schemes for the whole business to encourage participation from all areas of your organisation. Utilise your intranet, regular newsletters and lunch and learn sessions to promote idea generation and publicise the successful or more unusual test ideas.

Promote a culture of proving yourself wrong:

At Skyscanner they like to prove themselves wrong. People are generally poor at predicting the winning variant in experiments and so don’t be afraid of encouraging a culture that promotes being wrong. Publicise unusual tests and make it fun by asking people to vote for variants before a test begins. The danger with experiments is that we naturally suggest ideas that we think will win, when in fact it might be more powerful to suggest things that we don’t expect to succeed.

“If we’re talking culture then failure and success both need to be valued. Nothing kills experimentation and innovation more than massive fear that any non-success will be punished. Lip service to data driven and experimentation and user centric … but then bonusing or punishing people if revenue targets slip…It’s like transparency and “open door” and “speak truth to power” policies. Everyone gives them lip service, but it’s a damn sight harder and more painful to live those things than anyone ever plans, which results often in people saying one thing but doing the opposite”. – Tim Stewart – CRO Expert


The biggest gains and insights can often occur when we discover something unexpected. So by promoting a culture of proving yourself wrong you may generate more useful insights and get bigger uplifts than if you just stick to the ideas you think will win.

Aim for incremental improvements:

Experimentation is often about making small incremental improvements. However, as Guido Jansen – Psychologist, CRO Global Lead, points out our brains don’t work like that. People prefer to make large improvements rather than lots of little ones. But if every week we achieve a 0.5 per cent increase in conversion, over a whole year that would be a huge 30 per cent uplift.

As Manuel da Costa explains it’s important to understand people’s motivations and goals in your organisation. Psychology is critical for both customers and employees. If you don’t consider both you are going to struggle with your experimentation programme.


“To truly develop a culture of experimentation, you need to bring something to the table first – empathy and an understanding of the motivations and goals of the other people in the company that are not involved in optimisation. You need to see how their world is affected.

Secondly visibility of your work and accountability are key to your success.” Manuel da Costa (founder / Effective Experiments).

To create a culture of experimentation it is necessary for people to be comfortable with the process of testing, often with only small successes. Ensure you can measure these improvements and publicise how these small changes gradually add up to a major success for a culture of experimentation.


Remember that Hippo’s will want to kill your experimentation programme if they feel threatened or not involved in the process. Agreeing a transparent prioritisation process such as PIE will help you focus on more valuable tests, but it also makes it difficult for Hippo’s to undermine your approach.

Think about the Potential for each test. Look for pages that have high traffic but have never been properly evaluated and tested. These are likely to be ripe for experimentation and have the traffic to make it worthwhile.

Consider the Importance of a page or journey. How much money does your company throw at getting traffic to the page? PPC landing pages can cost a fortune to generate sufficient traffic to convert a reasonable number of customers and yet how well optimised are they? Such pages can be leaking thousands of pounds a month because of a poor user experience.

Finally, Ease of experimentation should be a factor. Sometimes you will have gaps in your schedule and so why not test the simple and straightforward ideas. How do you know that every component on your homepage is located in an optimal position or that they actually influence conversion at all? Sometimes it is good to test things just because you can and so that you can confirm they are positively influencing conversion. If you remove an element and nothing changes then you know it doesn’t actually need to be on your page. Remove shit and move on!


Notice nobody said that developing a culture of experimentation is easy, far from it. However, if you can agree a clear vision and set a unified goal you will at least give yourself a fighting chance. Being customer focused ensures the changes you make are more likely to be sustainable and setting a framework for experimentation will improve the effectiveness of your programme.

Analysis is still important and by investing in your analytics and data science you will establish trust in your insights that is essential for data-informed decision making. Creating small teams with autonomy to make decisions will help break down those silos that can disrupt your ability to innovate. Testing for the long-term will also allow you to evaluate concepts as well as variants in your experimentation programme. It’s important to have multiple streams to allow for both tactical and strategic tests.

Consider the psychology of your teams. Make experimentation fun, create some competition within and between teams to encourage the right kinds of behaviour. Communicate successes and failures and promote the idea of proving yourself wrong. Small improvements over time can result in huge uplifts and so make people comfortable with the nature of experimentation. Finally, make sure you have a transparent process for prioritisation so that everyone understands the criteria and how they can contribute to the testing programme. Make it as inclusive as possible so that everyone feels they can get involved if they want to.


736 thoughts on “Developing an Experimentation Culture