Good Tests, No Impact: The Missing Layer No One Is Talking About

It was nice to once again have a sunny backdrop in Leeds for an Experimentation North event. The first held since the clocks went back and one which undoubtedly faced stiff competition from beer gardens across the city. So it was heartening to see that even in the face of that competition, plenty of people came out to enjoy our latest event.

Manuel Da Costa, host of the CRO podcast Breaking The Echo Chamber and Director at business transformation company Efestra was coming at experimentation programmes from a slightly different angle to those we had seen before. And doing so in a genuinely interactive way. Each attendee were served with a bingo card which cited a lot of the common problems stakeholders face when conducting and scaling experimentation programmes. It was a list which unsurprisingly resonated with the audience.

The purpose was to highlight the faux reasons that you often hear within the industry when it comes to failing experimentation programmes. And at the same time provide a self-diagnosis as to where businesses lie in terms of failings around evidence integrity, synthesis and decision influence.

It was a talk which took a side swipe at vendors, leadership, agencies and even Ai for whom it is often so easy to focus on vanity metrics rather than the improved decisions made through garnering this more accurate layer of intelligence.

After using a sneaky trap to identify some of the more obvious answers to this problem, Manual highlighted how much of the traditional endeavours were based on 'Hope based Governance'. And however well-meaning this may be it is ultimately failing everyone's expectations. The 'missing' layer was the infrastructure that needs to be in place to drive and prescribe good governance. Moving away from a hope based model relying on an individuals capabilities and organisation and instead instilling processes to ensure experimentation turns into something which drives good decisions!

Our second speaker was Maks Brzozowski, a consultant with Eclipse Group Ltd and a consistently enthusiastic supporter of Experimentation North. He has worked with a variety of companys across a number of different industries, building and scaling experimentation programmes. And he was leveraging this knowledge to explain how different stages of maturity and development of such a programme require different metrics to measure their success.

He made the whole talk personal by introducing us to Martha. A first time buyer looking for the right house. In doing so how she assessed her needs and ultimately measured success in her house search. It was likened to an experimentation programme using the helpful buckets of outputs and outcomes to split the likely important metrics to measure velocity, quality, impact and cost.

The journey continued as Martha, now pregnant, looked up upscale and was once again looking for a home, this time with very different needs. It was a lovely analogy for a companys different requirements as their experimentation programme matures.

Perhaps the best advice came from the summary where Maks profoundly shared that an over-arching experimentation programme should be treated like an experiment. Be smart about the metrics used to measure it, with a primary, secondary and necessary guardrails.

As always we ended with a few thankyous and a shout out to upcoming events. With us heading quickly towards May then we called out Experimentation Elite, an important partner for us. Tickets are still available for their Space Academy event which talks about all things Ai and its use within CRO & experimentation. But importantly we'll also be supporting on the evening of the 6th May as we host a pub quiz as part of a MEGA meetup down in London.

There are also the Experimentation Elite Awards which are now live! It gives companies, agencies and practitioners an opportunity to be rewarded for the good work they are doing. It would be superb to see northern representation at these awards, we know there are great candidates, we hear about them and see them at our events. Get your entry in here now!

Check out our pictures from the event: