Lets Put That To The Panel

Despite balmy temperatures in Leeds, admittedly something I thought was unlikely competition to a northern based event, we once saw good numbers gather at the Flutter offices in Wellington Place.

For the first time we were mixing up the format, and alongside a speaker from our generous sponsors StatSig we were hosting our very first panel discussion featuring some of the finest minds in the industry.

If ever there was validation that we were on the right track with this event then it was the lengths our speaker Craig Sexauer had gone to be a part of it. Flying in from Seattle only the previous day, he had enjoyed the afternoon seeing the sights and sounds of Leeds.

His talk was on Building an Experimentation Culture and guided the audience through the varying stages of maturity a company goes through as they embark on an experimentation programme. Using some first-hand experience from working at StatSig, superb visuals from the tool to echo the point and even a small smattering of Data Science thrown in.

A panel of industry experts discuss Capturing More than just Metrics in Experimentation.

Following some questions from the audience we then kicked off our very first panel discussion as four chairs had been subtly added to the stage. Joining Craig was Jonny Longden (Chief Growth Officer, Speero) and Krista Rickard (Lead Digital Product Manager, Flutter UKI) with our moderator Sandeep Shah (Head of Product, Webtrends Optimize).

The discussion was based around the idea of Capturing MORE than just Metrics in Experimentation and considering things like brand reputation, customer safety and user experience.

Jonny brought in his experience from Boohoo to make a great point around how easy metrics can be to shift (for example through dramatic price reductions) but will inevitably have a knock on-effect on brand perception which is not necessarily evident through experiment metrics.

Leaning not only on her Flutter experience but also considerable experience at Capital One there were similar points made around customer safety and how product optimisations for immediate short term gain could have a longer term impact as customers miss safety tools.

It sparked an interesting discussion around short term vs long term and the importance of having a broad view of potential impact of changes.

This point was emphasized with a couple of great examples around experiments aimed at reducing call volumes and how that can potentially impact brand advocacy. Conversely a new messaging service and how it had wonderful conversion but cannibalised self-serve ultimately creating a more expensive operating model. Then another great point about understanding the trade-off between transactional benefits of a new feature factoring in the tech debt it can create.

A first pass at a panel discussion, the feedback has been good and gives us good confidence its something we should explore again. Please get in touch if there are any topics you'd like us to cover.

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

Want to see the full thing...we don't do this every time, but a HUGE shout out to @StatSig who recorded the event. Enjoy!

https://video.wixstatic.com/video/b047f9_6cbd32534d6040b7b9e0e69a7289d166/1080p/mp4/file.mp4

How To Develop a Culture of Experimentation - Craig Sexauer, StatSig

https://video.wixstatic.com/video/b047f9_ccb43ed51fcb4ff99bcaba1afd888f54/1080p/mp4/file.mp4

Our very first panel discussion - 'Exploring More Than Just Metrics in Experimentation'