How failed tests help you make better decisions | Craig Sullivan

Explicit content warning

10/04/22 • 46 min

It’s easier now than ever before for businesses to run CRO experiments, but what happens when great technology is misused? From not running enough tests to manipulating data to prove a favored hypothesis, there are many ways businesses use testing resources incorrectly — and screw up their growth in the process. So, how can we Unf*ck these issues?

Join me, Siobhan Solberg, and my co-host Russell McAthy as we chat with Optimal Visit’s Optimiser in Chief, Craig Sullivan, about all things testing. We get into why there is no such thing as a ‘failed’ test, the importance of optimizing your experiment programs, and how A/B testing can help you grow better tomatoes (literally).

In this episode:

Many more businesses are starting to run tests, but few run enough to drive real impact.
The more you test before setting updates live, the fewer mistakes you’ll make.
There’s no such thing as a failed test. Tests that don’t prove your hypothesis are as valuable as ‘successful’ tests — and can actually tell you more.
Tests should provide strong evidence that changing from what you’re doing now is the right idea.
Hunches and leaps of faith have their merit but need to be augmented by data, not driven by assumptions.
What should start-ups focus on if they don’t have adequate sample sizes or KPI outcomes?
How can businesses blend qualitative and quantitative data to generate better quality ideas?
Tests should never be run until a clear and measurable hypothesis that’s rooted in critical thinking has been written.
Manipulating data to support your hypothesis and personal biases leads to flawed business decisions.
The only way to truly understand your audience is to talk to them and run tests.
Accurately and authentically representing your audience drives better results and can change the way they engage with your business.
How to optimize experimentation programs in order to scale.
Big decisions can’t be made based on one A/B test that fails to take into account wider contexts.
Marketers should not ignore the importance of segmenting mobile and desktop users when running cross-device experiments.
Why governance and transparency are the largest things to be Unf*cked in testing.

Resources:

Craig on LinkedIn

Craig on Twitter

In this episode:

Many more businesses are starting to run tests, but few run enough to drive real impact.
The more you test before setting updates live, the fewer mistakes you’ll make.
There’s no such thing as a failed test. Tests that don’t prove your hypothesis are as valuable as ‘successful’ tests — and can actually tell you more.
Tests should provide strong evidence that changing from what you’re doing now is the right idea.
Hunches and leaps of faith have their merit but need to be augmented by data, not driven by assumptions.
What should start-ups focus on if they don’t have adequate sample sizes or KPI outcomes?
How can businesses blend qualitative and quantitative data to generate better quality ideas?
Tests should never be run until a clear and measurable hypothesis that’s rooted in critical thinking has been written.
Manipulating data to support your hypothesis and personal biases leads to flawed business decisions.
The only way to truly understand your audience is to talk to them and run tests.
Accurately and authentically representing your audience drives better results and can change the way they engage with your business.
How to optimize experimentation programs in order to scale.
Big decisions can’t be made based on one A/B test that fails to take into account wider contexts.
Marketers should not ignore the importance of segmenting mobile and desktop users when running cross-device experiments.
Why governance and transparency are the largest things to be Unf*cked in testing.

Resources:

Craig on LinkedIn

Craig on Twitter

Previous Episode

Why compliance should be your first thought, not an afterthought | Cory Underwood

Collecting customer data is something businesses take for granted. It’s an expectation: if they can’t gather data on their customers’ behavior online, how else are brands meant to optimize their user journeys? Yet as legal battles between Google Analytics and European data protection authorities show no signs of slowing down, global businesses risk losing access to data on a massive market. They need to reconsider their compliance strategies — and quickly. The shifting landscape of data security is impacting marketers, businesses, and consumers, but are the changes benefiting anyone?

In this episode of Marketing Un*fucked, Siobhan and Russell talk to Cory Underwood, Analytics Engineer at Search Discovery, about all things privacy when it comes to marketing, legal, and analytics. Listen in as Cory shares his expert views on why brands can no longer collect data on everything and deal with compliance later, how data privacy goes beyond analytics teams to wreak havoc for marketing teams, and the rocky future of international data transfers.

In this episode:

The legal issues Google Analytics is facing in Europe will get worse before they get better.
What is the impact on consumers when their analytics data is accessible by a foreign government?
The issue extends beyond Google Analytics to become an issue of American businesses versus EU laws and regulations.
Data security becomes alarmingly complex when laws and regulations differ from state to state and country to country.
Will hefty fines for breaching GDPR encourage businesses to act to avoid being caught off guard?
Legal councils, marketers, and IT teams must share knowledge to protect businesses and customers.
Is it possible to be 100% compliant when privacy requirements evolve rapidly?
A lack of clear and unbiased materials explaining the situation and its implications prevents marketers from understanding rulings.
Are marketing agencies aware enough of privacy regulations to make effective decisions?
The challenge of retooling systems with entirely new architectures when the rules are constantly changing.
Could the cost of maintaining compliance see small businesses priced out of the market?
Will we see an increase in the number of businesses building compliance teams?
Businesses will need to weigh up the value of their data versus the potential risk.
How differing legal language and definitions between states and countries cause confusion.
Marketers need to move on from talking about third-party cookies and email open rates.

Resources:

Cory’s Blog

Cory on LinkedIn

Cory on Twitter

Next Episode

Why data is not the solution to uncertainty | Rick Dronkers

What came first, the process or the insight? In an ideal world, a business’s processes would be informed by data, with teams seeking information to help them reach pre-established goals and objectives. But when it comes to data collection, can we really understand what we want to do with this information before we have it in front of us? And as clients become increasingly concerned about maintaining growth in a changing world, how do we convince them that collecting more data isn’t a business-saving silver bullet?

Join me, Siobhan Solberg, and my co-host Russell McAthy as we chat with Rick Dronkers, founder of Data to Value, about how to unf*ck marketing analytics. We get into how to get the most value out of our data, why people, not technology, are the answer to our problems, and the importance of learning to step back and embrace uncertainty. This episode is a fantastic look at the difficulties modern businesses face as technology evolves, even though we had a tech fail when we forgot to record the Zoom video (whoops).

In this episode:

The loop of turning data into actual value is often not completed as businesses don’t use what they collect to inform their actions.
Humans seek information to help control the uncontrollable, a desire that innovative technology firms selling insights-as-a-service appeal to.
Is there any truth to the idea that businesses should collect data on everything because it might be valuable in the future?
Why marketers should ask themselves what real-life action they’ll take when they collect specific data rather than letting the data guide their actions.
Businesses invest heavily in software because it’s perceived as a silver bullet solution, is an easier option than creating new processes, and is more controllable than people.
Do businesses rely too much on oversimplified, top-level data from dashboards to inform their decision-making?
To understand what data to collect, you must first understand your goals.
As economic outlooks change and growth starts to slow, CMOs should focus on prioritization and goal-setting to maintain upward momentum.
Data itself doesn’t improve conversion rates; instead, it provides insights that allow teams to make optimizations across processes that work together to generate growth.
Clients need to better differentiate between the metrics they should be monitoring and the results they view as success.
Why it’s impossible to hold a team accountable to an outcome metric and how to hold them responsible for practical output goals instead.
How do we support clients concerned about privacy, an issue which increases uncertainty?
Do marketers misunderstand the role of data analytics? Data is a tool to help businesses explore uncertain territories, optimize processes and answer questions.
What does the future of data collection and processing look like with the rise of machine learning?

Resources:
Rick’s Twitter

Rick’s LinkedIn

Life after GDPR