I want to really unpack something that came up from a conversation with a client just yesterday. I was talking to them about an A/B split test we did on a campaign.
This is something, if you’re not familiar with A/B split testing, an A/B split test is effectively running maybe some traffic to a particular landing page that maybe will serve a different landing page for 50% of the traffic and another landing page for 50%, and we’ll see what the conversion’s like. And you can do A/B split tests on things like email subject lines and all kinds of different things. If you’re listening to this podcast and you’re a CEO, founder or fractional CMO, you should really know what an A/B split test is.
But fundamentally, I have talked to a lot of clients over the years, they’ve wanted to do this. This particular client, very good size email list, we took a tranche of 15,000 people from the mailing list and we put them through this A/B split test.
And there was a gut feel from the owner and founder of the business that one side of this split test would win. And under my kind of breath and behind the scenes, I disagreed with him completely and I thought the other one would run. But I didn’t really voice it, because what I want to do is I want the data to actually tell us.
We ran this A/B split test for 14 days for these 15,000 people, and we saw double the conversion on my side, let’s call it A, than his gut feel, which was B. And we were able to prove with real data that this worked.
This is a really important thing. As a fractional CMO, you are there to bring marketing leadership. Founders and CEOs are amazing people. They’re entrepreneurs, they have ideas. They just go get it and it’s just like, let’s go, let’s go, let’s go, let’s go. But sometimes that can exhaust a marketing team. That can exhaust a marketing director. That can really fatigue the activity within the business, because what sometimes happens is there’s an idea on a Friday afternoon that needs to be implemented before close of day, ready for Monday morning, because they just had an idea.
And whilst we want to honour, respect, and choose to, you know, be led by the CEO and founder of the organisation that we’re working with, a fractional CMO’s role is to kind of pull the reins on some of these things sometimes and just pull back.
The idea around this A/B split test came from me basically saying, well, how do we know that that idea is going to work? His gut feel for this promotion that could have gone out to hundreds of thousands of people? We had this sort of, he had this gut feel that this promotion would work in this particular way, and I said, well, let’s just, can we just put a pin in that for 14 days? Let’s just run a test with a segment of non-buyers, and let’s send that segment of non-buyers the same message, but send them to a page that then has a split test around that particular offer. And we’re going to test your gut feel, but I also feel like we need to test it against something else that it was almost like the direct opposite.
And like I said, the conversion was amazing. The conversion on A, from those 15,000 was around 4%. And the conversion on B, which was his gut feel, was around 1.2%. And that for me, it was enough data to be able to say to the rest of the customer list, we need to now plan a campaign around A, because it’s proved that it works that way.
If you do an A/B split test like this, you need good numbers. If we’d done it maybe on a sample of around a 1,000, we probably would not have got the statistical data enough to be able to make the intelligent call for this. And what I mean by that is sometimes data can be skewed if you don’t have enough of a data set to be able to actually help people deduce what’s going on. You can have statistical anomalies within a data set. So if the data set is too small, then you need to think about getting that data set a little bit bigger.
Honestly, an A/B split test, for instance, on email marketing for email headlines, throw that A/B split test at 10,000 records before you make a decision. A campaign and an offer like this, again, 10,000 is a really good sweet spot for people to be able to come in, see, and you’re not going to get as many statistical anomalies in this.
This is just from my own experience over years of doing this, with different clients in different niches, different industries, with different messages, very different products and services as well, but it always comes back to the same thing. We need to test over a period of time, rather than just an instant hit and just see what happens. We need to communicate and test in a good way.
Plan that A/B split test as a campaign, not just as a simple promotion or a simple communication piece. Plan it, and run with it. Do that, you’ll find an offer that will, or a way in which your customers will respond, that probably goes against your gut, if you’re a CEO and founder, and you come up with these feel ideas. Trust the data and just let the CMO pull the reins a little bit, test it, and run with it. You’ll thank them afterwards.
Because if we can get more than almost a quadruple conversion rate on the split test that we ran, it shows that actually sometimes it pays to just be a little bit more patient with it and plan it properly.
So, A/B split tests, huge fan. Use a good statistical, you know, section of your audience, your list, whatever it might be. Don’t just do it on small volumes, do it on a good chunk of your volumes. Maybe run it twice as well. That’s the second, that’s the next thing that we’re going to do. We’re going to run it on another 15,000 just to see if we get a very similar kind of result.
That’s me. I’m Ant Hodges. I’m a fractional CMO. I work with clients who are doing over $1 million, scaling up to $50 million, and I’m helping to bring in marketing leadership to remove the CEO and founder from those Monday morning marketing meetings and all of the decisions around marketing, so they can focus on growing their business in the way that they want to grow it.
If you’d like to talk to me, then head over to www.anthodges.com or email me directly, cmo@anthodges.com. Let’s have a conversation. Let’s see if we can bring marketing leadership into your business, get you out of it, and see you start to scale.
