I’m not asking what AB testing is, I am asking how the actual statistics work. I’m going to use MrBeast as my example.
MrBeast switches up his thumbnails sometimes up to 5 times a video, most people being under the impression that it’s for AB testing. I’m aware that AB testing is meant to figure out which thumbnail gets clicked on the most. What I don’t understand is how this data is even verifiable or accurate in any way considering the amount of variables:
* The 10mil people (example number) who saw the first thumbnail are completely different from the set of 10mil who would see the second thumbnail.
* Those first 10mil people who already watched the video are very likely not going to watch the video again after the changed thumbnail.\*
* The YouTube algorithm more than likely has a large hand in who gets recommended certain content.
My main confusion is from the first two points. How are you going to determine what thumbnail works better if it’s not the same set of people choosing between two thumbnails? In my mind, the way I see a definitive answer of a choice between two things is to have one group of people choose between those things. But if you have two different groups (or more), and the first group already watched the video so they’re not going to click the second, I can’t wrap my head around how this is even useful information.
In terms of YouTube videos specifically, I imagine most of the views come pretty much within maybe 12 hours after posting, at least with MrBeast. After that, I imagine the growth of views starts to stagnate at a certain point or even slow down. This is another major reason why I don’t understand how AB testing works, because the data from the 4th thumbnail change is probably in their “slow down” phase of video views, so again, not understanding how it’s accurate or useful info.
\*To further my second point, let’s say MrBeast’s team expects that he will get 500mil views on his video. With the first thumbnail, 100mil people watched it. That means 100mil people are now removed from that pool of potential people clicking on the second thumbnail, because it is under the assumption they don’t plan to rewatch the video. So the statistics are now dwindled to 400mil who can click on a thumbnail vs 500mil.
In: Other
I’ve only used the system once or twice but I assume that both thumbnails get served together.
When you upload a video you add multiple thumbnails and the algorithm then shows different ones to different viewers. How it decides this I don’t know, it may be random or it may be based on markets or demographic groups etc
As the video owner I get a running report of how each thumbnail is performing over the initial period (I can’t remember how long the tests run for). At any point I can end the test and set the thumbnail if one is performing massively better than the other.
If you do it manually then I guess you just get an idea about what works. Maybe thumbs with your face work better or ones with a big red arrow or whatever.
Many people will scroll through their feed and miss one thumb but perhaps in a few days will scroll through again and have their eye caught by a different one.
When you get the kind of numbers MrBeast does there are all kinds of data you might pull out and even tiny tweaks will reap huge rewards.
Latest Answers