Your Hero Video A/B Test Is Answering the Wrong Question

Your A/B test between "hero video" and "hero image" is one of the lowest-information experiments you can run.

I don't mean stop running it. Video on a landing page reliably lifts conversion, and a low-information win beats the static default most B2B teams leave up for three years. But the question "does video beat an image?" is roughly as useful as asking "does copy beat no copy?" The answer is yes. You needed to know which copy.

Most landing page A/B tests are structured this way because they are easy to instrument. Two variants, one conversion metric, a traffic split, a winner. Clean. The problem is that "video won" tells you nothing about what to do next. Which video? How long? For what audience? At what stage of the buying cycle? Nobody knows, because you only had one video and the test didn't capture any of that.

The Test That Answers Nothing Useful

Here's what typically happens. A growth team wants to move the demo-page conversion rate. Someone suggests testing a video against the static screenshot. The video goes in. Traffic splits. After two or three weeks, the video variant has a higher form-fill rate and the test calls a winner. The team celebrates, ships the video, and reports the win in the next all-hands.

Six months later, the conversion rate is back to where it started. Not because the video stopped working. Because the video was the same for every visitor, and visitors are not the same.

The 50-person SaaS evaluating your product on an $800 MRR budget has a different objection than the 500-person enterprise running a six-month procurement cycle. The person who came from a paid brand search already knows what you do. The person who clicked through from a cold email doesn't know your category exists. Showing them the same 90-second product walkthrough and measuring the aggregate conversion rate is the organizational equivalent of A/B testing a shoe size. You learned something. You didn't learn anything useful.

The Variable That Actually Matters

The test worth running isn't video vs. image. It's variant A for segment A versus variant B for segment B, measured down to the deal.

This is a different test conceptually and a different build operationally. It requires knowing what segment a visitor belongs to before the page renders. It requires different clips. It requires the variant selection to be tied to the conversion event so you know which clip produced which form fill, which of those form fills became a qualified lead, and which of those leads eventually closed.

Most teams don't run this test because the instrumentation is harder than flipping a Webflow toggle. Wiring a variant ID to a HubSpot contact record is a Monday-morning project. Connecting that contact record to a deal stage without losing the video attribution event is a different project entirely. And producing three or four clips that speak to different segments, instead of one generic explainer, is where most teams stall out.

This is the wall I kept watching marketing teams hit. Not a lack of conviction about personalization. A lack of a system that makes the variant selection, the clip production, the attribution signal, and the CRM write all happen in the same breath.

Harloop's personalization engine exists because of this exact ceiling. The right clip per visitor, selected in real time based on company size, source, and stage. The attribution stack watches what happens after: did the visitor stay, did they fill the form, did the deal eventually close? When the demo page knows which variant a visitor saw and the CRM knows what that visitor became, the quarterly A/B test debrief turns into a table with real answers in it.

The teams that get ahead of this don't run fewer tests. They run tests that compound. Variant-to-segment data from this quarter trains the personalization logic for next quarter. The more it runs, the more precise the selection gets. That is a different kind of experiment than "video or image, pick one."

Run the binary test if you haven't. Ship the video. It will probably lift things. Just don't call it a personalization strategy. It's a starting condition.

Then figure out which video, for whom. That's the actual test.