How does the cookie window affect measuring creative performance?

It is hard to measure creative performance or compare the creative performance if all the creatives were not live at the same time (start to end). The problem which comes is the cookie window. The creatives which are live for a longer period of time have the advantage of a longer cookie window and the creatives which are live for a shorter period of time have the big disadvantage. This is a bigger issue if one of the creative is live only during towards the end of the campaign.

Looking at the example below, the creative C, was live during the last 4 days of the campaign and thus does not have the cookie window advantage which creative A and B do.

Thus, it is recommended to think through the methodology based on the cookie window before starting the analysis. One solution is that measure the results 30 days beyond the campaign end date, then all the creatives will have similar cookie windows. However, this solution is not applicable in all situations specially if there is a big seasonality issue of the product example: Christmas candy canes, holiday decorations, customized products for a special occasion like Mother 's Day etc.

Creative Optimizer

If we think about large advertisers - advertising over a billion impressions per month, optimization would be very hard or next to impossible. Possibly, there would be over 1,000 live placements and if there is more than 1 creative...GOOD LUCK with that.

But thanks to DART's DoubleClick Creative Optimizer you can set the the metric you want to optimize on - Click-through rate, conversion rate, revenue etc. What the optimizer does is that it dynamically moves the impressions to the creative which is delivering higher click-through rate, conversion rate or revenue...and the great thing about it is that it works at placement level. So, if on one particular placement creative A is performing better - it will get more impressions and if on another placement creative B is performing better - it will get higher impressions. One more good thing about this is that you can set a minimal threshold, example serve a minimum of 10% of total impression for each creative.

However, there are some drawbacks with every good system, one of the biggest that I have noticed is that you cannot start with a set creative split. It always starts with an even split. So, if you have 2 creatives it starts with a 50/50 split.

But hey, optimizing at creative and placement level and that too AUTOMATED, I will take it any day.

A/B Testing - Display Ad/Site/Newsletter...

Most of the analytics peeps keep talking in their jargon, one of the favorite is A/B Testing. The marketing folks most times start wondering what are they talking about.

A/B testing is essentially testing creatives - for the campaign. This could be applied to almost any form of advertising - Display, Site, Newsletters. The basic principle is keep an even split among all the creatives and try to measure the impact on the action - click through rate or conversion event. Some people think it is for only 2 creatives but one can use it for multiple 2,3, 4,5....

One important thing is that the sample size - it should be big enough to provide statistically significant results (t-test).

The first time I did a A/B test was a HomePage design for one of my clients. We were trying to measure which one of the two HomePage designs drove higher consumers to better use the entire website. Typically, on a Homepage you want information for existing customers, knowledge center and an acquisition section (to attract new customers). The banking websites do a pretty good job of this.

Since, then I have tested multiple forms of creative (display), emails (different forms of emails newsletters and then measuring the call to action rate), websites.