Calculating View-Through attribution: Challenges

After analyzing the Google Analytics "Content Report" for my blog, it became evident that the "Post-Impression" attribution post is the most popular post so far. So, I thought I would expand a bit more on that same topic, some of the common challenges that could be faced during the test set up and also interpreting the results.

If this test is being done for a very large advertiser, delivering in the range of billions of impressions in a month, then it is virtually impossible to create a test and control group as the reach of the campaign would be so high. It is highly likely that all consumers will be exposed to the advertising.

1. If all the ads are not third party served, it is impossible to create a test and control, as if there is a x% of Ads which are site served, creating a control group will be hard across the entire advertising network
   
2. Even with all the ads being served via third party ad servers, consumers could get exposed to the Ads on different computers, example work vs. home
   
3. After the results if the lift is negative i.e. it shows that showing the ads did not increase the conversion rate vs. control group – this indicates that the "Brand" has very high brand awareness
   
4. Interpreting the results could be another tough nut to crack; DoubleClick in one of their studies has shown that for Continental Airlines that there is a 67% post-impression attribution. Lately, it has been seen that the post-impression attribution has been pretty low due to increase in online advertising.
   
   1. One more variable could be seasonality, the post-impression attribution has been seen to change based on the different times of the year. The consumer mindset changes based on the seasonality for example it could be very high during the back to school season for school products like laptops, ipods etc

Heavy Clickers account for 50% of the clicks

A recent study conducted by Starcom, Tacoda and ComScore suggests that heavy clickers account for 50% of the clicks. Please follow the link below for details.

http://www.smvgroup.com/news_popup_flash.asp?pr=1643

Sample Size Estimation

Continuing about the testing section (A/B, Multivariate) “Sample Size” is very crucial in the analysis. From a statistics point of view, the aim is to demonstrate with 95% certainty that the true value of a parameter is within a distance an error range of the estimate, typically the value of error range is referred to as the 95% confidence interval. OR in simple English, the aim is to demonstrate with 95% certainty that the result (example: Conversion rate, Click rate etc) would hold true 95% of the time.
It would be good to explain with a simple example. If there is a test for a click-through rate on a banner, there should have a big enough sample size to estimate if x% CTR would be true 95% of the time with 95% certainty. One more example is if there is an A/B test for a homepage test measuring clicks on a call to action button it is important to have enough samples or views of the page to be able to estimate their conversion or click rate with 95% certainty.
As a rule of thumb, the lower the response rate, the higher the sample size needed for accurate estimation.
For someone who is interested in know the math behind the test, please follow the links below:
http://en.wikipedia.org/wiki/Sample_size
http://www.surveysystem.com/sscalc.htm
http://www.isixsigma.com/library/content/c000709a.asp

Collecting Consumer Data – Email, Zipcode, Area Code: Geo Targeting

Collecting data through various sources like Website, Call center or even Online Banners can provide a lot of information about your consumers. Some of the examples are:

Email Address: If a marketing group is able to get a list of clean email addresses, then a relationship marketing program can be initiated which would help create a long term relation with the consumers. Sending out a series of emails based on consumers needs, can help build a loyal group of consumers

Zipcode: Knowing the Zipcodes of a loyal group of consumers can help understand which Geographical areas have higher quality of leads, and then the marketing can be Geo targeted. With the current ability of Ad servers like DoubleClick and Atlas it is possible to geo target online display advertising.

Similar to Zipcode, Area code can provide a density map of loyal consumers and target the consumers to help increase the loyalty and sales of products.

Having more and more information or data about your consumers helps better understand them and thus get your marketing dollars to work.

In the online world, it is much easier to collect this data through various ways – Online banners, Website, Microsite.

Multivariate Analysis

Multivariate Analysis is used to test multiple options at once. These multiple things could be 5 different images on a webpage, 3 different background colors of the homepage of your site or 3 different body copies for a particular offer.

Offermatica is a vendor which was acquired by Omniture which is known to help most of the clients and agencies to do multivariate analysis. One of the examples of their website does a very good job of explaining how and why this kind of Analysis is very crucial in taking the correct decision while making any changes on a website.

http://www.omniture.com/en/product_tours/offermatica/mvt

Offermatica also has a demo which explains the A/B testing for various elements on the webpage

http://www.omniture.com/en/product_tours/offermatica/ab

How does the cookie window affect measuring creative performance?

It is hard to measure creative performance or compare the creative performance if all the creatives were not live at the same time (start to end). The problem which comes is the cookie window. The creatives which are live for a longer period of time have the advantage of a longer cookie window and the creatives which are live for a shorter period of time have the big disadvantage. This is a bigger issue if one of the creative is live only during towards the end of the campaign.

Looking at the example below, the creative C, was live during the last 4 days of the campaign and thus does not have the cookie window advantage which creative A and B do.

Thus, it is recommended to think through the methodology based on the cookie window before starting the analysis. One solution is that measure the results 30 days beyond the campaign end date, then all the creatives will have similar cookie windows. However, this solution is not applicable in all situations specially if there is a big seasonality issue of the product example: Christmas candy canes, holiday decorations, customized products for a special occasion like Mother 's Day etc.

Samples, how can we have a similar concept online?

I am sure everyone has had a free sample ice cream or candy bar. I just had two M&M ice cream bars while walking up and down on Michigan Ave. Anyone who has walked on Michigan Ave today between 12-2 has to have had it. They had trucks sitting on the road and they were giving them away. Even the trash cans on Michigan Ave were showing that – see picture below (I wanted to take a picture of the trucks but missed it). How measureable are these samples? Yes, we can model out how many ice cream bars were given out and use an average conversion rate to guesstimate the impact of the samples.

While walking back, I thought how we can have a similar concept online. Yes, we cannot give away free ice cream online but there are a lot of different products which can be sampled online. For example: When Sony or Microsoft release new video game for Playstation or Xbox, demo versions of the games should be available to download online for the gaming enthusiasts to play. After the demo, there should be a promotional offer to buy the full version of the game. This would be fully measureable – from number of demo games played to how many gamers bought the full version of the game. One more thing could be free versions of browser or software like picture organizer.

Even looking at upward in the funnel, there can be online banners and the click-through rate would help understand the initial engagement with the game. Nowadays even in-banner gaming is a possibility with the fancy rich-media banners.