CUIL is live

A new search engine was launched today, CUIL.
I personally have not used it a lot but I thought it would make sense to post a few links that provide a lot of insights about the new Search Engine.

http://searchengineland.com/080728-000100.php
http://searchengineland.com/080728-024035.php

Let's see if it really hits it big like Google.

Challenges in Online Advertising Measurement

Online Advertising is the only form of advertising that is measurable but there are a few challenges or drawbacks of online advertising that we need to understand. Some of the most common ones are listed below:

· Cookie Window: The cookie window needs to be set based on each business objective. Some businesses might have a longer purchase or research cycle which would mean consumers take more time to convert after they see the ad, example laptops computers. Whereas, there would be some business where the consumers would convert quicker. Thus, we cannot have a common cookie window for all the businesses.
· Cookie, Work vs. Home: Many consumers would see and click on the ad at home but might complete the transaction at home or vice versa. So, the cookie would not be able to close the loop and the advertiser would miss out on counting the revenue or the conversion based on the online advertising.
· Last Click: Typically, the ad servers reporting systems attribute the conversion to the last click and miss out on the multi-touch aspect. For example if a consumer clicks on a banner and then clicks on a Paid Search and then converts, the Paid Search gets 100% credit but the display ad also had some contribution in driving the consumer to the website and increase the consumer’s interest in the product.
· Cookie Deletion: It is knows that consumers tend to delete cookies from their computers. So, if the cookie is deleted, the ad server will not be able to connect the conversion to the ad unit that the consumer was exposed to or had clicked on. There is also an issue of 3rd party cookie deletion vs. 1st party cookie deletion. The 3rd party cookie deletion is more prevalent than the 1st party cookie deletion.
· Tags not firing: Several times the tags stop firing when expected example at the conversion event confirmation page – this could be due to many unexplained reasons and the sometimes all it needs is a tag refresh. It has also been seen that sometimes the IT personnel by mistake change the tag format and it does not work. One way to QC this problem is by using tools like HTTPlook.

Media Mix Modeling: Static vs. Dynamic

Media Mix modeling involving a lot of data collection and then a lot more number crunching and then comes the actionable insights based on the data.
Let's step back for a minute and think through this in some more detail. The data used to build these models goes as far back as 5 years (if you are using quarterly sales data) or at the very least 1 year (if you are lucky enough to get weekly sales data). So, let's assume on average 2-3 years of data is needed to build a media mix model.

Now, if you were to make decisions based on data which is 2-3 years old, are those assumptions still valid? The data that you are using is frozen in time 2 years ago. Just think through how much has the marketplace evolved since then? There are probably 2 new competitors in the arena and 1 of the big competitors has completely changed their marketing strategy. So, how actionable are the insights which would be derived based on this data? Probably not a whole not.

Thus, the amount of time and effort spend in create media mix models in probably not worth it.

There are some other options of using studies with Dynamic Logic or ComScore which might be able to provide similar insights and represent close to real time marketplace arena.

What attracts consumers: Free Samples, Coupons or Expensive, cool, sexy products

Like a few weeks ago, I was again walking up and down on Michigan Ave (in Chicago) and as a co-incidence M&M was again handing out their new ice cream bars. There were people all over with their ice cream. It was 90 degrees outside and I am sure consumers were enjoying it to the fullest. They did have the truck full of bars and people were consuming them fast.

While, walking back I passed the Apple store and I saw this long line – longer than the one for the M&M ice cream bars, I thought, "Wow! is Apple giving away something for free?" I asked someone in the line – "What is this line for" and she said "For the iPhone". I thought, on one end there are people in line to get free ice cream and on the other end people are standing in line to get the iPhone which is $199 and then a monthly service of $79. What a contrast!

Then a few more blocks near the Tribune building I see these two guys with the golden arches -"McDonalds" giving away coupons (Buy one get one free) for the Iced Coffee and there was no even a single person taking coupons from them.

It is just so amazing to see three such strong brands and the differences in how consumers perceive about them.

If I were to measure these three I think Apple is the clear winner as they were the only one making revenue and are locking down consumers with a 2 yr contract with AT&T, so not only is there a onetime cost but also a monthly cost and now with the new applications they are even charging for them as well.

Oh! and then there was this guy who had the M&M ice cream and the Coffee coupon, not sure if he bought the $199 iPhone.

Multivariate Analysis - Part II

Offermatica seems to be the industry leader in landing page multivariate analysis but I just found out about another tool called "Memetrics". This is a tool owned by Accenture and uses their Choice Modeling methodology.

They claim that Taguchi, which is a linear alogorithm is not the best way to predict human behaviour. Choice Modeling takes into account trade offs and preferences. Taguchi in a linear design and is a fractional factorial science which does not understand content interactions. While using Taguchi, enough sample for 16 versions of the page are required. Memetrics and choice modeling requires only enough sample for the number of attributes that will be tested and can then run a full factorial test in less time with less sample at a higher degree of confidence than Taguchi or Optimal Design (by Optimost) can.

Memetrics is the only tool which can easily optimize multiple outcomes, weighted outcomes and even offline outcomes. Memetrics is the only solution that is not a black box but rather a fully exposed, open analytical framework.

I personaly have not used Memetrics but it seems like a new technology that can be used for Multivariate testing.

Mobile Analytics: iPhone

I just installed the latest iPhone update yesterday and I am so excited with the new application. There is a lot to measure about this new application, a few metrics that I can think of are:

1. How many iPhone users started the install and how many actually completed it (the update takes about 1.5 to 2 hours, so it would be great to measure the completion rate)
2. Drilling down the consumer flow, the next thing to track would be how many users installed the applications
a. Frequency (1-3, 2-6, 7-10, 10+ applications)
b. Paid applications
c. Free applications
d. Applications by category
3. There are some “Westin” ads specially within the NY Times application, so measuring the Click-through rate on these ads would be a good metric to track

This new application has just changed the use of this product and obviously the sales of this product should go up dramatically as there were a lot of applications missing like Games, Mobile Banking (Bank of America), Newspapers (NYT), Social Media (Twitter, Facebook, Myspace)

The engagement with each of the above application would also be a good measure of how consumers are interacting with these applications.

Calculating View-Through attribution: Challenges

After analyzing the Google Analytics "Content Report" for my blog, it became evident that the "Post-Impression" attribution post is the most popular post so far. So, I thought I would expand a bit more on that same topic, some of the common challenges that could be faced during the test set up and also interpreting the results.

If this test is being done for a very large advertiser, delivering in the range of billions of impressions in a month, then it is virtually impossible to create a test and control group as the reach of the campaign would be so high. It is highly likely that all consumers will be exposed to the advertising.

1. If all the ads are not third party served, it is impossible to create a test and control, as if there is a x% of Ads which are site served, creating a control group will be hard across the entire advertising network
   
2. Even with all the ads being served via third party ad servers, consumers could get exposed to the Ads on different computers, example work vs. home
   
3. After the results if the lift is negative i.e. it shows that showing the ads did not increase the conversion rate vs. control group – this indicates that the "Brand" has very high brand awareness
   
4. Interpreting the results could be another tough nut to crack; DoubleClick in one of their studies has shown that for Continental Airlines that there is a 67% post-impression attribution. Lately, it has been seen that the post-impression attribution has been pretty low due to increase in online advertising.
   
   1. One more variable could be seasonality, the post-impression attribution has been seen to change based on the different times of the year. The consumer mindset changes based on the seasonality for example it could be very high during the back to school season for school products like laptops, ipods etc

Heavy Clickers account for 50% of the clicks

A recent study conducted by Starcom, Tacoda and ComScore suggests that heavy clickers account for 50% of the clicks. Please follow the link below for details.

http://www.smvgroup.com/news_popup_flash.asp?pr=1643

Sample Size Estimation

Continuing about the testing section (A/B, Multivariate) “Sample Size” is very crucial in the analysis. From a statistics point of view, the aim is to demonstrate with 95% certainty that the true value of a parameter is within a distance an error range of the estimate, typically the value of error range is referred to as the 95% confidence interval. OR in simple English, the aim is to demonstrate with 95% certainty that the result (example: Conversion rate, Click rate etc) would hold true 95% of the time.
It would be good to explain with a simple example. If there is a test for a click-through rate on a banner, there should have a big enough sample size to estimate if x% CTR would be true 95% of the time with 95% certainty. One more example is if there is an A/B test for a homepage test measuring clicks on a call to action button it is important to have enough samples or views of the page to be able to estimate their conversion or click rate with 95% certainty.
As a rule of thumb, the lower the response rate, the higher the sample size needed for accurate estimation.
For someone who is interested in know the math behind the test, please follow the links below:
http://en.wikipedia.org/wiki/Sample_size
http://www.surveysystem.com/sscalc.htm
http://www.isixsigma.com/library/content/c000709a.asp

Collecting Consumer Data – Email, Zipcode, Area Code: Geo Targeting

Collecting data through various sources like Website, Call center or even Online Banners can provide a lot of information about your consumers. Some of the examples are:

Email Address: If a marketing group is able to get a list of clean email addresses, then a relationship marketing program can be initiated which would help create a long term relation with the consumers. Sending out a series of emails based on consumers needs, can help build a loyal group of consumers

Zipcode: Knowing the Zipcodes of a loyal group of consumers can help understand which Geographical areas have higher quality of leads, and then the marketing can be Geo targeted. With the current ability of Ad servers like DoubleClick and Atlas it is possible to geo target online display advertising.

Similar to Zipcode, Area code can provide a density map of loyal consumers and target the consumers to help increase the loyalty and sales of products.

Having more and more information or data about your consumers helps better understand them and thus get your marketing dollars to work.

In the online world, it is much easier to collect this data through various ways – Online banners, Website, Microsite.

Multivariate Analysis

Multivariate Analysis is used to test multiple options at once. These multiple things could be 5 different images on a webpage, 3 different background colors of the homepage of your site or 3 different body copies for a particular offer.

Offermatica is a vendor which was acquired by Omniture which is known to help most of the clients and agencies to do multivariate analysis. One of the examples of their website does a very good job of explaining how and why this kind of Analysis is very crucial in taking the correct decision while making any changes on a website.

http://www.omniture.com/en/product_tours/offermatica/mvt

Offermatica also has a demo which explains the A/B testing for various elements on the webpage

http://www.omniture.com/en/product_tours/offermatica/ab

How does the cookie window affect measuring creative performance?

It is hard to measure creative performance or compare the creative performance if all the creatives were not live at the same time (start to end). The problem which comes is the cookie window. The creatives which are live for a longer period of time have the advantage of a longer cookie window and the creatives which are live for a shorter period of time have the big disadvantage. This is a bigger issue if one of the creative is live only during towards the end of the campaign.

Looking at the example below, the creative C, was live during the last 4 days of the campaign and thus does not have the cookie window advantage which creative A and B do.

Thus, it is recommended to think through the methodology based on the cookie window before starting the analysis. One solution is that measure the results 30 days beyond the campaign end date, then all the creatives will have similar cookie windows. However, this solution is not applicable in all situations specially if there is a big seasonality issue of the product example: Christmas candy canes, holiday decorations, customized products for a special occasion like Mother 's Day etc.

Samples, how can we have a similar concept online?

I am sure everyone has had a free sample ice cream or candy bar. I just had two M&M ice cream bars while walking up and down on Michigan Ave. Anyone who has walked on Michigan Ave today between 12-2 has to have had it. They had trucks sitting on the road and they were giving them away. Even the trash cans on Michigan Ave were showing that – see picture below (I wanted to take a picture of the trucks but missed it). How measureable are these samples? Yes, we can model out how many ice cream bars were given out and use an average conversion rate to guesstimate the impact of the samples.

While walking back, I thought how we can have a similar concept online. Yes, we cannot give away free ice cream online but there are a lot of different products which can be sampled online. For example: When Sony or Microsoft release new video game for Playstation or Xbox, demo versions of the games should be available to download online for the gaming enthusiasts to play. After the demo, there should be a promotional offer to buy the full version of the game. This would be fully measureable – from number of demo games played to how many gamers bought the full version of the game. One more thing could be free versions of browser or software like picture organizer.

Even looking at upward in the funnel, there can be online banners and the click-through rate would help understand the initial engagement with the game. Nowadays even in-banner gaming is a possibility with the fancy rich-media banners.