Design of Experiments for Web Analytics

After recently participating in a discussion with leading Search Engine Optimization (SEO) experts, I realized that there is an under-utilized market for using Six Sigma tools, such as Design of Experiments (DOE) to increase the value of web analytics for increasing sales, conversion rates, and Search Engine Optimization.

OFAT – One Factor at a Time

Most experts in marketing and SEO are familiar with the tactic of using A-B testing to improve traffic, goal conversion, and other website metrics. They may not realize that by using a well designed DOE, they could do multiple A-B tests simultaneously with a minimal amount of confounding while accelerating learning and results.

All Factors at Once

I found little evidence that Six Sigma tools like DOE are commonly being used in the SEO industry. One of the few examples of a case study using Six Sigma Methodology for SEO was to increase the conversion rate of song downloads from a music download website. The author was very open about the process and metrics that were used in the study. They demonstrated the use of the Six Sigma DMAIC (design, measure, analyze, improve, control) approach. From the study it appears they applied all the changes at once and measured the month end results. Applying all the proposed changes at once, while fast, sometimes provides misleading results.

For example (hypothetically), one or more of the changes could have had a negative individual effect of reducing the conversion ratio, while the net affect of all the changes could have still been positive. Reversing or adjusting the changes having a negative impact could have yielded an even higher net improvement, if they were individually quantified, which is not possible when making all the changes at once.

Multi-factorial Design of Experiments

Another option would have been to use a DOE to measure the effects of the individual factors and also interactions between the factors (proposed changes) on the results. Using the referenced study as an example, an experiment could be designed using four factors with two levels each:

Factor	+ High Level (current)	- Low Level (proposed)
Sample length of song	part of song (+)	whole song (-)
Button text	“buy now” (+)	“click here for free downloads” (-)
“downloading fees” text	displayed on website pages (+)	not displayed on website pages (-)
sales funnel process	8 steps (+)	3 steps (-)

Testing all of the 16 possible combinations (full factorial) of the factor levels would yield the most information about the individual effects and their interactions with each other. A full factorial experiment in this case is probably overkill.

Another option would be to run an 8 run fractional factorial experiment which will still provide useful insights to the individual main effects and some information on two factor interactions.

An example recipe for running the experiment follows. Each run is made one at a time, measuring the analytics results for a period of time of time under the conditions described by the run. For example, run 4 would have the following factor levels set:

• sample length of song = part of the song (represented by the + sign)
• button text = “buy now” (represented by the + sign)
• “downloading fees” text = not displayed on the web pages (represented by the – sign)
• sales funnel process = 3 steps (represented by the – sign)

Run #	Sample length of song	Button text	“downloading fees” text	sales funnel process
1	-	-	-	-
2	+	-	-	+
3	-	+	-	+
4	+	+	-	-
5	-	-	+	+
6	+	-	+	-
7	-	+	+	-
8	+	+	+	+

After completing the eight run experiment and collecting the analytic data (as recorded by analytics software, such as Google Analytics), the data would typically be analyzed with the help of JMP or MiniTab software.

The analysis should reveal the optimal combination of the factors studied to maximize the goal conversion rate. Additionally it should provide insight about which of the factors has the highest contribution to the improvement, information that can be used to hone in on additional related ideas to consider for future improvements.

In a market segment like Search Engine Optimization and website goal conversion, it is likely that the information learned from one experiment can be transferrable to other areas immediately.

Summary

Running Design of Experiments like the multifactorial one described above can provide valuable quantitative information about improvements that one can’t get otherwise, but it is not without a cost. DOE’s take time to set up, run, and analyze. They also require the use of experienced Six Sigma practitioners to effectively analyze and interpret the results.

In order to maximize the return on investment from this process, it is better suited for complex challenges where an organization might spend months making dozens to hundreds of individual changes to try to improve their results while not fully understanding the potential effect of the changes at the outset. The additional understanding gained about the contribution of individual changes and interactions can efficiently pinpoint the areas that have the most opportunity to yield the highest improvement.

photo credit: Alex E. Proimos / CC BY 2.0

Technorati Tags: SEO,Analytics,Six Sigma,Business Intelligence,Search Engine Optimization,DOE,Design of Experiments

Coke Meets ‘The Jetsons’ with new Robodispenser

The Jetsons Season 2 Volume 1- Rosie Flipped - watch more funny videos

Coca-Cola recently announced a revolutionary new soda dispenser they will soon pilot in selected areas. See a great comprehensive article in the June 8, 2009 issue of Information Week magazine. It is a significant announcement for many reasons.

• The Freestyle “Fountain” dispenses over 100 different beverages (sodas, juices, teas, and flavored waters) using thirty highly concentrated flavor cartridges.
• The system uses RFID tags on the flavor cartridges to track inventory
• The dispenser communicates with a central Corporate point-of-sale system and data warehouse via a dedicated wireless network recording quantity, types, time, and location of drinks sold

Customer Value

• More choices
• Special offers

Retailer Value

• Inventory optimization
• 100 choices in the same footprint as an eight to 12 drink dispenser
• Information about sales patterns that can be used to provide special offers to the customers
• Easier ordering process with recommendations based on ten day moving average of sales, cartridge inventory, and dispenser inventory.
• Access to a portal with visibility to the consumption data and statistics that can be sliced and diced by cartridge, drink, dispenser, hour, day, week, etc.

Value to Coca-Cola

• Faster, lower cost new product Research & Development, Market Research, product piloting and production rollout
• Remotely adjust beverage formulas
• Better understanding of regional customer preferences
• First to market; four years in development
• Better data to align supply with demand for Sales and Operations planning
• Competitive advantage with offer of high retailer and customer value

When Information Technology is applied effectively in close cooperation with Business Operations, improvement initiatives offer opportunities for significant gains for all parties involved from the supply chain to the customer.

One Strategy to Describe GM History in more than 11 Chapters

Leading up to a likely bankruptcy, GM Executives have had to make numerous big decisions recently about restructuring to maintain its future viability. Closing a large number of dealerships was one of the chosen strategies. In order to make smart decisions about which dealerships to close, the leaders would need two things: selection criteria aligned with the scope of the goal and a lot of data that links key performance indicators to dealership locations.

Despite the media reports of “successful” dealerships tagged for closure, the selection process is not likely as simple as just picking the bottom 40% of dealerships from a list of annual sales. Some accounts have described the targeted dealerships as “under-performing”. Under-performing is relative condition, especially in a huge organizational network implementing other significant changes that will impact dealership performance in the future, such as eliminating entire Makes (Pontiac) with multiple product lines.

Strategies like this one require careful consideration of a number of complex scenarios to minimize risk while maximizing the predictability and effectiveness. With over 6000 dealerships, it is unlikely that a comprehensive comparative review of each individual dealership can be conducted. More likely a review will be made of multiple combinations of aggregated key performance metrics like:

• sales history by make, model, and region
• forecast by make, model, region
• volume vs. mix
• up-sell/cross-sell statistics
• profitability by new/used car sales
• service labor vs. parts sales
• warrantee reimbursement
• dealership proximity concentration
• years in business - short term vs. long term performance
• customer loyalty
• average vehicle inventory
• regional demographics affecting future sales forecasts
• seasonal sales variation

*I’m not connected to the details of the actual selection process; I am supposing.

The aggregated data will be sliced and diced, repeatedly applying a great deal of analysis, criteria refinement, and simulations before lines are drawn and lists of targeted dealerships are made. The actual data specific to the selected locations are likely plugged back into the equation to solidify the estimate.

Executing a strategy like this without the right process would be a monumental task carrying high risk of missing expectations. It is necessary to have:

• strategy - clear understanding of the assumptions/expectations behind the selected strategy
• process - a process to identify the combination of complex criteria that will yield the optimum result in the face of many significant, changing, and sometimes unpredictable factors
• technology - a fast and flexible process to filter, sort, and aggregate massive amounts of demographic and performance data in any number of complex combinations and extrapolate that into specific dealership locations