Design of Experiments for Web Analytics

After recently participating in a discussion with leading Search Engine Optimization (SEO) experts, I realized that there is an under-utilized market for using Six Sigma tools, such as Design of Experiments (DOE) to increase the value of web analytics for increasing sales, conversion rates, and Search Engine Optimization.

OFAT – One Factor at a Time

Most experts in marketing and SEO are familiar with the tactic of using A-B testing to improve traffic, goal conversion, and other website metrics. They may not realize that by using a well designed DOE, they could do multiple A-B tests simultaneously with a minimal amount of confounding while accelerating learning and results.

All Factors at Once

I found little evidence that Six Sigma tools like DOE are commonly being used in the SEO industry. One of the few examples of a case study using Six Sigma Methodology for SEO was to increase the conversion rate of song downloads from a music download website. The author was very open about the process and metrics that were used in the study. They demonstrated the use of the Six Sigma DMAIC (design, measure, analyze, improve, control) approach. From the study it appears they applied all the changes at once and measured the month end results. Applying all the proposed changes at once, while fast, sometimes provides misleading results.

For example (hypothetically), one or more of the changes could have had a negative individual effect of reducing the conversion ratio, while the net affect of all the changes could have still been positive. Reversing or adjusting the changes having a negative impact could have yielded an even higher net improvement, if they were individually quantified, which is not possible when making all the changes at once.

Multi-factorial Design of Experiments

Another option would have been to use a DOE to measure the effects of the individual factors and also interactions between the factors (proposed changes) on the results. Using the referenced study as an example, an experiment could be designed using four factors with two levels each:

Factor	+ High Level (current)	- Low Level (proposed)
Sample length of song	part of song (+)	whole song (-)
Button text	“buy now” (+)	“click here for free downloads” (-)
“downloading fees” text	displayed on website pages (+)	not displayed on website pages (-)
sales funnel process	8 steps (+)	3 steps (-)

Testing all of the 16 possible combinations (full factorial) of the factor levels would yield the most information about the individual effects and their interactions with each other. A full factorial experiment in this case is probably overkill.

Another option would be to run an 8 run fractional factorial experiment which will still provide useful insights to the individual main effects and some information on two factor interactions.

An example recipe for running the experiment follows. Each run is made one at a time, measuring the analytics results for a period of time of time under the conditions described by the run. For example, run 4 would have the following factor levels set:

• sample length of song = part of the song (represented by the + sign)
• button text = “buy now” (represented by the + sign)
• “downloading fees” text = not displayed on the web pages (represented by the – sign)
• sales funnel process = 3 steps (represented by the – sign)

Run #	Sample length of song	Button text	“downloading fees” text	sales funnel process
1	-	-	-	-
2	+	-	-	+
3	-	+	-	+
4	+	+	-	-
5	-	-	+	+
6	+	-	+	-
7	-	+	+	-
8	+	+	+	+

After completing the eight run experiment and collecting the analytic data (as recorded by analytics software, such as Google Analytics), the data would typically be analyzed with the help of JMP or MiniTab software.

The analysis should reveal the optimal combination of the factors studied to maximize the goal conversion rate. Additionally it should provide insight about which of the factors has the highest contribution to the improvement, information that can be used to hone in on additional related ideas to consider for future improvements.

In a market segment like Search Engine Optimization and website goal conversion, it is likely that the information learned from one experiment can be transferrable to other areas immediately.

Summary

Running Design of Experiments like the multifactorial one described above can provide valuable quantitative information about improvements that one can’t get otherwise, but it is not without a cost. DOE’s take time to set up, run, and analyze. They also require the use of experienced Six Sigma practitioners to effectively analyze and interpret the results.

In order to maximize the return on investment from this process, it is better suited for complex challenges where an organization might spend months making dozens to hundreds of individual changes to try to improve their results while not fully understanding the potential effect of the changes at the outset. The additional understanding gained about the contribution of individual changes and interactions can efficiently pinpoint the areas that have the most opportunity to yield the highest improvement.

photo credit: Alex E. Proimos / CC BY 2.0

Technorati Tags: SEO,Analytics,Six Sigma,Business Intelligence,Search Engine Optimization,DOE,Design of Experiments