Design of Experiments for Web Analytics

After recently participating in a discussion with leading Search Engine Optimization (SEO) experts, I realized that there is an under-utilized market for using Six Sigma tools, such as Design of Experiments (DOE) to increase the value of web analytics for increasing sales, conversion rates, and Search Engine Optimization.

OFAT – One Factor at a Time

Most experts in marketing and SEO are familiar with the tactic of using A-B testing to improve traffic, goal conversion, and other website metrics. They may not realize that by using a well designed DOE, they could do multiple A-B tests simultaneously with a minimal amount of confounding while accelerating learning and results.

All Factors at Once

I found little evidence that Six Sigma tools like DOE are commonly being used in the SEO industry. One of the few examples of a case study using Six Sigma Methodology for SEO was to increase the conversion rate of song downloads from a music download website. The author was very open about the process and metrics that were used in the study. They demonstrated the use of the Six Sigma DMAIC (design, measure, analyze, improve, control) approach. From the study it appears they applied all the changes at once and measured the month end results. Applying all the proposed changes at once, while fast, sometimes provides misleading results.

For example (hypothetically), one or more of the changes could have had a negative individual effect of reducing the conversion ratio, while the net affect of all the changes could have still been positive. Reversing or adjusting the changes having a negative impact could have yielded an even higher net improvement, if they were individually quantified, which is not possible when making all the changes at once.

Multi-factorial Design of Experiments

Another option would have been to use a DOE to measure the effects of the individual factors and also interactions between the factors (proposed changes) on the results. Using the referenced study as an example, an experiment could be designed using four factors with two levels each:

Factor	+ High Level (current)	- Low Level (proposed)
Sample length of song	part of song (+)	whole song (-)
Button text	“buy now” (+)	“click here for free downloads” (-)
“downloading fees” text	displayed on website pages (+)	not displayed on website pages (-)
sales funnel process	8 steps (+)	3 steps (-)

Testing all of the 16 possible combinations (full factorial) of the factor levels would yield the most information about the individual effects and their interactions with each other. A full factorial experiment in this case is probably overkill.

Another option would be to run an 8 run fractional factorial experiment which will still provide useful insights to the individual main effects and some information on two factor interactions.

An example recipe for running the experiment follows. Each run is made one at a time, measuring the analytics results for a period of time of time under the conditions described by the run. For example, run 4 would have the following factor levels set:

• sample length of song = part of the song (represented by the + sign)
• button text = “buy now” (represented by the + sign)
• “downloading fees” text = not displayed on the web pages (represented by the – sign)
• sales funnel process = 3 steps (represented by the – sign)

Run #	Sample length of song	Button text	“downloading fees” text	sales funnel process
1	-	-	-	-
2	+	-	-	+
3	-	+	-	+
4	+	+	-	-
5	-	-	+	+
6	+	-	+	-
7	-	+	+	-
8	+	+	+	+

After completing the eight run experiment and collecting the analytic data (as recorded by analytics software, such as Google Analytics), the data would typically be analyzed with the help of JMP or MiniTab software.

The analysis should reveal the optimal combination of the factors studied to maximize the goal conversion rate. Additionally it should provide insight about which of the factors has the highest contribution to the improvement, information that can be used to hone in on additional related ideas to consider for future improvements.

In a market segment like Search Engine Optimization and website goal conversion, it is likely that the information learned from one experiment can be transferrable to other areas immediately.

Summary

Running Design of Experiments like the multifactorial one described above can provide valuable quantitative information about improvements that one can’t get otherwise, but it is not without a cost. DOE’s take time to set up, run, and analyze. They also require the use of experienced Six Sigma practitioners to effectively analyze and interpret the results.

In order to maximize the return on investment from this process, it is better suited for complex challenges where an organization might spend months making dozens to hundreds of individual changes to try to improve their results while not fully understanding the potential effect of the changes at the outset. The additional understanding gained about the contribution of individual changes and interactions can efficiently pinpoint the areas that have the most opportunity to yield the highest improvement.

photo credit: Alex E. Proimos / CC BY 2.0

Technorati Tags: SEO,Analytics,Six Sigma,Business Intelligence,Search Engine Optimization,DOE,Design of Experiments

Teaching through Demonstration on Kitchen Nightmares

A frequent tactic for improving restaurant quality on the TV show Kitchen Nightmares is to reduce the number of items on the menu. On the Anna Vincenzo’s episode the restaurant had an overwhelming 180 items. It might seem obvious that if you simply explained that to the owner the the point would be made, but often that isn’t the case. Gordon Ramsey went to the effort of having the kitchen staff prepare just one order of each item on the menu before the owner arrived for the day. When she showed up, every table and flat surface in the restaurant was completely covered with different entrees on it. There were dishes with food from wall to wall. It became immediately apparent to her that they had too many choices on the menu to be able to do an exceptional job for every customer order. No explanation necessary.

Problems with off-the-shelf Lean Learning Tools

Creating Lean learning tools for a mass public almost goes against the principles of learning through development. It is very difficult to find the simplest starting point for a learning tool that is applicable to the widest audience. The best learning tools break the concept down to the simplest form in the context that it will be applied - and using local terminology. No two organizations are alike so the conditions are always going to be different. For this reason, the learning experience will not reach its full potential with off-the-shelf learning tools.

Production Pacing Indicator Example

Even a simple display, like the one above, of a target vs. actual production count with red/yellow/green color indicators raises numerous questions:

• should pausing the elapsed time be allowed (like for during breaks and lunch)
• if paused, should it recalculate target based on total running time only
• should stopping automatically clear the display or retain it until restarted?
• should the counts restart every hour or be cumulative?
• what should be used as a target: pcs/hour, pcs/shift, pcs/day
• should the target rate be fixed, or allow adjustment throughout the shift?
• how often should the display update?
• should the color change be based on mathematical rounding of the target/actual calculation?
• Should it round up or down?
• Should it display the actual percentage to target or just the colors?

As the participants surface these questions relative to the decisions that were made in creating the off-the-shelf tool, it will do two things:

1. the participant’s learning or application of the learning may be inhibited by their experience with the tool and its “limitations”.
2. participants may question the validity of the tool because they see more “problems” with it, than benefits.

Do any of the following sound familiar?

• “It doesn’t do this…”
• “It doesn’t do that…”
• “It won’t work here, because…”
• “How does it handle this situation?”
• “Can you do that with it?”

Statements like those above distract the participants from the learning experience.

The best tools for learning are the ones that are created and customized specifically at the point of use, in collaboration with and for the participants engaged in the experience.