Transforming Tribal Knowledge to Standard Process

Organizations lament over the risky dependence on Tribal Knowledge to run their business. Extracting the knowledge from the key individuals is not as simple as asking them to write down everything they know about a process. People holding the Keys to the Kingdom are frequently making heavy use of intuition and hard won experience that can’t be dumped to documentation immediately upon request.

A deliberate structured iterative process is needed to reveal, define, and understand the processes and criteria that people use to solve daily problems, make decisions, and manage in their specific operational areas.

Here is one generalized variation of a method to extract and structure tribal knowledge into a repeatable process complete with decision logic.

1. Following the 80/20 rule, identify and document the 20% of the process that represents 80% of the normal conditions and decisions of the typical happy path.
2. Create a visual holding place, if one doesn’t already exist, for the data (actions, issues, and decisions) associated with the process - probably in time sequence. This can be as simple as a spreadsheet.
3. The objective of the visual system is to have a wider  perspective of the norm so that the exceptions and non-standard conditions are more readily exposed for investigation and clarification.
4. Apply what I call pattern-based process improvement and creative slicing and dicing of the data to begin to fill in the missing pieces that are currently only understood by the people with the “keys to the kingdom”.

This method is best executed with participation between the mentor (process expert) and at least one other person with “outside the process” eyes to distinguish between the intuitive knowledge and explicit knowledge. The additional participants are part of the audit process to verify if the process is being standardized in a form that can be operated by other non-experts.

 

photo credit: ben.fitzgerald / CC BY 2.0

Design of Experiments for Web Analytics

After recently participating in a discussion with leading Search Engine Optimization (SEO) experts, I realized that there is an under-utilized market for using Six Sigma tools, such as Design of Experiments (DOE) to increase the value of web analytics for increasing sales, conversion rates, and Search Engine Optimization.

OFAT – One Factor at a Time

Most experts in marketing and SEO are familiar with the tactic of using A-B testing to improve traffic, goal conversion, and other website metrics. They may not realize that by using a well designed DOE, they could do multiple A-B tests simultaneously with a minimal amount of confounding while accelerating learning and results.

All Factors at Once

I found little evidence that Six Sigma tools like DOE are commonly being used in the SEO industry. One of the few examples of a case study using Six Sigma Methodology for SEO was to increase the conversion rate of song downloads from a music download website. The author was very open about the process and metrics that were used in the study. They demonstrated the use of the Six Sigma DMAIC (design, measure, analyze, improve, control) approach. From the study it appears they applied all the changes at once and measured the month end results. Applying all the proposed changes at once, while fast, sometimes provides misleading results.

For example (hypothetically), one or more of the changes could have had a negative individual effect of reducing the conversion ratio, while the net affect of all the changes could have still been positive. Reversing or adjusting the changes having a negative impact could have yielded an even higher net improvement, if they were individually quantified, which is not possible when making all the changes at once.

Multi-factorial Design of Experiments

Another option would have been to use a DOE to measure the effects of the individual factors and also interactions between the factors (proposed changes) on the results. Using the referenced study as an example, an experiment could be designed using four factors with two levels each:

Factor	+ High Level (current)	- Low Level (proposed)
Sample length of song	part of song (+)	whole song (-)
Button text	“buy now” (+)	“click here for free downloads” (-)
“downloading fees” text	displayed on website pages (+)	not displayed on website pages (-)
sales funnel process	8 steps (+)	3 steps (-)

Testing all of the 16 possible combinations (full factorial) of the factor levels would yield the most information about the individual effects and their interactions with each other. A full factorial experiment in this case is probably overkill.

Another option would be to run an 8 run fractional factorial experiment which will still provide useful insights to the individual main effects and some information on two factor interactions.

An example recipe for running the experiment follows. Each run is made one at a time, measuring the analytics results for a period of time of time under the conditions described by the run. For example, run 4 would have the following factor levels set:

• sample length of song = part of the song (represented by the + sign)
• button text = “buy now” (represented by the + sign)
• “downloading fees” text = not displayed on the web pages (represented by the – sign)
• sales funnel process = 3 steps (represented by the – sign)

Run #	Sample length of song	Button text	“downloading fees” text	sales funnel process
1	-	-	-	-
2	+	-	-	+
3	-	+	-	+
4	+	+	-	-
5	-	-	+	+
6	+	-	+	-
7	-	+	+	-
8	+	+	+	+

After completing the eight run experiment and collecting the analytic data (as recorded by analytics software, such as Google Analytics), the data would typically be analyzed with the help of JMP or MiniTab software.

The analysis should reveal the optimal combination of the factors studied to maximize the goal conversion rate. Additionally it should provide insight about which of the factors has the highest contribution to the improvement, information that can be used to hone in on additional related ideas to consider for future improvements.

In a market segment like Search Engine Optimization and website goal conversion, it is likely that the information learned from one experiment can be transferrable to other areas immediately.

Summary

Running Design of Experiments like the multifactorial one described above can provide valuable quantitative information about improvements that one can’t get otherwise, but it is not without a cost. DOE’s take time to set up, run, and analyze. They also require the use of experienced Six Sigma practitioners to effectively analyze and interpret the results.

In order to maximize the return on investment from this process, it is better suited for complex challenges where an organization might spend months making dozens to hundreds of individual changes to try to improve their results while not fully understanding the potential effect of the changes at the outset. The additional understanding gained about the contribution of individual changes and interactions can efficiently pinpoint the areas that have the most opportunity to yield the highest improvement.

photo credit: Alex E. Proimos / CC BY 2.0

Technorati Tags: SEO,Analytics,Six Sigma,Business Intelligence,Search Engine Optimization,DOE,Design of Experiments

Closed Data Systems Limit Learning and Improvement Opportunities

Try taking data from a process you are working to improve out of the system you normally use and look at it in other ways. If it is in a proprietary spreadsheet or reporting system that has fixed standard reports and charts, take the raw data out and put it in a clean spreadsheet or database and look at the data from other perspectives.

Slice and Dice the data:

• Filter
• Sort
• Aggregate
• Graph
• SPC chart
• Pivot

Using fixed systems, while good for standardized daily analysis routines, can close our minds to opportunities, exceptions, patterns and trends that can lead to quick improvements and unexpected systemic breakthroughs.

photo credit: brian.chu / CC BY-ND 2.0

Information Transparency: is there a Wrong Time?

This is Part 1 of a multi-part series on transparency of information. There are probably two strong camps that would argue for or against it. I am squarely in the middle as I have found from experience there is both a right time and a wrong time for it. The first part explores the reasons why transparency can hinder productivity.

What might be some of the wrong times for transparency?

1. when first developing a process and working out the kinks
2. when the information is a poor measure of performance
3. when one doesn’t fully comprehend the implications of having the information freely available
4. when trying to “motivate” (i.e. bully) a team or person to do their job better by exposing information
5. when information can be misinterpreted

Depending on the process, you may be trying to discover which measures are good metrics to use to understand a cause and effect relationship. Many practitioners like myself will have seen occasions where the management and front-line employees alike have criticized the repeated changes in metrics as poor leadership – the “they don’t know what they are doing” syndrome. In reality, it is often best not to stick with metrics just because they were included in the original project plan or specification. If you take on an outcome based approach instead of a metric based approach, you will likely experience a more successful project or initiative overall.

Exposing information to the broadest audience too early can lead to the cancellation of a good project for political reasons or simply because the wrong data was used in the justification analysis.

Depending on the organizational culture early transparency of an evolving process can also cause a dramatic increase in resistance to change and project participation because of negative press. Or worse, if the management realizes the problem is worse than they expected, they may intervene in an unproductive way squashing a working PDCA process driven approach with a more command and control method.

Pattern-Based Process Improvement

Pattern-based process improvement is a practical method of making improvements where an exhaustive analysis and re-engineering exercise is prohibitive. The process consists of looking for patterns to identify key characteristics of a process that might provide valuable insight about opportunities to improve the process. A prerequisite to finding patterns is having data or information available for review.

Real world example

Objective: Improve how parts are scheduled across manufacturing work centers to reduce late orders and unnecessary expediting.

Background: A supervisor of a manufacturing facility schedules orders on a number of similar manufacturing work cells. He normally uses the following information to create the schedule based on his experience, a few rules of thumb, and some light calculations:

• order due date
• current schedule of orders on the work cells (capacity vs. loading)
• change-over time (the  time it takes to change the machine to accommodate a different component part or assembly)
• order quantity
• estimated production time (time it takes to process the parts through the work cell)
• physical size and shape of the part on order

Method: The supervisor uses intuition based on a large number of observations to make decisions (Bayesian statistics) when creating the schedule. The idea behind pattern-based process improvement is to begin to quantitatively blend the supervisors experiential knowledge and Bayesian intuition with the data to create standard work that improves the predictability of the scheduling process.

Start by looking at the historical data about how the work cells were scheduled in the past. This method works best when the data is in a spreadsheet or database so the data can be looked at from several angles. Sort and group the data multiple ways looking for patterns or trends – good and bad. How much you can learn from the exercise depends on how much information is available and how valuable that information is. For example:

• Are there part sizes or size ranges that are commonly scheduled on certain work cells more than others?
• Are there work cells that have more downtime, more change-over time, more schedule changes, or expedites?
• Is there a pattern to size vs. quantity or quantity vs. work cell?

If you can find a desirable pattern or a trend, determine if it is possible to create a rule that would make the pattern a more consistent part of the standard scheduling process. If the pattern is associated with a negative result, determine if there is a way to detect the pattern early, or eliminate its presence completely. It is important when using pattern-based process improvement, the output of the process is monitored carefully to fully understand the effects of the change. Sometimes changes introduce new issues. This should be looked at as an iterative improvement process, not a “set it and forget it” tactic.

Data Tells a Story, but are You Reading the Whole Story?

Eighteen years ago I learned an important lesson about the intersection of data and people. The data alone rarely tells the whole story.

The engineering manager was preparing his weekly report for the staff meeting. He asked me, the intern, to investigate a SPC (Statistical Process Control) chart gone wild and summarize my findings. The chart graphed several “critical” dimensional characteristics of a headlight reflector for one of the highest selling vehicles at the time. This was a simple task.

Injection molding is a complex process, so the machines were instrumented with several sensors to monitor and record important process parameters. I strolled down to the production floor control room and pulled up the historical data from the data acquisition system for the previous week. I saw a dramatic change in several readings which were consistent with the dimensional hiccup on the SPC chart. Based on the readings and my knowledge of the injection molding process, I could predict which process settings were probably changed by the technician.

Ahh, the Story is developing.

I went out to the machine where the parts were being made and checked the Process Deviation log to verify my predictions and see if the technicians documented any changes. For the most part they did, and I was correct in my analysis of which process settings were changed. I also verified the gage at the quality check station was calibrated and working properly; it was.

I summarized my findings in a report based on data I collected in the control room, the shop floor, the quality check area, and the SPC chart. Unfortunately, I was missing one source of data from my findings... the technician who made the changes.

When the manager’s weekly report came out with a small reference to the Dimensional problem, the 2nd shift technician was upset because he felt like he was being unfairly criticized for deviating from the standard approved process and for causing the problem. He left a “See me” note on my chair.

And now the rest of the Story

The technician “kindly” explained to me that the reason for the dimensional problem was that the mold, which is normally water cooled, developed a crack in the steel that caused water to pour out of the mold (bad). Normally the mold would be taken to the tool room to get repaired, but the part it was making was for one of the highest selling cars and the other duplicate mold was already in the tool room for routine maintenance.

In order to keep the mold cool and maintain a safe work environment (no water all over the floor and machine), they ran compressed air through the water lines. This required numerous changes to the machine settings to keep the dimensions in the functionally acceptable range. The dimensional variation was much different from normal, but was still acceptable for assembly and to the customer.

Now regardless of whether or not it is consistent with best practice to run the mold in the non-standard condition, it was obvious the technician was doing his best to work in the interest of the company. Unfortunately, by neglecting to check with the person responsible for making the process changes, my report left the impression the technician was just not doing what he was supposed to do – something others might assume to be evidence of bad intent, laziness, or some other negative trait. This could have been avoided by applying the following advice:

“In order to understand why somebody does something, you’ll find the answer faster if you look for what’s right about their behavior, rather than what’s wrong.”

Craig Henderson

The above quote was taken from the presentation Nobody Likes Bad Change by Craig Henderson. This is a philosophy that has guided my work since that day - long ago.