Locked lesson.

Upgrade

About this lesson

This lesson addresses how to execute and control each of the experimental runs in the study. It also explains the importance of the measurement system that is used.

Exercise files

Download this lesson’s related exercise files.

Conducting the Study.docx
60.5 KB Conducting the Study - Solution.docx
60.2 KB

Quick reference

Conducting the Study

The conduct of the experimental runs should be done according to the DOE study plan and the testing procedure.  The measurement system must be capable to provide accurate measurements and the process should be monitored to ensure it is precisely followed so that the statistical analysis can be completed.

When to use

The conduct of the DOE study is done once the planning is complete and the test articles or process configurations are ready.  This continues until all data is collected and recorded.

Instructions

Conducting the study is simply following the DOE study plan. The challenge in this portion of the study is to ensure the plan is followed and that accurate data is collected.  Normally the experimental runs are randomized to reduce the impact of noise in the testing and data collection process. Although the sequence of runs is randomized, the data values for the response factor must be correctly paired with the experiment control factor configurations of each run.

Where possible use normal production operators. These individuals will already be familiar with many of the steps that must be followed in the experimental test procedures.   Even so, conduct a pilot run with the procedure to be certain there are no errors or confusion about what is to be done on each run. The operators who are conducting the experimental runs should understand that they are participating in a DOE study and that there may be something they are asked to do as part of the study that would not be part of how they would normally build the product or operate the process. Some operators are reluctant to run configurations that they know will lead to an inferior result. An understanding of the DOE study, and possibly a waiver to allow them to deviate from standard procedures will help to overcome problems raised by the operators.

Another significant concern in any DOE study is the capability of the measurement system used to measure the response factor or variable.   The system needs to be accurate, precise, and stable through the full range of measurements. Unless the measurement system is a standard system used in production, a measurement systems analysis will probably be needed to ensure the measurement system is adequate. The measurement error in the system must be much lower than the measured value of the response factor or the recorded measurement will not be accurate and the statistical analysis will be in error.

Hints & tips

  • Some operators may try to rework a run result to make it better. Ensure that does not occur so your data will be valid. You may need to reassure the operator that they are not being tracked for scrap of defects when participating in the DOE study.
  • Even though the runs are done in random sequence, the data value must be correctly paired with the run configuration of control factors.
  • Don’t let operators' “batch” the runs and then record the data. They should record each data value for a run as soon as the run is completed. Batching greatly increases the chance that data values will be paired with the wrong configuration.
  • If you are not certain that the measurement system is capable, you should do a measurement system analysis.
Login to download
  • 00:00 Hi I'm Ray Sheen, well the study's been planned, and
  • 00:05 the samples are ready, so now it's time to actually conduct the tests.
  • 00:12 Of course the heart of the study is the experimental runs.
  • 00:17 Each run of a test sample, or
  • 00:18 process configuration creates a data point that will be used in the analysis.
  • 00:23 The experiment is the set of all of the runs.
  • 00:26 In an early lesson, we discussed how to determine the configuration for
  • 00:30 each sample.
  • 00:30 However, a best practice is to randomize those runs so
  • 00:34 that any noise associated with the order is minimized.
  • 00:37 The exception to that randomization is the center points.
  • 00:40 Those are sometimes set at intervals in the run sequence so
  • 00:44 that they can be used to check for drift or changes in the testing.
  • 00:47 Another practice I have learned over the years is to
  • 00:50 be sure that the test operators understand what you are doing.
  • 00:54 They don't have to understand the statistics, but they should know that this
  • 00:58 is part of an experiment, and that they need to follow the procedures.
  • 01:01 In particular,
  • 01:02 the data collection procedure may be different than what they're used to doing.
  • 01:06 Be sure that they understand what must be done,
  • 01:09 because incorrect data collection invalidates everything.
  • 01:13 Another issue that I've seen occur is the operator may want to fix a test or
  • 01:17 a run that have poor results.
  • 01:19 When the common practice in a process is for operators to rework a problem at
  • 01:23 the work station, they may try to do it on your runs.
  • 01:26 Doing that on these DOE runs will invalidate the data.
  • 01:30 And occasionally, I've had an operator question whether they should even do a run
  • 01:35 because they know it will create an unsatisfactory result.
  • 01:38 Assure them that we need that run done, and the data collected,
  • 01:42 so that the statistical analysis will be able to accurately predict when
  • 01:46 an unacceptable performance is likely to occur.
  • 01:49 Another point that has been briefly mentioned in the past, but
  • 01:53 it's focused on at this time is the measurement system that is sued for
  • 01:56 the responsive factor or variable.
  • 01:58 This measurement system must be stable and capable.
  • 02:02 That means that the measurement system is not one that is normally used.
  • 02:06 A measurement systems analysis should be done.
  • 02:08 If it is one that is normally used, the analysis should be reviewed to
  • 02:13 ensure it is capable to provide accurate data for these experimental runs.
  • 02:17 A final point about the experimental runs,
  • 02:20 pilot the procedure before you start collecting the actual data.
  • 02:23 There maybe a few steps that are unclear or have a problem.
  • 02:27 Get those worked out.
  • 02:28 Remember that you must have all the data points for the statistics to be valid.
  • 02:33 If you don't pilot the procedure in the first few runs or done in error, well,
  • 02:38 those will need to be repeated with new samples and may cause a delay.
  • 02:42 I wanna take a few moments to focus in on the measurement system error.
  • 02:47 A DOE study relies on data from the experimental runs when
  • 02:50 completing the statistical analysis.
  • 02:52 It's not a stretch to state that inaccurate data will lead to
  • 02:56 an inaccurate analysis.
  • 02:57 All processes have some level of inherent variation, and measurement is a process.
  • 03:02 When the measurement system is being used,
  • 03:05 the measurement value consists of two components.
  • 03:07 The actual value of what is being measured and the measurement system error.
  • 03:12 A measurement system analysis will show the accuracy, precision and
  • 03:15 stability of the measurement system.
  • 03:17 With that information,
  • 03:18 you can determine whether the measurements systems should be used in this study.
  • 03:22 You may need to switch measurement systems, or make improvements to that one,
  • 03:27 if the error is too large.
  • 03:29 As you can see from this diagram, when the measurement error is low,
  • 03:33 the measure value and true value are nearly equal.
  • 03:36 But when the measurement error is high, the measured value is no longer accurate.
  • 03:41 If you need more information about measurement system analysis,
  • 03:45 check out the GoSkills course that teaches that technique.
  • 03:48 Let's now highlight several aspects of controlling the DOE study.
  • 03:53 If the study is focused on an existing product, process or system,
  • 03:57 use the normal operators to do the actions required by your DOE study procedure.
  • 04:02 On the one hand,
  • 04:03 this will remove the use of atypical operators as a potential noise factor.
  • 04:08 However, in some cases the operator or
  • 04:11 category of operators may be one of your control factors.
  • 04:15 Obviously if that's the case you must specify which operators does which run.
  • 04:21 Make sure the operators are following the exact procedure and
  • 04:24 setting the control factors correctly for each run.
  • 04:27 For this reason,
  • 04:28 I prefer not to send a batch of samples to the operator to be tested.
  • 04:32 I don't wanna them to run items in the wrong order and
  • 04:36 thereby mix the data points.
  • 04:38 The statistical analysis requires that the data be precisely paired with
  • 04:42 the run configuration.
  • 04:44 And of course, make sure data is recorded correctly.
  • 04:48 I recommend that the data be recorded immediately after each run,
  • 04:51 to make sure that the data points do not become mixed.
  • 04:54 The extra oversight can create a Hawthorne effect, which is a term used for
  • 04:59 the impact of extra oversight in the process.
  • 05:02 The process operators become conscious that they are being watched, and
  • 05:06 therefore start to do things differently from their normal procedure.
  • 05:09 Make sure the operators understand that the oversight is not performance appraisal
  • 05:14 or a lack of trust in their ability.
  • 05:16 Rather, it's just part of the DOE study protocol and
  • 05:19 they should be following the study procedure using normal processes.
  • 05:24 The conduct of the test runs is at the heart of the DOE study.
  • 05:27 Everything depends upon collecting valid data and
  • 05:31 that occurs at this point in time.

Lesson notes are only available for subscribers.

PMI, PMP, CAPM and PMBOK are registered marks of the Project Management Institute, Inc.

Gift this course