Locked lesson.

Upgrade

About this lesson

The simplest experimental design approach is trial and error. If subject matters experts are generating the trial design, this can be successful. However, if the trial fails, this approach can lead to delays and overruns.

Exercise files

Download this lesson’s related exercise files.

Trial and Error.docx
61.2 KB Trial and Error - Solution.docx
60.7 KB

Quick reference

Trial and Error

Trial and Error is an experimental methodology where subject matter experts hypothesize the critical independent factors that will create a desired (or specifically undesired) response from the dependent factors.  The experiments are done with these factors.  If the experiments are not successful, another set of factors is selected.

When to use

Trial and error experimental approach is appropriate when there is a specific goal or response that is desired from the dependent factors and there are subject matter experts who can confidently select the appropriate independent factors for that experiment.

Instructions

When using the trial and error method, it is vital that the two conditions mentioned above are met.  First, the goal of the experiment will have a clear pass or fail criteria.  That is so that the experiment can be determined as a success or failure.  The second is that the subject matter experts must have adequate knowledge and experience to select the correct independent factors to be examined during the experiment.  If either of those conditions is not true, then this method will likely create delays and overruns to the project. 

If these independent factors selected do not create the desired response from the dependent factors, then a new set of independent factors must be selected.  This pattern is repeated until the desired response is achieved.  When there is an ambiguous goal, or the factors are just wild guesses, this approach often requires many iterations, leading to the delays and overruns.

When the aforementioned conditions are met, this approach will be the fastest and lowest cost experimental design approach.  By leveraging the expertise of the subject matter experts and focusing the experiments on a specific goal, the number of tests can be held to a minimum.

One caution with this approach is that the “solution” may not truly solve the problem.  If other factors are also significant, but not controlled in the experiment, the dependent factor results may be acceptable due to the uncontrolled factor rather than the independent factors that were tested. 

Hints & tips

  • The efficacy of this approach is highly dependent upon the knowledge and experience of the subject matter experts.  Work the best you can find, either internal or external and from any department in the business.
  • This is a difficult methodology to estimate.  An optimistic estimate is that the first set of factors are the correct factors and no further experiments are needed.  Most of the time that will be true, but when it isn’t, additional unplanned tests are needed and then this approach can quickly create massive delays and overruns.
  • If the goal is to gain knowledge about a technology or product, use a different approach.  This approach is best suited for testing a particular condition or to achieve a particular performance level.
Login to download
  • 00:04 Hi, I'm Ray Sheen.
  • 00:05 One of the experimental methods,
  • 00:07 in fact the most commonly used method, is trial and error.
  • 00:11 Let's take a look at it for a few moments.
  • 00:14 What do I mean by trial and error?
  • 00:17 In the trial and error method, a design or performance problem or
  • 00:21 a question is identified.
  • 00:23 Then subject matter experts who understand the product or
  • 00:26 process are consulted about the situation.
  • 00:29 These experts identify what they believe to be the most likely factor that is
  • 00:33 creating the problem or driving the performance of the product or process.
  • 00:37 That factor is then tested through several experiments to determine if it is
  • 00:41 the primary factor.
  • 00:42 If the results of the experiment do not show a clear connection between the factor
  • 00:46 and the product or process performance.
  • 00:49 Then the subject matter experts hypothesize a different factor and
  • 00:52 test the new factor with several experiments.
  • 00:56 This pattern continues until a factor or set of factors has been found
  • 01:00 that can explain the performance issue or create the level of desired performance.
  • 01:04 Once the factor or factor set is defined and
  • 01:07 understood, the experiments are completed.
  • 01:10 There are some advantages and
  • 01:12 benefits to this approach, along with some keys to making this approach successful.
  • 01:16 Let's look at these, first the benefits.
  • 01:19 This approach can be fast and the lowest cost approach.
  • 01:22 It may only take a few experiments to understand the relationship
  • 01:26 between the critical independent factor and the dependent factor.
  • 01:30 If the subject matter experts guess correctly, this will confirm their
  • 01:34 understanding of the product or process and validate any existing models.
  • 01:38 This approach uses the expertise of the subject matter experts.
  • 01:42 These individuals have knowledge and experience that should be leveraged to
  • 01:46 help the organization, and this is one of the ways that that knowledge can be used.
  • 01:50 Now let's consider the keys to success using this experimental approach.
  • 01:55 The obvious primary key is the access of subject matter experts.
  • 01:59 If they are not available then you're trusting to blind luck to guess what
  • 02:03 factors you should study.
  • 02:05 And just to be clear, the subject matter experts do not have to
  • 02:08 have advanced degrees in the product or process technology.
  • 02:12 Their expertise could be years of practical hands-on experience.
  • 02:16 The point is that they already understand a great deal about the product or
  • 02:20 process performance, and
  • 02:22 can therefore point the experiments in the right direction for testing and analysis.
  • 02:27 The other key is that this method works best when there is a specific problem or
  • 02:31 issue to be investigated.
  • 02:33 A general requirement to better understand the technology is not well suited for
  • 02:37 this approach.
  • 02:39 However, when the issue is focused,
  • 02:41 the subject matter experts can isolate the one or two factors for investigation.
  • 02:47 There are also some potential traps and pitfalls with this approach.
  • 02:51 One of these is it is almost impossible to predict
  • 02:54 how big the experimental effort will be when doing project planning.
  • 02:58 Will the first factor analyze to show the critical relationships between independent
  • 03:02 and dependent factors or will it take several tries to understand it?
  • 03:07 In my experience, when a project is planned using this approach,
  • 03:10 the plan generally assumes that the first set of experiments will be successful.
  • 03:14 When it's not the project is now delayed and over run creating additional problems.
  • 03:20 However, there is no rhyme or reason for presupposing how many attempts it
  • 03:25 will take, so the project plan in this area is a wild guess.
  • 03:30 Another pitfall is that the solution discovered may not really be the solution.
  • 03:35 This is especially true if only a few experimental tests are performed.
  • 03:39 There may be multiple factors that are affecting the performance, but
  • 03:42 the test is only controlling one or two of these.
  • 03:45 The test may be lucky that the others are aligned correctly for success, but
  • 03:50 the next time, those other factors may not be aligned.
  • 03:53 Since these experiments usually only look at one or
  • 03:55 two factors, interaction effects are often masked.
  • 04:00 Another pitfall is that the experiments chosen for
  • 04:02 testing will be based upon the knowledge and insight of the subject matter experts.
  • 04:07 If the problem is in the area where they do not have expertise,
  • 04:11 they may never guess the correct factors or set of factors.
  • 04:14 Limited knowledge and experience will limit the possible options for
  • 04:18 investigation.
  • 04:20 Finally, if the solution is not discovered within the first few sets of experiments,
  • 04:24 the subject matter experts are often put in a position of just guessing what to
  • 04:28 try next.
  • 04:29 At that point, the unstructured guesses can often lead to a very long and
  • 04:34 expensive test series until they stumble upon the right factors.
  • 04:38 Usually by that time, the project is under stress, and it's likely that poor
  • 04:42 decisions will be made with respect to the testing and design optimization.
  • 04:47 Although there are some pitfalls to watch out for,
  • 04:50 this is still the most commonly used approach.
  • 04:52 And it's definitely appropriate if you have an experienced subject matter expert
  • 04:56 and you've got a very specific design goal or problem to solve.

Lesson notes are only available for subscribers.

PMI, PMP, CAPM and PMBOK are registered marks of the Project Management Institute, Inc.

Gift this course