Retired course

This course has been retired and is no longer supported.

About this lesson

The statistic created in an hypothesis test is only 100% valid for the sample used to conduct the test. The application to the broader data set includes some uncertainty about the statistic in the entire data set. Therefore, it is possible to reach a wrong conclusion about the entire data set based upon the analysis of the sample. The elements of that risk are classified as Alpha risk and Beta risk. This lesson explains these risks and their significance.

Exercise files

Download this lesson’s related exercise files.

Alpha and Beta Risk.docx
61 KB Alpha and Beta Risk - Solution.docx
61.3 KB

Quick reference

Alpha and Beta Risk

When using inferential statistics, there is the possibility of making a wrong decision. This is known as alpha risk and beta risk.  The magnitude of these risks are related to the confidence level and the sampling approach.

When to use

Whenever inferential statistics are used, there is the possibility of alpha risk and beta risk.  The alpha risk is determined as soon as the confidence level is set and the beta risk is based upon the sampling approach and the characteristics of the data.

Instructions

Hypothesis testing provides a statistic that is used to either reject the Null hypothesis or fail to reject the Null hypothesis.  However, that decision is based upon comparing the statistic to some threshold value.  Depending upon where that value is set, there is the possibility of making a wrong decision.

  • False Positive – Rejecting the Null hypothesis when it is true.  This is known as the alpha risk or Type I error
  • False Negative – Failing to reject the Null hypothesis, even though it is not true.  This is known as the beta risk or Type II error.

A truth table reflecting these conditions is shown below:

Needless to say the desire is to always make the right decision.  But at times there is an overlap in the sample data sets and that leaves a zone where a statistical value could be reached with either truth condition.  In that case, the alpha and beta risks are inversely related.   When one gets larger the other gets smaller.  Consider the drawing below.  If the threshold line is moved to the right, the alpha risk will decrease but the beta risk will increase.  And the opposite will happen if the threshold line is moved to the left.

 

Increasing the sample size will reduce these risks because the confidence interval will be reduced.

The alpha risk is determined by subtracting the confidence level from 1.  So a confidence level of .95 will create an alpha risk of .05.  This is also referred to as the significance level.   The alpha risk creates a bias towards the Null hypothesis.  If a wrong decision is made, we prefer that we fail to recognize a difference rather than thinking we have seen a difference when one is not there; and then spending money and effort fixing a problem that does not exist.

The beta level will be determined by the differences in the data and the sampling approach used.  The Beta risk is referred to as the Power of a test.  The Power provides a sense of the effectiveness of the hypothesis test.

When conducting hypothesis testing, you must choose an alpha risk level and it is related to your confidence level.  Generally, the higher the impact, the higher the confidence level and therefore the smaller the alpha risk.  The table below shows some typical confidence levels and associated alpha risks for different business conditions.  Most Lean Six Sigma projects operate at the 95% confidence level which is a .05 alpha risk.

Effect of Error

Confidence Level

α Risk

Low Cost Rework

.9

.1

High Cost Rework

.95

.05

Personal Injury

.99

.01

Single Death

.999

.001

Multiple Deaths

.9999

.0001

Hints & tips

  • Lean Six Sigma projects normally use a .95 confidence level.  Coordinate closely with stakeholders and Black Belts before changing that value.
  • The principle of “innocent until proven guilty” is being applied to the Null hypothesis.  That is the alpha risk is so low, It is essentially saying that there is only a 5% chance of a False Positive.
Login to download
  • 00:04 Hi, I'm Ray Sheen.
  • 00:06 Even when you do everything right,
  • 00:08 you can get the wrong answer with a hypothesis test.
  • 00:12 And this is one of the problems we have whenever we're working with statistics.
  • 00:16 When we're working with hypothesis testing,
  • 00:18 this phenomena is known as the alpha risk and the beta risk.
  • 00:21 So let's consider what could happen with a hypothesis test.
  • 00:27 The hypothesis test will provide a statistic that we use to either reject
  • 00:31 the null hypothesis or fail to reject the null hypothesis.
  • 00:35 But depending upon the statistic value, it's possible to reach a wrong conclusion.
  • 00:40 For instance, it's very possible that there's an overlap
  • 00:43 in the statistic between the datasets where the null is true and
  • 00:46 the dataset with the alternative hypothesis is true.
  • 00:50 And if the statistic is in that area of overlap,
  • 00:53 it's quite likely that the wrong conclusion could be made.
  • 00:56 The condition where we reject the null hypothesis even though it is true
  • 01:01 is known as the alpha risk.
  • 01:03 And the condition where we fail to reject the null hypothesis
  • 01:06 even though it is not true is the beta risk.
  • 01:09 It's fairly obvious that these two risks are inversely related.
  • 01:13 When you lower one type of risk, you are likely to increase the other type.
  • 01:17 For instance, if we change the level of acceptable statistic so
  • 01:20 as to reduce the likelihood of the alpha risk which is that we'll reject the null
  • 01:24 hypothesis even though it's true, it would be offset by an increase in
  • 01:28 the risk that we fail to reject the null when it is not true.
  • 01:32 Now, we can get the effect of reducing both alpha risk and
  • 01:35 beta risk if we increase the sample size.
  • 01:38 This will decrease the confidence interval which will reduce the likelihood
  • 01:41 of an error.
  • 01:43 Let me show you what I mean about alpha risk and beta risk.
  • 01:46 This truth table considers four conditions.
  • 01:48 The top row is the case where the null hypothesis is true and
  • 01:52 the bottom row is the case where the alternative hypothesis is true.
  • 01:55 Then the left column is the case where the statistic
  • 01:57 would say that we failed to reject the null hypothesis and the right column
  • 02:01 is the case where the statistic says to reject the null hypothesis.
  • 02:06 So the upper left quadrant is the correction decision.
  • 02:09 In that case, the null hypothesis is true and we fail to reject the null hypothesis.
  • 02:15 And the lower right quadrant is a correct decision,
  • 02:18 the alternative hypothesis is true and we reject the null hypothesis.
  • 02:23 But now, let's look at the upper right quadrant.
  • 02:25 The null hypothesis was true but we rejected the null hypothesis.
  • 02:30 That is an alpha risk, or sometimes called a type I error.
  • 02:34 And the lower left corner is also a problem.
  • 02:37 In this case, the alternative hypothesis is true, but we failed to reject
  • 02:42 the null hypothesis, so we'll proceed as though the null hypothesis was true.
  • 02:47 That is the beta risk and it's called a type two II error.
  • 02:51 Now I think we can all start with the assumption
  • 02:53 that we want to make the right decision.
  • 02:55 But if there's going to be a mistake,
  • 02:56 I would rather that the mistake was that I didn't recognized the problem rather than
  • 03:00 I thought we've found the problem that was not there.
  • 03:03 When that happens, I'm likely to spend a lot of time and
  • 03:06 money fixing a problem that does not exist.
  • 03:09 So the acceptable hypothesis test statistic
  • 03:11 would be based upon the magnitude the alpha risk I want to accept.
  • 03:15 And that alpha risk value is directly related to a confidence level that we
  • 03:18 discussed in a previous lesson.
  • 03:21 Alpha is 1- confidence level.
  • 03:24 But the overall accuracy is based upon more than just the alpha risk.
  • 03:28 It's a combination of both risks.
  • 03:30 We normally set the alpha risk based upon the confidence interval.
  • 03:34 Then with the sample size and
  • 03:35 the nature of the data, we can determine the beta risk.
  • 03:39 Which leads to a measure of goodness for our test known as power.
  • 03:43 This is 1 minus the beta value.
  • 03:45 So let's talk a bit more about these terms, significance and power.
  • 03:50 The alpha risk is related to the significance term.
  • 03:53 Alpha, or as it's sometimes called,
  • 03:55 significance, is the likelihood of a false positive.
  • 03:58 We think something is there when it is not.
  • 04:01 Because of the typical values we select for confidence interval,
  • 04:04 the analysis is normally biased towards accepting the null hypothesis.
  • 04:09 The evidence of a difference in the dataset must be very clear for
  • 04:12 us to reject the null hypothesis.
  • 04:15 And the beta risk is related to the term power.
  • 04:18 The beta risk is the risk of a false negative.
  • 04:21 We say nothing is there when something really is there.
  • 04:24 The power factor for a test is a measure of the effectiveness of that test.
  • 04:30 Let me show you a diagram of this.
  • 04:32 Let's say that this distribution represents the dataset
  • 04:35 where the null hypothesis is true.
  • 04:37 And as part of our analysis,
  • 04:39 we're comparing it with this dataset where the alternative hypothesis is true.
  • 04:44 Notice that there is an area of overlap where if the test statistic is in that
  • 04:47 zone, we won't know which case is true.
  • 04:50 Now based upon the confidence level that I'm using, we can draw a threshold line.
  • 04:55 Any statistic value that is to the right of the line will reject the null
  • 04:58 hypothesis.
  • 04:59 But any time the statistic value is to the left of the line,
  • 05:02 we will fail to reject the null hypothesis.
  • 05:05 As you can see, there is a small zone of the dataset
  • 05:08 where we would reject the null hypothesis even though it is true.
  • 05:12 And in this case, there is a larger zone where we would fail to
  • 05:15 reject the null hypothesis even though the alternative hypothesis is true.
  • 05:19 Now in any case the differences between the two datasets is so
  • 05:22 great that there is no area of overlap.
  • 05:25 But also there are many cases where the overlap is so
  • 05:27 great that we almost never reject the null hypothesis.
  • 05:32 Okay, let's wrap up this discussion with a few comments
  • 05:35 about choosing your confidence level and alpha value.
  • 05:39 As we said, alpha and confidence level are directly related,
  • 05:43 alpha = 1- the confidence level.
  • 05:46 The alpha level is often chosen based upon the type of risk involved,
  • 05:49 with very high risk, a high confidence level is chosen.
  • 05:53 And of course this high confidence level will have a tendency
  • 05:56 to depress the beta value in the power of the test.
  • 06:01 Most of the time, Lean Six Sigma projects are using a confidence level of 0.95 and
  • 06:06 that's the level I'll be using for all of our examples.
  • 06:09 But if your project has higher levels of risk,
  • 06:12 you may want to change the confidence level.
  • 06:14 This table represents some of the commonly used values
  • 06:18 depending upon the risk of getting the analysis wrong.
  • 06:22 Alpha and beta risk help us to understand the likelihood of a false positive or
  • 06:26 a false negative when conducting a hypothesis test.

Lesson notes are only available for subscribers.

PMI, PMP, CAPM and PMBOK are registered marks of the Project Management Institute, Inc.

Gift this course