Locked lesson.

Upgrade

About this lesson

The ability of a hypothesis test to provide insight into the characteristics of a data population is based on the sample of data selected and some statistical characteristics of the sample and the population. The relationship between these gives rise to two measures that can be made concerning the validity of the hypothesis test. These measures are Significance and Power and will be discussed in this lesson.

Quick reference

Significance and Power

Significance and Power are measures of the reliability of the Alpha Risk and Beta Risk associated with a hypothesis test. Significance represents the ability to differentiate between data sets. Power represents the effectiveness of the hypothesis test.

When to use

Whenever inferential statistics are used, there is significance and power associated with the test. These are measures that provide insight into the reliability of the hypothesis test answer.

Instructions

Significance in hypothesis testing is a measure of the ability of the test to determine whether a difference exists between two data sets. We can think of significance in two ways, practical significance, and statistical significance. Practical significance is an indication that the difference between the two data sets will impact business performance. Statistical significance is an indication that there is a difference between the data sets that cannot be explained by normal random chance variation. Statistical difference may not lead to practical difference.  The business needs to assess whether the statistical difference is sufficient to impact business performance before making changes based on the difference.

 Hypothesis testing provides a statistic that is used to either reject the Null hypothesis or fail to reject the Null hypothesis.  However, that decision is based upon comparing the statistic to some threshold value.  Depending upon where that value is set, there is the possibility of making a wrong decision.

  • False Positive – Rejecting the Null hypothesis when it is true.  This is known as the alpha risk or Type I error
  • False Negative – Failing to reject the Null hypothesis, even though it is not true.  This is known as the beta risk or Type II error.

In some cases, the two data sets representing the hypothesis overlap.  Therefore, there are some data values that could have been generated in either data set. The ability to detect the difference is the alpha risk and is set based on the confidence level.

If the threshold line is moved to the tail of the data set representing the null hypothesis, the alpha risk will decrease but the beta risk will increase.  And the opposite will happen if the threshold line is moved in the other direction.

Increasing the sample size will reduce these risks because the confidence interval will be reduced with a more accurate mean and standard deviation.

The alpha risk is determined by subtracting the confidence level from 1.  So a confidence level of .95 will create an alpha risk of .05.  The alpha risk creates a bias toward the Null hypothesis.  If a wrong decision is made, we prefer that we fail to recognize a difference rather than thinking we have seen a difference when one is not there; and then spending money and effort fixing a problem that does not exist.

The beta level will be determined by the differences in the data and the sampling approach used.  The Beta risk is referred to as the Power of a test.  The Power provides a sense of the effectiveness of the hypothesis test.

When the two data sets do not overlap, the Beta risk goes to zero. The alpha risk is still determined by the selected confidence level.

Hints & tips

  • Increasing the number of data points will reduce uncertainty in the mean value and will often lead to a smaller standard deviation.  These will have a tendency to reduce the Beta risk which increases the Power of the test.
  • Depending upon the hypothesis test selected, the significance of the result can change. That is why we provide a table for selecting hypothesis tests – it is so you can select the one that is likely to have the best significance and power.
Login to download
  • 00:04 Hi, I'm Ray Sheen.
  • 00:06 Based upon the results of your alpha risk and beta risk,
  • 00:09 you can determine the measure of effectiveness of your analysis.
  • 00:13 These are referred to as significance and power.
  • 00:18 I'll start with significance.
  • 00:20 Significance is another way to say that the results of the analysis shows there is
  • 00:24 something different happening.
  • 00:27 But I want to divide that into two perspectives, practical significance and
  • 00:31 statistical significance.
  • 00:33 Practical significance means it's a significance that someone notices, and
  • 00:38 has business impact.
  • 00:40 There is a change in the technical or
  • 00:41 economic impact based upon the two conditions being studied.
  • 00:45 In fact, the difference will create enough benefit to offset any
  • 00:49 cost associated with implementing a change.
  • 00:52 That means it's worthwhile to make the change.
  • 00:54 That benefit could be financial or technical.
  • 00:57 I'll contrast that with statistical significance.
  • 01:00 In this case, the difference is large enough that it creates an effect in
  • 01:04 the data that cannot be explained away by just random variance.
  • 01:08 The difference can be seen in the data,
  • 01:10 even though it may not make any difference practically.
  • 01:14 And just to be clear, inferential statistics will be able to tell us
  • 01:18 if there is a statistically significant difference, the business will make
  • 01:22 a business decision about whether it is a practical difference for them.
  • 01:27 By the way, sometimes it is hard to perceive much difference between two sets
  • 01:31 of data, even though the analysis says it is statistically different.
  • 01:36 Other times, there appears to be a big difference,
  • 01:39 but the analysis says it's not really there.
  • 01:42 That is because the difference was well within the range of random variation.
  • 01:48 Let's take a look at a picture of significance.
  • 01:51 For starters, significance is related to the alpha risk.
  • 01:55 You may remember that that is the risk of the false positive.
  • 01:58 It is a condition when you say that there is a difference, but
  • 02:02 in reality there is not one.
  • 02:05 So we would be likely to do some big effort to treat something as different,
  • 02:09 and it would not have any effect because there was never any real difference.
  • 02:14 Significance creates a bias towards the null hypothesis.
  • 02:19 Let's look at an illustration.
  • 02:20 We'll start with a condition that would occur if the null hypothesis were true.
  • 02:25 Next, we will show a condition where the alternative hypothesis is true.
  • 02:30 Notice that we have a zone of overlap.
  • 02:33 If our data point falls in the zone, then we wouldn't know which was true,
  • 02:38 the null hypothesis or the alternative.
  • 02:42 Now, I'm going to draw a line that is our decision line.
  • 02:44 Everything to the left of the line is a case where we do not reject the null
  • 02:49 hypothesis, meaning we consider it true.
  • 02:52 Everything to the right of the decision line, we do reject the null hypothesis and
  • 02:57 consider that the alternative hypothesis is true.
  • 03:02 The decision line is drawn based upon the alpha risk.
  • 03:05 If our alpha risk is 95%, this shows the likelihood that we would reject
  • 03:10 the null hypothesis when we actually should have accepted it.
  • 03:15 So let's talk about power.
  • 03:17 Power is associated with the beta risk.
  • 03:20 This is the risk of a false negative.
  • 03:22 That means that the alternative hypothesis was true, but we didn't realize it.
  • 03:28 Therefore, we didn't make any changes because we still think the null
  • 03:32 hypothesis is true.
  • 03:34 Remember we are biasing the system in that manner, so that we don't waste time and
  • 03:38 money on changes that will have no effect.
  • 03:42 So a way to look at power,
  • 03:44 is that it is giving us a sense of effectiveness of the test.
  • 03:49 Let's bring back the illustration from the previous slide.
  • 03:52 We had noted that the alpha risk is that little section in black on the far
  • 03:56 right tail of the blue curve.
  • 03:59 Now we have a larger zone that we would fail to reject the null hypothesis,
  • 04:04 even though the alternative hypothesis is true.
  • 04:08 In many cases, the difference between the two datasets is so
  • 04:11 great that there is no area of overlap.
  • 04:14 But also, there will be cases where the overlap is so
  • 04:18 great they will almost never reject the null hypothesis.
  • 04:22 Significance and power are determined by the degree of overlap between
  • 04:27 the datasets, when the null hypothesis is true and the alternative is true.
  • 04:33 And by the determination of the alpha risk we're willing to take.
  • 04:36 Notice, you don't set the beta risk value or the power.
  • 04:40 Rather, it comes from the difference in the datasets and the alpha risk.
  • 04:46 Power and significance are measures that we use to discuss the effectiveness of
  • 04:51 the hypothesis test to discriminate between the null hypothesis and
  • 04:55 the alternative hypothesis.

Lesson notes are only available for subscribers.

PMI, PMP, CAPM and PMBOK are registered marks of the Project Management Institute, Inc.

Gift this course