What is the Subset (%) Significance Test?

You can now tick "Subset Significance Test" in your dashboard's settings (cog icon top left)

This will sig test ALL views in your dashboard where...

Values are set to either Row % or Column %
Filters are in either Rows or Columns

1: If Values = Row %, you'll see if each row's percentages are significantly higher (green) or lower (red) compared to the Total (%) row at the bottom.

2: However if Values = Column %, you'll test if each column's percentages are significanly different from the Total (%) column at the far-right.

This will appear as

green / red cells in tables
white up / down arrows in chart tooltip (for now)

This is also completely compatible with the Margin of Error feature (have both ticked if you want!)

Subset Sig-Test

Will determine if two overlapping percents are significantly different from each other. This test requires Row/Column 1 to be a subset of Row/Column 2. Or Row/Column 2 is the total. For Example, Row/Column 1 is all males and Row/Column 2 is the entire population (both males and females).

Q: Is Subset Row/Column % significantly different from Total Row/Column %?

1: Row/Column 1: Subset n / Base = XX%

2: Row/Column 2: Total n / Base = YY%

3: Variance: ( (Subset n + Total n) / (Subset Base + Total Base) * (1- (Subset n + Total n) / (Subset Base + Total Base) ) ) / (1 - 1/ (Subset Base + Total Base) )

4: T-Value: (ABS (Subset XX% - Total YY%) ) / SQRT (Variance * (1 / Subset Base - 1 / Total Base) )

5: P-Value: 2 * ( 1 - NORMSDIST ( ABS ( T-Value ) ) )

6: Significance Level = 1 - P-Value

7: Confidence Level = (80%, 90%, 95%)

A: If... Significance Level > Confidence Level, Then... Significantly Different = TRUE AND

If... Row/Column % < Total R/C %, Then... highlight Red OR
If... Row/Column % > Total R/C %, Then... highlight Green

Example:

Q: Is 18-40yrs Column % significantly different from Total Column for the choice: "No strong feelings"?

1: Column 1: 18-40yrs n=46 / 393= 12%

2: Column 2: Total n =150 / 1003 = 15%

3: Variance = ( (46 + 150) / (393 + 1003) * (1 - (46 + 150) / (393 + 1003) ) ) / (1 - 1/ (393 + 1003) ) = 0.12

4: T-Value = (ABS (12% - 15%) ) / SQRT (0.12 * (1 / 393 - 1 / 1003) ) = 2.377

5: P-Value = 2 * ( 1 - NORMSDIST ( ABS ( 2.377 ) ) ) = 1.74%

6: Significance Level = 1 - 1.74% = 98.26%

7: Confidence Level = (80%, 90%, 95%) = 95%

A: Column 1: 18-40yrs (12%) is significantly less [RED] than Column 2: Total (15%) for the choice: "No strong feelings"

Compatible Question Types

Sig Testing will work with ANY question which has Values set to either Row % or Column %. So works with Choice, Matrix, Rank as well as Scale, Numeric Scale, NPS, Score, and Hidden Variables but also works with Constant Sum, as long as the Values slot is set to Row % or Column %. Looped questions are also sig-tested.

Sig Testing will only work if the base size of the column / row being compared is less than (not equal to nor greater than) the base size of the total column / row being compared against, i.e. it's a subset of the total.

Sig Testing is best (and therefore only) applied when Filters are in either the Table Rows or Columns.

Here's 1 more example below.

1a: Matrix Default

Because statements are in the rows and choices are in the columns (no filters in either rows or columns), no sig testing is done on the default matrix configuration.

1b: Matrix Crosstab

However if we Target the statement: Lunch and cut by the global age Filters in the Rows, with values set to Row % (Matrix crosstab), we can see that 18-40yrs were sig more likely to add hot sauce 3 to 7 times p/week while 61+yrs were sig less likely to select those same choices.

Note 1: Sig testing when Values = Independent %, e.g. Males vs. Females, (not Subset %, e.g. Males vs. Total) will be developed next.

Note 2: Sig testing when Values = Independent Average will be developed last.