What are Significance Tested Filters [Subset Filters]?

User Journey: Subscriber > Project > Survey > Analyse V2 > Settings > Sig-Tested Cross Filters (%)

User Story: As an analysis user, when I've added a cross filter to my charts & tables, then I want to see if the cross filter percentages are significantly higher (green) or lower (red) compared to the Total (or "All") filter percentages.

For sig-testing...

We don't sig-test choices against each other
We also don’t sig-test ranks against each other.
Neither do we sig-test statements against each other
Nor do we sig-test loops against each other
Instead, we only sig-test filters against each other

Subset Sig-Test

Will determine if two overlapping percents are significantly different from each other. This test requires Filter 1 to be a subset of Filter 2. Or Filter 2 is the total. For Example, Filter 1 is all males and Filter 2 is the entire population (both males and females).

Q: Is Subset Filter % significantly different from Total Filter %?

1: Filter 1: Subset n / Base = XX%

2: Filter 2: Total n / Base = YY%

3: Variance: ( (Subset n + Total n) / (Subset Base + Total Base) * (1- (Subset n + Total n) / (Subset Base + Total Base) ) ) / (1 - 1/ (Subset Base + Total Base) )

4: T-Value: (ABS (Subset XX% - Total YY%) ) / SQRT (Variance * (1 / Subset Base - 1 / Total Base) )

5: P-Value: 2 * ( 1 - NORMSDIST ( ABS ( T-Value ) ) )

6: Significance Level = 1 - P-Value

7: Confidence Level = (80%, 90%, 95%)

A: If... Significance Level > Confidence Level, Then... Significantly Different = TRUE AND

If... Filter XX% < Total YY%, Then... highlight Red OR
If... Filter XX% > Total YY%, Then... highlight Green

Example 1

Q: Is 18-40yrs Filter significantly different from All Filter for the choice: "No strong feelings"?

1: Filter 1: 18-40yrs n=46 / 393= 12%

2: Filter 2: All n =150 / 1003 = 15%

3: Variance = ( (46 + 150) / (393 + 1003) * (1 - (46 + 150) / (393 + 1003) ) ) / (1 - 1/ (393 + 1003) ) = 0.12

4: T-Value = (ABS (12% - 15%) ) / SQRT (0.12 * (1 / 393 - 1 / 1003) ) = 2.377

5: P-Value = 2 * ( 1 - NORMSDIST ( ABS ( 2.377 ) ) ) = 1.74%

6: Significance Level = 1 - 1.74% = 98.26%

7: Confidence Level = (80%, 90%, 95%) = 95%

A: Filter 1: 18-40yrs (12%) is significantly less [RED] than Filter 2: All (15%) for the choice: "No strong feelings"

Example 2: Let's do that again!

Q: Is 18-40yrs Filter % significantly different from All Filter % for the choice: "A little passionate"?

1: Filter 1: 18-40yrs n=238 / 393 = 61%

2: Filter 2: All n=566 / 1003 = 56%

3: Variance = ( (238 + 566) / (393 + 1003) * (1 - (238 + 566) / (393 + 1003) ) ) / (1 - 1/ (393 + 1003) ) = 0.24

4: T-Value = (ABS (61% - 56%) ) / SQRT (0.24 * (1 / 393 - 1 / 1003) ) = 2.123

5: P-Value = 2 * ( 1 - NORMSDIST ( ABS ( 2.123 ) ) ) = 3.37%

6: Significance Level = 1 - 3.37% = 96.93%

7: Confidence Level = (80%, 90%, 95%)

A: Filter 1: 18-40yrs (61%) is significantly more [GREEN] than Filter 2: All (56%) for the choice: "A little passionate"

Example 3: And again!

Q: Is 41-60yrs Filter % significantly different from All Filter % for the choice: "No strong feelings"?

1: Filter 1: 41-60yrs n=49 / 393= 14%

2: Filter 2: All n=150 / 1003 = 15%

3: Variance = ( (49 + 150) / (393 + 1003) * (1 - (49 + 150) / (393 + 1003) ) ) / (1 - 1/ (393 + 1003) ) = 0.13

4: T-Value = (ABS (14% - 15%) ) / SQRT (0.13 * (1 / 393 - 1 / 1003) ) = 0.458

5: P-Value = 2 * ( 1 - NORMSDIST ( ABS ( 0.458 ) ) ) = 64.67%

6: Significance Level = 1 - 64.67% = 35.33%

7: Confidence Level = (80%, 90%, 95%) = 95%

A: Filter 1: 41-60yrs (14%) is NOT significantly different [NO COLOUR] than Filter 2: All (15%) for the choice: "No strong feelings".

Example 4: Last 1!!

Q: Is 41-60yrs Filter % significantly different from All Filter % for the choice: "A little passionate"?

1: Filter 1: 41-60yrs n=198 / 393 = 58%

2: Filter 2: All n=566 / 1003 = 56%

3: Variance = ( (198 + 566) / (393 + 1003) * (1 - (198 + 566) / (393 + 1003) ) ) / (1 - 1/ (393+ 1003) ) = 0.25

4: T-Value = (ABS (58% - 56%) ) / SQRT (0.25 * (1 / 393 - 1 / 1003) ) = 0.520

5: P-Value = 2 * ( 1 - NORMSDIST ( ABS ( 0.520 ) ) ) = 60.27%

6: Significance Level = 1 - 60.27% = 39.73%

7: Confidence Level = (80%, 90%, 95%) = 95%

A: Filter 1: 41-60yrs (58%) is NOT significantly different [NO COLOUR] than Filter 2: All (56%) for the choice: "A little passionate".

Finally...

Sig testing should be applied in 2 cases:

1: When...

Columns = Filters AND
Values = Column % (as above)

2: When...

Rows = Filters AND
Values = Row % (as below)

Table:

Stack Bar Chart

Group Bar Chart