Significance Testing.

Significance testing in market research is a statistical method used to determine if observed differences in survey data or experiments (like A/B testing) are real and actionable, rather than just random, chance occurrences.

It helps marketers confidently decide if a campaign, product, or trend is effective, typically requiring a p-value <0.05 (less than 5% probability of occurring by chance).

Every set of data tables Glow's platform delivers for clients features comprehensive statistical significance testing to help you quickly find out what is important within your data.

E.g. at a 95% confidence level, what is the likelihood that 31% of Male customers (n=146/464) being "Very Satisfied" is statistically less than the 35% "Very Satisfied" for All / Total customers (n=290/825)?

Glow Significance Testing.

In Glow's platform, we test to determine if two overlapping percents are significantly different. Sample 2 must be a subset of Sample 1. Or Sample 1 is the total, e.g. for 'Very satisfied', we can test:

Total: 35% vs Male: 31%
Total: 35% vs Female: 40%
Total: 35% vs. Metro: 34%
Total: 35% vs. Regional: 37%

To SIG test your dashboard:

Go to your dashboard settings (cog icon in dashboard bar, top left)
Tick Subset Significance Test
- The default Confidence Level is set to 95% but you can change this to whatever you want.
Click Apply
Click Save

Significance Testing is completely compatible with Margin of Error. Have both ticked if you want!

When Sig testing is applied, we can instantly see that Male: 31% is highlighted in red in the table and has a down arrow in the chart tooltip as it's statistically lower than the Total, while the Total: 35% is highlighted in grey to show it was significantly different to at least one of the variables in the table.

Conversely, if a subgroup is statistically higher than the Total (e.g. Female: 40%), it's highlighted in green in the table and has an up arrow in the chart tooltip.

Detailed Calculation

The exact calculation used to see if the Total 'Very satisfied' of 35% is statistically different from Males 'Very satisfied' of 31% is below:

Males: 146 / 464 = 31%
Total: 290 / 825 = 35%
Variance: ((146 + 290)/(464 + 825) * (1- (146 + 290)/(464 + 825)))/(1 - 1/ (464 + 825)) = 0.22
T-Value: (ABS (31% - 35%) ) / SQRT (Variance: 0.22 * (1 / 464 - 1 / 825) ) = 2.54
P-Value: 2 * ( 1 - NORMSDIST ( ABS ( T-Value: 2.54 ) ) ) = 1%
Significance Level = 1 - P-Value: 1% = 99%
Selected Confidence Level = 95%
Significance Level: 99% > Confidence Level: 95% = TRUE.

The P-Value in our test above shows there's less than a 1% chance that the ~4% difference is due to random chance. Therefore, at our confidence level of 95% confidence, we can conclude that there's a meaningful difference between Male: 31% and Total: 35% for "Very satisfied".

Why 95% Confidence Level?

95% is just a convention. Hence why it's defaulted in Glow's platform.

But feel free to deviate from 95% as needed. If you want stronger evidence of statistical difference, use 99% confidence level. If you don't care as much about strong evidence, pick a lower confidence value, like 85%.

But take care not to be too strict that you miss out on a potentially important finding, nor too lenient that you find significant differences when there is nothing really there.

Compatible Question Types

If ticked, the Overlapping % / Subset % test is calculated across ALL views with tables and charts in your dashboard where percentages are calculated and cross filters are added to tables, so visible in choice, matrix, & rank questions but not in text, location, or image upload questions.

SIG in Choice

In Choice questions, you'll see Sig-Testing applied when you add cross filters to your table columns.

SIG in Matrix

In Matrix questions, you won't see Sig-Testing applied to the default Matrix with statements in rows and choices in columns. You'll only see Sig Testing when you do a Matrix crosstab by selecting a statement (e.g. 'Free-to-air TV') in your "Target" slot above your table to the left, then adding cross filters (e.g. generation, gender, region, income) to your table rows.

Finally, just note that:

If Values = Columns % (default in Choice questions), every column of data in your table is tested against the 'Total (%)' column at the far right.
If Values = Row % (default in Matrix), every row of data is tested against the 'Total (%)' row at the very bottom.