Protobi automatically runs stats tests to identify “statistically significant” differences in the standard chart types:
- Baseline bar charts comparing the current scenario to the baseline scenario
- Crosstabs compare each column to either the Overall column (default) or to each other column (pairwise option)
Current vs Baseline
When you press to query (e.g. "Excellent" in q2) Protobi shows results for just those respondents. The percentages and solid color bars reflect the current scenario, (e.g. only those in "Excellent" health). The thin black outlines reflect the baseline scenario (all respondents).
Protobi shows blue triangles wherever the current scenario is significantly different from baseline. Here, the outward pointing triangle indicates that 49.6% is significantly higher than the baseline of 30.5%. Inward pointing triangles indicate the value for the current scenario is significantly lower than the value for the baseline scenario.
Set current filters as baseline
If you want to make a strict comparison between non-overlapping groups, you can change the baseline scenario. Protobi allows you to set current filters as the base instead of all respondents (initial base).
Press the toolbar button "Set base" to make the baseline scenario equal to the current scenario (e.g. respondents in "Excellent" health).
Now shift+press on "Excellent" to select those respondents who are NOT in "Excellent" health.
We can see above, we're now running a strong comparison between two distinct groups. The groups being those who are NOT in "Excellent" health (current scenario, solid bars) versus those who ARE in "Excellent" health (the baseline scenario, thin black outline).
Crosstabs
Crosstabs in Protobi use either Pairwise or Complement significance testing. Pairwise testing compares each column individually with each other column. Complement testing compares each column with the average of all other columns.
Descriptive statistical analysis (means, frequencies) and tests of differences (chi square, t-tests) within respondent types will be performed. Statistical significance will be set at p <0.05 by default, using 2-tailed tests.
Admins can change crosstab significance tests in Project properties. The default mode is Complement testing.
Complement
In Complement testing, each column is compared with the average of all other columns (excluding itself). Blue cells indicate the value is significantly higher in the specified column compared to all other columns, and grey indicates significantly lower.
Note: If showOverall is set to false on a question, there is no Overall column to compare against. So it falls back to pairwise. If you'd like to compare each column to the Overall distribution, set showOverall to true.
Pairwise
Pairwise testing compares each column with each other individual column. This mode shows detailed superscripts like a traditional crosstab.
Below we see that the percentage of respondents who rated their health as "Excellent" is much higher in column A (Very happy) than columns B, C or D.
P-value
To change the P-value for significance tests open the Project properties dialog. The default value is 0.05.
Note: Protobi limits significance testing to N>=10 to avoid testing when the sample size is too small.
