5. Supplementary Analyses

Overview

This document presents supplementary analyses for the DisInformeter scale development project.

Appendix — S4 COND experiment (migrated from 05-sup-analysis.qmd)

Appendix scope. This appendix lifts the Study 4 treatment-condition (COND) analysis from the legacy supplementary-analyses page, preserved here unchanged in substance as part of the Phase F retirement plan for 05-sup-analysis.qmd. The COND experiment evaluates whether the five FIMI-aligned manipulations (dismiss, distort, distract, dismay, divide) move respondents on DisInformeter outcomes, misinformation perceptions, and civic / political attitudes once country clustering is modelled. It sits in the prediction half of the pipeline because the affected outcomes include FIMI itself.

Study 4 randomised respondents to one of six conditions — control plus five FIMI-aligned treatments (dismiss, distort, distract, dismay, divide) — across 14 origin countries. The supplementary analyses below estimate, for every outcome captured after the manipulation, (a) how much variance lives between countries (intraclass correlation, ICC), and (b) what the marginal effect of each treatment arm is relative to control once that country clustering is modelled. Without accounting for country, the standard errors would be too small and the cross-national heterogeneity (which is substantively interesting) would be invisible.

Design and analytic strategy

We fit, separately for each outcome y, two multilevel models with country as a random intercept:

  • Null model: y ~ 1 + (1 | country)
  • Treatment model: y ~ cond + (1 | country)

cond is a six-level factor with control as the reference category, so each treatment yields one Wald contrast (treatment − control). Outcomes are z-standardised within the analytic sample before fitting, which means fixed-effect estimates can be read as standardised mean differences (close in interpretation to Cohen’s d). Random slopes of cond by country are not fit: with only 14 countries — the lower bound of what lme4 can reliably estimate variance components on — fixed-slope, random-intercept models are the conservative choice; cross-national moderation is examined descriptively via the country-level random intercepts and a sensitivity check that adds country fixed effects.

P-values use Satterthwaite degrees of freedom (lmerTest). Within each outcome family we apply the Benjamini-Hochberg FDR adjustment across the 5 treatment contrasts × k outcomes in that family. The three families are:

  1. DisInformeter scale outcomes — FIMI (8-item), FIMI_SHORT (4-item source-balanced), and the five DisInformeter constructs (GEN, THREAT_LONG, ABAND_LONG, PRAG_LONG, SUPR, SUPCH).
  2. Misinformation perceptions and behaviours — MISINFO_REGUL, MISINFO_FREE, MISINFO_CONF, MISINFO_POL, MISINFO_SRC, MISINFO_FOREIGN, CLICK_REASONS, MISINFO_ACTION, MISINFO_IMPACT, MISINFO_RESP.
  3. Civic and political attitudes — SOC_TRUST, INST_TRUST, POL_PARTIC, EU_SUPPORT, DEM_IMPORT, LIFE_SAT, LR_SCALE, POL_INTEREST, POL_EFF_EXT, POL_EFF_INT, RELIGIOSITY.

Working sample: 8,040 respondents across 15 countries and 6 conditions.

Randomisation check

A successful randomisation should produce cells of roughly equal size in every country × condition combination. We test this with a χ²-style goodness-of-fit comparison against an equal-allocation expectation.

The χ² test compares observed counts to an equal-allocation null. A non-significant result confirms randomisation balance held within each country; a significant result would warrant country-stratified treatment-effect estimates rather than a pooled fit.

Country clustering — null-model ICCs

Treatment effects (random-intercept models)

Effect-size table (FDR-adjusted)

Standardised treatment effects (vs. control) with random country intercept. FDR adjusted within each family of outcomes.
Family Outcome Arm b (z) 95% CI t p p (FDR) Flag
DisInformeter Abandonment dismiss 0.02 [-0.06, 0.09] 0.45 0.656 0.980
distort 0.00 [-0.07, 0.08] 0.09 0.928 0.992
distract 0.02 [-0.05, 0.09] 0.52 0.602 0.980
dismay -0.02 [-0.09, 0.06] -0.42 0.676 0.980
divide 0.00 [-0.07, 0.07] -0.06 0.952 0.992
China admiration dismiss -0.06 [-0.13, 0.02] -1.51 0.132 0.755
distort -0.02 [-0.10, 0.05] -0.67 0.503 0.980
distract 0.00 [-0.07, 0.07] -0.03 0.972 0.992
dismay -0.04 [-0.11, 0.03] -1.04 0.298 0.980
divide -0.05 [-0.12, 0.03] -1.28 0.201 0.893
FIMI (8 items) dismiss 0.06 [-0.01, 0.14] 1.59 0.111 0.739
distort 0.08 [0.01, 0.16] 2.22 0.026 0.210 •
distract 0.13 [0.05, 0.20] 3.39 <0.001 0.028 ↑
dismay 0.00 [-0.08, 0.07] -0.01 0.992 0.992
divide 0.03 [-0.05, 0.10] 0.69 0.491 0.980
FIMI short (4 items) dismiss 0.09 [0.02, 0.17] 2.39 0.017 0.171 •
distort 0.09 [0.02, 0.17] 2.39 0.017 0.171 •
distract 0.11 [0.03, 0.18] 2.78 0.005 0.109 •
dismay 0.02 [-0.06, 0.09] 0.50 0.617 0.980
divide 0.02 [-0.05, 0.10] 0.58 0.564 0.980
General negative affect dismiss -0.01 [-0.09, 0.06] -0.40 0.693 0.980
distort 0.01 [-0.06, 0.09] 0.35 0.727 0.980
distract 0.05 [-0.02, 0.12] 1.39 0.164 0.822
dismay 0.00 [-0.07, 0.07] -0.06 0.951 0.992
divide -0.01 [-0.08, 0.06] -0.34 0.735 0.980
Pragmatism dismiss -0.04 [-0.11, 0.04] -0.91 0.361 0.980
distort 0.01 [-0.06, 0.09] 0.37 0.709 0.980
distract 0.04 [-0.03, 0.12] 1.12 0.262 0.980
dismay -0.03 [-0.10, 0.05] -0.73 0.466 0.980
divide 0.00 [-0.08, 0.07] -0.08 0.932 0.992
Russia admiration dismiss 0.01 [-0.06, 0.08] 0.25 0.799 0.992
distort 0.01 [-0.06, 0.08] 0.40 0.686 0.980
distract 0.02 [-0.05, 0.09] 0.60 0.549 0.980
dismay 0.00 [-0.07, 0.07] -0.04 0.971 0.992
divide -0.02 [-0.09, 0.05] -0.52 0.603 0.980
Threat dismiss 0.00 [-0.07, 0.08] 0.09 0.928 0.992
distort -0.03 [-0.10, 0.04] -0.79 0.427 0.980
distract 0.03 [-0.05, 0.10] 0.72 0.470 0.980
dismay -0.01 [-0.08, 0.07] -0.20 0.845 0.992
divide -0.04 [-0.11, 0.03] -1.05 0.293 0.980
Misinfo perceptions Confidence detecting misinfo dismiss -0.04 [-0.12, 0.03] -1.12 0.263 0.821
distort 0.00 [-0.08, 0.08] 0.00 0.997 0.997
distract 0.01 [-0.07, 0.08] 0.21 0.837 0.930
dismay 0.01 [-0.06, 0.09] 0.30 0.767 0.930
divide -0.05 [-0.12, 0.03] -1.27 0.204 0.801
Foreign-actor topic salience dismiss -0.01 [-0.08, 0.07] -0.17 0.862 0.937
distort 0.01 [-0.06, 0.09] 0.30 0.767 0.930
distract 0.03 [-0.05, 0.10] 0.74 0.458 0.834
dismay -0.02 [-0.09, 0.06] -0.45 0.650 0.930
divide -0.03 [-0.10, 0.05] -0.68 0.494 0.834
Free-speech priority dismiss 0.04 [-0.04, 0.11] 0.96 0.337 0.831
distort 0.05 [-0.03, 0.12] 1.26 0.208 0.801
distract 0.05 [-0.02, 0.13] 1.38 0.167 0.801
dismay 0.05 [-0.03, 0.12] 1.18 0.237 0.821
divide 0.03 [-0.05, 0.10] 0.72 0.472 0.834
Misinfo seen as political dismiss -0.04 [-0.12, 0.03] -1.16 0.246 0.821
distort -0.04 [-0.11, 0.04] -1.00 0.315 0.831
distract -0.03 [-0.10, 0.05] -0.67 0.503 0.834
dismay -0.03 [-0.11, 0.04] -0.87 0.382 0.831
divide -0.07 [-0.15, 0.00] -1.87 0.062 0.801
Misinfo-response actions dismiss -0.01 [-0.08, 0.07] -0.22 0.830 0.930
distort -0.02 [-0.09, 0.06] -0.41 0.684 0.930
distract 0.01 [-0.06, 0.09] 0.39 0.696 0.930
dismay -0.05 [-0.12, 0.03] -1.30 0.192 0.801
divide 0.03 [-0.05, 0.10] 0.72 0.473 0.834
Perceived misinfo creators dismiss -0.02 [-0.10, 0.05] -0.62 0.538 0.841
distort -0.01 [-0.09, 0.06] -0.36 0.720 0.930
distract 0.06 [-0.01, 0.14] 1.65 0.098 0.801
dismay -0.01 [-0.08, 0.07] -0.22 0.829 0.930
divide -0.03 [-0.11, 0.04] -0.93 0.351 0.831
Perceived misinfo impact dismiss -0.06 [-0.13, 0.02] -1.51 0.132 0.801
distort -0.06 [-0.14, 0.01] -1.68 0.092 0.801
distract 0.03 [-0.05, 0.10] 0.66 0.509 0.834
dismay -0.02 [-0.10, 0.05] -0.65 0.517 0.834
divide 0.00 [-0.08, 0.07] -0.12 0.902 0.959
Pro-regulation of misinfo dismiss -0.05 [-0.13, 0.02] -1.33 0.183 0.801
distort -0.02 [-0.09, 0.06] -0.40 0.691 0.930
distract -0.03 [-0.11, 0.04] -0.88 0.378 0.831
dismay -0.09 [-0.16, -0.01] -2.21 0.027 0.801 •
divide -0.05 [-0.13, 0.02] -1.41 0.159 0.801
Reasons for clicking news dismiss -0.01 [-0.09, 0.06] -0.30 0.765 0.930
distort -0.02 [-0.09, 0.06] -0.43 0.667 0.930
distract 0.00 [-0.08, 0.07] -0.06 0.951 0.991
dismay -0.01 [-0.08, 0.07] -0.23 0.818 0.930
divide 0.04 [-0.04, 0.11] 1.03 0.304 0.831
Responsibility for prevention dismiss -0.08 [-0.15, -0.00] -2.04 0.041 0.801 •
distort -0.06 [-0.14, 0.01] -1.65 0.099 0.801
distract 0.00 [-0.08, 0.07] -0.02 0.986 0.997
dismay -0.03 [-0.11, 0.04] -0.90 0.369 0.831
divide -0.03 [-0.11, 0.04] -0.80 0.421 0.834
Civic / political External political efficacy dismiss -0.02 [-0.09, 0.06] -0.44 0.661 0.944
distort 0.01 [-0.07, 0.08] 0.24 0.813 0.944
distract -0.02 [-0.10, 0.05] -0.63 0.527 0.943
dismay -0.01 [-0.08, 0.07] -0.14 0.889 0.944
divide 0.05 [-0.03, 0.12] 1.26 0.207 0.892
Generalised social trust dismiss 0.01 [-0.07, 0.08] 0.16 0.877 0.944
distort 0.05 [-0.02, 0.13] 1.31 0.189 0.892
distract 0.01 [-0.06, 0.09] 0.33 0.743 0.944
dismay 0.04 [-0.04, 0.11] 0.93 0.353 0.892
divide 0.06 [-0.02, 0.13] 1.52 0.129 0.884
Importance of democracy dismiss -0.04 [-0.11, 0.04] -0.97 0.332 0.892
distort -0.11 [-0.18, -0.03] -2.74 0.006 0.341 •
distract 0.00 [-0.08, 0.07] -0.13 0.898 0.944
dismay -0.01 [-0.09, 0.06] -0.37 0.714 0.944
divide 0.01 [-0.07, 0.08] 0.23 0.820 0.944
Institutional trust dismiss -0.06 [-0.13, 0.01] -1.67 0.095 0.870
distort -0.02 [-0.09, 0.05] -0.60 0.548 0.943
distract -0.01 [-0.08, 0.06] -0.26 0.793 0.944
dismay -0.01 [-0.09, 0.06] -0.38 0.706 0.944
divide 0.01 [-0.06, 0.08] 0.23 0.818 0.944
Internal political efficacy dismiss 0.01 [-0.06, 0.09] 0.36 0.718 0.944
distort 0.04 [-0.03, 0.11] 1.08 0.280 0.892
distract 0.01 [-0.06, 0.09] 0.31 0.754 0.944
dismay 0.03 [-0.04, 0.11] 0.84 0.398 0.892
divide 0.04 [-0.03, 0.11] 1.08 0.279 0.892
Left–right self-placement dismiss -0.01 [-0.08, 0.07] -0.16 0.876 0.944
distort -0.04 [-0.12, 0.03] -1.08 0.279 0.892
distract -0.08 [-0.16, -0.01] -2.11 0.035 0.655 •
dismay -0.02 [-0.10, 0.05] -0.62 0.536 0.943
divide 0.00 [-0.08, 0.07] -0.10 0.917 0.944
Life satisfaction dismiss -0.03 [-0.11, 0.04] -0.88 0.381 0.892
distort -0.02 [-0.09, 0.05] -0.53 0.598 0.944
distract -0.03 [-0.10, 0.05] -0.75 0.453 0.923
dismay 0.00 [-0.08, 0.07] -0.07 0.944 0.944
divide -0.01 [-0.08, 0.06] -0.25 0.806 0.944
Political interest dismiss 0.02 [-0.05, 0.10] 0.60 0.548 0.943
distort -0.04 [-0.11, 0.04] -0.92 0.358 0.892
distract -0.04 [-0.11, 0.04] -0.99 0.320 0.892
dismay -0.03 [-0.11, 0.04] -0.83 0.406 0.892
divide -0.04 [-0.11, 0.04] -1.01 0.311 0.892
Political participation dismiss 0.06 [-0.02, 0.14] 1.56 0.119 0.884
distort 0.04 [-0.03, 0.12] 1.14 0.254 0.892
distract 0.04 [-0.04, 0.11] 1.00 0.319 0.892
dismay 0.02 [-0.06, 0.09] 0.42 0.673 0.944
divide 0.07 [-0.01, 0.14] 1.75 0.081 0.870
Religiosity dismiss -0.02 [-0.09, 0.06] -0.46 0.646 0.944
distort 0.03 [-0.05, 0.10] 0.68 0.499 0.943
distract 0.01 [-0.07, 0.08] 0.15 0.880 0.944
dismay 0.03 [-0.04, 0.10] 0.83 0.409 0.892
divide 0.07 [0.00, 0.15] 1.98 0.048 0.655 •
Support for EU enlargement dismiss -0.08 [-0.15, -0.00] -2.05 0.040 0.655 •
distort -0.03 [-0.11, 0.04] -0.80 0.422 0.892
distract 0.04 [-0.03, 0.12] 1.14 0.253 0.892
dismay -0.01 [-0.09, 0.06] -0.32 0.751 0.944
divide 0.00 [-0.07, 0.08] 0.08 0.933 0.944

Forest plots by outcome family

The forest plot is the most legible way to read 5 contrasts × k outcomes at once. Solid points are FDR-significant within the family; faded points are not.

Effect-size heatmap

Summary of the COND experiment

Per-arm summary of standardised effects across all 29 outcomes
Arm Outcomes tested FDR-sig (↑) FDR-sig (↓) Mean &#124;b&#124;
dismiss 29 0 0 0.03
distort 29 0 0 0.03
distract 29 1 0 0.03
dismay 29 0 0 0.02
divide 29 0 0 0.03

The Mean |b| column quantifies how much each manipulation moved the average outcome (in z units), while the directional columns show whether those movements are mostly upward or downward after FDR adjustment. Combined with the forest plots and heatmap, this is the headline picture of the COND experiment: which arms produced systematic shifts, and on which kind of outcome those shifts concentrated.

References