Reproducibility & methods

How to rebuild every figure and number, the analysis conventions, limitations, and the data-availability statement

The pipeline at a glance

Every artefact in this supplement is produced by a deterministic pipeline that runs from the harmonised, validated scales to the print-ready figures and tables. The validated measurement models themselves are estimated on the academic scale-development site and are not re-estimated here; this supplement consumes their outputs.

flowchart TD
  A["Raw study data<br/>(S1–S5)"] --> B["Harmonised scales<br/>data/wrangled_data/*.parquet"]
  B --> C["Validated SEM / BRMS fits<br/>(academic site; cached)"]
  C --> D["R/policy_report/00_run_all.R<br/>country aggregates → CSV"]
  D --> E["scripts/policy_report/F-*.R, A-*.R, T-*.R<br/>print-ready figures + tables"]
  E --> F["outputs/policy_report/<br/>(committed PNG / PDF / HTML / CSV)"]
  F --> G["This supplement<br/>quarto render full-annotated-code/07-policy-report"]

Figure 1

How to rebuild everything

# 1. Restore the exact R package environment (renv lockfile).
Rscript -e 'renv::restore()'

# 2. Build all country-level aggregate tables (writes outputs/policy_report/tables/*.csv).
Rscript R/policy_report/00_run_all.R

# 3. Build every figure and rendered table (writes outputs/policy_report/figures + tables/rendered).
for f in scripts/policy_report/F-*.R scripts/policy_report/A-*.R scripts/policy_report/T-*.R; do
  Rscript "$f"
done

# 4. Render this supplement to ../disinformeter-policy-report-supplement/ (sibling of the repo).
quarto render full-annotated-code/07-policy-report

The aggregate and figure stages run in well under a minute on a recent laptop, because all heavy SEM / multilevel-Bayesian estimation has already happened upstream. The orchestration entry point is shown below.

Code

R/policy_report/00_run_all.R

#' R/policy_report/00_run_all.R
#'
#' Orchestrates the policy-report analysis pipeline. Each `main_NN()` writes
#' one CSV (or RDS) to `outputs/policy_report/{tables,cache,geometries}/` and
#' runs its own acceptance checks. If any step fails, this script propagates
#' the error so the pipeline exits non-zero.
#'
#' Roadmap reference:
#'   docs/reports/2026-05-21_policy-report-content/03_new-analyses-roadmap.md §11
#'
#' Env vars honoured:
#'   POLICY_REPORT_PAR=1            Opt in to future.apply parallelism for
#'                                  the country bootstrap helpers (steps
#'                                  01..06, 09..13).
#'   POLICY_REPORT_SEM_WORKERS=N    Worker count for the country-stratified
#'                                  SEM (step 08). Default 12. Intentionally
#'                                  distinct from DISINFORMETER_SEM_WORKERS
#'                                  (used by the pooled FIMI prediction
#'                                  repository in 04c) so a parallel render
#'                                  does not collide with this pipeline.
#'
#' Step 07 (construct reference card) and 07b (Short DisInforMeter language
#' stub) implement roadmap §3.1 and §3.2 respectively. Step 07b writes a
#' STUB CSV — the language columns intentionally come out empty pending
#' translator hand-off (see the README block at the top of 07b's source).

local({
  here_root <- here::here()
  setwd(here_root)
})

source(here::here("R", "policy_report", "_helpers.R"))

policy_report_message("===========================================")
policy_report_message("Policy report analysis pipeline — START")
policy_report_message(sprintf("seed = %d, parallel = %s",
                              policy_report_seed(),
                              Sys.getenv("POLICY_REPORT_PAR", unset = "0")))
policy_report_message("===========================================")

ensure_policy_report_dirs()

source(here::here("R", "policy_report", "01_country_means_higherorder.R"))
main_01()

source(here::here("R", "policy_report", "02_country_means_fimi.R"))
main_02()

source(here::here("R", "policy_report", "03_country_means_long.R"))
main_03()

source(here::here("R", "policy_report", "04_s23_first_higher_order_means.R"))
main_04()

source(here::here("R", "policy_report", "05_country_fimi_difference.R"))
main_05()

source(here::here("R", "policy_report", "06_country_item_means.R"))
main_06()

source(here::here("R", "policy_report", "07_construct_reference_card.R"))
main_07()

source(here::here("R", "policy_report", "07b_short_disinformeter_languages.R"))
main_07b()

source(here::here("R", "policy_report", "08_country_stratified_sem.R"))
main_08()

source(here::here("R", "policy_report", "09_eu_pooled_benchmarks.R"))
main_09()

source(here::here("R", "policy_report", "10_sample_composition.R"))
main_10()

source(here::here("R", "policy_report", "11_forest_data.R"))
main_11()

source(here::here("R", "policy_report", "12_anchor_wide.R"))
main_12()

source(here::here("R", "policy_report", "13_country_geometries.R"))
main_13()

source(here::here("R", "policy_report", "14_nomological_network.R"))
main_14()

policy_report_message("===========================================")
policy_report_message("Policy report analysis pipeline — DONE")
policy_report_message("===========================================")

The supplement render is intentionally light: it embeds the committed artefacts and shows each producing script as folded, non-executed code (read live from the repository), and executes only inexpensive CSV reads. It does not re-fit the SEM / BRMS models, so it needs no model caches to render.

Analysis conventions

Table 1: Analysis conventions used throughout the supplement.

Convention	Value
Primary language / environment	R 4.4, packages pinned via `renv`
Reproducibility seed	42 (all bootstraps and resampling)
Confidence intervals	Non-parametric bootstrap (paired for the R − CH contrast)
Measurement estimator	MLR (robust ML); FIML for missing data
Map projection	EPSG:3035 (ETRS89 / LAEA Europe) — the EU-standard projection
Significance / adjustment	α = .05; FDR (Benjamini–Hochberg) for the 174 anchor correlations
Primary FIMI DV	Full FIMI (8 items); Russian / Chinese / Short as source & sensitivity variants

Limitations

Table 2: Current limitations (mirrors the report's Limitations section).

Limitation	What it means for use
All data are cross-sectional.	No causal effects or trends can be identified; relationships are associations. Trend, intervention, and causal claims require repeated waves.
The country set is broad but incomplete.	Several Member States are not yet covered; the 13-country set is a starting point, not full EU coverage.
Metric invariance holds; scalar invariance is weaker.	Relationship and profile comparisons are well supported; absolute latent-mean comparisons carry a caveat.
The source-specific result is bank-bounded.	The Russian vs Chinese asymmetry is specific to the fielded cases; it does not generalise to all actors. Treat it as a monitoring hypothesis.
Survey indicators are one layer.	Combine with platform, incident, and narrative evidence before operational prioritisation.

(Chen, 2007; Cheung & Rensvold, 2002; Shadish et al., 2002)

Next measurement priorities

The next phase should prioritise longitudinal measurement, expanded country coverage, refreshed source-specific FIMI item banks, and integration with DSA Article-40 research workflows. Methodologically: stabilise scalar comparability, evaluate weighting and sampling adjustments, refresh translations, and test whether domain changes predict later changes in FIMI detection.

Minimum caveat language for reuse

Important

If DisInforMeter figures are reused outside this supplement, they should carry three statements:

FIMI figures show detection of confirmed FIMI cases, where higher means better detection.
DisInforMeter domain scores show receptivity-relevant endorsement, where higher means more endorsement.
Country comparisons are population-level signals and do not license individual profiling.

Data availability & ethics

Code. The annotated analysis code is shown inline throughout this supplement and lives in the project repository (R/policy_report/, scripts/policy_report/).
Aggregate data. Every country-level aggregate is downloadable as CSV from Annex tables & downloads.
Participant-level data. Released where consent, anonymisation, and data-protection conditions permit, under an open reuse licence, as a DOI-stamped archive (Wilkinson et al., 2016).
Ethics & purpose. The DisInforMeter is a population-monitoring research instrument. It is not for individual risk-profiling, content moderation, or enforcement. See the boundary of interpretation.

A glossary of technical and policy terms is available on the Annex tables & downloads page.

Funded by the European Union (Grant Agreement No. 101132671). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

References

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Wilkinson, M. D., Dumontier, M., Aalbersberg, Ij. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., Silva Santos, L. B. da, Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., & Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18