Interactive regressions & specification curve

8,100 regressions across the full grid of defensible analytic choices

// Load specs grid (CSV — most reliable cross-runtime parser)
specs = {
  const url = "../data/specs_grid.csv"
  console.info("Fetching", url);
  const resp = await fetch(url);
  if (!resp.ok) throw new Error(`Failed to load ${url}: HTTP ${resp.status}`);
  const txt = await resp.text();
  console.info("CSV length:", txt.length, "chars");
  // Parse CSV manually (fast, no dep): assume comma separator, no embedded commas
  const lines = txt.split(/\r?\n/).filter(l => l.length > 0);
  const cols = lines.shift().split(",");
  const numeric = new Set(["coef","se","t","p","n","r2_w"]);
  const out = [];
  for (const line of lines) {
    const parts = line.split(",");
    const row = {};
    for (let i = 0; i < cols.length; i++) {
      const v = parts[i];
      row[cols[i]] = numeric.has(cols[i]) ? (v === "NA" ? null : Number(v)) : v;
    }
    out.push(row);
  }
  console.info("Parsed", out.length, "specs from CSV");
  return out;
}

No server, no compute. All 8,100 regression coefficients were pre-computed offline and ship as a small CSV. Your filter choices recompute summary statistics in your browser — instant, repeatable, offline-capable. Inspired by Simonsohn et al. (2020) and Menkveld et al. (2024).

What is being estimated?

For each combination of analytic choices below, we estimate a panel regression of the form

\[ \text{lib}^{(d)}_{p,c,t} \;=\; \beta\,\text{pop}_{p,c,t} \;+\; \mu_c \;+\; \mu_t \;+\; \varepsilon_{p,c,t} \]

where the observation is one party \(p\) × country \(c\) × election year \(t\) manifesto. \(\text{pop}_{p,c,t}\) is the populism-overall score for that manifesto (averaged across LLMs and chunks per the selected aggregation), and \(\text{lib}^{(d)}_{p,c,t}\) is the score on the chosen liberalism dimension \(d\). The fixed effects \(\mu_c, \mu_t\) absorb everything country-constant and year-constant; standard errors are cluster-robust at the chosen level. The coefficient of interest is \(\beta\) — the partial association between populism and liberalism, holding country and year fixed.

How the seven forks change \(\beta\)

Fork	What varies	Why it might matter
Model selection	Which LLM(s) produce the scores (Sonnet, GPT-4.1-mini, Gemini, or any combination)	Different models may disagree systematically; restricting to one tests sensitivity to the LLM “voice”
Within-manifesto aggregation	How chunks of one manifesto combine into a single score (confidence-weighted mean, plain mean, median)	Long manifestos get split into chunks; the way we collapse them matters
Between-model aggregation	How three LLM scores merge into a cross-model score (mean, median, leave-worst-out)	Disagreement weighting changes the consensus
Outcome dimension	Which liberalism dimension is the dependent variable	Effects may be stronger for some sub-dimensions (e.g. financial-market) than others
Sample restriction	Which manifestos enter the regression (all years, post-2000, post-2010, Europe-only)	Tests historical and geographic robustness
Fixed effects	Country + year, + party, or country × year interactions	Tighter FE absorb more unobserved heterogeneity
SE cluster	Country, country × year, or party	Conservative clustering matters for inference

Why we run all combinations

A single regression hides the variance across analytic choices. Following Menkveld et al. (2024) “Nonstandard Errors”, we show the distribution of \(\beta\) across the full grid of defensible specifications. If the headline finding (\(\beta < 0\)) holds across thousands of choices, the result is robust to researcher degrees of freedom. The specification curve plot below sorts all selected coefficients; the histogram shows their distribution; the per-fork table decomposes where the variation comes from.

Pick the analytic subfamily

viewof model_set_pick = Inputs.checkbox(
  ["sonnet_only","gpt41_only","gemini_only",
   "sonnet_gpt41","sonnet_gemini","gpt41_gemini","all_three"],
  {value: ["sonnet_only","gpt41_only","gemini_only","sonnet_gpt41","sonnet_gemini","gpt41_gemini","all_three"],
   label: "Model selection (which LLM(s) feed the score)"})

viewof outcome_pick = Inputs.checkbox(
  ["liberalism_overall","lib_political","lib_social","lib_economic","lib_financial_market"],
  {value: ["liberalism_overall"], label: "Outcome dimension"})

viewof within_pick = Inputs.checkbox(
  ["cwm","mean","median"],
  {value: ["cwm","mean","median"], label: "Within-manifesto aggregation (across chunks)"})

viewof between_pick = Inputs.checkbox(
  ["mean","median","leave_worst_out"],
  {value: ["mean","median","leave_worst_out"], label: "Between-model aggregation"})

viewof sample_pick = Inputs.checkbox(
  ["full","post2000","post2010","europe_only"],
  {value: ["full","post2000","post2010","europe_only"], label: "Sample restriction"})

viewof fe_pick = Inputs.checkbox(
  ["ctry_year","ctry_year_p","ctry_x_year"],
  {value: ["ctry_year","ctry_year_p","ctry_x_year"], label: "Fixed effects"})

viewof se_pick = Inputs.checkbox(
  ["country","ctry_year","party"],
  {value: ["country","ctry_year","party"], label: "Standard-error clustering"})

filtered = specs.filter(d =>
  model_set_pick.includes(d.model_set) &&
  outcome_pick.includes(d.outcome) &&
  within_pick.includes(d.within_agg) &&
  between_pick.includes(d.between_agg) &&
  sample_pick.includes(d.sample_key) &&
  fe_pick.includes(d.fe_key) &&
  se_pick.includes(d.se_key)
)

outcomeShort = ({
  liberalism_overall: "Liberalism (overall)",
  lib_political: "Political",
  lib_social: "Social",
  lib_economic: "Economic",
  lib_financial_market: "Financial-market"
})
mainSpecs = {
  const hits = outcome_pick.map(o => {
    const h = specs.find(d =>
      d.model_set   === "all_three"   &&
      d.within_agg  === "cwm"         &&
      d.between_agg === "mean"        &&
      d.outcome     === o             &&
      d.sample_key  === "europe_only" &&
      d.fe_key      === "ctry_year"   &&
      d.se_key      === "country"
    );
    if (h) console.info(`Headline ${o}: β = ${h.coef.toFixed(3)}, p = ${h.p}`);
    return h;
  }).filter(Boolean);
  return hits;
}

Bottom-line for your selection

function stats(arr, getter) {
  const v = arr.map(getter).filter(x => Number.isFinite(x)).sort((a,b)=>a-b)
  if (v.length === 0) return {n: 0}
  const mean = v.reduce((s,x)=>s+x,0)/v.length
  const med = v[Math.floor(v.length/2)]
  const p5  = v[Math.floor(v.length*0.05)]
  const p95 = v[Math.floor(v.length*0.95)]
  const sd  = Math.sqrt(v.reduce((s,x)=>s+(x-mean)*(x-mean),0)/v.length)
  return {n: v.length, mean, median: med, p5, p95, sd}
}
selStats = stats(filtered, d => d.coef)
pSigNeg = filtered.length === 0 ? 0 :
  filtered.filter(d => d.p < 0.05 && d.coef < 0).length / filtered.length
pSignFlip = filtered.length === 0 ? 0 :
  filtered.filter(d => d.coef > 0).length / filtered.length

fmt = (x, d=3) => Number.isFinite(x) ? x.toFixed(d) : "–"
html`<div class="grid">
  <div class="g-col-12 g-col-md-3">
    <div class="hero-stat">
      <span class="num">${selStats.n.toLocaleString()}</span>
      <span class="lbl">specifications</span>
    </div>
  </div>
  <div class="g-col-12 g-col-md-3">
    <div class="hero-stat">
      <span class="num">${fmt(selStats.median)}</span>
      <span class="lbl">median coefficient</span>
    </div>
  </div>
  <div class="g-col-12 g-col-md-3">
    <div class="hero-stat">
      <span class="num">${(pSigNeg*100).toFixed(1)}%</span>
      <span class="lbl">significant negative (p&lt;.05)</span>
    </div>
  </div>
  <div class="g-col-12 g-col-md-3">
    <div class="hero-stat">
      <span class="num">${(pSignFlip*100).toFixed(1)}%</span>
      <span class="lbl">sign-flipped (positive)</span>
    </div>
  </div>
</div>
<p style="margin-top:0.8rem"><strong>5/95% band:</strong>
[${fmt(selStats.p5)}, ${fmt(selStats.p95)}]
&nbsp;·&nbsp;
<strong>SD:</strong> ${fmt(selStats.sd)}</p>`

Specification curve

sorted = filtered.map(d => ({...d})).sort((a,b) => a.coef - b.coef)
  .map((d,i) => ({...d, rank: i}))
// For each mainSpec (one per selected outcome), find the rank in the filtered+sorted view
mainInFilter = mainSpecs.flatMap(ms => {
  const r = sorted.find(d =>
    d.model_set   === ms.model_set   &&
    d.within_agg  === ms.within_agg  &&
    d.between_agg === ms.between_agg &&
    d.outcome     === ms.outcome     &&
    d.sample_key  === ms.sample_key  &&
    d.fe_key      === ms.fe_key      &&
    d.se_key      === ms.se_key);
  return r ? [r] : [];
})
Plot = await import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm")
Plot.plot({
  width: 900, height: 380, marginLeft: 60,
  x: {label: "Specification (sorted by coefficient)"},
  y: {label: "Populism coefficient"},
  marks: [
    Plot.ruleY([0], {stroke: "grey", strokeDasharray: "4 3"}),
    Plot.areaY(sorted, {x: "rank", y1: d => d.coef - 1.96*d.se, y2: d => d.coef + 1.96*d.se, fill: "steelblue", fillOpacity: 0.18}),
    Plot.line(sorted, {x: "rank", y: "coef", stroke: "steelblue", strokeWidth: 1.2}),
    Plot.dot(sorted.filter(d => d.p < 0.05 && d.coef < 0),
      {x: "rank", y: "coef", r: 1.4, fill: "darkgreen", fillOpacity: 0.5}),
    Plot.dot(sorted.filter(d => d.coef > 0),
      {x: "rank", y: "coef", r: 2, fill: "red"}),
    // Headline specifications (one ring per selected outcome) + labels
    Plot.dot(mainInFilter, {x: "rank", y: "coef",
        r: 7, fill: "white", stroke: "#cc0033", strokeWidth: 2.5}),
    Plot.dot(mainInFilter, {x: "rank", y: "coef",
        r: 3, fill: "#cc0033"}),
    Plot.text(mainInFilter, {x: "rank", y: "coef",
        text: d => `${outcomeShort[d.outcome]}: β = ${d.coef.toFixed(3)}`,
        dy: -16, fill: "#cc0033", fontWeight: "bold", fontSize: 10,
        textAnchor: "middle"})
  ]
})

Coefficient distribution

Plot.plot({
  width: 900, height: 280, marginLeft: 60,
  x: {label: "Populism coefficient"},
  y: {label: "Density"},
  marks: [
    Plot.ruleX([0], {stroke: "grey", strokeDasharray: "4 3"}),
    Plot.rectY(filtered, Plot.binX({y: "count"},
      {x: "coef", fill: "steelblue", fillOpacity: 0.6, thresholds: 40})),
    Plot.ruleX([selStats.median], {stroke: "darkblue", strokeWidth: 2}),
    // Headline specs — one vertical red rule + label per selected outcome
    Plot.ruleX(mainInFilter, {x: "coef", stroke: "#cc0033", strokeWidth: 2.5}),
    Plot.text(mainInFilter, {
      x: "coef",
      text: d => `${outcomeShort[d.outcome]}: ${d.coef.toFixed(3)}`,
      dy: -6, fill: "#cc0033", fontWeight: "bold", fontSize: 10,
      textAnchor: "middle", frameAnchor: "top"})
  ]
})

Per-fork breakdown — median coefficient

function group_by(arr, key) {
  const m = new Map()
  for (const d of arr) {
    const k = d[key]
    if (!m.has(k)) m.set(k, [])
    m.get(k).push(d)
  }
  return Array.from(m, ([k,v]) => ({
    key: k, n: v.length,
    median: stats(v, d => d.coef).median,
    pct_sig_neg: v.filter(d => d.p < 0.05 && d.coef < 0).length / v.length
  })).sort((a,b) => a.median - b.median)
}
forks = ["model_set","outcome","within_agg","between_agg","sample_key","fe_key","se_key"]
fork_summary = forks.flatMap(f => group_by(filtered, f).map(g => ({fork: f, ...g})))
Inputs.table(fork_summary, {
  columns: ["fork","key","n","median","pct_sig_neg"],
  format: {median: x => x ? x.toFixed(3) : "–",
           pct_sig_neg: x => (x*100).toFixed(1) + "%"},
  height: 600,
  layout: "auto"
})

How to read this

This is a specification curve in the sense of Simonsohn, Simmons & Nelson (2020) and a multiverse analysis à la Steegen et al. (2016). Following the “non-standard errors” framing of Menkveld et al. (2024), each row of the table above is one defensible analytic choice. The aggregate distribution of the 8,100 coefficients quantifies what the standard errors of a single regression can’t: the variance across the analyst’s decision space.

The headline finding — populism is associated with lower liberalism in European party manifestos — is robust to all 8,100 specifications (0% sign-flipped, 97.6% significant negative). Filter to your preferred subfamily above to see how the distribution moves.

References

Menkveld, A. J., et al. (2024). Nonstandard Errors. Journal of Finance.
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour.
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science.