# Chapter refresh record: `ethnicity`

Ops checklist for regenerating the fits and figures behind [`book/ethnicity.md`](../../../book/ethnicity.md). Planning and variable decisions live in [`book/ethnicity_plan.md`](../../../book/ethnicity_plan.md); this file records how to rebuild. Board: **Task 167**.

---

## GSS mode refit (Task 175, 2026-11-01)

All GSS fits for this chapter were refit on the interviewer extract (`notebooks/gss_extract_2024_r3a_interviewer.hdf`, the default in `survey_data.GSS_DEFAULT_HDF`): 2021 dropped; 2022 and 2024 in-person and phone respondents only, weights raked to the full sample ([`gss_mode_decision.md`](../../../gss_mode_decision.md)). Chapter updated (b0f20e4); digest regenerated (a38c3ed). Before/after reports, change lists and figure pages: [`notebooks/tables/refresh/2026-10-31_mode/`](../../tables/refresh/2026-10-31_mode/); steps and lessons: [`chapter_update_review_plan.md`](../../../chapter_update_review_plan.md). Series with no GSS data after 2018 refit within sampling noise and their numbers were left as they were.

## Chapter metadata

| Field | Value |
|-------|-------|
| **Record id** | `ethnicity` |
| **Scope** | Variables in [`ethnicity_plan.md`](../../../book/ethnicity_plan.md) § Variables to process, plus the ANES and Pew additions from the targeted inventory |
| **GSS extract** | `notebooks/gss_extract_2024_r3a.hdf` (rebuilt 2026-10-13 to add `workhsp_rel`, `intlhsp_rel`, `wlthhsp_rel`) |
| **ANES extract** | `notebooks/anes_extract_anes_timeseries_cdf_stata_20260205.hdf` (rebuilt 2026-10-13: single-year items from [`anes_extract/timeseries.py`](../../anes_extract/timeseries.py) and derived `_rel` columns) |
| **Pew extract** | `notebooks/pew_extract.hdf` (rebuilt 2026-10-13 to add `pew_discrim_hisp`) |
| **Metadata** | [`variables.yaml`](../../variable_metadata/variables.yaml), group `ethnicity` |
| **EDA** | [`notebooks/eda_ethnicity.md`](../../eda_ethnicity.md) |
| **Related tasks** | **167** (chapter), **100** (stereotypes), **165** (`VCF0217` as a mean), **131** (nationality) |

**Production settings (Unity):** tune 4000, draws 4000, `target_accept` 0.95, 4 chains ([`cw_env.sh`](../../../slurm/cw_env.sh)). Retries for failed gates use draws 12000 with [`refit_array.sh`](../../../slurm/refit_array.sh).

**Raw data the extracts need** (gitignored): ANES 2000, 2012, 2016, 2020 and 2024 single-year Stata files in `data/raw/anes/`; Pew ATP Waves 24.5, 53, 84, 118, 143, 167 under `data/raw/pew/atp/`.

---

## Fits

GSS and ANES on Unity with [`survey_pipeline_array.sh`](../../../slurm/survey_pipeline_array.sh) (time model, model10, cohort trajectories). One list = one extract.

| Group | List (`slurm/`) | Extract | Coding |
|-------|-----------------|---------|--------|
| Social distance, stereotypes (GSS) | `variables_ethnicity_gss.txt` | GSS | default |
| Intermarriage, opposition | `variables_ethnicity_gss_oppose.txt` | GSS | `oppose` [4, 5] (`CW_CODING_TAG=oppose`) |
| Stereotypes, discrimination, influence (ANES) | `variables_ethnicity_anes.txt` | ANES | default |
| Influence in politics, all groups (Blacks `VCF9275`, Whites, Hispanics, Asians) | `variables_influence_anes.txt` | ANES | default (`VCF9275` too little; others too much); `CW_FORCE_RUN=1` after the 2026-10-13 extract change |
| Influence, too little | `variables_influence_anes_toolittle.txt` | ANES | `toolittle` [3] |

```bash
sbatch --array=0-4%4 slurm/survey_pipeline_array.sh slurm/variables_ethnicity_gss.txt gss_extract_2024_r3a.hdf
sbatch --array=0-1%4 --export=ALL,CW_CODING_TAG=oppose slurm/survey_pipeline_array.sh slurm/variables_ethnicity_gss_oppose.txt gss_extract_2024_r3a.hdf
sbatch --array=0-7%4 slurm/survey_pipeline_array.sh slurm/variables_ethnicity_anes.txt anes_extract_anes_timeseries_cdf_stata_20260205.hdf
```

**Pew** (time model only; no birth year in ATP files), local:

```bash
cd notebooks && python ../scripts/run_time_model.py --variable pew_discrim_hisp --extract pew_extract.hdf --draws 2000 --tune 2000 --target-accept 0.95
```

**Not in these lists:** `VCF0217` (fit under Task 97 as percent cold; refit after the Task 165 decision on a mean rating). `VCF0227` stays in the race chapter.

---

## Sampling log

| Variable | Run | Result |
|----------|-----|--------|
| `pew_discrim_hisp` | Local, time model, 2000/2000 @ 0.95 (2026-10-13) | 3 divergences; refit @ 0.99: passes (R̂ 1.012, ESS 632/701). 34.5% (2017) → 31.6% (2025), low 29.7% (2023) |
| `discrim_asian` | Local scout, 1000/1000 @ 0.95 (2026-10-13) | Pipeline runs end to end on 3 waves. Time model passes; model10 1 divergence (scout settings). 13.1% (2016) → 22.0% (2024); averaged split period 85%, cohort 15% |
| All three lists | Unity production, 2026-10-13 | 15 of 15 fits ran (`marhisp` resubmitted once: numba cache "stale file handle" on the shared filesystem, not a model problem). Gates (`converged`: R̂, ESS) pass for all but two; divergences 0.1–1% throughout, as for `marblk` |
| `violent_asian_rel` model10, `influence_asian` time model | Refit at draws 12000 ([`refit_ethnicity.txt`](../../../slurm/refit_ethnicity.txt)) | Both pass (R̂ 1.001, ESS > 4000) |

| Influence, all groups (`variables_influence_anes*.txt`) | Unity, 2026-10-13, after dropping 2000 (different question) and adding Whites and `VCF9275` 2024 | 13 of 14 pass; `influence_white` time model refit at draws 12000 ([`refit_influence.txt`](../../../slurm/refit_influence.txt)): passes. Too little, first year → 2024: Blacks 35.0 → 51.1 (2008), Hispanics 36.8 → 51.8 (2012), Asians 45.5 → 49.6 (2016), Whites 8.8 → 4.3 (2012) |
| `VCF0217` as a mean (Task 165), national and by `race` 1, 2, 3; `cold` coding | Unity, 2026-10-13 | National time model, `race` 1 model10 and `race` 3 time model refit at draws 12000, target accept 0.99 ([`refit_thermometers_default.txt`](../../../slurm/refit_thermometers_default.txt)): all pass. Mean rating 55.7 (1976) → 70.0 degrees (2024); averaged split +14.2 = cohort +1.6, period +12.5. By respondent race, 1976 → 2024: White 54.2 → 68.8, Black 62.8 → 69.7, Other 70.9 → 73.7 |

| Short series, 1990–2014 (`variables_ethnicity_gss_short.txt`: 1990–2000 group ratings `_rel` for Asian Americans, Hispanics (violent), Jews and Blacks (violent); `meltpot`, `meltpot1`, `mincult`, `ethclose`) | Unity, 2026-10-14 | 14 of 14 ran; all 28 fits pass at production settings |

**Not fit (published or weighted points, no model):** Gallup group relations and treatment, 2001–2025 (`scripts/parse_gallup_group_tables.py` → `data/raw/gallup_group_relations/gallup_group_tables_2001_2025.csv`; figures `scripts/plot_gallup_group_relations.py --respondents All White Black Hispanic`). Pew short items (`python pew_points.py` in `notebooks/` → `tables/pew/points/*.csv`).

**Artifact folder:** derived items without a `VCF` prefix (`violent_hisp_rel`, `violent_asian_rel`) resolve to `idata/gss/`, `tables/gss/`, `figs/gss/` although the data are ANES (`survey_data.infer_data_source_for_variable`). Readers and writers agree; look there.

---

## When new data arrive

- **GSS:** rebuild the extract (`make_gss_extract.py`), rerun both GSS lists.
- **ANES 2028:** add the year to `TIME_SERIES_FILES` / `CASE_ID_COLUMNS` and its source variables to `SUPPLEMENT_ITEMS` in `timeseries.py`; when the CDF includes these items, drop the supplement entries (the builder raises if both exist).
- **Pew:** download the new wave by hand, run `scripts/ingest_pew_downloads.py`, rebuild `pew_extract.py` (it finds the item by stem; check that `DISCRRIM_HISP` kept its name), refit.