Narrative label audit: coding guide (P5, step C)
This guide is for the two coders and the adjudicator. It holds no narrative text.
Record lookup: Iowa CrashData DB. Narrative is table znar, field NARRATIVE, key CRASH_KEY. Officer fields are table zpve (same key); do not open or copy them.
Coding happens in the web labeling tool on aiws. The tool reads ~/scratch/cellstate/r21/audit/sample.csv and fetches each narrative on aiws. Narrative text never leaves aiws.
1. Why we audit
The paper uses ten crash-type labels built from the police narrative and coded crash fields.
The reviewer asked how often the labels are right. We need precision (of the crashes a rule tags, how many truly fit) and recall (of the crashes that truly fit, how many the rule tags).
The machine labels are hidden from you. You code from the report alone.
2. Files on aiws
| File | Who opens it | What it holds |
sample.csv | labeling tool, analyst | row, crash_key, year, county, stratum, weight, machine labels L01-L10. No narrative text. The tool hides stratum and machine labels from coders |
audit_key.csv | analyst only | stratum, weight, machine labels and negation flags per row. Coders must not open it |
a01_sample.py | analyst | builds the sample (seed 20261101) |
a02_score.py | analyst | scores the finished sheets |
Each row shows audit_id, CRASH_KEY, the officer-coded manner of collision, the officer-coded traffic control and the narrative.
Officer names and badge numbers are masked as [OFFICER] and [ID]. If you still see a personal name, note the row in notes. Do not copy the name.
3. How to code a row
- Read the whole narrative once.
- For each label column, enter 1 if the label applies and 0 if it does not. Leave no blank cell.
- Use the coded manner and traffic control as given. Do not second-guess them. The audit tests the narrative part of each rule.
- Code what the narrative states. Do not infer an event that is not described.
- Fill the three question columns (section 5), your
coder_idand anynotes. - Code alone. Do not look at the other coder's sheet.
A crash can carry several labels. Code every label that applies.
4. Label dictionary
"Manner" means the officer-coded manner of collision (r6 mapping). "Signal" means any unit coded traffic signals.
The machine rule column states exactly what the code does (r7/b_narr.py regexes, r8/a_labels.py rules). The coder rule column states what you judge.
| Label | Paper name | Machine rule | Coder rule: code 1 when |
| L01_queue_rear | Queue rear-end | manner rear-end AND narrative matches queue wording (slowed, stopped or backed-up traffic ahead, stop and go, congestion, queue, sudden stop or brake, slammed brakes, line of traffic, stopped at a red light). A match is dropped if a negation word sits in the 40 characters before it | rear-end, and the narrative says the struck vehicle was slowing, stopped or in a line of traffic |
| L02_sudden_rear | Rear-end without queue wording | manner rear-end AND no queue wording (includes rear-ends with no narrative) | rear-end, and the narrative gives no slowing, stopped or queued traffic ahead |
| L03_sigchange_angle | Signal-change angle | manner broadside AND signal AND narrative mentions yellow or amber, or "light/signal changed/turned", or "turned green/yellow/red" | broadside at a signal, and the narrative says the signal was changing or yellow when a driver entered |
| L04_red_angle | Red-running angle | manner broadside AND signal AND (ran-signal contributing code OR red-light wording) AND no yellow/amber wording | broadside at a signal, and the narrative or code says a driver entered on red |
| L05_pl_yellow | Permissive left on yellow | (manner oncoming left OR failed-to-yield-left code with opposite directions) AND signal AND yellow/amber wording | a left-turn versus opposing crash at a signal, and the narrative mentions yellow |
| L06_left_opp_other | Other left versus opposing | same left-turn condition AND NOT (signal AND yellow wording) | a left-turn versus opposing crash that is not L05 |
| L07_pulled_out | Pulled out | (stop or yield control, OR driveway road type, OR signature S16 to S18) AND not signal AND (pulled-out or failed-to-yield wording OR failed-to-yield code 40, 41, 44, 47) | a driver entered from a stop, yield or driveway into a vehicle with priority |
| L08_did_not_see | Did not see (reported) | narrative matches "did not / could not / failed to / unable to see, notice or observe", "looked but", "blind spot". No negation filter | a driver is reported as saying or being found not to have seen the other party |
| L09_lc_sideswipe | Lane-change sideswipe | manner sideswipe same direction AND (lane change or merge wording OR lane-change code 9 OR lane-change contributing group) | same-direction sideswipe during a lane change or merge |
| L10_secondary | Secondary crash | narrative secondary wording (secondary, prior/earlier/another crash, traffic backed up from a crash, rubbernecking) OR SecondaryCrash flag OR traffic-backup circumstance OR signature S05 | the crash happened because of, or in the queue from, an earlier incident |
Notes on the rules:
- L08 reports a statement. Code 1 for "driver stated she did not see the car". Do not code a perceptual failure that nobody reported.
- L03 does not prove the signal change caused the crash. Code 1 only when the change is described at entry, not merely mentioned.
- The paper calls L02 "rear-end without queue wording". It does not assume a sudden stop. Section 5 asks for positive sudden-stop evidence separately.
5. Question columns
| Column | Code 1 when |
| Q_queue_wording | the narrative describes slowed, stopped or queued traffic ahead, in any wording |
| Q_sudden_stop_positive | the narrative positively says the lead vehicle stopped or braked suddenly or unexpectedly (not just "stopped") |
| Q_unclear | the narrative is too short, garbled or contradictory to code any label with confidence |
When Q_unclear is 1, still code each label with your best reading, and explain in notes.
6. Overlap and precedence
- Machine labels are not exclusive. Known pairs: L01 or L02 with L08 and L10; L03 with L04 (620 in all Polk and Linn records, 231 in the analysed set, because L03 accepts "light changed" while L04 only rejects yellow wording); L07 with L06 and L08.
- Within rear-ends, L01 and L02 are complements. Within left-turn versus opposing crashes, L05 and L06 are complements. Code each label on its own merits, so both can end up 0.
- The model's angle outcome adds L03 and L04, so a crash with both counts twice (212.5 direction-weighted crashes on modelled units).
- The crash-type signatures in r6 use first match (S05 secondary before manner). The labels do not.
7. Negation
The machine drops a queue, yellow or red-light match when one of these words sits in the 40 characters before it, inside the same clause: not, no, never, wasn't, weren't, denied, deny, didn't, did not, without, nor.
This can misfire both ways. "No brake lights, traffic was stopped" may drop a true queue. "Did not stop for the red light" may drop a true red-running.
Code what the sentence means. The NEG stratum oversamples these cases.
8. Sample design
The frame is the 21,181 analysed Polk and Linn crashes with a narrative (21,318 analysed, 137 without narrative).
Strata partition the frame. A crash goes to the first stratum it fits, in this order.
| Stratum | Rule | Frame N_h | Sample n_h | Weight N_h / n_h |
| NEG | a queue, yellow or red-light match fully removed by negation | 227 | 30 | 7.6 |
| MULTI | two or more machine labels | 3,602 | 40 | 90.1 |
| S_L01 to S_L10 | exactly one machine label | 3,627; 3,010; 426; 1,294; 375; 967; 1,048; 859; 1,122; 78 | 20 each | 3.9 to 181.4 |
| U_ANGLE | no label, manner broadside | 932 | 15 | 62.1 |
| U_OTHER | no label, any other manner | 3,614 | 15 | 240.9 |
Total sample 300. The unlabeled strata carry large weights, so recall intervals for rare labels will be wide. Add rows to U_ANGLE and U_OTHER if recall matters for a label.
9. Adjudication
- When both coders finish, the adjudicator copies agreeing codes into the Adjudication sheet and sets
agree_allto 1 for rows with full agreement. - For each disagreement, the adjudicator reads the narrative, applies this guide and enters the final code with a note.
- The adjudicator does not see the machine labels.
- Record rule ambiguities in
notes. They feed the next version of this guide.
10. Scoring
Run a02_score.py on aiws after adjudication. It reports per label:
- Agreement between coders: raw share and Cohen's kappa, unweighted, on the 300 rows.
- Machine precision = TP / (TP + FP) and recall = TP / (TP + FN) against the adjudicated codes, where each count is a sum of row weights (N_h / n_h). F1 as well.
- 95% intervals from a stratified bootstrap (2,000 resamples within strata, seed 7).
- For machine L02, the weighted share with Q_sudden_stop_positive = 1. That share says whether "sudden stop" can stay in the label name.
The pipeline was tested on simulated codes only (a02_scores_sim.json). Those numbers are not results.