"""Is the subway safer per ride? The analysis behind a SparkyData article. py src/build_article.py is-the-subway-safer-per-ride Crime on the subway, set against the number of rides, three ways: - the year to date: the NYPD's CompStat count of major felonies in the transit system from 1 January to 11 October 2026 against the same dates of 2025 (a count; the rides run only to 30 September, so no rate is made from it), and the NYPD's complaints whose premises is the subway, January to May 2026 (the last complete months in the year-to-date file), against the same months of 2025 and 2019; - calendar years 2014-2025: those complaints by offence per million paid subway rides; - the long view: the Mayor's Management Report's major felonies in the transit system, fiscal 1998 to fiscal 2026, per million rides in the same fiscal year, under each year's definitions. Rides are built month by month: up to 2019, each calendar year's official total (NYU Rudin's table of New York City Transit's counts to 2013, the MTA's yearly pages after) spread over its months by 2019's month pattern, which the MTA's own comparable-day figures give (2024's days, without 29 February); January and February 2020 together as 2020's official total less the daily file's March to December; from March 2020, the MTA's daily counts, which come within 0.2% of the official totals for 2021-2024 (checked here). Rides are paid entries, and fare evaders ride without being counted: each month's paid rides are divided by one minus the MTA's unpaid share for its quarter (``nycdata.mta.entries_from_paid``), and the comparisons with 2019 are made on those entries. Counts of recorded crimes carry no sampling margin; a change is named only when the approximate 90% interval of the ratio of two rates, as independent Poisson counts over their rides, excludes no change (``sparkylib.stats.count_change``), conditional on the rides. Every comparison the page names is then run again under three what-ifs (``results["sensitivity"]`` lists which apply to which): the fare-evasion estimates moved a fixed point against the claim (at least each quarterly margin the MTA publishes for the quarters compared; none is published for 2019), 2019's months rebuilt the other way, and late reports as they ran in 2025, lined up with the files' report-date cutoffs. Felony assault is also split by the NYPD's offence code, to set aside the codes that hold the workers the law has added since 2019. Complaints are reports to the police, so they move with where the police are and what they record, not only with crime. Writes results.json, viz.json, charts.json, data.csv and build.json into output/articles/is_the_subway_safer_per_ride/ and site/public/articles/is-the-subway-safer-per-ride/, and copies this script there as analysis.py. Deterministic: no sampling. """ from __future__ import annotations import re import pandas as pd from article_kit import Article, lines, r, strip from nycdata import crime, mta from sparkylib.stats import change_size, count_change, rate_ratio SLUG = "is-the-subway-safer-per-ride" SEED = 20261016 BASE, LAST = 2019, 2025 FIRST_YEAR = 2014 # the complaint files' first year WINDOW = range(1, 6) # January to May: the last complete months of 2026 PRE_LAW = range(1, 9) # January to August: 2022's months before the first widening (in force 25 September) COMPSTAT, COMPSTAT_WEEK = "nypd_compstat_citywide_2026_10_11", "2026-10-11" # the latest week before release EVASION_SHIFT = 0.01 # one point against each claim: at least every published quarterly margin compared #: The offences the story follows, as the complaint files name them. OFFENCES = {"six major": None, "felony assault": "FELONY ASSAULT", "robbery": "ROBBERY", "grand larceny": "GRAND LARCENY"} NAMES = {"six major": "The six major felonies", "felony assault": "Felony assault", "robbery": "Robbery", "grand larceny": "Grand larceny"} #: Felony-assault codes that hold the workers the law has added (police and peace officers; transit and #: other public service employees), set aside to test the comparison with 2019 against the widenings. FA_WORKERS = ("police or peace officer", "other public service employee") THEFT_OF_SERVICES = 478 # the NYPD's code for theft of services (fare evasion falls under it) #: The named comparisons a what-if moves, which the page says beside them: against last year, with 2026's #: unpaid share a point above the MTA's estimate and 2025's a point below, the six together read as a fall. MOVES_UNDER_WHAT_IF = [("window six major vs 2025", "evasion")] def offence_counts(d: pd.DataFrame, year: int, months=range(1, 13)) -> dict: """Complaints of each followed offence in the subway in ``year`` (the given months).""" x = d[(d["year"] == year) & d["m"].dt.month.isin(list(months))] out = {"six major": int(x.loc[x["major"], "n"].sum())} for key, name in OFFENCES.items(): if name: out[key] = int(x.loc[x["ofns_desc"].eq(name), "n"].sum()) out["all felonies"] = int(x.loc[x["law_cat_cd"].eq("FELONY"), "n"].sum()) return out def month_pattern(implied_2019: pd.Series) -> pd.Series: """2019's share of a year's rides in each month, from the MTA's comparable-day figures summed by month (``mta.implied_2019_by_month``; with ``skip_leap_day`` February has 28 days, as 2019's had).""" m = implied_2019.astype(float) return m / m.sum() def paid_rides(annual: pd.Series, daily: pd.Series, pattern: pd.Series) -> pd.Series: """Paid subway rides by (year, month): each year's official total spread by 2019's month pattern up to 2019; January and February 2020 together as 2020's official total less the daily file's March to December, split between the two by 2019's pattern (February with its leap day); the daily file from March 2020.""" out = {} for year in range(int(annual.index.min()), 2020): for month in range(1, 13): out[(year, month)] = float(annual.loc[year]) * float(pattern.loc[month]) later = {(int(y), int(m)): float(n) for (y, m), n in daily.items() if (int(y), int(m)) >= (2020, 3)} rest_2020 = [(2020, m) for m in range(3, 13)] if any(k not in later for k in rest_2020): raise AssertionError("the daily file does not cover March to December 2020") jan_feb = float(annual.loc[2020]) - sum(later[k] for k in rest_2020) weight = {1: float(pattern.loc[1]), 2: float(pattern.loc[2]) * 29 / 28} for month in (1, 2): out[(2020, month)] = jan_feb * weight[month] / (weight[1] + weight[2]) out.update(later) return pd.Series(out).sort_index() def months_of(year: int, months=range(1, 13)) -> list: """The (year, month) keys of the given months of ``year``.""" return [(year, m) for m in months] def fiscal_months(fy: int) -> list: """The (year, month) keys of fiscal year ``fy``, July to June.""" return months_of(fy - 1, range(7, 13)) + months_of(fy, range(1, 7)) def month_index(t: pd.Timestamp) -> int: """A month as a running count (year * 12 + month), for counting months between two dates.""" return int(t.year) * 12 + int(t.month) def quoted_figures(A: Article) -> dict: """Every figure and legal change the prose quotes from a report, found in its saved copy.""" Q = A.quotes() return { "compstat_transit": Q.quote(COMPSTAT, "nypd_compstat_citywide_2026-10-11.pdf", "Transit 50 41 22.0 182 141 29.1 1,697 1,670 1.6"), "mmr_fy2026": Q.quote("nyc_mmr_fy2026_nypd", "mmr_fy2026_nypd.pdf", "felony crime in the transit system decreased approximately two percent, from 2,215 to 2,176"), "mmr_fy1998_definition": Q.quote("nyc_mmr_fy2001_vol1", "mmr_fy2001_vol1.pdf", "Crime in the Transit System Major Felonies"), "transit_workers_2022": Q.quote("ny_a10491_2022", "ny_a10491_2022.pdf", "in relation to assaults upon certain employees of a transit agency or authority"), "transit_workers_2022_station_assistants": Q.quote("ny_a10491_2022", "ny_a10491_2022.pdf", "station customer assistant; person whose"), "transit_workers_2024": Q.quote("ny_s8305c_2024", "ny_s8305c_2024.pdf", "chapter 233 of the laws of 2022, is amended to read as follows"), "retail_worker_felony": Q.quote("ny_s8305c_2024", "ny_s8305c_2024.pdf", "Assault on a retail worker is a class E felony"), "retail_worker_in_force": Q.quote("ny_s8305c_2024", "ny_s8305c_2024.pdf", "one hundred eightieth day"), "rape_definition_2023": Q.quote("ny_a3340_2023", "ny_a3340_2023.pdf", "in relation to sex offenses"), "mmr_fy1998_preliminary": Q.quote("nyc_mmr_fy2001_vol1", "mmr_fy2001_vol1.pdf", "Figures based on preliminary data"), "mmr_transit_arrests": Q.quote("nyc_mmr_fy2026_nypd", "mmr_fy2026_nypd.pdf", "Arrests within the transit system NA 8,705 14,545 25,833 27,465"), "nypd_second_safest": Q.quote("nypd_release_q3_2026", "nypd_release_q3_2026.html", "2026 is the second safest year in the transit system in the last 15 years, excluding the pandemic"), # the history the story tells and the readings it checks (v0.9.6): each from its own saved document "nypd_q3_shares": Q.quote("nypd_release_q3_2026", "nypd_release_q3_2026.html", "Felony assaults and nonviolent grand larcenies are driving the crime in transit, accounting for " "30.7% (499) and 49.3% (803) of all transit crime year to date"), "merger_1995": Q.quote("nypd_transit_bureau_page", "nypd_transit_bureau_page_2026-10-15.html", "In 1995, the NYC Transit Police and the NYC Housing Authority Police, which had been independent " "police departments, merged with the New York City Police Department"), "six_million": Q.quote("nypd_transit_bureau_page", "nypd_transit_bureau_page_2026-10-15.html", "carries as many as six million passengers a day"), "bratton_1990": Q.quote("city_journal_gelinas_subways_2016", "city_journal_gelinas_subways_2016.html", "1990 when Bill Bratton came along and was hired to head up the transit police"), "fifty_felonies": Q.quote("city_journal_gelinas_subways_2016", "city_journal_gelinas_subways_2016.html", "the system used to have 50 felonies every day, and now it has five or six felonies a day"), "ssp_2022_date": Q.quote("nyc_mayor_subway_safety_plan_2022", "nyc_mayor_subway_safety_plan_2022-02-18.html", "February 18, 2022"), "ssp_2022_teams": Q.quote("nyc_mayor_subway_safety_plan_2022", "nyc_mayor_subway_safety_plan_2022-02-18.html", "Deploying up to 30 Joint Response Teams that bring together DHS, the Department of Health and " "Mental Hygiene, NYPD, and community-based providers"), "ssp_2022_rules": Q.quote("nyc_mayor_subway_safety_plan_2022", "nyc_mayor_subway_safety_plan_2022-02-18.html", "Training NYPD officers in our subway system to enforce the MTA and New York City Transit Authority"), "surge_2022_date": Q.quote("nyc_mayor_subway_actions_2022_10", "nyc_mayor_subway_actions_2022-10.html", "October 22, 2022"), "surge_2022": Q.quote("nyc_mayor_subway_actions_2022_10", "nyc_mayor_subway_actions_2022-10.html", "surge officer presence on platforms by approximately 1,200 additional overtime officer shifts each " "day on the subway"), "surge_2022_cameras": Q.quote("nyc_mayor_subway_actions_2022_10", "nyc_mayor_subway_actions_2022-10.html", "the MTA will continue to install cameras in each subway car"), "guard_2024": Q.quote("abc_national_guard_subway_2024_03", "abc_national_guard_subway_2024-03-07.html", "deployment of 750 National Guard members and 250 New York State and MTA police officers into the subway system"), "guard_2024_nypd": Q.quote("abc_national_guard_subway_2024_03", "abc_national_guard_subway_2024-03-07.html", "additional 1,000 NYPD officers ordered into the subway last month to conduct bag checks"), "overnight_2025": Q.quote("ny1_overnight_officers_2025_01", "ny1_overnight_officers_2025-01-16.html", "two officers working overtime will be assigned to each of the 150 trains that traverse the city overnight"), "overnight_2025_750": Q.quote("ny1_overnight_officers_2025_01", "ny1_overnight_officers_2025-01-16.html", "The NYPD will deploy 750 additional officers into the subway system"), "overnight_2025_2500": Q.quote("ny1_overnight_officers_2025_01", "ny1_overnight_officers_2025-01-16.html", "The NYPD has 2,500 officers assigned to transit stations around the city"), "osc_weekend": Q.quote("osc_subway_weekend_ridership_2026_04", "osc_subway_weekend_ridership_2026-04-16.html", "average weekend subway ridership was at 89% of the 2019 level"), "osc_weekday": Q.quote("osc_subway_weekend_ridership_2026_04", "osc_subway_weekend_ridership_2026-04-16.html", "average weekday subway ridership was at 74% of the 2019 level in 2025"), "vc_violence": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "subway violence increased markedly — by about 15% — from 2019 to 2024 and by 96% from 2014 to 2024"), "vc_tripled": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "Felony assaults on the subway have more than tripled since 2009"), "vc_animus": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "driven by noninstrumental violence, meaning motivated by animus rather than by monetary motives"), "vc_homeless": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "Nearly two-thirds of those with the largest number of subway arrests in 2022 and 2023 had a " "documented history of homelessness or mental illness"), "vc_fare": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "fare beating arrests are, in general, not very predictive of future subway violence"), "vc_definition": Q.quote("vital_city_subway_safety_2025_12", "vital_city_subway_safety_2025-12-16.html", "refers to the following offenses: murder, robbery, rape, and felony and misdemeanor assault"), "cr_20pct": Q.quote("city_reporter_vital_city_crime_2026_08", "city_reporter_vital_city_crime_2026-08-31.html", "Per million rides, subway crime is still about 20% higher than it was in 2019"), "cr_h1": Q.quote("city_reporter_vital_city_crime_2026_08", "city_reporter_vital_city_crime_2026-08-31.html", "There were 1,071 major felonies in the first half of the year across nearly 650 million rides"), "ny1_overall": Q.quote("ny1_subway_stabbings_2026_10_14", "ny1_subway_stabbings_2026-10-14.html", "Overall major crime on the subway rose less than 1% through September compared with the same time period last year"), "ny1_fa": Q.quote("ny1_subway_stabbings_2026_10_14", "ny1_subway_stabbings_2026-10-14.html", "But felony assault rose 11%"), } def main(argv: list[str] | None = None) -> int: """Run the analysis and write every file (no options).""" A = Article(SLUG, __file__, seed=SEED) # ---- the inputs: complaints, and rides month by month, paid and all entries d, late = crime.subway_complaints() codes, codes_late = crime.subway_complaints_by_code() cutoff = crime.subway_report_cutoffs() ride = mta.long() sub = ride[ride["Mode"] == "Subway"] daily = sub.groupby([sub["Date"].dt.year, sub["Date"].dt.month])["Count"].sum() last_ride_day = sub["Date"].max() annual = mta.subway_annual().set_index("year")["trips"] implied_2019 = mta.implied_2019_by_month("Subway", 2024, skip_leap_day=True) pattern = month_pattern(implied_2019) paid = paid_rides(annual, daily, pattern) entries = mta.entries_from_paid(paid) last_complaint = d["m"].max() if last_complaint.month < WINDOW.stop - 1: raise AssertionError("the complaint files do not reach the end of the window") def total(series: pd.Series, keys: list) -> float: return float(sum(series[k] for k in keys)) for year in (BASE, 2020): if abs(total(paid, months_of(year)) / annual.loc[year] - 1) > 1e-9: raise AssertionError(f"{year}'s months do not add up to its official total") # the daily file against the official totals, in the years both cover whole check = {str(y): r(100 * (total(paid, months_of(y)) / annual.loc[y] - 1), 2) for y in range(2021, int(annual.index.max()) + 1)} if any(abs(v) > 0.2 for v in check.values()): raise AssertionError(f"the daily file strays from the official totals: {check}") stray = max(abs(total(paid, months_of(y)) / annual.loc[y] - 1) for y in range(2021, int(annual.index.max()) + 1)) A.results["rides"] = {"daily_vs_official_pct": check, "daily_vs_official_max_pct": r(100 * stray, 2), "jan_feb_2020_m": r(total(paid, months_of(2020, (1, 2))) / 1e6, 1), "jan_feb_2019_m": r(total(paid, months_of(BASE, (1, 2))) / 1e6, 1), "official_latest_year": int(annual.index.max())} A.printed("rides.daily_vs_official_max_pct", f"{100 * stray:.1f}%") # ---- the year to date: CompStat's transit row, a count over the same dates of both years cs = crime.compstat(week=COMPSTAT_WEEK) tr = cs["rows"]["Transit"] nine = range(1, last_ride_day.month + (1 if last_ride_day.is_month_end else 0)) rides_now, rides_before = total(paid, months_of(LAST + 1, nine)), total(paid, months_of(LAST, nine)) A.results["now"] = {"week_end": cs["week_end"], "period": "1 January to the week's last day, in both years", "transit": count_change(tr["ytd"], tr["ytd_prior"]), "rides_through": last_ride_day.strftime("%Y-%m-%d"), "rides_change_pct": r(100 * (rides_now / rides_before - 1))} if A.results["now"]["transit"]["direction"] != "unclear": raise AssertionError("the prose says the year-to-date count is level with last year") A.printed("now.transit.now", fmt="int") A.printed("now.transit.before", fmt="int") A.printed("now.rides_change_pct", f"{100 * (rides_now / rides_before - 1):.0f}%") A.latest(cs["week_end"], "NYPD CompStat citywide report, major felonies in the transit system, year to date") A.results["quotes"] = quoted_figures(A) # ---- January to May: 2026 against 2025 and 2019, per paid ride and per entry win = {y: offence_counts(d, y, WINDOW) for y in (BASE, LAST, LAST + 1)} w_paid = {y: total(paid, months_of(y, WINDOW)) for y in win} w_entries = {y: total(entries, months_of(y, WINDOW)) for y in win} A.results["window"] = {"months": "January to May", "last_complaint_month": last_complaint.strftime("%Y-%m"), "late_reports": int(late["n"].sum()), "unpaid_share": {str(y): r(100 * (1 - w_paid[y] / w_entries[y])) for y in win}, "paid_m": {str(y): r(w_paid[y] / 1e6, 0) for y in win}, "entries_m": {str(y): r(w_entries[y] / 1e6, 0) for y in win}, "offences": {}} for key in list(OFFENCES) + ["all felonies"]: row = {"counts": {str(y): win[y][key] for y in win}, "per_m_entries": {str(y): r(1e6 * win[y][key] / w_entries[y], 2) for y in win}, "per_m_paid": {str(y): r(1e6 * win[y][key] / w_paid[y], 2) for y in win}} for base in (BASE, LAST): row[f"vs_{base}"] = count_change(win[LAST + 1][key], win[base][key], exposure_now=w_entries[LAST + 1], exposure_before=w_entries[base]) row[f"vs_{base}_paid"] = count_change(win[LAST + 1][key], win[base][key], exposure_now=w_paid[LAST + 1], exposure_before=w_paid[base]) A.results["window"]["offences"][key] = row w = A.results["window"]["offences"] A.claim("window.offences.felony assault.vs_2019", "up") A.claim("window.offences.six major.vs_2019", "up") # grand larceny, January to May against 2019: 13% lower per entry, but its range reaches no change once # the extra month-to-month swing of thefts is allowed for (recheck below), so the page calls it a gap # that could still be chance and leans on the whole of 2025 instead (owner, 13 October 2026, the rule # set on the violence article v1.1) gl_name = OFFENCES["grand larceny"] gl_monthly = d[d["ofns_desc"].eq(gl_name)].groupby([d["year"], d["m"].dt.month])["n"].sum() def gl_months(year: int, months) -> list[int]: return [int(gl_monthly.get((year, mo), 0)) for mo in months] A.recheck("window.offences.grand larceny.vs_2019", gl_months(BASE, WINDOW) + gl_months(LAST + 1, WINDOW), exposure_now=w_entries[LAST + 1], exposure_before=w_entries[BASE]) # all felonies are kept in the results and data.csv but not quoted as a reading of crime: in the subway # they now hold many offences the police find themselves (weapons, drugs), which follow enforcement # (2026-10-11 methodologist review) # the detailed records hold no murder with the subway as its place, so their six are in practice five murders = int(d.loc[d["ofns_desc"].str.startswith("MURDER", na=False), "n"].sum()) A.results["window"]["murders_in_records"] = murders if murders: raise AssertionError("the page says the detailed records hold no murder with the subway as its place") for key, want in (("robbery", "vs_2019"), ("six major", "vs_2025"), ("felony assault", "vs_2025")): if w[key][want]["direction"] != "unclear": raise AssertionError(f"the prose says {key} {want} is about the same") if w["grand larceny"]["vs_2019_paid"]["direction"] != "unclear" or w["robbery"]["vs_2019_paid"]["direction"] != "up": raise AssertionError("the page says on paid rides grand larceny is about the same as 2019 and robbery higher") def window_size(key: str, base: int, end: str = "change", paid_rides_only: bool = False) -> str: """A window change as the page prints it, rounded once from its counts and rides.""" ex = w_paid if paid_rides_only else w_entries node = w[key][f"vs_{base}_paid" if paid_rides_only else f"vs_{base}"] return change_size(node, exposure_now=ex[LAST + 1], exposure_before=ex[base], end=end) # felony assault by the NYPD's offence code: set aside the codes that hold the workers the law has added fa = codes[codes["ky_cd"].eq(106)] def fa_kinds(year: int, months=range(1, 13)) -> pd.Series: x = fa[(fa["year"] == year) & fa["m"].dt.month.isin(list(months))] return x.groupby("kind")["n"].sum() kinds = {y: fa_kinds(y, WINDOW) for y in win} for y in win: if int(kinds[y].sum()) != win[y]["felony assault"]: raise AssertionError(f"the felony assaults by code do not add up in January to May {y}") rest = {y: int(kinds[y].drop(list(FA_WORKERS), errors="ignore").sum()) for y in win} fa_rest = count_change(rest[LAST + 1], rest[BASE], exposure_now=w_entries[LAST + 1], exposure_before=w_entries[BASE]) rest_ratio, rest_lo, rest_hi = rate_ratio(rest[LAST + 1], rest[BASE], w_entries[LAST + 1], w_entries[BASE]) A.results["window"]["felony_assault_by_code"] = { "counts": {k: {str(y): int(kinds[y].get(k, 0)) for y in win} for k in sorted(set().union(*[set(v.index) for v in kinds.values()]))}, "set_aside": list(FA_WORKERS), "without_workers": {"counts": {str(y): rest[y] for y in win}, "vs_2019": fa_rest, "times_2019": r(rest_ratio, 3), "times_2019_lo": r(rest_lo, 3), "times_2019_hi": r(rest_hi, 3)}} A.claim("window.felony_assault_by_code.without_workers.vs_2019", "up") A.printed("window.felony_assault_by_code.without_workers.times_2019", f"{rest_ratio:.1f}") A.printed("window.felony_assault_by_code.without_workers.times_2019_lo", f"{rest_lo:.1f}") A.printed("window.felony_assault_by_code.without_workers.times_2019_hi", f"{rest_hi:.1f}") # before any of the widenings since 2019: January to August 2022 against the same months of 2019, per # entry (the first widening was in force from 25 September 2022); the style guide's test of a methods # break without the years after it (2026-10-11 methodologist review) pl = {y: offence_counts(d, y, PRE_LAW)["felony assault"] for y in (BASE, 2022)} pl_entries = {y: total(entries, months_of(y, PRE_LAW)) for y in (BASE, 2022)} pl_rest = {y: int(fa_kinds(y, PRE_LAW).drop(list(FA_WORKERS), errors="ignore").sum()) for y in (BASE, 2022)} pl_ratio, pl_lo, pl_hi = rate_ratio(pl[2022], pl[BASE], pl_entries[2022], pl_entries[BASE]) plr_ratio, plr_lo, plr_hi = rate_ratio(pl_rest[2022], pl_rest[BASE], pl_entries[2022], pl_entries[BASE]) A.results["window"]["before_the_widenings"] = { "months": "January to August", "years": [BASE, 2022], "counts": {str(y): pl[y] for y in pl}, "entries_m": {str(y): r(pl_entries[y] / 1e6, 1) for y in pl}, "vs_2019": count_change(pl[2022], pl[BASE], exposure_now=pl_entries[2022], exposure_before=pl_entries[BASE]), "times_2019": r(pl_ratio, 3), "times_2019_lo": r(pl_lo, 3), "times_2019_hi": r(pl_hi, 3), "without_workers": {"counts": {str(y): pl_rest[y] for y in pl_rest}, "vs_2019": count_change(pl_rest[2022], pl_rest[BASE], exposure_now=pl_entries[2022], exposure_before=pl_entries[BASE]), "times_2019": r(plr_ratio, 3), "times_2019_lo": r(plr_lo, 3), "times_2019_hi": r(plr_hi, 3)}} A.claim("window.before_the_widenings.vs_2019", "up") A.claim("window.before_the_widenings.without_workers.vs_2019", "up") A.printed("window.before_the_widenings.times_2019", f"{pl_ratio:.1f}") A.printed("window.before_the_widenings.times_2019_lo", f"{pl_lo:.1f}") A.printed("window.before_the_widenings.times_2019_hi", f"{pl_hi:.1f}") # ---- calendar years 2014-2025, per million paid rides years = list(range(FIRST_YEAR, LAST + 1)) cal = {y: offence_counts(d, y) for y in years} y_paid = {y: total(paid, months_of(y)) for y in years} y_entries = {y: total(entries, months_of(y)) for y in (BASE, LAST)} rates = {key: {str(y): r(1e6 * cal[y][key] / y_paid[y], 2) for y in years} for key in OFFENCES} A.results["years"] = {"counts": {str(y): cal[y] for y in years}, "paid_m": {str(y): r(y_paid[y] / 1e6, 0) for y in years}, "per_m_paid": rates, "unpaid_share": {str(y): r(100 * (1 - y_paid[y] / y_entries[y])) for y in (BASE, LAST)}} peak = max(years, key=lambda y: rates["six major"][str(y)]) A.results["years"]["six_major_peak"] = {"year": peak, "rate": rates["six major"][str(peak)]} if peak != 2020: raise AssertionError("the prose says major felonies per ride peaked in 2020, when ridership collapsed") A.printed("years.six_major_peak.rate", f"{1e6 * cal[peak]['six major'] / y_paid[peak]:.2f}") def per_ride_change(key: str, a: int, b: int) -> dict: return count_change(cal[a][key], cal[b][key], exposure_now=y_paid[a], exposure_before=y_paid[b]) fa_peak = max(years, key=lambda y: rates["felony assault"][str(y)]) if fa_peak != 2021: raise AssertionError("the prose says felony assaults per ride peaked in 2021") A.results["years"]["felony_assault_since_2021"] = per_ride_change("felony assault", LAST, 2021) A.results["years"]["felony_assault_vs_2019"] = per_ride_change("felony assault", LAST, BASE) A.claim("years.felony_assault_since_2021", "down") A.claim("years.felony_assault_vs_2019", "up") A.results["years"]["felony_assault_vs_2019_entry"] = count_change( cal[LAST]["felony assault"], cal[BASE]["felony assault"], exposure_now=y_entries[LAST], exposure_before=y_entries[BASE]) A.claim("years.felony_assault_vs_2019_entry", "up") year_kinds = {y: fa_kinds(y) for y in (BASE, LAST)} year_rest = {y: int(year_kinds[y].drop(list(FA_WORKERS), errors="ignore").sum()) for y in (BASE, LAST)} A.results["years"]["felony_assault_by_code"] = { "counts": {k: {str(y): int(year_kinds[y].get(k, 0)) for y in (BASE, LAST)} for k in sorted(set(year_kinds[BASE].index) | set(year_kinds[LAST].index))}, "without_workers_vs_2019_entry": count_change(year_rest[LAST], year_rest[BASE], exposure_now=y_entries[LAST], exposure_before=y_entries[BASE])} A.claim("years.felony_assault_by_code.without_workers_vs_2019_entry", "up") # grand larceny over the whole of 2025 against 2019, per entry: the second footing for "lower", since the # January-to-May range alone nears zero once extra month-to-month swing is allowed for (2026-10-11 review) A.results["years"]["grand_larceny_vs_2019_entry"] = count_change( cal[LAST]["grand larceny"], cal[BASE]["grand larceny"], exposure_now=y_entries[LAST], exposure_before=y_entries[BASE]) A.claim("years.grand_larceny_vs_2019_entry", "down") A.recheck("years.grand_larceny_vs_2019_entry", gl_months(BASE, range(1, 13)) + gl_months(LAST, range(1, 13)), exposure_now=y_entries[LAST], exposure_before=y_entries[BASE]) # the methods print both rechecked ranges, as sizes rounded from the stored ends rob = A.results["robustness"]["changes"] for key in ("window_offences_grand larceny_vs_2019", "years_grand_larceny_vs_2019_entry"): for end in ("lo_pct", "hi_pct"): A.printed(f"robustness.changes.{key}.{end}", f"{abs(rob[key][end]):.0f}%") if rob["window_offences_grand larceny_vs_2019"]["direction"] != "unclear" or rob["years_grand_larceny_vs_2019_entry"]["direction"] != "down": raise AssertionError("the page says January-to-May grand larceny could still be chance and the 2025 fall holds") A.printed("years.grand_larceny_vs_2019_entry.change_pct", change_size(A.results["years"]["grand_larceny_vs_2019_entry"], exposure_now=y_entries[LAST], exposure_before=y_entries[BASE])) for end in ("lo", "hi"): A.printed(f"years.grand_larceny_vs_2019_entry.{end}_pct", change_size(A.results["years"]["grand_larceny_vs_2019_entry"], exposure_now=y_entries[LAST], exposure_before=y_entries[BASE], end=end)) # the shapes the text describes: felony assault rising before 2019, robbery's jump in 2020 A.results["years"]["felony_assault_2019_vs_first"] = per_ride_change("felony assault", BASE, FIRST_YEAR) A.claim("years.felony_assault_2019_vs_first", "up") A.printed("years.felony_assault_2019_vs_first.change_pct", change_size(A.results["years"]["felony_assault_2019_vs_first"], exposure_now=y_paid[BASE], exposure_before=y_paid[FIRST_YEAR])) # policing in the transit system: the Mayor's Management Report's arrests, fiscal 2023 and 2026 m = re.search(r"Arrests within the transit system NA ([\d,]+) [\d,]+ [\d,]+ ([\d,]+)", A.results["quotes"]["mmr_transit_arrests"]["quote"]) if not m: raise AssertionError("the transit arrests are not in their quote") A.results["years"]["transit_arrests"] = {"fiscal_2023": int(m.group(1).replace(",", "")), "fiscal_2026": int(m.group(2).replace(",", ""))} A.printed("years.transit_arrests.fiscal_2023", fmt="int") A.printed("years.transit_arrests.fiscal_2026", fmt="int") A.results["years"]["robbery_2020_vs_2019"] = per_ride_change("robbery", 2020, BASE) A.claim("years.robbery_2020_vs_2019", "up") # felony assault had more than doubled per ride by 2020, before the widenings of the law since 2019 # (more transit workers from September 2022 and July 2024, retail workers from October 2024) A.results["years"]["felony_assault_2020_vs_2019"] = per_ride_change("felony assault", 2020, BASE) A.claim("years.felony_assault_2020_vs_2019", "up") if A.results["years"]["felony_assault_2020_vs_2019"]["lo_pct"] < 100: raise AssertionError("the prose says felony assault per ride had more than doubled by 2020") for y in (2021, LAST): A.printed(f"years.per_m_paid.felony assault.{y}", f"{1e6 * cal[y]['felony assault'] / y_paid[y]:.2f}") # theft of services (the code fare evasion falls under): an enforcement count, whose offence label changed tos = codes[codes["pd_cd"].eq(THEFT_OF_SERVICES)] A.results["years"]["theft_of_services"] = {str(y): int(tos.loc[tos["year"] == y, "n"].sum()) for y in years} for y in (BASE, LAST): A.printed(f"years.theft_of_services.{y}", fmt="int") for y in (LAST, BASE): A.printed(f"years.per_m_paid.six major.{y}", f"{1e6 * cal[y]['six major'] / y_paid[y]:.2f}") # ---- the long view: the Mayor's Management Report, fiscal 1998-2026 mm = crime.transit_major_felonies().set_index("fiscal_year")["n"] fy = {} for y, n in mm.items(): y = int(y) t = total(paid, fiscal_months(y)) how = ("daily file" if y >= 2021 else "annual totals and the daily file" if y == 2020 else "annual totals spread by 2019's month pattern") fy[y] = {"felonies": int(n), "paid_m": r(t / 1e6, 0), "per_m": r(1e6 * n / t, 2), "rides_from": how} first, last = min(fy), max(fy) A.results["fiscal"] = {"first": first, "last": last, "years": {str(y): v for y, v in fy.items()}, "definitions": "each year's: the six offences as the law defined them that year"} fr = A.results["fiscal"] def fiscal_paid(y: int) -> float: return total(paid, fiscal_months(y)) def rate_change(a: int, b: int) -> dict: return count_change(fy[a]["felonies"], fy[b]["felonies"], exposure_now=fiscal_paid(a), exposure_before=fiscal_paid(b)) fr["since_first"] = rate_change(last, first) fr["since_2019"] = rate_change(last, 2019) A.claim("fiscal.since_first", "down") for end in ("lo", "hi"): A.printed(f"fiscal.since_first.{end}_pct", change_size(fr["since_first"], 1, exposure_now=fiscal_paid(last), exposure_before=fiscal_paid(first), end=end)) fe = {y: total(entries, fiscal_months(y)) for y in (2019, last)} fr["unpaid_share"] = {str(y): r(100 * (1 - fiscal_paid(y) / fe[y])) for y in (2019, last)} fr["since_2019_per_entry"] = count_change(fy[last]["felonies"], fy[2019]["felonies"], exposure_now=fe[last], exposure_before=fe[2019]) if fr["since_2019_per_entry"]["direction"] != "unclear": raise AssertionError("the prose says fiscal 2026 is about level with fiscal 2019 per entry") peak_fy = max(fy, key=lambda y: fy[y]["per_m"] if y >= 2020 else 0) fr["pandemic_peak"] = {"year": peak_fy, "per_m": fy[peak_fy]["per_m"]} fr["since_peak"] = rate_change(last, peak_fy) A.claim("fiscal.since_peak", "down") low = min((y for y in fy if y < 2020), key=lambda y: fy[y]["per_m"]) fr["low_before_2020"] = {"year": low, "per_m": fy[low]["per_m"]} if low != 2010: raise AssertionError("the prose says the long fall bottomed out in fiscal 2010") def fiscal_rate(y: int) -> float: return 1e6 * fy[y]["felonies"] / fiscal_paid(y) A.printed("fiscal.low_before_2020.per_m", f"{fiscal_rate(low):.1f}") A.printed("fiscal.pandemic_peak.per_m", f"{fiscal_rate(peak_fy):.1f}") fr["fall_since_first_pct"] = r(-fr["since_first"]["change_pct"], 0) for y in (first, 2019, last): A.printed(f"fiscal.years.{y}.per_m", f"{fiscal_rate(y):.1f}") # felony assault's share of today's six (the offence the law has widened most), in the detailed records fr["felony_assault_share_2025_pct"] = r(100 * cal[LAST]["felony assault"] / cal[LAST]["six major"], 0) A.printed("fiscal.felony_assault_share_2025_pct", f"{100 * cal[LAST]['felony assault'] / cal[LAST]['six major']:.0f}%") rape = d[d["ofns_desc"].eq("RAPE") & d["major"]].groupby("year")["n"].sum() fr["rape_complaints_max_2014_2025"] = int(rape.loc[FIRST_YEAR:LAST].max()) A.printed("fiscal.rape_complaints_max_2014_2025", fmt="int") A.long_view(first, last, "Mayor's Management Report, major felony crime in the transit system, fiscal years, " "per million subway rides", why_shorter=("the Mayor's Management Report prints the transit series in this " "form from fiscal 1998; earlier counts survive only in " "secondary sources with other definitions")) # ---- the window's six together beside the fiscal year: the answer depends on the window A.printed("window.offences.six major.vs_2019.change_pct", window_size("six major", BASE)) A.printed("fiscal.since_2019_per_entry.change_pct", change_size(fr["since_2019_per_entry"], exposure_now=fe[last], exposure_before=fe[2019])) A.printed("window.offences.six major.vs_2025.change_pct", window_size("six major", LAST)) # both counts beside each change the story prints as a percentage (style guide, 15 October 2026, 4): # the reader sees the complaints, and the rides turn them into the rate for path in ("years.felony_assault_2019_vs_first", "years.grand_larceny_vs_2019_entry", "window.offences.six major.vs_2019", "window.offences.six major.vs_2025", "window.offences.grand larceny.vs_2019", "window.offences.felony assault.vs_2025", "fiscal.since_2019_per_entry"): for end in ("before", "now"): A.printed(f"{path}.{end}", fmt="int") # ---- the what-ifs: every comparison the page names, under each one that applies to it def entries_at(shift: float, keys: list) -> float: return total(mta.entries_from_paid(paid, shift=shift), keys) w26, w19, w25 = months_of(LAST + 1, WINDOW), months_of(BASE, WINDOW), months_of(LAST, WINDOW) named = { # name: (count now, count before, months now, months before, claim) "window felony assault vs 2019": (win[LAST + 1]["felony assault"], win[BASE]["felony assault"], w26, w19, "up"), "window felony assault without the workers' codes vs 2019": (rest[LAST + 1], rest[BASE], w26, w19, "up"), "window six major vs 2019": (win[LAST + 1]["six major"], win[BASE]["six major"], w26, w19, "up"), "window grand larceny vs 2019": (win[LAST + 1]["grand larceny"], win[BASE]["grand larceny"], w26, w19, "down"), "window robbery vs 2019": (win[LAST + 1]["robbery"], win[BASE]["robbery"], w26, w19, "level"), "window six major vs 2025": (win[LAST + 1]["six major"], win[LAST]["six major"], w26, w25, "level"), "window felony assault vs 2025": (win[LAST + 1]["felony assault"], win[LAST]["felony assault"], w26, w25, "level"), "year 2025 felony assault vs 2019": (cal[LAST]["felony assault"], cal[BASE]["felony assault"], months_of(LAST), months_of(BASE), "up"), "fiscal 2026 six major vs 2019": (fy[last]["felonies"], fy[2019]["felonies"], fiscal_months(last), fiscal_months(2019), "level"), } want_dir = {"up": "up", "down": "down", "level": "unclear"} def holds(claim: str, got: str) -> bool: return got == want_dir[claim] sens = {name: {"claim": claim} for name, (_, _, _, _, claim) in named.items()} # what-if 1: the evasion estimates a fixed point against the claim (both ways for a level claim) ev = mta.fare_evasion().set_index(["year", "quarter"]) compared = sorted({(y, (m - 1) // 3 + 1) for (_, _, a, b, _) in named.values() for (y, m) in a + b if y >= 2020}) moe = ev.loc[compared, "moe"] if moe.isna().any() or float(moe.max()) > EVASION_SHIFT: raise AssertionError("a published margin in the quarters compared exceeds the shift tested") for name, (now, before, a, b, claim) in named.items(): if a[0][0] < 2020 and b[0][0] < 2020: continue shifts = {"up": [EVASION_SHIFT], "down": [-EVASION_SHIFT], "level": [EVASION_SHIFT, -EVASION_SHIFT]}[claim] runs = [count_change(now, before, exposure_now=entries_at(s, a), exposure_before=entries_at(-s, b)) for s in shifts] sens[name]["evasion"] = [c["direction"] for c in runs] sens[name]["evasion_runs"] = runs # what-if 2: 2019's months rebuilt the first way (the comparable-day sums as they stand, leap day kept), for # the windows; for the fiscal year, every year's total spread by that pattern instead implied_full = mta.implied_2019_by_month("Subway", 2024) old_paid = pd.Series({(BASE, m): float(implied_full.loc[m]) for m in WINDOW}) old_entries = total(mta.entries_from_paid(old_paid), w19) paid_alt = paid_rides(annual, daily, month_pattern(implied_full)) entries_alt = mta.entries_from_paid(paid_alt) for name, (now, before, a, b, claim) in named.items(): if b == w19: c = count_change(now, before, exposure_now=w_entries[LAST + 1], exposure_before=old_entries) elif name.startswith("fiscal"): c = count_change(now, before, exposure_now=total(entries_alt, a), exposure_before=total(entries_alt, b)) else: continue sens[name]["rebuild"] = c["direction"] sens[name]["rebuild_run"] = c # what-if 3: late reports. The year-to-date file holds reports to its cutoff (30 June 2026), so May has had # one more month to be reported, as November 2025 had at the historic file's cutoff (31 December 2025). Each # window month takes the share of felonies reported late for the 2025 month in the same position, measured # in the year-to-date file, and its count is lifted to count / (1 - share), month by month. for k in ("historic", "ytd"): if not cutoff[k].is_month_end: raise AssertionError(f"the {k} file's report cutoff is not a month's end") fel_hist = d[d["law_cat_cd"].eq("FELONY")] fel_late = late[late["law_cat_cd"].eq("FELONY")] def late_share(month: pd.Timestamp) -> float: n_late = int(fel_late.loc[fel_late["m"].eq(month), "n"].sum()) n_on_time = int(fel_hist.loc[fel_hist["m"].eq(month), "n"].sum()) return n_late / (n_late + n_on_time) share_for = {} for m in WINDOW: after = month_index(cutoff["ytd"]) - month_index(pd.Timestamp(year=LAST + 1, month=m, day=1)) h = month_index(cutoff["historic"]) - after like = pd.Timestamp(year=(h - 1) // 12, month=(h - 1) % 12 + 1, day=1) share_for[m] = {"like": like.strftime("%Y-%m"), "share": late_share(like)} month_counts = {m: offence_counts(d, LAST + 1, [m]) for m in WINDOW} def lifted(key: str) -> float: return sum(month_counts[m][key] / (1 - share_for[m]["share"]) for m in WINDOW) rest_months = {m: int(fa_kinds(LAST + 1, [m]).drop(list(FA_WORKERS), errors="ignore").sum()) for m in WINDOW} lift = {key: lifted(key) / win[LAST + 1][key] for key in list(OFFENCES) + ["all felonies"]} lift["without workers"] = sum(rest_months[m] / (1 - share_for[m]["share"]) for m in WINDOW) / rest[LAST + 1] late_2025 = late[late["m"].dt.year.eq(LAST)] for name, (now, before, a, b, claim) in named.items(): if a == w26: key = ("without workers" if "without" in name else "all felonies" if "all felonies" in name else next(k for k in OFFENCES if k in name)) ex_b = w_entries[BASE] if b == w19 else w_entries[LAST] c = count_change(round(now * lift[key]), before, exposure_now=w_entries[LAST + 1], exposure_before=ex_b) elif name.startswith("year"): extra = int(late_2025.loc[late_2025["ofns_desc"].eq("FELONY ASSAULT"), "n"].sum()) c = count_change(now + extra, before, exposure_now=y_entries[LAST], exposure_before=y_entries[BASE]) else: continue sens[name]["late_reports"] = c["direction"] moved = sorted((n, k) for n, s in sens.items() for k in ("evasion", "rebuild", "late_reports") if k in s and not all(holds(s["claim"], g) for g in (s[k] if isinstance(s[k], list) else [s[k]]))) for n, k in moved: sens[n]["holds"] = False sens[n]["moves_under"] = k for n in sens: sens[n].setdefault("holds", True) if moved != sorted(MOVES_UNDER_WHAT_IF): raise AssertionError(f"the comparisons a what-if moves are {moved}, the page says {sorted(MOVES_UNDER_WHAT_IF)}") A.results["sensitivity"] = { "comparisons": sens, "evasion_shift_points": r(100 * EVASION_SHIFT, 1), "evasion_margins": {"max_points": r(100 * float(moe.max()), 1), "min_points": r(100 * float(moe.min()), 1), "quarters": f"{compared[0][0]} Q{compared[0][1]} to {compared[-1][0]} Q{compared[-1][1]}", "level": "not stated by the MTA", "before_2020": "none published; one point assumed"}, "late_reports": {"cutoffs": {k: v.strftime("%Y-%m-%d") for k, v in cutoff.items()}, "shares": {str(m): {"like": s["like"], "share_pct": r(100 * s["share"], 2)} for m, s in share_for.items()}, "december_2025_pct": r(100 * late_share(pd.Timestamp(year=LAST, month=12, day=1)), 1), "window_all_felonies_pct": r(100 * (lift["all felonies"] - 1), 2), "by_offence_pct": {k: r(100 * (v - 1), 2) for k, v in lift.items()}}, "rebuild": {"paid_2019_window_m": r(w_paid[BASE] / 1e6, 0), "paid_2019_window_other_m": r(float(old_paid.sum()) / 1e6, 0)}, } A.printed("sensitivity.evasion_margins.min_points", f"{100 * float(moe.min()):.1f}") A.printed("sensitivity.evasion_margins.max_points", f"{100 * float(moe.max()):.1f}") A.printed("sensitivity.late_reports.december_2025_pct", f"{100 * late_share(pd.Timestamp(year=LAST, month=12, day=1)):.0f}%") A.printed("sensitivity.late_reports.window_all_felonies_pct", f"{100 * (lift['all felonies'] - 1):.1f}%") ge = sens["window grand larceny vs 2019"]["evasion_runs"][0] A.results["window"]["grand_larceny_evasion_shifted"] = ge for end in ("lo", "hi"): A.printed(f"window.grand_larceny_evasion_shifted.{end}_pct", change_size(ge, exposure_now=entries_at(-EVASION_SHIFT, w26), exposure_before=entries_at(EVASION_SHIFT, w19), end=end)) # ---- what the page prints from the window ratio = 1 + w["felony assault"]["vs_2019"]["change_pct"] / 100 exact = (win[LAST + 1]["felony assault"] / w_entries[LAST + 1]) / (win[BASE]["felony assault"] / w_entries[BASE]) A.results["window"]["felony_assault_times_2019"] = r(exact, 3) A.printed("window.felony_assault_times_2019", f"{exact:.1f}") if abs(exact - ratio) > 0.001: raise AssertionError("the felony-assault ratio does not match its change") for y in (BASE, LAST + 1): A.printed(f"window.offences.felony assault.per_m_entries.{y}", f"{1e6 * win[y]['felony assault'] / w_entries[y]:.2f}") for key in ("six major", "grand larceny", "robbery", "felony assault"): for end in ("lo", "hi"): A.printed(f"window.offences.{key}.vs_2019.{end}_pct", window_size(key, BASE, end)) for key in ("grand larceny", "robbery"): for end in ("lo", "hi"): A.printed(f"window.offences.{key}.vs_2019_paid.{end}_pct", window_size(key, BASE, end, paid_rides_only=True)) for end in ("lo", "hi"): A.printed(f"window.offences.six major.vs_2025.{end}_pct", window_size("six major", LAST, end)) # the changes the page prints in whole percents, kept to three places as well, so a finding quoting one # rounds it once (grand larceny's 13.46% is stored as 13.5, which would print as 14%) for key, base in (("grand larceny", BASE), ("six major", BASE), ("six major", LAST)): ratio_k = (win[LAST + 1][key] / w_entries[LAST + 1]) / (win[base][key] / w_entries[base]) w[key][f"vs_{base}"]["change_pct_exact"] = r(100 * (ratio_k - 1), 3) A.printed("window.offences.grand larceny.vs_2019.change_pct", window_size("grand larceny", BASE)) for y in (BASE, LAST): A.printed(f"window.unpaid_share.{y}", f"{100 * (1 - w_paid[y] / w_entries[y]):.0f}%") # ---- the history's figures and the recent readings, each from its quote (v0.9.6; owner, 15 October: the # story in its history, the readings in the news checked). None enters a comparison of its own; the checks # below set each reading beside the page's own measure. def qnum(key: str, pattern: str) -> float: m = re.search(pattern, A.results["quotes"][key]["quote"]) if not m: raise AssertionError(f"{key}: {pattern!r} not in the quote") return float(m.group(1).replace(",", "")) A.results["history"] = { "surge_2022_shifts_a_day": int(qnum("surge_2022", r"approximately ([\d,]+) additional overtime")), "guard_2024_nypd_officers": int(qnum("guard_2024_nypd", r"additional ([\d,]+) NYPD officers")), "nypd_transit_officers_2025": int(qnum("overnight_2025_2500", r"has ([\d,]+) officers")), "weekend_share_of_2019_pct": int(qnum("osc_weekend", r"weekend subway ridership was at (\d+)% of the 2019")), "weekday_share_of_2019_pct": int(qnum("osc_weekday", r"weekday subway ridership was at (\d+)% of the 2019")), } for key, fmt in (("surge_2022_shifts_a_day", "{:,}"), ("guard_2024_nypd_officers", "{:,}"), ("nypd_transit_officers_2025", "{:,}"), ("weekend_share_of_2019_pct", "{}%"), ("weekday_share_of_2019_pct", "{}%")): A.printed(f"history.{key}", fmt.format(A.results["history"][key])) A.results["news"] = { "vital_city_violence_2019_2024_pct": int(qnum("vc_violence", r"by about (\d+)%")), "vital_city_violence_2014_2024_pct": int(qnum("vc_violence", r"and by (\d+)% from 2014")), "vital_city_per_ride_vs_2019_pct": int(qnum("cr_20pct", r"about (\d+)% higher")), "vital_city_h1_2026_felonies": int(qnum("cr_h1", r"There were ([\d,]+) major felonies")), "ny1_felony_assault_pct": int(qnum("ny1_fa", r"rose (\d+)%")), "ny1_overall_under_pct": int(qnum("ny1_overall", r"less than (\d+)%")), "nypd_felony_assault_share_pct": qnum("nypd_q3_shares", r"accounting for ([\d.]+)%"), "nypd_grand_larceny_share_pct": qnum("nypd_q3_shares", r"and ([\d.]+)% \(803\)"), } nw = A.results["news"] A.printed("news.vital_city_violence_2019_2024_pct", f"{nw['vital_city_violence_2019_2024_pct']}%") A.printed("news.vital_city_violence_2014_2024_pct", f"{nw['vital_city_violence_2014_2024_pct']}%") A.printed("news.vital_city_per_ride_vs_2019_pct", f"{nw['vital_city_per_ride_vs_2019_pct']}%") A.printed("news.vital_city_h1_2026_felonies", f"{nw['vital_city_h1_2026_felonies']:,}") A.printed("news.ny1_felony_assault_pct", f"{nw['ny1_felony_assault_pct']}%") A.printed("news.ny1_overall_under_pct", f"{nw['ny1_overall_under_pct']}%") A.printed("news.nypd_felony_assault_share_pct", f"{nw['nypd_felony_assault_share_pct']}%") A.printed("news.nypd_grand_larceny_share_pct", f"{nw['nypd_grand_larceny_share_pct']}%") # the checks: the six on paid rides alone, January to May against 2019 (the basis a per-ride reading without # fare evaders uses), and felony assault January to May against last year (a count-based "up 11%" through # September is set beside the page's per-entry comparison, whose direction is unclear, asserted above) A.printed("window.offences.six major.vs_2019_paid.change_pct", window_size("six major", BASE, paid_rides_only=True)) for end in ("lo", "hi"): A.printed(f"window.offences.six major.vs_2019_paid.{end}_pct", window_size("six major", BASE, end, paid_rides_only=True)) A.printed("window.offences.felony assault.vs_2025.change_pct", window_size("felony assault", LAST)) for end in ("lo", "hi"): A.printed(f"window.offences.felony assault.vs_2025.{end}_pct", window_size("felony assault", LAST, end)) # ---- a plausibility check across sources (not a reproduction of one statistic) A.reproduce("Major felonies in the transit system, fiscal 2026 (July 2025 to June 2026)", published=int(mm.loc[last]), published_moe90=None, reproduced=int(d[(d["m"] >= "2025-07-01") & (d["m"] <= "2026-06-01") & d["major"]]["n"].sum()), kind="count", tolerance=0.06 * int(mm.loc[last])) A.results["reproduction_source"] = "Mayor's Management Report, fiscal 2026 (NYPD chapter)" A.results["reproduction_note"] = ("A cross-source check, not a reproduction of one statistic: the complaint records count " "complaints whose premises is the subway, by the month the crime happened, with June " "2026 incomplete and late reports left out; the report counts the transit system under " "the Transit Bureau. The gap is 3.8%, inside a chosen tolerance of 6%.") # ---- data.csv rows = [] for y in years: for key in list(OFFENCES) + ["all felonies"]: rows.append({"series": "complaints, subway premises, calendar year", "period": str(y), "offence": key, "count": cal[y][key], "paid_rides_m": r(y_paid[y] / 1e6, 1), "entries_m": None}) for y in years: rows.append({"series": "complaints, subway premises, calendar year", "period": str(y), "offence": "theft of services (NYPD code 478)", "count": A.results["years"]["theft_of_services"][str(y)], "paid_rides_m": r(y_paid[y] / 1e6, 1), "entries_m": None}) for y in win: for key in list(OFFENCES) + ["all felonies"]: rows.append({"series": "complaints, subway premises, January to May", "period": str(y), "offence": key, "count": win[y][key], "paid_rides_m": r(w_paid[y] / 1e6, 1), "entries_m": r(w_entries[y] / 1e6, 1)}) for kind, n in kinds[y].items(): rows.append({"series": "complaints, subway premises, January to May", "period": str(y), "offence": f"felony assault: {kind}", "count": int(n), "paid_rides_m": r(w_paid[y] / 1e6, 1), "entries_m": r(w_entries[y] / 1e6, 1)}) for y, v in fy.items(): rows.append({"series": "Mayor's Management Report, transit system, fiscal year", "period": f"FY{y}", "offence": "six major", "count": v["felonies"], "paid_rides_m": v["paid_m"], "entries_m": None}) data = pd.DataFrame(rows) # ---- charts fys = [str(y) for y in sorted(fy)] # how much the uncounted fare evaders flatter the later years, from the unpaid shares themselves fiscal_bias = (1 - fr["unpaid_share"][str(2019)] / 100) / (1 - fr["unpaid_share"][str(last)] / 100) - 1 year_bias = (1 - A.results["years"]["unpaid_share"][str(BASE)] / 100) / (1 - A.results["years"]["unpaid_share"][str(LAST)] / 100) - 1 A.chart("long_view", lines( title=f"Per ride, major felonies in the transit system are {fr['fall_since_first_pct']:.0f}% below fiscal {first}", subtitle=f"Major felonies per million subway rides, fiscal years (July to June) {first} to {last}", x=[[y, y] for y in fys], series=[["rate", "Major felonies per million rides", "Per million rides"]], cells={"": {"rate": {y: fy[int(y)]["per_m"] for y in fys}}}, format="dec1", axis="Per million rides", x_ticks=[y for y in fys if int(y) % 4 == 2 or y == fys[-1]], x_axis="Fiscal year (July to June)", note="The six major felonies (murder, rape, robbery, felony assault, burglary, grand larceny) in the transit system, " "as the Mayor's Management Report prints them for each fiscal year (July to June), under each year's law, " "divided by the subway's paid rides in the same months: to 2019 each year's official total spread by 2019's " "month pattern, January and February 2020 as 2020's total less the daily counts from March, then the MTA's " f"daily counts. Fare evaders ride without being counted (about {fr['unpaid_share'][str(2019)]:.0f}% of entries " f"in fiscal 2019 and {fr['unpaid_share'][str(last)]:.0f}% in fiscal {last}, by the MTA's estimate), so the " f"latest years read about {100 * fiscal_bias:.0f}% high against 2019.", provenance="Mayor's Management Report, fiscal 2001-2026; MTA and NYU Rudin annual ridership; MTA daily ridership")) win_years = [str(y) for y in (BASE, LAST + 1)] strip_keys = ("six major", "grand larceny", "felony assault", "robbery") A.chart("window", strip( title="Against 2019, felony assault per ride has doubled; grand larceny is lower", subtitle="Crimes recorded in the subway per million entries, paid or not, January to May of 2019 and 2026", x=[[y, y] for y in win_years], series=[[k, NAMES[k], NAMES[k]] for k in strip_keys], cells={"": {k: {y: w[k]["per_m_entries"][y] for y in win_years} for k in strip_keys}}, format="dec2", axis="Per million entries", order="series", moe_label="rate", note="Complaints whose premises is the subway, by the month the crime happened, divided by subway entries: each " "month's paid rides over the share that paid, by the MTA's quarterly estimate of fare evasion. The six major " "felonies are murder, rape, robbery, felony assault, burglary and grand larceny; these records hold no murder " "with the subway as its place, so here they are five. 2019's months are rebuilt " "from the MTA's comparable-day figures, scaled to its official 2019 total.", provenance="NYPD Complaint Data (premises: subway), NYC Open Data; MTA ridership and fare-evasion estimates")) ys = [str(y) for y in years] A.chart("by_offence", lines( title="Felony assaults per ride doubled from 2019 and stayed high; robbery's 2020 jump has faded", subtitle=f"Crimes recorded in the subway per million paid rides, {FIRST_YEAR} to {LAST}", x=[[y, y] for y in ys], series=[[k, NAMES[k], NAMES[k]] for k in ("grand larceny", "felony assault", "robbery")], cells={"": {k: rates[k] for k in ("grand larceny", "felony assault", "robbery")}}, format="dec2", axis="Per million paid rides", x_ticks=[y for y in ys if int(y) % 2 == 1 or y == ys[-1]], note="Crimes recorded by the NYPD whose premises is the subway, by the year the crime happened, divided by the " "MTA's paid subway rides. Fare evaders ride without being counted (about " f"{A.results['years']['unpaid_share'][str(BASE)]:.0f}% of entries in {BASE} and " f"{A.results['years']['unpaid_share'][str(LAST)]:.0f}% in {LAST}, by the MTA's estimate), so the later years " f"read about {100 * year_bias:.0f}% high against {BASE}.", provenance="NYPD Complaint Data (premises: subway), NYC Open Data; MTA ridership")) A.write(data=data) print(f"{SLUG}: wrote {A.out}") return 0 if __name__ == "__main__": raise SystemExit(main())