{"cells":[{"cell_type":"markdown","id":"f911f198","metadata":{},"source":"# ๐Ÿฆต RSNA Knee: 0.926 LB, CoaTNet + fine-tune blend\n\n**0.926 on the public leaderboard** (top 25% of ~2,700 teams, scored by the two-arm\nversion of this notebook), one T4, internet off, about 6.5 hours. This notebook ports the\npublic **0.924** CoaTNet checkpoint, verifies it against the 58 radiologist-read studies,\nfine-tunes a second arm from it, adds the checkpoint's SWA twin as a third arm, and blends\nall three under two-pass test-time augmentation.\n\nThe checkpoint that made this worth doing: `dreaddevelopment/raptor-knee-widedense` (CC0) โ€”\na CoAtNet trained on soft language-model labels of the reports, scoring **0.924 as a single\nmodel** on the public leaderboard, weights and inference notebook fully public. Our first\nnotebook trained a small ResNet18 and scored 0.798, so three things happen here with it.\n\n1. **๐Ÿ” Reproduce it.** Port their preprocessing and head exactly, load their checkpoint,\n score it on the 58 studies that carry radiologist labels, and compare against the two\n numbers they published for it (0.9167 stored in the checkpoint, 0.9054 in their notebook's\n comments). We land between them.\n2. **๐ŸŽฏ Fine-tune it.** Continue training from their checkpoint on a different label set,\n `yunusgmsoy report_labels_v5`, which is a four-source merge that contains the 58 real\n annotations. A second arm trained on different labels is the one kind of diversity their\n own blending did not have; their arms shared labels and architecture, and blending bought\n them about +0.001 on the live board. Expect a small gain here too, not a second 0.92.\n3. **๐Ÿงช Blend and submit.** Weighted rank-mean of the arms, the weights picked on a\n held-out fifth of the studies and never on the 58.\n4. **๐Ÿงฌ Go further.** Eight fine-tune epochs (the 0.926 version stopped at three while the\n gate was still climbing), the dataset's SWA checkpoint as a third arm, and a second\n center-shifted pass over the test set โ€” each change measurable against what scored 0.926.\n\nTwo honesty rules, both inherited from the first notebook. The v5 label set contains the answers\nfor the 58 labelled studies, so whether those 58 may be used to *measure* depends entirely on\nwhether they were in training: `INCLUDE_GOLD=False` keeps them out and the 58 stay a\nmeasurement, `INCLUDE_GOLD=True` puts them in for the final submission run and the 58 stop\nbeing a measurement. And a mean AUC over 58 studies moves about 0.03 between runs of the same\nrecipe, so these numbers validate that a model works; they do not rank models that are close.\n\nEverything runs in this notebook on one T4 with the internet off: reading all 4,407 training\nstudies, arm-1 inference, fine-tuning, blending, and writing the submission. About 6.5 hours."},{"cell_type":"code","id":"3c53c5b5","metadata":{},"execution_count":null,"outputs":[],"source":"import os\n\nos.environ.setdefault(\"HF_HUB_OFFLINE\", \"1\")\nos.environ.setdefault(\"TRANSFORMERS_OFFLINE\", \"1\")\nos.environ.setdefault(\"HF_HUB_DISABLE_TELEMETRY\", \"1\")\n\nimport gc, glob, hashlib, json, math, pathlib, shutil, time, unicodedata, warnings\nfrom concurrent.futures import ThreadPoolExecutor\n\nimport cv2\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport pydicom\nimport timm\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom sklearn.metrics import roc_auc_score\n\nwarnings.filterwarnings(\"ignore\")\n\nT0 = time.time()\n\ndef log(msg):\n print(f\"[{time.time() - T0:7.1f}s] {msg}\", flush=True)\n\n# One session gets a fixed amount of wall clock, and the scored run reruns the whole notebook\n# against a test set nobody has seen. Every stage below checks these numbers, so a slow stage\n# shrinks the next one instead of killing the run. The fallback ladder: the notebook writes a\n# placeholder submission first, then arm-1 only (about 0.92), then the blend.\nHARD_DEADLINE = T0 + 8.5 * 3600 # 30 min of the 9 h session left for Kaggle teardown\nPREP_BUDGET_S = 2.5 * 3600 # pass-1 read budget, same number as the first notebook\nFT_END_BY = HARD_DEADLINE - 3.0 * 3600\nEVAL_END_BY = HARD_DEADLINE - 1.2 * 3600\nTEST_START_BY = HARD_DEADLINE - 1.0 * 3600\nFINAL_WRITE_BY = HARD_DEADLINE - 0.25 * 3600\n\n# The flag that decides what this run is. False: the 58 radiologist-labelled studies stay out of\n# fine-tuning, so every number against them is a measurement. True: they go in (their v5 labels\n# are the exact annotations), the run is the submission run, and the 58-study number printed is\n# contaminated by construction. Two pushes: first False to measure, then True to submit.\nINCLUDE_GOLD = True\n\nHOLDOUT_FRAC = 0.20 # report-hash holdout, excluded from fine-tune in BOTH modes\nW_WINDOWS = 12 # windows sampled per study per epoch\nEPOCHS_FT = 8 # v2 stopped at 3 while the holdout gate was still climbing\nACCUM = 4 # studies per optimizer step\nLR_FT = 1e-5\nWD_FT = 1e-4\nWARMUP_FT = 100\nSEED = 0\nREAD_THREADS = 12\nK_EVAL = 42 # windows per study at inference, raptor's scored setting\nK_GATE = 12 # cheaper windows for the per-epoch gate\nTTA_TEST = 2 # test-set passes per arm, center-shifted; gold/holdout stay 1\n\nTIMINGS = {}\n\n# The competition mounts under /kaggle/input/competitions/ on this image, and under\n# /kaggle/input/ on older ones.\nCOMP = next(p for p in map(pathlib.Path, [\n \"/kaggle/input/competitions/rsna-knee-abnormality-detection\",\n \"/kaggle/input/rsna-knee-abnormality-detection\",\n \"/tmp/rsnaknee\"]) if (p / \"train.csv\").exists())\n\nLABELS = [\"ACL\", \"MCL\", \"Medial Meniscus\", \"Lateral Meniscus\", \"Medial OA\", \"Lateral OA\",\n \"PF OA\", \"Effusion\", \"Synovitis\", \"Baker's\", \"Contusion\", \"Fracture\"]\n\ntrain = pd.read_csv(COMP / \"train.csv\")\ntrain_series = pd.read_csv(COMP / \"train_series.csv\")\ntest = pd.read_csv(COMP / \"test.csv\")\ntest_series = pd.read_csv(COMP / \"test_series.csv\")\n# train.csv / test.csv carry only StudyInstanceUID + Report (+labels); the series CSVs add\n# SeriesInstanceUID, so cast per-column rather than assuming both everywhere.\nfor d in (train, train_series, test, test_series):\n for c in (\"StudyInstanceUID\", \"SeriesInstanceUID\"):\n if c in d.columns:\n d[c] = d[c].astype(str)\n\nreports = train.set_index(\"StudyInstanceUID\").Report.fillna(\"\")\ngold = train[train[LABELS].notna().all(axis=1)].set_index(\"StudyInstanceUID\")[LABELS].astype(int)\n\ndev = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ntorch.backends.cudnn.benchmark = True\ntorch.backends.cuda.matmul.allow_tf32 = True\n\n# Scratch space for the memmap stack cache and the epoch checkpoints. On this image /kaggle/tmp\n# does not exist until something creates it, and shutil.disk_usage on a missing path raises.\nos.makedirs(\"/kaggle/tmp\", exist_ok=True)\n\nlog(f\"data root {COMP}\")\nlog(f\"train studies {len(train):,} train series {len(train_series):,}\")\nlog(f\"test studies {len(test):,} test series {len(test_series):,}\")\nlog(f\"labelled studies {len(gold)} ({len(gold) / len(train):.2%} of training)\")\nlog(f\"INCLUDE_GOLD {INCLUDE_GOLD} ({'the 58 are training rows, not a measurement' if INCLUDE_GOLD else 'the 58 are held out of fine-tuning and stay a measurement'})\")\nlog(f\"device {dev} | gpus {torch.cuda.device_count()} | \"\n f\"torch {torch.__version__} | timm {timm.__version__} | pydicom {pydicom.__version__} | \"\n f\"cv2 {cv2.__version__}\")\nif dev == \"cuda\":\n log(f\"gpu {torch.cuda.get_device_name(0)}\")\nlog(f\"disk free /kaggle/tmp {shutil.disk_usage('/kaggle/tmp').free / 1e9:.1f} GB | \"\n f\"RAM {os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES') / 1e9:.1f} GB\")"},{"cell_type":"markdown","id":"00ff1195","metadata":{},"source":"## ๐Ÿ“ Write a valid submission before doing anything else\n\nSame rule as the first notebook, for the same reason. When this is submitted, Kaggle runs it\nagain against a hidden test set, and a run that fails part way through scores nothing. The\nplaceholder of all 0.5 values is written now; two later rungs overwrite it, and neither rung\never touches submission.csv unless it has a complete result in hand."},{"cell_type":"code","id":"56aab0a0","metadata":{},"execution_count":null,"outputs":[],"source":"def write_submission(pred: pd.DataFrame, path=\"submission.csv\") -> pd.DataFrame:\n \"\"\"Rank inside each column, then write one row per test study.\n\n The score reads only the order of the values in a column, so replacing the numbers by their\n ranks throws nothing away. Feeding already-ranked values back through is harmless: the second\n ranking preserves the order of the first.\n \"\"\"\n sub = pred.rank(pct=True) if len(pred) > 1 else pred.copy()\n sub.index.name = \"StudyInstanceUID\"\n sub = sub.reindex(test.StudyInstanceUID).fillna(0.5).reset_index()\n sub = sub[[\"StudyInstanceUID\"] + LABELS]\n assert len(sub) == len(test), f\"{len(sub)} rows against {len(test)} test studies\"\n assert sub[LABELS].notna().all().all(), \"a prediction is missing\"\n sub.to_csv(path, index=False)\n return sub\n\nwrite_submission(pd.DataFrame(0.5, index=test.StudyInstanceUID, columns=LABELS))\nlog(\"wrote a placeholder submission.csv\")"},{"cell_type":"markdown","id":"c2fd1b97","metadata":{},"source":"## ๐Ÿงฐ Shared helpers\n\nCarried over from the first notebook: the HTML table display, the bounded file finder, the\nscorers, and the bootstrap. The bounded finder matters more than it looks: a recursive glob\nunder /kaggle/input walks the competition mount and its 700,000 DICOM files, which costs\nminutes for every lookup."},{"cell_type":"code","id":"703c1090","metadata":{},"execution_count":null,"outputs":[],"source":"from html import escape as _esc\nfrom IPython.display import display, HTML, Markdown\n\nTABLE_CSS = [\n {\"selector\": \"caption\", \"props\": [(\"caption-side\", \"top\"), (\"text-align\", \"left\"),\n (\"font-weight\", \"600\"), (\"font-size\", \"0.95rem\"),\n (\"padding\", \"0 0 0.45rem 0\"), (\"color\", \"#0b0b0b\")]},\n {\"selector\": \"th\", \"props\": [(\"background-color\", \"#f2f1ec\"), (\"color\", \"#0b0b0b\"),\n (\"font-weight\", \"600\"), (\"text-align\", \"right\"),\n (\"padding\", \"5px 11px\"), (\"border-bottom\", \"1px solid #c3c2b7\")]},\n {\"selector\": \"th.row_heading\", \"props\": [(\"text-align\", \"left\"),\n (\"background-color\", \"#fcfcfb\")]},\n {\"selector\": \"td\", \"props\": [(\"padding\", \"5px 11px\"), (\"text-align\", \"right\"),\n (\"border-bottom\", \"1px solid #e1e0d9\")]},\n {\"selector\": \"\", \"props\": [(\"border-collapse\", \"collapse\"), (\"font-size\", \"0.86rem\"),\n (\"font-variant-numeric\", \"tabular-nums\"), (\"margin\", \"0.3rem 0\")]},\n]\n\ndef _cell(v):\n if isinstance(v, (bool, np.bool_)):\n return \"yes\" if v else \"no\"\n if isinstance(v, (int, np.integer)):\n return f\"{v:,}\"\n if isinstance(v, (float, np.floating)):\n if not np.isfinite(v):\n return \"\"\n return f\"{v:,.3f}\" if abs(v) < 1e4 else f\"{v:,.0f}\"\n return str(v)\n\ndef show(df, caption=\"\", bars=None, vmin=None, vmax=None, hide_index=False):\n \"\"\"Display a table as HTML, optionally with in-cell bars (the baseline's shading option is\n dropped: this notebook shades nothing).\"\"\"\n st = (pd.DataFrame(df).style\n .set_caption(caption).set_table_styles(TABLE_CSS)\n .format(_cell, na_rep=\"\"))\n if bars is not None:\n st = st.bar(subset=bars, color=\"#cde2fb\", vmin=vmin, vmax=vmax)\n if hide_index:\n st = st.hide(axis=\"index\")\n display(st)\n\ndef facts(pairs, caption=\"\"):\n show(pd.DataFrame({\"value\": [v for _, v in pairs]}, index=[k for k, _ in pairs]), caption)\n\ndef note(md):\n display(Markdown(md))\n\ndef short_uid(u, n=10):\n u = str(u)\n return u if len(u) <= n else u[:n] + \"..\"\n\ndef fold(s: str) -> str:\n \"\"\"Lower case, strip accents, map the Turkish dotless i onto a plain i.\"\"\"\n s = str(s).replace(\"\\u0131\", \"i\").replace(\"\\u0130\", \"i\")\n s = unicodedata.normalize(\"NFKD\", s)\n return \"\".join(c for c in s if not unicodedata.combining(c)).lower()\n\nSEARCH = [p for p in map(pathlib.Path, [\"/kaggle/input/datasets\", \"/kaggle/input\", \"/tmp/rsnallm\"])\n if p.exists()]\n\ndef find_file(name: str):\n \"\"\"Find an attached dataset file without walking the competition folder.\"\"\"\n for root in SEARCH:\n for pat in (name, f\"*/{name}\", f\"*/*/{name}\", f\"*/*/*/{name}\"):\n for hit in root.glob(pat):\n if \"competitions\" not in hit.parts:\n return hit\n return None\n\ndef score_against_gold(pred: pd.DataFrame) -> pd.Series:\n p = pred.reindex(gold.index)\n return pd.Series({c: roc_auc_score(gold[c], p[c].fillna(0.0)) for c in LABELS})\n\ndef safe_auc(y, p):\n \"\"\"AUC, or nan when the column has no positives or no negatives to compare.\"\"\"\n y = np.asarray(y)\n if len(y) == 0 or y.min() == y.max():\n return np.nan\n return float(roc_auc_score(y, p))\n\ndef mean_auc(y, p):\n vals = [safe_auc((y[:, j] > 0.5).astype(int), p[:, j]) for j in range(len(LABELS))]\n vals = [v for v in vals if np.isfinite(v)]\n return float(np.mean(vals)) if vals else np.nan\n\ndef bootstrap_auc(y: pd.DataFrame, p: pd.DataFrame, n_boot=2000, seed=0):\n \"\"\"95% interval width for each finding's AUC and for the average, from resampling the rows.\"\"\"\n rng = np.random.default_rng(seed)\n per, mean = {c: [] for c in LABELS}, []\n for _ in range(n_boot):\n i = rng.integers(0, len(y), len(y))\n vals = []\n for c in LABELS:\n yc = y[c].values[i]\n if yc.min() == yc.max():\n continue\n a = roc_auc_score(yc, p[c].values[i])\n per[c].append(a); vals.append(a)\n if len(vals) == len(LABELS):\n mean.append(np.mean(vals))\n width = {c: float(np.diff(np.percentile(v, [2.5, 97.5]))[0]) for c, v in per.items()}\n return width, float(np.diff(np.percentile(mean, [2.5, 97.5]))[0])\n\ndef contains_the_answers(labels: pd.DataFrame) -> dict:\n on = labels.reindex(gold.index)\n off = labels.loc[~labels.index.isin(gold.index)]\n exact_on = float(np.isin(on.values, [0.0, 1.0]).mean())\n exact_off = float(np.isin(off.values, [0.0, 1.0]).mean())\n match = float((on.values == gold.values).mean())\n return {\"exactly 0 or 1 on the 58\": exact_on,\n \"exactly 0 or 1 elsewhere\": exact_off,\n \"equals the annotation\": match,\n \"distinct values elsewhere\": int(len(np.unique(off.values))),\n \"contains the answers\": bool(exact_on > 0.999 and match > 0.999 and exact_off < 0.01)}"},{"cell_type":"markdown","id":"fdabaa48","metadata":{},"source":"## ๐Ÿท๏ธ The fine-tuning labels\n\nOne published set is attached: `yunusgmsoy/rsna-knee-llm-labels-4-source-merged`, file\n`report_labels_v5.csv`. It merges four language-model label sets and, unlike every honest set\nthe first notebook audited, it carries the 58 radiologist annotations written in exactly. That\nis the reason this notebook picked it: real labels are the best labels there are. It is also\nthe reason the `INCLUDE_GOLD` flag exists, because the same rows cannot be both training data\nand the measuring stick."},{"cell_type":"code","id":"863d1461","metadata":{},"execution_count":null,"outputs":[],"source":"PUBLISHED = {\"four sets merged\": \"report_labels_v5.csv\"}\npublished, loaded = {}, []\nfor name, fname in PUBLISHED.items():\n hit = find_file(fname)\n if hit is None:\n loaded.append({\"labels\": name, \"file\": fname, \"found\": False, \"studies\": 0})\n continue\n d = pd.read_csv(hit).drop_duplicates(\"StudyInstanceUID\").set_index(\"StudyInstanceUID\")\n ok = all(c in d.columns for c in LABELS)\n if ok:\n published[name] = d[LABELS].astype(float)\n loaded.append({\"labels\": name, \"file\": hit.name, \"found\": True,\n \"studies\": int(len(d)) if ok else 0})\nshow(pd.DataFrame(loaded), \"What is attached\", hide_index=True)\nassert published, \"the v5 label file was not found among the attached datasets\"\n\nv5_name = next(iter(published))\nv5 = published[v5_name].reindex(train.StudyInstanceUID)\nLABEL_FILE = hit.name\n\ncheck = contains_the_answers(v5.dropna(subset=LABELS, how=\"all\"))\nshow(pd.DataFrame([check]), \"Does v5 contain the 58 answers, and is it soft elsewhere\")\nfacts([(\"studies covered\", int(v5[LABELS].notna().any(axis=1).sum())),\n (\"mean target across findings\", round(float(v5.stack().mean()), 3)),\n (\"share of cells exactly 0 or 1 off the 58\", round(check[\"exactly 0 or 1 elsewhere\"], 4))],\n f\"The fine-tuning target: {v5_name}\")\nnote(f\"Policy for this run: `INCLUDE_GOLD={INCLUDE_GOLD}`. \"\n + (\"The 58 annotated studies are inside fine-tuning, so every 58-study number printed below \"\n \"is contaminated by construction and is printed only to show the model saw them.\"\n if INCLUDE_GOLD else\n \"The 58 annotated studies are held out of fine-tuning, so the 58-study numbers below are \"\n \"honest measurements.\"))"},{"cell_type":"markdown","id":"da74a948","metadata":{},"source":"## โœ‚๏ธ The split\n\nTwo dispositions of the 4,407 studies, decided before anything is trained. A fifth of the\nstudies go to a holdout by a hash of the report text, which keeps studies sharing a report on\nthe same side; the holdout is excluded from fine-tuning in **both** flag modes, so epoch\nselection and the blend weight are always measured out-of-sample. The 58 gold studies are the\nsecond disposition: out of fine-tuning entirely when `INCLUDE_GOLD=False`, and so is any study\nwhose report matches a gold study's report, because the v5 labels of such a twin are the\ngold labels under another name. When `INCLUDE_GOLD=True` the gold rows and their twins go into\nfine-tuning; only the report-hash holdout stays out."},{"cell_type":"code","id":"f475f96d","metadata":{},"execution_count":null,"outputs":[],"source":"def bucket(sid: str) -> int:\n h = hashlib.md5(fold(reports.get(sid, sid)).encode()).hexdigest()\n return int(h[:8], 16) % 100\n\nall_train_ids = list(train.StudyInstanceUID)\ngold_ids = list(gold.index)\nholdout_ids = [s for s in all_train_ids if bucket(s) < HOLDOUT_FRAC * 100]\nft_base = [s for s in all_train_ids if s not in set(holdout_ids)]\n\ngold_text = set(reports.reindex(gold_ids).map(fold))\nif INCLUDE_GOLD:\n ft_ids = ft_base\nelse:\n ft_ids = [s for s in ft_base\n if s not in set(gold_ids) and fold(reports.get(s, \"\")) not in gold_text]\n\nif INCLUDE_GOLD:\n # The report-hash holdout keeps its 6 gold studies in both modes, so gold-in means\n # every non-holdout gold study, not all 58.\n assert (set(gold_ids) - set(holdout_ids)) <= set(ft_ids), \\\n \"gold-in mode but holdout-free gold is missing from fine-tuning\"\nelse:\n assert (set(ft_ids) & set(gold_ids)) == set(), \"gold rows leaked into an honest fine-tune set\"\nn_twins = len(ft_base) - len(ft_ids) if not INCLUDE_GOLD else \\\n sum(1 for s in ft_base if fold(reports.get(s, \"\")) in gold_text)\nlog(f\"holdout (both modes) {len(holdout_ids):,} gold inside it \"\n f\"{len(set(holdout_ids) & set(gold_ids))}\")\nlog(f\"fine-tune set {len(ft_ids):,} ({'gold + report twins included' if INCLUDE_GOLD else 'gold and report twins excluded'})\")\nlog(f\"report twins handled {n_twins}\")"},{"cell_type":"markdown","id":"995d4284","metadata":{},"source":"## ๐Ÿฆพ The raptor arm, ported\n\nEverything in the next two cells is ported from the public inference notebook of\n`dreaddevelopment/knee-mri-twelve-findings-from-a-single-model`, as exactly as the code can be\ncarried, because the weights were trained against this exact pipeline and any drift shows up\nas a silent score drop. What the pipeline does:\n\n- **Five fixed slots** per study, always in the same order: 18 slices from a sagittal\n fluid-sensitive series, 14 from a second sagittal series, 12 coronal fluid-sensitive,\n 8 coronal, 12 axial. A study with no series for a slot leaves it as zeros.\n- **Slices spread across 6 to 94 percent** of each series, not the middle: the collateral\n ligaments and the lateral meniscus live in the peripheral slices.\n- **Series ordered by geometry**, from `ImagePositionPatient` crossed with the in-plane axes,\n which is the only order that means anything (the first notebook measured filename order at a\n rank correlation of 0.043 with physical position).\n- **A 140 mm crop** using the pixel spacing, then resize to 336 px, so a knee fills the same\n fraction of the frame whatever the scanner's resolution.\n- **Three neighbouring slices per window**, 42 windows per study at 384 px, ImageNet norm.\n\nThe head pools those windows into twelve scores with **separate attention weights per finding**,\nso a cruciate tear can pick the two sagittal slices that show it while osteoarthritis reads the\ncoronal stack.\n\nDocumented deltas from their code, all deliberate: the weight-file path points at this\nnotebook's dataset mount; `apply_modality_lut` is imported from whichever pydicom location\nprovides it (pydicom 3 moved it); a `Fluid_Sensitive` value of NaN is treated as \"no\npreference\" instead of raising inside `int()` and silently dropping the study to 0.5; and\n`build_study` also returns a per-slice recipe so this notebook can rebuild a study's stack from\ndisk without re-decoding headers."},{"cell_type":"code","id":"39b3475f","metadata":{},"execution_count":null,"outputs":[],"source":"IMG = 336\nCROP_MM = 140.0\nSLOTS = [(\"Sagittal\", 1, 18), (\"Sagittal\", 0, 14), (\"Coronal\", 1, 12),\n (\"Coronal\", 0, 8), (\"Axial\", -1, 12)]\nMAXS = sum(s[2] for s in SLOTS) # 64\nNORM = \"imagenet\"\n_MEAN = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)\n_STD = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)\n\ndef build_backbone(arch, pretrained=False):\n # maxvit/maxxvit/coatnet are conv-attention hybrids: NO CLS token, NO interpolatable\n # pos-embed -> avg pool. The \"vit\" substring in \"coatnet\"/\"maxvit\" must NOT route them\n # down the ViT path (mirrors their finetune_raptor.py exactly).\n hybrid = arch.startswith((\"maxvit\", \"maxxvit\", \"coatnet\", \"coat_\", \"convnext\"))\n is_vit = (not hybrid) and any(k in arch for k in (\"vit\", \"deit\", \"dinov2\", \"eva\", \"beit\"))\n kw = dict(pretrained=pretrained, num_classes=0, in_chans=3)\n if is_vit:\n kw.update(global_pool=\"token\", dynamic_img_size=True)\n else:\n kw.update(global_pool=\"avg\")\n return timm.create_model(arch, **kw)\n\nclass RaptorClassifier(nn.Module):\n def __init__(self, backbone, F_dim=768, n=12, drop=0.2):\n super().__init__()\n self.backbone = backbone\n self.norm = nn.LayerNorm(F_dim)\n self.att = nn.Sequential(nn.Linear(F_dim, 256), nn.Tanh(), nn.Dropout(drop),\n nn.Linear(256, n))\n self.clsW = nn.Parameter(torch.zeros(n, F_dim))\n self.clsb = nn.Parameter(torch.zeros(n))\n nn.init.trunc_normal_(self.clsW, std=0.02)\n self.n = n\n\n def encode(self, x):\n B, K = x.shape[:2]\n f = self.backbone(x.flatten(0, 1))\n return f.view(B, K, -1)\n\n def head(self, feats):\n h = self.norm(feats)\n a = self.att(h)\n a = torch.softmax(a, dim=1)\n pooled = torch.einsum(\"bkn,bkf->bnf\", a, h)\n logits = (pooled * self.clsW).sum(-1) + self.clsb\n return logits\n\n def forward(self, x):\n return self.head(self.encode(x))\n\ndef load_model(pt_path, device):\n \"\"\"One model resident at a time is the discipline, not an optimisation.\n\n Their scored run OOM'd system RAM with DataParallel replicating modules across many studies;\n sequential single-model residency is what graded. Two 293 MB state dicts would not OOM on\n their own, but the discipline costs nothing and matches the scored configuration.\n \"\"\"\n ck = torch.load(pt_path, map_location=\"cpu\", weights_only=False)\n bb = build_backbone(ck[\"arch\"], pretrained=False)\n model = RaptorClassifier(bb, F_dim=bb.num_features)\n model.load_state_dict(ck[\"model\"], strict=True)\n model.eval().to(device)\n return model, int(ck.get(\"res\", 384)), ck\n\ndef _eval_centers(mask, D, k, shift=0):\n valid = np.where(mask > 0)[0]\n if len(valid) < 3:\n valid = np.arange(min(3, D))\n lo, hi = int(valid.min()), int(valid.max())\n cs = [c for c in range(lo + 1, hi) if c - 1 >= lo and c + 1 <= hi]\n if not cs:\n cs = [max(1, min((lo + hi) // 2, D - 2))]\n idx = np.linspace(0, len(cs) - 1, k).round().astype(int)\n if shift:\n idx = np.clip(idx + shift, 0, len(cs) - 1)\n return [cs[i] for i in idx]\n\ndef eval_windows(vol, mask, k, res, norm=NORM, shift=0):\n D = vol.shape[0]\n cs = _eval_centers(mask, D, k, shift=shift)\n wins = np.empty((len(cs), 3, res, res), np.float32)\n for j, c in enumerate(cs):\n c = max(1, min(c, D - 2))\n tri = np.stack([vol[c - 1], vol[c], vol[c + 1]], 0).astype(np.float32) / 255.0\n t = torch.from_numpy(tri)\n if t.shape[-1] != res:\n t = F.interpolate(t[None], size=(res, res), mode=\"bilinear\",\n align_corners=False)[0]\n wins[j] = t.numpy()\n x = torch.from_numpy(wins)\n if norm == \"imagenet\":\n x = (x - _MEAN) / _STD\n return x\n\n@torch.no_grad()\ndef infer_probs(model, xwins, device):\n x = xwins.unsqueeze(0).to(device)\n if device.startswith(\"cuda\"):\n # fp16 conv on T4 is fully cuDNN-supported (bf16 is NOT -> \"no engine\").\n try:\n with torch.autocast(\"cuda\", dtype=torch.float16):\n o = torch.sigmoid(model(x).float())\n return o[0].cpu().numpy()\n except RuntimeError:\n torch.cuda.empty_cache()\n o = torch.sigmoid(model(x).float())\n return o[0].cpu().numpy()\n o = torch.sigmoid(model(x).float())\n return o[0].cpu().numpy()\n\ndef infer_study(model, sid, k, shift=0):\n vol, mask = get_stack(sid)\n return infer_probs(model, eval_windows(vol, mask, k=k, res=RES, shift=shift), dev)\n\ndef infer_guarded(model, ids, k, deadline, tag):\n \"\"\"Evaluate studies one at a time, stopping if the projection crosses the deadline.\n\n The hidden-rerun reality is that the same notebook runs against an unknown test size on\n the same wall clock, so every eval larger than a handful of studies needs a stop rule, not\n a fixed size. Truncation is safe because callers shuffle `ids` first: what survives is a\n random sample of the set, so its AUC stays unbiased. Returns (ids_used, probs).\n \"\"\"\n probs, used, t0 = [], [], time.time()\n for i, s in enumerate(ids):\n if i and (time.time() - t0) / i * (len(ids) - i) > max(deadline - time.time(), 0):\n log(f\"{tag}: stopped at {i}/{len(ids)} โ€” projection crossed the deadline\")\n break\n probs.append(infer_study(model, s, k))\n used.append(s)\n if (i + 1) % 100 == 0:\n log(f\"{tag}: {i + 1}/{len(ids)} {(time.time() - t0) / (i + 1):.1f} s/study\")\n return used, (np.stack(probs) if probs else np.zeros((0, 12), np.float32))\n\ndef rankpct(x): # per-column percentile rank in [0,1]\n order = x.argsort(0).argsort(0).astype(np.float64)\n return order / max(1, (x.shape[0] - 1))\n\ndef blend_ranks(p2, p1, w):\n \"\"\"Weighted rank-mean, raptor's recipe: ranks, weighted, never probabilities.\"\"\"\n return w * rankpct(np.clip(p2, 0, 1)) + (1 - w) * rankpct(np.clip(p1, 0, 1))\n\ndef window_centers(mask):\n \"\"\"Every valid window centre, for training-time sampling.\"\"\"\n valid = np.where(mask > 0)[0]\n if len(valid) < 3:\n valid = np.arange(min(3, MAXS))\n lo, hi = int(valid.min()), int(valid.max())\n cs = [c for c in range(lo + 1, hi) if c - 1 >= lo and c + 1 <= hi]\n return cs or [max(1, min((lo + hi) // 2, MAXS - 2))]"},{"cell_type":"markdown","id":"cbf08521","metadata":{},"source":"### ๐Ÿ–ผ๏ธ The preprocessing, also ported"},{"cell_type":"code","id":"518ef3f7","metadata":{},"execution_count":null,"outputs":[],"source":"def _fs(row):\n \"\"\"NaN Fluid_Sensitive means no preference, not a crash.\n\n Their `_pick_series_for_slot` calls `int(r.get('Fluid_Sensitive', 0) or 0)`, and a NaN is\n truthy, so `int(nan)` raises, the study's per-study try/except catches it, and the study\n silently scores 0.5. Train-series rows carry NaNs, so the fix is part of the port, not an\n embellishment.\n \"\"\"\n v = row.get(\"Fluid_Sensitive\")\n return 0 if pd.isna(v) else int(v)\n\ndef _make_reader():\n import pydicom as _pd\n try:\n from pydicom.pixel_data_handlers.util import apply_modality_lut\n _lut_from = \"pydicom.pixel_data_handlers.util\"\n except ImportError:\n from pydicom.pixels.utils import apply_modality_lut\n _lut_from = \"pydicom.pixels.utils\"\n log(f\"modality LUT helper from {_lut_from} (pydicom {_pd.__version__})\")\n\n def order_and_meta(sdir):\n fs = list(pathlib.Path(sdir).glob(\"*.dcm\")); recs = []; ps_list = []\n for f in fs:\n try:\n h = _pd.dcmread(str(f), stop_before_pixels=True)\n iop = getattr(h, 'ImageOrientationPatient', None)\n ipp = getattr(h, 'ImagePositionPatient', None)\n if iop is not None and ipp is not None and len(iop) == 6:\n r = np.array(iop[:3], float); c = np.array(iop[3:], float)\n n = np.cross(r, c); pos = float(np.dot(np.array(ipp, float), n))\n else:\n pos = float(getattr(h, 'InstanceNumber', 0) or 0)\n ps = getattr(h, 'PixelSpacing', None); ps = float(ps[0]) if ps is not None else 0.5\n ps_list.append(ps); recs.append((pos, str(f), ps))\n except Exception:\n recs.append((0.0, str(f), 0.5))\n recs.sort(key=lambda x: x[0])\n med_ps = float(np.median(ps_list)) if ps_list else 0.5\n return [(f, ps) for _, f, ps in recs], med_ps\n\n def read_px(f):\n d = _pd.dcmread(f)\n a = apply_modality_lut(d.pixel_array, d).astype(np.float32)\n if str(getattr(d, 'PhotometricInterpretation', '')) == 'MONOCHROME1':\n a = a.max() - a\n return a\n\n def mm_crop_resize(a, ps):\n h, w = a.shape; cpx = int(round(CROP_MM / max(ps, 1e-3)))\n cpx = min(cpx, min(h, w)); y0 = (h - cpx) // 2; x0 = (w - cpx) // 2\n a = a[y0:y0 + cpx, x0:x0 + cpx]\n return cv2.resize(a, (IMG, IMG), interpolation=cv2.INTER_AREA)\n\n return order_and_meta, read_px, mm_crop_resize\n\nreader = _make_reader()\norder_and_meta, read_px, mm_crop_resize = reader\n\ndef _pick_series_for_slot(rows, plane, fluid, used):\n cands = [r for r in rows if r[\"Anatomical_Plane\"] == plane\n and r[\"SeriesInstanceUID\"] not in used]\n if fluid in (0, 1):\n pref = [r for r in cands if _fs(r) == fluid]\n if pref:\n return pref[0]\n return cands[0] if cands else None\n\ndef build_study(sid, ser_records, tsdir, reader):\n \"\"\"The 64-slot stack, its valid mask, and a recipe to rebuild it without re-decoding.\"\"\"\n order_and_meta, read_px, mm_crop_resize = reader\n rows = ser_records.get(sid, [])\n vol = np.zeros((MAXS, IMG, IMG), np.uint8); idx = 0; used = set()\n rec_f = [None] * MAXS; rec_ps = [0.0] * MAXS\n rec_lo = [0.0] * MAXS; rec_hi = [1.0] * MAXS\n for plane, fluid, k in SLOTS:\n r = _pick_series_for_slot(rows, plane, fluid, used)\n if r is None:\n idx += k; continue\n used.add(r['SeriesInstanceUID'])\n files, med_ps = order_and_meta(f\"{tsdir}/{sid}/{r['SeriesInstanceUID']}\")\n if not files:\n idx += k; continue\n n = len(files); lo, hi = int(n * 0.06), int(n * 0.94) - 1; hi = max(hi, lo)\n picks = np.linspace(lo, hi, k).round().astype(int) if n > 1 else [0] * k\n arrs = []; pss = []\n for p in picks:\n fp, ps = files[min(p, n - 1)]\n try:\n arrs.append(read_px(fp)); pss.append(ps)\n except Exception:\n arrs.append(None); pss.append(med_ps)\n valid = [a for a in arrs if a is not None]\n if valid:\n allpx = np.concatenate([a.ravel() for a in valid])\n loq, hiq = np.percentile(allpx, [2.0, 98.0])\n else:\n loq, hiq = 0.0, 1.0\n for j, (a, ps) in enumerate(zip(arrs, pss)):\n if idx >= MAXS: break\n if a is None:\n idx += 1; continue\n aw = np.clip((a - loq) / (hiq - loq + 1e-6), 0, 1)\n aw = mm_crop_resize(aw, ps if ps > 0 else med_ps)\n vol[idx] = (aw * 255).astype(np.uint8)\n rec_f[idx] = files[min(picks[j], n - 1)][0]\n rec_ps[idx] = ps if ps > 0 else med_ps\n rec_lo[idx] = float(loq); rec_hi[idx] = float(hiq)\n idx += 1\n if idx >= MAXS: break\n mask = (vol.reshape(MAXS, -1).sum(1) > 0).astype(np.uint8)\n return vol, mask, {\"files\": rec_f, \"pss\": rec_ps, \"loqs\": rec_lo, \"hiqs\": rec_hi}\n\ndef find_test_root():\n cands = [\"/kaggle/input/competitions/rsna-knee-abnormality-detection\",\n \"/kaggle/input/rsna-knee-abnormality-detection\"]\n for b in cands:\n if os.path.exists(b + \"/test.csv\"):\n return b\n for d, _, f in os.walk(\"/kaggle/input\"):\n if \"test.csv\" in f and (os.path.isdir(d + \"/test_series\") or os.path.isdir(d + \"/test_images\")):\n return d\n for d, _, f in os.walk(\"/kaggle/input\"):\n if \"test.csv\" in f:\n return d\n raise RuntimeError(\"no test root under /kaggle/input\")\n\nCKPT_FILE = \"raptor_ft_coatnet_v4_full.pt\"\nCKPT_FILE_SWA = \"raptor_ft_coatnet_v4_full_swa.pt\" # the dataset's second checkpoint: arm 3\n\ndef find_weight_file(fname):\n direct = [f\"/kaggle/input/raptor-knee-widedense/{fname}\",\n f\"/kaggle/input/raptor-knee-widedense/1/{fname}\"]\n for p in direct:\n if os.path.exists(p):\n return p\n for d in sorted(glob.glob(\"/kaggle/input/*/\")):\n if \"competition\" in d.lower():\n continue\n hits = glob.glob(os.path.join(d, \"**\", fname), recursive=True)\n if hits:\n return hits[0]\n raise RuntimeError(f\"{fname} not found under /kaggle/input\")\n\nTRAIN_DIR = COMP / \"train_series\"\nSER_TRAIN = {k: v.to_dict(\"records\") for k, v in train_series.groupby(\"StudyInstanceUID\")}\n\ndef get_stack(sid):\n \"\"\"(vol, mask) for any study: memmap row, recipe rebuild, or a fresh decode.\n\n The recipe path is bit-identical to a fresh decode because the per-slot window bounds are\n stored, not recomputed.\n \"\"\"\n if sid in idx_of and USE_MM:\n return np.asarray(STACKS[idx_of[sid]]), np.frombuffer(recipe[sid][\"mask\"], np.uint8)\n r = recipe.get(sid)\n if r is not None:\n vol = np.zeros((MAXS, IMG, IMG), np.uint8)\n for i in range(MAXS):\n if r[\"files\"][i] is None:\n continue\n a = read_px(r[\"files\"][i])\n aw = np.clip((a - r[\"loqs\"][i]) / (r[\"hiqs\"][i] - r[\"loqs\"][i] + 1e-6), 0, 1)\n vol[i] = (mm_crop_resize(aw, r[\"pss\"][i]) * 255).astype(np.uint8)\n return vol, np.frombuffer(r[\"mask\"], np.uint8)\n if sid in SER_TRAIN:\n vol, mask, _ = build_study(sid, SER_TRAIN, str(TRAIN_DIR), reader)\n else:\n vol, mask, _ = build_study(sid, SER_TEST, str(TEST_DIR), reader)\n return vol, mask"},{"cell_type":"markdown","id":"fdda5ef9","metadata":{},"source":"## ๐Ÿ“ฆ Load the checkpoint\n\nThe file is a dictionary, not a bare state dict, and it carries its own provenance: the\narchitecture name, the resolution it expects, the label order it was trained with, which epoch\nit came from, and the score its author measured on the 58. Every one of those is checked against\nthis notebook's own settings, because a mismatch here degrades silently otherwise."},{"cell_type":"code","id":"ecb03449","metadata":{},"execution_count":null,"outputs":[],"source":"t_load = time.time()\nweight_path = find_weight_file(CKPT_FILE)\narm1, RES, ck = load_model(weight_path, dev)\nck_meta = {k: v for k, v in ck.items() if k != \"model\"}\ndel ck; gc.collect()\n\nfacts([( \"checkpoint\", short_uid(weight_path.split(\"/\")[-1], 40)),\n (\"architecture\", str(ck_meta.get(\"arch\"))),\n (\"resolution\", int(ck_meta.get(\"res\", RES))),\n (\"trained epoch\", ck_meta.get(\"epoch\")),\n (\"their gold AUC, stored in the file\", round(float(ck_meta.get(\"gold_auc\", np.nan)), 4)),\n (\"their gold AUC, in their notebook comment\", 0.9054),\n (\"parameters\", f\"{sum(p.numel() for p in arm1.parameters()) / 1e6:.2f} M\")],\n \"What the checkpoint says about itself\")\nassert list(ck_meta[\"lab\"]) == LABELS, f\"label order drift: {ck_meta['lab']}\"\nassert RES == 384, f\"unexpected model resolution {RES}\"\nlog(f\"checkpoint loaded in {time.time() - t_load:.0f}s\")\n\nx = torch.randn(1, 4, 3, RES, RES, device=dev)\nwith torch.no_grad():\n _ = arm1(x) # warmup: cudnn.benchmark autotunes the first call, do not time it\n if dev == \"cuda\":\n torch.cuda.synchronize()\n t_s = time.time()\n for _ in range(4):\n _ = arm1(x)\n if dev == \"cuda\":\n torch.cuda.synchronize()\nif dev == \"cuda\":\n torch.cuda.synchronize()\nMS_WINDOW = (time.time() - t_s) / 4 * 1000\ndel x; gc.collect()\nlog(f\"sanity forward: {MS_WINDOW:.0f} ms/window at {RES}px -> \"\n f\"~{MS_WINDOW * K_EVAL / 1000:.1f} s/study at {K_EVAL} windows\")"},{"cell_type":"markdown","id":"e0661abc","metadata":{},"source":"## ๐Ÿ“š Pass 1: read every training study once\n\nThe arithmetic that decides how this is built: 4,407 studies of 64 slices at 336 by 336, one\nbyte per pixel, is **31.8 GB**. That cannot live in the RAM of a Kaggle box, so the stacks go to\na memory-mapped file on the fast local disk instead, and what stays in RAM is a small recipe per\nstudy: the 64 file paths, the pixel spacing and window bounds each slice was normalised with,\nand the valid window centres. With the recipe, any study can be rebuilt exactly, bounds and all,\nwithout re-reading a single header; if the disk does not have room, the notebook falls back to\nrebuilding from the recipe for every study and pays the decode time per epoch instead.\n\nThe read order is the 58 gold studies first, then a seeded shuffle of the rest, so if the time\nbudget truncates the pass, what was read is a random sample rather than the front of the\nalphabetical directory listing."},{"cell_type":"code","id":"c17a2d75","metadata":{},"execution_count":null,"outputs":[],"source":"rng = np.random.default_rng(SEED)\nrest = [s for s in all_train_ids if s not in set(gold_ids)]\ndecode_order = list(gold_ids) + list(rng.permutation(rest))\n\nMM_PATH = \"/kaggle/tmp/stacks.fbm\"\nDISK_NEED = int(len(all_train_ids) * MAXS * IMG * IMG) + int(2e9)\nUSE_MM = shutil.disk_usage(\"/kaggle/tmp\").free >= DISK_NEED\nidx_of, recipe = {}, {}\nif USE_MM:\n STACKS = np.memmap(MM_PATH, dtype=np.uint8, mode=\"w+\",\n shape=(len(all_train_ids), MAXS, IMG, IMG))\n log(f\"memmap {STACKS.shape} at {MM_PATH} ({STACKS.nbytes / 1e9:.1f} GB)\")\nelse:\n STACKS = None\n log(\"not enough disk for the memmap; every study will be rebuilt from its recipe\")\n\ndef decode_one(sid):\n vol, mask, rec = build_study(sid, SER_TRAIN, str(TRAIN_DIR), reader)\n return sid, vol, mask, rec\n\nt_p1 = time.time(); n_done = 0; truncated = False\nex = ThreadPoolExecutor(READ_THREADS)\n# Submit in bounded batches, never one future per study up front. A Future holds its result\n# alive until the Future object is dropped, and one decoded stack is 7.2 MB, so 4,407 queued\n# futures retain all 31.8 GB of decoded output in RAM at once and the kernel is killed at\n# study ~3,300. ex.map releases each result as it is consumed, so the in-flight footprint\n# stays at 2*READ_THREADS stacks.\nwhile True:\n if time.time() - T0 > PREP_BUDGET_S or time.time() > HARD_DEADLINE - 1800:\n truncated = True\n log(\"pass-1 budget reached; the rest will be decoded on demand\")\n break\n batch = decode_order[n_done:n_done + 2 * READ_THREADS]\n if not batch:\n break\n for sid, vol, mask, rec in ex.map(decode_one, batch):\n idx_of[sid] = n_done\n recipe[sid] = {\"mask\": mask.tobytes(), \"centers\": window_centers(mask), **rec}\n if USE_MM:\n STACKS[n_done] = vol\n n_done += 1\n if n_done % 250 == 0:\n rate = n_done / (time.time() - t_p1)\n log(f\"pass 1: {n_done:,}/{len(decode_order):,} {rate:.2f} studies/s \"\n f\"eta {(len(decode_order) - n_done) / max(rate, 1e-9) / 60:.0f} min\")\nex.shutdown(wait=False, cancel_futures=True)\nif USE_MM:\n STACKS.flush()\nTIMINGS[\"pass1_read_s\"] = round(time.time() - t_p1, 1)\nfacts([(\"studies decoded and cached\", n_done),\n (\"truncated by budget\", truncated),\n (\"stacks on disk (memmap)\", USE_MM),\n (\"recipe memory\", f\"{sum(sum(len(f) for f in r['files'] if f) for r in recipe.values()) / 1e6:.0f} MB of paths\")],\n \"Pass 1\")\nlog(f\"pass 1 done: {n_done:,} studies in {TIMINGS['pass1_read_s']:.0f}s\")"},{"cell_type":"markdown","id":"e9cf2a2f","metadata":{},"source":"## ๐Ÿ” Reproduction check: arm 1 against the radiologist\n\nThe number this whole notebook is anchored to. Their checkpoint stores 0.9167 for these 58\nstudies and their notebook's comment says 0.9054 for the same file; one of those two describes\na different run. Whatever this port prints here is the number that applies to this pipeline."},{"cell_type":"code","id":"68bf5efc","metadata":{},"execution_count":null,"outputs":[],"source":"TIMINGS[\"arm1_gold_start\"] = round(time.time() - T0, 1)\nt_g = time.time()\nrows = []\nfor i, s in enumerate(gold_ids):\n rows.append(infer_study(arm1, s, K_EVAL))\n if (i + 1) % 20 == 0:\n log(f\"arm-1 gold: {i + 1}/{len(gold_ids)}\")\nP1_gold = np.stack(rows)\npred1_gold = pd.DataFrame(P1_gold, index=gold_ids, columns=LABELS)\nauc1_gold = score_against_gold(pred1_gold)\nw_per1, w_mean1 = bootstrap_auc(gold, pred1_gold)\nTIMINGS[\"arm1_gold_s\"] = round(time.time() - t_g, 1)\nTIMINGS[\"s_per_study_eval\"] = TIMINGS[\"arm1_gold_s\"] / max(len(gold_ids), 1)\nlog(f\"eval speed: {TIMINGS['s_per_study_eval']:.1f} s/study at {K_EVAL} windows\")\n\nshow(pd.DataFrame({\"AUC on the 58\": auc1_gold}).sort_values(\"AUC on the 58\", ascending=False),\n f\"Arm 1 (their weights, this pipeline) against the radiologist, \"\n f\"mean {auc1_gold.mean():.3f} (95% CI width {w_mean1:.3f})\",\n bars=[\"AUC on the 58\"], vmin=0.4, vmax=1.0)\nrepro_ok = auc1_gold.mean() >= 0.88\nif repro_ok:\n log(f\"REPRODUCTION OK: arm-1 gold mean {auc1_gold.mean():.4f} \"\n f\"(their file says {float(ck_meta.get('gold_auc', np.nan)):.4f}, \"\n f\"their comment says 0.9054)\")\nelse:\n log(f\"REPRODUCTION FAILED: arm-1 gold mean {auc1_gold.mean():.4f} is far from \"\n f\"their 0.9054-0.9167. Read the preprocessing before trusting anything below.\")"},{"cell_type":"markdown","id":"8191d9e1","metadata":{},"source":"## ๐Ÿ“ˆ Arm 1 per finding: where the checkpoint is strong, and where it is not"},{"cell_type":"code","id":"93afec6f","metadata":{},"execution_count":null,"outputs":[],"source":"order1 = auc1_gold.sort_values()\nfig, ax = plt.subplots(figsize=(8, 4.5))\nbar_colors = [\"#d62728\" if v < 0.85 else \"#1f77b4\" for v in order1.values]\nax.barh(order1.index, order1.values, color=bar_colors)\nax.axvline(float(auc1_gold.mean()), color=\"k\", ls=\"--\", lw=1,\n label=f\"mean {auc1_gold.mean():.4f}\")\nax.set_xlim(0.5, 1.0)\nax.set_xlabel(\"AUC on the 58 radiologist studies\")\nax.set_title(\"Arm 1 (their checkpoint, this pipeline) per finding โ€” red is below 0.85\")\nax.legend(loc=\"lower right\")\nplt.tight_layout()\nplt.show()"},{"cell_type":"markdown","id":"73b470dc","metadata":{},"source":"## ๐Ÿš€ Rung 1: arm 1 on the test set\n\nThe first real submission, written the moment arm 1 has predicted the test set. On the public\nrun that is a handful of studies and takes seconds; on the scored run it is the hidden test, at\nroughly the per-study cost measured here. From this point on a scored run can only get better."},{"cell_type":"code","id":"64d04f76","metadata":{},"execution_count":null,"outputs":[],"source":"t_r1 = time.time()\nROOT = find_test_root()\nTEST_DIR = ROOT + (\"/test_series\" if os.path.isdir(ROOT + \"/test_series\") else \"/test_images\")\nSER_TEST = {k: v.to_dict(\"records\") for k, v in test_series.groupby(\"StudyInstanceUID\")}\nlog(f\"test root {ROOT} | series dir {TEST_DIR} | \"\n f\"Fluid_Sensitive column present: {'Fluid_Sensitive' in test_series.columns}\")\n\ntest_ids = list(test.StudyInstanceUID)\narm1_test = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\ntest_times = []\nn_fallback = 0\nfor i, s in enumerate(test_ids):\n t_s = time.time()\n try:\n arm1_test[i] = infer_study(arm1, s, K_EVAL)\n except Exception as e:\n n_fallback += 1\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n test_times.append(time.time() - t_s)\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-1 test: {i + 1}/{len(test_ids)} {np.median(test_times):.2f} s/study\")\nsub1 = write_submission(pd.DataFrame(arm1_test, index=test_ids, columns=LABELS))\nTIMINGS[\"rung1_s\"] = round(time.time() - t_r1, 1)\nlog(f\"RUNG 1 written: {len(sub1)} rows, arm 1 only, {n_fallback} fallback rows \"\n f\"({TIMINGS['rung1_s']:.0f}s)\")"},{"cell_type":"markdown","id":"88db2e1e","metadata":{},"source":"## ๐Ÿ“ Arm 1 on the holdout\n\nThe blend weight later is chosen here, not on the 58, so arm 1 needs scores for the same\nholdout the fine-tuned arm will be measured on. If pass 1 was truncated, the holdout shrinks to\nthe studies that were decoded, which is still a random sample of it."},{"cell_type":"code","id":"121a8c3e","metadata":{},"execution_count":null,"outputs":[],"source":"ho_ids = [s for s in holdout_ids if s in idx_of]\nif len(ho_ids) < len(holdout_ids):\n log(f\"holdout truncated to {len(ho_ids)} of {len(holdout_ids)} decoded studies\")\nrng.shuffle(ho_ids) # a time-guarded truncation below must stay a random sample\nt_h = time.time()\n# Deadline: the fine-tune owns the clock from FT_END_BY, so this eval must end 1.2 h before\n# it. At a pathological 41 s/study, 832 studies would cost 9.5 h โ€” the guard, not the list\n# length, decides how many actually run.\nho_ids, P1_ho = infer_guarded(arm1, ho_ids, K_EVAL, FT_END_BY - 1.2 * 3600, \"arm-1 holdout\") \\\n if ho_ids else ([], np.zeros((0, 12), np.float32))\nTIMINGS[\"arm1_holdout_s\"] = round(time.time() - t_h, 1)\nY_ho_bin = (v5.reindex(ho_ids).fillna(0.0).values > 0.5).astype(int) if ho_ids else np.zeros((0, 12), int)\narm1_ho_weak = mean_auc(Y_ho_bin, P1_ho) if ho_ids else np.nan\nlog(f\"arm 1 on the holdout: {len(ho_ids)} studies, weak-label mean AUC {arm1_ho_weak:.4f} \"\n f\"({TIMINGS['arm1_holdout_s']:.0f}s)\")\nnp.savez_compressed(\"/kaggle/tmp/arm1_holdout.npz\", ids=np.array(ho_ids), probs=P1_ho)"},{"cell_type":"markdown","id":"ff577393","metadata":{},"source":"## ๐Ÿ” Arm 1 on the test set, second pass (TTA)\n\nTest-time augmentation, kept cheap: the eval windows are evenly spaced over a study's valid\nrange, so a second pass with every center shifted one slice is a genuinely different view of\nthe same study, and averaging the two passes is the standard free lunch. Test only โ€” the\nholdout and the 58 stay single-pass so every selection number stays comparable to v2, and\nthe pass runs now because arm 1 is freed for the fine-tune right after this."},{"cell_type":"code","id":"d0a3a6a1","metadata":{},"execution_count":null,"outputs":[],"source":"arm1_test_b = None\nif TTA_TEST > 1 and test_ids and \\\n time.time() + len(test_ids) * float(np.median(test_times or [1.7])) < FT_END_BY - 1.2 * 3600:\n t_b = time.time()\n arm1_test_b = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\n for i, s in enumerate(test_ids):\n try:\n arm1_test_b[i] = infer_study(arm1, s, K_EVAL, shift=1)\n except Exception as e:\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-1 test pass 2: {i + 1}/{len(test_ids)}\")\n TIMINGS[\"arm1_test_tta_s\"] = round(time.time() - t_b, 1)\n log(f\"arm-1 TTA pass done ({TIMINGS['arm1_test_tta_s']:.0f}s)\")\nelse:\n log(\"arm-1 TTA pass skipped: TTA_TEST=1, or the clock belongs to the fine-tune\")"},{"cell_type":"markdown","id":"c88defca","metadata":{},"source":"## ๐ŸŽฏ Fine-tune arm 2 from the checkpoint\n\nThe second arm starts from their weights and continues on the v5 labels, with three deliberate\ndepartures from a from-scratch run. The learning rate is 1e-5, flat-ish and low, because the\nhead is already trained on a near-identical target and the risk is damage, not underfitting.\nThere is no horizontal flip, because a flip turns a left knee into a right knee and relabels\nmedial as lateral (the first notebook's finding), and no geometric augmentation at all, because\nthe 140 mm crop already fixes scale; the augmentation is which 12 of a study's valid windows\nare sampled this epoch. And the gate that picks the epoch is the report-hash holdout against\nthe v5 labels, in both flag modes, so selection is never done on the 58 and, in the\ninclude-gold run, never on data the model saw.\n\nOne study forward at a time (its 12 windows pooled by the attention head), four studies per\noptimizer step. Arm 1 is freed first: one model resident at a time."},{"cell_type":"code","id":"7b600cf6","metadata":{},"execution_count":null,"outputs":[],"source":"del arm1; gc.collect()\nif dev == \"cuda\":\n torch.cuda.empty_cache()\narm2, RES2, _ = load_model(weight_path, dev)\nassert RES2 == RES\n\nv5_ft = v5.reindex(ft_ids).fillna(0.0).astype(np.float32)\nY_ft = torch.from_numpy(v5_ft.values).to(dev)\nlog(f\"fine-tune set {len(ft_ids):,} studies, target mean {v5_ft.values.mean():.4f}\")\n\nopt = torch.optim.AdamW(arm2.parameters(), lr=LR_FT, weight_decay=WD_FT)\nTOTAL_STEPS = EPOCHS_FT * int(np.ceil(len(ft_ids) / ACCUM))\ndef lr_lambda(step):\n if step < WARMUP_FT:\n return step / max(1, WARMUP_FT)\n p = (step - WARMUP_FT) / max(1, TOTAL_STEPS - WARMUP_FT)\n return 0.5 * (1.0 + math.cos(math.pi * min(1.0, p)))\nsched = torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)\nscaler = torch.amp.GradScaler(\"cuda\", enabled=(dev == \"cuda\"))\n\n_MEAN_D, _STD_D = _MEAN.to(dev), _STD.to(dev)\n\ndef win_uint8(vol, centers):\n w = np.empty((len(centers), 3, IMG, IMG), np.uint8)\n for j, c in enumerate(centers):\n c = max(1, min(int(c), MAXS - 2))\n w[j, 0] = vol[c - 1]; w[j, 1] = vol[c]; w[j, 2] = vol[c + 1]\n return w\n\ndef windows_to_gpu(w):\n x = torch.from_numpy(w).to(dev).float().div_(255)\n if x.shape[-1] != RES:\n x = F.interpolate(x, size=(RES, RES), mode=\"bilinear\", align_corners=False)\n return (x - _MEAN_D) / _STD_D\n\ndef gate_auc(model, ids, y_bin, k):\n if not ids:\n return np.nan, np.zeros((0, len(LABELS)), np.float32)\n P = np.stack([infer_study(model, s, k) for s in ids])\n return mean_auc(y_bin, P), P\n\ng_eval = np.random.default_rng(SEED + 7)\n# The gate runs once per epoch, so its size is set from the measured eval speed, not a\n# constant: a ~3 min gate per epoch at whatever s/study the box actually delivers.\ns_per_eval = TIMINGS.get(\"s_per_study_eval\", 12.0) * K_GATE / K_EVAL\nn_gate = int(np.clip(180 / max(s_per_eval, 1e-6), 100, 500))\nho_eval_ids = list(g_eval.choice(ho_ids, min(n_gate, len(ho_ids)), replace=False)) if ho_ids else []\nY_ho_eval = (v5.reindex(ho_eval_ids).fillna(0.0).values > 0.5).astype(int) if ho_eval_ids else np.zeros((0, 12), int)\nlog(f\"epoch gate: {len(ho_eval_ids)} holdout studies at {K_GATE} windows \"\n f\"({s_per_eval:.1f} s/study measured at {K_EVAL})\")"},{"cell_type":"code","id":"8ab519d7","metadata":{},"execution_count":null,"outputs":[],"source":"history, snaps = [], []\nTIMINGS[\"finetune_start\"] = round(time.time() - T0, 1)\nfor epoch in range(EPOCHS_FT):\n if time.time() > FT_END_BY:\n log(\"fine-tune budget spent before this epoch; keeping what is trained\")\n break\n g = np.random.default_rng(SEED * 1000 + epoch)\n order = g.permutation(len(ft_ids))\n arm2.train()\n t_ep = time.time(); run = 0.0; seen = 0\n for n_i, i in enumerate(order):\n if time.time() > FT_END_BY:\n log(f\"epoch {epoch}: time guard mid-epoch at study {n_i}\")\n break\n if n_i == 50:\n per = (time.time() - t_ep) / 50\n if t_ep + per * len(order) > FT_END_BY:\n log(f\"epoch {epoch}: projected {per * len(order) / 60:.0f} min does not fit; \"\n f\"stopping fine-tune after {seen} studies\")\n break\n sid = ft_ids[i]\n vol, mask = get_stack(sid)\n pool = recipe[sid][\"centers\"] if sid in recipe else window_centers(mask)\n take = min(W_WINDOWS, len(pool))\n cs = g.choice(pool, size=take, replace=False) if take else [MAXS // 2]\n x = windows_to_gpu(win_uint8(vol, cs))\n with torch.autocast(\"cuda\", enabled=(dev == \"cuda\")):\n logits = arm2(x[None])\n loss = F.binary_cross_entropy_with_logits(logits[0], Y_ft[i]) / ACCUM\n scaler.scale(loss).backward()\n run += float(loss.item()) * ACCUM; seen += 1\n if seen % ACCUM == 0:\n scaler.unscale_(opt)\n torch.nn.utils.clip_grad_norm_(arm2.parameters(), 1.0)\n scaler.step(opt); scaler.update(); sched.step()\n opt.zero_grad(set_to_none=True)\n if seen == 0:\n break\n gate_v, _ = gate_auc(arm2, ho_eval_ids, Y_ho_eval, K_GATE)\n gold_v = np.nan\n # Per-epoch gold trend at the cheap K_GATE resolution only โ€” it selects nothing, and the\n # headline arm-2 gold number is measured at K_EVAL in the evaluation cell. 42 windows here\n # would cost ~40 min per epoch for a number this notebook never uses to decide anything.\n if not INCLUDE_GOLD and gold_ids and time.time() < FT_END_BY + 600:\n gold_v, _ = gate_auc(arm2, gold_ids, gold.values.astype(int), K_GATE)\n snaps.append((gate_v, {k: v.detach().cpu().clone() for k, v in arm2.state_dict().items()}))\n history.append({\"epoch\": epoch + 1, \"loss\": run / max(seen, 1),\n \"holdout_weak_auc\": gate_v, \"gold58_auc\": gold_v, \"studies\": seen})\n torch.save(snaps[-1][1], f\"/kaggle/tmp/ft_epoch{epoch}.pt\")\n log(f\"epoch {epoch + 1}/{EPOCHS_FT} loss {run / max(seen, 1):.4f} \"\n f\"holdout-weak AUC {gate_v:.4f} gold-58 {gold_v if np.isfinite(gold_v) else float('nan'):.4f} \"\n f\"({(time.time() - t_ep) / 60:.0f} min, {seen} studies)\")\nTIMINGS[\"finetune_s\"] = round(time.time() - T0 - TIMINGS[\"finetune_start\"], 1)\n\nhist_df = pd.DataFrame(history)\nif len(hist_df):\n show(hist_df, \"Fine-tune history (selection column: holdout_weak_auc)\")\n\nbest_epoch = -1\narm2_ready = False\nif snaps:\n best_epoch = int(max(range(len(snaps)), key=lambda i: snaps[i][0]))\n arm2.load_state_dict(snaps[best_epoch][1])\n arm2_ready = True\n log(f\"selected epoch {best_epoch + 1} by holdout-weak AUC \"\n f\"({snaps[best_epoch][0]:.4f} vs last {snaps[-1][0]:.4f})\")\nelse:\n log(\"no epoch completed; rung 1 stands\")"},{"cell_type":"markdown","id":"d1f1ac7d","metadata":{},"source":"## โš–๏ธ Arm 2 measured, then the blend weight\n\nArm 2 gets the same full-window measurement arm 1 got. In the honest mode the gold-58 numbers\nare the radiologist-grade check, with the bootstrap interval; in the include-gold mode they sit\nunder a contamination banner, printed so the log says plainly what the model saw.\n\nThe blend weight comes from the holdout, never from the 58. One caveat is worth saying in\nwords: both arms are trained on the same label family, so the holdout measures agreement with\nv5, and a gain measured against v5 is an upper bound on the gain against a radiologist. Their\nown live-board experience was that blending near-identical arms bought about +0.001."},{"cell_type":"code","id":"723ad247","metadata":{},"execution_count":null,"outputs":[],"source":"P2_ho = P2_gold = None\narm2_ho_weak = arm2_gold_mean = np.nan\nif arm2_ready and ho_ids and time.time() < EVAL_END_BY:\n t_a2 = time.time()\n used2, P2_ho = infer_guarded(arm2, ho_ids, K_EVAL, EVAL_END_BY, \"arm-2 holdout\")\n if len(used2) < len(ho_ids): # keep arm 1 / arm 2 / Y row-aligned for the blend\n ho_ids = used2\n P1_ho = P1_ho[:len(ho_ids)]\n Y_ho_bin = Y_ho_bin[:len(ho_ids)]\n arm2_ho_weak = mean_auc(Y_ho_bin, P2_ho) if ho_ids else np.nan\n log(f\"arm 2 on the holdout: weak-label mean AUC {arm2_ho_weak:.4f} \"\n f\"({time.time() - t_a2:.0f}s)\")\n # The 58 at 42 windows costs the same per study as any holdout eval; check the projection\n # before starting it rather than discovering the overrun at the deadline.\n est_gold = len(gold_ids) * TIMINGS.get(\"s_per_study_eval\", 12.0)\n if time.time() + est_gold < EVAL_END_BY:\n rows = []\n for i, s in enumerate(gold_ids):\n rows.append(infer_study(arm2, s, K_EVAL))\n P2_gold = np.stack(rows)\n pred2_gold = pd.DataFrame(P2_gold, index=gold_ids, columns=LABELS)\n auc2_gold = score_against_gold(pred2_gold)\n arm2_gold_mean = float(auc2_gold.mean())\n else:\n log(f\"arm-2 gold eval skipped: {est_gold:.0f}s projection past the eval budget\")\n if P2_gold is None:\n log(\"arm-2 gold numbers unavailable (eval budget spent)\")\n elif INCLUDE_GOLD:\n note(f\"**CONTAMINATED**: these 58 rows were in fine-tuning, so \"\n f\"{arm2_gold_mean:.3f} is not a measurement.\")\n else:\n w2, wm2 = bootstrap_auc(gold, pred2_gold)\n show(pd.DataFrame({\"arm 1\": auc1_gold, \"arm 2\": auc2_gold}).sort_values(\"arm 2\"),\n f\"The two arms against the radiologist: {auc1_gold.mean():.3f} vs \"\n f\"{arm2_gold_mean:.3f}\", bars=[\"arm 2\"], vmin=0.4, vmax=1.0)\n log(f\"arm-2 gold mean {arm2_gold_mean:.4f} (95% CI width {wm2:.3f})\")\nelif not arm2_ready:\n log(\"arm 2 skipped: no fine-tune epoch completed\")\nelse:\n log(\"arm 2 eval skipped: evaluation budget spent\")\n\nW_BLEND = 0.0\nblend_ho_weak = np.nan\nif P2_ho is not None and ho_ids:\n ws = np.round(np.arange(0, 1.001, 0.05), 2)\n scores = [mean_auc(Y_ho_bin, blend_ranks(P2_ho, P1_ho, w)) for w in ws]\n curve = pd.DataFrame({\"w (arm 2 weight)\": ws, \"holdout weak AUC\": scores})\n W_BLEND = float(curve.loc[curve[\"holdout weak AUC\"].idxmax(), \"w (arm 2 weight)\"])\n blend_ho_weak = float(curve[\"holdout weak AUC\"].max())\n show(curve, \"Blend weight, chosen on the holdout (never on the 58)\",\n bars=[\"holdout weak AUC\"], vmin=min(scores) - 0.005, vmax=max(scores) + 0.005)\n facts([(\"arm 1 alone (w=0)\", f\"{scores[0]:.4f}\"),\n (\"arm 2 alone (w=1)\", f\"{scores[-1]:.4f}\"),\n (f\"blend at w={W_BLEND}\", f\"{blend_ho_weak:.4f}\"),\n (\"blend gain over the better arm\", f\"{blend_ho_weak - max(scores[0], scores[-1]):+.4f}\")],\n \"What blending is worth on this label family\")\n note(\"Both arms share the same labels, architecture and preprocessing, so this gain is an \"\n \"upper bound on the radiologist-grade gain; the published arms that shared everything \"\n \"gained about +0.001 on the live board.\")\n if not INCLUDE_GOLD and P2_gold is not None:\n G = blend_ranks(P2_gold, P1_gold, W_BLEND)\n blend_gold = score_against_gold(pd.DataFrame(G, index=gold_ids, columns=LABELS))\n log(f\"blend gold mean {blend_gold.mean():.4f} (reported, not selected on)\")\nelse:\n log(\"blend skipped: arm 2 has no scores; W_BLEND stays 0.0\")"},{"cell_type":"markdown","id":"dcc3820a","metadata":{},"source":"## ๐Ÿ“ˆ Two arms per finding, and what the blend weight is worth"},{"cell_type":"code","id":"97168a3c","metadata":{},"execution_count":null,"outputs":[],"source":"if np.isfinite(arm2_gold_mean):\n fig, ax_l = plt.subplots(figsize=(7.5, 4.5))\n xs = np.arange(len(LABELS))\n ax_l.bar(xs - 0.2, auc1_gold.loc[LABELS].values, 0.4,\n label=f\"arm 1 (mean {auc1_gold.mean():.4f})\")\n ax_l.bar(xs + 0.2, auc2_gold.loc[LABELS].values, 0.4,\n label=f\"arm 2 (mean {arm2_gold_mean:.4f})\")\n ax_l.set_xticks(xs)\n ax_l.set_xticklabels(LABELS, rotation=45, ha=\"right\")\n ax_l.set_ylim(0.5, 1.0)\n ax_l.set_title(\"Per-finding AUC on the 58\" +\n (\" โ€” arm 2 saw them in training (contaminated)\" if INCLUDE_GOLD else \"\"))\n ax_l.legend(loc=\"lower right\")\n plt.tight_layout()\n plt.show()\nif np.isfinite(blend_ho_weak):\n fig, ax_r = plt.subplots(figsize=(6.5, 4))\n ax_r.plot(ws, scores, \"o-\")\n ax_r.axvline(W_BLEND, color=\"r\", ls=\"--\", lw=1, label=f\"w* = {W_BLEND:.2f}\")\n ax_r.set_xlabel(\"arm-2 weight in the rank blend\")\n ax_r.set_ylabel(\"holdout weak-label AUC\")\n ax_r.set_title(\"Blend weight, chosen on the holdout (never on the 58)\")\n ax_r.legend()\n plt.tight_layout()\n plt.show()\nif not np.isfinite(arm2_gold_mean) and not np.isfinite(blend_ho_weak):\n log(\"arm-2 measurements unavailable this run; no two-arm charts\")"},{"cell_type":"markdown","id":"c0516f81","metadata":{},"source":"## ๐Ÿ Rung 2: the blended submission\n\nAll-or-nothing, by the same rule the first notebook used for its model stage: the projected\ncost of arm 2 on the test set is measured from arm 1's per-study time, and if it does not fit\nbefore the final-write deadline, submission.csv is not touched. A scored run that loses arm 2\nstill has rung 1 on disk."},{"cell_type":"code","id":"dee05fc1","metadata":{},"execution_count":null,"outputs":[],"source":"ran_rung2 = False\nif arm2_ready and P2_ho is not None and time.time() < TEST_START_BY:\n per_study = float(np.median(test_times)) if test_times else 1.0\n projected = time.time() + len(test_ids) * per_study * 1.5 + 300\n if projected < FINAL_WRITE_BY:\n t_r2 = time.time()\n arm2_test = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\n for i, s in enumerate(test_ids):\n try:\n arm2_test[i] = infer_study(arm2, s, K_EVAL)\n except Exception as e:\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-2 test: {i + 1}/{len(test_ids)}\")\n ranks = blend_ranks(arm2_test, arm1_test, W_BLEND)\n sub2 = write_submission(pd.DataFrame(ranks, index=test_ids, columns=LABELS))\n assert np.isfinite(sub2[LABELS].values).all(), \"non-finite value in the blended submission\"\n TIMINGS[\"rung2_s\"] = round(time.time() - t_r2, 1)\n ran_rung2 = True\n log(f\"RUNG 2 written: blend at w={W_BLEND}, arm 1 + arm 2 ({TIMINGS['rung2_s']:.0f}s)\")\n else:\n log(f\"rung 2 skipped: projected finish {projected - T0:.0f}s is past the \"\n f\"final-write deadline; rung 1 stands\")\nelse:\n log(\"rung 2 skipped: no fine-tuned arm or too late; rung 1 stands\")"},{"cell_type":"markdown","id":"3149be59","metadata":{},"source":"## ๐Ÿ” Arm 2 on the test set, second pass (TTA)\n\nSame shifted-center second pass as arm 1 got, while arm 2 is still resident."},{"cell_type":"code","id":"46ab412c","metadata":{},"execution_count":null,"outputs":[],"source":"arm2_test_b = None\nif ran_rung2 and TTA_TEST > 1 and test_ids:\n per_study_b = float(np.median(test_times)) if test_times else 1.7\n if time.time() + len(test_ids) * per_study_b * 1.5 + 300 < FINAL_WRITE_BY - 3600:\n t_b2 = time.time()\n arm2_test_b = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\n for i, s in enumerate(test_ids):\n try:\n arm2_test_b[i] = infer_study(arm2, s, K_EVAL, shift=1)\n except Exception as e:\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-2 test pass 2: {i + 1}/{len(test_ids)}\")\n TIMINGS[\"arm2_test_tta_s\"] = round(time.time() - t_b2, 1)\n log(f\"arm-2 TTA pass done ({TIMINGS['arm2_test_tta_s']:.0f}s)\")\n else:\n log(\"arm-2 TTA pass skipped: the projection crosses the arm-3 window\")\nelse:\n log(\"arm-2 TTA pass skipped: rung 2 did not run, or TTA_TEST=1\")"},{"cell_type":"markdown","id":"1ca04425","metadata":{},"source":"## ๐Ÿงฌ Arm 3: the SWA checkpoint\n\nThe dataset ships a second checkpoint, `raptor_ft_coatnet_v4_full_swa.pt` โ€” stochastic\nweight averaging over the same training run. Same architecture, same label family, a\ndifferent point in weight space: the cheapest diversity there is. It gets no fine-tuning;\nit is a third opinion, measured exactly like the other two."},{"cell_type":"code","id":"8c4164df","metadata":{},"execution_count":null,"outputs":[],"source":"P3_ho = P3_gold = None\narm3_ho_weak = arm3_gold_mean = np.nan\nweight_path_swa = find_weight_file(CKPT_FILE_SWA)\narm3, RES3, ck3 = load_model(weight_path_swa, dev)\nassert RES3 == RES, f\"SWA checkpoint res {RES3} != {RES}\"\nif \"lab\" in ck3:\n assert list(ck3[\"lab\"]) == LABELS, \"SWA checkpoint label order differs\"\nlog(f\"arm 3 (SWA) loaded: stored gold_auc {ck3.get('gold_auc')}, epoch {ck3.get('epoch')}\")\nif ho_ids and time.time() < EVAL_END_BY:\n t_a3 = time.time()\n used3, P3_ho = infer_guarded(arm3, ho_ids, K_EVAL, EVAL_END_BY, \"arm-3 holdout\")\n if len(used3) < len(ho_ids): # keep every arm and Y row-aligned for the blend\n ho_ids = used3\n P1_ho = P1_ho[:len(ho_ids)]\n if P2_ho is not None:\n P2_ho = P2_ho[:len(ho_ids)]\n Y_ho_bin = Y_ho_bin[:len(ho_ids)]\n arm3_ho_weak = mean_auc(Y_ho_bin, P3_ho) if ho_ids else np.nan\n log(f\"arm 3 on the holdout: weak-label mean AUC {arm3_ho_weak:.4f} \"\n f\"({time.time() - t_a3:.0f}s)\")\n est_gold3 = len(gold_ids) * TIMINGS.get(\"s_per_study_eval\", 12.0)\n if time.time() + est_gold3 < EVAL_END_BY:\n P3_gold = np.stack([infer_study(arm3, s, K_EVAL) for s in gold_ids])\n auc3_gold = score_against_gold(pd.DataFrame(P3_gold, index=gold_ids, columns=LABELS))\n arm3_gold_mean = float(auc3_gold.mean())\n log(f\"arm-3 gold mean {arm3_gold_mean:.4f}\")\n else:\n log(\"arm-3 gold eval skipped: projection past the eval budget\")\nelse:\n log(\"arm 3 evals skipped: no decoded holdout, or the eval budget is spent\")"},{"cell_type":"markdown","id":"30d3a649","metadata":{},"source":"## ๐Ÿค The three-way blend, weights chosen on the holdout\n\nA simplex grid over (arm 1, arm 2, arm 3) in steps of 0.1, scored on the same holdout rows\nthe two-way weight came from. An arm with no holdout scores is forced to weight zero, so\nthe grid degrades to the two-way or one-way case instead of failing."},{"cell_type":"code","id":"d1038b5a","metadata":{},"execution_count":null,"outputs":[],"source":"W3 = (1.0, 0.0, 0.0)\nblend3_ho_weak = np.nan\nif ho_ids and (P2_ho is not None or P3_ho is not None):\n R1 = rankpct(np.clip(P1_ho, 0, 1))\n R2 = rankpct(np.clip(P2_ho, 0, 1)) if P2_ho is not None else None\n R3 = rankpct(np.clip(P3_ho, 0, 1)) if P3_ho is not None else None\n grid = []\n for w2i in range(0, 11):\n for w3i in range(0, 11 - w2i):\n w2, w3 = w2i / 10.0, w3i / 10.0\n if (R2 is None and w2 > 0) or (R3 is None and w3 > 0):\n continue\n w1 = round(1.0 - w2 - w3, 1)\n mix = w1 * R1 + (w2 * R2 if R2 is not None else 0.0) + \\\n (w3 * R3 if R3 is not None else 0.0)\n grid.append(((w1, w2, w3), mean_auc(Y_ho_bin, mix)))\n W3, blend3_ho_weak = max(grid, key=lambda t: t[1])\n g_lo = min(g[1] for g in grid) - 0.005\n g_hi = max(g[1] for g in grid) + 0.005\n top5 = sorted(grid, key=lambda t: -t[1])[:5]\n show(pd.DataFrame({\"w (arm 1)\": [t[0][0] for t in top5],\n \"w (arm 2)\": [t[0][1] for t in top5],\n \"w (arm 3)\": [t[0][2] for t in top5],\n \"holdout weak AUC\": [round(t[1], 4) for t in top5]}),\n f\"Best three-way blends on the holdout โ€” w* = {W3}, weak AUC {blend3_ho_weak:.4f}\",\n bars=[\"holdout weak AUC\"], vmin=g_lo, vmax=g_hi)\n log(f\"3-way blend: w*={W3} holdout weak AUC {blend3_ho_weak:.4f} \"\n f\"(two-way was {blend_ho_weak:.4f}, arm 1 alone {scores[0] if np.isfinite(blend_ho_weak) else arm1_ho_weak:.4f})\")\nelse:\n log(\"3-way blend skipped: no second arm has holdout scores; W3 stays (1, 0, 0)\")"},{"cell_type":"markdown","id":"fb811df7","metadata":{},"source":"## ๐Ÿ† Rung 3: the three-arm submission\n\nAll-or-nothing, same rule as rung 2: arm 3 predicts the test set (two center-shifted\npasses when TTA_TEST allows), the three arms blend at the holdout-chosen weights, and if\nthe projection does not fit before the final-write deadline, submission.csv is not\ntouched and rung 2 stands."},{"cell_type":"code","id":"18b27f7a","metadata":{},"execution_count":null,"outputs":[],"source":"ran_rung3 = False\nif ho_ids and P3_ho is not None and time.time() < TEST_START_BY:\n per_study3 = float(np.median(test_times)) if test_times else 1.7\n projected3 = time.time() + len(test_ids) * per_study3 * TTA_TEST * 1.5 + 300\n if projected3 < FINAL_WRITE_BY:\n t_r3 = time.time()\n arm3_test = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\n for i, s in enumerate(test_ids):\n try:\n arm3_test[i] = infer_study(arm3, s, K_EVAL)\n except Exception as e:\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-3 test: {i + 1}/{len(test_ids)}\")\n arm3_test_b = None\n if TTA_TEST > 1 and time.time() + len(test_ids) * per_study3 * 1.5 + 300 < FINAL_WRITE_BY:\n arm3_test_b = np.full((len(test_ids), len(LABELS)), 0.5, np.float32)\n for i, s in enumerate(test_ids):\n try:\n arm3_test_b[i] = infer_study(arm3, s, K_EVAL, shift=1)\n except Exception as e:\n log(f\" study {i} {short_uid(s)} FALLBACK ({type(e).__name__}: {e})\")\n if (i + 1) % 100 == 0 or i + 1 == len(test_ids):\n log(f\"arm-3 test pass 2: {i + 1}/{len(test_ids)}\")\n del arm3; gc.collect()\n if dev == \"cuda\":\n torch.cuda.empty_cache()\n r1 = rankpct(np.clip(arm1_test if arm1_test_b is None else (arm1_test + arm1_test_b) / 2, 0, 1))\n r2 = rankpct(np.clip(arm2_test if arm2_test_b is None else (arm2_test + arm2_test_b) / 2, 0, 1)) \\\n if ran_rung2 else None\n r3 = rankpct(np.clip(arm3_test if arm3_test_b is None else (arm3_test + arm3_test_b) / 2, 0, 1))\n w1, w2, w3 = W3\n w_sum = w1 + (w2 if r2 is not None else 0.0) + w3\n ranks3 = (w1 * r1 + (w2 * r2 if r2 is not None else 0.0) + w3 * r3) / w_sum\n sub3 = write_submission(pd.DataFrame(ranks3, index=test_ids, columns=LABELS))\n assert np.isfinite(sub3[LABELS].values).all(), \"non-finite value in the 3-way submission\"\n TIMINGS[\"rung3_s\"] = round(time.time() - t_r3, 1)\n ran_rung3 = True\n log(f\"RUNG 3 written: 3-way blend w={W3}, TTA passes={TTA_TEST} \"\n f\"({TIMINGS['rung3_s']:.0f}s)\")\n else:\n log(f\"rung 3 skipped: projected finish {projected3 - T0:.0f}s is past the \"\n f\"final-write deadline; rung 2 stands\")\nelse:\n log(\"rung 3 skipped: no arm-3 scores or too late; rung 2 stands\")"},{"cell_type":"markdown","id":"a8bc4c94","metadata":{},"source":"## ๐Ÿ–ผ๏ธ What one study looks like to the model"},{"cell_type":"code","id":"2d54cb78","metadata":{},"execution_count":null,"outputs":[],"source":"if USE_MM and n_done and gold_ids[0] in idx_of:\n vol0 = np.asarray(STACKS[idx_of[gold_ids[0]]])\n slot_names = [\"Sagittal FS\", \"Sagittal\", \"Coronal FS\", \"Coronal\", \"Axial\"]\n bounds = np.cumsum([s[2] for s in SLOTS])\n picks = np.linspace(0, MAXS - 1, 16).round().astype(int)\n fig, axes = plt.subplots(4, 4, figsize=(8.5, 8.5))\n for a, i in zip(axes.flat, picks):\n a.imshow(vol0[i], cmap=\"gray\")\n a.set_title(f\"slot {i}: {slot_names[int(np.searchsorted(bounds, i, side='right'))]}\",\n fontsize=8)\n a.axis(\"off\")\n fig.suptitle(f\"16 of {MAXS} slots, one radiologist-labelled study \"\n f\"({short_uid(gold_ids[0])}) โ€” decoded first in pass 1\")\n plt.tight_layout()\n plt.show()\nelse:\n log(\"no decoded stacks cached this run; skipping the study grid\")"},{"cell_type":"markdown","id":"7b86a702","metadata":{},"source":"## ๐Ÿ“Š The measurements this run leaves behind"},{"cell_type":"code","id":"02a04fad","metadata":{},"execution_count":null,"outputs":[],"source":"sub_final = pd.read_csv(\"submission.csv\")\nassert list(sub_final.columns) == [\"StudyInstanceUID\"] + LABELS\nassert len(sub_final) == len(test)\nassert sub_final[LABELS].notna().all().all()\n\nMEAS = {\n \"include_gold\": INCLUDE_GOLD,\n \"holdout_frac\": HOLDOUT_FRAC,\n \"n_decoded\": n_done,\n \"pass1_truncated\": truncated,\n \"use_memmap\": USE_MM,\n \"n_holdout_used\": len(ho_ids),\n \"n_train_ft\": len(ft_ids),\n \"epochs_run\": len(history),\n \"windows_per_epoch\": W_WINDOWS,\n \"batch_studies\": ACCUM,\n \"lr\": LR_FT,\n \"selected_epoch\": best_epoch + 1 if best_epoch >= 0 else None,\n \"ckpt_file\": CKPT_FILE,\n \"ckpt_gold_auc\": float(ck_meta.get(\"gold_auc\", np.nan)),\n \"ckpt_epoch\": ck_meta.get(\"epoch\"),\n \"arch\": str(ck_meta.get(\"arch\")),\n \"res\": RES,\n \"label_file\": LABEL_FILE,\n \"v5_contains_gold\": bool(check[\"contains the answers\"]),\n \"arm1_gold_mean\": float(auc1_gold.mean()),\n \"arm1_gold_ci95_width\": round(w_mean1, 4),\n \"arm1_holdout_weak\": None if not np.isfinite(arm1_ho_weak) else round(arm1_ho_weak, 4),\n \"arm2_gold_mean\": None if not np.isfinite(arm2_gold_mean) else round(arm2_gold_mean, 4),\n \"arm2_gold_contaminated\": INCLUDE_GOLD,\n \"arm2_holdout_weak\": None if not np.isfinite(arm2_ho_weak) else round(arm2_ho_weak, 4),\n \"blend_w\": W_BLEND,\n \"blend_holdout_weak\": None if not np.isfinite(blend_ho_weak) else round(blend_ho_weak, 4),\n \"arm3_swa_file\": CKPT_FILE_SWA,\n \"arm3_gold_mean\": None if not np.isfinite(arm3_gold_mean) else round(arm3_gold_mean, 4),\n \"arm3_holdout_weak\": None if not np.isfinite(arm3_ho_weak) else round(arm3_ho_weak, 4),\n \"blend3_w\": W3,\n \"blend3_holdout_weak\": None if not np.isfinite(blend3_ho_weak) else round(blend3_ho_weak, 4),\n \"tta_test_passes\": TTA_TEST,\n \"rung_written\": 3 if ran_rung3 else (2 if ran_rung2 else 1),\n \"per_finding_arm1_gold\": {c: round(float(v), 4) for c, v in auc1_gold.items()},\n \"timings\": TIMINGS,\n \"seed\": SEED,\n \"versions\": {\"torch\": torch.__version__, \"timm\": timm.__version__,\n \"pydicom\": pydicom.__version__, \"cv2\": cv2.__version__},\n}\nfacts([(\"run type\", \"submission (gold in fine-tune)\" if INCLUDE_GOLD else \"measurement (gold held out)\"),\n (\"studies decoded\", n_done),\n (\"fine-tune epochs\", len(history)),\n (\"arm 1 gold mean\", round(float(auc1_gold.mean()), 4)),\n (\"arm 2 gold mean\", None if not np.isfinite(arm2_gold_mean) else round(arm2_gold_mean, 4)),\n (\"arm 3 gold mean (SWA)\", None if not np.isfinite(arm3_gold_mean) else round(arm3_gold_mean, 4)),\n (\"blend weight (2-way)\", W_BLEND),\n (\"blend weights (3-way)\", W3),\n (\"submission rung\",\n \"3 (three-arm blend)\" if ran_rung3 else (\"2 (blend)\" if ran_rung2 else \"1 (arm 1 only)\"))],\n \"This run\")\nlog(\"MEASUREMENT_JSON \" + json.dumps(MEAS, separators=(\",\", \":\"), default=str))\nlog(f\"submission.csv ready: {len(sub_final)} rows x {len(sub_final.columns)} cols\")"},{"cell_type":"markdown","id":"f7430ce4","metadata":{},"source":"## โฑ๏ธ Where the hours went"},{"cell_type":"code","id":"3fc8e366","metadata":{},"execution_count":null,"outputs":[],"source":"phases = [(\"pass1_read_s\", \"pass 1: decode every study\"), (\"arm1_gold_s\", \"arm 1 on the 58\"),\n (\"arm1_holdout_s\", \"arm 1 on the holdout\"), (\"finetune_s\", \"fine-tune arm 2\"),\n (\"rung1_s\", \"rung 1: arm 1 on test\"), (\"arm1_test_tta_s\", \"arm 1 test, TTA pass 2\"),\n (\"rung2_s\", \"rung 2: blend on test\"), (\"arm2_test_tta_s\", \"arm 2 test, TTA pass 2\"),\n (\"rung3_s\", \"rung 3: three-arm test + blend\")]\ntimed = [(nm, TIMINGS[k] / 60.0) for k, nm in phases if k in TIMINGS]\nif timed:\n fig, ax = plt.subplots(figsize=(8, 3.2))\n ax.barh([t[0] for t in timed], [t[1] for t in timed])\n ax.set_xlabel(\"minutes\")\n ax.set_title(f\"one T4, total {(time.time() - T0) / 3600:.1f} h\")\n plt.tight_layout()\n plt.show()"},{"cell_type":"markdown","id":"88e43913","metadata":{},"source":"## ๐Ÿ’ก What the runs behind 0.926 settled\n\n- **The reproduction landed between their two published numbers.** This pipeline prints\n 0.9128 on the 58; the checkpoint stores 0.9167 and their notebook's comment says 0.9054\n for the same file. One of those described a different run.\n- **The 58-study gate predicts the leaderboard.** Blend gate 0.9170 here became 0.926 on\n the public board; the first notebook's 0.778 became 0.798. Same direction both times,\n about +0.01โ€“0.02, so the 58 stay the steering instrument โ€” for direction, not for\n ranking close models.\n- **Putting the 58 into training bought nothing out-of-sample.** Holdout weak AUC 0.9092\n with them in fine-tuning against 0.9097 with them out. The final submission still comes\n from the gold-in run: the holdout says it costs nothing, and 52 extra radiologist-read\n studies cannot hurt on the hidden test.\n- **A bug in the original pipeline, kept visible.** Their `_pick_series_for_slot` does\n `int(r.get('Fluid_Sensitive', 0) or 0)`; NaN is truthy, `int(nan)` raises, and their\n per-study try/except turns those studies into silent 0.5 submissions. This port treats\n NaN as no-preference and logs whether the column exists.\n- **The weak findings are the soft-tissue ones.** Synovitis (~0.78) and patellofemoral OA\n (~0.82) sit well below the rest; a mean over twelve findings hides that."},{"cell_type":"markdown","id":"a583873c","metadata":{},"source":"## ๐Ÿงญ What to do with this run\n\nTwo pushes are planned. This one ran with `INCLUDE_GOLD=False`, so its numbers are measurements:\nthe arm-1 reproduction against their two published numbers, arm 2's gain, the blend curve, all\nagainst both the holdout and the radiologist's 58. The next push flips the flag to `True`, the\n58 go into fine-tuning as the exact annotations they are, and that run is the submission.\n`MEASUREMENT_JSON` in the log carries `include_gold`, so the two runs stay distinguishable\nafterwards.\n\nAfter both runs land: `FINAL_RESULTS.md` gets a model-v2 section with the reproduction verdict,\nboth arms, the blend, both leaderboard scores and the timings; the README gets the second\nnotebook; and the runs decide whether a third idea (their SWA checkpoint as a third arm, or a\nlonger fine-tune on better labels) is worth a GPU-hour."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11.0"}},"nbformat":4,"nbformat_minor":5}