{"cells":[{"cell_type":"markdown","id":"56328ab4","metadata":{},"source":"# RSNA Knee: series-safe meniscus specialist V1, Fold 0 confirmation\n\nThis is a controlled Fold-0 experiment, not a leaderboard submission. It trains only\nthe medial and lateral meniscus targets using sagittal/coronal series. Every RGB triplet\ncontains adjacent, physically ordered slices from one SeriesInstanceUID. The notebook\nreuses the exact V1 folds, labels, and Fold-0 control OOF, then writes a paired OOF gate.\nA single run can never authorize the remaining folds: a second seed must reproduce the gain.\n\nRequired Kaggle inputs:\n\n- Competition: `rsna-knee-abnormality-detection`\n- Model: `metaresearch/dinov2/PyTorch/base/1`\n- Private dataset: `abisheksrivastav/rsna-knee-meniscus-series-safe-v1-assets`\n\nInternet must remain disabled. This is the seed-6203 confirmation of the completed\nseed-3407 Fold-0 run. Do not train folds 1-4 until both receipts are compared."},{"cell_type":"code","execution_count":null,"id":"dcbe02a8","metadata":{"lines_to_next_cell":2},"outputs":[],"source":"from __future__ import annotations\n\nimport gc\nimport hashlib\nimport json\nimport math\nimport os\nimport random\nimport re\nimport shutil\nimport time\nimport unicodedata\nfrom concurrent.futures import ThreadPoolExecutor\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport pydicom\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom IPython.display import display\nfrom sklearn.metrics import roc_auc_score"},{"cell_type":"markdown","id":"8f5b0ba2","metadata":{},"source":"## Configuration\n\nThe defaults target a Kaggle T4 notebook and leave a margin under the nine-hour limit.\nSet `DEBUG=True` only to verify the pipeline. This notebook never emits a submission."},{"cell_type":"code","execution_count":null,"id":"7277f2eb","metadata":{},"outputs":[],"source":"RUN_SEED = int(os.environ.get(\"RSNA_RUN_SEED\", \"6203\"))\nTRAIN_FOLD = int(os.environ.get(\"RSNA_TRAIN_FOLD\", \"0\"))\nDEBUG = os.environ.get(\"RSNA_DEBUG\", \"0\") == \"1\"\nDINO_VARIANT = \"base\"\n\nTARGETS = [\n    \"Medial Meniscus\",\n    \"Lateral Meniscus\",\n]\n\n# name, anatomical plane, fluid-sensitive flag, fat-suppression flag\nSLOTS = [\n    (\"SAG_FLUID_FS\", \"Sagittal\", 1, 1),\n    (\"COR_FLUID_FS\", \"Coronal\", 1, 1),\n    (\"SAG_FLUID_NOFS\", \"Sagittal\", 1, 0),\n]\nN_SLOTS = len(SLOTS)\n\nIMAGE_SIZE = 384\nGROUP_SIZE = 3\nN_GROUPS = 4\nCACHE_SLICES = GROUP_SIZE * N_GROUPS\nCROP_MM = 112.0\nPIXEL_THREADS = 12\nCACHE_IO_THREADS = 4\nCACHE_ROOT = Path(\"/kaggle/temp/rsna-knee-meniscus-series-safe-v1-cache\")\nif not Path(\"/kaggle\").is_dir():\n    CACHE_ROOT = Path(\"/tmp/rsna-knee-meniscus-series-safe-v1-cache\")\n\nN_FOLDS = 5\nEPOCHS = 6\nBATCH_STUDIES = 2\nEVAL_BATCH = 4\nUNFREEZE_LAST = 4\nLR_BACKBONE = 6e-6\nLR_HEAD = 4e-4\nWEIGHT_DECAY = 0.05\nRANK_LOSS_WEIGHT = 0.15\nGOLD_WEIGHT = 6.0\nLABEL_SMOOTHING = 0.015\nMIN_CONSENSUS_CONFIDENCE = 0.18\nSTEPS_PER_EPOCH = 550\nTIME_BUDGET_SECONDS = 8.15 * 3600\nTRAIN_STOP_SECONDS = 7.10 * 3600\nSAVE_FOLD_MODELS = True\nDELETE_TEMP_CACHE = True\nREQUIRE_EXTERNAL_LABELS = True\n\nif DEBUG:\n    N_FOLDS = 2\n    TRAIN_FOLD = 0\n    EPOCHS = 1\n    BATCH_STUDIES = 1\n    EVAL_BATCH = 1\n    N_GROUPS = 1\n    CACHE_SLICES = GROUP_SIZE\n    UNFREEZE_LAST = 1\n\nSTART_TIME = time.time()\nRUN_STATUS_PATH = Path(\"MENISCUS_SERIES_SAFE_V1_STATUS.json\")\n\n\ndef log(message: str) -> None:\n    elapsed = time.time() - START_TIME\n    print(f\"[{elapsed:7.1f}s] {message}\", flush=True)\n\n\ndef write_run_status(stage: str, **details: object) -> None:\n    payload = {\n        \"version\": \"meniscus-series-safe-v1-status\",\n        \"stage\": stage,\n        \"elapsed_seconds\": round(time.time() - START_TIME, 1),\n        \"seed\": int(RUN_SEED),\n        \"fold\": int(TRAIN_FOLD),\n        **details,\n    }\n    temporary = RUN_STATUS_PATH.with_suffix(\".tmp\")\n    temporary.write_text(json.dumps(payload, indent=2) + \"\\n\", encoding=\"utf-8\")\n    os.replace(temporary, RUN_STATUS_PATH)\n\n\ndef seed_everything(seed: int) -> None:\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n\n\nif not 0 <= TRAIN_FOLD < N_FOLDS:\n    raise ValueError(f\"TRAIN_FOLD must be in [0, {N_FOLDS - 1}], got {TRAIN_FOLD}\")\n\nseed_everything(RUN_SEED + TRAIN_FOLD)\ntorch.backends.cudnn.benchmark = True\nlog(\n    f\"torch={torch.__version__} cuda={torch.cuda.is_available()} \"\n    f\"seed={RUN_SEED} fold={TRAIN_FOLD} variant={DINO_VARIANT}\"\n)"},{"cell_type":"markdown","id":"6e31d84b","metadata":{},"source":"## Locate offline inputs"},{"cell_type":"code","execution_count":null,"id":"19b6a500","metadata":{"lines_to_next_cell":2},"outputs":[],"source":"def find_competition_root() -> Path:\n    candidates = [\n        Path(\"/kaggle/input/competitions/rsna-knee-abnormality-detection\"),\n        Path(\"/kaggle/input/rsna-knee-abnormality-detection\"),\n        Path(\"input\"),\n        Path(\"input_meta\"),\n        Path(\".\"),\n    ]\n    for path in candidates:\n        if (path / \"train.csv\").is_file() and (path / \"test.csv\").is_file():\n            return path\n\n    base = Path(\"/kaggle/input\")\n    if base.is_dir():\n        for train_csv in base.glob(\"**/train.csv\"):\n            path = train_csv.parent\n            if (path / \"test.csv\").is_file() and (path / \"train_series.csv\").is_file():\n                return path\n    raise FileNotFoundError(\"Could not locate the competition input directory\")\n\n\ndef find_dinov2_root(variant: str = DINO_VARIANT) -> Path:\n    base = Path(\"/kaggle/input\")\n    hits: list[Path] = []\n    if base.is_dir():\n        direct_candidates = [\n            base / f\"models/metaresearch/dinov2/pytorch/{variant}/1\",\n            base / f\"dinov2/pytorch/{variant}/1\",\n        ]\n        hits.extend(path for path in direct_candidates if (path / \"config.json\").is_file())\n\n        # Avoid recursively walking the very large competition DICOM directories.\n        for root, directories, files in os.walk(base):\n            directories[:] = [\n                name for name in directories if name not in (\"train_series\", \"test_series\")\n            ]\n            if \"config.json\" in files and \"dinov2\" in root.lower():\n                hits.append(Path(root))\n    hits = list(dict.fromkeys(hits))\n    preferred = [path for path in hits if variant.lower() in str(path).lower()]\n    if preferred:\n        return preferred[0]\n    if hits:\n        return hits[0]\n    mounts = []\n    if base.is_dir():\n        mounts = sorted(path.name for path in base.iterdir() if path.is_dir())\n    raise FileNotFoundError(\n        \"DINOv2 weights were not found. This is a Kaggle Model, not a Dataset. \"\n        \"Open https://www.kaggle.com/models/metaresearch/dinov2 and attach \"\n        f\"PyTorch / {variant} / version 1 to this notebook. \"\n        f\"Visible /kaggle/input mounts: {mounts}\"\n    )\n\n\nROOT = find_competition_root()\nlog(f\"competition root: {ROOT}\")\n\n# Fail before the expensive DICOM cache pass when the offline model input is missing.\nDINO_ROOT = None\nif Path(\"/kaggle/input\").is_dir() or not DEBUG:\n    DINO_ROOT = find_dinov2_root(DINO_VARIANT)\n    log(f\"DINOv2 root: {DINO_ROOT}\")\n\ntrain_df = pd.read_csv(ROOT / \"train.csv\")\ntest_df = pd.read_csv(ROOT / \"test.csv\")\ntrain_series_df = pd.read_csv(ROOT / \"train_series.csv\")\ntest_series_df = pd.read_csv(ROOT / \"test_series.csv\")\n\nif DEBUG:\n    train_df = train_df.iloc[:96].copy()\n    train_series_df = train_series_df[\n        train_series_df[\"StudyInstanceUID\"].isin(train_df[\"StudyInstanceUID\"])\n    ].copy()\n\nlog(f\"train={train_df.shape} train_series={train_series_df.shape} test={test_df.shape}\")\n\n\nPath(\"TRAINER_ONLY_DO_NOT_SUBMIT.txt\").write_text(\n    \"This is a controlled Fold-0 meniscus experiment. It does not create submission.csv.\\n\"\n    \"Do not train the remaining folds unless two seeds pass the paired OOF gate.\\n\"\n)\nlog(\"artifact-only trainer: submission.csv will not be created\")\n\n\ndef find_asset_file(filename: str) -> Path:\n    \"\"\"Find a small attached asset without walking the 570 GB DICOM tree.\"\"\"\n    base = Path(\"/kaggle/input\")\n    dataset_roots = [\n        base / \"rsna-knee-meniscus-series-safe-v1-assets\",\n        base / \"datasets\" / \"abisheksrivastav\" / \"rsna-knee-meniscus-series-safe-v1-assets\",\n    ]\n    direct = [root / filename for root in dataset_roots]\n    for root in dataset_roots:\n        versions = root / \"versions\"\n        if versions.is_dir():\n            direct.extend(\n                version / filename\n                for version in sorted(versions.iterdir(), reverse=True)\n                if version.is_dir()\n            )\n    for path in direct:\n        if path.is_file():\n            return path\n    if base.is_dir():\n        for mount in base.iterdir():\n            if not mount.is_dir() or mount.name in {\n                \"rsna-knee-abnormality-detection\",\n                \"competitions\",\n            }:\n                continue\n            candidate = mount / filename\n            if candidate.is_file():\n                return candidate\n            versions = mount / \"versions\"\n            if versions.is_dir():\n                for version in versions.iterdir():\n                    candidate = version / filename\n                    if candidate.is_file():\n                        return candidate\n    local_candidates = [\n        Path(\"artifacts\") / \"meniscus_series_safe_v1_assets\" / filename,\n        Path(\"../../../artifacts\") / \"meniscus_series_safe_v1_assets\" / filename,\n    ]\n    for local in local_candidates:\n        if local.is_file():\n            return local.resolve()\n    mounts = sorted(path.name for path in base.iterdir() if path.is_dir()) if base.is_dir() else []\n    raise FileNotFoundError(\n        f\"Missing {filename}. Attach the private Kaggle Dataset \"\n        \"abisheksrivastav/rsna-knee-meniscus-series-safe-v1-assets. \"\n        f\"Visible mounts: {mounts}\"\n    )\n\n\ndef file_sha256(path: Path) -> str:\n    digest = hashlib.sha256()\n    with path.open(\"rb\") as handle:\n        for block in iter(lambda: handle.read(1024 * 1024), b\"\"):\n            digest.update(block)\n    return digest.hexdigest()\n\n\ndef validate_asset_contract() -> dict[str, object]:\n    manifest_path = find_asset_file(\"manifest.json\")\n    manifest = json.loads(manifest_path.read_text(encoding=\"utf-8\"))\n    expected_version = \"rsna-knee-meniscus-series-safe-v1-assets\"\n    if manifest.get(\"version\") != expected_version:\n        raise RuntimeError(\n            f\"asset version mismatch: expected {expected_version}, got {manifest.get('version')}\"\n        )\n    for filename, expected_hash in manifest.get(\"sha256\", {}).items():\n        path = find_asset_file(filename)\n        actual_hash = file_sha256(path)\n        if actual_hash != expected_hash:\n            raise RuntimeError(f\"asset hash mismatch for {filename}\")\n    log(f\"validated paired-control asset contract: {manifest_path}\")\n    return manifest\n\n\nASSET_MANIFEST = validate_asset_contract()\nwrite_run_status(\n    \"preflight_complete\",\n    asset_version=ASSET_MANIFEST[\"version\"],\n    model_root=str(DINO_ROOT),\n)"},{"cell_type":"markdown","id":"606e2d16","metadata":{},"source":"## Report-derived supervision\n\nA mounted `report_labels*.csv` is preferred. The fallback below is intentionally\nconservative: silence receives little weight, explicit negation receives medium weight,\nand a named positive finding receives high weight. The 58 expert-labelled studies always\noverride report labels."},{"cell_type":"code","execution_count":null,"id":"6899897e","metadata":{"lines_to_next_cell":2},"outputs":[],"source":"_TRANSLATE = str.maketrans(\n    {\n        \"\\u0131\": \"i\", \"\\u0130\": \"i\", \"\\u015f\": \"s\", \"\\u015e\": \"s\",\n        \"\\u011f\": \"g\", \"\\u011e\": \"g\", \"\\u00fc\": \"u\", \"\\u00dc\": \"u\",\n        \"\\u00f6\": \"o\", \"\\u00d6\": \"o\", \"\\u00e7\": \"c\", \"\\u00c7\": \"c\",\n        \"\\u00df\": \"ss\",\n    }\n)\n\n\ndef normalize_report(text: object) -> str:\n    if not isinstance(text, str):\n        return \"\"\n    text = text.translate(_TRANSLATE)\n    text = unicodedata.normalize(\"NFKD\", text)\n    text = \"\".join(ch for ch in text if not unicodedata.combining(ch))\n    text = text.lower()\n    text = re.sub(r\"[^a-z0-9'./;:,+-]+\", \" \", text)\n    return re.sub(r\"\\s+\", \" \", text).strip()\n\n\nNEGATION = re.compile(\n    r\"\\b(no|not|without|absent|absence|negative|normal|intact|preserved|\"\n    r\"unremarkable|free of|geen|zonder|kein|keine|ohne|sin|ausencia|\"\n    r\"pas de|sans|assenza|senza|yok|izlenmedi|saptanmadi|gorulmedi|\"\n    r\"mevcut degil|aucun|aucune)\\b\"\n)\nTEAR = re.compile(\n    r\"\\b(tear|torn|ruptur|rotura|scheur|riss|dechir|yirtik|kopma|sprain|desgarro|lesion)\\w*\\b\"\n)\nMENISCUS = re.compile(r\"\\b(menisc|menisk)\\w*\\b\")\nMEDIAL = re.compile(r\"\\b(medial|mediale|interno|internal|inner|binnen|innen|ic)\\b\")\nLATERAL = re.compile(r\"\\b(lateral|laterale|externo|external|outer|buiten|aussen|dis)\\b\")\nOA_WORD = re.compile(\n    r\"\\b(osteoarth|arthros|artros|gonarthros|chondr|condropat|cartilage|\"\n    r\"kraakbeen|knorpel|degenerat)\\w*\\b\"\n)\nPF_WORD = re.compile(\n    r\"\\b(patellofemoral|patello femoral|femoropatell|retropatell|patella|rotul|trochlea)\\w*\\b\"\n)\n\nANCHORS = {\n    \"ACL\": re.compile(\n        r\"\\b(acl|anterior cruciate|cruzado anterior|lca|voorste kruisband|vorderes kreuzband|on capraz)\\b\"\n    ),\n    \"MCL\": re.compile(\n        r\"\\b(mcl|medial collateral|colateral medial|colateral interno|innenband|ic yan bag)\\b\"\n    ),\n    \"Effusion\": re.compile(\n        r\"\\b(effusion|joint fluid|derrame|gewrichtsvocht|erguss|epanchement|versamento|efuzyon)\\w*\\b\"\n    ),\n    \"Synovitis\": re.compile(r\"\\b(synovit|sinovit|synovial hypertrophy|synovial proliferation)\\w*\\b\"),\n    \"Baker's\": re.compile(r\"\\b(baker|popliteal cyst|quiste popliteo|bakercyste|bakerzyste)\\w*\\b\"),\n    \"Contusion\": re.compile(\n        r\"\\b(contusion|bone bruise|bone marrow edema|marrow edema|knochenmarkodem|kemik iligi odem)\\w*\\b\"\n    ),\n    \"Fracture\": re.compile(r\"\\b(fractur|fractuur|fraktur|frattur|kirik|avulsion)\\w*\\b\"),\n}\n\n\ndef report_clauses(text: str) -> list[str]:\n    return [part.strip() for part in re.split(r\"[.;:\\n\\r]+\", text) if part.strip()]\n\n\ndef nearby(clause: str, pattern: re.Pattern[str], start: int, end: int, window: int = 120) -> bool:\n    region = clause[max(0, start - window) : min(len(clause), end + window)]\n    return bool(pattern.search(region))\n\n\ndef negated(clause: str, start: int, end: int) -> bool:\n    region = clause[max(0, start - 90) : min(len(clause), end + 90)]\n    return bool(NEGATION.search(region))\n\n\ndef grade_positive(clause: str) -> float:\n    if re.search(r\"\\b(trace|minimal|tiny|slight|grade 1|grade i)\\b\", clause):\n        return 0.62\n    if re.search(r\"\\b(marked|severe|large|complete|grade 3|grade iii)\\b\", clause):\n        return 0.96\n    return 0.86\n\n\ndef fallback_report_labels(report: object) -> dict[str, float]:\n    clauses = report_clauses(normalize_report(report))\n    result: dict[str, float] = {}\n    for target in TARGETS:\n        score, confidence = 0.35, 0.12\n        explicit_negative = False\n        for clause in clauses:\n            candidates: list[re.Match[str]] = []\n            positive = False\n            if target in (\"ACL\", \"MCL\"):\n                candidates = list(ANCHORS[target].finditer(clause))\n                positive = bool(candidates and TEAR.search(clause))\n            elif target in (\"Medial Meniscus\", \"Lateral Meniscus\"):\n                side = MEDIAL if target.startswith(\"Medial\") else LATERAL\n                candidates = list(MENISCUS.finditer(clause))\n                positive = any(\n                    nearby(clause, side, match.start(), match.end())\n                    and nearby(clause, TEAR, match.start(), match.end())\n                    for match in candidates\n                )\n            elif target in (\"Medial OA\", \"Lateral OA\", \"PF OA\"):\n                side = MEDIAL if target == \"Medial OA\" else LATERAL if target == \"Lateral OA\" else PF_WORD\n                candidates = list(OA_WORD.finditer(clause))\n                positive = any(nearby(clause, side, match.start(), match.end()) for match in candidates)\n            else:\n                candidates = list(ANCHORS[target].finditer(clause))\n                positive = bool(candidates)\n            for match in candidates:\n                if negated(clause, match.start(), match.end()):\n                    explicit_negative = True\n            if positive and not any(negated(clause, m.start(), m.end()) for m in candidates):\n                score = max(score, grade_positive(clause))\n                confidence = 0.88\n        if score == 0.35 and explicit_negative:\n            score, confidence = 0.08, 0.68\n        result[target] = score\n        result[target + \"__conf\"] = confidence\n    return result\n\n\ndef fallback_label_frame(frame: pd.DataFrame) -> pd.DataFrame:\n    labels = pd.DataFrame([fallback_report_labels(report) for report in frame[\"Report\"].fillna(\"\")])\n    labels.insert(0, \"StudyInstanceUID\", frame[\"StudyInstanceUID\"].values)\n    return labels.set_index(\"StudyInstanceUID\")\n\n\ndef find_report_label_tables() -> list[Path]:\n    base = Path(\"/kaggle/input\")\n    if not base.is_dir():\n        return []\n    expected = [\n        (\"pilkwang\", \"rsna-knee-llm-labels\", [\"report_labels_v2.csv\"]),\n        (\n            \"stevenleehans\",\n            \"rsna-knee-llm-report-labels\",\n            [\"llm_labels_v4_blend.csv\", \"llm_labels_v2.csv\"],\n        ),\n        (\n            \"lixin73\",\n            \"rsna-knee-llm-report-labels-sol56\",\n            [\"report_labels_gpt56sol.csv\"],\n        ),\n    ]\n    candidates: list[Path] = []\n    for owner, slug, filenames in expected:\n        dataset_roots = [base / slug, base / \"datasets\" / owner / slug]\n        for dataset_root in dataset_roots:\n            search_roots = [dataset_root]\n            versions = dataset_root / \"versions\"\n            if versions.is_dir():\n                search_roots.extend(\n                    path for path in versions.iterdir() if path.is_dir()\n                )\n            for search_root in search_roots:\n                for filename in filenames:\n                    candidate = search_root / filename\n                    if candidate.is_file():\n                        candidates.append(candidate)\n                        break\n                if candidates and candidates[-1].parent == search_root:\n                    break\n            if candidates and slug in str(candidates[-1]):\n                break\n    candidates = sorted(set(candidates))\n    selected: list[Path] = []\n    seen_hashes: set[str] = set()\n    for path in candidates:\n        try:\n            probe = pd.read_csv(path, nrows=64)\n        except Exception:\n            continue\n        if \"StudyInstanceUID\" not in probe or not all(target in probe for target in TARGETS):\n            continue\n        if \"Report\" in probe or len(probe) < 8:\n            continue\n        digest = hashlib.sha256(path.read_bytes()).hexdigest()\n        if digest in seen_hashes:\n            continue\n        seen_hashes.add(digest)\n        selected.append(path)\n    return selected\n\n\ndef read_label_source(path: Path, index: pd.Index) -> tuple[pd.DataFrame, pd.DataFrame]:\n    raw = pd.read_csv(path).drop_duplicates(\"StudyInstanceUID\").set_index(\"StudyInstanceUID\")\n    values = pd.DataFrame(np.nan, index=index, columns=TARGETS, dtype=np.float32)\n    confidence = pd.DataFrame(0.0, index=index, columns=TARGETS, dtype=np.float32)\n    overlap = index.intersection(raw.index)\n    for target in TARGETS:\n        target_values = pd.to_numeric(raw.loc[overlap, target], errors=\"coerce\").clip(0, 1)\n        unknown = target_values.isna()\n        verdict = target + \"__verdict\"\n        if verdict in raw:\n            unknown |= raw.loc[overlap, verdict].astype(str).str.upper().isin(\n                {\"UNK\", \"UNKNOWN\", \"NOT ADDRESSED\", \"NA\"}\n            )\n        values.loc[overlap, target] = target_values.where(~unknown).astype(np.float32)\n        confidence_column = target + \"__conf\"\n        if confidence_column in raw:\n            source_confidence = pd.to_numeric(\n                raw.loc[overlap, confidence_column], errors=\"coerce\"\n            ).fillna(0.0).clip(0, 1)\n        else:\n            source_confidence = pd.Series(0.82, index=overlap)\n        confidence.loc[overlap, target] = source_confidence.where(~unknown, 0.0)\n    return values, confidence\n\n\ndef build_report_labels(frame: pd.DataFrame) -> tuple[pd.DataFrame, str]:\n    fallback = fallback_label_frame(frame)\n    sources = find_report_label_tables()\n    if not sources:\n        return fallback, \"multilingual rules fallback\"\n    value_frames, confidence_frames = zip(\n        *(read_label_source(path, fallback.index) for path in sources)\n    )\n    diagnostics = []\n    for target in TARGETS:\n        values = np.stack([table[target].to_numpy(float) for table in value_frames], axis=1)\n        source_confidence = np.stack(\n            [table[target].to_numpy(float) for table in confidence_frames], axis=1\n        )\n        known = np.isfinite(values)\n        support = known.mean(axis=1)\n        with np.errstate(invalid=\"ignore\"):\n            consensus = np.nanmedian(values, axis=1)\n            spread = np.nanstd(values, axis=1)\n            mean_confidence = np.nanmean(np.where(known, source_confidence, np.nan), axis=1)\n        has_external = known.any(axis=1)\n        agreement = np.exp(-6.0 * np.nan_to_num(spread, nan=1.0))\n        extremity = 2.0 * np.abs(np.nan_to_num(consensus, nan=0.5) - 0.5)\n        consensus_confidence = (\n            np.nan_to_num(mean_confidence)\n            * agreement\n            * (0.55 + 0.45 * support)\n            * (0.55 + 0.45 * extremity)\n        )\n        accepted = has_external & (consensus_confidence >= MIN_CONSENSUS_CONFIDENCE)\n        fallback.loc[accepted, target] = consensus[accepted]\n        fallback.loc[accepted, target + \"__conf\"] = consensus_confidence[accepted]\n        rejected = has_external & ~accepted\n        fallback.loc[rejected, target] = 0.5\n        fallback.loc[rejected, target + \"__conf\"] = 0.0\n        diagnostics.append(\n            {\n                \"target\": target,\n                \"external_coverage\": float(has_external.mean()),\n                \"accepted_coverage\": float(accepted.mean()),\n                \"mean_agreement\": float(agreement[has_external].mean()),\n            }\n        )\n\n    # Silence is not a negative. Only fill missing Synovitis from a high-confidence\n    # Effusion consensus, at half weight, because the findings overlap but are not equal.\n    syn_conf = fallback[\"Synovitis__conf\"].to_numpy(float)\n    eff_conf = fallback[\"Effusion__conf\"].to_numpy(float)\n    fill = (syn_conf <= 0) & (eff_conf >= 0.55)\n    fallback.loc[fill, \"Synovitis\"] = fallback.loc[fill, \"Effusion\"]\n    fallback.loc[fill, \"Synovitis__conf\"] = 0.5 * fallback.loc[fill, \"Effusion__conf\"]\n    pd.DataFrame(diagnostics).to_csv(\"label_consensus_diagnostics.csv\", index=False)\n    fallback.reset_index().to_csv(\"report_label_consensus.csv\", index=False)\n    return fallback, \"; \".join(str(path) for path in sources)\n\n\nlabel_path = find_asset_file(\"report_label_consensus.csv\")\nreport_labels = pd.read_csv(label_path, dtype={\"StudyInstanceUID\": str})\nrequired_label_columns = TARGETS + [target + \"__conf\" for target in TARGETS]\nmissing_label_columns = [column for column in required_label_columns if column not in report_labels]\nif missing_label_columns:\n    raise RuntimeError(f\"report label asset is missing columns: {missing_label_columns}\")\nif report_labels[\"StudyInstanceUID\"].duplicated().any():\n    raise RuntimeError(\"report label asset contains duplicate study IDs\")\nreport_labels = report_labels.set_index(\"StudyInstanceUID\")\nlabel_source = str(label_path)\nlog(f\"fixed label source: {label_source}\")\ngold = train_df.set_index(\"StudyInstanceUID\")[TARGETS]\ngold = gold[gold.notna().all(axis=1)]\nlog(f\"gold studies: {len(gold)}\")\n\nlabel_diagnostics = []\nfor target in TARGETS:\n    y = gold[target].astype(int).to_numpy()\n    p = report_labels.loc[gold.index, target].astype(float).to_numpy()\n    auc = roc_auc_score(y, p) if len(np.unique(y)) == 2 else np.nan\n    label_diagnostics.append({\"target\": target, \"gold_auc\": auc})\ndisplay(pd.DataFrame(label_diagnostics).round(3))"},{"cell_type":"markdown","id":"1d25b770","metadata":{},"source":"## DICOM ordering and multi-plane cache\n\nFile names are SOP UIDs and are not slice order. Each selected series is sorted using\n`ImageOrientationPatient` and `ImagePositionPatient`. Four centers span the central 60%;\nevery center becomes the adjacent triplet `(c-1, c, c+1)` from that same series. Images\nare cropped to a 112 mm field of view and resized to 384 px. This is the intervention:\nthe old 44-slice Raptor corpus could place slices from different series in one triplet."},{"cell_type":"code","execution_count":null,"id":"a0af82a3","metadata":{"lines_to_next_cell":2},"outputs":[],"source":"SLOT_TO_INDEX = {\n    (plane, int(fluid), int(fat_suppression)): index\n    for index, (_, plane, fluid, fat_suppression) in enumerate(SLOTS)\n}\n\nFATSAT_OPTIONS = {\"FS\", \"FATSAT\", \"FAT_SAT\", \"FSAT\"}\nFATSAT_PATTERN = re.compile(\n    r\"\\bfs\\b|fatsat|fat sat|\\bstir\\b|\\bspair\\b|\\bspir\\b|water excit|\"\n    r\"\\btirm\\b|\\bfatsup\\b\"\n)\nT1_PATTERN = re.compile(r\"\\bt1\\b|\\bt1w\\b\")\nT2_PATTERN = re.compile(r\"\\bt2\\b|\\bt2w\\b\")\nPD_PATTERN = re.compile(r\"\\bpd\\b|\\bpdw\\b|proton|\\bdp\\b|dens\")\n\n\ndef series_protocol(row: object, path: Path, files: list[Path]) -> tuple[int, int]:\n    \"\"\"Recover fluid weighting and fat suppression from one representative header.\"\"\"\n    fluid = int(getattr(row, \"Fluid_Sensitive\", 0) or 0)\n    fat_suppression = int(getattr(row, \"Fat_Suppression\", 0) or 0)\n    if not files:\n        return fluid, fat_suppression\n    try:\n        ds = pydicom.dcmread(str(files[len(files) // 2]), stop_before_pixels=True, force=True)\n    except Exception:\n        return fluid, fat_suppression\n\n    description = \" \".join(\n        str(getattr(ds, name, \"\") or \"\") for name in (\"SeriesDescription\", \"SequenceName\")\n    ).lower()\n    description = re.sub(r\"[_\\-.]+\", \" \", description)\n    raw_scan_options = getattr(ds, \"ScanOptions\", None)\n    scan_options = set()\n    if isinstance(raw_scan_options, str):\n        scan_options = {\n            token.strip().upper()\n            for token in re.split(r\"[\\\\,| ]+\", raw_scan_options)\n            if token.strip()\n        }\n    elif raw_scan_options is not None:\n        scan_options = {str(value).strip().upper() for value in raw_scan_options}\n    if FATSAT_PATTERN.search(description) or scan_options.intersection(FATSAT_OPTIONS):\n        fat_suppression = 1\n\n    repetition = pd.to_numeric(getattr(ds, \"RepetitionTime\", np.nan), errors=\"coerce\")\n    echo = pd.to_numeric(getattr(ds, \"EchoTime\", np.nan), errors=\"coerce\")\n    named_t1 = bool(T1_PATTERN.search(description))\n    named_t2 = bool(T2_PATTERN.search(description))\n    named_pd = bool(PD_PATTERN.search(description))\n    if named_t1 and not named_t2 and not named_pd:\n        fluid = 0\n    elif named_t2 or named_pd:\n        fluid = 1\n    elif np.isfinite(repetition) and repetition < 800:\n        fluid = 0\n    elif (np.isfinite(echo) and echo > 60) or (\n        np.isfinite(repetition) and repetition >= 800\n    ):\n        fluid = 1\n    return fluid, fat_suppression\n\n\ndef select_series(frame: pd.DataFrame, split: str) -> dict[str, dict[int, Path]]:\n    \"\"\"Choose the thickest matching series for each recovered protocol slot.\"\"\"\n    jobs = list(frame.itertuples(index=False))\n\n    def inspect(row: object):\n        study = str(row.StudyInstanceUID)\n        series = str(row.SeriesInstanceUID)\n        path = ROOT / split / study / series\n        try:\n            files = sorted(\n                (Path(entry.path) for entry in os.scandir(path) if entry.name.lower().endswith(\".dcm\")),\n                key=lambda item: item.name,\n            )\n        except FileNotFoundError:\n            files = []\n        fluid, fat_suppression = series_protocol(row, path, files)\n        key = SLOT_TO_INDEX.get((str(row.Anatomical_Plane), fluid, fat_suppression))\n        return study, key, path, len(files)\n\n    selected: dict[str, dict[int, tuple[Path, int]]] = {}\n    with ThreadPoolExecutor(max_workers=PIXEL_THREADS) as pool:\n        for study, key, path, count in pool.map(inspect, jobs):\n            if key is None or count == 0:\n                continue\n            previous = selected.setdefault(study, {}).get(key)\n            if previous is None or count > previous[1]:\n                selected[study][key] = (path, count)\n    return {\n        study: {slot: value[0] for slot, value in slots.items()}\n        for study, slots in selected.items()\n    }\n\n\ndef geometry_scalar(ds: pydicom.dataset.FileDataset) -> float | None:\n    try:\n        orientation = np.asarray(ds.ImageOrientationPatient, dtype=np.float64)\n        position = np.asarray(ds.ImagePositionPatient, dtype=np.float64)\n        normal = np.cross(orientation[:3], orientation[3:])\n        return float(np.dot(position, normal))\n    except Exception:\n        return None\n\n\ndef ordered_series_files(path: Path) -> tuple[list[Path], pydicom.dataset.FileDataset | None]:\n    records = []\n    first_header = None\n    for file_path in path.glob(\"*.dcm\"):\n        try:\n            ds = pydicom.dcmread(str(file_path), stop_before_pixels=True, force=True)\n            if first_header is None:\n                first_header = ds\n            records.append((file_path, geometry_scalar(ds), int(getattr(ds, \"InstanceNumber\", 0) or 0)))\n        except Exception:\n            records.append((file_path, None, 0))\n    geometry_count = sum(scalar is not None for _, scalar, _ in records)\n    if geometry_count >= 2:\n        records.sort(key=lambda item: (item[1] is None, item[1] or 0.0, item[0].name))\n    else:\n        records.sort(key=lambda item: (item[2], item[0].name))\n    files = [item[0] for item in records]\n    if files:\n        try:\n            first_header = pydicom.dcmread(\n                str(files[len(files) // 2]), stop_before_pixels=True, force=True\n            )\n        except Exception:\n            pass\n    return files, first_header\n\n\ndef adjacent_group_indices(n_files: int, n_groups: int = N_GROUPS) -> np.ndarray:\n    \"\"\"Return flattened, central, adjacent triplets from one ordered series.\"\"\"\n    if n_files <= 0:\n        raise ValueError(\"a series must contain at least one DICOM\")\n    if n_files < GROUP_SIZE:\n        centers = np.linspace(0, n_files - 1, n_groups).round().astype(int)\n    else:\n        low = max(1, int(round(0.20 * (n_files - 1))))\n        high = min(n_files - 2, int(round(0.80 * (n_files - 1))))\n        high = max(low, high)\n        centers = np.linspace(low, high, n_groups).round().astype(int)\n    groups = []\n    for center in centers:\n        group = np.clip(np.arange(center - 1, center + 2), 0, n_files - 1)\n        groups.append(group)\n    result = np.concatenate(groups).astype(int)\n    if result.shape != (CACHE_SLICES,):\n        raise AssertionError(f\"expected {CACHE_SLICES} indices, got {result.shape}\")\n    return result\n\n\ndef center_crop_physical(image: np.ndarray, spacing: object) -> np.ndarray:\n    try:\n        spacing_values = np.asarray(spacing, dtype=np.float32)\n        mm_per_pixel = float(np.mean(spacing_values[:2]))\n    except Exception:\n        return image\n    if not np.isfinite(mm_per_pixel) or mm_per_pixel <= 0:\n        return image\n    crop = int(round(CROP_MM / mm_per_pixel))\n    height, width = image.shape[-2:]\n    if crop < 32 or crop >= min(height, width):\n        return image\n    top = (height - crop) // 2\n    left = (width - crop) // 2\n    return image[top : top + crop, left : left + crop]\n\n\ndef header_laterality(header: pydicom.dataset.FileDataset | None) -> str:\n    if header is None:\n        return \"\"\n    tagged = str(\n        getattr(header, \"ImageLaterality\", None) or getattr(header, \"Laterality\", \"\")\n    ).strip().upper()[:1]\n    if tagged in (\"L\", \"R\"):\n        return tagged\n    try:\n        orientation = np.asarray(header.ImageOrientationPatient, dtype=np.float64)\n        position = np.asarray(header.ImagePositionPatient, dtype=np.float64)\n        spacing = np.asarray(header.PixelSpacing, dtype=np.float64)\n        rows = float(header.Rows)\n        columns = float(header.Columns)\n        center = (\n            position[:3]\n            + orientation[:3] * spacing[1] * columns / 2\n            + orientation[3:6] * spacing[0] * rows / 2\n        )\n        if abs(float(center[0])) >= 20.0:\n            return \"R\" if center[0] < 0 else \"L\"\n    except Exception:\n        pass\n    return \"\"\n\n\ndef read_series(path: Path, plane: str) -> np.ndarray | None:\n    files, header = ordered_series_files(path)\n    if not files:\n        return None\n    arrays: list[np.ndarray | None] = []\n    sample_indices = adjacent_group_indices(len(files))\n    for group in range(N_GROUPS):\n        triplet = sample_indices[group * GROUP_SIZE : (group + 1) * GROUP_SIZE]\n        if int(triplet.max() - triplet.min()) > 2:\n            raise AssertionError(f\"non-adjacent triplet generated for {path}: {triplet}\")\n    for index in sample_indices:\n        try:\n            ds = pydicom.dcmread(str(files[int(index)]), force=True)\n            image = ds.pixel_array.astype(np.float32)\n            image = image * float(getattr(ds, \"RescaleSlope\", 1.0) or 1.0)\n            image = image + float(getattr(ds, \"RescaleIntercept\", 0.0) or 0.0)\n            if str(getattr(ds, \"PhotometricInterpretation\", \"\")) == \"MONOCHROME1\":\n                image = image.max() - image\n            image = center_crop_physical(image, getattr(ds, \"PixelSpacing\", None))\n            tensor = torch.from_numpy(np.ascontiguousarray(image))[None, None]\n            resized = F.interpolate(\n                tensor, size=(IMAGE_SIZE, IMAGE_SIZE), mode=\"bilinear\", align_corners=False\n            )[0, 0].numpy()\n            arrays.append(resized)\n        except Exception:\n            arrays.append(None)\n    valid_indices = [index for index, array in enumerate(arrays) if array is not None]\n    if not valid_indices:\n        return None\n    for index, array in enumerate(arrays):\n        if array is None:\n            nearest = min(valid_indices, key=lambda valid_index: abs(valid_index - index))\n            arrays[index] = arrays[nearest]\n    valid = [array for array in arrays if array is not None]\n    values = np.concatenate([array.reshape(-1) for array in valid])\n    low, high = np.percentile(values, [1, 99])\n    scale = max(float(high - low), 1e-6)\n    volume = np.stack(arrays)\n    volume = np.clip((volume - low) / scale, 0, 1)\n    laterality = header_laterality(header)\n    if laterality == \"R\":\n        if plane in (\"Coronal\", \"Axial\"):\n            volume = volume[..., ::-1]\n        elif plane == \"Sagittal\":\n            volume = volume[::-1]\n    return np.ascontiguousarray((volume * 255).round().astype(np.uint8))\n\n\nclass CompressedStudyCache:\n    def __init__(self, root: Path, studies: list[str]):\n        self.root = root\n        self.studies = studies\n\n    def path(self, index: int, group: int) -> Path:\n        return self.root / f\"{index:05d}_g{group}.npz\"\n\n    def load_group(self, indices: np.ndarray, group: int) -> np.ndarray:\n        def load(index: int) -> np.ndarray:\n            with np.load(self.path(int(index), group), allow_pickle=False) as item:\n                return item[\"images\"]\n\n        with ThreadPoolExecutor(max_workers=min(CACHE_IO_THREADS, max(len(indices), 1))) as pool:\n            rows = list(pool.map(load, indices.tolist()))\n        return np.stack(rows) if rows else np.zeros(\n            (0, N_SLOTS, GROUP_SIZE, IMAGE_SIZE, IMAGE_SIZE), dtype=np.uint8\n        )\n\n\ndef build_cache(\n    studies: list[str], selected: dict[str, dict[int, Path]], tag: str\n) -> tuple[CompressedStudyCache, np.ndarray]:\n    root = CACHE_ROOT / (\n        f\"{tag}_{IMAGE_SIZE}px_{CACHE_SLICES}sl_{int(CROP_MM)}mm_\"\n        + \"_\".join(slot[0] for slot in SLOTS)\n    )\n    root.mkdir(parents=True, exist_ok=True)\n    mask = np.zeros((len(studies), N_SLOTS), dtype=np.float32)\n    dense_gb = len(studies) * N_SLOTS * CACHE_SLICES * IMAGE_SIZE**2 / 1024**3\n    free_gb = shutil.disk_usage(root).free / 1024**3\n    log(\n        f\"{tag}: compressed cache for {len(studies)} studies; \"\n        f\"dense equivalent={dense_gb:.1f} GB, free temporary disk={free_gb:.1f} GB\"\n    )\n    write_run_status(\n        \"cache_running\",\n        cache_split=tag,\n        cache_completed=0,\n        cache_total=len(studies),\n        dense_equivalent_gb=round(dense_gb, 2),\n    )\n\n    def cache_one(item: tuple[int, str]) -> tuple[int, np.ndarray]:\n        index, study = item\n        paths = [root / f\"{index:05d}_g{group}.npz\" for group in range(N_GROUPS)]\n        if all(path.is_file() for path in paths):\n            with np.load(paths[0], allow_pickle=False) as saved:\n                return index, saved[\"mask\"].astype(np.float32)\n\n        images = np.zeros(\n            (N_SLOTS, CACHE_SLICES, IMAGE_SIZE, IMAGE_SIZE), dtype=np.uint8\n        )\n        study_mask = np.zeros(N_SLOTS, dtype=np.float32)\n        for slot_index, path in selected.get(study, {}).items():\n            plane = SLOTS[slot_index][1]\n            volume = read_series(path, plane)\n            if volume is not None:\n                images[slot_index] = volume\n                study_mask[slot_index] = 1.0\n        for group, path in enumerate(paths):\n            start = group * GROUP_SIZE\n            temporary = path.with_suffix(\".tmp\")\n            with open(temporary, \"wb\") as handle:\n                np.savez_compressed(\n                    handle,\n                    images=images[:, start : start + GROUP_SIZE],\n                    mask=study_mask,\n                )\n            os.replace(temporary, path)\n        return index, study_mask\n\n    jobs = list(enumerate(studies))\n    with ThreadPoolExecutor(max_workers=CACHE_IO_THREADS) as pool:\n        for completed, (index, study_mask) in enumerate(pool.map(cache_one, jobs), 1):\n            mask[index] = study_mask\n            if completed % 100 == 0 or completed == len(jobs):\n                log(f\"{tag}: cached {completed}/{len(jobs)} studies\")\n                write_run_status(\n                    \"cache_running\",\n                    cache_split=tag,\n                    cache_completed=completed,\n                    cache_total=len(studies),\n                )\n    write_run_status(\n        \"cache_complete\",\n        cache_split=tag,\n        cache_completed=len(studies),\n        cache_total=len(studies),\n    )\n    return CompressedStudyCache(root, studies), mask\n\n\ntrain_studies = train_df[\"StudyInstanceUID\"].astype(str).tolist()\ntrain_selected = select_series(train_series_df, \"train_series\")\n\nseries_audit = []\nfor study, slots in train_selected.items():\n    for slot_index, path in slots.items():\n        if path.name == study or path.parent.name != study:\n            raise RuntimeError(f\"unexpected series path layout for {study}: {path}\")\n        series_audit.append(\n            {\n                \"StudyInstanceUID\": study,\n                \"slot\": SLOTS[slot_index][0],\n                \"plane\": SLOTS[slot_index][1],\n                \"SeriesInstanceUID\": path.name,\n            }\n        )\npd.DataFrame(series_audit).to_csv(\"selected_series_audit.csv\", index=False)\nlog(f\"series-safe audit: {len(series_audit)} selected study/slot series\")\n\ntrain_cache, train_mask = build_cache(train_studies, train_selected, \"train\")\nlog(\n    f\"slot coverage train={train_mask.mean():.1%}; \"\n    f\"empty studies train={(train_mask.sum(1) == 0).sum()}\"\n)"},{"cell_type":"markdown","id":"c543dd7c","metadata":{},"source":"## DINOv2 study model\n\nEach non-empty sequence slot becomes one adjacent three-slice RGB input. DINOv2 encodes\nthree sagittal/coronal fluid-sensitive slots; each target has its own attention query.\nEmpty slots are excluded from both the encoder and the attention softmax."},{"cell_type":"code","execution_count":null,"id":"548e584e","metadata":{},"outputs":[],"source":"TARGET_SLOT_PRIOR = torch.tensor(\n    [\n        # sagFS corFS sagNOFS\n        [1.5, 1.1, 1.2],  # Medial meniscus\n        [1.5, 1.1, 1.2],  # Lateral meniscus\n    ],\n    dtype=torch.float32,\n)\n\n\nclass DiagnosisSlotHead(nn.Module):\n    def __init__(self, input_dim: int, hidden_dim: int = 384, dropout: float = 0.25):\n        super().__init__()\n        self.projection = nn.Sequential(\n            nn.LayerNorm(input_dim), nn.Linear(input_dim, hidden_dim), nn.GELU()\n        )\n        self.slot_embedding = nn.Parameter(torch.randn(N_SLOTS, hidden_dim) * 0.02)\n        self.queries = nn.Parameter(torch.randn(len(TARGETS), hidden_dim) * 0.02)\n        self.prior_strength = nn.Parameter(torch.tensor(1.0))\n        self.fusion = nn.Sequential(\n            nn.LayerNorm(hidden_dim * 3),\n            nn.Linear(hidden_dim * 3, hidden_dim),\n            nn.GELU(),\n            nn.Dropout(dropout),\n        )\n        self.output_weight = nn.Parameter(torch.randn(len(TARGETS), hidden_dim) * 0.02)\n        self.output_bias = nn.Parameter(torch.zeros(len(TARGETS)))\n        self.dropout = nn.Dropout(dropout)\n        self.scale = hidden_dim**-0.5\n        self.register_buffer(\"target_slot_prior\", TARGET_SLOT_PRIOR)\n\n    def forward(self, features: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:\n        hidden = self.projection(features) + self.slot_embedding[None]\n        attention = torch.einsum(\"bsh,th->bts\", hidden, self.queries) * self.scale\n        attention = attention + self.prior_strength.clamp(0.0, 3.0) * self.target_slot_prior[None]\n        attention = attention.masked_fill(mask[:, None] < 0.5, -1e4).softmax(dim=-1)\n        context = self.dropout(torch.einsum(\"bts,bsh->bth\", attention, hidden))\n        denominator = mask.sum(dim=1, keepdim=True).clamp_min(1.0)\n        mean = (hidden * mask[:, :, None]).sum(dim=1) / denominator\n        maximum = hidden.masked_fill(mask[:, :, None] < 0.5, -1e4).max(dim=1).values\n        global_context = torch.cat([mean, maximum], dim=-1)[:, None].expand(\n            -1, len(TARGETS), -1\n        )\n        fused = self.fusion(torch.cat([context, global_context], dim=-1))\n        return (fused * self.output_weight[None]).sum(dim=-1) + self.output_bias\n\n\nclass KneeDINOv2(nn.Module):\n    def __init__(self, backbone: nn.Module):\n        super().__init__()\n        self.backbone = backbone\n        self.head = DiagnosisSlotHead(int(backbone.config.hidden_size) * 2)\n        self.register_buffer(\n            \"image_mean\", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)\n        )\n        self.register_buffer(\n            \"image_std\", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)\n        )\n\n    def forward(self, images: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:\n        batch, slots = images.shape[:2]\n        flat = images.reshape(batch * slots, *images.shape[2:]).float().div(255.0)\n        valid = mask.reshape(-1) > 0.5\n        features = torch.zeros(\n            (batch * slots, self.backbone.config.hidden_size * 2),\n            device=images.device,\n            dtype=torch.float32,\n        )\n        if valid.any():\n            normalized = (flat[valid] - self.image_mean) / self.image_std\n            output = self.backbone(pixel_values=normalized).last_hidden_state\n            encoded = torch.cat([output[:, 0], output[:, 1:].mean(dim=1)], dim=1)\n            features[valid] = encoded.float()\n        return self.head(features.reshape(batch, slots, -1), mask)\n\n\ndef build_model() -> KneeDINOv2:\n    from transformers import AutoModel\n\n    source = DINO_ROOT or find_dinov2_root(DINO_VARIANT)\n    backbone = AutoModel.from_pretrained(str(source), local_files_only=True)\n    if hasattr(backbone, \"gradient_checkpointing_enable\"):\n        try:\n            backbone.gradient_checkpointing_enable(\n                gradient_checkpointing_kwargs={\"use_reentrant\": False}\n            )\n        except TypeError:\n            backbone.gradient_checkpointing_enable()\n    for parameter in backbone.parameters():\n        parameter.requires_grad = False\n    layers = backbone.encoder.layer\n    for layer in layers[max(0, len(layers) - UNFREEZE_LAST) :]:\n        for parameter in layer.parameters():\n            parameter.requires_grad = True\n    for parameter in backbone.layernorm.parameters():\n        parameter.requires_grad = True\n    trainable = sum(parameter.numel() for parameter in backbone.parameters() if parameter.requires_grad)\n    log(f\"DINOv2 source={source}; trainable backbone parameters={trainable / 1e6:.1f}M\")\n    return KneeDINOv2(backbone)\n\n\ndef take_group(\n    cache: CompressedStudyCache, indices: np.ndarray, group: int\n) -> torch.Tensor:\n    return torch.from_numpy(cache.load_group(indices, group))\n\n\ndef augment(images: torch.Tensor) -> torch.Tensor:\n    batch, slots, channels, height, width = images.shape\n    flat = images.float().reshape(batch * slots, channels, height, width)\n    angles = (torch.rand(batch, device=images.device) * 2 - 1) * math.radians(7.0)\n    scales = 1.0 + (torch.rand(batch, device=images.device) * 2 - 1) * 0.07\n    shifts = (torch.rand(batch, 2, device=images.device) * 2 - 1) * 0.04\n    cosine = torch.cos(angles) / scales\n    sine = torch.sin(angles) / scales\n    theta = torch.zeros(batch, 2, 3, device=images.device)\n    theta[:, 0, 0], theta[:, 0, 1] = cosine, -sine\n    theta[:, 1, 0], theta[:, 1, 1] = sine, cosine\n    theta[:, :, 2] = shifts\n    theta = theta.repeat_interleave(slots, dim=0)\n    grid = F.affine_grid(theta, flat.shape, align_corners=False)\n    flat = F.grid_sample(flat, grid, mode=\"bilinear\", padding_mode=\"zeros\", align_corners=False)\n    intensity = 1.0 + (torch.rand(batch * slots, 1, 1, 1, device=images.device) * 2 - 1) * 0.10\n    flat = (flat * intensity).clamp(0, 255)\n    return flat.reshape(batch, slots, channels, height, width).round().to(torch.uint8)\n\n\ndef pairwise_rank_loss(logits: torch.Tensor, labels: torch.Tensor, weights: torch.Tensor) -> torch.Tensor:\n    losses = []\n    for target_index in range(logits.shape[1]):\n        confident = weights[:, target_index] >= 0.6\n        positives = logits[(labels[:, target_index] >= 0.6) & confident, target_index]\n        negatives = logits[(labels[:, target_index] <= 0.4) & confident, target_index]\n        if len(positives) and len(negatives):\n            losses.append(F.softplus(-(positives[:, None] - negatives[None, :])).mean())\n    return torch.stack(losses).mean() if losses else logits.new_tensor(0.0)\n\n\ndef macro_auc(\n    labels: np.ndarray,\n    predictions: np.ndarray,\n    weights: np.ndarray | None = None,\n) -> float:\n    scores = []\n    for index in range(labels.shape[1]):\n        known = np.isfinite(labels[:, index]) & np.isfinite(predictions[:, index])\n        if weights is not None:\n            known &= weights[:, index] > 0\n        target_labels = (labels[known, index] >= 0.5).astype(np.uint8)\n        if len(np.unique(target_labels)) == 2:\n            scores.append(roc_auc_score(target_labels, predictions[known, index]))\n    return float(np.mean(scores)) if scores else float(\"nan\")\n\n\ndef rank_columns(values: np.ndarray) -> np.ndarray:\n    return pd.DataFrame(values).rank(method=\"average\", pct=True).to_numpy(np.float64)\n\n\ndef binary_auc(labels: np.ndarray, predictions: np.ndarray, weights: np.ndarray) -> float:\n    known = (weights > 0) & np.isfinite(labels) & np.isfinite(predictions)\n    truth = (labels[known] >= 0.5).astype(np.uint8)\n    if np.unique(truth).size < 2:\n        return float(\"nan\")\n    return float(roc_auc_score(truth, predictions[known]))\n\n\ndef write_paired_oof_gate(\n    valid_ids: list[str],\n    labels: np.ndarray,\n    weights: np.ndarray,\n    candidate: np.ndarray,\n    expected_evaluable_ids: list[str] | None = None,\n) -> dict[str, object]:\n    \"\"\"Cross-fit target-wise blend weights against the fixed Fold-0 control.\"\"\"\n    control_path = find_asset_file(\"control_fold0_oof.csv\")\n    control = pd.read_csv(control_path, dtype={\"StudyInstanceUID\": str}).set_index(\n        \"StudyInstanceUID\"\n    )\n    missing = [study for study in valid_ids if study not in control.index]\n    if missing:\n        raise RuntimeError(f\"control Fold-0 OOF is missing {len(missing)} candidate studies\")\n    raw_control_coverage = len(valid_ids) / max(len(control), 1)\n    expected_evaluable = set(expected_evaluable_ids or valid_ids)\n    evaluable_coverage = len(set(valid_ids) & expected_evaluable) / max(\n        len(expected_evaluable), 1\n    )\n    if evaluable_coverage < 0.98:\n        log(\n            f\"WARNING: candidate covers only {evaluable_coverage:.1%} of evaluable Fold 0; \"\n            \"the gate will fail. Raw control-row coverage is {raw_control_coverage:.1%}.\"\n        )\n    control_values = control.loc[valid_ids, TARGETS].to_numpy(np.float64)\n    control_rank = rank_columns(control_values)\n    candidate_rank = rank_columns(candidate)\n    halves = np.asarray(\n        [int(hashlib.sha256(study.encode()).hexdigest()[-1], 16) % 2 for study in valid_ids],\n        dtype=np.uint8,\n    )\n    blend = np.full_like(candidate_rank, np.nan)\n    rows = []\n    weight_grid = np.arange(0.0, 0.36, 0.05)\n\n    for target_index, target in enumerate(TARGETS):\n        selected_weights = []\n        for validation_half in (0, 1):\n            choose = halves != validation_half\n            validate = halves == validation_half\n            best_weight, best_objective = 0.0, -np.inf\n            for candidate_weight in weight_grid:\n                mixed = (\n                    (1.0 - candidate_weight) * control_rank[:, target_index]\n                    + candidate_weight * candidate_rank[:, target_index]\n                )\n                score = binary_auc(\n                    labels[choose, target_index],\n                    mixed[choose],\n                    weights[choose, target_index],\n                )\n                objective = score - 0.0015 * float(candidate_weight)\n                if np.isfinite(objective) and objective > best_objective:\n                    best_weight = float(candidate_weight)\n                    best_objective = objective\n            blend[validate, target_index] = (\n                (1.0 - best_weight) * control_rank[validate, target_index]\n                + best_weight * candidate_rank[validate, target_index]\n            )\n            selected_weights.append(best_weight)\n\n        control_auc = binary_auc(\n            labels[:, target_index], control_rank[:, target_index], weights[:, target_index]\n        )\n        candidate_auc = binary_auc(\n            labels[:, target_index], candidate_rank[:, target_index], weights[:, target_index]\n        )\n        blend_auc = binary_auc(\n            labels[:, target_index], blend[:, target_index], weights[:, target_index]\n        )\n        correlation = float(\n            np.corrcoef(control_rank[:, target_index], candidate_rank[:, target_index])[0, 1]\n        )\n        rows.append(\n            {\n                \"target\": target,\n                \"control_auc\": control_auc,\n                \"candidate_auc\": candidate_auc,\n                \"candidate_gain\": candidate_auc - control_auc,\n                \"crossfit_blend_auc\": blend_auc,\n                \"crossfit_blend_gain\": blend_auc - control_auc,\n                \"rank_correlation\": correlation,\n                \"weight_for_half0\": selected_weights[0],\n                \"weight_for_half1\": selected_weights[1],\n            }\n        )\n\n    metrics = pd.DataFrame(rows)\n    metrics.to_csv(\"meniscus_series_safe_v1_gate_targets.csv\", index=False)\n    control_macro = float(metrics[\"control_auc\"].mean())\n    candidate_macro = float(metrics[\"candidate_auc\"].mean())\n    blend_macro = float(metrics[\"crossfit_blend_auc\"].mean())\n    blend_gain = blend_macro - control_macro\n    minimum_target_gain = float(metrics[\"crossfit_blend_gain\"].min())\n    maximum_target_gain = float(metrics[\"crossfit_blend_gain\"].max())\n    seed_thresholds_passed = bool(\n        evaluable_coverage >= 0.98\n        and blend_gain >= 0.002\n        and minimum_target_gain >= -0.002\n        and maximum_target_gain >= 0.004\n    )\n    receipt = {\n        \"version\": \"meniscus-series-safe-v1-paired-gate\",\n        \"fold\": int(TRAIN_FOLD),\n        \"seed\": int(RUN_SEED),\n        \"held_studies\": len(valid_ids),\n        \"control_fold_rows\": int(len(control)),\n        \"raw_control_fold_coverage\": raw_control_coverage,\n        \"evaluable_control_rows\": int(len(expected_evaluable)),\n        \"evaluable_control_coverage\": evaluable_coverage,\n        \"targets\": TARGETS,\n        \"control\": str(control_path),\n        \"truth\": \"fixed confidence-weighted report labels with expert labels overriding gold rows\",\n        \"control_macro_auc\": control_macro,\n        \"candidate_macro_auc\": candidate_macro,\n        \"candidate_gain\": candidate_macro - control_macro,\n        \"crossfit_blend_macro_auc\": blend_macro,\n        \"crossfit_blend_gain\": blend_gain,\n        \"minimum_target_gain\": minimum_target_gain,\n        \"maximum_target_gain\": maximum_target_gain,\n        \"thresholds\": {\n            \"crossfit_blend_gain\": 0.002,\n            \"minimum_target_gain\": -0.002,\n            \"one_target_gain\": 0.004,\n        },\n        \"seed_thresholds_passed\": seed_thresholds_passed,\n        \"confirmation_run\": True,\n        \"train_remaining_folds\": False,\n        \"required_next_step\": (\n            \"Seed 6203 passed. Compare it with seed 3407 before authorizing folds 1-4.\"\n            if seed_thresholds_passed\n            else \"Confirmation failed; reject this candidate and do not train folds 1-4.\"\n        ),\n        \"score_guarantee\": False,\n    }\n    Path(\"MENISCUS_SERIES_SAFE_V1_GATE.json\").write_text(\n        json.dumps(receipt, indent=2) + \"\\n\", encoding=\"utf-8\"\n    )\n    log(\"paired gate: \" + json.dumps(receipt))\n    return receipt\n\n\n@torch.inference_mode()\ndef predict(\n    model: KneeDINOv2,\n    cache: CompressedStudyCache,\n    mask: np.ndarray,\n    indices: np.ndarray,\n    device: torch.device,\n) -> np.ndarray:\n    model.eval()\n    predictions = []\n    for offset in range(0, len(indices), EVAL_BATCH):\n        selection = indices[offset : offset + EVAL_BATCH]\n        batch_mask = torch.from_numpy(mask[selection]).to(device)\n        group_probabilities = []\n        for group in range(N_GROUPS):\n            images = take_group(cache, selection, group).to(device)\n            with torch.autocast(device_type=device.type, enabled=device.type == \"cuda\"):\n                group_logits = model(images, batch_mask).float()\n            group_probabilities.append(torch.sigmoid(group_logits))\n        stacked = torch.stack(group_probabilities, dim=1)\n        pooled = stacked.mean(dim=1)\n        predictions.append(pooled.cpu().numpy())\n    if not predictions:\n        return np.zeros((0, len(TARGETS)), dtype=np.float32)\n    return np.concatenate(predictions)"},{"cell_type":"markdown","id":"f2c9dacd","metadata":{},"source":"## Targets and leakage-controlled folds\n\nDuplicate reports and studies from the same identifiable scanner remain in one fold.\nThis is a harder validation split, but it prevents report templates and site acquisition\nstyle from masquerading as image generalization."},{"cell_type":"code","execution_count":null,"id":"8ffa4aaa","metadata":{},"outputs":[],"source":"Y = np.zeros((len(train_studies), len(TARGETS)), dtype=np.float32)\nW = np.zeros_like(Y)\nfor index, study in enumerate(train_studies):\n    if study in gold.index:\n        Y[index] = gold.loc[study, TARGETS].to_numpy(dtype=np.float32)\n        W[index] = GOLD_WEIGHT\n    elif study in report_labels.index:\n        Y[index] = report_labels.loc[study, TARGETS].to_numpy(dtype=np.float32)\n        confidence_columns = [target + \"__conf\" for target in TARGETS]\n        confidence = report_labels.loc[study, confidence_columns].to_numpy(dtype=np.float32)\n        W[index] = np.where(confidence > 0, 0.15 + 0.85 * confidence, 0.0)\n\nusable = np.where((W.sum(axis=1) > 0) & (train_mask.sum(axis=1) > 0))[0]\nlog(f\"usable studies={len(usable)}/{len(train_studies)}\")\n\nreports = train_df.set_index(\"StudyInstanceUID\")[\"Report\"].fillna(\"\")\n\n\ndef scanner_fingerprint(study: str) -> str | None:\n    slots = train_selected.get(study, {})\n    if not slots:\n        return None\n    folder = slots[sorted(slots)[0]]\n    files = sorted(folder.glob(\"*.dcm\"))\n    if not files:\n        return None\n    try:\n        ds = pydicom.dcmread(\n            str(files[len(files) // 2]), stop_before_pixels=True, force=True\n        )\n    except Exception:\n        return None\n    identity = [\n        str(getattr(ds, name, \"\") or \"\").strip().lower()\n        for name in (\"InstitutionName\", \"StationName\", \"DeviceSerialNumber\")\n    ]\n    if not any(identity):\n        return None\n    protocol = [\n        str(getattr(ds, name, \"\") or \"\").strip().lower()\n        for name in (\"Manufacturer\", \"ManufacturerModelName\", \"MagneticFieldStrength\")\n    ]\n    return hashlib.sha256(\"|\".join(identity + protocol).encode()).hexdigest()\n\n\ndef grouped_multilabel_folds(indices: np.ndarray) -> np.ndarray:\n    \"\"\"Balance targets over connected report/scanner groups.\"\"\"\n    parent = {int(index): int(index) for index in indices}\n\n    def find(index: int) -> int:\n        while parent[index] != index:\n            parent[index] = parent[parent[index]]\n            index = parent[index]\n        return index\n\n    def union(left: int, right: int) -> None:\n        left_root, right_root = find(left), find(right)\n        if left_root != right_root:\n            parent[right_root] = left_root\n\n    token_owner: dict[str, int] = {}\n    scanner_records = []\n    for index in indices:\n        index = int(index)\n        report = normalize_report(reports.get(train_studies[index], \"\"))\n        report_key = hashlib.sha256((report or train_studies[index]).encode()).hexdigest()\n        scanner_key = scanner_fingerprint(train_studies[index])\n        tokens = [\"report::\" + report_key]\n        if scanner_key is not None:\n            tokens.append(\"scanner::\" + scanner_key)\n        for token in tokens:\n            if token in token_owner:\n                union(index, token_owner[token])\n            else:\n                token_owner[token] = index\n        scanner_records.append(\n            {\"StudyInstanceUID\": train_studies[index], \"scanner_group\": scanner_key or \"\"}\n        )\n\n    grouped: dict[str, list[int]] = {}\n    for index in indices:\n        root = str(find(int(index)))\n        grouped.setdefault(root, []).append(int(index))\n    pd.DataFrame(scanner_records).to_csv(\"scanner_groups.csv\", index=False)\n\n    known = W > 0\n    positive = (Y >= 0.5) & known\n    target_positive = positive[indices].sum(axis=0).astype(np.float64)\n    target_size = len(indices) / N_FOLDS\n    fold_size = np.zeros(N_FOLDS, dtype=np.float64)\n    fold_positive = np.zeros((N_FOLDS, len(TARGETS)), dtype=np.float64)\n    fold_of = np.full(len(train_studies), -1, dtype=np.int16)\n\n    items = []\n    for key, members in grouped.items():\n        members_array = np.asarray(members, dtype=int)\n        positives = positive[members_array].sum(axis=0).astype(np.float64)\n        rarity = float(np.sum(positives / np.maximum(target_positive, 1.0)))\n        items.append((key, members_array, positives, rarity))\n    items.sort(key=lambda item: (-item[3], -len(item[1]), item[0]))\n\n    rng = np.random.default_rng(RUN_SEED)\n    fold_order = rng.permutation(N_FOLDS)\n    for _, members, positives, _ in items:\n        costs = []\n        for fold in fold_order:\n            candidate_size = fold_size.copy()\n            candidate_positive = fold_positive.copy()\n            candidate_size[fold] += len(members)\n            candidate_positive[fold] += positives\n            size_cost = np.mean((candidate_size / max(target_size, 1.0) - 1.0) ** 2)\n            expected_positive = np.maximum(target_positive / N_FOLDS, 1.0)\n            positive_cost = np.mean(\n                (candidate_positive / expected_positive[None, :] - 1.0) ** 2\n            )\n            costs.append((positive_cost + 0.15 * size_cost, int(fold)))\n        _, chosen = min(costs)\n        fold_of[members] = chosen\n        fold_size[chosen] += len(members)\n        fold_positive[chosen] += positives\n    return fold_of\n\n\nfixed_fold_path = find_asset_file(\"folds.csv\")\nfixed_folds = pd.read_csv(fixed_fold_path, dtype={\"StudyInstanceUID\": str})\nif fixed_folds[\"StudyInstanceUID\"].duplicated().any():\n    raise RuntimeError(\"fixed folds contain duplicate study IDs\")\nfixed_fold_map = fixed_folds.set_index(\"StudyInstanceUID\")[\"fold\"]\nmissing_fold_ids = [study for study in train_studies if study not in fixed_fold_map.index]\nif missing_fold_ids:\n    raise RuntimeError(f\"fixed folds are missing {len(missing_fold_ids)} studies\")\nfold_of = fixed_fold_map.loc[train_studies].to_numpy(dtype=np.int16)\nif set(np.unique(fold_of)) != set(range(N_FOLDS)):\n    raise RuntimeError(f\"expected fixed folds 0..{N_FOLDS - 1}, got {np.unique(fold_of)}\")\nlog(f\"reused paired-control fold assignment: {fixed_fold_path}\")\nfold_records = pd.DataFrame(\n    {\n        \"StudyInstanceUID\": train_studies,\n        \"fold\": fold_of,\n        \"usable\": np.isin(np.arange(len(train_studies)), usable).astype(np.uint8),\n    }\n)\nfold_records.to_csv(\"folds.csv\", index=False)\nfold_table = pd.DataFrame(\n    {\"fold\": fold_of[usable], \"gold\": [train_studies[index] in gold.index for index in usable]}\n).groupby(\"fold\").agg(studies=(\"fold\", \"size\"), gold=(\"gold\", \"sum\"))\ndisplay(fold_table)\n\n\ndef balanced_epoch_batches(\n    train_indices: np.ndarray, steps: int, rng: np.random.Generator\n) -> list[np.ndarray]:\n    \"\"\"Build positive/negative pairs so the AUC surrogate is active in every step.\"\"\"\n    pools = []\n    for target_index in range(len(TARGETS)):\n        confident = W[train_indices, target_index] >= 0.45\n        positive = train_indices[confident & (Y[train_indices, target_index] >= 0.60)]\n        negative = train_indices[confident & (Y[train_indices, target_index] <= 0.40)]\n        if len(positive) and len(negative):\n            pools.append((target_index, positive, negative))\n    if not pools:\n        raise RuntimeError(\"No confident positive/negative target pools were found\")\n\n    rarity = np.asarray(\n        [1.0 / math.sqrt(max(len(positive), 1)) for _, positive, _ in pools], dtype=float\n    )\n    probability = rarity / rarity.sum()\n    batches = []\n    for _ in range(steps):\n        pool_index = int(rng.choice(len(pools), p=probability))\n        _, positive, negative = pools[pool_index]\n        selection = [int(rng.choice(positive)), int(rng.choice(negative))]\n        while len(selection) < BATCH_STUDIES:\n            selection.append(int(rng.choice(train_indices)))\n        rng.shuffle(selection)\n        batches.append(np.asarray(selection, dtype=int))\n    return batches\n\n\ndef train_fold(\n    fold: int,\n    train_indices: np.ndarray,\n    valid_indices: np.ndarray,\n    device: torch.device,\n) -> tuple[dict[str, torch.Tensor], list[dict[str, float]]]:\n    seed_everything(RUN_SEED + fold)\n    model = build_model().to(device)\n    optimizer = torch.optim.AdamW(\n        [\n            {\"params\": [p for p in model.backbone.parameters() if p.requires_grad], \"lr\": LR_BACKBONE},\n            {\"params\": model.head.parameters(), \"lr\": LR_HEAD},\n        ],\n        weight_decay=WEIGHT_DECAY,\n    )\n    steps_per_epoch = min(STEPS_PER_EPOCH, max(1, len(train_indices) // BATCH_STUDIES))\n    elapsed_before_training = time.time() - START_TIME\n    if elapsed_before_training > 6.5 * 3600:\n        steps_per_epoch = min(steps_per_epoch, 100)\n        log(\"cache was slow; emergency 100-step epoch enabled so an OOF gate is still written\")\n    elif elapsed_before_training > 5.0 * 3600:\n        steps_per_epoch = min(steps_per_epoch, 300)\n        log(\"cache was slow; reduced each epoch to 300 steps\")\n    scheduler = torch.optim.lr_scheduler.OneCycleLR(\n        optimizer,\n        max_lr=[LR_BACKBONE, LR_HEAD],\n        total_steps=steps_per_epoch * EPOCHS,\n        pct_start=0.15,\n    )\n    scaler = torch.amp.GradScaler(\"cuda\", enabled=device.type == \"cuda\")\n    best_score, best_state, history = -np.inf, None, []\n    partial_stem = f\"meniscus_series_safe_base_seed{RUN_SEED}_fold{fold}\"\n    prevalence = (\n        (Y[train_indices] * W[train_indices]).sum(axis=0)\n        / W[train_indices].sum(axis=0).clip(min=1e-6)\n    ).clip(0.03, 0.97)\n    positive_scale = np.sqrt(0.5 / prevalence).clip(0.5, 3.0)\n    negative_scale = np.sqrt(0.5 / (1.0 - prevalence)).clip(0.5, 3.0)\n\n    for epoch in range(EPOCHS):\n        model.train()\n        epoch_rng = np.random.default_rng(RUN_SEED * 1000 + fold * 100 + epoch)\n        batches = balanced_epoch_batches(train_indices, steps_per_epoch, epoch_rng)\n        losses = []\n        budget_hit = False\n        for step, selection in enumerate(batches):\n            if step >= 100 and step % 25 == 0 and time.time() - START_TIME > TRAIN_STOP_SECONDS:\n                log(f\"time budget reached inside epoch {epoch + 1} after {step} steps\")\n                budget_hit = True\n                break\n            group = int(epoch_rng.integers(0, N_GROUPS))\n            images = augment(take_group(train_cache, selection, group).to(device))\n            batch_mask = torch.from_numpy(train_mask[selection]).to(device)\n            labels = torch.from_numpy(Y[selection]).to(device)\n            weights = torch.from_numpy(W[selection]).to(device)\n            labels_for_bce = labels * (1.0 - 2.0 * LABEL_SMOOTHING) + LABEL_SMOOTHING\n            class_scale = labels * torch.as_tensor(\n                positive_scale, device=device\n            ) + (1.0 - labels) * torch.as_tensor(negative_scale, device=device)\n            optimizer.zero_grad(set_to_none=True)\n            with torch.autocast(device_type=device.type, enabled=device.type == \"cuda\"):\n                logits = model(images, batch_mask)\n                bce = F.binary_cross_entropy_with_logits(\n                    logits, labels_for_bce, reduction=\"none\"\n                )\n                effective_weight = weights * class_scale\n                loss = (bce * effective_weight).sum() / effective_weight.sum().clamp_min(1.0)\n                loss = loss + RANK_LOSS_WEIGHT * pairwise_rank_loss(logits.float(), labels, weights)\n            scaler.scale(loss).backward()\n            scaler.unscale_(optimizer)\n            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n            scaler.step(optimizer)\n            scaler.update()\n            scheduler.step()\n            losses.append(float(loss.detach().cpu()))\n\n        valid_predictions = predict(model, train_cache, train_mask, valid_indices, device)\n        score = macro_auc(Y[valid_indices], valid_predictions, W[valid_indices])\n        gold_indices = np.array(\n            [index for index in valid_indices if train_studies[index] in gold.index], dtype=int\n        )\n        gold_score = float(\"nan\")\n        if len(gold_indices) >= 6:\n            gold_y = gold.loc[[train_studies[index] for index in gold_indices], TARGETS].to_numpy(\n                dtype=np.int8\n            )\n            gold_predictions = predict(model, train_cache, train_mask, gold_indices, device)\n            gold_score = macro_auc(gold_y, gold_predictions)\n        mean_loss = float(np.mean(losses))\n        history.append(\n            {\"fold\": fold, \"epoch\": epoch + 1, \"loss\": mean_loss,\n             \"weak_auc\": score, \"gold_auc\": gold_score}\n        )\n        log(\n            f\"fold={fold} epoch={epoch + 1}/{EPOCHS} loss={mean_loss:.4f} \"\n            f\"weak_auc={score:.4f} gold_auc={gold_score:.4f}\"\n        )\n        if np.isfinite(score) and score > best_score:\n            best_score = score\n            best_state = {key: value.detach().cpu().clone() for key, value in model.state_dict().items()}\n            partial_checkpoint = {\n                \"model\": best_state,\n                \"seed\": RUN_SEED,\n                \"fold\": fold,\n                \"targets\": TARGETS,\n                \"partial\": True,\n                \"epoch\": epoch + 1,\n                \"weak_auc\": score,\n            }\n            temporary_checkpoint = Path(partial_stem + \".partial.tmp\")\n            torch.save(partial_checkpoint, temporary_checkpoint)\n            os.replace(temporary_checkpoint, Path(partial_stem + \".partial.pt\"))\n            valid_ids = np.asarray(train_studies)[valid_indices].astype(str).tolist()\n            pd.DataFrame(valid_predictions, columns=TARGETS, index=valid_ids).rename_axis(\n                \"StudyInstanceUID\"\n            ).to_csv(partial_stem + \".partial_oof.csv\")\n            gate = write_paired_oof_gate(\n                valid_ids,\n                Y[valid_indices],\n                W[valid_indices],\n                valid_predictions,\n                np.asarray(train_studies)[\n                    (fold_of == fold) & (W.sum(axis=1) > 0)\n                ].astype(str).tolist(),\n            )\n            pd.DataFrame(history).to_csv(\"training_history.partial.csv\", index=False)\n            write_run_status(\n                \"training_checkpointed\",\n                epoch=epoch + 1,\n                epochs_planned=EPOCHS,\n                steps_completed=len(losses),\n                best_weak_auc=best_score,\n                seed_thresholds_passed=gate[\"seed_thresholds_passed\"],\n                partial_checkpoint=partial_stem + \".partial.pt\",\n            )\n        else:\n            write_run_status(\n                \"training_epoch_complete\",\n                epoch=epoch + 1,\n                epochs_planned=EPOCHS,\n                steps_completed=len(losses),\n                best_weak_auc=best_score,\n            )\n        if budget_hit or time.time() - START_TIME > TRAIN_STOP_SECONDS:\n            log(\"time budget reached during fold\")\n            break\n\n    if best_state is None:\n        best_state = {key: value.detach().cpu().clone() for key, value in model.state_dict().items()}\n    del model, optimizer, scheduler, scaler\n    gc.collect()\n    if device.type == \"cuda\":\n        torch.cuda.empty_cache()\n    return best_state, history"},{"cell_type":"markdown","id":"5f2abe21","metadata":{},"source":"## Train the selected fold and export artifacts"},{"cell_type":"code","execution_count":null,"id":"b1dee321","metadata":{"lines_to_next_cell":2},"outputs":[],"source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nif device.type != \"cuda\" and not DEBUG:\n    raise RuntimeError(\"Enable a Kaggle GPU accelerator before running the full notebook\")\n\noof = np.full((len(train_studies), len(TARGETS)), np.nan, dtype=np.float32)\nall_history = []\n\nfor fold in [TRAIN_FOLD]:\n    train_indices = usable[fold_of[usable] != fold]\n    valid_indices = usable[fold_of[usable] == fold]\n    if len(train_indices) < BATCH_STUDIES or len(valid_indices) == 0:\n        raise RuntimeError(\n            f\"fold {fold} is empty or too small: train={len(train_indices)} \"\n            f\"valid={len(valid_indices)}\"\n        )\n    log(f\"starting fold={fold}: train={len(train_indices)} valid={len(valid_indices)}\")\n    write_run_status(\n        \"training_start\",\n        train_studies=len(train_indices),\n        valid_studies=len(valid_indices),\n        epochs=EPOCHS,\n        steps_per_epoch=STEPS_PER_EPOCH,\n    )\n    fold_start = time.time()\n    state, history = train_fold(fold, train_indices, valid_indices, device)\n    all_history.extend(history)\n\n    model = build_model().to(device)\n    model.load_state_dict(state)\n    oof[valid_indices] = predict(model, train_cache, train_mask, valid_indices, device)\n    artifact_stem = f\"meniscus_series_safe_base_seed{RUN_SEED}_fold{fold}\"\n    if SAVE_FOLD_MODELS:\n        torch.save(\n            {\n                \"model\": state,\n                \"seed\": RUN_SEED,\n                \"fold\": fold,\n                \"targets\": TARGETS,\n                \"config\": {\n                    \"architecture\": \"dinov2-base-series-safe-meniscus-attention-v1\",\n                    \"variant\": DINO_VARIANT,\n                    \"image_size\": IMAGE_SIZE,\n                    \"crop_mm\": CROP_MM,\n                    \"group_size\": GROUP_SIZE,\n                    \"n_groups\": N_GROUPS,\n                    \"slices\": CACHE_SLICES,\n                    \"unfreeze_last\": UNFREEZE_LAST,\n                    \"slots\": [slot[0] for slot in SLOTS],\n                    \"slice_sampling\": \"four physically ordered adjacent triplets per series\",\n                    \"series_boundary_policy\": \"one SeriesInstanceUID per slot and triplet\",\n                    \"pool\": \"mean_probability_over_triplets\",\n                    \"label_source\": label_source,\n                    \"label_consensus\": \"fixed-from-control-v1\",\n                    \"fold_contract\": \"fixed-paired-control-folds-v1\",\n                    \"target_slot_prior\": TARGET_SLOT_PRIOR.tolist(),\n                },\n            },\n            artifact_stem + \".pt\",\n        )\n    np.save(artifact_stem + \"_oof.npy\", oof)\n    pd.DataFrame(\n        oof[valid_indices], columns=TARGETS, index=np.asarray(train_studies)[valid_indices]\n    ).rename_axis(\"StudyInstanceUID\").to_csv(artifact_stem + \"_oof.csv\")\n    valid_ids = np.asarray(train_studies)[valid_indices].astype(str).tolist()\n    gate_receipt = write_paired_oof_gate(\n        valid_ids,\n        Y[valid_indices],\n        W[valid_indices],\n        oof[valid_indices],\n        np.asarray(train_studies)[\n            (fold_of == fold) & (W.sum(axis=1) > 0)\n        ].astype(str).tolist(),\n    )\n    del model, state\n    gc.collect()\n    if device.type == \"cuda\":\n        torch.cuda.empty_cache()\n\n    fold_seconds = time.time() - fold_start\n    log(f\"finished seed={RUN_SEED} fold={fold} in {fold_seconds / 60:.1f} min\")\n\nhistory_df = pd.DataFrame(all_history)\nhistory_df.to_csv(f\"training_history_seed{RUN_SEED}_fold{TRAIN_FOLD}.csv\", index=False)\ngold_positions = np.array(\n    [index for index, study in enumerate(train_studies) if study in gold.index], dtype=int\n)\nseen = gold_positions[np.isfinite(oof[gold_positions]).all(axis=1)]\nif len(seen) >= 8:\n    gold_y = gold.loc[[train_studies[index] for index in seen], TARGETS].to_numpy(dtype=np.int8)\n    log(f\"gold OOF macro AUC on {len(seen)} studies: {macro_auc(gold_y, oof[seen]):.4f}\")\n\nmanifest = {\n    \"artifact_type\": \"single_fold_training_output\",\n    \"not_for_submission\": True,\n    \"seed\": RUN_SEED,\n    \"fold\": TRAIN_FOLD,\n    \"checkpoint\": artifact_stem + \".pt\",\n    \"oof_csv\": artifact_stem + \"_oof.csv\",\n    \"gate\": \"MENISCUS_SERIES_SAFE_V1_GATE.json\",\n    \"seed_thresholds_passed\": gate_receipt[\"seed_thresholds_passed\"],\n    \"train_remaining_folds\": False,\n    \"required_next_step\": gate_receipt[\"required_next_step\"],\n}\nPath(f\"training_manifest_seed{RUN_SEED}_fold{TRAIN_FOLD}.json\").write_text(\n    json.dumps(manifest, indent=2) + \"\\n\"\n)\nwrite_run_status(\n    \"complete\",\n    checkpoint=artifact_stem + \".pt\",\n    oof_csv=artifact_stem + \"_oof.csv\",\n    gate=\"MENISCUS_SERIES_SAFE_V1_GATE.json\",\n    seed_thresholds_passed=gate_receipt[\"seed_thresholds_passed\"],\n)\nlog(f\"complete: artifact-only seed={RUN_SEED}, fold={TRAIN_FOLD}; no submission.csv\")\ndisplay(history_df.tail())\n\nif DELETE_TEMP_CACHE:\n    cache_root = train_cache.root\n    if cache_root.is_dir() and CACHE_ROOT in cache_root.parents:\n        shutil.rmtree(cache_root)\n    log(\"removed temporary compressed image caches\")"},{"cell_type":"markdown","id":"dfc89688","metadata":{},"source":"## Required run order\n\nRead `MENISCUS_SERIES_SAFE_V1_GATE.json`. This confirmation must reproduce the\nseed-3407 blend gain on both targets. Even a pass leaves `train_remaining_folds=false`;\ncompare both receipts before generating folds 1-4 and an inference extension."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":5}