{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"cv2_notes":{"source":"rsna-knee-abnormality-detection-cv1.ipynb","builder":"build_cv2_notebook.py","cache_version":"cv2-orient-roi-letterbox-256-12-v1"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 0. Configuration\n\nRun one stage at a time. The submission stage must have internet disabled and must receive model\nweights and decoder dependencies through attached Kaggle Datasets. Training all folds is deliberate;\nset `TRAIN_FOLDS=[0]` only for a smoke test, never for the final ensemble.\n","metadata":{}},{"cell_type":"code","source":"import os, re, sys, gc, json, math, time, warnings, hashlib, unicodedata, itertools\nfrom pathlib import Path\nfrom collections import Counter\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nwarnings.filterwarnings('ignore')\npd.set_option('display.width', 200)\npd.set_option('display.max_columns', 80)\n\n\nclass CFG:\n    # ------------------------------ choose exactly one stage\n    # The previous Kaggle run failed with \"Submission File Not Found\" because the\n    # notebook was committed with STAGE='eda'. Only the 'infer' stage writes\n    # submission.csv, so a code competition re-run had nothing to score.\n    # Anything you submit must be committed with STAGE = 'infer'.\n    STAGE = 'infer'  # eda | teacher | preprocess | train | infer\n    VALID_STAGES = {'eda', 'teacher', 'preprocess', 'train', 'infer'}\n\n    # ------------------------------ paths\n    # Kaggle normally mounts the competition at /kaggle/input/<slug>; the\n    # /competitions/ prefix only exists in some environments. Both are probed at the\n    # bottom of this cell, so a wrong literal here is no longer fatal.\n    comp_dir = Path('/kaggle/input/rsna-knee-abnormality-detection')\n    work_dir = Path('/kaggle/working')\n    teacher_dir = work_dir / 'rsna_teacher_v2'\n    cache_dir = work_dir / 'rsna_cache_v2'\n    ckpt_dir = work_dir / 'rsna_models_v2'\n\n    # Attach published stage outputs here. Do not recursively scan the 570 GB competition mount.\n    extra_teacher_dirs = []  # e.g. [Path('/kaggle/input/rsna-teacher-v2')]\n    extra_cache_dirs = []    # e.g. [Path('/kaggle/input/rsna-cache-v2-shard00'), ...]\n    extra_ckpt_dirs = []     # e.g. [Path('/kaggle/input/rsna-models-v2')]\n    # Directories of .whl files for offline DICOM decoders (see the preflight cell).\n    wheel_dirs = []          # e.g. [Path('/kaggle/input/pydicom-decoder-wheels')]\n\n    # ------------------------------ preprocessing\n    CACHE_VERSION = 'cv2-orient-roi-letterbox-256-12-v1'\n    IMG_SIZE = 256\n    N_SLICES = 12\n    VIEWS = ['SAG_FS', 'SAG_NFS', 'COR_FS', 'COR_NFS', 'AX']\n    N_VIEWS = len(VIEWS)\n    SHARD_ID = 0\n    NUM_SHARDS = 8\n    PREPROCESS_SMOKE_ONLY = False\n\n    # ------------------------------ model\n    backbone = [\n        'tf_efficientnetv2_s.in21k_ft_in1k',\n        'tf_efficientnetv2_s',\n        'efficientnetv2_rw_s',\n        'tf_efficientnetv2_b0',\n    ]\n    pretrained = True\n    drop_rate = 0.25\n    drop_path_rate = 0.15\n    grad_ckpt = True\n\n    # ------------------------------ training\n    folds = 5\n    TRAIN_FOLDS = None       # None = all folds; [0] = smoke test\n    mixed_epochs = 6         # hard + report-derived targets\n    hard_epochs = 2          # final hard-label-only fine-tuning\n    batch_size = 2\n    infer_batch_size = 4\n    accum = 4\n    lr = 3e-4\n    backbone_lr = 8e-5\n    wd = 1e-2\n    pseudo_w = 0.45\n    pseudo_conf_floor = 0.20\n    aux_w = 0.20\n    max_pos_weight = 5.0\n    ema_decay = 0.999\n    seed = 42\n    num_workers = 4\n    amp = True\n\n    # ------------------------------ submission robustness\n    # Fit a CPU model on the series descriptors (planes / fluid-sensitivity / fat-sat)\n    # that are available for the test set, so the notebook still submits something\n    # better than 0.5 before any image checkpoint exists.\n    USE_METADATA_FALLBACK = True\n    # Horizontal flip is NOT a valid TTA here: preprocessing maps every knee onto a\n    # shared lateral -> medial axis, so a flip swaps \"Medial OA\" with \"Lateral OA\".\n    TTA_HFLIP = False\n\n\nassert CFG.STAGE in CFG.VALID_STAGES, (CFG.STAGE, CFG.VALID_STAGES)\nCFG.RUN_EDA = CFG.STAGE == 'eda'\nCFG.RUN_TEACHER = CFG.STAGE == 'teacher'\nCFG.RUN_PREPROCESS = CFG.STAGE == 'preprocess'\nCFG.RUN_TRAIN = CFG.STAGE == 'train'\nCFG.RUN_INFER = CFG.STAGE == 'infer'\n\nTARGETS = ['ACL', 'MCL', 'Medial Meniscus', 'Lateral Meniscus',\n           'Medial OA', 'Lateral OA', 'PF OA', 'Effusion',\n           'Synovitis', \"Baker's\", 'Contusion', 'Fracture']\nWEIGHT_COLS = [f'w__{c}' for c in TARGETS]\nOBS_COLS = [f'obs__{c}' for c in TARGETS]\n\nfor d in [CFG.work_dir, CFG.teacher_dir, CFG.cache_dir, CFG.ckpt_dir]:\n    d.mkdir(parents=True, exist_ok=True)\n\n# ------------------------------------------------------------------ plotting theme\nICEFIRE = sns.color_palette('icefire', 12)\nICE, FIRE, MIDLINE = ICEFIRE[2], ICEFIRE[9], '#39414d'\nCMAP = 'icefire'\nCMAP_SEQ = sns.blend_palette(['#eef1f6', ICE, FIRE], as_cmap=True)\nsns.set_theme(style='white', palette=ICEFIRE, font_scale=0.95)\nmpl.rcParams.update({\n    'figure.dpi': 120, 'savefig.dpi': 120,\n    'axes.spines.top': False, 'axes.spines.right': False,\n    'axes.edgecolor': '#c8cdd4', 'axes.labelcolor': MIDLINE,\n    'axes.titlesize': 12, 'axes.titleweight': 'bold', 'axes.titlepad': 10,\n    'text.color': MIDLINE, 'xtick.color': MIDLINE, 'ytick.color': MIDLINE,\n    'grid.color': '#e8eaee', 'axes.grid': True, 'axes.grid.axis': 'y',\n    'legend.frameon': False, 'figure.facecolor': 'white',\n})\n\n\ndef grad(n, reverse=False):\n    cm = plt.get_cmap(CMAP)\n    cold = np.linspace(0.02, 0.28, n - n // 2)\n    warm = np.linspace(0.78, 0.99, n // 2)\n    cols = [cm(p) for p in np.concatenate([cold, warm])][:n]\n    return cols[::-1] if reverse else cols\n\n\ndef seed_everything(seed=42):\n    import random\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    try:\n        import torch\n        torch.manual_seed(seed)\n        torch.cuda.manual_seed_all(seed)\n    except ImportError:\n        pass\n\n\ndef find_artifact(name, primary, extras):\n    hits = [Path(primary) / name] + [Path(d) / name for d in extras]\n    hits = [p for p in hits if p.exists()]\n    if not hits:\n        raise FileNotFoundError(\n            f'{name} not found. Attach the preceding-stage dataset and list its directory in CFG.')\n    if len(hits) > 1:\n        print(f'[artifact] multiple copies of {name}; using {hits[0]}')\n    return hits[0]\n\n\nseed_everything(CFG.seed)\nif not (CFG.comp_dir / 'test.csv').exists():\n    candidates = [Path('/kaggle/input/rsna-knee-abnormality-detection'),\n                  Path('/kaggle/input/competitions/rsna-knee-abnormality-detection')]\n    candidates += sorted(Path('/kaggle/input').glob('*knee*'))\n    candidates += sorted(Path('/kaggle/input/competitions').glob('*knee*'))\n    for _c in candidates:\n        if (_c / 'test.csv').exists():\n            CFG.comp_dir = _c\n            break\nprint('stage:', CFG.STAGE, '| comp_dir:', CFG.comp_dir, '| exists:', CFG.comp_dir.exists())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:17.077259Z","iopub.execute_input":"2026-08-05T19:16:17.077485Z","iopub.status.idle":"2026-08-05T19:16:23.639072Z","shell.execute_reply.started":"2026-08-05T19:16:17.077463Z","shell.execute_reply":"2026-08-05T19:16:23.638366Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Environment and decoder preflight\n\nJPEG Lossless and JPEG 2000 both occur in this data, and the Kaggle base image does not\nalways ship a decoder for them. Rather than guessing from importable module names, this\ncell samples real DICOMs, groups them by transfer syntax, and **attempts an actual pixel\ndecode of each**, because Pillow covers JPEG 2000 in most builds but never JPEG Lossless\nP14.\n\nIt runs before the submission safety net, so it refuses to raise during inference: a\nmissing decoder costs score, but crashing here costs the entire submission. Only the\npreprocess stage, which produces no submission, fails hard.\n","metadata":{}},{"cell_type":"code","source":"import subprocess\nimport pydicom\n\nprint('pydicom:', pydicom.__version__)\n\n# ---------------------------------------------------------------- 1. offline wheels\n# Internet is disabled at submission time, so a missing decoder has to arrive as an\n# attached Kaggle Dataset of .whl files. Put those directories in CFG.wheel_dirs and\n# they are installed here with --no-index (no network).\nfor _d in [Path(x) for x in getattr(CFG, 'wheel_dirs', [])]:\n    _whl = sorted(_d.glob('*.whl'))\n    if not _whl:\n        print(f'[warn] no .whl files under {_d}')\n        continue\n    _r = subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '--no-index',\n                         f'--find-links={_d}', *[str(w) for w in _whl]],\n                        capture_output=True, text=True)\n    print(f'[wheels] {_d}: {len(_whl)} wheels -> exit {_r.returncode}')\n    if _r.returncode:\n        print(_r.stderr[-1000:])\n\n# ------------------------------------------------------- 2. which plugins are present\ndecoders = {}\nfor mod in ['pylibjpeg', 'openjpeg', 'libjpeg', 'gdcm', 'PIL']:\n    try:\n        __import__(mod)\n        decoders[mod] = 'ok'\n    except Exception as e:\n        decoders[mod] = f'MISSING ({type(e).__name__})'\nfor k, v in decoders.items():\n    print(f'  {k:<10}: {v}')\n\n# --------------------------------------------- 3. ground truth: actually decode a file\n# Module names are a poor proxy. Pillow, for instance, covers JPEG 2000 in most builds\n# but never JPEG Lossless P14 - only a real decode tells you which syntaxes you can read.\nSYNTAX_NAMES = {\n    '1.2.840.10008.1.2': 'Implicit VR Little Endian',\n    '1.2.840.10008.1.2.1': 'Explicit VR Little Endian',\n    '1.2.840.10008.1.2.1.99': 'Deflated Explicit VR LE',\n    '1.2.840.10008.1.2.2': 'Explicit VR Big Endian',\n    '1.2.840.10008.1.2.4.50': 'JPEG Baseline',\n    '1.2.840.10008.1.2.4.51': 'JPEG Extended',\n    '1.2.840.10008.1.2.4.57': 'JPEG Lossless P14',\n    '1.2.840.10008.1.2.4.70': 'JPEG Lossless P14 SV1',\n    '1.2.840.10008.1.2.4.80': 'JPEG-LS Lossless',\n    '1.2.840.10008.1.2.4.81': 'JPEG-LS Near-lossless',\n    '1.2.840.10008.1.2.4.90': 'JPEG 2000 Lossless',\n    '1.2.840.10008.1.2.4.91': 'JPEG 2000',\n}\n\n\ndef probe_decoders(roots, max_series=48, per_series=1):\n    \"\"\"Sample DICOMs, group them by transfer syntax, and try a real pixel decode.\"\"\"\n    files = []\n    for root in roots:\n        root = Path(root)\n        if not root.exists():\n            continue\n        studies = list(itertools.islice((p for p in root.iterdir() if p.is_dir()), 400))\n        series = [s for st in studies for s in itertools.islice(\n            (q for q in st.iterdir() if q.is_dir()), 3)]\n        if not series:\n            continue\n        step = max(1, len(series) // max_series)\n        for s in series[::step][:max_series]:\n            files.extend(sorted(s.glob('*.dcm'))[:per_series])\n    by_syntax = {}\n    for f in files:\n        try:\n            ds = pydicom.dcmread(str(f), stop_before_pixels=True)\n            uid = str(ds.file_meta.TransferSyntaxUID)\n        except Exception:\n            continue\n        by_syntax.setdefault(uid, []).append(f)\n    out = {}\n    for uid, fs in by_syntax.items():\n        ok, why = False, ''\n        for f in fs[:3]:\n            try:\n                _ = pydicom.dcmread(str(f)).pixel_array\n                ok = True\n                break\n            except Exception as e:\n                why = f'{type(e).__name__}: {str(e)[:110]}'\n        out[uid] = {'name': SYNTAX_NAMES.get(uid, uid), 'n_sampled': len(fs),\n                    'decodes': ok, 'error': '' if ok else why}\n    return out\n\n\nDECODE_PROBE, compressed_decoder_ok = {}, True\n_roots = [CFG.comp_dir / 'test_series', CFG.comp_dir / 'train_series']\nif any(Path(r).exists() for r in _roots):\n    DECODE_PROBE = probe_decoders(_roots)\n    if DECODE_PROBE:\n        _tbl = pd.DataFrame(DECODE_PROBE).T.sort_values('decodes')\n        print('\\ntransfer syntaxes actually present, and whether they decode here:')\n        print(_tbl[['name', 'n_sampled', 'decodes', 'error']].to_string())\n        _broken = [v['name'] for v in DECODE_PROBE.values() if not v['decodes']]\n        compressed_decoder_ok = not _broken\n    else:\n        print('\\n[warn] found no readable DICOM to probe; decoder status unknown')\nelse:\n    print('\\n[warn] no train_series/test_series mount; skipping the decode probe')\n\n# --------------------------------------------------------------- 4. act on the result\n# This cell runs BEFORE the submission safety net, so raising here means Kaggle gets no\n# submission.csv at all. Only the preprocess stage - which produces no submission - is\n# allowed to fail hard. Inference warns and lets the per-study fallback absorb the loss.\nif not compressed_decoder_ok:\n    _msg = (f'Cannot decode: {_broken}. Install python-gdcm and/or pylibjpeg + '\n            'pylibjpeg-libjpeg + pylibjpeg-openjpeg. With internet on, pip install '\n            'them directly; for an offline submission, pip download the wheels, '\n            'publish them as a Kaggle Dataset, attach it, and set CFG.wheel_dirs.')\n    if CFG.RUN_PREPROCESS:\n        raise RuntimeError(_msg)\n    if CFG.RUN_INFER:\n        print(f'\\n[SEVERE] {_msg}\\n[SEVERE] Studies using those syntaxes will fall back '\n              'to the metadata prediction, which will cost real score. Fix this before '\n              'your final submission.')\n\ntry:\n    import torch\n    gpu = torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'cpu only'\n    print(f'torch: {torch.__version__} | CUDA: {torch.cuda.is_available()} | {gpu}')\nexcept ImportError:\n    print('torch unavailable (acceptable only for EDA/teacher/preprocess)')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:23.640405Z","iopub.execute_input":"2026-08-05T19:16:23.640882Z","iopub.status.idle":"2026-08-05T19:16:27.634521Z","shell.execute_reply.started":"2026-08-05T19:16:23.640857Z","shell.execute_reply":"2026-08-05T19:16:27.633633Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"required = ['train.csv', 'train_series.csv', 'test.csv', 'test_series.csv', 'sample_submission.csv']\nmissing = [x for x in required if not (CFG.comp_dir / x).exists()]\nif missing:\n    raise FileNotFoundError(f'Competition mount is incomplete: {missing}')\n\ntrain = pd.read_csv(CFG.comp_dir / 'train.csv')\ntrain_series = pd.read_csv(CFG.comp_dir / 'train_series.csv')\ntest = pd.read_csv(CFG.comp_dir / 'test.csv')\ntest_series = pd.read_csv(CFG.comp_dir / 'test_series.csv')\nsample_sub = pd.read_csv(CFG.comp_dir / 'sample_submission.csv')\n\n# These were hard asserts. On the hidden re-run an assert means no submission.csv at\n# all, so anything that does not actually break inference is now only a warning.\nif list(sample_sub.columns) != ['StudyInstanceUID'] + TARGETS:\n    print('[warn] sample_submission column order differs from the documented one:',\n          list(sample_sub.columns))\nif 'Report' in test.columns:\n    print('[warn] test.csv contains Report; the image strategy still ignores it.')\nassert train.StudyInstanceUID.is_unique and test.StudyInstanceUID.is_unique\n\nobserved = train[TARGETS].notna()\nhas_any_label = observed.any(axis=1)\nhas_all_labels = observed.all(axis=1)\nhas_report = train['Report'].fillna('').str.strip().str.len() > 0\n_label_values = train[TARGETS].where(observed).stack().astype(float).values\nif len(_label_values) and not np.isin(_label_values, [0.0, 1.0]).all():\n    raise ValueError('Every observed target must be binary 0/1.')\n\nprint(f'train studies: {len(train):,} | train series: {len(train_series):,}')\nprint(f'test example studies: {len(test):,} | test series: {len(test_series):,}')\nprint(f'any observed label: {has_any_label.sum():,} | all 12: {has_all_labels.sum():,} '\n      f'| report present: {has_report.sum():,}')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:27.635541Z","iopub.execute_input":"2026-08-05T19:16:27.635882Z","iopub.status.idle":"2026-08-05T19:16:27.877116Z","shell.execute_reply.started":"2026-08-05T19:16:27.635848Z","shell.execute_reply":"2026-08-05T19:16:27.876401Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ============================== SUBMISSION SAFETY NET ==============================\n# Why the last run failed: a code competition re-runs the whole notebook and then looks\n# for /kaggle/working/submission.csv. If the writing cell is skipped (wrong STAGE) or\n# any earlier cell raises, the file does not exist and the submission is rejected no\n# matter how good the model is.\n#\n# Rule from here on: a VALID submission.csv exists on disk from this cell onwards, in\n# every stage. Later cells only ever overwrite it with something better.\n#\n# This cell deliberately re-reads its own inputs instead of trusting the loader cell\n# above. A safety net that needs an earlier cell to have succeeded is not a safety net,\n# and it must also work when you execute this cell on its own.\n\nif 'CFG' not in globals() or 'TARGETS' not in globals():\n    raise RuntimeError('Run the configuration cell (Part 0) first, or use Run All.')\n\n\ndef _resolve_comp_dir():\n    \"\"\"Find the competition mount even if the loader cell never ran.\"\"\"\n    seen, roots = set(), []\n    for c in [getattr(CFG, 'comp_dir', None),\n              Path('/kaggle/input/rsna-knee-abnormality-detection'),\n              Path('/kaggle/input/competitions/rsna-knee-abnormality-detection'),\n              *sorted(Path('/kaggle/input').glob('*knee*')),\n              *sorted(Path('/kaggle/input/competitions').glob('*knee*'))]:\n        if c is not None and str(c) not in seen:\n            seen.add(str(c))\n            roots.append(Path(c))\n    for r in roots:\n        if (r / 'test.csv').exists():\n            return r\n    raise FileNotFoundError(\n        'test.csv not found under any candidate mount. Attach the competition data '\n        f'(Add Input -> Competitions), then re-run. Searched: {[str(r) for r in roots]}')\n\n\n_root = _resolve_comp_dir()\nif _root != getattr(CFG, 'comp_dir', None):\n    print(f'[info] comp_dir corrected to {_root}')\n    CFG.comp_dir = _root\n\nif not isinstance(globals().get('test'), pd.DataFrame):\n    test = pd.read_csv(_root / 'test.csv')\nif not isinstance(globals().get('sample_sub'), pd.DataFrame):\n    _p = _root / 'sample_submission.csv'\n    sample_sub = pd.read_csv(_p) if _p.exists() else None\n\nSUB_PATH = (Path('/kaggle/working') if Path('/kaggle/working').exists() else Path('.')) / 'submission.csv'\nSUB_COLS = ['StudyInstanceUID'] + TARGETS\n\n# The row set the grader expects. sample_submission is authoritative when it agrees\n# with test.csv; otherwise test.csv wins, because that is what defines the studies.\nif (isinstance(sample_sub, pd.DataFrame)\n        and 'StudyInstanceUID' in sample_sub.columns\n        and len(sample_sub) == len(test)\n        and set(sample_sub.StudyInstanceUID) == set(test.StudyInstanceUID)):\n    SUB_INDEX = sample_sub[['StudyInstanceUID']].copy()\nelse:\n    if sample_sub is not None:\n        print('[warn] sample_submission and test.csv disagree; using test.csv as the row set')\n    SUB_INDEX = test[['StudyInstanceUID']].copy()\n\n\ndef write_submission(pred_df, note=''):\n    \"\"\"Align any prediction frame to the official rows/columns, then write the file.\n\n    Missing or non-finite cells fall back to 0.5 rather than propagating NaN, so a\n    partially successful model still produces a scorable file.\n    \"\"\"\n    out = SUB_INDEX.merge(pred_df, on='StudyInstanceUID', how='left')\n    for c in TARGETS:\n        if c not in out.columns:\n            out[c] = 0.5\n    out[TARGETS] = (out[TARGETS].apply(pd.to_numeric, errors='coerce')\n                                .replace([np.inf, -np.inf], np.nan)\n                                .fillna(0.5).clip(0.0, 1.0))\n    out = out[SUB_COLS]\n    assert len(out) == len(SUB_INDEX), (len(out), len(SUB_INDEX))\n    assert not out[TARGETS].isna().any().any()\n    out.to_csv(SUB_PATH, index=False)\n    print(f'submission.csv written {out.shape} -> {SUB_PATH} {note}')\n    return out\n\n\n_baseline = SUB_INDEX.copy()\nfor _c in TARGETS:\n    _baseline[_c] = 0.5\nsubmission = write_submission(_baseline, '(constant 0.5 safety net)')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:27.878018Z","iopub.execute_input":"2026-08-05T19:16:27.878738Z","iopub.status.idle":"2026-08-05T19:16:27.912491Z","shell.execute_reply.started":"2026-08-05T19:16:27.878715Z","shell.execute_reply":"2026-08-05T19:16:27.911598Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Part 1 — EDA\n\nNot decoration: every plot below answers a question that changes the code further down.\n\n## 1.1 How much of the data is actually labelled","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    per_target = pd.DataFrame(index=TARGETS)\n    per_target['n_observed'] = observed.sum()\n    per_target['missing_pct'] = (1 - observed.mean()) * 100\n    per_target['n_pos'] = train[TARGETS].fillna(0).sum().astype(int)\n    per_target['n_neg'] = per_target.n_observed - per_target.n_pos\n    per_target['prevalence_observed'] = per_target.n_pos / per_target.n_observed.clip(lower=1)\n    per_target['expected_pos_per_fold'] = per_target.n_pos / CFG.folds\n    per_target['expected_neg_per_fold'] = per_target.n_neg / CFG.folds\n    display(per_target.style.background_gradient(cmap=CMAP_SEQ,\n                                                  subset=['prevalence_observed'])\n                            .format({'missing_pct': '{:.1f}',\n                                     'prevalence_observed': '{:.3f}',\n                                     'expected_pos_per_fold': '{:.1f}',\n                                     'expected_neg_per_fold': '{:.1f}'}))\n\n    known_per_study = observed.sum(axis=1)\n    print('\\nknown labels per study:')\n    print(known_per_study.value_counts().sort_index().to_string())\n    print(f'neither report nor any label: {(~has_report & ~has_any_label).sum():,}')\n\n    # Compatibility aliases used only by the original EDA plots below.\n    has_label = has_all_labels\n    lab = train[has_all_labels].copy()\n    prev = pd.Series({c: train.loc[observed[c], c].mean() for c in TARGETS}).sort_values(ascending=False)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:27.913533Z","iopub.execute_input":"2026-08-05T19:16:27.913894Z","iopub.status.idle":"2026-08-05T19:16:27.924617Z","shell.execute_reply.started":"2026-08-05T19:16:27.913872Z","shell.execute_reply":"2026-08-05T19:16:27.923955Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    fig, axes = plt.subplots(1, 2, figsize=(15, 4.6))\n\n    ax = axes[0]\n    ax.barh(prev.index[::-1], prev.values[::-1] * 100, color=grad(12), edgecolor='white', height=.75)\n    ax.set_xlabel('positive rate, %'); ax.set_title('Prevalence of each target (hard labels)')\n    ax.grid(axis='x'); ax.grid(axis='y', visible=False)\n    for i, v in enumerate(prev.values[::-1]):\n        ax.text(v * 100 + .35, i, f'{v*100:.1f}', va='center', fontsize=8, color=MIDLINE)\n\n    ax = axes[1]\n    npl = lab[TARGETS].sum(axis=1)\n    vals, cnts = np.unique(npl, return_counts=True)\n    ax.bar(vals, cnts, color=grad(len(vals)), edgecolor='white')\n    ax.set_xlabel('abnormalities per study'); ax.set_ylabel('studies')\n    ax.set_title(f'Multi-label density (mean = {npl.mean():.2f})')\n    plt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:27.926349Z","iopub.execute_input":"2026-08-05T19:16:27.927164Z","iopub.status.idle":"2026-08-05T19:16:27.940977Z","shell.execute_reply.started":"2026-08-05T19:16:27.927113Z","shell.execute_reply":"2026-08-05T19:16:27.940342Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.2 Co-occurrence — use lift, not conditional probability\n\n`P(col | row)` mostly reflects the prevalence of the column, so it looks dramatic without saying much.\nLift, `P(A,B) / P(A)P(B)`, isolates the actual association. Pairs well above 1 (typically\n`Medial OA` ↔ `Lateral OA`, `ACL` ↔ `Contusion`) will be learnt jointly by a shared trunk; isolated\ntargets such as `Fracture` need their own attention.","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    P = lab[TARGETS].mean().values\n    J = (lab[TARGETS].T.values @ lab[TARGETS].values) / len(lab)\n    lift = pd.DataFrame(J / (P[:, None] * P[None, :] + 1e-9), index=TARGETS, columns=TARGETS)\n\n    fig, ax = plt.subplots(figsize=(10, 8))\n    sns.heatmap(lift, annot=True, fmt='.1f', cmap=CMAP, center=1.0, vmin=0, vmax=3,\n                mask=np.eye(12, dtype=bool), linewidths=.6, linecolor='white', ax=ax,\n                annot_kws={'size': 8}, cbar_kws={'label': 'lift = P(A,B) / P(A)P(B)', 'shrink': .8})\n    ax.set_title('Co-occurrence lift — above 1 means the pair appears together more often than chance')\n    plt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:16:27.941726Z","iopub.execute_input":"2026-08-05T19:16:27.942009Z","iopub.status.idle":"2026-08-05T19:16:27.956809Z","shell.execute_reply.started":"2026-08-05T19:16:27.941976Z","shell.execute_reply":"2026-08-05T19:16:27.956193Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.3 Series inventory — which views a study actually has\n\nThe question that decides the architecture: **for how many studies does each\n(plane × fluid-sensitive × fat-sat) combination exist?** If axial is present for only 60% of studies,\nthe model must handle a missing view through masking rather than crash or read zeros as signal.","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    def view_key(r):\n        plane = str(r['Anatomical_Plane'])[:3].upper()\n        return f\"{plane}_{'FS' if r['Fluid_Sensitive'] == 1 else 'T1'}{'_FAT' if r['Fat_Suppression'] == 1 else ''}\"\n\n    ts = train_series.copy()\n    ts['view'] = ts.apply(view_key, axis=1)\n\n    combo = (ts.groupby(['Anatomical_Plane', 'Fluid_Sensitive', 'Fat_Suppression'])\n               .size().reset_index(name='n_series').sort_values('n_series', ascending=False))\n    combo['pct'] = (combo.n_series / len(ts) * 100).round(1)\n    print('Series type combinations:'); display(combo)\n\n    cov = ts.assign(one=1).pivot_table(index='StudyInstanceUID', columns='view',\n                                       values='one', aggfunc='sum', fill_value=0)\n    coverage = pd.DataFrame({\n        'studies_with_view': (cov > 0).sum(),\n        'pct_studies': ((cov > 0).mean() * 100).round(1),\n        'mean_series_when_present': cov.replace(0, np.nan).mean().round(2),\n    }).sort_values('pct_studies', ascending=False)\n\n    fig, axes = plt.subplots(1, 2, figsize=(15, 4.4))\n    ax = axes[0]\n    ax.barh(coverage.index[::-1], coverage.pct_studies.values[::-1],\n            color=grad(len(coverage)), edgecolor='white', height=.7)\n    ax.axvline(80, color=FIRE, ls='--', lw=1.2, label='80% — below this, make the view optional')\n    ax.set_xlabel('% of studies containing the view'); ax.set_title('View coverage per study')\n    ax.grid(axis='x'); ax.grid(axis='y', visible=False); ax.legend(fontsize=8)\n\n    ax = axes[1]\n    plane_cov = (ts.pivot_table(index='StudyInstanceUID', columns='Anatomical_Plane',\n                                values='Fluid_Sensitive', aggfunc='size', fill_value=0) > 0)\n    pc = (plane_cov.mean() * 100).sort_values(ascending=False)\n    ax.bar(pc.index, pc.values, color=grad(len(pc)), edgecolor='white', width=.55)\n    ax.set_ylim(0, 108); ax.set_ylabel('% of studies'); ax.set_title('Coverage by anatomical plane')\n    for i, v in enumerate(pc.values):\n        ax.text(i, v + 1.8, f'{v:.1f}', ha='center', fontsize=9, color=MIDLINE)\n    plt.tight_layout(); plt.show()\n\n    display(coverage)\n    print(f'studies with all three planes           : {plane_cov.all(axis=1).mean()*100:.1f}%')\n    print(f\"studies with any fluid-sensitive series : \"\n          f\"{ts.groupby('StudyInstanceUID')['Fluid_Sensitive'].max().mean()*100:.1f}%\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:01.323214Z","iopub.execute_input":"2026-08-05T19:18:01.324069Z","iopub.status.idle":"2026-08-05T19:18:01.339241Z","shell.execute_reply.started":"2026-08-05T19:18:01.324031Z","shell.execute_reply":"2026-08-05T19:18:01.338448Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.4 DICOM audit — tags, transfer syntax, laterality\n\nThe organisers kept 86 tags. Which ones matters a great deal: can we read the knee side\n(`Laterality` / `ImageLaterality`), can we sort slices geometrically (`ImagePositionPatient` +\n`ImageOrientationPatient`), is `PatientID` present (fold leakage if one patient has several studies)?","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    def sample_dcm_paths(root, n_series=60, per_series=2, seed=0):\n        rng = np.random.default_rng(seed)\n        series_dirs = [p for p in root.glob('*/*') if p.is_dir()]\n        if not series_dirs:\n            return []\n        pick = rng.choice(len(series_dirs), size=min(n_series, len(series_dirs)), replace=False)\n        return [f for i in pick for f in sorted(series_dirs[i].glob('*.dcm'))[:per_series]]\n\n\n    paths = sample_dcm_paths(CFG.comp_dir / 'train_series')\n    print('scanning files:', len(paths))\n\n    tag_counter, rows = Counter(), []\n    for p in paths:\n        try:\n            ds = pydicom.dcmread(str(p), stop_before_pixels=True)\n        except Exception as e:\n            print('read failed:', p.name, e); continue\n        for el in ds:\n            tag_counter[el.keyword or str(el.tag)] += 1\n        rows.append({\n            'Rows': getattr(ds, 'Rows', None), 'Columns': getattr(ds, 'Columns', None),\n            'PixelSpacing': str(getattr(ds, 'PixelSpacing', None)),\n            'SliceThickness': getattr(ds, 'SliceThickness', None),\n            'Photometric': getattr(ds, 'PhotometricInterpretation', None),\n            'Laterality': getattr(ds, 'Laterality', None) or getattr(ds, 'ImageLaterality', None),\n            'SeriesDescription': getattr(ds, 'SeriesDescription', None),\n            'FieldStrength': getattr(ds, 'MagneticFieldStrength', None),\n            'Manufacturer': getattr(ds, 'Manufacturer', None),\n            'TransferSyntax': str(ds.file_meta.TransferSyntaxUID) if hasattr(ds, 'file_meta') else None,\n            'has_IPP': hasattr(ds, 'ImagePositionPatient'),\n            'has_IOP': hasattr(ds, 'ImageOrientationPatient'),\n            'has_PatientID': hasattr(ds, 'PatientID'),\n        })\n\n    meta = pd.DataFrame(rows)\n    print('\\n=== available tags, top 40 by frequency ===')\n    print(pd.Series(tag_counter).sort_values(ascending=False).head(40).to_string())\n    for c in ['Laterality', 'Photometric', 'TransferSyntax', 'Manufacturer',\n              'has_IPP', 'has_IOP', 'has_PatientID']:\n        print(f'\\n--- {c} ---')\n        print(meta[c].value_counts(dropna=False).head(8).to_string())\n    print('\\n--- geometry ---')\n    print(meta[['Rows', 'Columns', 'SliceThickness']].describe().round(2).to_string())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:02.617098Z","iopub.execute_input":"2026-08-05T19:18:02.61751Z","iopub.status.idle":"2026-08-05T19:18:02.628093Z","shell.execute_reply.started":"2026-08-05T19:18:02.617482Z","shell.execute_reply":"2026-08-05T19:18:02.6272Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    fig, axes = plt.subplots(1, 3, figsize=(15, 3.8))\n    for ax, col, title in zip(axes, ['Laterality', 'Photometric', 'Manufacturer'],\n                              ['Knee side', 'Photometric interpretation', 'Scanner vendor']):\n        vc = meta[col].fillna('missing').value_counts().head(6)\n        ax.bar(vc.index.astype(str), vc.values, color=grad(max(len(vc), 3)), edgecolor='white', width=.6)\n        ax.set_title(title); ax.tick_params(axis='x', rotation=20, labelsize=8)\n    plt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:02.806708Z","iopub.execute_input":"2026-08-05T19:18:02.807245Z","iopub.status.idle":"2026-08-05T19:18:02.812259Z","shell.execute_reply.started":"2026-08-05T19:18:02.80722Z","shell.execute_reply":"2026-08-05T19:18:02.811514Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Audit checklist**\n\n- Count observed positives *and negatives per target*; a row need not have all twelve labels.\n- Verify pixel decoding, not only header reading, for every observed transfer syntax.\n- Use `PatientID` when retained and exact normalized report duplicates as fold groups.\n- Treat `Fluid_Sensitive=0` as “not marked fluid-sensitive,” not automatically as T1.\n- Compare CSV series identifiers to directories and count physical/unique slices before selecting a series.\n- Inspect canonical coronal/axial/sagittal montages on both left and right knees before a long cache run.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    if meta['has_PatientID'].any():\n        print('PatientID exists in the headers -> collect it during preprocessing and group folds by it.')\n    else:\n        print('PatientID is not among the retained tags.')\n\n    rep = train['Report'].fillna('').str.strip()\n    long_rep = rep[rep.str.len() > 30]\n    dup = long_rep.duplicated(keep=False)\n    print(f'\\nexactly duplicated non-empty reports: {dup.sum():,} rows '\n          f'({long_rep[dup].nunique():,} unique texts)')\n    if dup.sum():\n        print('-> those studies must share a fold; handled in Part 4 via a report hash group.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:03.171089Z","iopub.execute_input":"2026-08-05T19:18:03.171912Z","iopub.status.idle":"2026-08-05T19:18:03.178488Z","shell.execute_reply.started":"2026-08-05T19:18:03.171872Z","shell.execute_reply":"2026-08-05T19:18:03.177884Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.6 Reports — length, language, keyword signal\n\nHand-written stop-word heuristics make a poor language detector: `la ` occurs in French, Spanish and\nItalian alike, and an `if/elif` chain bakes in an ordering bias. The estimate below is at least\nsymmetric, and it is only used for orientation — the teacher in Part 3 relies on **character n-grams,\nwhich are language-agnostic by construction**.","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    rep_len = train['Report'].fillna('').str.split().str.len()\n    print(rep_len.describe().round(1).to_string())\n\n    LANG_HINTS = {\n        'en': [' the ', ' and ', ' with ', ' there is ', ' no evidence '],\n        'nl': [' de ', ' het ', ' en ', ' geen ', ' voorste ', ' kruisband '],\n        'de': [' der ', ' die ', ' das ', ' und ', ' kein ', ' kreuzband '],\n        'fr': [' le ', ' la ', ' des ', ' avec ', ' pas de ', ' croise '],\n        'es': [' el ', ' los ', ' con ', ' sin ', ' rodilla ', ' menisco '],\n        'pt': [' do ', ' da ', ' com ', ' sem ', ' joelho ', ' menisco '],\n        'it': [' il ', ' della ', ' con ', ' senza ', ' ginocchio ', ' menisco '],\n        'tr': [' ve ', ' ile ', ' yok', ' diz ', ' menisk'],\n    }\n\n    def guess_lang(t):\n        if not isinstance(t, str) or len(t.strip()) < 15:\n            return 'empty'\n        s = ' ' + t.lower() + ' '\n        scores = {k: sum(s.count(w) for w in v) for k, v in LANG_HINTS.items()}\n        best, val = max(scores.items(), key=lambda kv: kv[1])\n        return best if val > 0 else 'other'\n\n    train['lang_guess'] = train['Report'].apply(guess_lang)\n    lc = train['lang_guess'].value_counts()\n\n    fig, axes = plt.subplots(1, 2, figsize=(15, 4))\n    ax = axes[0]\n    ax.hist(rep_len.clip(0, rep_len.quantile(.99)), bins=50, color=ICE, edgecolor='white')\n    ax.axvline(rep_len.median(), color=FIRE, ls='--', lw=1.4,\n               label=f'median = {rep_len.median():.0f} words')\n    ax.set_xlabel('report length, words'); ax.set_ylabel('studies')\n    ax.set_title('Report length'); ax.legend(fontsize=8)\n\n    ax = axes[1]\n    ax.bar(lc.index, lc.values, color=grad(max(len(lc), 3)), edgecolor='white', width=.6)\n    ax.set_title('Estimated report language (rough)'); ax.tick_params(axis='x', rotation=30)\n    plt.tight_layout(); plt.show()\n\n    print('\\n=== sample report ===')\n    if (has_label & has_report).any():\n        print(train.loc[has_label & has_report, 'Report'].iloc[0][:800])\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:03.507001Z","iopub.execute_input":"2026-08-05T19:18:03.507569Z","iopub.status.idle":"2026-08-05T19:18:03.523498Z","shell.execute_reply.started":"2026-08-05T19:18:03.507529Z","shell.execute_reply":"2026-08-05T19:18:03.522653Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1.7 Raw DICOM visual/decode audit\n\nThis preview is intentionally raw: it sorts slices geometrically and tests pixel decoding, but the\ncanonical laterality/physical-space transform is previewed in the preprocessing smoke stage. Limit the\nfigure to six series so a study with many acquisitions cannot create an enormous notebook output.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_EDA:\n    def read_pixels(path):\n        ds = pydicom.dcmread(str(path))\n        img = ds.pixel_array.astype(np.float32)\n        if img.ndim == 3:\n            img = img[..., 0] if img.shape[-1] <= 4 else img[0]\n        img = img * float(getattr(ds, 'RescaleSlope', 1) or 1) + \\\n              float(getattr(ds, 'RescaleIntercept', 0) or 0)\n        if str(getattr(ds, 'PhotometricInterpretation', '')) == 'MONOCHROME1':\n            img = img.max() - img\n        lo, hi = np.percentile(img, [0.5, 99.5])\n        return np.clip((img - lo) / (hi - lo + 1e-6), 0, 1), ds\n\n    demo_study = train_series['StudyInstanceUID'].iloc[0]\n    sers = train_series[train_series.StudyInstanceUID == demo_study].head(6)\n    fig, axes = plt.subplots(len(sers), 5, figsize=(15, 3 * len(sers)), squeeze=False)\n    for row_i, (_, s) in enumerate(sers.iterrows()):\n        folder = CFG.comp_dir / 'train_series' / str(demo_study) / str(s.SeriesInstanceUID)\n        headers = []\n        for p in folder.glob('*.dcm'):\n            try:\n                ds = pydicom.dcmread(str(p), stop_before_pixels=True)\n                iop = np.asarray(ds.ImageOrientationPatient, float)\n                normal = np.cross(iop[:3], iop[3:])\n                position = float(np.dot(np.asarray(ds.ImagePositionPatient, float), normal))\n            except Exception:\n                try:\n                    ds = pydicom.dcmread(str(p), stop_before_pixels=True)\n                    position = float(getattr(ds, 'InstanceNumber', 0) or 0)\n                except Exception:\n                    continue\n            headers.append((position, str(p), p))\n        headers.sort(key=lambda x: (x[0], x[1]))\n        pick = np.linspace(0, len(headers) - 1, 5).round().astype(int) if headers else []\n        for col_i, j in enumerate(pick):\n            try:\n                img, ds = read_pixels(headers[j][2])\n                axes[row_i, col_i].imshow(img, cmap='gray')\n                if col_i == 0:\n                    side = getattr(ds, 'Laterality', None) or getattr(ds, 'ImageLaterality', '?')\n                    axes[row_i, col_i].set_title(\n                        f'{s.Anatomical_Plane} | fluid={s.Fluid_Sensitive} '\n                        f'| fat={s.Fat_Suppression} | side={side}',\n                        fontsize=8, loc='left', color=FIRE)\n            except Exception as e:\n                axes[row_i, col_i].text(.5, .5, str(e)[:40], ha='center', fontsize=6)\n            axes[row_i, col_i].axis('off'); axes[row_i, col_i].grid(False)\n    plt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:03.823241Z","iopub.execute_input":"2026-08-05T19:18:03.823904Z","iopub.status.idle":"2026-08-05T19:18:03.84021Z","shell.execute_reply.started":"2026-08-05T19:18:03.823868Z","shell.execute_reply":"2026-08-05T19:18:03.839421Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Part 2 — study/series data model\n\nCV2 keeps complementary contrasts instead of forcing one series per plane:\n\n| Slot | Selection | Main signal |\n|---|---|---|\n| `SAG_FS` | sagittal, fat-suppressed; fluid-sensitive preferred | ACL, contusion, effusion |\n| `SAG_NFS` | sagittal, non-fat-suppressed; fluid-sensitive preferred | menisci, cartilage, fracture |\n| `COR_FS` | coronal, fat-suppressed; fluid-sensitive preferred | MCL, contusion, OA |\n| `COR_NFS` | coronal, non-fat-suppressed; fluid-sensitive preferred | menisci, OA, fracture |\n| `AX` | axial; fluid-sensitive and fat suppression preferred | PF OA, synovitis, effusion, Baker's |\n\nSelections are unique, use actual `.dcm` counts, and reject obvious single-slice/localizer series.\nThe learned target-aware pooling can ignore an unavailable or unhelpful slot using masks.\n","metadata":{}},{"cell_type":"code","source":"VIEW_SPEC = {\n    'SAG_FS':  dict(plane='Sagittal', fat=1),\n    'SAG_NFS': dict(plane='Sagittal', fat=0),\n    'COR_FS':  dict(plane='Coronal',  fat=1),\n    'COR_NFS': dict(plane='Coronal',  fat=0),\n    'AX':      dict(plane='Axial',    fat=None),\n}\n\n\ndef _series_file_count(root, study, series):\n    return sum(1 for _ in (Path(root) / str(study) / str(series)).glob('*.dcm'))\n\n\ndef assign_views(series_df: pd.DataFrame) -> dict:\n    \"\"\"Choose distinct, deterministic series for the five sequence slots.\"\"\"\n    g = series_df.copy()\n    for c in ['Fluid_Sensitive', 'Fat_Suppression', 'n_files']:\n        if c not in g:\n            g[c] = 0\n    g['Fluid_Sensitive'] = pd.to_numeric(g.Fluid_Sensitive, errors='coerce').fillna(0).astype(int)\n    g['Fat_Suppression'] = pd.to_numeric(g.Fat_Suppression, errors='coerce').fillna(0).astype(int)\n    g['n_files'] = pd.to_numeric(g.n_files, errors='coerce').fillna(0).astype(int)\n    used, out = set(), {}\n\n    for slot, spec in VIEW_SPEC.items():\n        cand = g[(g.Anatomical_Plane == spec['plane']) & (~g.SeriesInstanceUID.isin(used))].copy()\n        if spec['fat'] is not None:\n            cand = cand[cand.Fat_Suppression == spec['fat']]\n        cand = cand[cand.n_files >= 4]  # excludes most scouts/localizers\n        if cand.empty:\n            out[slot] = None\n            continue\n        cand['fluid_rank'] = (cand.Fluid_Sensitive != 1).astype(int)\n        cand = cand.sort_values(['fluid_rank', 'n_files', 'SeriesInstanceUID'],\n                                ascending=[True, False, True], kind='stable')\n        sid = str(cand.iloc[0].SeriesInstanceUID)\n        out[slot] = sid\n        used.add(sid)\n    return out\n\n\ndef build_manifest(series_df: pd.DataFrame, root: Path, count_files=True) -> pd.DataFrame:\n    df = series_df.drop_duplicates(['StudyInstanceUID', 'SeriesInstanceUID']).copy()\n    if count_files:\n        df['n_files'] = [_series_file_count(root, r.StudyInstanceUID, r.SeriesInstanceUID)\n                         for r in df.itertuples()]\n    elif 'n_files' not in df:\n        raise ValueError('n_files is required for deterministic series selection')\n    rows = []\n    for study, g in df.groupby('StudyInstanceUID', sort=False):\n        rec = {'StudyInstanceUID': study, 'n_series': len(g),\n               'n_dicom_total': int(g.n_files.sum())}\n        rec.update(assign_views(g))\n        rows.append(rec)\n    return pd.DataFrame(rows)\n\n\n_demo = pd.DataFrame([\n    dict(SeriesInstanceUID='a', Anatomical_Plane='Sagittal', Fluid_Sensitive=1,\n         Fat_Suppression=1, n_files=30),\n    dict(SeriesInstanceUID='b', Anatomical_Plane='Sagittal', Fluid_Sensitive=1,\n         Fat_Suppression=0, n_files=28),\n    dict(SeriesInstanceUID='c', Anatomical_Plane='Coronal', Fluid_Sensitive=1,\n         Fat_Suppression=1, n_files=25),\n])\n_assigned = assign_views(_demo)\nassert _assigned['SAG_FS'] == 'a' and _assigned['SAG_NFS'] == 'b'\nassert len([x for x in _assigned.values() if x is not None]) == len(set(x for x in _assigned.values() if x))\n\nmanifest = None\nif CFG.RUN_EDA or CFG.RUN_PREPROCESS:\n    manifest = build_manifest(train_series, CFG.comp_dir / 'train_series', count_files=True)\n    fill = (manifest[CFG.VIEWS].notna().mean() * 100).round(1)\n    print('view fill rate (%):', fill.to_dict())\n    print('studies with no selected view:', int(manifest[CFG.VIEWS].isna().all(axis=1).sum()))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:04.147549Z","iopub.execute_input":"2026-08-05T19:18:04.148668Z","iopub.status.idle":"2026-08-05T19:18:04.184331Z","shell.execute_reply.started":"2026-08-05T19:18:04.148625Z","shell.execute_reply":"2026-08-05T19:18:04.183366Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2.1 Geometry-aware DICOM preprocessing\n\nThe loader now:\n\n1. deduplicates physical slice positions and orders depth in patient coordinates;\n2. uses IOP direction cosines to standardize the two in-plane axes;\n3. maps left and right knees onto the same lateral→medial coordinate;\n4. rescales pixel axes to physical spacing, crops the shared foreground ROI, then letterboxes\n   (never stretches) into a square;\n5. returns a slice-valid mask and records the preprocessing version in every cache item.\n\nOblique acquisitions are handled by selecting the array axis most aligned with the requested patient\ncoordinate. Always inspect a left/right montage: DICOM orientation bugs are high-impact and visual.\n","metadata":{}},{"cell_type":"code","source":"import cv2\ncv2.setNumThreads(0)\n\n\ndef _side(headers):\n    for ds in headers:\n        v = str(getattr(ds, 'Laterality', '') or getattr(ds, 'ImageLaterality', '')).upper()\n        if v in {'L', 'R'}:\n            return v\n    xs = []\n    for ds in headers:\n        try:\n            xs.append(float(ds.ImagePositionPatient[0]))\n        except Exception:\n            pass\n    if xs:\n        return 'R' if float(np.median(xs)) < 0 else 'L'\n    return 'R'  # unknown: preserve pixels instead of applying an arbitrary mirror\n\n\ndef _position(ds, plane, side):\n    \"\"\"Monotonic patient-coordinate position for deterministic depth order.\"\"\"\n    try:\n        ipp = np.asarray(ds.ImagePositionPatient, np.float64)\n        if plane == 'Sagittal':\n            return float(ipp[0] * (1 if side == 'R' else -1))  # lateral -> medial\n        if plane == 'Coronal':\n            return float(ipp[1])                               # anterior -> posterior\n        return float(ipp[2])                                   # inferior -> superior\n    except Exception:\n        return float(getattr(ds, 'InstanceNumber', 0) or 0)\n\n\ndef _orient_frame(a, ds, plane, side):\n    \"\"\"Map array H/W to a stable patient-coordinate display convention.\"\"\"\n    try:\n        iop = np.asarray(ds.ImageOrientationPatient, np.float64)\n        width_dir, height_dir = iop[:3].copy(), iop[3:].copy()\n    except Exception:\n        # Common radiological convention fallback.\n        if side == 'L' and plane in {'Coronal', 'Axial'}:\n            a = a[:, ::-1]\n        return np.ascontiguousarray(a), False\n\n    side_factor = 1.0 if side == 'R' else -1.0\n    if plane == 'Sagittal':\n        h_axis, h_sign, v_axis, v_sign = 1, +1, 2, -1  # posterior right; inferior down\n    elif plane == 'Coronal':\n        h_axis, h_sign, v_axis, v_sign = 0, side_factor, 2, -1  # lateral -> medial\n    else:\n        h_axis, h_sign, v_axis, v_sign = 0, side_factor, 1, +1  # lateral -> medial; posterior down\n\n    # If the desired horizontal patient axis is carried by array rows, transpose first.\n    transposed = abs(height_dir[h_axis]) > abs(width_dir[h_axis])\n    if transposed:\n        a = a.T\n        width_dir, height_dir = height_dir, width_dir\n    if width_dir[h_axis] * h_sign < 0:\n        a = a[:, ::-1]\n    if height_dir[v_axis] * v_sign < 0:\n        a = a[::-1, :]\n    return np.ascontiguousarray(a), transposed\n\n\ndef _resize_physical(a, ds, transposed=False):\n    ps = getattr(ds, 'PixelSpacing', None)\n    sy, sx = (float(ps[0]), float(ps[1])) if ps and len(ps) >= 2 else (1.0, 1.0)\n    if transposed:\n        sy, sx = sx, sy\n    base = max(min(sy, sx), 1e-6)\n    nh = max(1, int(round(a.shape[0] * sy / base)))\n    nw = max(1, int(round(a.shape[1] * sx / base)))\n    if (nh, nw) != a.shape:\n        interp = cv2.INTER_AREA if nh < a.shape[0] or nw < a.shape[1] else cv2.INTER_LINEAR\n        a = cv2.resize(a, (nw, nh), interpolation=interp)\n    return a\n\n\ndef _shared_foreground_bbox(vol, pad=0.08):\n    mean_img = np.mean(vol, axis=0)\n    positive = mean_img[mean_img > np.min(mean_img)]\n    if positive.size < 64:\n        return 0, vol.shape[1], 0, vol.shape[2]\n    threshold = np.percentile(positive, 12)\n    mask = (mean_img > threshold).astype(np.uint8)\n    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, np.ones((9, 9), np.uint8))\n    ys, xs = np.where(mask > 0)\n    if len(xs) < 64:\n        return 0, vol.shape[1], 0, vol.shape[2]\n    y0, y1, x0, x1 = ys.min(), ys.max() + 1, xs.min(), xs.max() + 1\n    py, px = int((y1 - y0) * pad), int((x1 - x0) * pad)\n    return max(0, y0 - py), min(vol.shape[1], y1 + py), \\\n           max(0, x0 - px), min(vol.shape[2], x1 + px)\n\n\ndef _letterbox(a, size):\n    scale = min(size / a.shape[0], size / a.shape[1])\n    nh, nw = max(1, int(round(a.shape[0] * scale))), max(1, int(round(a.shape[1] * scale)))\n    interp = cv2.INTER_AREA if scale < 1 else cv2.INTER_LINEAR\n    b = cv2.resize(a, (nw, nh), interpolation=interp)\n    out = np.zeros((size, size), dtype=b.dtype)\n    y0, x0 = (size - nh) // 2, (size - nw) // 2\n    out[y0:y0 + nh, x0:x0 + nw] = b\n    return out\n\n\ndef load_series_volume(series_dir: Path, plane: str, size=CFG.IMG_SIZE,\n                       n_slices=CFG.N_SLICES):\n    \"\"\"Return `(uint8[D,H,W], valid_slice_mask[D])`, or `(None, None)`.\"\"\"\n    files = list(Path(series_dir).glob('*.dcm'))\n    if not files:\n        return None, None\n\n    pairs = []\n    for f in files:\n        try:\n            ds = pydicom.dcmread(str(f), stop_before_pixels=True)\n            pairs.append((f, ds))\n        except Exception:\n            continue\n    if not pairs:\n        return None, None\n\n    # Reject embedded localizers/mixed geometry inside a nominal series.\n    def geometry_key(ds):\n        try:\n            iop = tuple(np.round(np.asarray(ds.ImageOrientationPatient, float), 3))\n        except Exception:\n            iop = None\n        return (int(getattr(ds, 'Rows', 0) or 0),\n                int(getattr(ds, 'Columns', 0) or 0), iop)\n    geom_counts = Counter(geometry_key(ds) for _, ds in pairs)\n    dominant = max(geom_counts, key=lambda k: (geom_counts[k], k[2] is not None,\n                                               k[0] * k[1], str(k)))\n    pairs = [(f, ds) for f, ds in pairs if geometry_key(ds) == dominant]\n\n    side = _side([x[1] for x in pairs])\n    keyed = sorted(((_position(ds, plane, side), str(f), f, ds) for f, ds in pairs),\n                   key=lambda x: (x[0], x[1]))\n    unique = []\n    for item in keyed:\n        if not unique or abs(item[0] - unique[-1][0]) > 1e-3:\n            unique.append(item)\n    if not unique:\n        return None, None\n\n    # Short series are spread across the fixed depth grid. Unfilled positions are\n    # nearest-filled for 2.5D context but remain masked out of attention/encoding.\n    n_take = min(n_slices, len(unique))\n    take = np.unique(np.linspace(0, len(unique) - 1, n_take).round().astype(int))\n    out_positions = np.linspace(0, n_slices - 1, len(take)).round().astype(int)\n    frames, valid = [None] * n_slices, np.zeros(n_slices, np.uint8)\n    for out_i, src_i in zip(out_positions, take):\n        _, _, f, _ = unique[src_i]\n        try:\n            ds = pydicom.dcmread(str(f))\n            a = ds.pixel_array.astype(np.float32)\n            if a.ndim == 3:\n                a = a[..., 0] if a.shape[-1] <= 4 else a[0]\n            a = a * float(getattr(ds, 'RescaleSlope', 1) or 1) + \\\n                float(getattr(ds, 'RescaleIntercept', 0) or 0)\n            if str(getattr(ds, 'PhotometricInterpretation', '')) == 'MONOCHROME1':\n                a = a.max() - a\n            a, transposed = _orient_frame(a, ds, plane, side)\n            a = _resize_physical(a, ds, transposed)\n            frames[out_i] = a\n            valid[out_i] = 1\n        except Exception:\n            frames[out_i] = None\n\n    good = [x for x in frames if x is not None]\n    if not good:\n        return None, None\n    # Orientation/spacing should make shapes identical; center-pad rare one-pixel discrepancies.\n    mh, mw = max(x.shape[0] for x in good), max(x.shape[1] for x in good)\n    aligned = []\n    valid_positions = np.flatnonzero(valid)\n    for out_i, a in enumerate(frames):\n        if a is None:\n            nearest = valid_positions[np.argmin(np.abs(valid_positions - out_i))]\n            a = frames[int(nearest)]\n        canvas = np.zeros((mh, mw), np.float32)\n        y0, x0 = (mh - a.shape[0]) // 2, (mw - a.shape[1]) // 2\n        canvas[y0:y0 + a.shape[0], x0:x0 + a.shape[1]] = a\n        aligned.append(canvas)\n    vol = np.stack(aligned)\n\n    vox = vol[valid.astype(bool)]\n    lo, hi = np.percentile(vox, [0.5, 99.5])\n    vol = np.clip((vol - lo) / (hi - lo + 1e-6), 0, 1)\n    y0, y1, x0, x1 = _shared_foreground_bbox(vol[valid.astype(bool)])\n    vol = np.stack([_letterbox(a[y0:y1, x0:x1], size) for a in vol])\n    vol = (vol * 255).round().astype(np.uint8)\n    return np.ascontiguousarray(vol), valid\n\n\ndef _valid_cached(path):\n    try:\n        with np.load(path, allow_pickle=False) as z:\n            vm, sm, vol = z['view_mask'], z['slice_mask'], z['vol']\n            return (str(z['version'].item()) == CFG.CACHE_VERSION and\n                    vol.shape == (CFG.N_VIEWS, CFG.N_SLICES, CFG.IMG_SIZE, CFG.IMG_SIZE) and\n                    vol.dtype == np.uint8 and vm.shape == (CFG.N_VIEWS,) and\n                    sm.shape == (CFG.N_VIEWS, CFG.N_SLICES) and\n                    np.array_equal(vm.astype(bool), sm.astype(bool).any(1)) and\n                    bool(vm.any()))\n    except Exception:\n        return False\n\n\ndef cache_study(study: str, mrow, root: Path, out_dir: Path):\n    out_dir.mkdir(parents=True, exist_ok=True)\n    dst = out_dir / f'{study}.npz'\n    if dst.exists() and _valid_cached(dst):\n        return True, 'cached'\n    vol = np.zeros((CFG.N_VIEWS, CFG.N_SLICES, CFG.IMG_SIZE, CFG.IMG_SIZE), np.uint8)\n    view_mask = np.zeros(CFG.N_VIEWS, np.uint8)\n    slice_mask = np.zeros((CFG.N_VIEWS, CFG.N_SLICES), np.uint8)\n    for vi, slot in enumerate(CFG.VIEWS):\n        sid = mrow.get(slot)\n        if not isinstance(sid, str) or not sid:\n            continue\n        v, sm = load_series_volume(root / str(study) / sid, VIEW_SPEC[slot]['plane'])\n        if v is not None:\n            vol[vi], slice_mask[vi], view_mask[vi] = v, sm, 1\n    if not view_mask.any():\n        return False, 'no_decodable_view'\n    tmp = dst.with_suffix('.tmp.npz')\n    np.savez_compressed(tmp, vol=vol, view_mask=view_mask, slice_mask=slice_mask,\n                        version=np.asarray(CFG.CACHE_VERSION))\n    os.replace(tmp, dst)\n    return True, 'written'\n\n\nprint('geometry-aware preprocessing helpers ready:', CFG.CACHE_VERSION)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:04.472522Z","iopub.execute_input":"2026-08-05T19:18:04.47323Z","iopub.status.idle":"2026-08-05T19:18:04.715431Z","shell.execute_reply.started":"2026-08-05T19:18:04.4732Z","shell.execute_reply":"2026-08-05T19:18:04.714537Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Preprocessing validation\n\nThe smoke montage only runs in the preprocessing stage. Inspect at least one left and one right knee\nin each plane. A cache version is embedded in each item, so changing orientation/geometry cannot\nsilently reuse stale arrays.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_PREPROCESS and CFG.PREPROCESS_SMOKE_ONLY:\n    assert manifest is not None and len(manifest)\n    _row = manifest.iloc[0]\n    _out = CFG.cache_dir / f'shard{CFG.SHARD_ID:02d}'\n    _t0 = time.time()\n    _ok, _why = cache_study(str(_row.StudyInstanceUID), _row,\n                            CFG.comp_dir / 'train_series', _out)\n    print('smoke:', _ok, _why, f'{time.time() - _t0:.1f}s')\n    if not _ok:\n        raise RuntimeError(f'Preprocessing smoke test failed: {_why}')\n    with np.load(_out / f'{_row.StudyInstanceUID}.npz') as z:\n        vol, vm, sm = z['vol'], z['view_mask'], z['slice_mask']\n    fig, axes = plt.subplots(CFG.N_VIEWS, 6, figsize=(14, 2.1 * CFG.N_VIEWS))\n    show = np.linspace(0, CFG.N_SLICES - 1, 6).astype(int)\n    for v in range(CFG.N_VIEWS):\n        for c, s in enumerate(show):\n            axes[v, c].imshow(vol[v, s], cmap='gray', vmin=0, vmax=255)\n            axes[v, c].axis('off')\n            axes[v, c].grid(False)\n            if c == 0:\n                axes[v, c].set_title(f'{CFG.VIEWS[v]} view={vm[v]} valid={sm[v].sum()}',\n                                     loc='left', fontsize=8, color=FIRE)\n    plt.tight_layout(); plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:04.804569Z","iopub.execute_input":"2026-08-05T19:18:04.805061Z","iopub.status.idle":"2026-08-05T19:18:04.812105Z","shell.execute_reply.started":"2026-08-05T19:18:04.805037Z","shell.execute_reply":"2026-08-05T19:18:04.81127Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_PREPROCESS and not CFG.PREPROCESS_SMOKE_ONLY:\n    assert manifest is not None and len(manifest)\n    if not (0 <= CFG.SHARD_ID < CFG.NUM_SHARDS):\n        raise ValueError((CFG.SHARD_ID, CFG.NUM_SHARDS))\n\n    def shard_of(uid):\n        return int(hashlib.sha1(str(uid).encode()).hexdigest()[:12], 16) % CFG.NUM_SHARDS\n\n    out_dir = CFG.cache_dir / f'shard{CFG.SHARD_ID:02d}'\n    todo = manifest[manifest.StudyInstanceUID.map(shard_of) == CFG.SHARD_ID].reset_index(drop=True)\n    print(f'stable shard {CFG.SHARD_ID}/{CFG.NUM_SHARDS}: {len(todo):,} studies -> {out_dir}')\n    root = CFG.comp_dir / 'train_series'\n    records, t0 = [], time.time()\n    for i, r in todo.iterrows():\n        try:\n            ok, reason = cache_study(str(r.StudyInstanceUID), r, root, out_dir)\n        except Exception as e:\n            ok, reason = False, f'{type(e).__name__}: {str(e)[:160]}'\n        records.append({'StudyInstanceUID': r.StudyInstanceUID, 'ok': int(ok), 'reason': reason})\n        if (i + 1) % 50 == 0:\n            elapsed = time.time() - t0\n            print(f'{i+1:,}/{len(todo):,} | ok={sum(x[\"ok\"] for x in records):,} '\n                  f'| {elapsed/60:.1f} min | ETA '\n                  f'{elapsed/(i+1)*(len(todo)-i-1)/60:.1f} min', flush=True)\n\n    cache_log = pd.DataFrame(records)\n    log_path = CFG.work_dir / f'cache_manifest_shard{CFG.SHARD_ID:02d}.csv'\n    cache_log.to_csv(log_path, index=False)\n    failures = cache_log[cache_log.ok == 0]\n    print('cache complete:', cache_log.ok.sum(), '/', len(cache_log), '| log:', log_path)\n    if len(failures):\n        display(failures.head(20))\n        raise RuntimeError(f'{len(failures)} studies have no valid cache; fix before training.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:04.96151Z","iopub.execute_input":"2026-08-05T19:18:04.962488Z","iopub.status.idle":"2026-08-05T19:18:04.975946Z","shell.execute_reply.started":"2026-08-05T19:18:04.962444Z","shell.execute_reply":"2026-08-05T19:18:04.975237Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> **Stage workflow:** run all stable hash shards, publish each `/kaggle/working/rsna_cache_v2/shardXX`\n> directory and its manifest, then attach all shards to training. Do not train until coverage is 100%\n> for the requested rows.\n\n# Part 3 — leakage-safe report supervision\n\nFolds are constructed first from every observed positive and negative target, grouping studies that\nshare a retained `PatientID` or an exact normalized non-empty report. For image fold `f`, its report\nteacher is fitted only on observed labels outside `f`; it fills missing labels for training rows and\nfully unlabelled rows. Validation always uses original observed labels only.\n\nThe character/word TF-IDF vocabulary may use all *unlabelled text* (transductive representation\nlearning), but no held-out target enters the fold-specific classifier. Blank reports use zero rule\nfeatures and are never sent to `predict_proba`.\n","metadata":{}},{"cell_type":"code","source":"LEXICON = {\n    'ACL': dict(\n        pos=[r'\\bacl\\b', r'anterior cruciate', r'ligament(?:um)? crois[eé] ant[eé]rieur', r'\\blca\\b',\n             r'vorder\\w* kreuzband', r'voorste kruisband', r'\\bvkb\\b',\n             r'ligamento cruzado anterior', r'legamento crociato anteriore',\n             r'ön çapraz bağ', r'\\böçb\\b'],\n        cond=[r'tear|rupture|torn|ruptu\\w*|scheur|riss|rotura|rottura|lesi[oó]n|d[eé]chirure|yırtık|kopma']),\n    'MCL': dict(\n        pos=[r'\\bmcl\\b', r'medial collateral', r'ligament(?:um)? lat[eé]ral interne',\n             r'inner\\w* seitenband', r'mediale?s? kollateralband', r'mediale collaterale band',\n             r'ligamento colateral medial', r'legamento collaterale mediale', r'medial kollateral'],\n        cond=[r'tear|rupture|torn|sprain|ruptu\\w*|riss|distors|lesi[oó]n|d[eé]chirure|yırtık']),\n    'Medial Meniscus': dict(\n        pos=[r'medial menisc\\w+', r'm[eé]nisque interne', r'innenmeniskus', r'mediale meniscus',\n             r'menisco (?:medial|interno)', r'iç menisküs', r'\\bmm\\b'],\n        cond=[r'tear|torn|ruptu\\w*|riss|scheur|rotura|lesi[oó]n|d[eé]chirure|yırtık|degenerat']),\n    'Lateral Meniscus': dict(\n        pos=[r'lateral menisc\\w+', r'm[eé]nisque externe', r'aussenmeniskus|außenmeniskus',\n             r'laterale meniscus', r'menisco (?:lateral|externo)', r'dış menisküs', r'\\blm\\b'],\n        cond=[r'tear|torn|ruptu\\w*|riss|scheur|rotura|lesi[oó]n|d[eé]chirure|yırtık|degenerat']),\n    'Medial OA': dict(\n        pos=[r'medial (?:compartment|tibiofemoral|femorotibial)', r'compartiment interne',\n             r'mediale?n? (?:kompartiment|gelenkspalt)', r'mediale compartiment',\n             r'compartimento medial'],\n        cond=[r'osteoarthrit|arthros|gonarthros|arthrose|artrosis|artrose|cartilage loss|'\n              r'chondropath|chondral|knorpel|kraakbeen|kıkırdak|osteofit|osteophyt|joint space narrow']),\n    'Lateral OA': dict(\n        pos=[r'lateral (?:compartment|tibiofemoral|femorotibial)', r'compartiment externe',\n             r'lateral\\w* (?:kompartiment|gelenkspalt)', r'laterale compartiment',\n             r'compartimento lateral'],\n        cond=[r'osteoarthrit|arthros|gonarthros|arthrose|artrosis|artrose|cartilage loss|'\n              r'chondropath|chondral|knorpel|kraakbeen|kıkırdak|osteofit|osteophyt|joint space narrow']),\n    'PF OA': dict(\n        pos=[r'patellofemoral', r'f[eé]moro-?patellaire', r'retropatell\\w+', r'patellofemoraal',\n             r'patelofemoral', r'femoropatelar', r'patellar cartilage', r'patella\\w* knorpel'],\n        cond=[r'osteoarthrit|arthros|arthrose|artrosis|chondropath|chondral|chondromalac|'\n              r'cartilage|knorpel|kraakbeen|osteophyt|osteofit']),\n    'Effusion': dict(\n        pos=[r'effusion', r'joint fluid', r'[ée]panchement', r'gelenkerguss|erguss', r'gewrichtsvocht',\n             r'derrame(?: articular)?', r'versamento', r'eklem s[ıi]v[ıi]s[ıi]|efüzyon', r'hydrops'],\n        cond=[]),\n    'Synovitis': dict(\n        pos=[r'synovit\\w+', r'synovial (?:thickening|proliferation|hypertroph)', r'synovialitis',\n             r'sinovitis', r'sinovite', r'synovite', r'sinovit'],\n        cond=[]),\n    \"Baker's\": dict(\n        pos=[r\"baker'?s? cyst\", r'popliteal cyst', r'kyste (?:de )?baker|kyste poplit[eé]',\n             r'baker[- ]?zyste|poplitealzyste', r'bakercyste', r'quiste de baker',\n             r'cisti di baker', r'baker kisti', r'gastrocnemio-?semimembranosus'],\n        cond=[]),\n    'Contusion': dict(\n        pos=[r'contusion', r'bone bruise', r'bone marrow (?:edema|oedema)', r'knochenmark[soö]dem',\n             r'beenmerg[oö]edeem', r'[oó]edema (?:[oó]seo|de m[eé]dula)', r'contus[aã]o',\n             r'contusione', r'kemik ili[gğ]i [oö]dem', r'trabecular (?:edema|injury)'],\n        cond=[]),\n    'Fracture': dict(\n        pos=[r'fracture', r'fraktur', r'fractuur', r'fractura', r'frattura', r'k[ıi]r[ıi][kg]',\n             r'avulsion', r'segond'],\n        cond=[]),\n}\n\nNEG_CUES = [\n    r'\\bno\\b', r'\\bnot\\b', r'\\bwithout\\b', r'\\bintact\\b', r'\\bnormal\\w*', r'\\bunremarkable\\b',\n    r'\\bnegative\\b', r'\\bfree of\\b', r'\\bruled out\\b', r'\\bgeen\\b', r'\\bkein\\w*', r'\\bohne\\b',\n    r'\\bintakt\\b', r\"\\bpas d[eu']\", r'\\bsans\\b', r'\\baucun\\w*', r'\\babsence\\b', r'\\bsin\\b',\n    r'\\bsem\\b', r'\\bsenza\\b', r'\\bnessun\\w*', r'\\byok(?:tur)?\\b', r'\\bdo[gğ]al\\b',\n    r'\\bregelrecht\\b', r'\\bunauff[aä]llig\\b', r'\\bnegatif\\b',\n]\nNEG_RE = re.compile('|'.join(NEG_CUES), re.I)\nCLAUSE_SPLIT = re.compile(r'[.;:\\n]')\n\n\ndef rule_score(text: str) -> dict:\n    \"\"\"Rule-based estimate per label: 0.03 / 0.10 / 0.15 / 0.90.\n\n    Key detail: negation is searched only inside the current clause. Otherwise\n    \"Kein Erguss. Innenmeniskus Riss\" marks the meniscal tear as negated.\n    \"\"\"\n    if not isinstance(text, str) or not text.strip():\n        return {t: 0.0 for t in TARGETS}\n    t = ' ' + re.sub(r'\\s+', ' ', text.lower()) + ' '\n    out = {}\n    for label, spec in LEXICON.items():\n        pos_re = re.compile('|'.join(spec['pos']), re.I)\n        cond_re = re.compile('|'.join(spec['cond']), re.I) if spec['cond'] else None\n        hits = negated = conditioned = 0\n        for m in pos_re.finditer(t):\n            hits += 1\n            left = CLAUSE_SPLIT.split(t[max(0, m.start() - 80):m.start()])[-1]\n            right = CLAUSE_SPLIT.split(t[m.end():m.end() + 110])[0]\n            if NEG_RE.search(left) or NEG_RE.search(right[:40]):\n                negated += 1\n            if cond_re is None or cond_re.search(left) or cond_re.search(right):\n                conditioned += 1\n        if hits == 0:\n            out[label] = 0.03                       # never mentioned\n        elif conditioned == 0:\n            out[label] = 0.15                       # mentioned, no finding word nearby\n        elif negated >= conditioned:\n            out[label] = 0.10                       # explicitly negated\n        else:\n            out[label] = 0.90                       # asserted\n    return out","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:12.081262Z","iopub.execute_input":"2026-08-05T19:18:12.081775Z","iopub.status.idle":"2026-08-05T19:18:12.095645Z","shell.execute_reply.started":"2026-08-05T19:18:12.081736Z","shell.execute_reply":"2026-08-05T19:18:12.094904Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- self-test: rerun this after every edit to the lexicon ---\nSANITY = {\n    'EN pos': 'Complete tear of the anterior cruciate ligament. Medial meniscus posterior horn tear. '\n              'Small joint effusion.',\n    'EN neg': 'The anterior cruciate ligament is intact. No meniscal tear. No joint effusion.',\n    'NL pos': 'Ruptuur van de voorste kruisband. Scheur mediale meniscus achterhoorn. Bakercyste aanwezig.',\n    'DE mix': 'Vorderes Kreuzband intakt. Kein Erguss. Innenmeniskus Riss im Hinterhorn.',\n    'FR mix': \"Rupture du ligament croisé antérieur. Pas d'épanchement articulaire. Kyste de Baker.\",\n    'ES pos': 'Rotura del menisco medial. Derrame articular moderado. Artrosis del compartimento medial.',\n}\n_s = pd.DataFrame({k: rule_score(v) for k, v in SANITY.items()}).T[TARGETS]\n\nassert _s.loc['EN pos', ['ACL', 'Medial Meniscus', 'Effusion']].min() > .5\nassert _s.loc['EN neg', ['ACL', 'Medial Meniscus', 'Effusion']].max() < .5\nassert _s.loc['DE mix', 'Medial Meniscus'] > .5 and _s.loc['DE mix', 'Effusion'] < .5\nassert _s.loc['FR mix', 'Effusion'] < .5 and _s.loc['FR mix', \"Baker's\"] > .5\nassert _s.loc['NL pos', ['ACL', 'Medial Meniscus', \"Baker's\"]].min() > .5\nprint('rule_score self-test passed — clause-scoped negation and French elision both handled')\ndisplay(_s.style.background_gradient(cmap=CMAP_SEQ, vmin=0, vmax=1).format('{:.2f}'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:12.224601Z","iopub.execute_input":"2026-08-05T19:18:12.225047Z","iopub.status.idle":"2026-08-05T19:18:12.328043Z","shell.execute_reply.started":"2026-08-05T19:18:12.225023Z","shell.execute_reply":"2026-08-05T19:18:12.327465Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_TEACHER:\n    from sklearn.metrics import roc_auc_score, average_precision_score\n\n    t0 = time.time()\n    rules = pd.DataFrame([rule_score(x) for x in train['Report']], index=train.index)[TARGETS]\n    rules = rules.fillna(0.0).astype(np.float32)\n    print(f'rules computed in {time.time()-t0:.1f}s | finite: {np.isfinite(rules.values).all()}')\n\n    ev = []\n    for c in TARGETS:\n        m = observed[c] & has_report\n        y, p = train.loc[m, c].astype(int).values, rules.loc[m, c].values\n        if len(np.unique(y)) < 2:\n            continue\n        ev.append({'label': c, 'n': int(m.sum()), 'n_pos': int(y.sum()),\n                   'rule_auc': roc_auc_score(y, p),\n                   'rule_ap': average_precision_score(y, p)})\n    ev = pd.DataFrame(ev).sort_values('rule_auc')\n    display(ev.style.background_gradient(cmap=CMAP_SEQ, subset=['rule_auc'], vmin=.5, vmax=1)\n                    .format({'rule_auc': '{:.3f}', 'rule_ap': '{:.3f}'}))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:12.416501Z","iopub.execute_input":"2026-08-05T19:18:12.417069Z","iopub.status.idle":"2026-08-05T19:18:12.423464Z","shell.execute_reply.started":"2026-08-05T19:18:12.417046Z","shell.execute_reply":"2026-08-05T19:18:12.422889Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.1 Patient/report groups and partial-label multilabel folds\n\nPositive and negative observations are separate stratification features. This matters because\nmissing does not mean negative. A small union-find joins studies by retained `PatientID` and by exact\nnormalized report duplicates before the greedy group assignment.\n","metadata":{}},{"cell_type":"code","source":"def _normal_report(text):\n    if not isinstance(text, str):\n        return ''\n    text = unicodedata.normalize('NFKC', text).casefold()\n    return re.sub(r'\\s+', ' ', text).strip()\n\n\ndef collect_patient_ids(studies, series_df, root):\n    \"\"\"Read at most one header per study; return blank when PatientID was stripped.\"\"\"\n    first_series = series_df.drop_duplicates('StudyInstanceUID').set_index('StudyInstanceUID') \\\n                            .SeriesInstanceUID.to_dict()\n    out = {}\n    for i, uid in enumerate(studies):\n        sid = first_series.get(uid)\n        pid = ''\n        if sid is not None:\n            files = list((Path(root) / str(uid) / str(sid)).glob('*.dcm'))\n            if files:\n                try:\n                    ds = pydicom.dcmread(str(files[0]), stop_before_pixels=True,\n                                         specific_tags=['PatientID'])\n                    pid = str(getattr(ds, 'PatientID', '') or '').strip()\n                except Exception:\n                    pass\n        out[uid] = pid\n        if (i + 1) % 1000 == 0:\n            print('patient headers:', i + 1, '/', len(studies), flush=True)\n    return pd.Series(out)\n\n\nclass UnionFind:\n    def __init__(self, n):\n        self.p = np.arange(n)\n    def find(self, x):\n        while self.p[x] != x:\n            self.p[x] = self.p[self.p[x]]\n            x = self.p[x]\n        return int(x)\n    def union(self, a, b):\n        a, b = self.find(a), self.find(b)\n        if a != b:\n            self.p[b] = a\n\n\ndef make_groups(df, patient_ids):\n    uf = UnionFind(len(df))\n    pid_values = patient_ids.reindex(df.StudyInstanceUID).fillna('').astype(str)\n    # Some anonymizers replace every PatientID with one constant. Do not merge an\n    # implausibly large fraction of the dataset into one artificial patient.\n    pid_counts = pid_values[pid_values != ''].value_counts()\n    oversized = pid_counts[pid_counts > max(25, int(.01 * len(df)))].index\n    pid_values = pid_values.mask(pid_values.isin(oversized), '')\n    if len(oversized):\n        print(f'ignored {len(oversized)} implausibly common PatientID value(s)')\n    report_values = df.Report.map(_normal_report).where(lambda s: s.str.len() > 30, '')\n    for values in [pid_values.values, report_values.values]:\n        seen = {}\n        for i, value in enumerate(values):\n            if not value:\n                continue\n            if value in seen:\n                uf.union(i, seen[value])\n            else:\n                seen[value] = i\n    return np.asarray([f'g{uf.find(i):07d}' for i in range(len(df))])\n\n\ndef group_multilabel_folds(df, groups, n_splits=5, seed=42):\n    \"\"\"Greedy group split balancing observed positives and observed negatives per target.\"\"\"\n    obs = df[TARGETS].notna().values\n    pos = (df[TARGETS].fillna(0).values > .5) & obs\n    feat = np.concatenate([pos, obs & ~pos], axis=1).astype(np.int16)\n    tmp = pd.DataFrame(feat)\n    tmp['group'] = groups\n    group_feat = tmp.groupby('group').sum()\n    group_size = pd.Series(groups).value_counts().reindex(group_feat.index).values\n    labelled_group = group_feat.values.sum(1) > 0\n\n    X = group_feat.values[labelled_group]\n    names = group_feat.index.values[labelled_group]\n    sizes = group_size[labelled_group]\n    total = X.sum(0).astype(float)\n    rarity = (X / np.maximum(total, 1)).sum(1)\n    rng = np.random.default_rng(seed)\n    tie = rng.random(len(X))\n    order = np.lexsort((tie, -sizes, -rarity))\n\n    assigned = np.zeros((n_splits, X.shape[1]), float)\n    fold_sizes = np.zeros(n_splits, float)\n    target = total / n_splits\n    target_size = sizes.sum() / n_splits\n    group_fold = {}\n    for j in order:\n        active = X[j] > 0\n        deficit = ((target[None, :] - assigned)[:, active] /\n                   np.maximum(target[active], 1)).sum(1) if active.any() else np.zeros(n_splits)\n        size_need = (target_size - fold_sizes) / max(target_size, 1)\n        score = deficit + 0.25 * size_need\n        best = np.flatnonzero(score == score.max())\n        f = int(best[np.argmin(fold_sizes[best])])\n        group_fold[names[j]] = f\n        assigned[f] += X[j]\n        fold_sizes[f] += sizes[j]\n\n    # Purely unlabelled groups remain -1 and are used in every fold's training set.\n    return np.asarray([group_fold.get(g, -1) for g in groups], dtype=np.int8)\n\n\ndef fold_balance(df):\n    rows = []\n    for f in sorted(x for x in df.fold.unique() if x >= 0):\n        for c in TARGETS:\n            m = (df.fold == f) & df[c].notna()\n            rows.append({'fold': f, 'target': c, 'n': int(m.sum()),\n                         'pos': int(df.loc[m, c].sum()),\n                         'neg': int(m.sum() - df.loc[m, c].sum())})\n    return pd.DataFrame(rows)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:12.750273Z","iopub.execute_input":"2026-08-05T19:18:12.750955Z","iopub.status.idle":"2026-08-05T19:18:12.770572Z","shell.execute_reply.started":"2026-08-05T19:18:12.750918Z","shell.execute_reply":"2026-08-05T19:18:12.769692Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_TEACHER:\n    patient_ids = collect_patient_ids(train.StudyInstanceUID.tolist(), train_series,\n                                      CFG.comp_dir / 'train_series')\n    groups = make_groups(train, patient_ids)\n    folds = group_multilabel_folds(train, groups, CFG.folds, CFG.seed)\n    fold_df = train[['StudyInstanceUID'] + TARGETS].copy()\n    fold_df['group'] = groups\n    fold_df['fold'] = folds\n    fold_df.to_csv(CFG.teacher_dir / 'folds.csv', index=False)\n\n    bal = fold_balance(fold_df)\n    display(bal.pivot(index='fold', columns='target', values='pos'))\n    bad = bal[(bal['pos'] == 0) | (bal['neg'] == 0)]\n    print('fold sizes (labelled groups):', fold_df[fold_df.fold >= 0].fold.value_counts().sort_index().to_dict())\n    print('unlabelled-only rows used by every training fold:', int((fold_df.fold < 0).sum()))\n    if len(bad):\n        display(bad)\n        raise RuntimeError('At least one fold/target lacks a positive or negative; reduce CFG.folds.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:12.921467Z","iopub.execute_input":"2026-08-05T19:18:12.921846Z","iopub.status.idle":"2026-08-05T19:18:12.927843Z","shell.execute_reply.started":"2026-08-05T19:18:12.921821Z","shell.execute_reply":"2026-08-05T19:18:12.926941Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3.2 Fold-specific TF-IDF teachers\n\nCharacter 3–5 grams handle multilingual morphology; word 1–2 grams retain phrase context. Balanced\nlogistic regression is useful for ranking rare findings, then its logits are adjusted back to the\ntraining prior before becoming BCE soft targets. Pseudo-label weights depend on grouped OOF AUC and\nsample confidence. Known cells always keep weight 1.0.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_TEACHER:\n    from scipy import sparse\n    from sklearn.feature_extraction.text import TfidfVectorizer\n    from sklearn.linear_model import LogisticRegression\n    from sklearn.metrics import roc_auc_score\n\n    txt = train.Report.fillna('').map(_normal_report).values\n    report_rows = has_report.values\n    vec_char = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5), min_df=4,\n                               max_features=350_000, sublinear_tf=True, dtype=np.float32)\n    vec_word = TfidfVectorizer(analyzer='word', ngram_range=(1, 2), min_df=3,\n                               max_features=200_000, sublinear_tf=True, dtype=np.float32)\n    X = sparse.hstack([\n        vec_char.fit_transform(txt),\n        vec_word.fit_transform(txt),\n        sparse.csr_matrix(rules.fillna(0).values.astype(np.float32)),\n    ], format='csr')\n    assert np.isfinite(X.data).all()\n    print('teacher feature matrix:', X.shape, '| nnz:', f'{X.nnz:,}')\n\n    def prior_correct(p, prevalence):\n        p = np.clip(np.asarray(p), 1e-5, 1 - 1e-5)\n        prevalence = float(np.clip(prevalence, 1e-4, 1 - 1e-4))\n        logit = np.log(p / (1 - p)) + np.log(prevalence / (1 - prevalence))\n        return (1 / (1 + np.exp(-logit))).astype(np.float32)\n\n    def fit_predict_teacher(X, y, fit_mask, pred_mask, fallback):\n        yfit = y[fit_mask].astype(int)\n        n_pos, n_neg = int(yfit.sum()), int(len(yfit) - yfit.sum())\n        out = np.asarray(fallback, np.float32).copy()\n        if min(n_pos, n_neg) < 2:\n            return out\n        model = LogisticRegression(C=3.0, max_iter=1500, class_weight='balanced',\n                                   solver='liblinear', random_state=CFG.seed)\n        model.fit(X[fit_mask], yfit)\n        raw = model.predict_proba(X[pred_mask])[:, 1]\n        out[pred_mask] = prior_correct(raw, yfit.mean())\n        return out\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:13.290333Z","iopub.execute_input":"2026-08-05T19:18:13.290646Z","iopub.status.idle":"2026-08-05T19:18:13.298719Z","shell.execute_reply.started":"2026-08-05T19:18:13.290623Z","shell.execute_reply":"2026-08-05T19:18:13.298039Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_TEACHER:\n    n, T, FOLDS = len(train), len(TARGETS), list(range(CFG.folds))\n    teacher_pred = np.full((CFG.folds, n, T), np.nan, np.float32)\n    teacher_fit_ok = np.zeros((CFG.folds, T), bool)\n    oof = np.full((n, T), np.nan, np.float32)\n\n    for f in FOLDS:\n        print(f'\\nreport teacher for image fold {f}')\n        for ti, c in enumerate(TARGETS):\n            fit_mask = (train[c].notna().values & report_rows & (folds != f))\n            yfit = train.loc[fit_mask, c].astype(int).values\n            teacher_fit_ok[f, ti] = (len(yfit) > 0 and yfit.sum() >= 2 and\n                                      (len(yfit) - yfit.sum()) >= 2)\n            fallback = rules[c].values.astype(np.float32)\n            pred = fit_predict_teacher(X, train[c].fillna(0).values,\n                                       fit_mask, report_rows, fallback)\n            teacher_pred[f, :, ti] = pred\n            val_mask = train[c].notna().values & report_rows & (folds == f)\n            oof[val_mask, ti] = pred[val_mask]\n\n    quality = []\n    for ti, c in enumerate(TARGETS):\n        m = np.isfinite(oof[:, ti]) & train[c].notna().values\n        y = train.loc[m, c].astype(int).values\n        auc = roc_auc_score(y, oof[m, ti]) if len(np.unique(y)) == 2 else 0.5\n        quality.append(auc)\n    quality = np.asarray(quality, np.float32)\n    qdf = pd.DataFrame({'target': TARGETS,\n                        'grouped_oof_auc_diagnostic': quality}).sort_values(\n                            'grouped_oof_auc_diagnostic')\n    display(qdf.style.background_gradient(cmap=CMAP_SEQ,\n                                          subset=['grouped_oof_auc_diagnostic'], vmin=.5, vmax=1)\n                 .format({'grouped_oof_auc_diagnostic': '{:.4f}'}))\n    qdf.to_csv(CFG.teacher_dir / 'teacher_quality.csv', index=False)\n\n    original = train[TARGETS].values.astype(np.float32)\n    obs_np = observed.values\n    for f in FOLDS:\n        pred = teacher_pred[f]\n        target = np.where(obs_np, original, pred).astype(np.float32)\n        confidence = np.abs(pred - .5) * 2\n        # OOF AUC is diagnostic only: using held-out outcomes to scale this fold's\n        # training loss would leak validation information through 12 scalars.\n        fit_scale = np.where(teacher_fit_ok[f], 1.0, 0.25)[None, :]\n        pseudo_weight = (CFG.pseudo_w * fit_scale *\n                         (CFG.pseudo_conf_floor + (1 - CFG.pseudo_conf_floor) * confidence))\n        pseudo_allowed = report_rows[:, None] & (folds[:, None] != f)\n        weights = np.where(obs_np, 1.0, np.where(pseudo_allowed, pseudo_weight, 0.0)) \\\n                    .astype(np.float32)\n        out = pd.DataFrame({'StudyInstanceUID': train.StudyInstanceUID,\n                            'fold': folds, 'group': groups})\n        out[TARGETS] = target\n        out[WEIGHT_COLS] = weights\n        out[OBS_COLS] = obs_np.astype(np.uint8)\n        out = out[weights.sum(1) > 0].reset_index(drop=True)\n        out.to_csv(CFG.teacher_dir / f'targets_fold{f}.csv', index=False)\n        print(f'fold {f}: wrote {len(out):,} supervised rows')\n\n    meta_out = {\n        'cache_version': CFG.CACHE_VERSION,\n        'folds': CFG.folds,\n        'targets': TARGETS,\n        'teacher_oof_auc': dict(zip(TARGETS, map(float, quality))),\n    }\n    with (CFG.teacher_dir / 'teacher_meta.json').open('w') as f:\n        json.dump(meta_out, f, indent=2)\n    print('teacher artifacts:', CFG.teacher_dir)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:13.567565Z","iopub.execute_input":"2026-08-05T19:18:13.567893Z","iopub.status.idle":"2026-08-05T19:18:13.579888Z","shell.execute_reply.started":"2026-08-05T19:18:13.567872Z","shell.execute_reply":"2026-08-05T19:18:13.579202Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Part 4 — target-aware 2.5D student\n\n```\n(B,V,D,H,W)\n  → adjacent-slice 2.5D channels\n  → shared pretrained encoder + backbone-specific normalization\n  → add view and normalized depth embeddings\n  → 12 learned slice queries: (B,V,T,D)\n  → 12 learned view queries:  (B,T,V)\n  → 12 target-specific linear heads\n```\n\nACL, MCL, PF OA and Baker's cyst no longer share the same attention distribution. Clinical view priors\nonly initialize the learned view logits; masks and data decide the final weights. The auxiliary loss is\nrouted only to plausible target/view pairs. Final epochs use observed hard targets only.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_TRAIN or CFG.RUN_INFER:\n    import copy\n    import torch\n    import torch.nn as nn\n    import torch.nn.functional as F\n    from torch.utils.data import Dataset, DataLoader, WeightedRandomSampler\n    import timm\n\n    DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n    if DEVICE != 'cuda':\n        raise RuntimeError('Train/infer stage requires a GPU for this configuration.')\n    torch.backends.cudnn.benchmark = True\n    torch.backends.cuda.matmul.allow_tf32 = True\n    print('device:', DEVICE, '| timm:', timm.__version__, '| torch:', torch.__version__)\n\n    _NEW_AMP = hasattr(torch.amp, 'GradScaler')\n\n    def make_scaler():\n        return (torch.amp.GradScaler('cuda', enabled=CFG.amp) if _NEW_AMP\n                else torch.cuda.amp.GradScaler(enabled=CFG.amp))\n\n    def autocast():\n        return (torch.amp.autocast('cuda', enabled=CFG.amp) if _NEW_AMP\n                else torch.cuda.amp.autocast(enabled=CFG.amp))\n\n    def resolve_backbone(candidates, pretrained):\n        available = set(timm.list_models())\n        weighted = set(timm.list_models(pretrained=True))\n        for name in candidates:\n            base = name.split('.')[0]\n            if pretrained and name in weighted:\n                return name\n            if not pretrained and (name in available or base in available):\n                return name if name in available else base\n        for name in candidates:\n            base = name.split('.')[0]\n            if base in available:\n                return base\n        raise RuntimeError(f'No configured backbone exists in timm {timm.__version__}: {candidates}')\n\n    BACKBONE = resolve_backbone(CFG.backbone, CFG.pretrained if CFG.RUN_TRAIN else False)\n    print('backbone:', BACKBONE)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:14.308665Z","iopub.execute_input":"2026-08-05T19:18:14.309089Z","iopub.status.idle":"2026-08-05T19:18:21.573925Z","shell.execute_reply.started":"2026-08-05T19:18:14.309063Z","shell.execute_reply":"2026-08-05T19:18:21.573079Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_TRAIN or CFG.RUN_INFER:\n    _VALIDATED_CACHE_PATHS = set()\n\n    def discover_cache(cache_dirs):\n        index = {}\n        duplicates = []\n        for d in cache_dirs:\n            d = Path(d)\n            if not d.exists():\n                continue\n            for p in d.rglob('*.npz'):\n                if p.name.endswith('.tmp.npz'):\n                    continue\n                if p.stem in index:\n                    duplicates.append(p.stem)\n                else:\n                    index[p.stem] = p\n        if duplicates:\n            print(f'[cache] {len(set(duplicates))} duplicate UIDs; first configured directory wins')\n        return index\n\n\n    class KneeDataset(Dataset):\n        def __init__(self, df, cache_dirs, train_mode, hard_only=False, strict=True):\n            self.df = df.reset_index(drop=True).copy()\n            self.train_mode = train_mode\n            self.hard_only = hard_only\n            self.index = discover_cache(cache_dirs)\n            missing = [str(x) for x in self.df.StudyInstanceUID if str(x) not in self.index]\n            if missing and strict:\n                raise FileNotFoundError(\n                    f'{len(missing)}/{len(self.df)} requested cache items are missing; first: {missing[:5]}. '\n                    'Attach every shard. Blank-study fallback is intentionally disabled.')\n            self.df = self.df[self.df.StudyInstanceUID.astype(str).isin(self.index)].reset_index(drop=True)\n            invalid = []\n            for uid in self.df.StudyInstanceUID.astype(str):\n                p = self.index[uid]\n                key = (str(p), p.stat().st_size, p.stat().st_mtime_ns)\n                if key not in _VALIDATED_CACHE_PATHS:\n                    if not _valid_cached(p):\n                        invalid.append(str(p))\n                    else:\n                        _VALIDATED_CACHE_PATHS.add(key)\n            if invalid:\n                raise RuntimeError(f'{len(invalid)} stale/corrupt cache files; first: {invalid[:5]}')\n            if hard_only:\n                w = self.df[WEIGHT_COLS].values\n                self.df = self.df[(w >= .999).any(1)].reset_index(drop=True)\n            print(f'dataset: {len(self.df):,} rows | hard_only={hard_only} | cache={len(self.index):,}')\n\n        def __len__(self):\n            return len(self.df)\n\n        @staticmethod\n        def _augment(vol, view_mask):\n            out = np.zeros_like(vol)\n            for v in range(CFG.N_VIEWS):\n                if not view_mask[v]:\n                    continue\n                angle = np.random.uniform(-8, 8)\n                scale = np.random.uniform(.93, 1.07)\n                tx, ty = np.random.uniform(-.04, .04, 2) * CFG.IMG_SIZE\n                M = cv2.getRotationMatrix2D((CFG.IMG_SIZE / 2, CFG.IMG_SIZE / 2), angle, scale)\n                M[0, 2] += tx\n                M[1, 2] += ty\n                for d in range(CFG.N_SLICES):\n                    out[v, d] = cv2.warpAffine(vol[v, d], M,\n                                               (CFG.IMG_SIZE, CFG.IMG_SIZE),\n                                               flags=cv2.INTER_LINEAR,\n                                               borderMode=cv2.BORDER_CONSTANT)\n            x = out.astype(np.float32) / 255.0\n            # Contrast/intensity varies independently by sequence.\n            gain = np.random.uniform(.88, 1.12, (CFG.N_VIEWS, 1, 1, 1))\n            bias = np.random.uniform(-.05, .05, (CFG.N_VIEWS, 1, 1, 1))\n            return np.clip(x * gain + bias, 0, 1).astype(np.float32)\n\n        def __getitem__(self, i):\n            r = self.df.iloc[i]\n            path = self.index[str(r.StudyInstanceUID)]\n            with np.load(path, allow_pickle=False) as z:\n                vol = z['vol']\n                view_mask = z['view_mask'].astype(np.float32)\n                slice_mask = z['slice_mask'].astype(np.float32)\n            if not view_mask.any():\n                raise RuntimeError(f'cache item has no valid view: {path}')\n            x = self._augment(vol, view_mask) if self.train_mode else vol.astype(np.float32) / 255.0\n            y = r[TARGETS].values.astype(np.float32)\n            w = r[WEIGHT_COLS].values.astype(np.float32)\n            if self.hard_only:\n                w = w * (w >= .999)\n            return {\n                'x': torch.from_numpy(np.ascontiguousarray(x)),\n                'view_mask': torch.from_numpy(view_mask),\n                'slice_mask': torch.from_numpy(slice_mask),\n                'y': torch.from_numpy(y),\n                'w': torch.from_numpy(w),\n                'idx': i,\n            }\n\n\n    def masked_softmax(logits, mask, dim=-1):\n        \"\"\"AMP-safe masked softmax; an all-masked row returns exactly zero.\"\"\"\n        mask = torch.broadcast_to(mask.to(device=logits.device, dtype=torch.bool), logits.shape)\n        z = logits.float().masked_fill(~mask, -torch.inf)\n        zmax = z.amax(dim=dim, keepdim=True)\n        zmax = torch.where(torch.isfinite(zmax), zmax, torch.zeros_like(zmax))\n        p = torch.exp(z - zmax) * mask.to(z.dtype)\n        den = p.sum(dim=dim, keepdim=True)\n        p = torch.where(den > 0, p / den.clamp_min(torch.finfo(p.dtype).tiny),\n                        torch.zeros_like(p))\n        return p.to(logits.dtype)\n\n\n    # Rows=targets, columns=CFG.VIEWS. Used for initialization and auxiliary routing only.\n    _R = {\n        'ACL':             [1, 1, 1, 0, 0],\n        'MCL':             [0, 0, 1, 1, 0],\n        'Medial Meniscus': [1, 1, 1, 1, 0],\n        'Lateral Meniscus':[1, 1, 1, 1, 0],\n        'Medial OA':       [1, 1, 1, 1, 1],\n        'Lateral OA':      [1, 1, 1, 1, 1],\n        'PF OA':           [1, 1, 0, 0, 1],\n        'Effusion':        [1, 0, 1, 0, 1],\n        'Synovitis':       [1, 0, 1, 0, 1],\n        \"Baker's\":        [1, 1, 0, 0, 1],\n        'Contusion':       [1, 0, 1, 0, 1],\n        'Fracture':        [1, 1, 1, 1, 1],\n    }\n    TARGET_VIEW_ROUTE = torch.tensor([_R[c] for c in TARGETS], dtype=torch.float32)\n\n\n    class KneeNet(nn.Module):\n        def __init__(self, backbone=BACKBONE, pretrained=False):\n            super().__init__()\n            self.backbone_name = backbone\n            self.enc = timm.create_model(backbone, pretrained=pretrained, num_classes=0,\n                                         in_chans=3, drop_rate=CFG.drop_rate,\n                                         drop_path_rate=CFG.drop_path_rate)\n            if CFG.grad_ckpt:\n                try:\n                    self.enc.set_grad_checkpointing(True)\n                except Exception:\n                    pass\n            C, T, V = self.enc.num_features, len(TARGETS), CFG.N_VIEWS\n            try:\n                pcfg = timm.data.resolve_model_data_config(self.enc)\n            except Exception:\n                pcfg = getattr(self.enc, 'pretrained_cfg', {}) or {}\n            mean = torch.tensor(pcfg.get('mean', (0.485, 0.456, 0.406))).view(1, 3, 1, 1)\n            std = torch.tensor(pcfg.get('std', (0.229, 0.224, 0.225))).view(1, 3, 1, 1)\n            self.register_buffer('input_mean', mean, persistent=True)\n            self.register_buffer('input_std', std, persistent=True)\n\n            self.view_emb = nn.Parameter(torch.randn(V, C) * .02)\n            self.depth_mlp = nn.Sequential(nn.Linear(1, C), nn.Tanh(), nn.Linear(C, C))\n            self.slice_query = nn.Parameter(torch.randn(T, C) * .02)\n            self.view_query = nn.Parameter(torch.randn(T, C) * .02)\n            init_route = torch.where(TARGET_VIEW_ROUTE > 0, 0.75, -0.75)\n            self.view_prior = nn.Parameter(init_route)\n            self.norm = nn.LayerNorm(C)\n            self.drop = nn.Dropout(CFG.drop_rate)\n            self.head_weight = nn.Parameter(torch.randn(T, C) * (1 / math.sqrt(C)))\n            self.head_bias = nn.Parameter(torch.zeros(T))\n\n        @staticmethod\n        def to_25d(x):\n            prev = torch.cat([x[:, :, :1], x[:, :, :-1]], 2)\n            nxt = torch.cat([x[:, :, 1:], x[:, :, -1:]], 2)\n            return torch.stack([prev, x, nxt], 3)\n\n        def forward(self, x, view_mask, slice_mask):\n            B, V, D, H, W = x.shape\n            x = self.to_25d(x).reshape(B * V * D, 3, H, W)\n            x = (x - self.input_mean) / self.input_std\n            valid = ((slice_mask > 0) & (view_mask[:, :, None] > 0)).reshape(-1)\n            valid_idx = valid.nonzero(as_tuple=False).squeeze(1)\n            if not len(valid_idx):\n                raise RuntimeError('A batch contains no valid DICOM slices')\n            encoded = self.enc(x.index_select(0, valid_idx))\n            flat = encoded.new_zeros((B * V * D, self.enc.num_features))\n            flat.index_copy_(0, valid_idx, encoded)\n            f = flat.reshape(B, V, D, -1)\n            depth = torch.linspace(-1, 1, D, device=f.device, dtype=f.dtype).view(D, 1)\n            f = self.norm(f + self.view_emb[None, :, None, :] +\n                          self.depth_mlp(depth)[None, None, :, :])\n\n            # Target-specific attention over depth: B,V,T,D.\n            s = torch.einsum('bvdc,tc->bvtd', f, self.slice_query) / math.sqrt(f.shape[-1])\n            a_slice = masked_softmax(s.permute(0, 1, 3, 2),\n                                     slice_mask[:, :, None, :], dim=-1)\n            fv = torch.einsum('bvtd,bvdc->bvtc', a_slice, f)\n\n            # Target-specific attention over sequence slots: B,T,V.\n            sv = torch.einsum('bvtc,tc->btv', fv, self.view_query) / math.sqrt(f.shape[-1])\n            sv = sv + self.view_prior[None, :, :]\n            a_view = masked_softmax(sv, view_mask[:, None, :], dim=-1)\n            ft = torch.einsum('btv,bvtc->btc', a_view, fv)\n            logits = (self.drop(ft) * self.head_weight[None]).sum(-1) + self.head_bias\n            aux = (self.drop(fv) * self.head_weight[None, None]).sum(-1) + \\\n                  self.head_bias[None, None]\n            return logits, aux, a_view, a_slice\n\n\n    @torch.no_grad()\n    def ema_update(ema_model, model, decay, num_updates):\n        # Avoid an initial-model bias when a fold has only a few hundred updates.\n        decay = min(float(decay), (1 + num_updates) / (10 + num_updates))\n        ema_state, state = ema_model.state_dict(), model.state_dict()\n        for key, value in ema_state.items():\n            src = state[key].detach()\n            if value.dtype.is_floating_point:\n                value.mul_(decay).add_(src, alpha=1 - decay)\n            else:\n                value.copy_(src)\n\n\n    def masked_multitask_loss(logits, aux, y, weight, view_mask, pos_weight):\n        \"\"\"Macro target reduction; pseudo reliability is not renormalized away.\"\"\"\n        valid = weight > 0\n        raw = F.binary_cross_entropy_with_logits(logits, y, reduction='none',\n                                                 pos_weight=pos_weight)\n        denom = valid.sum(0)\n        main_per_target = (raw * weight).sum(0) / denom.clamp_min(1)\n        present = denom > 0\n        main = main_per_target[present].mean() if present.any() else logits.sum() * 0\n\n        aux_raw = F.binary_cross_entropy_with_logits(aux, y[:, None, :].expand_as(aux),\n                                                     reduction='none', pos_weight=pos_weight)\n        route = TARGET_VIEW_ROUTE.T.to(aux.device)[None]\n        routed_view = view_mask[:, :, None] * route\n        per_study_aux = (aux_raw * routed_view).sum(1) / routed_view.sum(1).clamp_min(1)\n        aux_valid = valid & (routed_view.sum(1) > 0)\n        aux_denom = aux_valid.sum(0)\n        aux_per_target = (per_study_aux * weight * aux_valid).sum(0) / aux_denom.clamp_min(1)\n        aux_present = aux_denom > 0\n        aux_loss = (aux_per_target[aux_present].mean() if aux_present.any()\n                    else logits.sum() * 0)\n        return main + CFG.aux_w * aux_loss, main.detach(), aux_loss.detach()\n\n\n    print('dataset and target-aware model defined')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:21.575285Z","iopub.execute_input":"2026-08-05T19:18:21.576123Z","iopub.status.idle":"2026-08-05T19:18:21.618373Z","shell.execute_reply.started":"2026-08-05T19:18:21.576098Z","shell.execute_reply":"2026-08-05T19:18:21.617812Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Model smoke test\n\nThe smoke test checks exact output shapes, finite loss, normalization buffers, and zero attention for\nmasked slices/views. It runs for both training and inference stages before loading a long job.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_TRAIN or CFG.RUN_INFER:\n    _m = KneeNet(pretrained=False).to(DEVICE)\n    _B = min(CFG.batch_size, 2)\n    _x = torch.rand(_B, CFG.N_VIEWS, CFG.N_SLICES, CFG.IMG_SIZE, CFG.IMG_SIZE, device=DEVICE)\n    _vm = torch.ones(_B, CFG.N_VIEWS, device=DEVICE)\n    _vm[:, -1] = 0\n    _sm = torch.ones(_B, CFG.N_VIEWS, CFG.N_SLICES, device=DEVICE)\n    _sm[:, -1] = 0\n    _sm[:, 0, -2:] = 0\n    _y = (torch.rand(_B, len(TARGETS), device=DEVICE) > .7).float()\n    _w = torch.ones_like(_y)\n    with autocast():\n        _logit, _aux, _av, _as = _m(_x, _vm, _sm)\n        _loss, _, _ = masked_multitask_loss(_logit, _aux, _y, _w, _vm,\n                                            torch.ones(len(TARGETS), device=DEVICE))\n    _loss.backward()\n    assert _logit.shape == (_B, len(TARGETS))\n    assert _aux.shape == (_B, CFG.N_VIEWS, len(TARGETS))\n    assert _av.shape == (_B, len(TARGETS), CFG.N_VIEWS)\n    assert torch.isfinite(_loss)\n    assert float(_av[:, :, -1].abs().max()) < 1e-5\n    assert float(_as[:, -1].abs().max()) < 1e-5\n    print(f'smoke ok | params={sum(p.numel() for p in _m.parameters())/1e6:.1f}M '\n          f'| loss={float(_loss):.4f} | peak={torch.cuda.max_memory_allocated()/2**30:.2f} GiB')\n    del _m, _x, _vm, _sm, _y, _w, _logit, _aux, _av, _as, _loss\n    gc.collect(); torch.cuda.empty_cache()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:21.619214Z","iopub.execute_input":"2026-08-05T19:18:21.619468Z","iopub.status.idle":"2026-08-05T19:18:55.979261Z","shell.execute_reply.started":"2026-08-05T19:18:21.619447Z","shell.execute_reply":"2026-08-05T19:18:55.9783Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.RUN_TRAIN:\n    from sklearn.metrics import roc_auc_score\n\n    cache_dirs = [CFG.cache_dir] + list(CFG.extra_cache_dirs)\n\n    def load_fold_targets(fold):\n        p = find_artifact(f'targets_fold{fold}.csv', CFG.teacher_dir, CFG.extra_teacher_dirs)\n        df = pd.read_csv(p)\n        required = {'StudyInstanceUID', 'fold', *TARGETS, *WEIGHT_COLS}\n        if not required.issubset(df.columns):\n            raise ValueError(f'{p} is missing {sorted(required - set(df.columns))}')\n        return df\n\n    def macro_auc(y, p, observed_mask):\n        per = {}\n        for i, c in enumerate(TARGETS):\n            m = observed_mask[:, i].astype(bool)\n            yi = y[m, i].astype(int)\n            per[c] = (np.nan if len(np.unique(yi)) < 2\n                      else roc_auc_score(yi, p[m, i]))\n        if any(not np.isfinite(v) for v in per.values()):\n            raise RuntimeError(f'Invalid per-target validation AUC: {per}')\n        return float(np.mean(list(per.values()))), per\n\n    def make_sampler(df, hard_only):\n        y = df[TARGETS].values\n        w = df[WEIGHT_COLS].values\n        if hard_only:\n            active = w >= .999\n        else:\n            active = w > 0\n        positive = (y > .6) & active\n        prevalence = positive.sum(0) / np.maximum(active.sum(0), 1)\n        rarity = np.sqrt(1 / np.maximum(prevalence, .01))\n        sw = 1 + (positive * rarity[None]).max(1)\n        sw = np.clip(sw, 1, 6)\n        return WeightedRandomSampler(torch.as_tensor(sw, dtype=torch.double),\n                                     num_samples=len(sw), replacement=True)\n\n    def make_loader(ds, batch_size, train_mode):\n        sampler = make_sampler(ds.df, ds.hard_only) if train_mode else None\n        return DataLoader(ds, batch_size=batch_size, sampler=sampler, shuffle=False,\n                          num_workers=CFG.num_workers, pin_memory=True, drop_last=False,\n                          persistent_workers=CFG.num_workers > 0)\n\n    def validation(model, loader):\n        model.eval(); P, Y, O = [], [], []\n        with torch.inference_mode():\n            for b in loader:\n                with autocast():\n                    logits, _, _, _ = model(b['x'].to(DEVICE, non_blocking=True),\n                                            b['view_mask'].to(DEVICE, non_blocking=True),\n                                            b['slice_mask'].to(DEVICE, non_blocking=True))\n                P.append(logits.float().sigmoid().cpu().numpy())\n                Y.append(b['y'].numpy())\n                O.append((b['w'].numpy() >= .999).astype(np.uint8))\n        return macro_auc(np.concatenate(Y), np.concatenate(P), np.concatenate(O))\n\n    def train_fold(fold):\n        df = load_fold_targets(fold)\n        tr = df[df.fold != fold].reset_index(drop=True)  # includes fold=-1 pseudo-only studies\n        va = df[df.fold == fold].reset_index(drop=True)\n        if not len(va):\n            raise RuntimeError(f'fold {fold} has no validation rows')\n\n        ds_mix = KneeDataset(tr, cache_dirs, train_mode=True, hard_only=False, strict=True)\n        ds_hard = KneeDataset(tr, cache_dirs, train_mode=True, hard_only=True, strict=True)\n        ds_val = KneeDataset(va, cache_dirs, train_mode=False, hard_only=False, strict=True)\n        dl_mix = make_loader(ds_mix, CFG.batch_size, True)\n        dl_hard = make_loader(ds_hard, CFG.batch_size, True)\n        dl_val = make_loader(ds_val, CFG.batch_size, False)\n\n        hard = tr[WEIGHT_COLS].values >= .999\n        y = tr[TARGETS].values\n        pos = ((y > .5) & hard).sum(0)\n        neg = ((y <= .5) & hard).sum(0)\n        pos_weight = np.sqrt((neg + 1) / (pos + 1)).clip(1, CFG.max_pos_weight)\n        pos_weight = torch.tensor(pos_weight, dtype=torch.float32, device=DEVICE)\n        print(f'\\n=== fold {fold} | mix={len(ds_mix):,} hard={len(ds_hard):,} val={len(ds_val):,} ===')\n        print('capped sqrt pos_weight:', dict(zip(TARGETS, pos_weight.cpu().numpy().round(2))))\n\n        model = KneeNet(pretrained=CFG.pretrained).to(DEVICE)\n        ema = copy.deepcopy(model).eval()\n        for p in ema.parameters():\n            p.requires_grad_(False)\n        bb = [p for n, p in model.named_parameters() if n.startswith('enc.')]\n        hd = [p for n, p in model.named_parameters() if not n.startswith('enc.')]\n        opt = torch.optim.AdamW([{'params': bb, 'lr': CFG.backbone_lr},\n                                 {'params': hd, 'lr': CFG.lr}], weight_decay=CFG.wd)\n\n        loaders = [dl_mix] * CFG.mixed_epochs + [dl_hard] * CFG.hard_epochs\n        updates_per_epoch = [math.ceil(len(dl) / CFG.accum) for dl in loaders]\n        total_updates = sum(updates_per_epoch)\n        sched = torch.optim.lr_scheduler.OneCycleLR(\n            opt, max_lr=[CFG.backbone_lr, CFG.lr], total_steps=total_updates,\n            pct_start=.10, div_factor=10, final_div_factor=100)\n        scaler = make_scaler()\n        history, best, update_count = [], -np.inf, 0\n\n        for epoch, dl in enumerate(loaders):\n            model.train(); opt.zero_grad(set_to_none=True)\n            running, n_batches, t0 = 0.0, 0, time.time()\n            group_start = 0\n            for it, b in enumerate(dl):\n                # Scale by the actual final accumulation group size, not always CFG.accum.\n                group_end = min(group_start + CFG.accum, len(dl))\n                group_size = group_end - group_start\n                x = b['x'].to(DEVICE, non_blocking=True)\n                vm = b['view_mask'].to(DEVICE, non_blocking=True)\n                sm = b['slice_mask'].to(DEVICE, non_blocking=True)\n                yy = b['y'].to(DEVICE, non_blocking=True)\n                ww = b['w'].to(DEVICE, non_blocking=True)\n                with autocast():\n                    logits, aux, _, _ = model(x, vm, sm)\n                    loss, main_loss, aux_loss = masked_multitask_loss(\n                        logits, aux, yy, ww, vm, pos_weight)\n                    scaled_loss = loss / group_size\n                scaler.scale(scaled_loss).backward()\n                running += float(loss.detach()); n_batches += 1\n\n                do_step = (it + 1 == group_end)\n                if do_step:\n                    scaler.unscale_(opt)\n                    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n                    scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True)\n                    sched.step(); update_count += 1\n                    ema_update(ema, model, CFG.ema_decay, update_count)\n                    group_start = it + 1\n\n            auc, per = validation(ema, dl_val)\n            row = {'epoch': epoch, 'phase': 'mixed' if epoch < CFG.mixed_epochs else 'hard',\n                   'loss': running / max(n_batches, 1), 'auc': auc, **per}\n            history.append(row)\n            print(f'epoch {epoch:02d} {row[\"phase\"]:<5} | loss={row[\"loss\"]:.4f} '\n                  f'| macro AUC={auc:.4f} | {time.time()-t0:.0f}s', flush=True)\n            if auc > best:\n                best = auc\n                payload = {\n                    'state_dict': ema.state_dict(), 'backbone': BACKBONE,\n                    'views': CFG.VIEWS, 'targets': TARGETS,\n                    'img_size': CFG.IMG_SIZE, 'n_slices': CFG.N_SLICES,\n                    'cache_version': CFG.CACHE_VERSION, 'fold': fold,\n                    'epoch': epoch, 'val_auc': auc, 'per_target_auc': per,\n                }\n                torch.save(payload, CFG.ckpt_dir / f'knee_cv2_f{fold}.pt')\n\n        hist = pd.DataFrame(history)\n        hist.to_csv(CFG.ckpt_dir / f'history_f{fold}.csv', index=False)\n        print(f'fold {fold} best macro AUC: {best:.4f}')\n        del model, ema, opt, scaler, dl_mix, dl_hard, dl_val\n        gc.collect(); torch.cuda.empty_cache()\n        return best\n\n    train_folds = list(range(CFG.folds)) if CFG.TRAIN_FOLDS is None else list(CFG.TRAIN_FOLDS)\n    scores = [train_fold(f) for f in train_folds]\n    print('CV folds:', train_folds, '| scores:', np.round(scores, 4),\n          '| mean:', float(np.mean(scores)))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:55.981458Z","iopub.execute_input":"2026-08-05T19:18:55.982118Z","iopub.status.idle":"2026-08-05T19:18:56.006397Z","shell.execute_reply.started":"2026-08-05T19:18:55.982086Z","shell.execute_reply":"2026-08-05T19:18:56.005691Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5.0 Protocol-metadata fallback — a scorable submission before any GPU training\n\n`test_series.csv` ships with the test set, so plane / fluid-sensitivity / fat-suppression\ncounts are legitimately available at inference time. Which sequences a radiographer\nacquired is not independent of the clinical question, so these descriptors carry a small\nbut real signal, and the model costs a couple of CPU minutes with no weights to attach.\n\nTwo honest caveats: the folds below are stratified, **not** grouped by patient/report, so\nthe printed OOF AUC is optimistic; and a target whose cross-validated AUC does not clear\n0.53 is left at 0.5 rather than shipped. Treat this as the floor the image model has to\nbeat, not as a solution.\n","metadata":{}},{"cell_type":"code","source":"if CFG.RUN_INFER and CFG.USE_METADATA_FALLBACK:\n    from sklearn.ensemble import HistGradientBoostingClassifier\n    from sklearn.model_selection import StratifiedKFold\n    from sklearn.metrics import roc_auc_score\n\n    PLANES = ['Sagittal', 'Coronal', 'Axial']\n\n    def protocol_features(series_df):\n        g = series_df.copy()\n        g['Fluid_Sensitive'] = pd.to_numeric(g.Fluid_Sensitive, errors='coerce').fillna(0).astype(int)\n        g['Fat_Suppression'] = pd.to_numeric(g.Fat_Suppression, errors='coerce').fillna(0).astype(int)\n        g['Anatomical_Plane'] = g.Anatomical_Plane.astype(str).str.strip().str.title()\n        recs = {}\n        for study, s in g.groupby('StudyInstanceUID', sort=False):\n            f = {'n_series': len(s),\n                 'n_fluid': int((s.Fluid_Sensitive == 1).sum()),\n                 'n_fatsup': int((s.Fat_Suppression == 1).sum()),\n                 'frac_fluid': float((s.Fluid_Sensitive == 1).mean()),\n                 'frac_fatsup': float((s.Fat_Suppression == 1).mean())}\n            for p in PLANES:\n                sp = s[s.Anatomical_Plane == p]\n                f[f'n_{p}'] = len(sp)\n                f[f'n_{p}_fluid'] = int((sp.Fluid_Sensitive == 1).sum())\n                f[f'n_{p}_fat'] = int((sp.Fat_Suppression == 1).sum())\n                f[f'n_{p}_fluid_fat'] = int(((sp.Fluid_Sensitive == 1) &\n                                             (sp.Fat_Suppression == 1)).sum())\n            recs[study] = f\n        out = pd.DataFrame.from_dict(recs, orient='index')\n        out['n_other_plane'] = out.n_series - out[[f'n_{p}' for p in PLANES]].sum(axis=1)\n        return out.astype(np.float32)\n\n    try:\n        Xtr_meta = protocol_features(train_series).reindex(train.StudyInstanceUID).fillna(0.0)\n        Xte_meta = protocol_features(test_series).reindex(SUB_INDEX.StudyInstanceUID).fillna(0.0)\n        Xte_meta = Xte_meta[Xtr_meta.columns]\n        print('protocol features:', Xtr_meta.shape[1], '| test rows:', len(Xte_meta))\n\n        meta_pred = pd.DataFrame(0.5, index=pd.Index(SUB_INDEX.StudyInstanceUID),\n                                 columns=TARGETS, dtype=np.float32)\n        rows = []\n        for c in TARGETS:\n            m = train[c].notna().values\n            y = train.loc[m, c].astype(int).values\n            Xc = Xtr_meta.values[m]\n            if len(np.unique(y)) < 2 or int(np.bincount(y, minlength=2).min()) < 25:\n                rows.append({'target': c, 'n': int(m.sum()), 'n_pos': int(y.sum()),\n                             'oof_auc': np.nan, 'used': False})\n                continue\n            oof = np.zeros(len(y), np.float32)\n            te = np.zeros(len(Xte_meta), np.float32)\n            skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=CFG.seed)\n            for tr_i, va_i in skf.split(Xc, y):\n                clf = HistGradientBoostingClassifier(\n                    max_iter=300, learning_rate=0.06, max_leaf_nodes=15,\n                    min_samples_leaf=40, l2_regularization=1.0,\n                    early_stopping=True, validation_fraction=0.15,\n                    random_state=CFG.seed)\n                clf.fit(Xc[tr_i], y[tr_i])\n                oof[va_i] = clf.predict_proba(Xc[va_i])[:, 1]\n                te += clf.predict_proba(Xte_meta.values)[:, 1] / skf.get_n_splits()\n            auc = float(roc_auc_score(y, oof))\n            used = auc > 0.53          # do not ship noise as if it were signal\n            if used:\n                meta_pred[c] = te\n            rows.append({'target': c, 'n': int(m.sum()), 'n_pos': int(y.sum()),\n                         'oof_auc': auc, 'used': used})\n\n        meta_oof = pd.DataFrame(rows).sort_values('oof_auc', ascending=False)\n        # Write first, pretty-print second: a rendering problem must never cost the file.\n        submission = write_submission(meta_pred.reset_index(names='StudyInstanceUID'),\n                                      '(protocol-metadata fallback)')\n        print(f'metadata-only macro AUC (stratified OOF, optimistic): '\n              f'{meta_oof.oof_auc.mean(skipna=True):.4f} over '\n              f'{int(meta_oof.used.sum())}/{len(TARGETS)} usable targets')\n        try:\n            display(meta_oof.style.format({'oof_auc': '{:.3f}'}))\n        except Exception:\n            print(meta_oof.round(3).to_string(index=False))\n    except Exception as e:\n        print(f'[error] metadata fallback failed ({type(e).__name__}: {e}); '\n              'the 0.5 safety-net submission is kept.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:18:56.007444Z","iopub.execute_input":"2026-08-05T19:18:56.007683Z","iopub.status.idle":"2026-08-05T19:19:07.178425Z","shell.execute_reply.started":"2026-08-05T19:18:56.007664Z","shell.execute_reply":"2026-08-05T19:19:07.177809Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Part 5 — inference and submission\n\nAll fold models are loaded together and each decoded test batch is passed through them sequentially.\nThis avoids decoding every DICOM once per checkpoint. Fold outputs are percentile-ranked per target\nbefore averaging, matching the rank-based metric while reducing calibration differences.\n\nThe cell validates checkpoint preprocessing metadata, requires at least one checkpoint, requires every\nstudy to have a decoded view, and refuses non-finite or constant target predictions.\n","metadata":{}},{"cell_type":"code","source":"# ===================================================================================\n# Image-model inference. This cell can only IMPROVE the submission that already exists\n# on disk: every failure path leaves the previous file in place instead of raising and\n# leaving Kaggle with nothing to score.\n# ===================================================================================\nif CFG.RUN_INFER:\n    import traceback\n\n    class TestDataset(Dataset):\n        \"\"\"Reports ok=0 instead of raising, so one undecodable study cannot kill the\n        other ~1300 and take the whole submission with it.\"\"\"\n\n        def __init__(self, manifest_df, root):\n            self.df = manifest_df.reset_index(drop=True)\n            self.root = Path(root)\n\n        def __len__(self):\n            return len(self.df)\n\n        def __getitem__(self, i):\n            r = self.df.iloc[i]\n            vol = np.zeros((CFG.N_VIEWS, CFG.N_SLICES, CFG.IMG_SIZE, CFG.IMG_SIZE), np.uint8)\n            vm = np.zeros(CFG.N_VIEWS, np.float32)\n            sm = np.zeros((CFG.N_VIEWS, CFG.N_SLICES), np.float32)\n            for vi, slot in enumerate(CFG.VIEWS):\n                sid = r.get(slot)\n                if not isinstance(sid, str) or not sid:\n                    continue\n                try:\n                    v, s = load_series_volume(self.root / str(r.StudyInstanceUID) / sid,\n                                              VIEW_SPEC[slot]['plane'])\n                except Exception:\n                    v, s = None, None\n                if v is not None:\n                    vol[vi], sm[vi], vm[vi] = v, s, 1\n            ok = int(bool(vm.any()))\n            if not ok:\n                # Keep the tensor well formed so masked attention cannot produce NaN;\n                # the row is excluded from the ensemble by row_ok below.\n                vm[0], sm[0, 0] = 1.0, 1.0\n            return {'x': torch.from_numpy(vol.astype(np.float32) / 255.0),\n                    'view_mask': torch.from_numpy(vm),\n                    'slice_mask': torch.from_numpy(sm),\n                    'idx': i, 'ok': ok}\n\n    def _rank01(a):\n        \"\"\"Per-target percentile rank. AUC is rank-based, so this removes calibration\n        differences between folds before averaging.\"\"\"\n        return pd.DataFrame(np.asarray(a, np.float64)).rank(method='average',\n                                                            pct=True).to_numpy(np.float64)\n\n    try:\n        test_manifest = test.merge(\n            build_manifest(test_series, CFG.comp_dir / 'test_series', count_files=True),\n            on='StudyInstanceUID', how='left', validate='one_to_one')\n        n_noview = int(test_manifest[CFG.VIEWS].isna().all(axis=1).sum())\n        if n_noview:\n            print(f'[warn] {n_noview} test studies have no selectable series; '\n                  'they keep the fallback prediction')\n\n        ckpt_paths = []\n        for d in [CFG.ckpt_dir] + list(CFG.extra_ckpt_dirs):\n            if Path(d).exists():\n                ckpt_paths.extend(Path(d).rglob('knee_cv2_f*.pt'))\n        ckpt_paths = sorted(set(ckpt_paths))\n        if not ckpt_paths:\n            raise FileNotFoundError(\n                'no knee_cv2_f*.pt checkpoints found. Run STAGE=\"train\", publish '\n                '/kaggle/working/rsna_models_v2 as a Kaggle Dataset, attach it, and list '\n                'it in CFG.extra_ckpt_dirs.')\n        print('checkpoints:', [str(x) for x in ckpt_paths])\n\n        models, seen_folds = [], set()\n        for path in ckpt_paths:\n            try:\n                ck = torch.load(path, map_location='cpu', weights_only=False)\n            except TypeError:  # torch < 2.0\n                ck = torch.load(path, map_location='cpu')\n            expected = {'views': CFG.VIEWS, 'targets': TARGETS, 'img_size': CFG.IMG_SIZE,\n                        'n_slices': CFG.N_SLICES, 'cache_version': CFG.CACHE_VERSION}\n            mismatch = {k: (ck.get(k), v) for k, v in expected.items() if ck.get(k) != v}\n            if mismatch:\n                print(f'[warn] skipping {path}: preprocessing mismatch {mismatch}')\n                continue\n            if ck.get('fold') in seen_folds:\n                print(f'[warn] skipping duplicate fold-{ck.get(\"fold\")} checkpoint: {path}')\n                continue\n            seen_folds.add(ck.get('fold'))\n            model = KneeNet(backbone=ck['backbone'], pretrained=False).to(DEVICE)\n            model.load_state_dict(ck['state_dict'], strict=True)\n            model.eval()\n            models.append(model)\n        if not models:\n            raise RuntimeError('every checkpoint was rejected; nothing to run')\n        if len(models) < CFG.folds:\n            print(f'[warning] {len(models)}/{CFG.folds} folds loaded; train all folds '\n                  'for the final entry')\n        print(f'loaded {len(models)} models | GPU weights: '\n              f'{torch.cuda.memory_allocated() / 2 ** 30:.2f} GiB')\n\n        ds = TestDataset(test_manifest, CFG.comp_dir / 'test_series')\n        dl = DataLoader(ds, batch_size=CFG.infer_batch_size, shuffle=False,\n                        num_workers=CFG.num_workers, pin_memory=True,\n                        persistent_workers=CFG.num_workers > 0)\n        fold_pred = np.full((len(models), len(ds), len(TARGETS)), 0.5, np.float32)\n        row_ok = np.zeros(len(ds), bool)\n        t0 = time.time()\n        with torch.inference_mode():\n            for step, b in enumerate(dl):\n                x = b['x'].to(DEVICE, non_blocking=True)\n                vm = b['view_mask'].to(DEVICE, non_blocking=True)\n                sm = b['slice_mask'].to(DEVICE, non_blocking=True)\n                idx = b['idx'].numpy()\n                for mi, model in enumerate(models):\n                    with autocast():\n                        logits, _, _, _ = model(x, vm, sm)\n                    p = logits.float().sigmoid().cpu().numpy()\n                    if CFG.TTA_HFLIP:\n                        with autocast():\n                            lf, _, _, _ = model(torch.flip(x, dims=[-1]), vm, sm)\n                        p = 0.5 * (p + lf.float().sigmoid().cpu().numpy())\n                    fold_pred[mi, idx] = p\n                row_ok[idx] = b['ok'].numpy().astype(bool)\n                if (step + 1) % 25 == 0:\n                    el = time.time() - t0\n                    done = min((step + 1) * CFG.infer_batch_size, len(ds))\n                    print(f'{done}/{len(ds)} | {el / 60:.1f} min | '\n                          f'ETA {el / done * (len(ds) - done) / 60:.1f} min', flush=True)\n\n        if not row_ok.any():\n            raise RuntimeError('no test study produced a decodable view')\n\n        ranked = np.stack([_rank01(np.nan_to_num(p, nan=0.5, posinf=1.0, neginf=0.0))\n                           for p in fold_pred]).mean(0)\n        img_df = pd.DataFrame(ranked, columns=TARGETS)\n        img_df.insert(0, 'StudyInstanceUID', ds.df.StudyInstanceUID.values)\n        img_df = img_df[row_ok].set_index('StudyInstanceUID')\n\n        # Start from the best fallback available, put it on the same rank scale, then\n        # overwrite the studies the image model could actually read.\n        if 'meta_pred' in globals():\n            final = meta_pred.copy()\n        else:\n            final = pd.DataFrame(0.5, index=pd.Index(SUB_INDEX.StudyInstanceUID),\n                                 columns=TARGETS)\n        # float64 throughout: assigning float64 into a float32 column raises in\n        # pandas 3 and warns in pandas 2, which would abort the run at the last step.\n        final = final[TARGETS].astype(np.float64)\n        final[TARGETS] = _rank01(final[TARGETS].to_numpy(np.float64))\n        common = final.index.intersection(img_df.index)\n        final.loc[common, TARGETS] = img_df.loc[common, TARGETS].to_numpy(np.float64)\n        final = final.reset_index(names='StudyInstanceUID')\n\n        const = [c for c in TARGETS if final[c].nunique() < 2]\n        if const:\n            print(f'[warn] constant predictions for {const} — check the checkpoints')\n        print(f'image rows used: {int(row_ok.sum())}/{len(ds)} | '\n              f'inference {(time.time() - t0) / 60:.1f} min')\n        submission = write_submission(final, '(image ensemble)')\n    except Exception:\n        traceback.print_exc()\n        print('[error] image inference did not complete; the existing submission.csv '\n              '(metadata or 0.5 baseline) is kept, so the run still scores.')\n\n# ---------------------------------------------------------------- final verification\n_check = pd.read_csv(SUB_PATH)\nassert list(_check.columns) == SUB_COLS, list(_check.columns)\nassert len(_check) == len(SUB_INDEX) and not _check[TARGETS].isna().any().any()\nprint(f'FINAL submission.csv: {_check.shape} at {SUB_PATH}')\ndisplay(_check.head())\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-05T19:19:07.179524Z","iopub.execute_input":"2026-08-05T19:19:07.179848Z","iopub.status.idle":"2026-08-05T19:19:07.313952Z","shell.execute_reply.started":"2026-08-05T19:19:07.179824Z","shell.execute_reply":"2026-08-05T19:19:07.313124Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Score-improvement ladder\n\nRun ablations against the same grouped folds; do not assume every larger model helps.\n\n1. **Make CV trustworthy first:** inspect per-target fold positives/negatives, report-teacher OOF AUC,\n   cache coverage, and left/right orientation montages.\n2. **Baseline ablation:** CV1 attention versus CV2 target-aware attention with identical cache/backbone.\n3. **Resolution/depth:** compare 256×12 with 288×16. Meniscus/OA often benefit, but measure runtime.\n4. **Teacher:** replace TF-IDF with an offline multilingual encoder, still cross-fitted by the same\n   groups. Never send competition reports to an external API.\n5. **Series MIL:** keep the top two qualifying series per slot and add a series-level attention layer.\n6. **Backbone/seed ensemble:** train a complementary ConvNeXt/MaxViT or second seed and rank-blend.\n7. **Efficiency entry:** use EfficientNetV2-B0, 192×10, fewer folds and two/three core slots; time it\n   on a full-size synthetic/test manifest because the example test has only three studies.\n\nThreshold tuning cannot improve ROC AUC. Calibration is useful only when it improves cross-model\nblending; any monotonic recalibration leaves an individual target's AUC unchanged.\n","metadata":{}}]}