{"cells":[{"cell_type":"markdown","metadata":{},"source":"# RSNA Knee Abnormality Detection — detailed baseline (v2.6)\n\nPredicts per-study probabilities for **12 knee abnormalities** from knee MRI\nDICOM series + radiology report text. Evaluation: macro average ROC-AUC.\n\nDesigned for the competition constraints:\n\n- total notebook runtime <= 9 h (GPU recommended; CPU fallback config included,\n  plus an automatic runtime budget guard)\n- **internet DISABLED at scoring** -> torch is installed from an attached\n  wheel dataset (`nururrr/rsna-torch-wheels-cu126`, torch 2.7.1+cu126 for\n  Pascal GPUs) and pretrained weights are\n  attached as a dataset (`nururrr/rsna-backbone`, dumped via `CFG.save_weights`)\n- all attached datasets must be **publicly available** (competition rule)\n- output file must be `submission.csv` with the exact header\n\n> This notebook was verified end-to-end on the real competition data (dev\n> runs on Kaggle GPUs): final config 3-fold DenseNet121 + report text on the\n> 58 fully-labeled studies, val macro AUC 0.5256 (fold AUCs 0.486/0.489/0.603,\n> weak labels dominate the signal). v2.5 added low-weight pseudo labels mined\n> from radiology reports of unlabeled studies (~76% agreement on the labeled\n> subset, 500 extra studies); v2.6 makes the environment **scoring-compliant**\n> (internet disabled: offline torch wheels + CPU fallback instead of crash).\n\n## What is inside (why)\n\n| Step | Choice | Rationale |\n|---|---|---|\n| Series selection | Sagittal + Coronal, fluid-sensitive first | ACL/MCL/menisci best on sagittal; coronal adds MCL/OA signal; fluid-sensitive (T2/PD/STIR) shows effusion, synovitis, contusion |\n| DICOM decoding | VOI LUT, MONOCHROME1 fix, rescale slope/intercept, percentile windowing | correct intensity semantics across 16 sites |\n| Physical resampling | resample by PixelSpacing to ~0.5 mm/px | normalizes anatomy scale across scanners/sites |\n| Disk cache | uint8 `.npy` per series | decode once, epoch loop becomes I/O-light |\n| Patient-aware CV | StratifiedGroupKFold on patient_id | prevents leakage when a patient has several studies |\n| Slice aggregation | attention pooling over valid slices | learn which slices matter instead of blind mean |\n| Text branch | char n-gram hashing (stateless, offline) + MLP head | uses the paired radiology report, multilingual by construction, zero pretrained dependency |\n| Pseudo labels | report-keyword weak labels on unlabeled studies (weight 0.3, NaN targets masked) | only ~58/4407 studies have full labels; report text agrees with true labels ~76% |\n| Training | EMA, warmup+cosine, grad clip, label smoothing, AMP, early stop | stable, better final weights |\n| Inference | sliding-window TTA + fold ensemble | robust probabilities |\n| Runtime budget | stop adding folds if > 8.5 h elapsed | never breach the 9 h limit |\n| Offline torch | install 2.4.1+cu118 from attached wheel dataset (`--no-index`), PyPI only as dev fallback | scoring has internet disabled; preinstalled cu128 torch lacks kernels on P100/T4-era GPUs |\n\nRun order:  config -> data -> pseudo labels -> EDA -> CV splits -> series\nselection -> preprocessing + cache -> text features -> dataset -> model ->\ntrain folds -> inference -> submission\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"import gc, glob, json, math, os, random, re, time, warnings\nfrom collections import Counter\n\n# ---- GPU check + torch reinstall (OFFLINE-SAFE) ----\n# Scoring runs with internet DISABLED, so prefer installing torch from the\n# attached wheel dataset (rsna-torch-wheels-cu126) with --no-index; the PyPI\n# fallback only works during development. Why: the preinstalled torch\n# 2.10.0+cu128 has no CUDA kernels for older GPUs (e.g. P100 sm_6.0 ->\n# \"no kernel image is available\"; PyTorch >= 2.8 dropped sm_50-sm_60 for\n# CUDA 12.8+ builds), so we downgrade to 2.7.1+cu126 - the last build with\n# Pascal kernels. Install is --no-deps: the cu126 wheel relies on the pip\n# nvidia-cu12 runtime libs the kernel already ships with preinstalled torch\n# (checked by _nvidia_libs_ready before installing).\nimport subprocess, sys\n\nTORCH_VER, TORCHVISION_VER = \"2.7.1+cu126\", \"0.22.1+cu126\"\n\ndef install_torch(use_index):\n    # Returns True on success. use_index=False -> local wheels only.\n    if use_index:\n        args = [sys.executable, \"-m\", \"pip\", \"install\", \"-q\",\n                f\"torch=={TORCH_VER}\", f\"torchvision=={TORCHVISION_VER}\",\n                \"--index-url\", \"https://download.pytorch.org/whl/cu126\"]\n    else:\n        # kaggle auto-extracts the uploaded wheels_cu126.zip; the dataset\n        # ships torch/torchvision 2.7.1+cu126 AND the exact nvidia-cu12\n        # dependency wheels (env images drop some of their libs, e.g.\n        # libcusparseLt.so.0), so --no-deps install of all of them is safe.\n        py_tag = f\"cp{sys.version_info.major}{sys.version_info.minor}\"\n        whls = sorted(glob.glob(\n            \"/kaggle/input/**/wheels_cu126/*.whl\", recursive=True))\n        whls = [w for w in whls\n                if w.split(\"/\")[-1].startswith((\"torch-\", \"torchvision-\",\n                                                \"nvidia_\"))\n                and (f\"-{py_tag}-\" in w or \"py3-none-\" in w)]\n        if not whls:\n            print(\"  (no wheels found under wheels_cu126/ in attached datasets)\")\n            return False\n        args = [sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"--no-index\",\n                \"--no-deps\", *whls]\n    return subprocess.run(args).returncode == 0\n\ngpu = subprocess.run([\"nvidia-smi\", \"--query-gpu=name,compute_cap\",\n                      \"--format=csv,noheader\"], capture_output=True, text=True)\nreinstalled = False\nif gpu.returncode == 0 and gpu.stdout.strip():\n    name_s, cap_s = gpu.stdout.strip().splitlines()[0].rsplit(\",\", 1)\n    cap = tuple(int(x) for x in cap_s.strip().split(\".\"))\n    print(f\"GPU: {name_s.strip()} | compute capability: {cap[0]}.{cap[1]}\")\n    if cap[0] < 8:\n        if install_torch(use_index=False):\n            print(f\"installed torch {TORCH_VER} from attached wheel dataset\")\n            reinstalled = True\n        elif install_torch(use_index=True):\n            print(f\"installed torch {TORCH_VER} from PyPI (needs internet)\")\n            reinstalled = True\n        else:\n            print(\"WARNING: torch reinstall failed - continuing with preinstalled torch\")\nelse:\n    print(\"GPU: nvidia-smi unavailable\")\n\n# after a reinstall the kernel's LD_LIBRARY_PATH may miss some nvidia-cu12\n# lib dirs (e.g. libcusparseLt.so.0 from nvidia-cusparselt-cu12), so expose\n# them all before importing torch\nif reinstalled:\n    nvdirs = sorted(glob.glob(\n        \"/usr/local/lib/python3.12/dist-packages/nvidia/*/lib\"))\n    if nvdirs:\n        os.environ[\"LD_LIBRARY_PATH\"] = \":\".join(\n            nvdirs + [os.environ.get(\"LD_LIBRARY_PATH\", \"\")])\n        print(f\"LD_LIBRARY_PATH extended with {len(nvdirs)} nvidia lib dirs\")\n    # some kernel images also drop the .so soname symlinks of the nvidia-cu12\n    # packages (e.g. libcusparseLt.so.0); recreate them from the real files\n    made = 0\n    for d in nvdirs:\n        for f in sorted(glob.glob(f\"{d}/*.so.*\")):\n            b = os.path.basename(f)\n            s = b.rsplit(\".\", 2)[0]\n            if (b.startswith(\"lib\") and s != b\n                    and not os.path.exists(os.path.join(d, s))):\n                try:\n                    os.symlink(b, os.path.join(d, s))\n                    made += 1\n                except OSError:\n                    pass\n    if made:\n        print(f\"created {made} missing nvidia .so soname symlinks\")\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport pydicom\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import StratifiedKFold, KFold\n\ntry:\n    from sklearn.model_selection import StratifiedGroupKFold\n    HAS_SGKF = True\nexcept ImportError:\n    HAS_SGKF = False\n\nwarnings.filterwarnings(\"ignore\")\n\nSTART_TIME = time.time()\nSEED = 42\n\n\ndef seed_everything(seed=SEED):\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n\n\nseed_everything()\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"PyTorch:\", torch.__version__, \"| device:\", DEVICE)\nif DEVICE == \"cuda\":\n    try:\n        x = torch.ones(4, device=DEVICE)\n        (x + x).cpu()\n        torch.cuda.synchronize()\n        print(\"CUDA kernel check: OK\")\n    except Exception as e:\n        print(f\"CUDA kernel check FAILED ({e})\")\n        print(\"-> falling back to CPU (slow; config will be shrunk)\")\n        DEVICE = \"cpu\"\n\n\nclass CFG:\n    # ---- data ----\n    # Kaggle mounts competition data either as /kaggle/input/<slug> or\n    # /kaggle/input/competitions/<slug> - pick whichever is present.\n    data_dir  = next((p for p in (\"/kaggle/input/rsna-knee-abnormality-detection\",\n                                  \"/kaggle/input/competitions/rsna-knee-abnormality-detection\")\n                      if os.path.exists(os.path.join(p, \"train.csv\"))),\n                     \"/kaggle/input/rsna-knee-abnormality-detection\")\n    train_dir = f\"{data_dir}/train_series\"\n    test_dir  = f\"{data_dir}/test_series\"\n    out_dir   = \"/kaggle/working\"\n    cache_dir = \"/kaggle/working/cache\"\n\n    # ---- model ----\n    backbone        = \"densenet121\"          # densenet121 | efficientnet_b0\n    # attached backbone weights (offline scoring): auto-detected from the\n    # rsna-backbone dataset; None (random init) only if nothing is attached\n    pretrained_path = next(iter(sorted(glob.glob(\n        \"/kaggle/input/rsna-backbone/**/*.pth\", recursive=True)) or\n        glob.glob(\"/kaggle/input/**/densenet121_backbone.pth\", recursive=True) or []),\n        None)\n    save_weights    = False                  # DEV ONLY: dump ImageNet backbone to out_dir\n    use_compile     = False                  # torch.compile (GPU, optional speedup)\n    use_text        = True                   # radiology report branch\n\n    # ---- preprocessing ----\n    img_size         = 224\n    target_px_mm     = 0.5                   # physical pixel target for resampling\n    n_slices         = 24                    # slices sampled per series\n    max_slices_cache = 48                    # slices kept in the on-disk cache\n    n_series         = 2                     # series per study (1 = sagittal, 2 = + coronal)\n\n    # ---- training ----\n    folds           = 3\n    epochs          = 4\n    freeze_epochs   = 1                      # head-only first epoch(s)\n    patience        = 2                      # early stopping on val macro AUC\n    ema_decay       = 0.99\n    lr_head         = 1e-3\n    lr_body         = 3e-5\n    weight_decay    = 1e-2\n    warmup_frac     = 0.05\n    grad_clip       = 1.0\n    label_smoothing = 0.02\n    batch_size      = 4                      # samples per step (each = n_slices images)\n    frames_chunk    = 64                     # max frames per GPU conv chunk (OOM guard)\n    num_workers     = 2\n    use_amp         = torch.cuda.is_available()\n\n    # ---- inference / limits ----\n    tta_grids    = 3 if torch.cuda.is_available() else 1\n    budget_hours = 8.5                       # skip further folds beyond this\n    subset       = 1.0                       # fraction of labeled studies (quick tests)\n\n    # ---- pseudo labels (weak labels from radiology reports) ----\n    use_pseudo    = True                     # add report-keyword labels of unlabeled studies\n    pseudo_max    = 500                      # max pseudo studies added\n    pseudo_weight = 0.3                      # loss weight for pseudo labels\n\n\nif DEVICE == \"cpu\":\n    # CPU fallback (e.g. scoring without usable CUDA): shrink the workload so\n    # a valid submission still fits the 9 h budget (weak, but not a crash).\n    print(\"CPU mode: shrinking config (n_series=1, batch=2, folds=1, epochs=1, TTA=1)\")\n    CFG.n_series   = 1\n    CFG.folds      = 1\n    CFG.epochs     = 1\n    CFG.batch_size = 2\n    CFG.tta_grids  = 1\n    CFG.use_amp    = False\n\n\nIMAGENET_MEAN = (0.485, 0.456, 0.406)\nIMAGENET_STD = (0.229, 0.224, 0.225)\n\nos.makedirs(CFG.cache_dir, exist_ok=True)\nos.makedirs(CFG.out_dir, exist_ok=True)\ntorch.backends.cudnn.benchmark = False\ntorch.backends.cudnn.enabled = True\ntorch.set_num_threads(min(os.cpu_count() or 4, 8))\n\n\ndef elapsed_hours():\n    return (time.time() - START_TIME) / 3600.0\n\n\ndef free_memory():\n    gc.collect()\n    if torch.cuda.is_available():\n        torch.cuda.empty_cache()\n\n\nprint(f\"config: backbone={CFG.backbone} folds={CFG.folds} epochs={CFG.epochs} \"\n      f\"n_series={CFG.n_series} text={CFG.use_text} tta={CFG.tta_grids}\")\n"},{"cell_type":"markdown","metadata":{},"source":"## 1. Data overview\n\n`train.csv`: one row per study (`StudyInstanceUID`, `PatientSex`, free-text\n`Report`, 12 binary labels). **Only a subset of training studies carries\nlabels** — we train on labeled rows and keep the reports for the text branch.\n`train_series.csv` / `test_series.csv`: series descriptors\n(`Fluid_Sensitive`, `Fat_Suppression`, `Anatomical_Plane`). DICOM slices:\n`train_series/<Study>/<Series>/*.dcm`."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"train_df   = pd.read_csv(f\"{CFG.data_dir}/train.csv\")\ntrain_ser  = pd.read_csv(f\"{CFG.data_dir}/train_series.csv\")\ntest_ser   = pd.read_csv(f\"{CFG.data_dir}/test_series.csv\")\nsample_sub = pd.read_csv(f\"{CFG.data_dir}/sample_submission.csv\")\n\nTARGETS = [c for c in sample_sub.columns if c != \"StudyInstanceUID\"]\nassert len(TARGETS) == 12, TARGETS\n\nlabeled = train_df.dropna(subset=TARGETS).copy()\nlabeled = labeled.drop_duplicates(subset=\"StudyInstanceUID\").reset_index(drop=True)\n\n# subset MUST be applied here (before CV splits are built), otherwise the\n# fold indices computed later would point at the wrong rows\nif CFG.subset < 1.0:\n    rng = np.random.RandomState(SEED)\n    keep = rng.choice(len(labeled), max(1, int(len(labeled) * CFG.subset)), replace=False)\n    labeled = labeled.iloc[keep].reset_index(drop=True)\n    print(f\"[subset] -> {len(labeled)} labeled studies\")\n\nprint(\"train.csv   :\", train_df.shape, \"| columns:\", list(train_df.columns))\nprint(\"labeled     :\", len(labeled), \"of\", len(train_df), \"studies\")\nprint(\"test_series :\", test_ser.shape, \"| test studies:\", test_ser.StudyInstanceUID.nunique())\nprint(\"targets     :\", TARGETS)\nprint()\nprint(\"report char-length stats (train):\")\nprint(train_df[\"Report\"].fillna(\"\").str.len().describe().round(0).to_string())\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# ---- weak labels from radiology reports (multi-language keywords) ----\n# Only ~58 of 4407 train studies carry all 12 labels; the rest are all-NaN.\n# Keyword rules over the Report text give noisy (but useful) labels, verified\n# below against the true labels (~76% agreement). Used as low-weight\n# auxiliary training targets; undecided targets are masked from the loss.\nPSEUDO_RULES = {\n    \"ACL\": (\n        [r\"anterior cruciate\", r\"\\bacl\\b\", r\"cruzado anterior\",\n         r\"vordere[sn]? kreuzband\", r\"crois[ée] ant[ée]rieur\", r\"çapraz ba\"],\n        [r\"intact\", r\"preserv\", r\"normal\", r\"unremarkable\", r\"without\",\n         r\"no tear\", r\"not torn\", r\"sin rotura\", r\"no rotura\", r\"no rasg\",\n         r\"no hay\", r\"nicht\", r\"keine\", r\"aucune\", r\"sans\", r\"no evidencia\",\n         r\"no evid\", r\"denied\", r\"rule[ds]? out\", r\"excluded\", r\"no increased intensity\"],\n    ),\n    \"MCL\": (\n        [r\"medial collateral\", r\"\\bmcl\\b\", r\"colateral medial\",\n         r\"inner[es] seitenband\", r\"collat[ée]ral m[ée]dial\"],\n        [r\"intact\", r\"preserv\", r\"normal\", r\"unremarkable\", r\"without\",\n         r\"no tear\", r\"not torn\", r\"no rasg\", r\"no hay\", r\"nicht\", r\"keine\",\n         r\"aucune\", r\"sans\", r\"denied\", r\"excluded\", r\"no increased intensity\"],\n    ),\n    \"Medial Meniscus\": (\n        [r\"medial menisc\", r\"menisco interno\", r\"medial meniscal\",\n         r\"innenmeniskus\", r\"m[ée]nisque interne\"],\n        [r\"intact\", r\"preserv\", r\"normal\", r\"unremarkable\", r\"without\",\n         r\"no tear\", r\"not torn\", r\"sin rotura\", r\"no rotura\", r\"no rasg\",\n         r\"no hay\", r\"nicht\", r\"keine\", r\"aucune\", r\"sans\", r\"denied\", r\"excluded\"],\n    ),\n    \"Lateral Meniscus\": (\n        [r\"lateral menisc\", r\"menisco externo\", r\"lateral meniscal\",\n         r\"au[ss]enmeniskus\", r\"m[ée]nisque externe\"],\n        [r\"intact\", r\"preserv\", r\"normal\", r\"unremarkable\", r\"without\",\n         r\"no tear\", r\"not torn\", r\"sin rotura\", r\"no rotura\", r\"no rasg\",\n         r\"no hay\", r\"nicht\", r\"keine\", r\"aucune\", r\"sans\", r\"denied\", r\"excluded\"],\n    ),\n    \"Medial OA\": (\n        [r\"medial (compartment|comparti\\w*|femorotibial|joint space|tibiofemoral).{0,60}(narrow|arthros|gonarthros|osteophyt|cartilage loss|chondral loss|degenerat|spur|sclerosis)\",\n         r\"(medial|m[ée]diale) (gon)?arthros\\w*\", r\"medial osteoarthritis\",\n         r\"tricompartmental|all three compartments|all 3 compartments\"],\n        [r\"normal\", r\"unremarkable\", r\"preserved\", r\"no narrowing\",\n         r\"no joint space\", r\"no osteophyt\", r\"no spur\",\n         r\"without .{0,30}(narrow|arthros)\", r\"no .{0,30}(arthros|narrow)\",\n         r\"sin .{0,30}(artrosis|estrech|osteofit)\", r\"no .{0,30}osteoarthritis\",\n         r\"intact\", r\"no .{0,30}gonarthrose\", r\"no hay .{0,30}artrosis\"],\n    ),\n    \"Lateral OA\": (\n        [r\"lateral (compartment|comparti\\w*|femorotibial|joint space|tibiofemoral).{0,60}(narrow|arthros|gonarthros|osteophyt|cartilage loss|chondral loss|degenerat|spur|sclerosis)\",\n         r\"(lateral|l[ée]g[ée]re) (gon)?arthros\\w*\", r\"lateral osteoarthritis\",\n         r\"tricompartmental|all three compartments|all 3 compartments\"],\n        [r\"normal\", r\"unremarkable\", r\"preserved\", r\"no narrowing\",\n         r\"no joint space\", r\"no osteophyt\", r\"no spur\",\n         r\"without .{0,30}(narrow|arthros)\", r\"no .{0,30}(arthros|narrow)\",\n         r\"sin .{0,30}(artrosis|estrech|osteofit)\", r\"no .{0,30}osteoarthritis\",\n         r\"intact\", r\"no .{0,30}gonarthrose\"],\n    ),\n    \"PF OA\": (\n        [r\"patellofemoral.{0,60}(arthros|chondromalacia|narrow|osteophyt|cartilage loss|chondral loss|degenerat|spur|sclerosis|cliff)\",\n         r\"tricompartmental|all three compartments|all 3 compartments\",\n         r\"retropatell\\w* (arthrose|artrose|narrow|degenerat)\",\n         r\"(chondromalacia|cartilage loss|chondral loss|narrow).{0,40}patellofemoral\"],\n        [r\"normal\", r\"unremarkable\", r\"congru\", r\"preserved\", r\"intact\",\n         r\"no .{0,30}(arthros|osteoarthritis|chondr|narrow)\", r\"without .{0,30}(arthros|chondr|narrow)\",\n         r\"sin .{0,30}(artrosis|condr|estrech)\", r\"no .{0,30}gonarthrose\",\n         r\"no hay .{0,30}(artrosis|condr)\"],\n    ),\n    \"Effusion\": (\n        [r\"effusion\", r\"derrame\", r\"erguss\", r\"[ée]panchement\", r\"efusi[oó]n\",\n         r\"efüzyon\", r\"joint fluid\", r\"articular fluid\", r\"fluid accumulation\",\n         r\"hidrartrosis\", r\"hydrarthrosis\", r\"effusie\", r\"synovial fluid\"],\n        [r\"no .{0,25}effusion\", r\"no .{0,25}derrame\", r\"no .{0,25}erguss\",\n         r\"no .{0,25}[ée]panchement\", r\"no joint fluid\", r\"no fluid\",\n         r\"sin derrame\", r\"without .{0,25}effusion\", r\"dry joint\",\n         r\"no .{0,25}efusi\", r\"keine .{0,25}erguss\", r\"nicht .{0,25}erguss\"],\n    ),\n    \"Synovitis\": (\n        [r\"synovitis\", r\"sinovitis\", r\"synovial\", r\"pannus\", r\"sinovial\"],\n        [r\"no .{0,25}synovitis\", r\"no .{0,25}sinovitis\", r\"no .{0,25}synovial\",\n         r\"no synovial\", r\"without .{0,25}synovial\", r\"normal synovial\",\n         r\"sin sinovitis\", r\"no sinovial\", r\"no .{0,25}sinovial\",\n         r\"keine .{0,25}synovitis\", r\"nicht .{0,25}synovitis\"],\n    ),\n    \"Baker's\": (\n        [r\"baker\", r\"popliteal cyst\", r\"popliteus cyst\", r\"poplitea cyst\",\n         r\"quiste popl\", r\"poplitealzyste\", r\"kyste popl\", r\"bakers cyste\"],\n        [r\"no .{0,25}baker\", r\"no .{0,25}poplit\", r\"sin .{0,25}quiste\",\n         r\"no cyst\", r\"without .{0,25}cyst\", r\"no .{0,25}cyst\", r\"keine .{0,25}zyste\",\n         r\"nicht .{0,25}zyste\"],\n    ),\n    \"Contusion\": (\n        [r\"contusion\", r\"contusi\", r\"bone bruise\", r\"bone contusion\",\n         r\"knochenkontusion\", r\"bone bruising\", r\"bruis[ei]ng\"],\n        [r\"no .{0,35}(contusion|bruise|bruis)\", r\"without .{0,35}(contusion|bruise)\",\n         r\"sin .{0,35}contusion\", r\"no .{0,25}contusion\", r\"no bone bruise\",\n         r\"keine .{0,35}kontusion\", r\"nicht .{0,35}kontusion\",\n         r\"no hay .{0,35}contusion\"],\n    ),\n    \"Fracture\": (\n        [r\"fracture\", r\"fractura\", r\"\\bfx\\b\", r\"fractuur\", r\"fraktur\", r\"fractur\"],\n        [r\"no .{0,25}fracture\", r\"no .{0,25}fractura\", r\"no acute fracture\",\n         r\"no evidence of fracture\", r\"no evidencia de fractura\",\n         r\"no .{0,25}fraktur\", r\"nicht .{0,25}fraktur\", r\"keine .{0,25}fraktur\",\n         r\"without .{0,25}fracture\", r\"sin fractura\", r\"no .{0,25}fx\\b\"],\n    ),\n}\n\n\ndef report_label(text, pos_pats, neg_terms):\n    if not isinstance(text, str) or not text.strip():\n        return None\n    t = re.sub(r\"\\s+\", \" \", text).lower()\n    negs = []\n    for n in neg_terms:\n        for m in re.finditer(n, t):\n            negs.append((m.start(), m.end()))\n    pos_hit = 0\n    for p in pos_pats:\n        for m in re.finditer(p, t):\n            if any(n_end >= m.start() - 30 and n_start <= m.end() + 40\n                   for n_start, n_end in negs):\n                continue\n            pos_hit += 1\n    if pos_hit > 0:\n        return 1\n    return 0 if negs else None\n\n\ndef make_pseudo(df, rules):\n    out = {t: [] for t in rules}\n    for _, r in df.iterrows():\n        for t in rules:\n            out[t].append(report_label(r[\"Report\"], *rules[t]))\n    p = pd.DataFrame(out)\n    p.insert(0, \"StudyInstanceUID\", df[\"StudyInstanceUID\"].values)\n    return p\n\n\npseudo = make_pseudo(train_df, PSEUDO_RULES)\npseudo[\"n_dec\"] = pseudo[TARGETS].notna().sum(axis=1)\n\nif CFG.use_pseudo:\n    agree = pseudo.set_index(\"StudyInstanceUID\").loc[labeled.StudyInstanceUID][TARGETS]\n    y_true = labeled.set_index(\"StudyInstanceUID\")[TARGETS]\n    both = agree.notna() & y_true.notna()\n    n_dec = int(both.sum().sum())\n    eq = ((agree.to_numpy() == y_true.to_numpy()) & both.to_numpy())\n    print(\"pseudo-vs-true agreement (labeled studies):\")\n    print(f\"  overall {eq.sum() / max(n_dec, 1):.3f}  decisions {n_dec}\")\n    per = {}\n    for t in TARGETS:\n        m = both[t]\n        per[t] = (agree[t][m] == y_true[t][m]).mean() if m.any() else float(\"nan\")\n    print(\"  \" + \"  \".join(f\"{t}: {v:.2f}\" for t, v in per.items()))\n\n    pool = pseudo[(pseudo[\"n_dec\"] >= 6) & ~pseudo.StudyInstanceUID.isin(labeled.StudyInstanceUID)]\n    pool = pool.sort_values([\"n_dec\", \"StudyInstanceUID\"], ascending=[False, True])\n    pseudo_selected = pool.head(CFG.pseudo_max).copy()\n    print(f\"pseudo: {len(pseudo_selected)} studies selected \"\n          f\"({int(pseudo_selected['n_dec'].sum())} decisions) \"\n          f\"of {len(pool)} eligible\")\n    TRAIN_UIDS = list(labeled.StudyInstanceUID) + list(pseudo_selected.StudyInstanceUID)\n    tr_label = pd.concat([labeled, pseudo_selected[TARGETS + [\"StudyInstanceUID\"]]],\n                         ignore_index=True)\n    pseudo_w = {u: CFG.pseudo_weight for u in pseudo_selected.StudyInstanceUID}\nelse:\n    pseudo_selected = pd.DataFrame()\n    TRAIN_UIDS = list(labeled.StudyInstanceUID)\n    tr_label = labeled\n    pseudo_w = {}\nprint(\"train studies (labeled + pseudo):\", len(TRAIN_UIDS))\n"},{"cell_type":"markdown","metadata":{},"source":"## 2. EDA\n\nLabel prevalence, label correlations, series/plane/fluid composition, and a\nDICOM header sample (transfer syntax, pixel spacing, photometric\ninterpretation) — helps you sanity-check the data and spot preprocessing\nissues early (e.g. compressed DICOM that needs pylibjpeg)."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"print(\"== label prevalence (labeled subset) ==\")\nprev = (labeled[TARGETS] == 1).mean().sort_values(ascending=False)\nprint(prev.round(3).to_string())\nprint()\nprint(\"== label correlation (Pearson, labeled subset) ==\")\nprint(labeled[TARGETS].corr().round(2).to_string())\nprint()\nprint(\"== series composition (train) ==\")\nprint(\"series per study:\",\n      train_ser.groupby(\"StudyInstanceUID\").size().describe().round(2).to_string())\nprint()\nprint(train_ser.Anatomical_Plane.value_counts().to_string())\nprint()\nprint(\"plane x fluid-sensitive:\")\nprint(pd.crosstab(train_ser.Anatomical_Plane, train_ser.Fluid_Sensitive).to_string())\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"print(\"== DICOM header sample (up to 1200 random series, no pixel decode) ==\")\nrng = np.random.RandomState(0)\nsids = rng.choice(train_ser.SeriesInstanceUID.unique(),\n                  min(1200, train_ser.SeriesInstanceUID.nunique()), replace=False)\nn_slices, tss, pis, sps, shapes = [], Counter(), Counter(), [], []\nbad = 0\nfor sid in sids:\n    row = train_ser[train_ser.SeriesInstanceUID == sid].iloc[0]\n    d = os.path.join(CFG.train_dir, row.StudyInstanceUID, sid)\n    files = glob.glob(os.path.join(d, \"*.dcm\"))\n    n_slices.append(len(files))\n    if not files:\n        continue\n    try:\n        ds = pydicom.dcmread(files[0], stop_before_pixels=True)\n        tss[str(ds.file_meta.TransferSyntaxUID)] += 1\n        pis[str(getattr(ds, \"PhotometricInterpretation\", None))] += 1\n        sp = getattr(ds, \"PixelSpacing\", None)\n        sps.append((float(sp[0]), float(sp[1])) if sp and len(sp) >= 2 else (None, None))\n        shapes.append((int(getattr(ds, \"Rows\", 0)), int(getattr(ds, \"Columns\", 0))))\n    except Exception:\n        bad += 1\n\nn_slices = np.array(n_slices)\nprint(\"slices per series:\", \"min\", n_slices.min(), \"| median\", int(np.median(n_slices)),\n      \"| max\", n_slices.max(), \"| >100 slices:\", int((n_slices > 100).mean() * 100), \"%\")\nprint(\"unreadable sample files:\", bad)\nprint()\nprint(\"transfer syntaxes:\")\nfor ts, c in tss.most_common(6):\n    comp = \"COMPRESSED\" if ts.startswith(\"1.2.840.10008.1.2.4\") else \"uncompressed\"\n    print(f\"  {ts}  x{c}  [{comp}]\")\nprint(\"photometric:\", dict(pis))\nsps = np.array([s for s in sps if s[0] is not None])\nprint(\"pixel spacing (mm), rows/cols of sample:\",\n      sps.min(0).round(3) if len(sps) else None, \"to\",\n      sps.max(0).round(3) if len(sps) else None,\n      \"| shapes:\", sorted(set(shapes))[:10])\n"},{"cell_type":"markdown","metadata":{},"source":"## 3. Patient-aware cross-validation\n\nA patient can appear in several studies, so naive study-level folds leak.\nWe group by `patient_id` (if present) with `StratifiedGroupKFold`, stratifying\non the label count. Falls back to `StratifiedKFold` / `KFold` automatically."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"PAT_COL = \"patient_id\" if \"patient_id\" in train_df.columns else None\nprint(\"patient column detected:\", PAT_COL)\nif PAT_COL:\n    print(\"patients in train:\", labeled[PAT_COL].nunique(),\n          \"| studies/patient:\", labeled.groupby(PAT_COL).size().describe().round(2).to_string())\n    multi = (labeled.groupby(PAT_COL).size() > 1).mean()\n    print(f\"patients with >1 study: {multi * 100:.1f}%  <-- why we group by patient\")\n\ny_sum = labeled[TARGETS].sum(axis=1)\ntry:\n    bins = pd.qcut(y_sum, 4, labels=False, duplicates=\"drop\")\nexcept Exception:\n    bins = None\n\nif CFG.folds > len(labeled):\n    CFG.folds = max(1, len(labeled))\n    print(f\"folds capped to {CFG.folds} (too few labeled studies)\")\n\nif CFG.folds < 2:\n    # folds=1 quick test: patient-aware single holdout (80% train / 20% val)\n    pat = labeled[PAT_COL].values if PAT_COL else np.arange(len(labeled))\n    pats = np.unique(pat)\n    n_tr = max(1, min(len(pats) - 1, int(len(pats) * 0.8)))\n    tr_pats = set(np.random.RandomState(SEED).permutation(pats)[:n_tr])\n    tr_idx = np.where(np.isin(pat, list(tr_pats)))[0]\n    va_idx = np.where(~np.isin(pat, list(tr_pats)))[0]\n    splits = [(tr_idx, va_idx)]\n    print(\"using single holdout split (folds=1, patient-aware)\")\nelif PAT_COL and HAS_SGKF:\n    try:\n        groups = labeled[PAT_COL].values\n        skf = StratifiedGroupKFold(n_splits=CFG.folds, shuffle=True, random_state=SEED)\n        y_split = bins.values if bins is not None else np.zeros(len(labeled), dtype=int)\n        splits = list(skf.split(labeled, y_split, groups=groups))\n        print(\"using StratifiedGroupKFold (patient-aware)\")\n    except Exception as e:\n        print(\"StratifiedGroupKFold failed:\", e, \"-> falling back\")\n        if bins is not None:\n            splits = list(StratifiedKFold(n_splits=CFG.folds, shuffle=True,\n                                          random_state=SEED).split(labeled, bins))\n            print(\"using StratifiedKFold\")\n        else:\n            splits = list(KFold(n_splits=CFG.folds, shuffle=True, random_state=SEED).split(labeled))\n            print(\"using KFold\")\nelif bins is not None:\n    splits = list(StratifiedKFold(n_splits=CFG.folds, shuffle=True,\n                                  random_state=SEED).split(labeled, bins))\n    print(\"using StratifiedKFold\")\nelse:\n    splits = list(KFold(n_splits=CFG.folds, shuffle=True, random_state=SEED).split(labeled))\n    print(\"using KFold\")\n\nassert len(splits) == CFG.folds and len(splits[0][0]) > 0\nfor f, (tr, va) in enumerate(splits):\n    print(f\"  fold {f}: train studies={len(tr)} val studies={len(va)}\")\n"},{"cell_type":"markdown","metadata":{},"source":"## 4. Series selection\n\nKnee MRI sequences ranked by diagnostic value:\n\n1. **Sagittal fluid-sensitive** (PD/STIR/T2-FS): menisci, ACL, cartilage, effusion\n2. **Coronal fluid-sensitive**: MCL, medial/lateral compartments, menisci\n3. Other sagittal / coronal (T1, non-FS)\n4. Axial (patellofemoral, patella)\n\nWe take up to `CFG.n_series` series per study following this preference order.\nIf a study has no series at all it is dropped from training and predicted 0.5\nat test time."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"PLANE_PREF = [\"Sagittal\", \"Coronal\", \"Axial\"]\n\n\ndef pick_series(series_df, uid, n_series=CFG.n_series):\n    rows = series_df[series_df.StudyInstanceUID == uid]\n    if len(rows) == 0:\n        return []\n    chosen = []\n    for plane in PLANE_PREF:\n        sub = rows[rows.Anatomical_Plane == plane]\n        if len(sub) == 0:\n            continue\n        fluid = sub[sub.Fluid_Sensitive == 1]\n        sub = fluid if len(fluid) > 0 else sub\n        sub = sub.sort_values(\"Fat_Suppression\", ascending=False)\n        chosen.append(sub.iloc[0].SeriesInstanceUID)\n        if len(chosen) >= n_series:\n            break\n    return chosen\n\n\ntrain_map = {u: pick_series(train_ser, u) for u in TRAIN_UIDS}\ntest_map = {u: pick_series(test_ser, u) for u in test_ser.StudyInstanceUID.unique()}\n\nn_tr = sum(1 for v in train_map.values() if v)\nn_te = sum(1 for v in test_map.values() if v)\nprint(f\"train studies with >=1 series: {n_tr}/{len(train_map)}\")\nprint(f\"test  studies with >=1 series: {n_te}/{len(test_map)}\")\n\n\ndef plane_stats(series_df, smap, name):\n    chosen = [s for v in smap.values() for s in v]\n    df = series_df[series_df.SeriesInstanceUID.isin(chosen)]\n    print(f\"{name}: chosen series per plane:\")\n    print(df.Anatomical_Plane.value_counts().to_string())\n    print(df.Fluid_Sensitive.value_counts().rename(\"fluid\").to_string())\n\n\nplane_stats(train_ser, train_map, \"train\")\nplane_stats(test_ser, test_map, \"test\")\n"},{"cell_type":"markdown","metadata":{},"source":"## 5. Robust DICOM -> array + disk cache\n\n`read_dicom_arr` applies, in order:\n\n- `apply_voi_lut` if present (wrapped in try: some slices have none)\n- MONOCHROME1 inversion (white-on-black -> black-on-white)\n- multi-frame -> middle slice\n- rescale slope/intercept (raw stored values -> physical MR units)\n\nThen `process_series` windows each slice at percentiles 0.5–99.5, **resamples\nby pixel spacing** to a common physical scale (`target_px_mm`), and resizes to\n224x224. Results are cached as uint8 `.npy` (one file per series) so the epoch\nloop and repeated runs stay fast. Unreadable slices (compressed DICOM without\ncodec) are skipped and counted."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def read_dicom_arr(path):\n    \"\"\"-> (float32 2D array, pydicom Dataset). Applies VOI LUT, MONOCHROME1\n    inversion, multi-frame handling and rescale slope/intercept.\"\"\"\n    ds = pydicom.dcmread(path, force=True)\n    arr = ds.pixel_array\n    arr = np.squeeze(np.asarray(arr, dtype=np.float32))\n    try:\n        from pydicom.pixel_data_handlers.util import apply_voi_lut\n        arr = np.asarray(apply_voi_lut(arr, ds), dtype=np.float32)\n    except Exception:\n        pass\n    if arr.ndim != 2:                    # multiframe -> keep middle slice\n        arr = arr[arr.shape[0] // 2]\n    if getattr(ds, \"PhotometricInterpretation\", \"\") == \"MONOCHROME1\":\n        arr = arr.max() - arr\n    slope = float(getattr(ds, \"RescaleSlope\", 1) or 1)\n    intercept = float(getattr(ds, \"RescaleIntercept\", 0) or 0)\n    return arr * slope + intercept, ds\n\n\ndef normalize_window(arr, lo_p=0.5, hi_p=99.5):\n    lo, hi = np.percentile(arr, [lo_p, hi_p])\n    if hi <= lo:\n        hi = lo + 1e-6\n    arr = np.clip((arr - lo) / (hi - lo), 0.0, 1.0)\n    return (arr * 255).astype(np.uint8)\n\n\ndef resample_physical(arr, ds, target_mm=CFG.target_px_mm):\n    \"\"\"Resample so pixel spacing is ~target_mm on both axes (anatomy scale\n    consistency across sites). Falls back to direct resize if no spacing.\"\"\"\n    sp = getattr(ds, \"PixelSpacing\", None)\n    h, w = arr.shape[:2]\n    if sp and len(sp) >= 2 and float(sp[0]) > 0 and float(sp[1]) > 0:\n        new_h = int(round(h * float(sp[0]) / target_mm))\n        new_w = int(round(w * float(sp[1]) / target_mm))\n        new_h, new_w = max(64, min(new_h, 1600)), max(64, min(new_w, 1600))\n        interp = cv2.INTER_CUBIC if (new_h, new_w) > (h, w) else cv2.INTER_AREA\n        arr = cv2.resize(arr, (new_w, new_h), interpolation=interp)\n    return arr\n\n\ndef process_series(series_dir, counter):\n    files = glob.glob(os.path.join(series_dir, \"*.dcm\"))\n    if not files:\n        if counter:\n            counter[\"empty_dir\"] += 1\n        return None\n\n    def keyf(p):\n        m = re.search(r\"(\\d+)\", os.path.basename(p))\n        return int(m.group(1)) if m else 10 ** 18\n\n    files = sorted(files, key=keyf)\n    if len(files) > CFG.max_slices_cache:\n        idx = np.linspace(0, len(files) - 1, CFG.max_slices_cache).astype(int)\n        files = [files[i] for i in idx]\n    out = []\n    for f in files:\n        try:\n            arr, ds = read_dicom_arr(f)\n            arr = normalize_window(resample_physical(arr, ds))\n            arr = cv2.resize(arr, (CFG.img_size, CFG.img_size), interpolation=cv2.INTER_AREA)\n            out.append(arr)\n        except Exception:\n            if counter:\n                counter[\"unreadable\"] += 1\n            continue\n    if not out:\n        if counter:\n            counter[\"all_failed\"] += 1\n        return None\n    return np.stack(out)                 # (S,H,W) uint8\n\n\ndef get_cached(uid, sid, base_dir, force=False):\n    path = os.path.join(CFG.cache_dir, f\"{uid}__{sid}.npy\")\n    if os.path.exists(path) and not force:\n        return np.load(path)\n    arr = process_series(os.path.join(base_dir, uid, sid), None)\n    if arr is not None:\n        np.save(path, arr)\n    return arr\n\n\ndef build_cache(samples, base_dir):\n    t0 = time.time()\n    cnt = Counter()\n    for i, (uid, sid) in enumerate(samples):\n        path = os.path.join(CFG.cache_dir, f\"{uid}__{sid}.npy\")\n        if not os.path.exists(path):\n            arr = process_series(os.path.join(base_dir, uid, sid), cnt)\n            if arr is not None:\n                np.save(path, arr)\n        if (i + 1) % 200 == 0:\n            print(f\"  cached {i+1}/{len(samples)}  ({time.time()-t0:.0f}s)\")\n    if cnt:\n        print(\"  cache issues:\", dict(cnt))\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"train_samples = [(u, s) for u, sv in train_map.items() for s in sv]\ntest_samples  = [(u, s) for u, sv in test_map.items() for s in sv]\nprint(\"train samples:\", len(train_samples), \"| test samples:\", len(test_samples))\n\n# Cache version stamp: cached .npy are tied to img_size / max_slices / px_mm.\n# If the config changed, stale caches would crash with shape errors - clear them.\nos.makedirs(CFG.cache_dir, exist_ok=True)   # defensive: do not depend on cell order\nCACHE_PARAMS = {\"img_size\": CFG.img_size, \"max_slices\": CFG.max_slices_cache,\n                \"px_mm\": CFG.target_px_mm}\nmeta_path = os.path.join(CFG.cache_dir, \"cache_meta.json\")\nold_meta = {}\nif os.path.exists(meta_path):\n    try:\n        with open(meta_path, encoding=\"utf-8\") as f:\n            old_meta = json.load(f)\n    except Exception:\n        old_meta = {}\nif old_meta != CACHE_PARAMS:\n    cleared = 0\n    for p in glob.glob(os.path.join(CFG.cache_dir, \"*.npy\")):\n        os.remove(p)\n        cleared += 1\n    with open(meta_path, \"w\", encoding=\"utf-8\") as f:\n        json.dump(CACHE_PARAMS, f)\n    if cleared or not old_meta:\n        print(f\"cache prepared ({cleared} old files removed, config: {CACHE_PARAMS})\")\nelse:\n    print(\"cache version OK\")\n\nest_gb = ((len(train_samples) + len(test_samples)) * CFG.max_slices_cache\n          * CFG.img_size * CFG.img_size / 1e9)\nprint(f\"estimated cache size: {est_gb:.1f} GB \"\n      f\"({CFG.max_slices_cache} slices x {CFG.img_size}^2 uint8 per series)\")\nif est_gb > 15:\n    print(\"WARNING: cache may be large for /kaggle/working - if disk fills up,\")\n    print(\"         set CFG.max_slices_cache=32 and re-run this cell\")\n\nprint(\"caching train series...\")\nbuild_cache(train_samples, CFG.train_dir)\nprint(\"caching test series...\")\nbuild_cache(test_samples, CFG.test_dir)\n\ntrain_samples = [(u, s) for u, s in train_samples if get_cached(u, s, CFG.train_dir) is not None]\ntest_samples  = [(u, s) for u, s in test_samples  if get_cached(u, s, CFG.test_dir) is not None]\nprint(\"valid train samples:\", len(train_samples), \"| valid test samples:\", len(test_samples))\n\ncache_files = glob.glob(os.path.join(CFG.cache_dir, \"*.npy\"))\ntotal_mb = sum(os.path.getsize(p) for p in cache_files) / 1e6\nprint(f\"cache: {len(cache_files)} files, {total_mb:.0f} MB\")\n"},{"cell_type":"markdown","metadata":{},"source":"## 6. Text branch (radiology report)\n\nReports come in ~12 languages, so we avoid vocabularies: a **stateless\nchar-gram HashingVectorizer** (2–4 char n-grams, 512 dims, binary) is\nmultilingual by construction and needs no pretrained model — perfect for the\noffline constraint. The hashed vector feeds an MLP that is concatenated with\nCNN features before the final head.\n\nIf the test set has no reports (`test.csv` absent or no `Report` column),\nthe test text is set to zeros: the model still works, image-only. Set\n`CFG.use_text=False` for a pure-vision model."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"TEXT_DIM = 512\nTEXT_ON = CFG.use_text and \"Report\" in train_df.columns\nprint(\"text branch:\", \"ON\" if TEXT_ON else \"OFF\")\n\nif TEXT_ON:\n    from sklearn.feature_extraction.text import HashingVectorizer\n    vec = HashingVectorizer(n_features=TEXT_DIM, analyzer=\"char_wb\",\n                            ngram_range=(2, 4), binary=True, norm=\"l2\",\n                            alternate_sign=False)\n\n    def build_text_lookup(df):\n        rep = df[[\"StudyInstanceUID\", \"Report\"]].copy()\n        rep[\"Report\"] = rep[\"Report\"].fillna(\"\").astype(str)\n        X = vec.transform(rep[\"Report\"].values).toarray().astype(np.float32)\n        return {uid: X[i] for i, uid in enumerate(rep.StudyInstanceUID)}\n\n    text_lookup = build_text_lookup(train_df)\n    print(f\"train text features: {len(text_lookup)} studies x {TEXT_DIM}\")\n\n    text_lookup_test = None\n    test_csv = os.path.join(CFG.data_dir, \"test.csv\")\n    if os.path.exists(test_csv):\n        tdf = pd.read_csv(test_csv)\n        if \"Report\" in tdf.columns:\n            text_lookup_test = build_text_lookup(tdf)\n            print(f\"test reports found: {len(text_lookup_test)} studies (real text)\")\n        else:\n            print(\"test.csv exists but has no Report -> zeros at inference\")\n    else:\n        print(\"no test.csv -> zeros at inference\")\nelse:\n    text_lookup, text_lookup_test = None, None\n\n\ndef text_row(uid, lookup, dim=TEXT_DIM):\n    v = lookup.get(uid) if lookup else None\n    return v if v is not None else np.zeros(dim, dtype=np.float32)\n"},{"cell_type":"markdown","metadata":{},"source":"## 7. Dataset\n\nEach sample = one (study, series) stack. A fixed window of `CFG.n_slices`\nslices is drawn: random window in training, sliding window (`offset`) in\nvalidation/TTA. Augmentations (horizontal flip, small rotation, brightness\n+ contrast jitter) are applied per slice; padded slots are masked out so\nattention pooling ignores them."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"class KneeDataset(Dataset):\n    def __init__(self, samples, base_dir, text_lookup=None, train_mode=True, offset=0):\n        self.samples = samples\n        self.base_dir = base_dir\n        self.text_lookup = text_lookup\n        self.train_mode = train_mode\n        self.offset = offset\n\n    def __len__(self):\n        return len(self.samples)\n\n    def __getitem__(self, i):\n        uid, sid = self.samples[i]\n        arr = get_cached(uid, sid, self.base_dir)      # (S,H,W) uint8\n        S, K, H, W = len(arr), CFG.n_slices, CFG.img_size, CFG.img_size\n        if self.train_mode and S > K:\n            start = np.random.randint(0, S - K + 1)\n            idx = np.arange(start, start + K)\n        elif S >= K:\n            step = max(1, S - K + 1)\n            start = min(self.offset, step - 1)\n            idx = np.arange(start, start + K)\n        else:\n            idx = np.linspace(0, S - 1, K).astype(int)\n\n        real = min(S, K)\n        imgs = np.zeros((K, H, W), dtype=np.float32)\n        imgs[:real] = arr[idx[:real]].astype(np.float32) / 255.0\n        for k in range(real):\n            img = imgs[k]\n            if self.train_mode:\n                if random.random() < 0.5:\n                    img = img[:, ::-1]\n                angle = random.uniform(-7, 7)\n                if abs(angle) > 0.1:\n                    M = cv2.getRotationMatrix2D((W / 2, H / 2), angle, 1.0)\n                    img = cv2.warpAffine(img, M, (W, H), flags=cv2.INTER_LINEAR,\n                                         borderMode=cv2.BORDER_REPLICATE)\n                img = np.clip(img * random.uniform(0.9, 1.1) + random.uniform(-0.04, 0.04), 0, 1)\n            imgs[k] = (img - IMAGENET_MEAN[0]) / IMAGENET_STD[0]\n\n        mask = np.zeros(K, dtype=np.bool_)\n        mask[:real] = True\n        text = text_row(uid, self.text_lookup)\n        return (torch.from_numpy(imgs).float(), torch.from_numpy(mask),\n                torch.from_numpy(text), i)\n"},{"cell_type":"markdown","metadata":{},"source":"## 8. Model\n\n- Backbone: ImageNet-pretrained 2D CNN (DenseNet-121 default, EfficientNet-B0\n  option). Weights must be attached for offline scoring — see helper cell at\n  the end; without them the model falls back to random init.\n- **Slice attention**: each slice's feature vector gets an attention score\n  (small Linear); masked slots are set to -inf; softmax over valid slices;\n  weighted average -> series feature.\n- **Text MLP**: hashed report vector -> ReLU(128) -> series feature concat.\n- Head: dropout 0.2 + Linear(D + 512 or D, 12).\n\n**Offline note:** torchvision cannot download weights at scoring time. Save\nthem during development (`CFG.save_weights = True`, last cell), attach as a\ndataset, set `CFG.pretrained_path`."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def load_backbone(backbone):\n    pretrained = bool(CFG.pretrained_path) and os.path.exists(CFG.pretrained_path)\n    if backbone == \"densenet121\":\n        from torchvision.models import densenet121, DenseNet121_Weights\n        if not pretrained:\n            try:\n                base = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)\n            except Exception as e:\n                print(\"WARNING: ImageNet download failed -> random init:\", e)\n                base = densenet121(weights=None)\n        else:\n            base = densenet121(weights=None)\n        D = 1024\n    elif backbone == \"efficientnet_b0\":\n        from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights\n        if not pretrained:\n            try:\n                base = efficientnet_b0(weights=EfficientNet_B0_Weights.IMAGENET1K_V1)\n            except Exception as e:\n                print(\"WARNING: ImageNet download failed -> random init:\", e)\n                base = efficientnet_b0(weights=None)\n        else:\n            base = efficientnet_b0(weights=None)\n        D = 1280\n    else:\n        raise ValueError(f\"unknown backbone: {backbone}\")\n    if pretrained:\n        sd = torch.load(CFG.pretrained_path, map_location=\"cpu\")\n        sd = {k: v for k, v in sd.items() if k in base.state_dict()}\n        base.load_state_dict(sd, strict=False)\n        print(\"loaded attached pretrained weights\")\n    return base, D\n\n\nclass KneeModel(nn.Module):\n    def __init__(self, backbone=\"densenet121\", num_classes=12, text_dim=0):\n        super().__init__()\n        base, D = load_backbone(backbone)\n        self.features = base.features\n        self.text_dim = text_dim\n        self.attn = nn.Linear(D, 1)\n        self.head = nn.Sequential(nn.Dropout(0.2), nn.Linear(D + text_dim, num_classes))\n\n    def forward(self, x, mask, text=None):\n        # x: (B,K,H,W)  mask: (B,K)  text: (B,TEXT_DIM) or None\n        # Frames are run through the backbone in chunks of CFG.frames_chunk so\n        # GPU memory stays bounded regardless of batch size / series length.\n        B, K = x.shape[:2]\n        flat = x.view(-1, 1, x.size(2), x.size(3))\n        feats = []\n        for c in range(0, len(flat), CFG.frames_chunk):\n            x3 = flat[c:c + CFG.frames_chunk].expand(-1, 3, -1, -1)\n            f = self.features(x3)\n            feats.append(F.adaptive_avg_pool2d(f, 1).flatten(1))\n        feats = torch.cat(feats, 0).view(B, K, -1)\n        att = self.attn(feats).squeeze(-1)\n        att = att.masked_fill(~mask, float(\"-inf\"))\n        att = F.softmax(att, dim=1).unsqueeze(-1)\n        pooled = (feats * att).sum(1)                 # (B, D)\n        if self.text_dim and text is not None:\n            pooled = torch.cat([pooled, text], dim=1)\n        return self.head(pooled)\n\n\nclass EMA:\n    \"\"\"Exponential moving average of model weights (evaluated, not trained).\"\"\"\n\n    def __init__(self, model, decay=0.99):\n        self.decay = decay\n        self.shadow = {k: v.detach().clone() for k, v in model.state_dict().items()}\n\n    def update(self, model):\n        for k, v in model.state_dict().items():\n            if v.dtype != torch.float32:     # e.g. BN num_batches_tracked (long)\n                self.shadow[k].copy_(v.detach())\n                continue\n            self.shadow[k].mul_(self.decay).add_(v.detach(), alpha=1 - self.decay)\n\n    def swap(self, model):\n        backup = {k: v.detach().clone() for k, v in model.state_dict().items()}\n        model.load_state_dict(self.shadow)\n        return backup\n\n    def restore(self, model, backup):\n        model.load_state_dict(backup)\n\n\ndef count_params(m):\n    total = sum(p.numel() for p in m.parameters())\n    train = sum(p.numel() for p in m.parameters() if p.requires_grad)\n    return total, train\n"},{"cell_type":"markdown","metadata":{},"source":"## 9. Training loop\n\n- Loss: BCE + 0.02 label smoothing on study-level logits\n- AdamW (head 1e-3 / body 3e-5), linear warmup (5%) + cosine decay\n- Epoch 1: head-only (backbone frozen); epochs 2+: full fine-tune\n- AMP on GPU, gradient clipping at 1.0, EMA decay 0.99 (validation uses EMA)\n- Early stopping on val macro AUC (patience), best EMA state per fold saved\n- Per-label AUC printed each epoch for debugging weak classes\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def sigmoid(z):\n    \"\"\"Stable sigmoid probabilities from logits (clip guards exp overflow).\"\"\"\n    return 1.0 / (1.0 + np.exp(-np.clip(z, -30, 30)))\n\n\ndef macro_auc(y_df, probs):\n    scores = []\n    for j, t in enumerate(TARGETS):\n        try:\n            s = roc_auc_score(y_df[t].values, probs[:, j])\n        except ValueError:\n            continue\n        if np.isfinite(s):                 # sklearn may return NaN for single-class y\n            scores.append(s)\n    return float(np.mean(scores)) if scores else 0.0\n\n\ndef per_label_auc(y_df, probs):\n    out = {}\n    for j, t in enumerate(TARGETS):\n        if y_df[t].nunique() > 1:\n            try:\n                s = roc_auc_score(y_df[t].values, probs[:, j])\n            except ValueError:\n                continue\n            if np.isfinite(s):\n                out[t] = round(s, 4)\n    return out\n\n\ndef predict_studies(model, samples, gt_ids, base_dir, text_lookup=None, grids=CFG.tta_grids):\n    \"\"\"Study-level logits (N,12). grids>1 slides the slice window (TTA).\"\"\"\n    model.eval()\n    pos = {u: i for i, u in enumerate(gt_ids)}\n    sums = np.zeros((len(gt_ids), len(TARGETS)))\n    cnts = np.zeros(len(gt_ids))\n    for g in range(grids):\n        offset = int(round(g * CFG.n_slices / max(grids - 1, 1)))\n        ds = KneeDataset(samples, base_dir, text_lookup=text_lookup,\n                         train_mode=False, offset=offset)\n        dl = DataLoader(ds, batch_size=CFG.batch_size, shuffle=False, num_workers=0)\n        with torch.no_grad():\n            for imgs, mask, text, idxs in dl:\n                imgs, mask = imgs.to(DEVICE), mask.to(DEVICE)\n                text = text.to(DEVICE) if model.text_dim else None\n                B = imgs.size(0)\n                logits = model(imgs, mask, text)\n                for b in range(B):\n                    p = pos[samples[idxs[b]][0]]\n                    sums[p] += logits[b].cpu().numpy()\n                    cnts[p] += 1\n    return sums / np.maximum(cnts, 1)[:, None]\n\n\ndef train_fold(fold, tr_samples, va_samples, va_ids, label_df, text_lookup, w_map=None):\n    seed_everything(SEED + fold)\n    model = KneeModel(backbone=CFG.backbone, num_classes=len(TARGETS),\n                      text_dim=TEXT_DIM if TEXT_ON else 0)\n    if CFG.use_compile and torch.cuda.is_available():\n        try:\n            model = torch.compile(model)\n        except Exception:\n            pass\n    model = model.to(DEVICE)\n    total_p, train_p = count_params(model)\n    print(f\"  params: total={total_p/1e6:.1f}M trainable={train_p/1e6:.1f}M\")\n\n    tr_ids = list(dict.fromkeys(u for u, _ in tr_samples))\n    label_arr = label_df.set_index(\"StudyInstanceUID\").reindex(tr_ids)\n    labs = label_arr[TARGETS].values.astype(np.float32)\n    # pseudo-label rows can be NaN on undecided targets -> mask from the loss\n    lab_mask = ~np.isnan(labs)\n    labs = np.nan_to_num(labs)\n    labels = torch.tensor(labs, dtype=torch.float32, device=DEVICE)\n    label_mask = torch.tensor(lab_mask, dtype=torch.bool, device=DEVICE)\n    if w_map:\n        w = torch.tensor([w_map.get(u, 1.0) for u in tr_ids], dtype=torch.float32,\n                         device=DEVICE)\n    else:\n        w = torch.ones(len(tr_ids), dtype=torch.float32, device=DEVICE)\n    pos_of_uid = {u: i for i, u in enumerate(tr_ids)}\n\n    head_params, body_params = [], []\n    for name, p in model.named_parameters():\n        if name.startswith(\"classifier\") or name.startswith(\"head\") or name.startswith(\"attn\"):\n            head_params.append(p)\n        else:\n            body_params.append(p)\n\n    optimizer = torch.optim.AdamW(\n        [{\"params\": body_params, \"lr\": CFG.lr_body},\n         {\"params\": head_params, \"lr\": CFG.lr_head}],\n        weight_decay=CFG.weight_decay)\n\n    total_steps = CFG.epochs * ((len(tr_samples) + CFG.batch_size - 1) // CFG.batch_size)\n    warmup_steps = int(total_steps * CFG.warmup_frac)\n\n    def lr_lambda(step):\n        if step < warmup_steps:\n            return 0.1 + 0.9 * step / max(1, warmup_steps)\n        t = (step - warmup_steps) / max(1, total_steps - warmup_steps)\n        return 0.5 * (1.0 + math.cos(math.pi * t))\n\n    scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)\n    scaler = torch.cuda.amp.GradScaler(enabled=CFG.use_amp)\n\n    def bce_loss(logits, targets, wgt=None, msk=None):\n        # BCEWithLogits with manual label smoothing (the label_smoothing kwarg\n        # is not supported on every torch version); masked+weighted so pseudo\n        # rows (NaN targets) and low-weight pseudo labels are handled\n        y = targets * (1.0 - CFG.label_smoothing) + 0.5 * CFG.label_smoothing\n        loss = F.binary_cross_entropy_with_logits(logits, y, reduction=\"none\")\n        if wgt is not None:\n            loss = loss * wgt[:, None]\n        if msk is not None:\n            m = msk.float()\n            loss = loss * m\n        else:\n            m = torch.ones_like(loss)\n        return loss.sum() / m.sum().clamp_min(1.0)\n\n    ema = EMA(model, CFG.ema_decay)\n\n    best_auc, best_ema_state, no_improve = -1.0, None, 0\n\n    for ep in range(1, CFG.epochs + 1):\n        freeze = ep <= CFG.freeze_epochs\n        for p in body_params:\n            p.requires_grad_(not freeze)\n        ds_tr = KneeDataset(tr_samples, CFG.train_dir,\n                            text_lookup=text_lookup, train_mode=True)\n        dl_tr = DataLoader(ds_tr, batch_size=CFG.batch_size, shuffle=True,\n                           num_workers=CFG.num_workers, pin_memory=(DEVICE == \"cuda\"))\n        model.train()\n        t0 = time.time()\n        for step, (imgs, mask, text, idxs) in enumerate(dl_tr):\n            imgs, mask = imgs.to(DEVICE), mask.to(DEVICE)\n            text = text.to(DEVICE) if model.text_dim else None\n            B = imgs.size(0)\n            with torch.autocast(\"cuda\", enabled=CFG.use_amp):\n                logits = model(imgs, mask, text)          # (B,12) per sample\n                # per-study logits indexed by study position in tr_ids\n                # (tensor sized len(tr_ids), NOT batch size: batch samples can\n                # belong to studies with position > B)\n                study_logits = torch.zeros(len(tr_ids), len(TARGETS), device=DEVICE)\n                counts = torch.zeros(len(tr_ids), device=DEVICE)\n                for b in range(B):\n                    p = pos_of_uid[tr_samples[idxs[b]][0]]\n                    study_logits[p] += logits[b]\n                    counts[p] += 1\n                # loss only over studies present in THIS batch\n                sel = torch.nonzero(counts > 0).squeeze(1)\n                if len(sel) == 0:\n                    continue\n                study_logits = study_logits[sel] / counts[sel].clamp_min(1).unsqueeze(1)\n                loss = bce_loss(study_logits, labels[sel], w[sel], label_mask[sel])\n            scaler.scale(loss).backward()\n            if CFG.grad_clip > 0:\n                scaler.unscale_(optimizer)\n                torch.nn.utils.clip_grad_norm_(model.parameters(), CFG.grad_clip)\n            scaler.step(optimizer)\n            scaler.update()\n            optimizer.zero_grad()\n            scheduler.step()\n            ema.update(model)\n            if step % 50 == 0:\n                print(f\"  fold{fold} ep{ep} step{step}/{len(dl_tr)} loss={loss.item():.4f}\")\n        del dl_tr\n\n        backup = ema.swap(model)\n        va_logits = predict_studies(model, va_samples, va_ids, CFG.train_dir,\n                                    text_lookup, grids=1)\n        ema.restore(model, backup)\n        va_prob = sigmoid(va_logits)\n        auc = macro_auc(label_df.set_index(\"StudyInstanceUID\").loc[va_ids][TARGETS], va_prob)\n        pla = per_label_auc(label_df.set_index(\"StudyInstanceUID\").loc[va_ids][TARGETS], va_prob)\n        print(f\"  fold{fold} ep{ep} val_macro_auc={auc:.4f}  ({time.time()-t0:.0f}s/epoch)\")\n        print(\"  per-label:\", \" \".join(f\"{k}:{v:.4f}\" for k, v in pla.items()))\n        if auc > best_auc + 1e-4:\n            best_auc, no_improve = auc, 0\n            best_ema_state = {k: v.detach().cpu().clone() for k, v in ema.shadow.items()}\n        else:\n            no_improve += 1\n            if no_improve >= CFG.patience:\n                print(f\"  early stop at epoch {ep}\")\n                break\n\n    if best_ema_state is None:             # no improvement at all -> keep EMA\n        best_ema_state = {k: v.detach().cpu().clone() for k, v in ema.shadow.items()}\n    torch.save(best_ema_state, f\"{CFG.out_dir}/knee_fold{fold}.pt\")\n    return best_auc\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"results = []\nfor fold, (tr_idx, va_idx) in enumerate(splits):\n    if elapsed_hours() > CFG.budget_hours:\n        print(f\"[budget] {elapsed_hours():.1f}h elapsed > {CFG.budget_hours}h\")\n        print(\"  -> skipping remaining folds\")\n        break\n    tr_u = set(labeled.iloc[tr_idx].StudyInstanceUID)\n    if CFG.use_pseudo and pseudo_w:\n        # pseudo studies of this fold: all except val studies\n        tr_u |= set(pseudo_selected.StudyInstanceUID) - set(labeled.iloc[va_idx].StudyInstanceUID)\n    va_u = list(labeled.iloc[va_idx].StudyInstanceUID)\n    tr_samples = [(u, s) for u, s in train_samples if u in tr_u]\n    va_samples = [(u, s) for u, s in train_samples if u in va_u]\n    print(f\"== fold {fold}: train studies={len(tr_u)} val studies={len(va_u)} \"\n          f\"(elapsed {elapsed_hours():.2f}h) ==\")\n    t0 = time.time()\n    auc = train_fold(fold, tr_samples, va_samples, va_u, tr_label, text_lookup, pseudo_w)\n    results.append(auc)\n    print(f\"== fold {fold} done, val macro AUC={auc:.4f} ({time.time()-t0:.0f}s) ==\")\n    free_memory()\n\nif results:\n    print(\"fold AUCs:\", [round(a, 4) for a in results],\n          \"-> mean\", round(float(np.mean(results)), 4))\nelse:\n    print(\"WARNING: no fold completed (budget/subset too small)\")\n"},{"cell_type":"markdown","metadata":{},"source":"## 10. Inference -> submission.csv\n\nEnsemble over trained folds (mean of sigmoid probabilities) with sliding-window\nTTA. The final submission is merged against `sample_submission` so every test\nrow is present; studies without usable data default to 0.5."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"test_ids = sorted(set(u for u, _ in test_samples) | set(test_map.keys()))\nprint(f\"test studies: {len(test_ids)} | test samples: {len(test_samples)}\")\nif len(test_samples) == 0:\n    print(\"WARNING: no valid test series -> submission will be all 0.5\")\n\ngrids = CFG.tta_grids if elapsed_hours() < CFG.budget_hours else 1\nall_probs = np.zeros((len(test_ids), len(TARGETS)))\nused = 0\nfor fold in range(CFG.folds):\n    ckpt = os.path.join(CFG.out_dir, f\"knee_fold{fold}.pt\")\n    if not os.path.exists(ckpt):\n        continue\n    model = KneeModel(backbone=CFG.backbone, num_classes=len(TARGETS),\n                      text_dim=TEXT_DIM if TEXT_ON else 0).to(DEVICE)\n    model.load_state_dict(torch.load(ckpt, map_location=\"cpu\"))\n    logits = predict_studies(model, test_samples, test_ids, CFG.test_dir,\n                             text_lookup_test, grids=grids)\n    all_probs += sigmoid(logits)\n    used += 1\n    del model\n    free_memory()\n    print(f\"fold {fold} inference done (elapsed {elapsed_hours():.2f}h)\")\n\nif used == 0:\n    print(\"WARNING: no fold checkpoints found -> submission will be all 0.5\")\n    all_probs = np.full((len(test_ids), len(TARGETS)), 0.5)\nelse:\n    all_probs /= used\n    print(f\"ensembled {used} fold(s), grids={grids}\")\n"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"prob_df = pd.DataFrame(all_probs, columns=TARGETS)\nprob_df[\"StudyInstanceUID\"] = test_ids\nsub = sample_sub[[\"StudyInstanceUID\"]].merge(prob_df, on=\"StudyInstanceUID\", how=\"left\").fillna(0.5)\nsub = sub[sample_sub.columns]\n\nassert len(sub) == len(sample_sub)\nassert list(sub.columns) == list(sample_sub.columns)\nassert sub.isna().sum().sum() == 0\nassert (sub[TARGETS] >= 0).all().all() and (sub[TARGETS] <= 1).all().all()\n\nsub.to_csv(f\"{CFG.out_dir}/submission.csv\", index=False)\nprint(\"saved /kaggle/working/submission.csv | rows:\", len(sub))\nprint(sub.head())\nprint()\nprint(\"predicted probability stats (sanity: no NaN/constant columns):\")\nprint(sub[TARGETS].agg([\"mean\", \"min\", \"max\"]).round(3).to_string())\n"},{"cell_type":"markdown","metadata":{},"source":"## 11. Offline-weights helper (dev only) + timing\n\nSet `CFG.save_weights = True` in the config cell and run the notebook **once\nduring development (internet ON)** — the last cell dumps the ImageNet backbone\nto `/kaggle/working`. Then create a dataset from that file (Output tab ->\nCreate dataset, or download & upload), attach it to this notebook and set\n`CFG.pretrained_path` to e.g.\n`/kaggle/input/knee-backbone/densenet121_backbone.pth` (set\n`CFG.save_weights = False` again).\n\nAlso useful for the **Efficiency award**: it is scored as\n`AUC/(Benchmark-maxAUC) + RuntimeSeconds/32400`, i.e. faster is better —\ncaching, no heavy TTA and small ensembles all help. Check the elapsed time\nbelow and tighten `CFG` if needed."},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# ---- DEV ONLY: save backbone weights for offline scoring ----\nif CFG.save_weights:\n    base, _ = load_backbone(CFG.backbone)\n    torch.save(base.state_dict(), f\"{CFG.out_dir}/{CFG.backbone}_backbone.pth\")\n    print(\"saved ->\", f\"{CFG.out_dir}/{CFG.backbone}_backbone.pth\")\n\nprint(f\"total runtime: {elapsed_hours():.2f} h\")\nprint(\"submission ready: /kaggle/working/submission.csv\")\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11"}},"nbformat":4,"nbformat_minor":5}