{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":99552,"databundleVersionId":13851420},{"sourceType":"datasetVersion","sourceId":15040734,"datasetId":9628522,"databundleVersionId":15919893},{"sourceType":"datasetVersion","sourceId":3610416,"datasetId":2126553,"databundleVersionId":3663963},{"sourceType":"datasetVersion","sourceId":15126861,"datasetId":9686495,"databundleVersionId":16015164},{"sourceType":"datasetVersion","sourceId":15126867,"datasetId":9686499,"databundleVersionId":16015170},{"sourceType":"datasetVersion","sourceId":15020557,"datasetId":9615023,"databundleVersionId":15897969},{"sourceType":"datasetVersion","sourceId":14998015,"datasetId":9600370,"databundleVersionId":15872863},{"sourceType":"modelInstanceVersion","sourceId":612683,"databundleVersionId":14140664,"modelInstanceId":460275,"modelId":476073}],"dockerImageVersionId":31287,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# V-Ultimate: Physics-Constrained, “Do-No-Harm” CT Image Restoration\n\n---\n\n## Research Map — A One-Minute Overview\n\n**What this project does** \nThis project develops **V-Ultimate**, a 2.5D U-Net for CT/CTA restoration. \nThe model is trained on **physics-simulated degradations** rather than paired real-world corrupted/clean scans. The training idea is inspired by **Liu et al. (2025), *Denoising the Deep Sky***, which showed that when perfect real pairs are unavailable, physically grounded corruption models can be used to generate meaningful supervision.\n\n**Why this problem is difficult** \nIn medical imaging, “sharper” does not automatically mean “better.” \nA restoration model can improve visual appearance while still harming downstream interpretation. It may:\n\n- exaggerate normal tissue so that it looks suspicious,\n- suppress subtle but real pathology,\n- or introduce changes that make a diagnosis less reliable.\n\nSo the real challenge is not simply to enhance CT images, but to do so **under control**.\n\n**How this notebook validates the model** \nThe project is organized around four experiments, each addressing one key reviewer question:\n\n| Experiment | Question it answers | How it answers it |\n|-----------|----------------------|-------------------|\n| **Clinical Rescue Matrix** | **Q1. Does it actually help?** | 50 OOD CT/CTA cases, multiple baselines, and an independent clinical proxy judge |\n| **Monte Carlo Noise Stability** | **Q2. Could the gain be random luck?** | 100 cases × 10 random seeds to test reproducibility under stochastic noise |\n| **TotalSegmentator Overlap** | **Q3. Is it modifying the whole image indiscriminately?** | Anatomical overlap analysis of where the model changes the scan |\n| **Mayo Cross-Domain Test** | **Q4. Does it generalize beyond the training domain?** | External evaluation on real paired low-dose CT from a different dataset |\n\n**Core contribution** \nThe contribution of this project is not just a restoration network. It is a **controlled restoration framework** with three layers of evidence:\n\n- **controlled intervention** rather than unrestricted enhancement,\n- **noise-stability analysis** rather than single-run claims,\n- **anatomical localization of edits** rather than black-box visual improvement.\n\n> **In one sentence:** \n> This project does not aim to produce the strongest-looking enhancement. It aims to build a form of medical image restoration that is **testable, constrained, and interpretable**: one that can show downstream clinical-proxy benefit in OOD data, reveal where its stability limits are under noise, and demonstrate that its edits are anatomically concentrated rather than randomly distributed across the image.\n\n---\n\n## What “Do-No-Harm” Means in This Project\n\nIn this notebook, **Do-No-Harm** does **not** mean refusing to change the image at all. \nIt means **avoiding misleading or uncontrolled edits**.\n\nThat principle is implemented through explicit mathematical constraints, including:\n\n- a **bounded residual budget** (maximum edit range capped at 15%),\n- an **identity hard lock** for near-zero degradation,\n- and **frequency-domain safeguards** that discourage unrealistic over-correction.\n\nIn other words, the model is allowed to intervene, but only within a controlled and auditable range.\n\n---\n\n## Notebook Structure\n\n| Section | Content |\n|--------|---------|\n| **Training Brief** | Overview of the V-Ultimate training pipeline and full training code |\n| **Framework Setup** | Environment, configuration, model loading, utility functions, and traditional baselines |\n| **Part I: Core Evidence** | Clinical Rescue Matrix → Monte Carlo Stability → TotalSegmentator Overlap |\n| **Part II: Generalization Evidence** | Mayo cross-domain validation on real paired low-dose CT |\n| **Supplementary Experiments** | Pilot comparison and additional OOD evaluation analyses |\n\n---","metadata":{}},{"cell_type":"markdown","source":"# ═══════════════════════════════════════════════════════════════\n# Part I: V-Ultimate New Training Deep-Dive (FiLM + Authority Map)\n# ═══════════════════════════════════════════════════════════════\n\n> ⚠️ Training on Kaggle GPU takes several hours. The evaluation section of this notebook typically loads pre-trained weights (`deblur_ultimate_film_auth_best.pt`) directly — no need to retrain each time.\n\n---\n\n## Theoretical Foundation: Physics-Based Degradation Synthesis + Safety-Constrained Restoration\n\nThe core idea of this project: when real \"clean–degraded\" paired medical images are hard to obtain, we use a **physics-inspired degradation model** to synthesise training data, then train a **controlled-modification (do-no-harm)** restoration network.\n\nWe approximate CT image degradation with three components:\n\n- **PSF blur**: Gaussian blur approximates finite-resolution imaging systems\n- **Poisson-Gaussian noise**: simulates quantum and electronic noise in low-dose X-ray\n- **Motion artefacts**: simulates directional blur from patient head movement\n\nThe network learns to restore degraded images back to clean, subject to a strict principle: **modify less when degradation is mild, modify nothing when there is no degradation, and be conservative in uncertain regions.**\n\n---\n\n## 8-Step Training Code Walkthrough\n\n### Step 0: Environment Initialisation\n- Detect GPU\n- Enable **AMP automatic mixed precision**\n- `cv2.setNumThreads(0)` to avoid conflicts between OpenCV and DataLoader\n\n### Step 1: Configuration & Physical Boundaries\nKey parameters:\n\n- Image size [(64, 448, 448)](cci:1://file:///tmp/update_results_md.py:9:0-11:76)\n- HU range `[-1024, 3072]`\n- Degradation levels `[0, 1, 3, 5, 8]`\n- Identity injection `P_IDENTITY = 0.20`\n- Residual budget `RES_MIN=0.02, RES_MAX=0.15`\n- Iatrogenic penalty `W_CHANGE_ID = 10.0`\n- Low-degradation edit constraint `W_LOW_T_EDIT`\n- Authority map TV regularisation `W_AUTH_TV`\n\n### Step 2: Data Triage — Build Train/Val Firewall\n- Keep CT/CTA only\n- Exclude localisers\n- Build UID lists for training and validation sets\n- Use `load_series_volume` to convert DICOM series to uniformly-sized 3D volumes\n\n### Step 3: Physical Degradation Engine\nRandomly apply the following degradations to clean slices:\n\n- Gaussian PSF blur\n- Poisson-Gaussian low-dose noise\n- Optional motion artefact\n\nThis synthesises clean–degraded training pairs on the fly.\n\n### Step 4: Anatomy-Aware Dataset\n- 2.5D input: slices `z-1, z, z+1`\n- Additional `t_norm` degradation-level channel\n- Generates `meta = [t_norm, do_motion, dose_clean, is_identity]` per sample\n- Rejects pure-air patches; prioritises patches with anatomical content\n\n### Step 5: New V-Ultimate Architecture\nTwo key mechanisms added over the original version:\n\n1. **FiLM Conditioning** \n Feeds `meta` into multiple encoder/decoder layers for conditional feature modulation. \n This allows the model to adapt its restoration strategy based on degradation type, motion, noise regime, and identity flag — the model \"knows\" what kind of input it is processing.\n\n2. **Pixel-wise Authority Map** \n The network additionally predicts a per-pixel authority map, determining how much each spatial location is allowed to change. The final modification budget is:\n\n `rmax_map = base_rmax × authority`\n\n The model becomes more conservative in uncertain regions and more confident where its predictions are reliable.\n\nPreserved from the original design:\n\n- **SEBlock channel attention**\n- **BatchNorm-free residual blocks** (preserves HU physical scale)\n- **Nearest-neighbour upsample + Conv** (eliminates checkerboard artefacts)\n- **Residual prediction + tanh clamp**\n- **Identity hard-lock** (zero modification when `t_norm ≈ 0`)\n\n### Step 6: Loss Functions\nTotal loss combines:\n\n- **Charbonnier**: pixel-level accuracy\n- **SSIM**: structural similarity\n- **Sobel**: edge fidelity\n- **Laplacian**: fine-structure fidelity\n- **FFT frequency-domain guard**: prevents over-smoothing\n- **Iatrogenic penalty**: minimise modification on identity samples (10× weight)\n- **Low-degradation edit penalty**: stay conservative when degradation is mild\n- **Authority map TV regularisation**: keeps the authority map spatially smooth and stable\n\n### Step 7: Validation Module\nPSNR is computed on the validation set every few epochs. \nIf a new best is achieved, the checkpoint is saved as:\n\n- `deblur_ultimate_film_auth_best.pt`\n\nThe final-epoch weights are also saved as:\n\n- `deblur_ultimate_film_auth_last.pt`\n\n### Step 8: Training Loop\nTraining uses:\n\n- AdamW optimiser\n- CosineAnnealing learning rate schedule\n- Gradient clipping\n- AMP (automatic mixed precision)\n\nTotal parameter updates depend on configuration, e.g.:\n\n- `14 × 250 = 3,500` updates (standard)\n- `20 × 350 = 7,000` updates (extended)\n\nThe goal of this training scheme is not simply to maximise PSNR, but to learn a **condition-aware, controlled-modification, iatrogenic-bias-minimising** restoration model.\n","metadata":{}},{"cell_type":"markdown","source":"# §2 V-Ultimate Model Architecture — Structure of the AI \"Brain\" (this cell is for your understanding only, highly probably no need to explain to judges)\n\n## Why Do We Need to Define the Architecture?\n\nAn AI model's \"knowledge\" is stored in a weights file — think of it as millions of numbers. But those numbers only make sense when arranged according to a specific **structure**, just like a dictionary's content (the weights) must match the dictionary's format (the architecture) to be usable.\n\nSo even when pre-trained weights already exist, we must first define the architecture before we can load those weights into it.\n\n## What the Model Name Means\n\n**DeblurUNet25D_Ultimate**:\n- **Deblur** = remove blur and noise\n- **UNet** = a classic image-processing AI architecture (see below)\n- **2.5D** = a processing strategy between 2D and 3D (see below)\n- **Ultimate** = final version after multiple rounds of iteration\n\n## What is a UNet? (Shaped Like the Letter U)\n\n\n\n```\nInput image Encoder (compress) Decoder (reconstruct) Output 448×448 → [32] → [64] → [128] → [256] (deepest, most abstract) ↓ ↓ ↓ pool2× pool2× pool2× (halved at each stage)\n\n[256] → [128] → [64] → [32] → Output ↑ ↑ ↑ upsample2× upsample2× upsample2× + skip connections\n```\n\n\n**Analogy**: imagine restoring a blurry photograph.\n- **Encoder** (left half): \"understands\" the image — progressively extracts abstract features from raw pixels (e.g., \"there is a line here\" → \"this is a blood vessel\")\n- **Decoder** (right half): \"reconstructs\" the image — progressively recovers concrete pixels from abstract concepts\n- **Skip Connections**: fine-grained detail from the encoder is passed directly to the decoder, preventing information loss during the compression stage\n\n## What Does 2.5D Mean?\n\nCT scans are **three-dimensional** (a stack of slices), but full 3D processing is prohibitively memory-intensive. Our compromise:\n\n**4-channel input**:\n\n| Channel | Content | Why it is needed |\n|---------|---------|-----------------|\n| ch0 | Previous slice `vol[z-1]` | Context for \"what is above\" |\n| ch1 | **Current slice** `vol[z]` | The target slice to restore |\n| ch2 | Next slice `vol[z+1]` | Context for \"what is below\" |\n| ch3 | Degradation parameter `t_norm` | Tells the model \"how blurry this image is\" (higher = more degraded) |\n\nEach inference processes one slice (2D) while using information from adjacent slices (half-3D) — hence **2.5D**.\n\n## Three Key Design Choices\n\n### 1. No Normalisation Layers\n\nMost AI models include BatchNorm or InstanceNorm for training stability. In medical imaging, however, **pixel values represent a physical quantity** (HU = tissue density). Normalisation layers distort this scale, making HU values in the output unreliable. We deliberately omit them.\n\n### 2. Residual Prediction + Clamping\n\nThe model does not output the restored image directly. Instead, it predicts a **correction term** (residual):\n\n```\nRestored image = Degraded image + residual × amplitude\n```\n\n- The residual is constrained to `[-1, 1]` via `tanh`\n- The amplitude `r_max` is dynamically scaled by degradation level (more blur → larger allowed budget)\n- Range: `r_max ∈ [0.02, 0.15]`, meaning the model can modify at most 15% of the original value\n**Why?** Safety. If the model predicted the full image and made an error, the output could be entirely unlike a CT scan. With residual + clamping, even a wrong prediction cannot stray far from the original input.\n\n### 3. Hard Identity Lock\n\n```python\nif t_norm <= 1e-8:\n return input image # zero modification guaranteed\n```\n\nIf the model is told \"this image has no degradation\" (t = 0), it is guaranteed to output the input unchanged. Zero degradation = zero intervention. This is a mathematical safety guarantee.\n\n## SE Block (Channel Attention)\n\nBefore entering the UNet, all 4 input channels pass through a Squeeze-and-Excite Block:\n\nEach channel is compressed to a single scalar (global average pooling)\nA small fully-connected network learns \"which channels matter more\"\nImportant channels are amplified; less important channels are suppressed\n\n**Analogy**：among the 4 channels, the current slice (ch1) is typically most informative and the time parameter (ch3) second. The SE Block lets the model learn this priority automatically, rather than treating all channels equally.","metadata":{}},{"cell_type":"code","source":"# =====================================================================\n# V-Ultimate FULL TRAINING CELL \n# =====================================================================\n\nimport os, gc, math, time, random\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport pydicom\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader, Sampler\nfrom contextlib import nullcontext\nimport torch.fft\n\n# -----------------------------\n# Environment\n# -----------------------------\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(\"Device:\", device)\n\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = lambda: torch.amp.autocast(\"cuda\") if USE_AMP else nullcontext()\nscaler = torch.amp.GradScaler(\"cuda\") if USE_AMP else None\n\ntorch.backends.cudnn.benchmark = True\nif device.type == \"cuda\":\n try:\n torch.backends.cuda.matmul.allow_tf32 = True\n torch.backends.cudnn.allow_tf32 = True\n torch.set_float32_matmul_precision(\"high\")\n except Exception:\n pass\n\n# ============================================================\n# Configuration\n# ============================================================\n\n# --- Data paths ---\nRSNA_DATA_ROOT = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\"\nTRAIN_LOCALIZERS_CSV = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train_localizers.csv\"\nMETA_CSV = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\"\n\nOUT_TRAIN_UIDS = \"/kaggle/working/train_uids_ultimate.csv\"\nOUT_VAL_UIDS = \"/kaggle/working/val_uids_ultimate.csv\"\n\nSAVE_BEST = \"/kaggle/working/deblur_ultimate_film_auth_best.pt\"\nSAVE_LAST = \"/kaggle/working/deblur_ultimate_film_auth_last.pt\"\n\n# --- Training scale ---\nSEED = 2026\nN_TRAIN_UIDS = 300\nN_VAL_UIDS = 20\nEPOCHS = 24\nBATCH_SIZE = 32\nBATCHES_PER_EPOCH = 400\nNUM_WORKERS = 4\n\n# --- Optimiser ---\nLR = 2e-4\nWEIGHT_DECAY = 1e-4\nGRAD_CLIP = 1.0\n\n# --- Image parameters ---\nTARGET_D, TARGET_H, TARGET_W = 64, 448, 448\nPATCH_SIZE = 112\nPATCHES_PER_SLICE = 1\n\nHU_MIN, HU_MAX = -1024.0, 3072.0\nHU_RANGE = HU_MAX - HU_MIN\n\n# --- Degradation parameters ---\nDIFFUSION_ALPHA = 0.20\nBLUR_LEVELS = [0, 1, 3, 5, 8]\nBLUR_LEVEL_MAX = float(max(BLUR_LEVELS))\nBLUR_T_MAX = BLUR_LEVEL_MAX\n\nP_IDENTITY = 0.20\nENABLE_MOTION = True\nP_MOTION = 0.15\n\nPEAK_RANGE_QUARTER, SIGMA_E_QUARTER = (3000.0, 6000.0), (0.01, 0.02)\nPEAK_RANGE_EXTREME, SIGMA_E_EXTREME = (1000.0, 3000.0), (0.02, 0.04)\n\nREGIME_PROBS = {\"clean\": 0.30, \"typical\": 0.45, \"hard\": 0.15, \"motion\": 0.10}\n\n# --- Anatomy-aware patch sampling ---\nANATOMY_REJECT_TRIES = 15\nANATOMY_MEAN_TH = 0.05\nANATOMY_STD_TH = 0.02\nP_RANDOM_PATCH = 0.10\n\n# --- Loss weights ---\nW_CHARBONNIER = 1.0\nW_SSIM = 0.20\nW_SOBEL = 0.10\nW_LAP = 0.05\nW_FFT = 0.05\nFFT_FCUTOFF = 0.20\nFFT_ONLY_IF_T_LE = 10.0\n\nW_CHANGE_ID = 10.0\nW_LOW_T_EDIT = 2.0\nLOW_T_THR = 0.20\nW_AUTH_TV = 1e-3\n\nRES_MIN, RES_MAX = 0.02, 0.15\n\n# ============================================================\n# Random seed\n# ============================================================\n\ndef seed_all(seed):\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(seed)\n\nseed_all(SEED)\n\ndef _normalize_probs(d):\n s = float(sum(d.values()))\n return {k: float(v) / s for k, v in d.items()}\n\nREGIME_PROBS = _normalize_probs(REGIME_PROBS)\n\n# ============================================================\n# UID list construction\n# ============================================================\n\ndef build_ct_only_uid_lists(meta_csv, rsna_series_root, localizers_csv, n_train, n_val, seed):\n rsna_uids = set(\n [u for u in os.listdir(rsna_series_root)\n if os.path.isdir(os.path.join(rsna_series_root, u)) and not u.startswith(\".\")]\n )\n\n localizer_uids = set()\n if localizers_csv and os.path.exists(localizers_csv):\n df_loc = pd.read_csv(localizers_csv)\n localizer_uids = set(df_loc[df_loc.columns[0]].astype(str).tolist())\n\n meta = pd.read_csv(meta_csv)\n meta_ct = meta[meta[\"Modality\"].isin({\"CT\", \"CTA\"})]\n\n ct_candidates = [\n u for u in set(meta_ct[\"SeriesInstanceUID\"].astype(str))\n if u in rsna_uids and u not in localizer_uids\n ]\n\n rng = random.Random(seed)\n rng.shuffle(ct_candidates)\n\n train_uids = ct_candidates[:min(n_train, len(ct_candidates))]\n rem_ct = [u for u in ct_candidates if u not in set(train_uids)]\n rng.shuffle(rem_ct)\n val_uids = rem_ct[:min(n_val, len(rem_ct))]\n return train_uids, val_uids\n\n# ============================================================\n# DICOM loading\n# ============================================================\n\ndef get_sorted_dicom_files(series_path):\n files = [f for f in os.listdir(series_path) if not f.startswith(\".\")]\n pairs, ok = [], True\n for f in files:\n try:\n ds = pydicom.dcmread(os.path.join(series_path, f), stop_before_pixels=True, force=True)\n if getattr(ds, \"InstanceNumber\", None) is None:\n ok = False\n break\n pairs.append((int(ds.InstanceNumber), os.path.join(series_path, f)))\n except Exception:\n ok = False\n break\n if ok and len(pairs) == len(files):\n return [p[1] for p in sorted(pairs, key=lambda x: x[0])]\n return [os.path.join(series_path, f) for f in sorted(files)]\n\ndef load_series_volume(uid, series_root, target_shape=(64, 448, 448)):\n series_path = os.path.join(series_root, uid)\n if not os.path.isdir(series_path):\n return None\n\n dcm_files = get_sorted_dicom_files(series_path)\n tD, tH, tW = target_shape\n if len(dcm_files) < 10:\n return None\n\n if len(dcm_files) != tD:\n idxs = np.linspace(0, len(dcm_files) - 1, tD).astype(int)\n dcm_files = [dcm_files[i] for i in idxs]\n\n slices = []\n for fp in dcm_files:\n try:\n ds = pydicom.dcmread(fp, force=True)\n hu = ds.pixel_array.astype(np.float32) * float(getattr(ds, \"RescaleSlope\", 1.0)) \\\n + float(getattr(ds, \"RescaleIntercept\", 0.0))\n x = (np.clip(hu, HU_MIN, HU_MAX) - HU_MIN) / HU_RANGE\n x = cv2.resize(x, (tW, tH), interpolation=cv2.INTER_LINEAR)\n slices.append(x.astype(np.float32))\n except Exception:\n continue\n\n if len(slices) < int(0.8 * tD):\n return None\n\n while len(slices) < tD:\n slices.append(slices[-1].copy())\n\n return np.stack(slices[:tD], axis=0).astype(np.float32)\n\nclass VolumeLRU:\n def __init__(self, max_items=12):\n self.max_items = int(max_items)\n self.cache = {}\n self.order = []\n\n def get(self, key):\n if key not in self.cache:\n return None\n self.order.remove(key)\n self.order.append(key)\n return self.cache[key]\n\n def put(self, key, value):\n if key in self.cache:\n self.order.remove(key)\n self.cache[key] = value\n self.order.append(key)\n if len(self.order) > self.max_items:\n old = self.order.pop(0)\n self.cache.pop(old, None)\n\n# ============================================================\n# Degradation engine\n# ============================================================\n\ndef gaussian_psf_surrogate(img01, blur_level, alpha=0.20):\n if blur_level <= 0:\n return img01\n sigma = math.sqrt(max(1e-8, 2.0 * alpha * float(blur_level)))\n return np.clip(\n cv2.GaussianBlur(img01, (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE),\n 0.0, 1.0\n )\n\ndef motion_artifact_surrogate(img01, length=None, angle=None):\n length = length or random.choice([3, 5, 7, 9, 11])\n if length <= 1:\n return img01\n angle = angle or random.uniform(0, 180)\n\n k = np.zeros((length, length), dtype=np.float32)\n c = length // 2\n cos_a, sin_a = np.cos(np.radians(angle)), np.sin(np.radians(angle))\n for i in range(length):\n x = int(c + (i - c) * cos_a)\n y = int(c + (i - c) * sin_a)\n if 0 <= x < length and 0 <= y < length:\n k[y, x] = 1.0\n if k.sum() > 0:\n k /= k.sum()\n\n return np.clip(cv2.filter2D(img01, -1, k, borderType=cv2.BORDER_REPLICATE), 0.0, 1.0)\n\ndef mixed_poisson_gaussian(img01, mode=\"quarter\"):\n if mode == \"clean\":\n return img01\n peak_rng, sigma_rng = (\n (PEAK_RANGE_EXTREME, SIGMA_E_EXTREME) if mode == \"extreme\"\n else (PEAK_RANGE_QUARTER, SIGMA_E_QUARTER)\n )\n peak = random.uniform(*peak_rng)\n sigma_e = random.uniform(*sigma_rng)\n noisy_p = np.random.poisson(np.clip(img01 * peak, 0, None)).astype(np.float32) / peak\n noisy_g = np.random.randn(*img01.shape).astype(np.float32) * sigma_e\n return np.clip(noisy_p + noisy_g, 0.0, 1.0)\n\ndef _choice_weighted(prob_dict):\n r = random.random()\n acc = 0.0\n for k, p in prob_dict.items():\n acc += p\n if r <= acc:\n return k\n return list(prob_dict.keys())[-1]\n\ndef _sample_regime_params():\n reg = _choice_weighted(REGIME_PROBS)\n if reg == \"clean\":\n return 0, \"clean\", False\n if reg == \"typical\":\n return random.choice([1, 3, 5]), \"quarter\", False\n if reg == \"hard\":\n return random.choice([3, 5, 8]), \"extreme\", False\n if reg == \"motion\":\n return random.choice([1, 3, 5]), random.choice([\"quarter\", \"extreme\"]), True\n return 3, \"quarter\", False\n\n# ============================================================\n# Dataset\n# ============================================================\n\nclass CTDeblur25D(Dataset):\n def __init__(self, uids, series_root, target_shape=(64, 448, 448), patch_size=112, patches_per_slice=1):\n self.uids = list(uids)\n self.series_root = series_root\n self.target_shape = target_shape\n self.patch_size = int(patch_size)\n self.patches_per_slice = int(patches_per_slice)\n self.cache = VolumeLRU(max_items=8)\n self.items = [(ui, z) for ui in range(len(self.uids)) for z in range(1, target_shape[0] - 1)]\n\n def __len__(self):\n return len(self.items)\n\n def _sample_patch_xy(self, cent, ps):\n H, W = cent.shape\n if random.random() < P_RANDOM_PATCH:\n return np.random.randint(0, H - ps + 1), np.random.randint(0, W - ps + 1)\n\n for _ in range(ANATOMY_REJECT_TRIES):\n y = np.random.randint(0, H - ps + 1)\n x = np.random.randint(0, W - ps + 1)\n patch = cent[y:y+ps, x:x+ps]\n if patch.mean() > ANATOMY_MEAN_TH and patch.std() > ANATOMY_STD_TH:\n return y, x\n\n return (H - ps) // 2, (W - ps) // 2\n\n def __getitem__(self, idx):\n ui, z = self.items[idx]\n uid = self.uids[ui]\n\n vol = self.cache.get(uid)\n if vol is None:\n vol = load_series_volume(uid, self.series_root, self.target_shape)\n if vol is None:\n return self.__getitem__(random.randint(0, len(self.items) - 1))\n self.cache.put(uid, vol)\n\n ps = self.patch_size\n y, x = self._sample_patch_xy(vol[z], ps)\n\n clean = vol[z][y:y+ps, x:x+ps].copy()\n pp = vol[z-1][y:y+ps, x:x+ps].copy()\n cc = vol[z][y:y+ps, x:x+ps].copy()\n nn_ = vol[z+1][y:y+ps, x:x+ps].copy()\n\n is_identity = 0.0\n if random.random() < P_IDENTITY:\n blur_level, dose_mode, do_motion, is_identity = 0, \"clean\", False, 1.0\n else:\n blur_level, dose_mode, do_motion = _sample_regime_params()\n do_motion = do_motion and ENABLE_MOTION and (random.random() < P_MOTION)\n\n bp = gaussian_psf_surrogate(pp, blur_level, alpha=DIFFUSION_ALPHA)\n bc = gaussian_psf_surrogate(cc, blur_level, alpha=DIFFUSION_ALPHA)\n bn = gaussian_psf_surrogate(nn_, blur_level, alpha=DIFFUSION_ALPHA)\n\n if do_motion:\n L = random.choice([3, 5, 7, 9, 11])\n A = random.uniform(0, 180)\n bp = motion_artifact_surrogate(bp, L, A)\n bc = motion_artifact_surrogate(bc, L, A)\n bn = motion_artifact_surrogate(bn, L, A)\n\n if dose_mode != \"clean\":\n bp = mixed_poisson_gaussian(bp, dose_mode)\n bc = mixed_poisson_gaussian(bc, dose_mode)\n bn = mixed_poisson_gaussian(bn, dose_mode)\n\n cp = clean.copy()\n if random.random() > 0.5:\n cp, bp, bc, bn = cp[::-1].copy(), bp[::-1].copy(), bc[::-1].copy(), bn[::-1].copy()\n if random.random() > 0.5:\n cp, bp, bc, bn = cp[:, ::-1].copy(), bp[:, ::-1].copy(), bc[:, ::-1].copy(), bn[:, ::-1].copy()\n k = random.randint(0, 3)\n if k > 0:\n cp, bp, bc, bn = np.rot90(cp, k).copy(), np.rot90(bp, k).copy(), np.rot90(bc, k).copy(), np.rot90(bn, k).copy()\n\n t_norm = float(blur_level) / BLUR_LEVEL_MAX if BLUR_LEVEL_MAX > 0 else 0.0\n\n inp = np.stack([bp, bc, bn, np.full_like(bc, t_norm, dtype=np.float32)], axis=0).astype(np.float32)\n tgt = cp[np.newaxis, ...].astype(np.float32)\n\n meta = np.array([\n t_norm,\n 1.0 if do_motion else 0.0,\n 1.0 if dose_mode == \"clean\" else 0.0,\n is_identity\n ], dtype=np.float32)\n\n return (\n torch.from_numpy(inp).float(),\n torch.from_numpy(tgt).float(),\n torch.from_numpy(meta).float()\n )\n\nclass UIDBatchSampler(Sampler):\n def __init__(self, dataset, batch_size, seed=42, batches_per_epoch=None):\n self.dataset = dataset\n self.batch_size = int(batch_size)\n self.rng = random.Random(seed)\n\n self.by_ui = {}\n for idx, (ui, z) in enumerate(dataset.items):\n self.by_ui.setdefault(ui, []).append(idx)\n\n self.ui_keys = list(self.by_ui.keys())\n self.batches_per_epoch = int(batches_per_epoch) if batches_per_epoch else len(dataset) // self.batch_size\n\n def __len__(self):\n return self.batches_per_epoch\n\n def __iter__(self):\n for _ in range(self.batches_per_epoch):\n ui = self.rng.choice(self.ui_keys)\n pool = self.by_ui[ui]\n if len(pool) >= self.batch_size:\n yield self.rng.sample(pool, self.batch_size)\n else:\n yield [self.rng.choice(pool) for _ in range(self.batch_size)]\n\n# ============================================================\n# New model: FiLM + authority map\n# ============================================================\n\nclass SEBlock(nn.Module):\n def __init__(self, c, r=4):\n super().__init__()\n self.fc = nn.Sequential(\n nn.AdaptiveAvgPool2d(1),\n nn.Conv2d(c, max(1, c // r), 1, bias=False),\n nn.ReLU(inplace=True),\n nn.Conv2d(max(1, c // r), c, 1, bias=False),\n nn.Sigmoid(),\n )\n\n def forward(self, x):\n return x * self.fc(x)\n\nclass ResBlockPhysics(nn.Module):\n def __init__(self, ic, oc):\n super().__init__()\n self.conv = nn.Sequential(\n nn.Conv2d(ic, oc, 3, padding=1, bias=True),\n nn.ReLU(inplace=True),\n nn.Conv2d(oc, oc, 3, padding=1, bias=True),\n )\n self.shortcut = nn.Conv2d(ic, oc, 1, bias=True) if ic != oc else nn.Identity()\n\n def forward(self, x):\n return F.relu(self.conv(x) + self.shortcut(x), inplace=True)\n\nclass UpsamplePhysicsUltimate(nn.Module):\n def __init__(self, ic, oc):\n super().__init__()\n self.up = nn.Sequential(\n nn.Upsample(scale_factor=2.0, mode=\"nearest\"),\n nn.Conv2d(ic, oc, 3, padding=1, bias=True),\n nn.ReLU(inplace=True),\n )\n\n def forward(self, x):\n return self.up(x)\n\nclass FiLM2d(nn.Module):\n def __init__(self, channels, meta_dim=4, hidden=64):\n super().__init__()\n self.net = nn.Sequential(\n nn.Linear(meta_dim, hidden),\n nn.ReLU(inplace=True),\n nn.Linear(hidden, channels * 2),\n )\n self.channels = channels\n\n def forward(self, x, meta):\n gb = self.net(meta)\n gamma, beta = torch.chunk(gb, 2, dim=1)\n gamma = gamma.view(-1, self.channels, 1, 1)\n beta = beta.view(-1, self.channels, 1, 1)\n return x * (1.0 + gamma) + beta\n\nclass DeblurUNet25D_Ultimate(nn.Module):\n def __init__(\n self,\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=0.02,\n res_max=0.15,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n ):\n super().__init__()\n self.res_min = float(res_min)\n self.res_max = float(res_max)\n self.meta_dim = int(meta_dim)\n\n c = [base, base * 2, base * 4, base * 8]\n\n self.se = SEBlock(in_ch)\n\n self.enc1 = ResBlockPhysics(in_ch, c[0])\n self.enc2 = ResBlockPhysics(c[0], c[1])\n self.enc3 = ResBlockPhysics(c[1], c[2])\n self.enc4 = ResBlockPhysics(c[2], c[3])\n self.pool = nn.MaxPool2d(2)\n\n self.up3 = UpsamplePhysicsUltimate(c[3], c[2])\n self.dec3 = ResBlockPhysics(c[2] * 2, c[2])\n\n self.up2 = UpsamplePhysicsUltimate(c[2], c[1])\n self.dec2 = ResBlockPhysics(c[1] * 2, c[1])\n\n self.up1 = UpsamplePhysicsUltimate(c[1], c[0])\n self.dec1 = ResBlockPhysics(c[0] * 2, c[0])\n\n self.film_e1 = FiLM2d(c[0], meta_dim=meta_dim, hidden=film_hidden)\n self.film_e2 = FiLM2d(c[1], meta_dim=meta_dim, hidden=film_hidden)\n self.film_e3 = FiLM2d(c[2], meta_dim=meta_dim, hidden=film_hidden)\n self.film_e4 = FiLM2d(c[3], meta_dim=meta_dim, hidden=film_hidden)\n self.film_d3 = FiLM2d(c[2], meta_dim=meta_dim, hidden=film_hidden)\n self.film_d2 = FiLM2d(c[1], meta_dim=meta_dim, hidden=film_hidden)\n self.film_d1 = FiLM2d(c[0], meta_dim=meta_dim, hidden=film_hidden)\n\n self.out_conv = nn.Conv2d(c[0], out_ch, 1, bias=True)\n\n self.auth_head = nn.Sequential(\n nn.Conv2d(c[0], c[0] // 2, 3, padding=1, bias=True),\n nn.ReLU(inplace=True),\n nn.Conv2d(c[0] // 2, 1, 1, bias=True),\n )\n nn.init.constant_(self.auth_head[-1].bias, float(authority_bias_init))\n\n def forward(self, x, meta=None, return_aux=False):\n bc = x[:, 1:2]\n tch = x[:, 3:4]\n\n if meta is None:\n B = x.shape[0]\n t_scalar = torch.mean(tch, dim=(2, 3)).view(B, 1)\n zeros = torch.zeros(B, 3, device=x.device, dtype=x.dtype)\n meta = torch.cat([t_scalar, zeros], dim=1)\n\n e1 = self.enc1(self.se(x))\n e1 = self.film_e1(e1, meta)\n\n e2 = self.enc2(self.pool(e1))\n e2 = self.film_e2(e2, meta)\n\n e3 = self.enc3(self.pool(e2))\n e3 = self.film_e3(e3, meta)\n\n e4 = self.enc4(self.pool(e3))\n e4 = self.film_e4(e4, meta)\n\n d3 = self.dec3(torch.cat([self.up3(e4), e3], dim=1))\n d3 = self.film_d3(d3, meta)\n\n d2 = self.dec2(torch.cat([self.up2(d3), e2], dim=1))\n d2 = self.film_d2(d2, meta)\n\n d1 = self.dec1(torch.cat([self.up1(d2), e1], dim=1))\n d1 = self.film_d1(d1, meta)\n\n residual = torch.tanh(self.out_conv(d1))\n\n base_rmax = self.res_min + (self.res_max - self.res_min) * tch\n authority = torch.sigmoid(self.auth_head(d1))\n rmax_map = base_rmax * authority\n\n pred_soft = (bc + residual * rmax_map).clamp(0.0, 1.0)\n pred = torch.where(tch <= 1e-8, bc, pred_soft)\n\n if return_aux:\n return pred, {\n \"authority\": authority,\n \"rmax_map\": rmax_map,\n \"residual\": residual,\n }\n return pred\n\n# ============================================================\n# Loss functions\n# ============================================================\n\ndef charbonnier_loss(pred, target, eps=1e-3):\n return torch.mean(torch.sqrt((pred - target) ** 2 + eps ** 2))\n\ndef fft_spectrum_loss(pred, target, fft_mask):\n pred_fft = torch.fft.rfft2(pred.float(), dim=(-2, -1), norm=\"ortho\")\n tgt_fft = torch.fft.rfft2(target.float(), dim=(-2, -1), norm=\"ortho\")\n m = fft_mask.view(1, 1, *fft_mask.shape)\n return charbonnier_loss(torch.abs(pred_fft) * m, torch.abs(tgt_fft) * m)\n\ndef ssim_loss(pred, target, window_size=11):\n C1, C2 = 0.01 ** 2, 0.03 ** 2\n pad = window_size // 2\n\n mu_x = F.avg_pool2d(pred, window_size, stride=1, padding=pad)\n mu_y = F.avg_pool2d(target, window_size, stride=1, padding=pad)\n\n sigma_x2 = F.avg_pool2d(pred ** 2, window_size, stride=1, padding=pad) - mu_x ** 2\n sigma_y2 = F.avg_pool2d(target ** 2, window_size, stride=1, padding=pad) - mu_y ** 2\n sigma_xy = F.avg_pool2d(pred * target, window_size, stride=1, padding=pad) - mu_x * mu_y\n\n ssim_map = ((2 * mu_x * mu_y + C1) * (2 * sigma_xy + C2)) / (\n (mu_x ** 2 + mu_y ** 2 + C1) * (sigma_x2 + sigma_y2 + C2) + 1e-8\n )\n return 1.0 - ssim_map.mean()\n\ndef _make_fft_mask(H, W, fcut=0.20, device=\"cpu\"):\n fy = torch.fft.fftfreq(H, d=1.0, device=device).view(H, 1).abs()\n fx = torch.fft.rfftfreq(W, d=1.0, device=device).view(1, W // 2 + 1).abs()\n return (torch.sqrt(fx * fx + fy * fy) >= fcut).float()\n\nclass UltimatePhysicsLoss(nn.Module):\n def __init__(self, patch_size=112):\n super().__init__()\n self.sobel_x = torch.tensor([[[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]]]).view(1, 1, 3, 3).to(device)\n self.sobel_y = torch.tensor([[[-1., -2., -1.], [0., 0., 0.], [1., 2., 1.]]]).view(1, 1, 3, 3).to(device)\n self.lap = torch.tensor([[[0., 1., 0.], [1., -4., 1.], [0., 1., 0.]]]).view(1, 1, 3, 3).to(device)\n self.register_buffer(\"fft_mask\", _make_fft_mask(patch_size, patch_size, fcut=FFT_FCUTOFF, device=device))\n\n def forward(self, pred, target, allow_fft=False):\n total = W_CHARBONNIER * charbonnier_loss(pred, target)\n total += W_SSIM * ssim_loss(pred, target)\n\n p_pad = F.pad(pred, (1, 1, 1, 1), mode=\"replicate\")\n t_pad = F.pad(target, (1, 1, 1, 1), mode=\"replicate\")\n\n total += W_SOBEL * (\n charbonnier_loss(F.conv2d(p_pad, self.sobel_x), F.conv2d(t_pad, self.sobel_x))\n + charbonnier_loss(F.conv2d(p_pad, self.sobel_y), F.conv2d(t_pad, self.sobel_y))\n )\n total += W_LAP * charbonnier_loss(F.conv2d(p_pad, self.lap), F.conv2d(t_pad, self.lap))\n\n if allow_fft:\n total += W_FFT * fft_spectrum_loss(pred, target, self.fft_mask)\n return total\n\ndef alg_humility_penalty(pred, inp, is_id):\n center = inp[:, 1:2]\n per_sample = torch.mean(torch.abs(pred - center), dim=(1, 2, 3))\n return torch.mean(per_sample * is_id * W_CHANGE_ID)\n\ndef low_t_edit_penalty(pred, inp, meta):\n center = inp[:, 1:2]\n t_norm = meta[:, 0]\n low_mask = (t_norm <= LOW_T_THR).float()\n per_sample = torch.mean(torch.abs(pred - center), dim=(1, 2, 3))\n return torch.mean(per_sample * low_mask * W_LOW_T_EDIT)\n\ndef authority_tv_penalty(authority):\n dy = torch.abs(authority[:, :, 1:, :] - authority[:, :, :-1, :]).mean()\n dx = torch.abs(authority[:, :, :, 1:] - authority[:, :, :, :-1]).mean()\n return (dx + dy) * W_AUTH_TV\n\n# ============================================================\n# Validation\n# ============================================================\n\n@torch.no_grad()\ndef eval_model_psnr(model, val_uids, series_root, target_shape=(64, 448, 448), max_uids=8):\n model.eval()\n scores = []\n cache = VolumeLRU(max_items=2)\n\n for uid in list(val_uids)[:max_uids]:\n vol = cache.get(uid)\n if vol is None:\n vol = load_series_volume(uid, series_root, target_shape)\n if vol is None:\n continue\n cache.put(uid, vol)\n\n D = vol.shape[0]\n for blur_level in [3, 8]:\n for z in range(1, D - 1, 8):\n cl = vol[z].astype(np.float32)\n prev = vol[z - 1].astype(np.float32)\n cent = vol[z].astype(np.float32)\n next_ = vol[z + 1].astype(np.float32)\n\n bp = mixed_poisson_gaussian(gaussian_psf_surrogate(prev, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bc = mixed_poisson_gaussian(gaussian_psf_surrogate(cent, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bn = mixed_poisson_gaussian(gaussian_psf_surrogate(next_, blur_level, DIFFUSION_ALPHA), \"quarter\")\n\n t_norm = float(blur_level) / BLUR_LEVEL_MAX\n inp_np = np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n inp_t = torch.from_numpy(inp_np).unsqueeze(0).to(device)\n\n meta_np = np.array([[t_norm, 0.0, 0.0, 0.0]], dtype=np.float32)\n meta_t = torch.from_numpy(meta_np).to(device)\n\n with AMP_CTX():\n pred = model(inp_t, meta=meta_t)[0, 0].float().cpu().numpy()\n\n mse = float(np.mean((pred - cl) ** 2))\n scores.append(99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse))\n\n return float(np.mean(scores)) if scores else None\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T00:49:15.730759Z","iopub.execute_input":"2026-03-11T00:49:15.731124Z","iopub.status.idle":"2026-03-11T00:49:15.827597Z","shell.execute_reply.started":"2026-03-11T00:49:15.731094Z","shell.execute_reply":"2026-03-11T00:49:15.826702Z"}},"outputs":[{"name":"stdout","text":"Device: cuda\n","output_type":"stream"}],"execution_count":15},{"cell_type":"code","source":"\n# ============================================================\n# Pre-training checks\n# ============================================================\n\nprint(\"\\n=== 0. Pre-training checks ===\")\nprint(\"RSNA_DATA_ROOT exists?\", os.path.exists(RSNA_DATA_ROOT))\nprint(\"META_CSV exists?\", os.path.exists(META_CSV))\nprint(\"TRAIN_LOCALIZERS_CSV exists?\", os.path.exists(TRAIN_LOCALIZERS_CSV))\n\n# ============================================================\n# Data preparation\n# ============================================================\n\nprint(\"\\n=== 1. Build data firewall ===\")\ntrain_uids, val_uids = build_ct_only_uid_lists(\n META_CSV, RSNA_DATA_ROOT, TRAIN_LOCALIZERS_CSV, N_TRAIN_UIDS, N_VAL_UIDS, SEED\n)\npd.DataFrame({\"SeriesInstanceUID\": train_uids}).to_csv(OUT_TRAIN_UIDS, index=False)\npd.DataFrame({\"SeriesInstanceUID\": val_uids}).to_csv(OUT_VAL_UIDS, index=False)\nprint(f\"Train: {len(train_uids)} | Val: {len(val_uids)}\")\n\nprint(\"\\n=== 2. Initialise dataset ===\")\ntrain_ds = CTDeblur25D(\n train_uids, RSNA_DATA_ROOT, (TARGET_D, TARGET_H, TARGET_W), PATCH_SIZE, PATCHES_PER_SLICE\n)\ntrain_loader = DataLoader(\n train_ds,\n batch_sampler=UIDBatchSampler(train_ds, BATCH_SIZE, SEED, BATCHES_PER_EPOCH),\n num_workers=NUM_WORKERS,\n pin_memory=True,\n)\n\n# ============================================================\n# Model initialisation\n# ============================================================\n\nprint(\"\\n=== 3. Initialise new model ===\")\nmodel = DeblurUNet25D_Ultimate(\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=RES_MIN,\n res_max=RES_MAX,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n).to(device)\n\nprint(\"Model parameters:\", f\"{sum(p.numel() for p in model.parameters() if p.requires_grad):,}\")\nprint(\"has auth_head?\", hasattr(model, \"auth_head\"))\nprint(\"has film_e1?\", hasattr(model, \"film_e1\"))\n\nassert hasattr(model, \"auth_head\"), \"Not the new model: missing auth_head\"\nassert hasattr(model, \"film_e1\"), \"Not the new model: missing FiLM module\"\n\ncrit = UltimatePhysicsLoss(patch_size=PATCH_SIZE).to(device)\nopt = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay=WEIGHT_DECAY)\nsched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=EPOCHS)\nBEST_PSNR = -1.0\n\n# ============================================================\n# Training\n# ============================================================\n\nprint(\"\\n=== 4. Start training ===\")\nfor ep in range(1, EPOCHS + 1):\n model.train()\n losses = []\n t0 = time.time()\n\n for b, (inp, tgt, meta) in enumerate(train_loader, 1):\n inp = inp.to(device, non_blocking=True)\n tgt = tgt.to(device, non_blocking=True)\n meta = meta.to(device, non_blocking=True)\n\n opt.zero_grad(set_to_none=True)\n\n with AMP_CTX():\n pred, aux = model(inp, meta=meta, return_aux=True)\n\n t_scalar = meta[:, 0].mean().item() * BLUR_LEVEL_MAX\n is_motion = meta[:, 1]\n is_id = meta[:, 3]\n\n allow_fft = (t_scalar <= FFT_ONLY_IF_T_LE) and not bool((is_motion > 0.5).any().item())\n\n loss_main = crit(pred, tgt, allow_fft=allow_fft)\n loss_id = alg_humility_penalty(pred, inp, is_id)\n loss_lowt = low_t_edit_penalty(pred, inp, meta)\n loss_auth_tv = authority_tv_penalty(aux[\"authority\"])\n\n loss = loss_main + loss_id + loss_lowt + loss_auth_tv\n\n if USE_AMP:\n scaler.scale(loss).backward()\n scaler.unscale_(opt)\n nn.utils.clip_grad_norm_(model.parameters(), GRAD_CLIP)\n scaler.step(opt)\n scaler.update()\n else:\n loss.backward()\n nn.utils.clip_grad_norm_(model.parameters(), GRAD_CLIP)\n opt.step()\n\n losses.append(float(loss.item()))\n\n if b == 1 or b % 50 == 0 or b == len(train_loader):\n print(\n f\" [Epoch {ep:02d} | Batch {b:03d}/{len(train_loader)}] \"\n f\"Loss={np.mean(losses[-20:]):.4f} \"\n f\"(main={float(loss_main.item()):.4f}, id={float(loss_id.item()):.4f}, \"\n f\"lowt={float(loss_lowt.item()):.4f}, auth_tv={float(loss_auth_tv.item()):.6f})\"\n )\n\n sched.step()\n print(f\"Epoch {ep:02d} done | Loss={np.mean(losses):.5f} | Time={(time.time() - t0)/60:.1f} min\")\n\n if ep % 3 == 0 or ep == EPOCHS:\n psnr_val = eval_model_psnr(\n model, val_uids, RSNA_DATA_ROOT, (TARGET_D, TARGET_H, TARGET_W), max_uids=8\n )\n if psnr_val is not None:\n if psnr_val > BEST_PSNR:\n BEST_PSNR = psnr_val\n torch.save(\n {\n \"model\": model.state_dict(),\n \"train_uids\": train_uids,\n \"val_uids\": val_uids,\n \"epoch\": ep,\n \"best_val_psnr\": BEST_PSNR,\n \"config\": {\n \"res_min\": RES_MIN,\n \"res_max\": RES_MAX,\n \"meta_dim\": 4,\n \"film_hidden\": 64,\n \"authority_bias_init\": 2.0,\n },\n },\n SAVE_BEST,\n )\n print(f\" [Val] PSNR={psnr_val:.2f} dB ★ NEW BEST\")\n else:\n print(f\" [Val] PSNR={psnr_val:.2f} dB\")\n\ntorch.save(\n {\n \"model\": model.state_dict(),\n \"train_uids\": train_uids,\n \"val_uids\": val_uids,\n \"epoch\": EPOCHS,\n \"best_val_psnr\": BEST_PSNR,\n \"config\": {\n \"res_min\": RES_MIN,\n \"res_max\": RES_MAX,\n \"meta_dim\": 4,\n \"film_hidden\": 64,\n \"authority_bias_init\": 2.0,\n },\n },\n SAVE_LAST,\n)\n\nprint(\"\\n=== 5. Training complete ===\")\nprint(\"BEST:\", SAVE_BEST, \"| exists?\", os.path.exists(SAVE_BEST))\nprint(\"LAST:\", SAVE_LAST, \"| exists?\", os.path.exists(SAVE_LAST))\nprint(\"Best val PSNR:\", BEST_PSNR)\n\n# ============================================================\n# Reload best checkpoint — sanity check\n# ============================================================\n\nprint(\"\\n=== 6. Reload best checkpoint — sanity check ===\")\nassert os.path.exists(SAVE_BEST), f\"Missing best checkpoint: {SAVE_BEST}\"\n\nckpt = torch.load(SAVE_BEST, map_location=\"cpu\")\ntest_model = DeblurUNet25D_Ultimate(\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=RES_MIN,\n res_max=RES_MAX,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n)\ntest_model.load_state_dict(ckpt[\"model\"], strict=True)\n\nprint(\"Reload best checkpoint: OK\")\nprint(\"test_model has auth_head?\", hasattr(test_model, \"auth_head\"))\nprint(\"test_model has film_e1?\", hasattr(test_model, \"film_e1\"))\n\n# Expose model for downstream CRM / Mayo cells\nmodel_25d = model.eval()\n\nprint(\"\\n✅ All done. Downstream evaluation cells should reference: model_25d\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T17:19:49.246683Z","iopub.execute_input":"2026-03-10T17:19:49.24692Z","iopub.status.idle":"2026-03-10T19:28:14.690882Z","shell.execute_reply.started":"2026-03-10T17:19:49.246898Z","shell.execute_reply":"2026-03-10T19:28:14.689554Z"}},"outputs":[{"name":"stdout","text":"Device: cuda\n\n=== 0. 训练前检查 ===\nRSNA_DATA_ROOT exists? True\nMETA_CSV exists? True\nTRAIN_LOCALIZERS_CSV exists? True\n\n=== 1. 构建数据防火墙 ===\n训练集: 300 | 验证集: 20\n\n=== 2. 初始化数据集 ===\n\n=== 3. 初始化新版模型 ===\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/torch/backends/__init__.py:46: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)\n self.setter(val)\n","output_type":"stream"},{"name":"stdout","text":"模型参数量: 2,326,186\nhas auth_head? True\nhas film_e1? True\n\n=== 4. 开始训练 ===\n [Epoch 01 | Batch 001/400] Loss=0.0634 (main=0.0629, id=0.0000, lowt=0.0005, auth_tv=0.000000)\n [Epoch 01 | Batch 050/400] Loss=0.0408 (main=0.0501, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 01 | Batch 100/400] Loss=0.0392 (main=0.0414, id=0.0000, lowt=0.0003, auth_tv=0.000000)\n [Epoch 01 | Batch 150/400] Loss=0.0388 (main=0.0352, id=0.0000, lowt=0.0007, auth_tv=0.000000)\n [Epoch 01 | Batch 200/400] Loss=0.0411 (main=0.0331, id=0.0000, lowt=0.0005, auth_tv=0.000000)\n [Epoch 01 | Batch 250/400] Loss=0.0401 (main=0.0333, id=0.0000, lowt=0.0006, auth_tv=0.000000)\n [Epoch 01 | Batch 300/400] Loss=0.0403 (main=0.0355, id=0.0000, lowt=0.0013, auth_tv=0.000000)\n [Epoch 01 | Batch 350/400] Loss=0.0373 (main=0.0223, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 01 | Batch 400/400] Loss=0.0398 (main=0.0478, id=0.0000, lowt=0.0004, auth_tv=0.000000)\nEpoch 01 done | Loss=0.03993 | Time=8.8 min\n [Epoch 02 | Batch 001/400] Loss=0.0307 (main=0.0297, id=0.0000, lowt=0.0010, auth_tv=0.000000)\n [Epoch 02 | Batch 050/400] Loss=0.0377 (main=0.0357, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 02 | Batch 100/400] Loss=0.0387 (main=0.0336, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 02 | Batch 150/400] Loss=0.0381 (main=0.0387, id=0.0000, lowt=0.0009, auth_tv=0.000000)\n [Epoch 02 | Batch 200/400] Loss=0.0411 (main=0.0433, id=0.0000, lowt=0.0005, auth_tv=0.000000)\n [Epoch 02 | Batch 250/400] Loss=0.0398 (main=0.0373, id=0.0000, lowt=0.0019, auth_tv=0.000000)\n [Epoch 02 | Batch 300/400] Loss=0.0389 (main=0.0320, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 02 | Batch 350/400] Loss=0.0406 (main=0.0410, id=0.0000, lowt=0.0013, auth_tv=0.000001)\n [Epoch 02 | Batch 400/400] Loss=0.0378 (main=0.0264, id=0.0000, lowt=0.0011, auth_tv=0.000000)\nEpoch 02 done | Loss=0.03896 | Time=6.2 min\n [Epoch 03 | Batch 001/400] Loss=0.0332 (main=0.0327, id=0.0000, lowt=0.0005, auth_tv=0.000001)\n [Epoch 03 | Batch 050/400] Loss=0.0392 (main=0.0381, id=0.0000, lowt=0.0007, auth_tv=0.000001)\n [Epoch 03 | Batch 100/400] Loss=0.0388 (main=0.0333, id=0.0000, lowt=0.0010, auth_tv=0.000001)\n [Epoch 03 | Batch 150/400] Loss=0.0383 (main=0.0424, id=0.0000, lowt=0.0010, auth_tv=0.000002)\n [Epoch 03 | Batch 200/400] Loss=0.0351 (main=0.0326, id=0.0000, lowt=0.0010, auth_tv=0.000002)\n [Epoch 03 | Batch 250/400] Loss=0.0343 (main=0.0402, id=0.0000, lowt=0.0013, auth_tv=0.000002)\n [Epoch 03 | Batch 300/400] Loss=0.0282 (main=0.0297, id=0.0000, lowt=0.0012, auth_tv=0.000001)\n [Epoch 03 | Batch 350/400] Loss=0.0290 (main=0.0234, id=0.0000, lowt=0.0017, auth_tv=0.000001)\n [Epoch 03 | Batch 400/400] Loss=0.0295 (main=0.0333, id=0.0000, lowt=0.0014, auth_tv=0.000001)\nEpoch 03 done | Loss=0.03370 | Time=5.4 min\n [Val] PSNR=41.51 dB ★ NEW BEST\n [Epoch 04 | Batch 001/400] Loss=0.0256 (main=0.0237, id=0.0000, lowt=0.0019, auth_tv=0.000000)\n [Epoch 04 | Batch 050/400] Loss=0.0274 (main=0.0220, id=0.0000, lowt=0.0012, auth_tv=0.000000)\n [Epoch 04 | Batch 100/400] Loss=0.0248 (main=0.0321, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 04 | Batch 150/400] Loss=0.0231 (main=0.0185, id=0.0000, lowt=0.0013, auth_tv=0.000000)\n [Epoch 04 | Batch 200/400] Loss=0.0245 (main=0.0149, id=0.0000, lowt=0.0019, auth_tv=0.000000)\n [Epoch 04 | Batch 250/400] Loss=0.0244 (main=0.0238, id=0.0000, lowt=0.0016, auth_tv=0.000001)\n [Epoch 04 | Batch 300/400] Loss=0.0259 (main=0.0144, id=0.0000, lowt=0.0009, auth_tv=0.000000)\n [Epoch 04 | Batch 350/400] Loss=0.0221 (main=0.0158, id=0.0000, lowt=0.0026, auth_tv=0.000000)\n [Epoch 04 | Batch 400/400] Loss=0.0216 (main=0.0174, id=0.0000, lowt=0.0008, auth_tv=0.000000)\nEpoch 04 done | Loss=0.02378 | Time=5.0 min\n [Epoch 05 | Batch 001/400] Loss=0.0186 (main=0.0155, id=0.0000, lowt=0.0031, auth_tv=0.000000)\n [Epoch 05 | Batch 050/400] Loss=0.0225 (main=0.0170, id=0.0000, lowt=0.0024, auth_tv=0.000000)\n [Epoch 05 | Batch 100/400] Loss=0.0203 (main=0.0134, id=0.0000, lowt=0.0015, auth_tv=0.000000)\n [Epoch 05 | Batch 150/400] Loss=0.0229 (main=0.0501, id=0.0000, lowt=0.0009, auth_tv=0.000001)\n [Epoch 05 | Batch 200/400] Loss=0.0201 (main=0.0258, id=0.0000, lowt=0.0014, auth_tv=0.000001)\n [Epoch 05 | Batch 250/400] Loss=0.0204 (main=0.0254, id=0.0000, lowt=0.0010, auth_tv=0.000001)\n [Epoch 05 | Batch 300/400] Loss=0.0223 (main=0.0224, id=0.0000, lowt=0.0043, auth_tv=0.000000)\n [Epoch 05 | Batch 350/400] Loss=0.0199 (main=0.0184, id=0.0000, lowt=0.0018, auth_tv=0.000000)\n [Epoch 05 | Batch 400/400] Loss=0.0211 (main=0.0205, id=0.0000, lowt=0.0022, auth_tv=0.000000)\nEpoch 05 done | Loss=0.02150 | Time=4.8 min\n [Epoch 06 | Batch 001/400] Loss=0.0270 (main=0.0235, id=0.0000, lowt=0.0035, auth_tv=0.000000)\n [Epoch 06 | Batch 050/400] Loss=0.0211 (main=0.0230, id=0.0000, lowt=0.0020, auth_tv=0.000001)\n [Epoch 06 | Batch 100/400] Loss=0.0213 (main=0.0167, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 06 | Batch 150/400] Loss=0.0230 (main=0.0223, id=0.0000, lowt=0.0013, auth_tv=0.000001)\n [Epoch 06 | Batch 200/400] Loss=0.0208 (main=0.0295, id=0.0000, lowt=0.0022, auth_tv=0.000000)\n [Epoch 06 | Batch 250/400] Loss=0.0216 (main=0.0167, id=0.0000, lowt=0.0035, auth_tv=0.000001)\n [Epoch 06 | Batch 300/400] Loss=0.0194 (main=0.0243, id=0.0000, lowt=0.0027, auth_tv=0.000001)\n [Epoch 06 | Batch 350/400] Loss=0.0209 (main=0.0130, id=0.0000, lowt=0.0044, auth_tv=0.000000)\n [Epoch 06 | Batch 400/400] Loss=0.0178 (main=0.0176, id=0.0000, lowt=0.0027, auth_tv=0.000001)\nEpoch 06 done | Loss=0.02081 | Time=5.0 min\n [Val] PSNR=44.37 dB ★ NEW BEST\n [Epoch 07 | Batch 001/400] Loss=0.0330 (main=0.0302, id=0.0000, lowt=0.0028, auth_tv=0.000000)\n [Epoch 07 | Batch 050/400] Loss=0.0203 (main=0.0196, id=0.0000, lowt=0.0032, auth_tv=0.000001)\n [Epoch 07 | Batch 100/400] Loss=0.0181 (main=0.0202, id=0.0000, lowt=0.0025, auth_tv=0.000001)\n [Epoch 07 | Batch 150/400] Loss=0.0198 (main=0.0229, id=0.0000, lowt=0.0024, auth_tv=0.000001)\n [Epoch 07 | Batch 200/400] Loss=0.0188 (main=0.0175, id=0.0000, lowt=0.0030, auth_tv=0.000000)\n [Epoch 07 | Batch 250/400] Loss=0.0190 (main=0.0158, id=0.0000, lowt=0.0028, auth_tv=0.000000)\n [Epoch 07 | Batch 300/400] Loss=0.0199 (main=0.0156, id=0.0000, lowt=0.0019, auth_tv=0.000000)\n [Epoch 07 | Batch 350/400] Loss=0.0190 (main=0.0172, id=0.0000, lowt=0.0008, auth_tv=0.000001)\n [Epoch 07 | Batch 400/400] Loss=0.0196 (main=0.0125, id=0.0000, lowt=0.0019, auth_tv=0.000001)\nEpoch 07 done | Loss=0.01964 | Time=4.8 min\n [Epoch 08 | Batch 001/400] Loss=0.0239 (main=0.0221, id=0.0000, lowt=0.0017, auth_tv=0.000001)\n [Epoch 08 | Batch 050/400] Loss=0.0195 (main=0.0179, id=0.0000, lowt=0.0008, auth_tv=0.000001)\n [Epoch 08 | Batch 100/400] Loss=0.0205 (main=0.0176, id=0.0000, lowt=0.0022, auth_tv=0.000001)\n [Epoch 08 | Batch 150/400] Loss=0.0217 (main=0.0162, id=0.0000, lowt=0.0045, auth_tv=0.000001)\n [Epoch 08 | Batch 200/400] Loss=0.0193 (main=0.0142, id=0.0000, lowt=0.0016, auth_tv=0.000001)\n [Epoch 08 | Batch 250/400] Loss=0.0198 (main=0.0174, id=0.0000, lowt=0.0024, auth_tv=0.000001)\n [Epoch 08 | Batch 300/400] Loss=0.0201 (main=0.0175, id=0.0000, lowt=0.0008, auth_tv=0.000001)\n [Epoch 08 | Batch 350/400] Loss=0.0176 (main=0.0102, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 08 | Batch 400/400] Loss=0.0182 (main=0.0158, id=0.0000, lowt=0.0025, auth_tv=0.000001)\nEpoch 08 done | Loss=0.01956 | Time=5.0 min\n [Epoch 09 | Batch 001/400] Loss=0.0197 (main=0.0160, id=0.0000, lowt=0.0036, auth_tv=0.000001)\n [Epoch 09 | Batch 050/400] Loss=0.0183 (main=0.0152, id=0.0000, lowt=0.0019, auth_tv=0.000001)\n [Epoch 09 | Batch 100/400] Loss=0.0187 (main=0.0162, id=0.0000, lowt=0.0023, auth_tv=0.000000)\n [Epoch 09 | Batch 150/400] Loss=0.0177 (main=0.0194, id=0.0000, lowt=0.0017, auth_tv=0.000001)\n [Epoch 09 | Batch 200/400] Loss=0.0190 (main=0.0162, id=0.0000, lowt=0.0000, auth_tv=0.000001)\n [Epoch 09 | Batch 250/400] Loss=0.0185 (main=0.0162, id=0.0000, lowt=0.0008, auth_tv=0.000001)\n [Epoch 09 | Batch 300/400] Loss=0.0180 (main=0.0147, id=0.0000, lowt=0.0006, auth_tv=0.000001)\n [Epoch 09 | Batch 350/400] Loss=0.0191 (main=0.0195, id=0.0000, lowt=0.0031, auth_tv=0.000001)\n [Epoch 09 | Batch 400/400] Loss=0.0192 (main=0.0111, id=0.0000, lowt=0.0012, auth_tv=0.000001)\nEpoch 09 done | Loss=0.01849 | Time=4.9 min\n [Val] PSNR=44.74 dB ★ NEW BEST\n [Epoch 10 | Batch 001/400] Loss=0.0265 (main=0.0233, id=0.0000, lowt=0.0032, auth_tv=0.000001)\n [Epoch 10 | Batch 050/400] Loss=0.0167 (main=0.0117, id=0.0000, lowt=0.0015, auth_tv=0.000001)\n [Epoch 10 | Batch 100/400] Loss=0.0176 (main=0.0152, id=0.0000, lowt=0.0020, auth_tv=0.000001)\n [Epoch 10 | Batch 150/400] Loss=0.0169 (main=0.0143, id=0.0000, lowt=0.0013, auth_tv=0.000001)\n [Epoch 10 | Batch 200/400] Loss=0.0177 (main=0.0140, id=0.0000, lowt=0.0026, auth_tv=0.000001)\n [Epoch 10 | Batch 250/400] Loss=0.0189 (main=0.0198, id=0.0000, lowt=0.0022, auth_tv=0.000001)\n [Epoch 10 | Batch 300/400] Loss=0.0169 (main=0.0109, id=0.0000, lowt=0.0029, auth_tv=0.000001)\n [Epoch 10 | Batch 350/400] Loss=0.0199 (main=0.0135, id=0.0000, lowt=0.0021, auth_tv=0.000000)\n [Epoch 10 | Batch 400/400] Loss=0.0185 (main=0.0139, id=0.0000, lowt=0.0000, auth_tv=0.000001)\nEpoch 10 done | Loss=0.01799 | Time=4.8 min\n [Epoch 11 | Batch 001/400] Loss=0.0231 (main=0.0221, id=0.0000, lowt=0.0010, auth_tv=0.000001)\n [Epoch 11 | Batch 050/400] Loss=0.0201 (main=0.0157, id=0.0000, lowt=0.0003, auth_tv=0.000001)\n [Epoch 11 | Batch 100/400] Loss=0.0165 (main=0.0121, id=0.0000, lowt=0.0011, auth_tv=0.000000)\n [Epoch 11 | Batch 150/400] Loss=0.0187 (main=0.0176, id=0.0000, lowt=0.0020, auth_tv=0.000001)\n [Epoch 11 | Batch 200/400] Loss=0.0196 (main=0.0183, id=0.0000, lowt=0.0039, auth_tv=0.000001)\n [Epoch 11 | Batch 250/400] Loss=0.0176 (main=0.0180, id=0.0000, lowt=0.0023, auth_tv=0.000000)\n [Epoch 11 | Batch 300/400] Loss=0.0186 (main=0.0156, id=0.0000, lowt=0.0009, auth_tv=0.000001)\n [Epoch 11 | Batch 350/400] Loss=0.0181 (main=0.0119, id=0.0000, lowt=0.0044, auth_tv=0.000000)\n [Epoch 11 | Batch 400/400] Loss=0.0176 (main=0.0122, id=0.0000, lowt=0.0023, auth_tv=0.000000)\nEpoch 11 done | Loss=0.01800 | Time=4.9 min\n [Epoch 12 | Batch 001/400] Loss=0.0214 (main=0.0184, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 12 | Batch 050/400] Loss=0.0167 (main=0.0134, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 12 | Batch 100/400] Loss=0.0181 (main=0.0178, id=0.0000, lowt=0.0025, auth_tv=0.000001)\n [Epoch 12 | Batch 150/400] Loss=0.0178 (main=0.0185, id=0.0000, lowt=0.0020, auth_tv=0.000001)\n [Epoch 12 | Batch 200/400] Loss=0.0178 (main=0.0119, id=0.0000, lowt=0.0011, auth_tv=0.000000)\n [Epoch 12 | Batch 250/400] Loss=0.0173 (main=0.0212, id=0.0000, lowt=0.0011, auth_tv=0.000000)\n [Epoch 12 | Batch 300/400] Loss=0.0173 (main=0.0151, id=0.0000, lowt=0.0023, auth_tv=0.000001)\n [Epoch 12 | Batch 350/400] Loss=0.0178 (main=0.0115, id=0.0000, lowt=0.0047, auth_tv=0.000000)\n [Epoch 12 | Batch 400/400] Loss=0.0180 (main=0.0172, id=0.0000, lowt=0.0027, auth_tv=0.000001)\nEpoch 12 done | Loss=0.01763 | Time=4.8 min\n [Val] PSNR=45.62 dB ★ NEW BEST\n [Epoch 13 | Batch 001/400] Loss=0.0139 (main=0.0122, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 13 | Batch 050/400] Loss=0.0175 (main=0.0186, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 13 | Batch 100/400] Loss=0.0182 (main=0.0134, id=0.0000, lowt=0.0027, auth_tv=0.000000)\n [Epoch 13 | Batch 150/400] Loss=0.0177 (main=0.0122, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 13 | Batch 200/400] Loss=0.0173 (main=0.0108, id=0.0000, lowt=0.0000, auth_tv=0.000000)\n [Epoch 13 | Batch 250/400] Loss=0.0190 (main=0.0193, id=0.0000, lowt=0.0023, auth_tv=0.000000)\n [Epoch 13 | Batch 300/400] Loss=0.0167 (main=0.0135, id=0.0000, lowt=0.0009, auth_tv=0.000000)\n [Epoch 13 | Batch 350/400] Loss=0.0174 (main=0.0162, id=0.0000, lowt=0.0019, auth_tv=0.000000)\n [Epoch 13 | Batch 400/400] Loss=0.0164 (main=0.0104, id=0.0000, lowt=0.0035, auth_tv=0.000000)\nEpoch 13 done | Loss=0.01730 | Time=4.9 min\n [Epoch 14 | Batch 001/400] Loss=0.0206 (main=0.0186, id=0.0000, lowt=0.0020, auth_tv=0.000001)\n [Epoch 14 | Batch 050/400] Loss=0.0173 (main=0.0119, id=0.0000, lowt=0.0016, auth_tv=0.000000)\n [Epoch 14 | Batch 100/400] Loss=0.0173 (main=0.0102, id=0.0000, lowt=0.0023, auth_tv=0.000000)\n [Epoch 14 | Batch 150/400] Loss=0.0166 (main=0.0145, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 14 | Batch 200/400] Loss=0.0162 (main=0.0120, id=0.0000, lowt=0.0031, auth_tv=0.000000)\n [Epoch 14 | Batch 250/400] Loss=0.0173 (main=0.0184, id=0.0000, lowt=0.0030, auth_tv=0.000000)\n [Epoch 14 | Batch 300/400] Loss=0.0170 (main=0.0182, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 14 | Batch 350/400] Loss=0.0171 (main=0.0102, id=0.0000, lowt=0.0023, auth_tv=0.000000)\n [Epoch 14 | Batch 400/400] Loss=0.0167 (main=0.0139, id=0.0000, lowt=0.0025, auth_tv=0.000000)\nEpoch 14 done | Loss=0.01709 | Time=5.1 min\n [Epoch 15 | Batch 001/400] Loss=0.0164 (main=0.0144, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 15 | Batch 050/400] Loss=0.0165 (main=0.0104, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 15 | Batch 100/400] Loss=0.0171 (main=0.0080, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 15 | Batch 150/400] Loss=0.0159 (main=0.0118, id=0.0000, lowt=0.0026, auth_tv=0.000000)\n [Epoch 15 | Batch 200/400] Loss=0.0171 (main=0.0111, id=0.0000, lowt=0.0028, auth_tv=0.000000)\n [Epoch 15 | Batch 250/400] Loss=0.0163 (main=0.0131, id=0.0000, lowt=0.0035, auth_tv=0.000000)\n [Epoch 15 | Batch 300/400] Loss=0.0176 (main=0.0164, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 15 | Batch 350/400] Loss=0.0163 (main=0.0127, id=0.0000, lowt=0.0007, auth_tv=0.000000)\n [Epoch 15 | Batch 400/400] Loss=0.0176 (main=0.0143, id=0.0000, lowt=0.0018, auth_tv=0.000000)\nEpoch 15 done | Loss=0.01677 | Time=4.9 min\n [Val] PSNR=45.92 dB ★ NEW BEST\n [Epoch 16 | Batch 001/400] Loss=0.0145 (main=0.0110, id=0.0000, lowt=0.0034, auth_tv=0.000000)\n [Epoch 16 | Batch 050/400] Loss=0.0166 (main=0.0164, id=0.0000, lowt=0.0028, auth_tv=0.000000)\n [Epoch 16 | Batch 100/400] Loss=0.0177 (main=0.0168, id=0.0000, lowt=0.0013, auth_tv=0.000000)\n [Epoch 16 | Batch 150/400] Loss=0.0178 (main=0.0160, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 16 | Batch 200/400] Loss=0.0162 (main=0.0115, id=0.0000, lowt=0.0006, auth_tv=0.000000)\n [Epoch 16 | Batch 250/400] Loss=0.0167 (main=0.0179, id=0.0000, lowt=0.0010, auth_tv=0.000000)\n [Epoch 16 | Batch 300/400] Loss=0.0162 (main=0.0163, id=0.0000, lowt=0.0013, auth_tv=0.000000)\n [Epoch 16 | Batch 350/400] Loss=0.0171 (main=0.0184, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 16 | Batch 400/400] Loss=0.0154 (main=0.0141, id=0.0000, lowt=0.0014, auth_tv=0.000000)\nEpoch 16 done | Loss=0.01685 | Time=4.9 min\n [Epoch 17 | Batch 001/400] Loss=0.0180 (main=0.0160, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 17 | Batch 050/400] Loss=0.0161 (main=0.0163, id=0.0000, lowt=0.0026, auth_tv=0.000000)\n [Epoch 17 | Batch 100/400] Loss=0.0174 (main=0.0201, id=0.0000, lowt=0.0032, auth_tv=0.000000)\n [Epoch 17 | Batch 150/400] Loss=0.0171 (main=0.0099, id=0.0000, lowt=0.0024, auth_tv=0.000000)\n [Epoch 17 | Batch 200/400] Loss=0.0184 (main=0.0137, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 17 | Batch 250/400] Loss=0.0159 (main=0.0118, id=0.0000, lowt=0.0024, auth_tv=0.000000)\n [Epoch 17 | Batch 300/400] Loss=0.0162 (main=0.0113, id=0.0000, lowt=0.0033, auth_tv=0.000000)\n [Epoch 17 | Batch 350/400] Loss=0.0168 (main=0.0137, id=0.0000, lowt=0.0035, auth_tv=0.000000)\n [Epoch 17 | Batch 400/400] Loss=0.0179 (main=0.0162, id=0.0000, lowt=0.0021, auth_tv=0.000000)\nEpoch 17 done | Loss=0.01693 | Time=4.9 min\n [Epoch 18 | Batch 001/400] Loss=0.0153 (main=0.0133, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 18 | Batch 050/400] Loss=0.0165 (main=0.0166, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 18 | Batch 100/400] Loss=0.0168 (main=0.0161, id=0.0000, lowt=0.0015, auth_tv=0.000000)\n [Epoch 18 | Batch 150/400] Loss=0.0158 (main=0.0156, id=0.0000, lowt=0.0016, auth_tv=0.000000)\n [Epoch 18 | Batch 200/400] Loss=0.0168 (main=0.0156, id=0.0000, lowt=0.0008, auth_tv=0.000000)\n [Epoch 18 | Batch 250/400] Loss=0.0164 (main=0.0088, id=0.0000, lowt=0.0038, auth_tv=0.000000)\n [Epoch 18 | Batch 300/400] Loss=0.0181 (main=0.0147, id=0.0000, lowt=0.0046, auth_tv=0.000000)\n [Epoch 18 | Batch 350/400] Loss=0.0163 (main=0.0104, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 18 | Batch 400/400] Loss=0.0178 (main=0.0156, id=0.0000, lowt=0.0014, auth_tv=0.000000)\nEpoch 18 done | Loss=0.01679 | Time=5.0 min\n [Val] PSNR=45.97 dB ★ NEW BEST\n [Epoch 19 | Batch 001/400] Loss=0.0171 (main=0.0161, id=0.0000, lowt=0.0011, auth_tv=0.000000)\n [Epoch 19 | Batch 050/400] Loss=0.0175 (main=0.0123, id=0.0000, lowt=0.0015, auth_tv=0.000000)\n [Epoch 19 | Batch 100/400] Loss=0.0163 (main=0.0134, id=0.0000, lowt=0.0015, auth_tv=0.000000)\n [Epoch 19 | Batch 150/400] Loss=0.0158 (main=0.0128, id=0.0000, lowt=0.0016, auth_tv=0.000000)\n [Epoch 19 | Batch 200/400] Loss=0.0175 (main=0.0167, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 19 | Batch 250/400] Loss=0.0164 (main=0.0193, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 19 | Batch 300/400] Loss=0.0166 (main=0.0115, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 19 | Batch 350/400] Loss=0.0152 (main=0.0155, id=0.0000, lowt=0.0012, auth_tv=0.000000)\n [Epoch 19 | Batch 400/400] Loss=0.0177 (main=0.0162, id=0.0000, lowt=0.0009, auth_tv=0.000000)\nEpoch 19 done | Loss=0.01655 | Time=4.7 min\n [Epoch 20 | Batch 001/400] Loss=0.0131 (main=0.0114, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 20 | Batch 050/400] Loss=0.0170 (main=0.0124, id=0.0000, lowt=0.0009, auth_tv=0.000000)\n [Epoch 20 | Batch 100/400] Loss=0.0161 (main=0.0157, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 20 | Batch 150/400] Loss=0.0174 (main=0.0116, id=0.0000, lowt=0.0014, auth_tv=0.000000)\n [Epoch 20 | Batch 200/400] Loss=0.0160 (main=0.0122, id=0.0000, lowt=0.0018, auth_tv=0.000000)\n [Epoch 20 | Batch 250/400] Loss=0.0159 (main=0.0152, id=0.0000, lowt=0.0046, auth_tv=0.000000)\n [Epoch 20 | Batch 300/400] Loss=0.0153 (main=0.0148, id=0.0000, lowt=0.0015, auth_tv=0.000000)\n [Epoch 20 | Batch 350/400] Loss=0.0169 (main=0.0096, id=0.0000, lowt=0.0028, auth_tv=0.000000)\n [Epoch 20 | Batch 400/400] Loss=0.0168 (main=0.0136, id=0.0000, lowt=0.0014, auth_tv=0.000000)\nEpoch 20 done | Loss=0.01648 | Time=4.8 min\n [Epoch 21 | Batch 001/400] Loss=0.0112 (main=0.0094, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 21 | Batch 050/400] Loss=0.0171 (main=0.0115, id=0.0000, lowt=0.0024, auth_tv=0.000000)\n [Epoch 21 | Batch 100/400] Loss=0.0165 (main=0.0144, id=0.0000, lowt=0.0020, auth_tv=0.000000)\n [Epoch 21 | Batch 150/400] Loss=0.0168 (main=0.0157, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 21 | Batch 200/400] Loss=0.0165 (main=0.0120, id=0.0000, lowt=0.0018, auth_tv=0.000000)\n [Epoch 21 | Batch 250/400] Loss=0.0161 (main=0.0093, id=0.0000, lowt=0.0009, auth_tv=0.000000)\n [Epoch 21 | Batch 300/400] Loss=0.0161 (main=0.0133, id=0.0000, lowt=0.0013, auth_tv=0.000000)\n [Epoch 21 | Batch 350/400] Loss=0.0169 (main=0.0081, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 21 | Batch 400/400] Loss=0.0169 (main=0.0133, id=0.0000, lowt=0.0029, auth_tv=0.000000)\nEpoch 21 done | Loss=0.01654 | Time=5.0 min\n [Val] PSNR=46.27 dB ★ NEW BEST\n [Epoch 22 | Batch 001/400] Loss=0.0155 (main=0.0130, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 22 | Batch 050/400] Loss=0.0190 (main=0.0111, id=0.0000, lowt=0.0021, auth_tv=0.000000)\n [Epoch 22 | Batch 100/400] Loss=0.0171 (main=0.0216, id=0.0000, lowt=0.0026, auth_tv=0.000000)\n [Epoch 22 | Batch 150/400] Loss=0.0157 (main=0.0133, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 22 | Batch 200/400] Loss=0.0160 (main=0.0156, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 22 | Batch 250/400] Loss=0.0165 (main=0.0173, id=0.0000, lowt=0.0021, auth_tv=0.000000)\n [Epoch 22 | Batch 300/400] Loss=0.0170 (main=0.0179, id=0.0000, lowt=0.0045, auth_tv=0.000000)\n [Epoch 22 | Batch 350/400] Loss=0.0164 (main=0.0139, id=0.0000, lowt=0.0021, auth_tv=0.000000)\n [Epoch 22 | Batch 400/400] Loss=0.0169 (main=0.0135, id=0.0000, lowt=0.0009, auth_tv=0.000000)\nEpoch 22 done | Loss=0.01668 | Time=5.3 min\n [Epoch 23 | Batch 001/400] Loss=0.0156 (main=0.0139, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 23 | Batch 050/400] Loss=0.0165 (main=0.0123, id=0.0000, lowt=0.0027, auth_tv=0.000000)\n [Epoch 23 | Batch 100/400] Loss=0.0163 (main=0.0069, id=0.0000, lowt=0.0017, auth_tv=0.000000)\n [Epoch 23 | Batch 150/400] Loss=0.0166 (main=0.0109, id=0.0000, lowt=0.0004, auth_tv=0.000000)\n [Epoch 23 | Batch 200/400] Loss=0.0158 (main=0.0131, id=0.0000, lowt=0.0044, auth_tv=0.000000)\n [Epoch 23 | Batch 250/400] Loss=0.0164 (main=0.0165, id=0.0000, lowt=0.0012, auth_tv=0.000000)\n [Epoch 23 | Batch 300/400] Loss=0.0157 (main=0.0103, id=0.0000, lowt=0.0034, auth_tv=0.000000)\n [Epoch 23 | Batch 350/400] Loss=0.0160 (main=0.0127, id=0.0000, lowt=0.0044, auth_tv=0.000000)\n [Epoch 23 | Batch 400/400] Loss=0.0183 (main=0.0128, id=0.0000, lowt=0.0039, auth_tv=0.000000)\nEpoch 23 done | Loss=0.01655 | Time=4.8 min\n [Epoch 24 | Batch 001/400] Loss=0.0158 (main=0.0133, id=0.0000, lowt=0.0025, auth_tv=0.000000)\n [Epoch 24 | Batch 050/400] Loss=0.0168 (main=0.0089, id=0.0000, lowt=0.0031, auth_tv=0.000000)\n [Epoch 24 | Batch 100/400] Loss=0.0173 (main=0.0157, id=0.0000, lowt=0.0012, auth_tv=0.000000)\n [Epoch 24 | Batch 150/400] Loss=0.0173 (main=0.0133, id=0.0000, lowt=0.0029, auth_tv=0.000000)\n [Epoch 24 | Batch 200/400] Loss=0.0158 (main=0.0136, id=0.0000, lowt=0.0010, auth_tv=0.000000)\n [Epoch 24 | Batch 250/400] Loss=0.0177 (main=0.0163, id=0.0000, lowt=0.0024, auth_tv=0.000000)\n [Epoch 24 | Batch 300/400] Loss=0.0163 (main=0.0172, id=0.0000, lowt=0.0036, auth_tv=0.000000)\n [Epoch 24 | Batch 350/400] Loss=0.0162 (main=0.0119, id=0.0000, lowt=0.0006, auth_tv=0.000000)\n [Epoch 24 | Batch 400/400] Loss=0.0156 (main=0.0158, id=0.0000, lowt=0.0039, auth_tv=0.000000)\nEpoch 24 done | Loss=0.01674 | Time=5.1 min\n [Val] PSNR=46.35 dB ★ NEW BEST\n\n=== 5. 训练完成 ===\nBEST: /kaggle/working/deblur_ultimate_film_auth_best.pt | exists? True\nLAST: /kaggle/working/deblur_ultimate_film_auth_last.pt | exists? True\nbest val psnr: 46.346922057560946\n\n=== 6. 回读 best checkpoint 检查 ===\nreload best ckpt ok\ntest_model has auth_head? True\ntest_model has film_e1? True\n\n✅ 全部完成。后续评估请使用：model_25d\n","output_type":"stream"}],"execution_count":1},{"cell_type":"markdown","source":"# Scientific-Method Training Design for V-Ultimate\n\n## Why this training design was introduced\n\nThe goal of this training strategy is **not simply to make the model stronger on paper**, but to improve it in a way that is **scientifically defensible**.\n\nA common mistake in medical-AI projects is:\n\n1. train a model,\n2. test it on the final evaluation set,\n3. see where it fails,\n4. then tune the model to fix those failures.\n\nThat is **not good science**, because the test set stops being a true test. \nOnce you use the final test set to guide model improvement, the evaluation is no longer independent.\n\nThis notebook avoids that problem by introducing a **three-way internal split** and a **two-stage training procedure**.\n\n---\n\n## Core idea in one sentence\n\n> First train a general model, then analyze its failures **only on a separate internal refinement split**, and finally perform targeted refinement **without ever touching the external test set**.\n\n---\n\n## The three-way split\n\nThe data are divided into three disjoint groups:\n\n| Split | Size | Role |\n|------|------|------|\n| **Train** | 300 CT/CTA series | Used for the main/base training |\n| **Refine** | 60 CT/CTA series | Used only to discover failure modes and mine hard cases |\n| **Validation** | 20 CT/CTA series | Used to monitor whether the refined model really improves |\n| **External test** | untouched | Reserved for later CRM / Mayo / Monte Carlo evaluation only |\n\nThis design matters because each split has a different job:\n\n- **Train** teaches the model the general restoration task.\n- **Refine** acts like a “practice exam” where we are allowed to inspect mistakes.\n- **Validation** checks whether the fixes actually generalize.\n- **External test** remains completely clean and unbiased.\n\nSo the model is improved using evidence, but **without leaking information from the final test set**.\n\n---\n\n## Two-stage training pipeline\n\n## Stage A — Base learning\n\nStage A is the general training phase.\n\nThe model is trained on the **300-series Train split** using a broad mixture of simulated degradations:\n\n- clean / identity cases,\n- typical blur + quarter-dose noise,\n- harder low-photon noise,\n- motion corruption.\n\nThe purpose of Stage A is to teach the model the **overall physics restoration task**:\n\n- how blur behaves,\n- how Poisson-Gaussian noise behaves,\n- how to recover edges without breaking HU-scale structure,\n- and how to remain conservative when degradation is weak.\n\nYou can think of Stage A as teaching the model the **basic language of restoration**.\n\n### What Stage A is trying to learn\n\nThe model is not asked to hallucinate a brand-new CT slice. \nInstead, it learns a **bounded correction** to the degraded center slice.\n\nThat means Stage A is mainly about learning:\n\n- where blur tends to remove detail,\n- where noise should be reduced,\n- and where the safest action is to do almost nothing.\n\n---\n\n## Stage B — Failure-driven refinement\n\nAfter Stage A finishes, the model is **not immediately pushed to the external benchmark**.\n\nInstead, it is evaluated on the **Refine split only**. \nThis is the crucial scientific step.\n\nThe question becomes:\n\n> On a separate internal set that the model did not train on, which patients are still difficult?\n\nThose difficult cases are then mined and used for **targeted refinement**.\n\nThis is the purpose of Stage B.\n\n---\n\n## How hard cases are mined\n\nFor each UID in the **Refine split**, the Stage A model is run on controlled degraded examples. \nTwo signals are recorded:\n\n| Signal | Meaning |\n|------|------|\n| **PSNR mean** | Lower PSNR means the reconstruction is still weak |\n| **Over-edit mean** | Larger edit magnitude means the model may be changing too much |\n\nThese are combined into a **hard score**:\n\n- **low PSNR** suggests the case is hard to restore,\n- **high over-edit** suggests the model may be unstable or too aggressive.\n\nSo a hard case is not just a case where the model is inaccurate. \nIt is also a case where the model may be **unsafe**.\n\nThis is important for a “Do-No-Harm” project.\n\n---\n\n## What Stage B does differently\n\nStage B does **not** retrain from scratch. \nInstead, it starts from the **best Stage A checkpoint** and then refines it with a more targeted curriculum.\n\nThe Stage B training set is:\n\n- all original **Train UIDs**, plus\n- the mined **hard UIDs from the Refine split**.\n\nThe hard UIDs are also **repeated multiple times** (`HARD_UID_REPEAT = 3`), so the model sees them more often.\n\nThat means Stage B is telling the model:\n\n> “Keep everything you already learned, but spend extra attention on the kinds of cases that previously caused trouble.”\n\n---\n\n## Why the degradation is changed in Stage B\n\nStage B also makes the task slightly more demanding.\n\nCompared with Stage A:\n\n- the probability of **hard** degradation is increased,\n- the probability of **clean identity** cases is slightly reduced,\n- motion cases remain present,\n- and the training becomes more focused on harder examples.\n\nThis is a controlled way to make the model more robust.\n\nIt is similar to how a student studies:\n\n- first by learning normal examples,\n- then by revisiting the questions they got wrong,\n- especially the harder ones.\n\n---\n\n## Why the penalties are strengthened in Stage B\n\nStage B also strengthens the “be careful” terms:\n\n- `W_LOW_T_EDIT_STAGE_B` is increased,\n- `W_AUTH_TV_STAGE_B` is increased.\n\nThese do two things:\n\n### 1. Stronger low-degradation restraint\nIf a case has only weak degradation, the model is penalized more for making unnecessary edits.\n\nThis helps reduce the classic medical-image risk of **over-correction**.\n\n### 2. Stronger authority-map smoothness\nThe authority map is encouraged to change more smoothly across space.\n\nThat means the model is less likely to produce scattered, patchy, or noisy edit permissions.\n\nSo Stage B is not only about improving recovery. \nIt is also about improving **control**.\n\n---\n\n## Why this is scientifically stronger than “just train longer”\n\nA simple way to improve a model would be to:\n\n- make it larger,\n- train for more epochs,\n- or keep tuning until the metrics look better.\n\nBut that does not necessarily teach us **why** the model improves.\n\nThis two-stage design is stronger because it follows a real scientific logic:\n\n### Step 1 — Form a base model\nTrain a first version on a clean training split.\n\n### Step 2 — Observe failures\nEvaluate it on a separate internal split that was not used for fitting.\n\n### Step 3 — Identify failure modes\nMine the cases where the model is inaccurate or over-edits.\n\n### Step 4 — Apply a targeted intervention\nRefine training specifically toward those failure modes.\n\n### Step 5 — Re-check on validation\nUse a separate validation split to test whether the refinement actually helps.\n\nThis is much closer to the logic of a real experiment:\n\n> observe → hypothesize → intervene → verify\n\nrather than:\n\n> keep tuning until something looks good\n\n---\n\n## Why this does not leak the external test set\n\nThis point is very important.\n\nThe **external test benchmarks** — such as:\n\n- Clinical Rescue Matrix,\n- Monte Carlo stability analysis,\n- TotalSegmentator overlap,\n- Mayo cross-domain validation,\n\nare **not used** to select hard cases, tune the loss, or choose training samples.\n\nAll failure-driven refinement is based only on the internal **Refine split**.\n\nThat means when the final model is later tested on OOD or cross-domain data, those results are still meaningful.\n\nSo the notebook can honestly claim:\n\n> the model was improved using observed failure modes, but the final test evidence remained independent.\n\n---\n\n## How the training design fits the “Do-No-Harm” philosophy\n\nThis project is not trying to maximize visual sharpness at all costs.\n\nIts design says:\n\n- recover useful detail,\n- but keep edits bounded,\n- learn from difficult examples,\n- and specifically pay attention to where the model may over-edit.\n\nThat is why Stage B is not just “hard-example boosting.” \nIt is really **failure-aware safety refinement**.\n\nThe hard-case mining score includes both:\n\n- reconstruction weakness,\n- and edit aggressiveness.\n\nSo the refinement process is aimed at a model that is not only more effective, but also more clinically controlled.\n\n---\n\n## Intuitive analogy\n\nA simple analogy is a student preparing for an exam:\n\n- **Stage A**: learn the whole textbook.\n- **Refine split**: take a mock exam and inspect mistakes.\n- **Stage B**: review the mistakes and practice weak topics more often.\n- **Validation**: take another clean mock exam to see whether the review helped.\n- **Final test**: only after all that, take the real exam.\n\nThat is exactly what this training design is doing.\n\n---\n\n## What the final output means\n\nAt the end of training, the notebook saves:\n\n- `deblur_stageA_best.pt`\n- `deblur_stageB_best.pt`\n\nIn most cases, the recommended checkpoint for downstream experiments is:\n\n- **`deblur_stageB_best.pt`**\n\nbecause it includes both:\n\n1. the general restoration ability learned in Stage A, and \n2. the targeted failure-driven refinement learned in Stage B.\n\nSo the final model is not simply “a later checkpoint.” \nIt is a model that has gone through a more careful scientific improvement process.\n\n---\n\n## Short takeaway\n\n> This optimized training design makes V-Ultimate stronger in a scientifically responsible way: it first learns the general restoration task, then studies its mistakes on a separate refinement split, and finally improves itself through targeted hard-case refinement — all while keeping the external test set untouched.\n\nThat is why this design is better than simply saying, “we trained a little more and the numbers went up.”","metadata":{}},{"cell_type":"code","source":"# =====================================================================\n# V-Ultimate scientific-method training cell\n# Two-stage training:\n# Stage A: base training on TRAIN split\n# Stage B: failure-driven refinement on REFINE split only\n#\n# Goal:\n# improve the model using observed failure modes\n# without leaking information from the external test set\n# =====================================================================\n\nimport os, gc, math, time, random, json\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport pydicom\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader, Sampler\nfrom contextlib import nullcontext\nimport torch.fft\n\n# -----------------------------\n# environment\n# -----------------------------\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(\"Device:\", device)\n\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = lambda: torch.amp.autocast(\"cuda\") if USE_AMP else nullcontext()\nscaler = torch.amp.GradScaler(\"cuda\") if USE_AMP else None\n\ntorch.backends.cudnn.benchmark = True\nif device.type == \"cuda\":\n try:\n torch.backends.cuda.matmul.allow_tf32 = True\n torch.backends.cudnn.allow_tf32 = True\n torch.set_float32_matmul_precision(\"high\")\n except Exception:\n pass\n\n# -----------------------------\n# require architecture\n# -----------------------------\nif \"DeblurUNet25D_Ultimate\" not in globals():\n raise RuntimeError(\"请先运行新版 architecture cell（FiLM + authority map）。\")\n\n# ============================================================\n# config\n# ============================================================\n\n# --- paths ---\nRSNA_DATA_ROOT = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\"\nTRAIN_LOCALIZERS_CSV = \"/kaggle/input/rsna-intracranial-aneurysm-detection/train_localizers.csv\"\nMETA_CSV = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\"\n\nOUTDIR = \"/kaggle/working/vultimate_scientific\"\nos.makedirs(OUTDIR, exist_ok=True)\n\nOUT_TRAIN_UIDS = os.path.join(OUTDIR, \"train_uids.csv\")\nOUT_REFINE_UIDS = os.path.join(OUTDIR, \"refine_uids.csv\")\nOUT_VAL_UIDS = os.path.join(OUTDIR, \"val_uids.csv\")\nOUT_HARD_UIDS = os.path.join(OUTDIR, \"hard_uids_stageB.csv\")\n\nSAVE_STAGEA_BEST = os.path.join(OUTDIR, \"deblur_stageA_best.pt\")\nSAVE_STAGEA_LAST = os.path.join(OUTDIR, \"deblur_stageA_last.pt\")\nSAVE_STAGEB_BEST = os.path.join(OUTDIR, \"deblur_stageB_best.pt\")\nSAVE_STAGEB_LAST = os.path.join(OUTDIR, \"deblur_stageB_last.pt\")\n\n# --- dataset size ---\nSEED = 2026\nN_TRAIN_UIDS = 300\nN_REFINE_UIDS = 60\nN_VAL_UIDS = 20\n\n# --- stage A training ---\nEPOCHS_STAGE_A = 18\nBATCHES_PER_EPOCH_A = 300\n\n# --- stage B refinement ---\nEPOCHS_STAGE_B = 8\nBATCHES_PER_EPOCH_B = 220\nN_HARD_UIDS = 24 # 从 refine 集里挑多少个 hardest cases\nHARD_UID_REPEAT = 3 # Stage B 中对 hard UID 做重复强化\n\n# --- optimizer ---\nBATCH_SIZE = 32\nNUM_WORKERS = 4\nLR_STAGE_A = 2e-4\nLR_STAGE_B = 8e-5\nWEIGHT_DECAY = 1e-4\nGRAD_CLIP = 1.0\n\n# --- image ---\nTARGET_D, TARGET_H, TARGET_W = 64, 448, 448\nPATCH_SIZE = 112\nPATCHES_PER_SLICE = 1\n\nHU_MIN, HU_MAX = -1024.0, 3072.0\nHU_RANGE = HU_MAX - HU_MIN\n\n# --- degradation ---\nDIFFUSION_ALPHA = 0.20\nBLUR_LEVELS = [0, 1, 3, 5, 8]\nBLUR_LEVEL_MAX = float(max(BLUR_LEVELS))\nBLUR_T_MAX = BLUR_LEVEL_MAX\n\nP_IDENTITY = 0.20\nENABLE_MOTION = True\nP_MOTION = 0.15\n\nPEAK_RANGE_QUARTER, SIGMA_E_QUARTER = (3000.0, 6000.0), (0.01, 0.02)\nPEAK_RANGE_EXTREME, SIGMA_E_EXTREME = (1000.0, 3000.0), (0.02, 0.04)\n\nREGIME_PROBS_STAGE_A = {\"clean\": 0.30, \"typical\": 0.45, \"hard\": 0.15, \"motion\": 0.10}\nREGIME_PROBS_STAGE_B = {\"clean\": 0.20, \"typical\": 0.35, \"hard\": 0.30, \"motion\": 0.15}\n\n# --- anatomy-aware crop ---\nANATOMY_REJECT_TRIES = 15\nANATOMY_MEAN_TH = 0.05\nANATOMY_STD_TH = 0.02\nP_RANDOM_PATCH = 0.10\n\n# --- loss weights ---\nW_CHARBONNIER = 1.0\nW_SSIM = 0.20\nW_SOBEL = 0.10\nW_LAP = 0.05\nW_FFT = 0.05\nFFT_FCUTOFF = 0.20\nFFT_ONLY_IF_T_LE = 10.0\n\nW_CHANGE_ID = 10.0\nW_LOW_T_EDIT = 2.0\nLOW_T_THR = 0.20\nW_AUTH_TV = 1e-3\n\n# stage B 强一点，专门压过修复\nW_LOW_T_EDIT_STAGE_B = 3.0\nW_AUTH_TV_STAGE_B = 2e-3\n\nRES_MIN, RES_MAX = 0.02, 0.15\n\n# --- hard-case mining ---\nMINE_EVAL_MAX_UIDS = None # None = all refine uids\nMINE_STRIDE_Z = 8\nMINE_BLUR_LEVELS = [3, 8]\nHARD_SCORE_W_PSNR = 1.0\nHARD_SCORE_W_OVEREDIT = 2.0\n\n# ============================================================\n# seed\n# ============================================================\n\ndef seed_all(seed):\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n\nseed_all(SEED)\n\ndef _normalize_probs(d):\n s = float(sum(d.values()))\n return {k: float(v) / s for k, v in d.items()}\n\nREGIME_PROBS_STAGE_A = _normalize_probs(REGIME_PROBS_STAGE_A)\nREGIME_PROBS_STAGE_B = _normalize_probs(REGIME_PROBS_STAGE_B)\n\n# ============================================================\n# split logic\n# scientific-method version:\n# train -> stage A base learning\n# refine -> failure analysis + stage B targeted improvement\n# val -> held-out internal validation only\n# external Mayo is untouched\n# ============================================================\n\ndef build_ct_uid_lists_3way(meta_csv, rsna_series_root, localizers_csv, n_train, n_refine, n_val, seed):\n rsna_uids = set(\n [u for u in os.listdir(rsna_series_root)\n if os.path.isdir(os.path.join(rsna_series_root, u)) and not u.startswith(\".\")]\n )\n\n localizer_uids = set()\n if localizers_csv and os.path.exists(localizers_csv):\n df_loc = pd.read_csv(localizers_csv)\n localizer_uids = set(df_loc[df_loc.columns[0]].astype(str).tolist())\n\n meta = pd.read_csv(meta_csv)\n meta_ct = meta[meta[\"Modality\"].isin({\"CT\", \"CTA\"})]\n\n ct_candidates = [\n u for u in set(meta_ct[\"SeriesInstanceUID\"].astype(str))\n if u in rsna_uids and u not in localizer_uids\n ]\n\n rng = random.Random(seed)\n rng.shuffle(ct_candidates)\n\n total_need = n_train + n_refine + n_val\n picked = ct_candidates[:min(total_need, len(ct_candidates))]\n\n train_uids = picked[:min(n_train, len(picked))]\n rem = picked[len(train_uids):]\n\n refine_uids = rem[:min(n_refine, len(rem))]\n rem2 = rem[len(refine_uids):]\n\n val_uids = rem2[:min(n_val, len(rem2))]\n\n return train_uids, refine_uids, val_uids\n\n# ============================================================\n# DICOM loading\n# ============================================================\n\ndef get_sorted_dicom_files(series_path):\n files = [f for f in os.listdir(series_path) if not f.startswith(\".\")]\n pairs, ok = [], True\n for f in files:\n try:\n ds = pydicom.dcmread(os.path.join(series_path, f), stop_before_pixels=True, force=True)\n if getattr(ds, \"InstanceNumber\", None) is None:\n ok = False\n break\n pairs.append((int(ds.InstanceNumber), os.path.join(series_path, f)))\n except Exception:\n ok = False\n break\n\n if ok and len(pairs) == len(files):\n return [p[1] for p in sorted(pairs, key=lambda x: x[0])]\n\n return [os.path.join(series_path, f) for f in sorted(files)]\n\ndef load_series_volume(uid, series_root, target_shape=(64, 448, 448)):\n series_path = os.path.join(series_root, uid)\n if not os.path.isdir(series_path):\n return None\n\n dcm_files = get_sorted_dicom_files(series_path)\n tD, tH, tW = target_shape\n if len(dcm_files) < 10:\n return None\n\n if len(dcm_files) != tD:\n idxs = np.linspace(0, len(dcm_files) - 1, tD).astype(int)\n dcm_files = [dcm_files[i] for i in idxs]\n\n slices = []\n for fp in dcm_files:\n try:\n ds = pydicom.dcmread(fp, force=True)\n hu = ds.pixel_array.astype(np.float32) * float(getattr(ds, \"RescaleSlope\", 1.0)) \\\n + float(getattr(ds, \"RescaleIntercept\", 0.0))\n x = (np.clip(hu, HU_MIN, HU_MAX) - HU_MIN) / HU_RANGE\n x = cv2.resize(x, (tW, tH), interpolation=cv2.INTER_LINEAR)\n slices.append(x.astype(np.float32))\n except Exception:\n continue\n\n if len(slices) < int(0.8 * tD):\n return None\n\n while len(slices) < tD:\n slices.append(slices[-1].copy())\n\n return np.stack(slices[:tD], axis=0).astype(np.float32)\n\nclass VolumeLRU:\n def __init__(self, max_items=12):\n self.max_items = int(max_items)\n self.cache = {}\n self.order = []\n\n def get(self, key):\n if key not in self.cache:\n return None\n self.order.remove(key)\n self.order.append(key)\n return self.cache[key]\n\n def put(self, key, value):\n if key in self.cache:\n self.order.remove(key)\n self.cache[key] = value\n self.order.append(key)\n if len(self.order) > self.max_items:\n old = self.order.pop(0)\n self.cache.pop(old, None)\n\n# ============================================================\n# physics degradation\n# ============================================================\n\ndef gaussian_psf_surrogate(img01, blur_level, alpha=0.20):\n if blur_level <= 0:\n return img01\n sigma = math.sqrt(max(1e-8, 2.0 * alpha * float(blur_level)))\n return np.clip(\n cv2.GaussianBlur(img01, (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE),\n 0.0, 1.0\n )\n\ndef motion_artifact_surrogate(img01, length=None, angle=None):\n length = length or random.choice([3, 5, 7, 9, 11])\n if length <= 1:\n return img01\n angle = angle or random.uniform(0, 180)\n\n k = np.zeros((length, length), dtype=np.float32)\n c = length // 2\n cos_a, sin_a = np.cos(np.radians(angle)), np.sin(np.radians(angle))\n for i in range(length):\n x = int(c + (i - c) * cos_a)\n y = int(c + (i - c) * sin_a)\n if 0 <= x < length and 0 <= y < length:\n k[y, x] = 1.0\n if k.sum() > 0:\n k /= k.sum()\n\n return np.clip(cv2.filter2D(img01, -1, k, borderType=cv2.BORDER_REPLICATE), 0.0, 1.0)\n\ndef mixed_poisson_gaussian(img01, mode=\"quarter\"):\n if mode == \"clean\":\n return img01\n\n peak_rng, sigma_rng = (\n (PEAK_RANGE_EXTREME, SIGMA_E_EXTREME) if mode == \"extreme\"\n else (PEAK_RANGE_QUARTER, SIGMA_E_QUARTER)\n )\n peak = random.uniform(*peak_rng)\n sigma_e = random.uniform(*sigma_rng)\n noisy_p = np.random.poisson(np.clip(img01 * peak, 0, None)).astype(np.float32) / peak\n noisy_g = np.random.randn(*img01.shape).astype(np.float32) * sigma_e\n return np.clip(noisy_p + noisy_g, 0.0, 1.0)\n\ndef _choice_weighted(prob_dict):\n r = random.random()\n acc = 0.0\n for k, p in prob_dict.items():\n acc += p\n if r <= acc:\n return k\n return list(prob_dict.keys())[-1]\n\ndef _sample_regime_params(prob_dict):\n reg = _choice_weighted(prob_dict)\n if reg == \"clean\":\n return 0, \"clean\", False\n if reg == \"typical\":\n return random.choice([1, 3, 5]), \"quarter\", False\n if reg == \"hard\":\n return random.choice([3, 5, 8]), \"extreme\", False\n if reg == \"motion\":\n return random.choice([1, 3, 5]), random.choice([\"quarter\", \"extreme\"]), True\n return 3, \"quarter\", False\n\n# ============================================================\n# datasets\n# ============================================================\n\nclass CTDeblur25D(Dataset):\n def __init__(\n self,\n uids,\n series_root,\n target_shape=(64, 448, 448),\n patch_size=112,\n patches_per_slice=1,\n regime_probs=None,\n p_identity=0.20,\n enable_motion=True,\n p_motion=0.15,\n hard_uid_set=None,\n hard_uid_repeat=1,\n ):\n self.uids = list(uids)\n self.series_root = series_root\n self.target_shape = target_shape\n self.patch_size = int(patch_size)\n self.patches_per_slice = int(patches_per_slice)\n self.regime_probs = _normalize_probs(regime_probs or REGIME_PROBS_STAGE_A)\n self.p_identity = float(p_identity)\n self.enable_motion = bool(enable_motion)\n self.p_motion = float(p_motion)\n self.cache = VolumeLRU(max_items=8)\n\n self.hard_uid_set = set(hard_uid_set or [])\n self.hard_uid_repeat = int(max(1, hard_uid_repeat))\n\n items = []\n for ui in range(len(self.uids)):\n uid = self.uids[ui]\n rep = self.hard_uid_repeat if uid in self.hard_uid_set else 1\n for _ in range(rep):\n items.extend([(ui, z) for z in range(1, target_shape[0] - 1)])\n self.items = items\n\n def __len__(self):\n return len(self.items)\n\n def _sample_patch_xy(self, cent, ps):\n H, W = cent.shape\n if random.random() < P_RANDOM_PATCH:\n return np.random.randint(0, H - ps + 1), np.random.randint(0, W - ps + 1)\n\n for _ in range(ANATOMY_REJECT_TRIES):\n y = np.random.randint(0, H - ps + 1)\n x = np.random.randint(0, W - ps + 1)\n patch = cent[y:y+ps, x:x+ps]\n if patch.mean() > ANATOMY_MEAN_TH and patch.std() > ANATOMY_STD_TH:\n return y, x\n\n return (H - ps) // 2, (W - ps) // 2\n\n def __getitem__(self, idx):\n ui, z = self.items[idx]\n uid = self.uids[ui]\n\n vol = self.cache.get(uid)\n if vol is None:\n vol = load_series_volume(uid, self.series_root, self.target_shape)\n if vol is None:\n return self.__getitem__(random.randint(0, len(self.items) - 1))\n self.cache.put(uid, vol)\n\n ps = self.patch_size\n y, x = self._sample_patch_xy(vol[z], ps)\n\n clean = vol[z][y:y+ps, x:x+ps].copy()\n pp = vol[z-1][y:y+ps, x:x+ps].copy()\n cc = vol[z][y:y+ps, x:x+ps].copy()\n nn_ = vol[z+1][y:y+ps, x:x+ps].copy()\n\n is_identity = 0.0\n if random.random() < self.p_identity:\n blur_level, dose_mode, do_motion, is_identity = 0, \"clean\", False, 1.0\n else:\n blur_level, dose_mode, do_motion = _sample_regime_params(self.regime_probs)\n do_motion = do_motion and self.enable_motion and (random.random() < self.p_motion)\n\n bp = gaussian_psf_surrogate(pp, blur_level, alpha=DIFFUSION_ALPHA)\n bc = gaussian_psf_surrogate(cc, blur_level, alpha=DIFFUSION_ALPHA)\n bn = gaussian_psf_surrogate(nn_, blur_level, alpha=DIFFUSION_ALPHA)\n\n if do_motion:\n L = random.choice([3, 5, 7, 9, 11])\n A = random.uniform(0, 180)\n bp = motion_artifact_surrogate(bp, L, A)\n bc = motion_artifact_surrogate(bc, L, A)\n bn = motion_artifact_surrogate(bn, L, A)\n\n if dose_mode != \"clean\":\n bp = mixed_poisson_gaussian(bp, dose_mode)\n bc = mixed_poisson_gaussian(bc, dose_mode)\n bn = mixed_poisson_gaussian(bn, dose_mode)\n\n cp = clean.copy()\n if random.random() > 0.5:\n cp, bp, bc, bn = cp[::-1].copy(), bp[::-1].copy(), bc[::-1].copy(), bn[::-1].copy()\n if random.random() > 0.5:\n cp, bp, bc, bn = cp[:, ::-1].copy(), bp[:, ::-1].copy(), bc[:, ::-1].copy(), bn[:, ::-1].copy()\n k = random.randint(0, 3)\n if k > 0:\n cp, bp, bc, bn = np.rot90(cp, k).copy(), np.rot90(bp, k).copy(), np.rot90(bc, k).copy(), np.rot90(bn, k).copy()\n\n t_norm = float(blur_level) / BLUR_LEVEL_MAX if BLUR_LEVEL_MAX > 0 else 0.0\n\n inp = np.stack([bp, bc, bn, np.full_like(bc, t_norm, dtype=np.float32)], axis=0).astype(np.float32)\n tgt = cp[np.newaxis, ...].astype(np.float32)\n\n meta = np.array([\n t_norm,\n 1.0 if do_motion else 0.0,\n 1.0 if dose_mode == \"clean\" else 0.0,\n is_identity\n ], dtype=np.float32)\n\n return (\n torch.from_numpy(inp).float(),\n torch.from_numpy(tgt).float(),\n torch.from_numpy(meta).float()\n )\n\nclass UIDBatchSampler(Sampler):\n def __init__(self, dataset, batch_size, seed=42, batches_per_epoch=None):\n self.dataset = dataset\n self.batch_size = int(batch_size)\n self.rng = random.Random(seed)\n self.by_ui = {}\n\n for idx, (ui, z) in enumerate(dataset.items):\n self.by_ui.setdefault(ui, []).append(idx)\n\n self.ui_keys = list(self.by_ui.keys())\n self.batches_per_epoch = int(batches_per_epoch) if batches_per_epoch else len(dataset) // self.batch_size\n\n def __len__(self):\n return self.batches_per_epoch\n\n def __iter__(self):\n for _ in range(self.batches_per_epoch):\n ui = self.rng.choice(self.ui_keys)\n pool = self.by_ui[ui]\n if len(pool) >= self.batch_size:\n yield self.rng.sample(pool, self.batch_size)\n else:\n yield [self.rng.choice(pool) for _ in range(self.batch_size)]\n\n# ============================================================\n# loss\n# ============================================================\n\ndef charbonnier_loss(pred, target, eps=1e-3):\n return torch.mean(torch.sqrt((pred - target) ** 2 + eps ** 2))\n\ndef fft_spectrum_loss(pred, target, fft_mask):\n pred_fft = torch.fft.rfft2(pred.float(), dim=(-2, -1), norm=\"ortho\")\n tgt_fft = torch.fft.rfft2(target.float(), dim=(-2, -1), norm=\"ortho\")\n m = fft_mask.view(1, 1, *fft_mask.shape)\n return charbonnier_loss(torch.abs(pred_fft) * m, torch.abs(tgt_fft) * m)\n\ndef ssim_loss(pred, target, window_size=11):\n C1, C2 = 0.01 ** 2, 0.03 ** 2\n pad = window_size // 2\n\n mu_x = F.avg_pool2d(pred, window_size, stride=1, padding=pad)\n mu_y = F.avg_pool2d(target, window_size, stride=1, padding=pad)\n\n sigma_x2 = F.avg_pool2d(pred ** 2, window_size, stride=1, padding=pad) - mu_x ** 2\n sigma_y2 = F.avg_pool2d(target ** 2, window_size, stride=1, padding=pad) - mu_y ** 2\n sigma_xy = F.avg_pool2d(pred * target, window_size, stride=1, padding=pad) - mu_x * mu_y\n\n ssim_map = ((2 * mu_x * mu_y + C1) * (2 * sigma_xy + C2)) / (\n (mu_x ** 2 + mu_y ** 2 + C1) * (sigma_x2 + sigma_y2 + C2) + 1e-8\n )\n return 1.0 - ssim_map.mean()\n\ndef _make_fft_mask(H, W, fcut=0.20, device=\"cpu\"):\n fy = torch.fft.fftfreq(H, d=1.0, device=device).view(H, 1).abs()\n fx = torch.fft.rfftfreq(W, d=1.0, device=device).view(1, W // 2 + 1).abs()\n return (torch.sqrt(fx * fx + fy * fy) >= fcut).float()\n\nclass UltimatePhysicsLoss(nn.Module):\n def __init__(self, patch_size=112):\n super().__init__()\n self.sobel_x = torch.tensor([[[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]]]).view(1, 1, 3, 3).to(device)\n self.sobel_y = torch.tensor([[[-1., -2., -1.], [0., 0., 0.], [1., 2., 1.]]]).view(1, 1, 3, 3).to(device)\n self.lap = torch.tensor([[[0., 1., 0.], [1., -4., 1.], [0., 1., 0.]]]).view(1, 1, 3, 3).to(device)\n self.register_buffer(\"fft_mask\", _make_fft_mask(patch_size, patch_size, fcut=FFT_FCUTOFF, device=device))\n\n def forward(self, pred, target, allow_fft=False):\n total = W_CHARBONNIER * charbonnier_loss(pred, target)\n total += W_SSIM * ssim_loss(pred, target)\n\n p_pad = F.pad(pred, (1, 1, 1, 1), mode=\"replicate\")\n t_pad = F.pad(target, (1, 1, 1, 1), mode=\"replicate\")\n\n total += W_SOBEL * (\n charbonnier_loss(F.conv2d(p_pad, self.sobel_x), F.conv2d(t_pad, self.sobel_x))\n + charbonnier_loss(F.conv2d(p_pad, self.sobel_y), F.conv2d(t_pad, self.sobel_y))\n )\n total += W_LAP * charbonnier_loss(F.conv2d(p_pad, self.lap), F.conv2d(t_pad, self.lap))\n\n if allow_fft:\n total += W_FFT * fft_spectrum_loss(pred, target, self.fft_mask)\n return total\n\ndef alg_humility_penalty(pred, inp, is_id, weight_change_id):\n center = inp[:, 1:2]\n per_sample = torch.mean(torch.abs(pred - center), dim=(1, 2, 3))\n return torch.mean(per_sample * is_id * weight_change_id)\n\ndef low_t_edit_penalty(pred, inp, meta, weight_low_t):\n center = inp[:, 1:2]\n t_norm = meta[:, 0]\n low_mask = (t_norm <= LOW_T_THR).float()\n per_sample = torch.mean(torch.abs(pred - center), dim=(1, 2, 3))\n return torch.mean(per_sample * low_mask * weight_low_t)\n\ndef authority_tv_penalty(authority, weight_auth_tv):\n dy = torch.abs(authority[:, :, 1:, :] - authority[:, :, :-1, :]).mean()\n dx = torch.abs(authority[:, :, :, 1:] - authority[:, :, :, :-1]).mean()\n return (dx + dy) * weight_auth_tv\n\n# ============================================================\n# validation\n# ============================================================\n\n@torch.no_grad()\ndef eval_model_psnr(model, val_uids, series_root, target_shape=(64, 448, 448), max_uids=8):\n model.eval()\n scores = []\n cache = VolumeLRU(max_items=2)\n\n for uid in list(val_uids)[:max_uids]:\n vol = cache.get(uid)\n if vol is None:\n vol = load_series_volume(uid, series_root, target_shape)\n if vol is None:\n continue\n cache.put(uid, vol)\n\n D = vol.shape[0]\n for blur_level in [3, 8]:\n for z in range(1, D - 1, 8):\n cl = vol[z].astype(np.float32)\n prev = vol[z - 1].astype(np.float32)\n cent = vol[z].astype(np.float32)\n next_ = vol[z + 1].astype(np.float32)\n\n bp = mixed_poisson_gaussian(gaussian_psf_surrogate(prev, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bc = mixed_poisson_gaussian(gaussian_psf_surrogate(cent, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bn = mixed_poisson_gaussian(gaussian_psf_surrogate(next_, blur_level, DIFFUSION_ALPHA), \"quarter\")\n\n t_norm = float(blur_level) / BLUR_LEVEL_MAX\n inp_np = np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n meta_np = np.array([[t_norm, 0.0, 0.0, 0.0]], dtype=np.float32)\n\n inp_t = torch.from_numpy(inp_np).unsqueeze(0).to(device)\n meta_t = torch.from_numpy(meta_np).to(device)\n\n with AMP_CTX():\n pred = model(inp_t, meta=meta_t)[0, 0].float().cpu().numpy()\n\n mse = float(np.mean((pred - cl) ** 2))\n scores.append(99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse))\n\n return float(np.mean(scores)) if scores else None\n\n# ============================================================\n# hard-case mining\n# only on refine split\n# ============================================================\n\n@torch.no_grad()\ndef mine_hard_uids(model, refine_uids, series_root, target_shape=(64, 448, 448), max_uids=None):\n model.eval()\n cache = VolumeLRU(max_items=2)\n rows = []\n\n use_uids = list(refine_uids)\n if max_uids is not None:\n use_uids = use_uids[:max_uids]\n\n for uid in use_uids:\n vol = cache.get(uid)\n if vol is None:\n vol = load_series_volume(uid, series_root, target_shape)\n if vol is None:\n continue\n cache.put(uid, vol)\n\n D = vol.shape[0]\n psnr_scores = []\n overedit_scores = []\n\n for blur_level in MINE_BLUR_LEVELS:\n for z in range(1, D - 1, MINE_STRIDE_Z):\n cl = vol[z].astype(np.float32)\n prev = vol[z - 1].astype(np.float32)\n cent = vol[z].astype(np.float32)\n next_ = vol[z + 1].astype(np.float32)\n\n bp = mixed_poisson_gaussian(gaussian_psf_surrogate(prev, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bc = mixed_poisson_gaussian(gaussian_psf_surrogate(cent, blur_level, DIFFUSION_ALPHA), \"quarter\")\n bn = mixed_poisson_gaussian(gaussian_psf_surrogate(next_, blur_level, DIFFUSION_ALPHA), \"quarter\")\n\n t_norm = float(blur_level) / BLUR_LEVEL_MAX\n inp_np = np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n meta_np = np.array([[t_norm, 0.0, 0.0, 0.0]], dtype=np.float32)\n\n inp_t = torch.from_numpy(inp_np).unsqueeze(0).to(device)\n meta_t = torch.from_numpy(meta_np).to(device)\n\n with AMP_CTX():\n pred, aux = model(inp_t, meta=meta_t, return_aux=True)\n pred_np = pred[0, 0].float().cpu().numpy()\n\n mse = float(np.mean((pred_np - cl) ** 2))\n psnr = 99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse)\n psnr_scores.append(psnr)\n\n overedit = float(np.mean(np.abs(pred_np - bc)))\n overedit_scores.append(overedit)\n\n if len(psnr_scores) == 0:\n continue\n\n psnr_mean = float(np.mean(psnr_scores))\n overedit_mean = float(np.mean(overedit_scores))\n\n rows.append({\n \"uid\": uid,\n \"psnr_mean\": psnr_mean,\n \"overedit_mean\": overedit_mean,\n })\n\n df_hard = pd.DataFrame(rows)\n if len(df_hard) == 0:\n return [], df_hard\n\n # 分数越大越难：低 PSNR + 高 overedit\n psnr_norm = (df_hard[\"psnr_mean\"].max() - df_hard[\"psnr_mean\"])\n if psnr_norm.max() > 0:\n psnr_norm = psnr_norm / (psnr_norm.max() + 1e-8)\n\n over_norm = df_hard[\"overedit_mean\"]\n if over_norm.max() > 0:\n over_norm = over_norm / (over_norm.max() + 1e-8)\n\n df_hard[\"hard_score\"] = HARD_SCORE_W_PSNR * psnr_norm + HARD_SCORE_W_OVEREDIT * over_norm\n df_hard = df_hard.sort_values(\"hard_score\", ascending=False).reset_index(drop=True)\n\n hard_uids = df_hard[\"uid\"].tolist()[:min(N_HARD_UIDS, len(df_hard))]\n return hard_uids, df_hard\n\n# ============================================================\n# generic training loop\n# ============================================================\n\ndef run_training_stage(\n stage_name,\n model,\n train_loader,\n val_uids,\n save_best,\n save_last,\n epochs,\n lr,\n weight_low_t,\n weight_auth_tv,\n extra_metadata=None,\n):\n opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=WEIGHT_DECAY)\n sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=epochs)\n crit = UltimatePhysicsLoss(patch_size=PATCH_SIZE).to(device)\n\n best_psnr = -1.0\n hist = []\n\n print(f\"\\n=== {stage_name} ===\")\n print(\"lr:\", lr, \"| epochs:\", epochs)\n\n for ep in range(1, epochs + 1):\n model.train()\n losses = []\n t0 = time.time()\n\n for b, (inp, tgt, meta) in enumerate(train_loader, 1):\n inp = inp.to(device, non_blocking=True)\n tgt = tgt.to(device, non_blocking=True)\n meta = meta.to(device, non_blocking=True)\n\n opt.zero_grad(set_to_none=True)\n\n with AMP_CTX():\n pred, aux = model(inp, meta=meta, return_aux=True)\n\n t_scalar = meta[:, 0].mean().item() * BLUR_LEVEL_MAX\n is_motion = meta[:, 1]\n is_id = meta[:, 3]\n\n allow_fft = (t_scalar <= FFT_ONLY_IF_T_LE) and not bool((is_motion > 0.5).any().item())\n\n loss_main = crit(pred, tgt, allow_fft=allow_fft)\n loss_id = alg_humility_penalty(pred, inp, is_id, W_CHANGE_ID)\n loss_lowt = low_t_edit_penalty(pred, inp, meta, weight_low_t)\n loss_auth = authority_tv_penalty(aux[\"authority\"], weight_auth_tv)\n\n loss = loss_main + loss_id + loss_lowt + loss_auth\n\n if USE_AMP:\n scaler.scale(loss).backward()\n scaler.unscale_(opt)\n nn.utils.clip_grad_norm_(model.parameters(), GRAD_CLIP)\n scaler.step(opt)\n scaler.update()\n else:\n loss.backward()\n nn.utils.clip_grad_norm_(model.parameters(), GRAD_CLIP)\n opt.step()\n\n losses.append(float(loss.item()))\n\n if b == 1 or b % 50 == 0 or b == len(train_loader):\n print(\n f\" [{stage_name} | Epoch {ep:02d} | Batch {b:03d}/{len(train_loader)}] \"\n f\"Loss={np.mean(losses[-20:]):.4f} \"\n f\"(main={float(loss_main.item()):.4f}, id={float(loss_id.item()):.4f}, \"\n f\"lowt={float(loss_lowt.item()):.4f}, auth={float(loss_auth.item()):.6f})\"\n )\n\n sched.step()\n epoch_loss = float(np.mean(losses)) if len(losses) else np.nan\n print(f\"{stage_name} Epoch {ep:02d} done | Loss={epoch_loss:.5f} | Time={(time.time()-t0)/60:.1f} min\")\n\n row = {\"stage\": stage_name, \"epoch\": ep, \"train_loss\": epoch_loss}\n\n if ep % 3 == 0 or ep == epochs:\n psnr_val = eval_model_psnr(model, val_uids, RSNA_DATA_ROOT, (TARGET_D, TARGET_H, TARGET_W), max_uids=8)\n row[\"val_psnr\"] = psnr_val\n\n if psnr_val is not None:\n if psnr_val > best_psnr:\n best_psnr = psnr_val\n pack = {\n \"model\": model.state_dict(),\n \"epoch\": ep,\n \"best_val_psnr\": best_psnr,\n \"stage\": stage_name,\n \"extra_metadata\": extra_metadata or {},\n }\n torch.save(pack, save_best)\n print(f\" [Val] PSNR={psnr_val:.2f} dB ★ NEW BEST\")\n else:\n print(f\" [Val] PSNR={psnr_val:.2f} dB\")\n\n hist.append(row)\n\n final_pack = {\n \"model\": model.state_dict(),\n \"epoch\": epochs,\n \"best_val_psnr\": best_psnr,\n \"stage\": stage_name,\n \"extra_metadata\": extra_metadata or {},\n }\n torch.save(final_pack, save_last)\n\n return best_psnr, pd.DataFrame(hist)\n\n# ============================================================\n# 1) split data\n# ============================================================\n\nprint(\"\\n=== Step 1. Build scientific train/refine/val split ===\")\ntrain_uids, refine_uids, val_uids = build_ct_uid_lists_3way(\n META_CSV,\n RSNA_DATA_ROOT,\n TRAIN_LOCALIZERS_CSV,\n N_TRAIN_UIDS,\n N_REFINE_UIDS,\n N_VAL_UIDS,\n SEED,\n)\n\npd.DataFrame({\"SeriesInstanceUID\": train_uids}).to_csv(OUT_TRAIN_UIDS, index=False)\npd.DataFrame({\"SeriesInstanceUID\": refine_uids}).to_csv(OUT_REFINE_UIDS, index=False)\npd.DataFrame({\"SeriesInstanceUID\": val_uids}).to_csv(OUT_VAL_UIDS, index=False)\n\nprint(f\"train={len(train_uids)} | refine={len(refine_uids)} | val={len(val_uids)}\")\n\n# ============================================================\n# 2) Stage A dataset\n# ============================================================\n\nprint(\"\\n=== Step 2. Stage A dataset ===\")\ntrain_ds_A = CTDeblur25D(\n train_uids,\n RSNA_DATA_ROOT,\n (TARGET_D, TARGET_H, TARGET_W),\n PATCH_SIZE,\n PATCHES_PER_SLICE,\n regime_probs=REGIME_PROBS_STAGE_A,\n p_identity=P_IDENTITY,\n enable_motion=ENABLE_MOTION,\n p_motion=P_MOTION,\n hard_uid_set=None,\n hard_uid_repeat=1,\n)\n\ntrain_loader_A = DataLoader(\n train_ds_A,\n batch_sampler=UIDBatchSampler(train_ds_A, BATCH_SIZE, SEED, BATCHES_PER_EPOCH_A),\n num_workers=NUM_WORKERS,\n pin_memory=True,\n)\n\n# ============================================================\n# 3) init model\n# ============================================================\n\nprint(\"\\n=== Step 3. Initialize model ===\")\nmodel = DeblurUNet25D_Ultimate(\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=RES_MIN,\n res_max=RES_MAX,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n).to(device)\n\nprint(\"Trainable params:\", f\"{sum(p.numel() for p in model.parameters() if p.requires_grad):,}\")\n\n# ============================================================\n# 4) Stage A: base training\n# ============================================================\n\nstageA_meta = {\n \"seed\": SEED,\n \"train_uids\": train_uids,\n \"refine_uids\": refine_uids,\n \"val_uids\": val_uids,\n \"regime_probs\": REGIME_PROBS_STAGE_A,\n \"phase\": \"base_training\",\n}\n\nbestA, histA = run_training_stage(\n stage_name=\"StageA_Base\",\n model=model,\n train_loader=train_loader_A,\n val_uids=val_uids,\n save_best=SAVE_STAGEA_BEST,\n save_last=SAVE_STAGEA_LAST,\n epochs=EPOCHS_STAGE_A,\n lr=LR_STAGE_A,\n weight_low_t=W_LOW_T_EDIT,\n weight_auth_tv=W_AUTH_TV,\n extra_metadata=stageA_meta,\n)\n\nhistA.to_csv(os.path.join(OUTDIR, \"history_stageA.csv\"), index=False)\n\nprint(\"\\nStage A best val PSNR:\", bestA)\n\n# ============================================================\n# 5) Failure analysis on REFINE split only\n# ============================================================\n\nprint(\"\\n=== Step 5. Mine hard cases on refine split only ===\")\nif os.path.exists(SAVE_STAGEA_BEST):\n packA = torch.load(SAVE_STAGEA_BEST, map_location=\"cpu\")\n model.load_state_dict(packA[\"model\"], strict=True)\n model = model.to(device).eval()\n\nhard_uids, df_hard = mine_hard_uids(\n model,\n refine_uids,\n RSNA_DATA_ROOT,\n (TARGET_D, TARGET_H, TARGET_W),\n max_uids=MINE_EVAL_MAX_UIDS,\n)\n\ndf_hard.to_csv(os.path.join(OUTDIR, \"refine_uid_difficulty.csv\"), index=False)\npd.DataFrame({\"SeriesInstanceUID\": hard_uids}).to_csv(OUT_HARD_UIDS, index=False)\n\nprint(f\"Hard UIDs selected for Stage B: {len(hard_uids)}\")\ndisplay(df_hard.head(20))\n\n# ============================================================\n# 6) Stage B dataset\n# only uses:\n# - original TRAIN split\n# - failure-derived hard UID list from REFINE split analysis\n# no external test leakage\n# ============================================================\n\nprint(\"\\n=== Step 6. Stage B refinement dataset ===\")\nstageB_uids = list(train_uids) + list(hard_uids)\n\ntrain_ds_B = CTDeblur25D(\n stageB_uids,\n RSNA_DATA_ROOT,\n (TARGET_D, TARGET_H, TARGET_W),\n PATCH_SIZE,\n PATCHES_PER_SLICE,\n regime_probs=REGIME_PROBS_STAGE_B,\n p_identity=P_IDENTITY,\n enable_motion=ENABLE_MOTION,\n p_motion=P_MOTION,\n hard_uid_set=set(hard_uids),\n hard_uid_repeat=HARD_UID_REPEAT,\n)\n\ntrain_loader_B = DataLoader(\n train_ds_B,\n batch_sampler=UIDBatchSampler(train_ds_B, BATCH_SIZE, SEED + 101, BATCHES_PER_EPOCH_B),\n num_workers=NUM_WORKERS,\n pin_memory=True,\n)\n\nprint(f\"Stage B total UID list = {len(stageB_uids)} (train + hard refine)\")\nprint(f\"Hard UID repeat = {HARD_UID_REPEAT}\")\n\n# ============================================================\n# 7) load Stage A best and do Stage B refinement\n# ============================================================\n\nprint(\"\\n=== Step 7. Stage B refinement ===\")\nif os.path.exists(SAVE_STAGEA_BEST):\n packA = torch.load(SAVE_STAGEA_BEST, map_location=\"cpu\")\n model.load_state_dict(packA[\"model\"], strict=True)\n model = model.to(device)\n\nstageB_meta = {\n \"seed\": SEED,\n \"train_uids\": train_uids,\n \"refine_uids\": refine_uids,\n \"val_uids\": val_uids,\n \"hard_uids\": hard_uids,\n \"regime_probs\": REGIME_PROBS_STAGE_B,\n \"phase\": \"failure_driven_refinement\",\n \"note\": \"hard cases mined only from refine split, not from external test set\",\n}\n\nbestB, histB = run_training_stage(\n stage_name=\"StageB_Refine\",\n model=model,\n train_loader=train_loader_B,\n val_uids=val_uids,\n save_best=SAVE_STAGEB_BEST,\n save_last=SAVE_STAGEB_LAST,\n epochs=EPOCHS_STAGE_B,\n lr=LR_STAGE_B,\n weight_low_t=W_LOW_T_EDIT_STAGE_B,\n weight_auth_tv=W_AUTH_TV_STAGE_B,\n extra_metadata=stageB_meta,\n)\n\nhistB.to_csv(os.path.join(OUTDIR, \"history_stageB.csv\"), index=False)\n\n# ============================================================\n# 8) final summary\n# ============================================================\n\nsummary = {\n \"train_uids\": len(train_uids),\n \"refine_uids\": len(refine_uids),\n \"val_uids\": len(val_uids),\n \"hard_uids_stageB\": len(hard_uids),\n \"best_stageA_val_psnr\": bestA,\n \"best_stageB_val_psnr\": bestB,\n \"stageA_best_ckpt\": SAVE_STAGEA_BEST,\n \"stageB_best_ckpt\": SAVE_STAGEB_BEST,\n}\n\nwith open(os.path.join(OUTDIR, \"training_summary.json\"), \"w\") as f:\n json.dump(summary, f, indent=2)\n\nprint(\"\\n\" + \"=\" * 90)\nprint(\"Scientific-method training finished\")\nprint(\"=\" * 90)\nfor k, v in summary.items():\n print(f\"{k}: {v}\")\n\nprint(\"\\nSaved files:\")\nfor p in sorted(os.listdir(OUTDIR)):\n print(\" -\", os.path.join(OUTDIR, p))\n\nprint(\"\\n✅ 推荐用于后续评估的权重：\")\nprint(\" \", SAVE_STAGEB_BEST if os.path.exists(SAVE_STAGEB_BEST) else SAVE_STAGEA_BEST)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:34:38.965726Z","iopub.execute_input":"2026-03-10T19:34:38.966556Z","iopub.status.idle":"2026-03-10T21:24:38.534173Z","shell.execute_reply.started":"2026-03-10T19:34:38.966524Z","shell.execute_reply":"2026-03-10T21:24:38.533183Z"}},"outputs":[{"name":"stdout","text":"Device: cuda\n\n=== Step 1. Build scientific train/refine/val split ===\ntrain=300 | refine=60 | val=20\n\n=== Step 2. Stage A dataset ===\n\n=== Step 3. Initialize model ===\nTrainable params: 2,326,186\n\n=== StageA_Base ===\nlr: 0.0002 | epochs: 18\n [StageA_Base | Epoch 01 | Batch 001/300] Loss=0.0564 (main=0.0559, id=0.0000, lowt=0.0005, auth=0.000000)\n [StageA_Base | Epoch 01 | Batch 050/300] Loss=0.0412 (main=0.0481, id=0.0000, lowt=0.0010, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 01 | Batch 100/300] Loss=0.0401 (main=0.0464, id=0.0000, lowt=0.0003, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 01 | Batch 150/300] Loss=0.0386 (main=0.0342, id=0.0000, lowt=0.0007, auth=0.000000)\n [StageA_Base | Epoch 01 | Batch 200/300] Loss=0.0423 (main=0.0328, id=0.0000, lowt=0.0005, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 01 | Batch 250/300] Loss=0.0417 (main=0.0334, id=0.0000, lowt=0.0009, auth=0.000000)\n [StageA_Base | Epoch 01 | Batch 300/300] Loss=0.0408 (main=0.0398, id=0.0000, lowt=0.0004, auth=0.000000)\nStageA_Base Epoch 01 done | Loss=0.04070 | Time=7.1 min\n [StageA_Base | Epoch 02 | Batch 001/300] Loss=0.0347 (main=0.0343, id=0.0000, lowt=0.0004, auth=0.000000)\n [StageA_Base | Epoch 02 | Batch 050/300] Loss=0.0371 (main=0.0354, id=0.0000, lowt=0.0004, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 02 | Batch 100/300] Loss=0.0377 (main=0.0316, id=0.0000, lowt=0.0011, auth=0.000000)\n [StageA_Base | Epoch 02 | Batch 150/300] Loss=0.0409 (main=0.0359, id=0.0000, lowt=0.0015, auth=0.000000)\n [StageA_Base | Epoch 02 | Batch 200/300] Loss=0.0414 (main=0.0510, id=0.0000, lowt=0.0003, auth=0.000000)\n [StageA_Base | Epoch 02 | Batch 250/300] Loss=0.0389 (main=0.0390, id=0.0000, lowt=0.0009, auth=0.000000)\n [StageA_Base | Epoch 02 | Batch 300/300] Loss=0.0389 (main=0.0329, id=0.0000, lowt=0.0009, auth=0.000000)\nStageA_Base Epoch 02 done | Loss=0.03896 | Time=5.0 min\n [StageA_Base | Epoch 03 | Batch 001/300] Loss=0.0338 (main=0.0331, id=0.0000, lowt=0.0007, auth=0.000000)\n [StageA_Base | Epoch 03 | Batch 050/300] Loss=0.0389 (main=0.0385, id=0.0000, lowt=0.0008, auth=0.000000)\n [StageA_Base | Epoch 03 | Batch 100/300] Loss=0.0402 (main=0.0344, id=0.0000, lowt=0.0011, auth=0.000000)\n [StageA_Base | Epoch 03 | Batch 150/300] Loss=0.0400 (main=0.0453, id=0.0000, lowt=0.0009, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 03 | Batch 200/300] Loss=0.0397 (main=0.0390, id=0.0000, lowt=0.0005, auth=0.000001)\n [StageA_Base | Epoch 03 | Batch 250/300] Loss=0.0393 (main=0.0410, id=0.0000, lowt=0.0012, auth=0.000001)\n [StageA_Base | Epoch 03 | Batch 300/300] Loss=0.0368 (main=0.0441, id=0.0000, lowt=0.0008, auth=0.000002)\nStageA_Base Epoch 03 done | Loss=0.03891 | Time=4.6 min\n [Val] PSNR=37.87 dB ★ NEW BEST\n [StageA_Base | Epoch 04 | Batch 001/300] Loss=0.0376 (main=0.0363, id=0.0000, lowt=0.0013, auth=0.000001)\n [StageA_Base | Epoch 04 | Batch 050/300] Loss=0.0363 (main=0.0326, id=0.0000, lowt=0.0007, auth=0.000002)\n [StageA_Base | Epoch 04 | Batch 100/300] Loss=0.0332 (main=0.0254, id=0.0000, lowt=0.0013, auth=0.000001)\n [StageA_Base | Epoch 04 | Batch 150/300] Loss=0.0306 (main=0.0250, id=0.0000, lowt=0.0009, auth=0.000001)\n [StageA_Base | Epoch 04 | Batch 200/300] Loss=0.0307 (main=0.0210, id=0.0000, lowt=0.0014, auth=0.000001)\n [StageA_Base | Epoch 04 | Batch 250/300] Loss=0.0276 (main=0.0279, id=0.0000, lowt=0.0011, auth=0.000001)\n [StageA_Base | Epoch 04 | Batch 300/300] Loss=0.0289 (main=0.0201, id=0.0000, lowt=0.0005, auth=0.000000)\nStageA_Base Epoch 04 done | Loss=0.03112 | Time=3.7 min\n [StageA_Base | Epoch 05 | Batch 001/300] Loss=0.0207 (main=0.0184, id=0.0000, lowt=0.0023, auth=0.000000)\n [StageA_Base | Epoch 05 | Batch 050/300] Loss=0.0285 (main=0.0212, id=0.0000, lowt=0.0017, auth=0.000000)\n [StageA_Base | Epoch 05 | Batch 100/300] Loss=0.0256 (main=0.0166, id=0.0000, lowt=0.0012, auth=0.000000)\n [StageA_Base | Epoch 05 | Batch 150/300] Loss=0.0248 (main=0.0234, id=0.0000, lowt=0.0014, auth=0.000000)\n [StageA_Base | Epoch 05 | Batch 200/300] Loss=0.0241 (main=0.0298, id=0.0000, lowt=0.0011, auth=0.000001)\n [StageA_Base | Epoch 05 | Batch 250/300] Loss=0.0261 (main=0.0202, id=0.0000, lowt=0.0012, auth=0.000000)\n [StageA_Base | Epoch 05 | Batch 300/300] Loss=0.0246 (main=0.0210, id=0.0000, lowt=0.0043, auth=0.000000)\nStageA_Base Epoch 05 done | Loss=0.02586 | Time=3.8 min\n [StageA_Base | Epoch 06 | Batch 001/300] Loss=0.0243 (main=0.0203, id=0.0000, lowt=0.0040, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 050/300] Loss=0.0251 (main=0.0274, id=0.0000, lowt=0.0017, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 100/300] Loss=0.0219 (main=0.0223, id=0.0000, lowt=0.0023, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 150/300] Loss=0.0225 (main=0.0167, id=0.0000, lowt=0.0013, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 200/300] Loss=0.0220 (main=0.0156, id=0.0000, lowt=0.0033, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 250/300] Loss=0.0227 (main=0.0171, id=0.0000, lowt=0.0033, auth=0.000000)\n [StageA_Base | Epoch 06 | Batch 300/300] Loss=0.0212 (main=0.0328, id=0.0000, lowt=0.0022, auth=0.000001)\nStageA_Base Epoch 06 done | Loss=0.02314 | Time=3.8 min\n [Val] PSNR=42.71 dB ★ NEW BEST\n [StageA_Base | Epoch 07 | Batch 001/300] Loss=0.0209 (main=0.0190, id=0.0000, lowt=0.0019, auth=0.000000)\n [StageA_Base | Epoch 07 | Batch 050/300] Loss=0.0205 (main=0.0187, id=0.0000, lowt=0.0033, auth=0.000000)\n [StageA_Base | Epoch 07 | Batch 100/300] Loss=0.0196 (main=0.0201, id=0.0000, lowt=0.0027, auth=0.000000)\n [StageA_Base | Epoch 07 | Batch 150/300] Loss=0.0220 (main=0.0158, id=0.0000, lowt=0.0030, auth=0.000000)\n [StageA_Base | Epoch 07 | Batch 200/300] Loss=0.0217 (main=0.0269, id=0.0000, lowt=0.0019, auth=0.000001)\n [StageA_Base | Epoch 07 | Batch 250/300] Loss=0.0215 (main=0.0198, id=0.0000, lowt=0.0022, auth=0.000000)\n [StageA_Base | Epoch 07 | Batch 300/300] Loss=0.0216 (main=0.0177, id=0.0000, lowt=0.0015, auth=0.000000)\nStageA_Base Epoch 07 done | Loss=0.02144 | Time=3.7 min\n [StageA_Base | Epoch 08 | Batch 001/300] Loss=0.0181 (main=0.0162, id=0.0000, lowt=0.0019, auth=0.000000)\n [StageA_Base | Epoch 08 | Batch 050/300] Loss=0.0240 (main=0.0208, id=0.0000, lowt=0.0010, auth=0.000001)\n [StageA_Base | Epoch 08 | Batch 100/300] Loss=0.0222 (main=0.0187, id=0.0000, lowt=0.0025, auth=0.000001)\n [StageA_Base | Epoch 08 | Batch 150/300] Loss=0.0231 (main=0.0245, id=0.0000, lowt=0.0034, auth=0.000001)\n [StageA_Base | Epoch 08 | Batch 200/300] Loss=0.0198 (main=0.0164, id=0.0000, lowt=0.0012, auth=0.000000)\n [StageA_Base | Epoch 08 | Batch 250/300] Loss=0.0223 (main=0.0220, id=0.0000, lowt=0.0022, auth=0.000001)\n [StageA_Base | Epoch 08 | Batch 300/300] Loss=0.0197 (main=0.0128, id=0.0000, lowt=0.0010, auth=0.000000)\nStageA_Base Epoch 08 done | Loss=0.02105 | Time=3.8 min\n [StageA_Base | Epoch 09 | Batch 001/300] Loss=0.0198 (main=0.0163, id=0.0000, lowt=0.0035, auth=0.000000)\n [StageA_Base | Epoch 09 | Batch 050/300] Loss=0.0194 (main=0.0140, id=0.0000, lowt=0.0023, auth=0.000000)\n [StageA_Base | Epoch 09 | Batch 100/300] Loss=0.0191 (main=0.0227, id=0.0000, lowt=0.0018, auth=0.000001)\n [StageA_Base | Epoch 09 | Batch 150/300] Loss=0.0208 (main=0.0257, id=0.0000, lowt=0.0015, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 09 | Batch 200/300] Loss=0.0225 (main=0.0143, id=0.0000, lowt=0.0000, auth=0.000000)\n [StageA_Base | Epoch 09 | Batch 250/300] Loss=0.0212 (main=0.0201, id=0.0000, lowt=0.0008, auth=0.000001)\n [StageA_Base | Epoch 09 | Batch 300/300] Loss=0.0194 (main=0.0189, id=0.0000, lowt=0.0005, auth=0.000001)\nStageA_Base Epoch 09 done | Loss=0.02033 | Time=3.8 min\n [Val] PSNR=44.12 dB ★ NEW BEST\n [StageA_Base | Epoch 10 | Batch 001/300] Loss=0.0195 (main=0.0161, id=0.0000, lowt=0.0034, auth=0.000000)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 10 | Batch 050/300] Loss=0.0199 (main=0.0241, id=0.0000, lowt=0.0010, auth=0.000001)\n [StageA_Base | Epoch 10 | Batch 100/300] Loss=0.0188 (main=0.0136, id=0.0000, lowt=0.0022, auth=0.000001)\n [StageA_Base | Epoch 10 | Batch 150/300] Loss=0.0195 (main=0.0173, id=0.0000, lowt=0.0012, auth=0.000001)\n [StageA_Base | Epoch 10 | Batch 200/300] Loss=0.0188 (main=0.0217, id=0.0000, lowt=0.0018, auth=0.000001)\n [StageA_Base | Epoch 10 | Batch 250/300] Loss=0.0189 (main=0.0187, id=0.0000, lowt=0.0024, auth=0.000001)\n [StageA_Base | Epoch 10 | Batch 300/300] Loss=0.0200 (main=0.0146, id=0.0000, lowt=0.0024, auth=0.000001)\nStageA_Base Epoch 10 done | Loss=0.01943 | Time=3.7 min\n [StageA_Base | Epoch 11 | Batch 001/300] Loss=0.0167 (main=0.0152, id=0.0000, lowt=0.0015, auth=0.000001)\n [StageA_Base | Epoch 11 | Batch 050/300] Loss=0.0194 (main=0.0174, id=0.0000, lowt=0.0003, auth=0.000002)\n [StageA_Base | Epoch 11 | Batch 100/300] Loss=0.0180 (main=0.0146, id=0.0000, lowt=0.0009, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 11 | Batch 150/300] Loss=0.0192 (main=0.0162, id=0.0000, lowt=0.0024, auth=0.000001)\n [StageA_Base | Epoch 11 | Batch 200/300] Loss=0.0201 (main=0.0160, id=0.0000, lowt=0.0041, auth=0.000001)\n [StageA_Base | Epoch 11 | Batch 250/300] Loss=0.0195 (main=0.0191, id=0.0000, lowt=0.0018, auth=0.000001)\n [StageA_Base | Epoch 11 | Batch 300/300] Loss=0.0190 (main=0.0157, id=0.0000, lowt=0.0008, auth=0.000001)\nStageA_Base Epoch 11 done | Loss=0.01911 | Time=3.7 min\n [StageA_Base | Epoch 12 | Batch 001/300] Loss=0.0157 (main=0.0123, id=0.0000, lowt=0.0033, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 050/300] Loss=0.0180 (main=0.0192, id=0.0000, lowt=0.0011, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 100/300] Loss=0.0189 (main=0.0166, id=0.0000, lowt=0.0027, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 150/300] Loss=0.0185 (main=0.0163, id=0.0000, lowt=0.0020, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 200/300] Loss=0.0174 (main=0.0158, id=0.0000, lowt=0.0008, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 250/300] Loss=0.0180 (main=0.0149, id=0.0000, lowt=0.0008, auth=0.000001)\n [StageA_Base | Epoch 12 | Batch 300/300] Loss=0.0188 (main=0.0202, id=0.0000, lowt=0.0017, auth=0.000002)\nStageA_Base Epoch 12 done | Loss=0.01859 | Time=3.8 min\n [Val] PSNR=44.18 dB ★ NEW BEST\n [StageA_Base | Epoch 13 | Batch 001/300] Loss=0.0153 (main=0.0135, id=0.0000, lowt=0.0018, auth=0.000001)\n [StageA_Base | Epoch 13 | Batch 050/300] Loss=0.0194 (main=0.0182, id=0.0000, lowt=0.0024, auth=0.000001)\n [StageA_Base | Epoch 13 | Batch 100/300] Loss=0.0178 (main=0.0166, id=0.0000, lowt=0.0017, auth=0.000002)\n [StageA_Base | Epoch 13 | Batch 150/300] Loss=0.0183 (main=0.0132, id=0.0000, lowt=0.0023, auth=0.000001)\n [StageA_Base | Epoch 13 | Batch 200/300] Loss=0.0190 (main=0.0123, id=0.0000, lowt=0.0000, auth=0.000001)\n [StageA_Base | Epoch 13 | Batch 250/300] Loss=0.0187 (main=0.0155, id=0.0000, lowt=0.0027, auth=0.000001)\n [StageA_Base | Epoch 13 | Batch 300/300] Loss=0.0174 (main=0.0152, id=0.0000, lowt=0.0008, auth=0.000001)\nStageA_Base Epoch 13 done | Loss=0.01814 | Time=3.7 min\n [StageA_Base | Epoch 14 | Batch 001/300] Loss=0.0154 (main=0.0131, id=0.0000, lowt=0.0022, auth=0.000000)\n [StageA_Base | Epoch 14 | Batch 050/300] Loss=0.0176 (main=0.0139, id=0.0000, lowt=0.0015, auth=0.000001)\n [StageA_Base | Epoch 14 | Batch 100/300] Loss=0.0191 (main=0.0122, id=0.0000, lowt=0.0017, auth=0.000001)\n [StageA_Base | Epoch 14 | Batch 150/300] Loss=0.0177 (main=0.0148, id=0.0000, lowt=0.0024, auth=0.000001)\n [StageA_Base | Epoch 14 | Batch 200/300] Loss=0.0166 (main=0.0157, id=0.0000, lowt=0.0027, auth=0.000001)\n [StageA_Base | Epoch 14 | Batch 250/300] Loss=0.0173 (main=0.0213, id=0.0000, lowt=0.0027, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 14 | Batch 300/300] Loss=0.0173 (main=0.0111, id=0.0000, lowt=0.0028, auth=0.000000)\nStageA_Base Epoch 14 done | Loss=0.01811 | Time=3.7 min\n [StageA_Base | Epoch 15 | Batch 001/300] Loss=0.0216 (main=0.0201, id=0.0000, lowt=0.0015, auth=0.000002)\n [StageA_Base | Epoch 15 | Batch 050/300] Loss=0.0178 (main=0.0119, id=0.0000, lowt=0.0024, auth=0.000001)\n [StageA_Base | Epoch 15 | Batch 100/300] Loss=0.0180 (main=0.0084, id=0.0000, lowt=0.0012, auth=0.000000)\n [StageA_Base | Epoch 15 | Batch 150/300] Loss=0.0161 (main=0.0136, id=0.0000, lowt=0.0023, auth=0.000001)\n [StageA_Base | Epoch 15 | Batch 200/300] Loss=0.0166 (main=0.0092, id=0.0000, lowt=0.0026, auth=0.000000)\n [StageA_Base | Epoch 15 | Batch 250/300] Loss=0.0166 (main=0.0164, id=0.0000, lowt=0.0028, auth=0.000001)\n [StageA_Base | Epoch 15 | Batch 300/300] Loss=0.0158 (main=0.0192, id=0.0000, lowt=0.0024, auth=0.000002)\nStageA_Base Epoch 15 done | Loss=0.01725 | Time=3.9 min\n [Val] PSNR=44.86 dB ★ NEW BEST\n [StageA_Base | Epoch 16 | Batch 001/300] Loss=0.0173 (main=0.0150, id=0.0000, lowt=0.0023, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 16 | Batch 050/300] Loss=0.0162 (main=0.0139, id=0.0000, lowt=0.0039, auth=0.000001)\n [StageA_Base | Epoch 16 | Batch 100/300] Loss=0.0171 (main=0.0146, id=0.0000, lowt=0.0014, auth=0.000001)\n [StageA_Base | Epoch 16 | Batch 150/300] Loss=0.0167 (main=0.0186, id=0.0000, lowt=0.0016, auth=0.000001)\n [StageA_Base | Epoch 16 | Batch 200/300] Loss=0.0161 (main=0.0128, id=0.0000, lowt=0.0007, auth=0.000001)\n [StageA_Base | Epoch 16 | Batch 250/300] Loss=0.0185 (main=0.0164, id=0.0000, lowt=0.0010, auth=0.000001)\n [StageA_Base | Epoch 16 | Batch 300/300] Loss=0.0178 (main=0.0184, id=0.0000, lowt=0.0011, auth=0.000002)\nStageA_Base Epoch 16 done | Loss=0.01731 | Time=3.9 min\n [StageA_Base | Epoch 17 | Batch 001/300] Loss=0.0155 (main=0.0131, id=0.0000, lowt=0.0024, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 17 | Batch 050/300] Loss=0.0166 (main=0.0132, id=0.0000, lowt=0.0029, auth=0.000001)\n [StageA_Base | Epoch 17 | Batch 100/300] Loss=0.0171 (main=0.0145, id=0.0000, lowt=0.0041, auth=0.000001)\n [StageA_Base | Epoch 17 | Batch 150/300] Loss=0.0169 (main=0.0126, id=0.0000, lowt=0.0020, auth=0.000001)\n [StageA_Base | Epoch 17 | Batch 200/300] Loss=0.0187 (main=0.0166, id=0.0000, lowt=0.0015, auth=0.000001)\n [StageA_Base | Epoch 17 | Batch 250/300] Loss=0.0179 (main=0.0127, id=0.0000, lowt=0.0023, auth=0.000001)\n [StageA_Base | Epoch 17 | Batch 300/300] Loss=0.0182 (main=0.0166, id=0.0000, lowt=0.0024, auth=0.000001)\nStageA_Base Epoch 17 done | Loss=0.01766 | Time=4.5 min\n [StageA_Base | Epoch 18 | Batch 001/300] Loss=0.0172 (main=0.0151, id=0.0000, lowt=0.0021, auth=0.000001)\n [StageA_Base | Epoch 18 | Batch 050/300] Loss=0.0175 (main=0.0161, id=0.0000, lowt=0.0021, auth=0.000001)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageA_Base | Epoch 18 | Batch 100/300] Loss=0.0169 (main=0.0120, id=0.0000, lowt=0.0021, auth=0.000001)\n [StageA_Base | Epoch 18 | Batch 150/300] Loss=0.0170 (main=0.0142, id=0.0000, lowt=0.0016, auth=0.000001)\n [StageA_Base | Epoch 18 | Batch 200/300] Loss=0.0182 (main=0.0178, id=0.0000, lowt=0.0008, auth=0.000001)\n [StageA_Base | Epoch 18 | Batch 250/300] Loss=0.0164 (main=0.0104, id=0.0000, lowt=0.0034, auth=0.000000)\n [StageA_Base | Epoch 18 | Batch 300/300] Loss=0.0172 (main=0.0169, id=0.0000, lowt=0.0042, auth=0.000001)\nStageA_Base Epoch 18 done | Loss=0.01734 | Time=3.9 min\n [Val] PSNR=45.04 dB ★ NEW BEST\n\nStage A best val PSNR: 45.037856024328335\n\n=== Step 5. Mine hard cases on refine split only ===\nHard UIDs selected for Stage B: 24\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" uid psnr_mean \\\n0 1.2.826.0.1.3680043.8.498.11626515658801775324... 41.587017 \n1 1.2.826.0.1.3680043.8.498.85657836759065233468... 42.378001 \n2 1.2.826.0.1.3680043.8.498.13330094446695328993... 40.547437 \n3 1.2.826.0.1.3680043.8.498.36751575608626599293... 42.595342 \n4 1.2.826.0.1.3680043.8.498.12600056406312244714... 41.995032 \n5 1.2.826.0.1.3680043.8.498.11183727176682315478... 42.384594 \n6 1.2.826.0.1.3680043.8.498.77640992220078275934... 43.564771 \n7 1.2.826.0.1.3680043.8.498.73559758536294067151... 41.724661 \n8 1.2.826.0.1.3680043.8.498.94667171722410517833... 42.733100 \n9 1.2.826.0.1.3680043.8.498.13016287603897726863... 42.957814 \n10 1.2.826.0.1.3680043.8.498.12439782289683573985... 41.383155 \n11 1.2.826.0.1.3680043.8.498.60535230017781034976... 43.702799 \n12 1.2.826.0.1.3680043.8.498.42914518105599695138... 43.864598 \n13 1.2.826.0.1.3680043.8.498.24311511963019370797... 43.787381 \n14 1.2.826.0.1.3680043.8.498.12153575965064183006... 43.097272 \n15 1.2.826.0.1.3680043.8.498.11968949928784170488... 44.432831 \n16 1.2.826.0.1.3680043.8.498.10595568885979229712... 43.443026 \n17 1.2.826.0.1.3680043.8.498.80865903852554884079... 44.400794 \n18 1.2.826.0.1.3680043.8.498.90000252095920683908... 44.504854 \n19 1.2.826.0.1.3680043.8.498.64528265314234570880... 43.958598 \n\n overedit_mean hard_score \n0 0.012948 2.851041 \n1 0.013036 2.849462 \n2 0.012769 2.843603 \n3 0.012794 2.808168 \n4 0.012357 2.752579 \n5 0.012404 2.752334 \n6 0.012467 2.739445 \n7 0.012175 2.729810 \n8 0.012291 2.728344 \n9 0.012221 2.713248 \n10 0.011987 2.707605 \n11 0.012259 2.704881 \n12 0.012195 2.691972 \n13 0.012025 2.667394 \n14 0.011863 2.655714 \n15 0.011952 2.643707 \n16 0.011806 2.640392 \n17 0.011772 2.616796 \n18 0.011764 2.613554 \n19 0.011695 2.613465 ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
uidpsnr_meanoveredit_meanhard_score
01.2.826.0.1.3680043.8.498.11626515658801775324...41.5870170.0129482.851041
11.2.826.0.1.3680043.8.498.85657836759065233468...42.3780010.0130362.849462
21.2.826.0.1.3680043.8.498.13330094446695328993...40.5474370.0127692.843603
31.2.826.0.1.3680043.8.498.36751575608626599293...42.5953420.0127942.808168
41.2.826.0.1.3680043.8.498.12600056406312244714...41.9950320.0123572.752579
51.2.826.0.1.3680043.8.498.11183727176682315478...42.3845940.0124042.752334
61.2.826.0.1.3680043.8.498.77640992220078275934...43.5647710.0124672.739445
71.2.826.0.1.3680043.8.498.73559758536294067151...41.7246610.0121752.729810
81.2.826.0.1.3680043.8.498.94667171722410517833...42.7331000.0122912.728344
91.2.826.0.1.3680043.8.498.13016287603897726863...42.9578140.0122212.713248
101.2.826.0.1.3680043.8.498.12439782289683573985...41.3831550.0119872.707605
111.2.826.0.1.3680043.8.498.60535230017781034976...43.7027990.0122592.704881
121.2.826.0.1.3680043.8.498.42914518105599695138...43.8645980.0121952.691972
131.2.826.0.1.3680043.8.498.24311511963019370797...43.7873810.0120252.667394
141.2.826.0.1.3680043.8.498.12153575965064183006...43.0972720.0118632.655714
151.2.826.0.1.3680043.8.498.11968949928784170488...44.4328310.0119522.643707
161.2.826.0.1.3680043.8.498.10595568885979229712...43.4430260.0118062.640392
171.2.826.0.1.3680043.8.498.80865903852554884079...44.4007940.0117722.616796
181.2.826.0.1.3680043.8.498.90000252095920683908...44.5048540.0117642.613554
191.2.826.0.1.3680043.8.498.64528265314234570880...43.9585980.0116952.613465
\n
"},"metadata":{}},{"name":"stdout","text":"\n=== Step 6. Stage B refinement dataset ===\nStage B total UID list = 324 (train + hard refine)\nHard UID repeat = 3\n\n=== Step 7. Stage B refinement ===\n\n=== StageB_Refine ===\nlr: 8e-05 | epochs: 8\n [StageB_Refine | Epoch 01 | Batch 001/220] Loss=0.0239 (main=0.0193, id=0.0000, lowt=0.0045, auth=0.000003)\n [StageB_Refine | Epoch 01 | Batch 050/220] Loss=0.0221 (main=0.0195, id=0.0000, lowt=0.0004, auth=0.000004)\n [StageB_Refine | Epoch 01 | Batch 100/220] Loss=0.0202 (main=0.0216, id=0.0000, lowt=0.0022, auth=0.000004)\n [StageB_Refine | Epoch 01 | Batch 150/220] Loss=0.0203 (main=0.0165, id=0.0000, lowt=0.0033, auth=0.000002)\n [StageB_Refine | Epoch 01 | Batch 200/220] Loss=0.0205 (main=0.0186, id=0.0000, lowt=0.0021, auth=0.000002)\n [StageB_Refine | Epoch 01 | Batch 220/220] Loss=0.0215 (main=0.0159, id=0.0000, lowt=0.0011, auth=0.000003)\nStageB_Refine Epoch 01 done | Loss=0.02075 | Time=3.3 min\n [StageB_Refine | Epoch 02 | Batch 001/220] Loss=0.0197 (main=0.0181, id=0.0000, lowt=0.0016, auth=0.000003)\n [StageB_Refine | Epoch 02 | Batch 050/220] Loss=0.0212 (main=0.0160, id=0.0000, lowt=0.0040, auth=0.000002)\n [StageB_Refine | Epoch 02 | Batch 100/220] Loss=0.0200 (main=0.0194, id=0.0000, lowt=0.0035, auth=0.000002)\n [StageB_Refine | Epoch 02 | Batch 150/220] Loss=0.0200 (main=0.0188, id=0.0000, lowt=0.0025, auth=0.000003)\n [StageB_Refine | Epoch 02 | Batch 200/220] Loss=0.0201 (main=0.0234, id=0.0000, lowt=0.0024, auth=0.000003)\n [StageB_Refine | Epoch 02 | Batch 220/220] Loss=0.0212 (main=0.0169, id=0.0000, lowt=0.0023, auth=0.000003)\nStageB_Refine Epoch 02 done | Loss=0.02067 | Time=3.2 min\n [StageB_Refine | Epoch 03 | Batch 001/220] Loss=0.0198 (main=0.0163, id=0.0000, lowt=0.0035, auth=0.000002)\n [StageB_Refine | Epoch 03 | Batch 050/220] Loss=0.0215 (main=0.0180, id=0.0000, lowt=0.0020, auth=0.000004)\n [StageB_Refine | Epoch 03 | Batch 100/220] Loss=0.0204 (main=0.0198, id=0.0000, lowt=0.0030, auth=0.000004)\n [StageB_Refine | Epoch 03 | Batch 150/220] Loss=0.0192 (main=0.0150, id=0.0000, lowt=0.0006, auth=0.000002)\n [StageB_Refine | Epoch 03 | Batch 200/220] Loss=0.0179 (main=0.0116, id=0.0000, lowt=0.0027, auth=0.000001)\n [StageB_Refine | Epoch 03 | Batch 220/220] Loss=0.0201 (main=0.0185, id=0.0000, lowt=0.0026, auth=0.000002)\nStageB_Refine Epoch 03 done | Loss=0.01997 | Time=3.1 min\n [Val] PSNR=45.07 dB ★ NEW BEST\n [StageB_Refine | Epoch 04 | Batch 001/220] Loss=0.0184 (main=0.0177, id=0.0000, lowt=0.0006, auth=0.000003)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageB_Refine | Epoch 04 | Batch 050/220] Loss=0.0215 (main=0.0201, id=0.0000, lowt=0.0032, auth=0.000003)\n [StageB_Refine | Epoch 04 | Batch 100/220] Loss=0.0202 (main=0.0136, id=0.0000, lowt=0.0020, auth=0.000001)\n [StageB_Refine | Epoch 04 | Batch 150/220] Loss=0.0202 (main=0.0187, id=0.0000, lowt=0.0020, auth=0.000004)\n [StageB_Refine | Epoch 04 | Batch 200/220] Loss=0.0204 (main=0.0177, id=0.0000, lowt=0.0030, auth=0.000002)\n [StageB_Refine | Epoch 04 | Batch 220/220] Loss=0.0206 (main=0.0163, id=0.0000, lowt=0.0029, auth=0.000002)\nStageB_Refine Epoch 04 done | Loss=0.02044 | Time=3.1 min\n [StageB_Refine | Epoch 05 | Batch 001/220] Loss=0.0236 (main=0.0220, id=0.0000, lowt=0.0016, auth=0.000003)\n [StageB_Refine | Epoch 05 | Batch 050/220] Loss=0.0196 (main=0.0129, id=0.0000, lowt=0.0014, auth=0.000001)\n [StageB_Refine | Epoch 05 | Batch 100/220] Loss=0.0197 (main=0.0196, id=0.0000, lowt=0.0015, auth=0.000003)\n [StageB_Refine | Epoch 05 | Batch 150/220] Loss=0.0188 (main=0.0162, id=0.0000, lowt=0.0004, auth=0.000003)\n [StageB_Refine | Epoch 05 | Batch 200/220] Loss=0.0201 (main=0.0180, id=0.0000, lowt=0.0020, auth=0.000002)\n [StageB_Refine | Epoch 05 | Batch 220/220] Loss=0.0199 (main=0.0170, id=0.0000, lowt=0.0024, auth=0.000003)\nStageB_Refine Epoch 05 done | Loss=0.01969 | Time=3.2 min\n [StageB_Refine | Epoch 06 | Batch 001/220] Loss=0.0224 (main=0.0205, id=0.0000, lowt=0.0018, auth=0.000003)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageB_Refine | Epoch 06 | Batch 050/220] Loss=0.0200 (main=0.0166, id=0.0000, lowt=0.0042, auth=0.000003)\n [StageB_Refine | Epoch 06 | Batch 100/220] Loss=0.0198 (main=0.0153, id=0.0000, lowt=0.0010, auth=0.000003)\n [StageB_Refine | Epoch 06 | Batch 150/220] Loss=0.0194 (main=0.0182, id=0.0000, lowt=0.0029, auth=0.000003)\n [StageB_Refine | Epoch 06 | Batch 200/220] Loss=0.0192 (main=0.0139, id=0.0000, lowt=0.0011, auth=0.000002)\n [StageB_Refine | Epoch 06 | Batch 220/220] Loss=0.0204 (main=0.0207, id=0.0000, lowt=0.0034, auth=0.000004)\nStageB_Refine Epoch 06 done | Loss=0.01971 | Time=3.3 min\n [Val] PSNR=45.19 dB ★ NEW BEST\n [StageB_Refine | Epoch 07 | Batch 001/220] Loss=0.0218 (main=0.0187, id=0.0000, lowt=0.0031, auth=0.000002)\n [StageB_Refine | Epoch 07 | Batch 050/220] Loss=0.0209 (main=0.0206, id=0.0000, lowt=0.0045, auth=0.000003)\n","output_type":"stream"},{"name":"stderr","text":"/usr/local/lib/python3.12/dist-packages/pydicom/pixels/utils.py:222: UserWarning: A value of 'None' for (0028,0008) 'Number of Frames' is invalid, assuming 1 frame\n warn_and_log(\n","output_type":"stream"},{"name":"stdout","text":" [StageB_Refine | Epoch 07 | Batch 100/220] Loss=0.0198 (main=0.0117, id=0.0000, lowt=0.0009, auth=0.000001)\n [StageB_Refine | Epoch 07 | Batch 150/220] Loss=0.0196 (main=0.0184, id=0.0000, lowt=0.0037, auth=0.000003)\n [StageB_Refine | Epoch 07 | Batch 200/220] Loss=0.0193 (main=0.0161, id=0.0000, lowt=0.0018, auth=0.000003)\n [StageB_Refine | Epoch 07 | Batch 220/220] Loss=0.0198 (main=0.0143, id=0.0000, lowt=0.0029, auth=0.000001)\nStageB_Refine Epoch 07 done | Loss=0.01978 | Time=2.8 min\n [StageB_Refine | Epoch 08 | Batch 001/220] Loss=0.0216 (main=0.0193, id=0.0000, lowt=0.0023, auth=0.000004)\n [StageB_Refine | Epoch 08 | Batch 050/220] Loss=0.0194 (main=0.0148, id=0.0000, lowt=0.0019, auth=0.000002)\n [StageB_Refine | Epoch 08 | Batch 100/220] Loss=0.0203 (main=0.0147, id=0.0000, lowt=0.0017, auth=0.000002)\n [StageB_Refine | Epoch 08 | Batch 150/220] Loss=0.0202 (main=0.0127, id=0.0000, lowt=0.0019, auth=0.000002)\n [StageB_Refine | Epoch 08 | Batch 200/220] Loss=0.0208 (main=0.0200, id=0.0000, lowt=0.0026, auth=0.000004)\n [StageB_Refine | Epoch 08 | Batch 220/220] Loss=0.0191 (main=0.0174, id=0.0000, lowt=0.0029, auth=0.000002)\nStageB_Refine Epoch 08 done | Loss=0.01988 | Time=2.9 min\n [Val] PSNR=45.33 dB ★ NEW BEST\n\n==========================================================================================\nScientific-method training finished\n==========================================================================================\ntrain_uids: 300\nrefine_uids: 60\nval_uids: 20\nhard_uids_stageB: 24\nbest_stageA_val_psnr: 45.037856024328335\nbest_stageB_val_psnr: 45.326924368298805\nstageA_best_ckpt: /kaggle/working/vultimate_scientific/deblur_stageA_best.pt\nstageB_best_ckpt: /kaggle/working/vultimate_scientific/deblur_stageB_best.pt\n\nSaved files:\n - /kaggle/working/vultimate_scientific/deblur_stageA_best.pt\n - /kaggle/working/vultimate_scientific/deblur_stageA_last.pt\n - /kaggle/working/vultimate_scientific/deblur_stageB_best.pt\n - /kaggle/working/vultimate_scientific/deblur_stageB_last.pt\n - /kaggle/working/vultimate_scientific/hard_uids_stageB.csv\n - /kaggle/working/vultimate_scientific/history_stageA.csv\n - /kaggle/working/vultimate_scientific/history_stageB.csv\n - /kaggle/working/vultimate_scientific/refine_uid_difficulty.csv\n - /kaggle/working/vultimate_scientific/refine_uids.csv\n - /kaggle/working/vultimate_scientific/train_uids.csv\n - /kaggle/working/vultimate_scientific/training_summary.json\n - /kaggle/working/vultimate_scientific/val_uids.csv\n\n✅ 推荐用于后续评估的权重：\n /kaggle/working/vultimate_scientific/deblur_stageB_best.pt\n","output_type":"stream"}],"execution_count":5},{"cell_type":"code","source":"# ============================================================\n# 0) Imports & 环境\n# ============================================================\nimport os, sys, gc, math, time, random, hashlib, inspect, shutil, subprocess\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport pydicom\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom contextlib import nullcontext\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = (lambda: torch.amp.autocast(\"cuda\")) if USE_AMP else (lambda: nullcontext())\n\nprint(\"device:\", device)\nprint(\"USE_AMP:\", USE_AMP)","metadata":{"execution":{"iopub.status.busy":"2026-03-11T00:43:30.219963Z","iopub.execute_input":"2026-03-11T00:43:30.220596Z","iopub.status.idle":"2026-03-11T00:43:35.440835Z","shell.execute_reply.started":"2026-03-11T00:43:30.220564Z","shell.execute_reply":"2026-03-11T00:43:35.439968Z"},"trusted":true},"outputs":[{"name":"stdout","text":"device: cuda\nUSE_AMP: True\n","output_type":"stream"}],"execution_count":1},{"cell_type":"markdown","source":"# §1 Configuration — Telling the Notebook Where the Data Is and How the Experiments Are Set Up\n\n## What does “configuration” mean?\n\nBefore any experiment can run, the notebook needs a clear set of instructions:\n\n- Where is the CT data stored?\n- Where is the trained model checkpoint?\n- Which external clinical judge should be used?\n- How many cases should each experiment evaluate?\n- What degradation settings should be used to simulate lower-quality scans?\n\nThis section is the project’s **control panel**. \nIt does not perform analysis by itself, but it tells every later cell how to behave.\n\n---\n\n## Key configuration groups\n\n### 1. Data and model paths\n\n| Parameter | What it points to | Why it matters |\n|----------|-------------------|----------------|\n| `RSNA_DATA_ROOT` | The RSNA competition CT/CTA scan folders | This is the main source of medical image data in DICOM format |\n| `META_CSV` | The metadata table for the RSNA dataset | Used to identify scan modality and build fair evaluation pools |\n| `CKPT_PATH` | The trained **V-Ultimate checkpoint** | This file contains the learned model weights — the model’s “memory” from training |\n| `PREDICTION_PY` / `FLAYER_DIR` | The independent clinical judge model | This is a separate aneurysm-detection AI used to evaluate whether restoration improves downstream diagnostic signal |\n\n---\n\n### 2. Image geometry and HU range\n\n| Parameter | Value | Meaning |\n|----------|-------|---------|\n| `TARGET_SHAPE = (64, 448, 448)` | 64 slices, each 448×448 pixels | All scans are standardized to the same shape so they can be processed consistently |\n| `HU_MIN = -1024`, `HU_MAX = 3072` | Hounsfield Unit range | HU is the physical intensity scale used in CT: air is about -1000, water is 0, dense bone is much higher |\n\nBecause different CT scans can have different slice counts and resolutions, the notebook converts them into a common format before evaluation.\n\nThis is important because the neural network expects a fixed input size.\n\n---\n\n### 3. Degradation settings — how the notebook simulates image damage\n\n| Parameter | Value | Physical meaning |\n|----------|-------|------------------|\n| `EVAL_T = 8.0` | diffusion time | A larger value means stronger blur; in the PSF-inspired formulation, blur strength grows with time |\n| `EVAL_DOSE = \"quarter\"` | quarter-dose | Simulates a much noisier low-dose acquisition, roughly corresponding to reduced photon counts |\n| `LAM = 0.20` | diffusion coefficient | Controls how quickly blur spreads in the synthetic degradation model |\n| `BLUR_T_MAX = 8.0` | maximum blur level | Used to normalize blur severity when passing degradation strength into the model |\n\nThese settings make the evaluation controlled and repeatable. \nEvery method is tested under the **same synthetic damage conditions**, so the comparison is fair.\n\n---\n\n### 4. Experiment size\n\n| Parameter | Value | Purpose |\n|----------|-------|---------|\n| `N_PILOT_COMPARE = 10` | 10 cases | A small pilot comparison for quick checking |\n| `N_OOD_EVAL = 50` | 50 cases | Main out-of-distribution evaluation set |\n| `N_MC_CASES = 100` with `MC_SEEDS = 10` | 100 × 10 = 1000 runs | Monte Carlo stability test to see whether effects are reproducible across noise realizations |\n| `N_TOTALSEG_CASES = 40` | 40 cases | Anatomical overlap analysis with TotalSegmentator |\n\nThese numbers balance two goals:\n\n- enough cases to make the results meaningful,\n- but still feasible within Kaggle time and GPU limits.\n\n---\n\n### 5. Output directory\n\nThe notebook saves intermediate tables and final summaries to the working directory.\n\nThis allows the results to be:\n\n- inspected later,\n- reused in downstream cells,\n- and exported as evidence for the final report.\n\n---\n\n## Why this section matters\n\nA strong scientific notebook does not rely on hidden settings or manual clicking. \nAll important choices are written down explicitly here.\n\nThat makes the experiments:\n\n- **transparent**\n- **reproducible**\n- **easy to audit**\n- **easy to modify if needed**\n\nIn other words, this section makes the rest of the notebook trustworthy.\n\n---\n\n## Expected startup check\n\nAt the end of the configuration cell, the notebook prints whether the critical files exist:\n\n```text\nCKPT_PATH exists: True\nRSNA_DATA_ROOT exists: True\nMETA_CSV exists: True","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 1) Config\n# ============================================================\nfrom pathlib import Path\nimport os\n\nRSNA_DATA_ROOT = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\"\nMETA_CSV = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\"\nTRAIN_LOCALIZERS_CSV = \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train_localizers.csv\" # optional\n\n# ------------------------------------------------------------\n# Final model checkpoint: use the best Stage-B scientific model\n# ------------------------------------------------------------\nCKPT_PATH = \"/kaggle/input/datasets/linguoyuemma/deblur-stageb-best-pt/deblur_stageB_best.pt\"\n\n# Optional: split files from the scientific training pipeline\nTRAIN_UIDS_CSV = \"/kaggle/input/datasets/linguoyuemma/refine-uids/train_uids.csv\"\nREFINE_UIDS_CSV = \"/kaggle/input/datasets/linguoyuemma/refine-uids/refine_uids.csv\"\nVAL_UIDS_CSV = \"/kaggle/input/datasets/linguoyuemma/refine-uids/val_uids.csv\"\n\n# Clinical Judge (9th place flayer)\nPREDICTION_PY = \"/kaggle/input/datasets/mingzeli2009/rsna-prediction/prediction.py\"\nMODEL_BASE = \"/kaggle/input/models/tom99763/9th-place-models-rsna-iad/pytorch/default/1\"\nFLAYER_DIR = f\"{MODEL_BASE}/flayer/outputs_heatmap_aux_v1_acc2\"\n\n# ------------------------------------------------------------\n# Output directory\n# ------------------------------------------------------------\nOUTDIR = Path(\"/kaggle/working/vultimate_scientific_eval\")\nOUTDIR.mkdir(parents=True, exist_ok=True)\n\n# ------------------------------------------------------------\n# Data\n# ------------------------------------------------------------\nTARGET_D, TARGET_H, TARGET_W = 64, 448, 448\nTARGET_SHAPE = (TARGET_D, TARGET_H, TARGET_W)\n\nHU_MIN, HU_MAX = -1024.0, 3072.0\nHU_RANGE = HU_MAX - HU_MIN\nKEEP_MODALITIES = {\"CT\", \"CTA\"}\n\n# ------------------------------------------------------------\n# Degradation settings for evaluation\n# Keep these aligned with the training-time degradation model\n# ------------------------------------------------------------\nEVAL_T = 8.0\nEVAL_DOSE = \"quarter\"\nDIFFUSION_ALPHA = 0.20\nLAM = DIFFUSION_ALPHA # backward compatibility for older cells\nBLUR_T_MAX = 8.0\nRESTORE_BATCH = 16\n\n# ------------------------------------------------------------\n# Evaluation scale\n# ------------------------------------------------------------\nSEED_CASES = 2026\n\nN_PILOT_COMPARE = 10 # quick sanity check\nN_OOD_EVAL = 50 # main CRM size; can later increase to 100/200\nN_MC_CASES = 100 # Monte Carlo stability test\nMC_SEEDS = [10, 42, 23, 55, 83, 9999, 7, 11, 19, 29]\n\n# ------------------------------------------------------------\n# TotalSegmentator\n# Set this to the actual number you want to run\n# ------------------------------------------------------------\nN_TOTALSEG_CASES = 40\nTOTALSEG_TASK = \"total\"\nTOTALSEG_CHANGE_THR = 0.05\n\n# ------------------------------------------------------------\n# Diagnostics\n# ------------------------------------------------------------\nprint(\"CKPT_PATH exists:\", os.path.exists(CKPT_PATH))\nprint(\"TRAIN_UIDS_CSV exists:\", os.path.exists(TRAIN_UIDS_CSV))\nprint(\"REFINE_UIDS_CSV exists:\", os.path.exists(REFINE_UIDS_CSV))\nprint(\"VAL_UIDS_CSV exists:\", os.path.exists(VAL_UIDS_CSV))\nprint(\"RSNA_DATA_ROOT exists:\", os.path.exists(RSNA_DATA_ROOT))\nprint(\"META_CSV exists:\", os.path.exists(META_CSV))\nprint(\"OUTDIR:\", OUTDIR)","metadata":{"execution":{"iopub.status.busy":"2026-03-11T00:50:08.081404Z","iopub.execute_input":"2026-03-11T00:50:08.08176Z","iopub.status.idle":"2026-03-11T00:50:08.101414Z","shell.execute_reply.started":"2026-03-11T00:50:08.081731Z","shell.execute_reply":"2026-03-11T00:50:08.100417Z"},"trusted":true},"outputs":[{"name":"stdout","text":"CKPT_PATH exists: True\nTRAIN_UIDS_CSV exists: True\nREFINE_UIDS_CSV exists: True\nVAL_UIDS_CSV exists: True\nRSNA_DATA_ROOT exists: True\nMETA_CSV exists: True\nOUTDIR: /kaggle/working/vultimate_scientific_eval\n","output_type":"stream"}],"execution_count":17},{"cell_type":"markdown","source":"# §3 Load Model Weights + Clinical Judge\n\n## What This Step Does\n\nThis section performs two essential tasks.\n\n---\n\n### 1. Load the V-Ultimate Checkpoint\n\nA trained AI model stores what it has learned in a checkpoint file ([.pt](cci:7://file:///Users/andrewlin/Downloads/neuroexplain/deblur_25d.pt:0:0-0:0)) — think of it as the model's long-term memory.\n\n checkpoint file → read learned parameters → load into model architecture → model is ready to use\n\n`strict=True` means every parameter in the checkpoint must match the architecture exactly. If any layer mismatches, loading fails immediately. This is a deliberate safety check — it prevents silently loading weights into the wrong model structure.\n\nThe checkpoint may also contain `train_uids`: the case IDs used during training. These are critical for evaluation, because all downstream experiments must exclude training cases and test only on unseen data.\n\n---\n\n### 2. Load the Clinical Judge (`FlayerClassifier`)\n\nThe core evaluation question in this notebook is not:\n\n> \"Does the restored CT look sharper?\"\n\nIt is:\n\n> \"Does the restored CT preserve or improve clinically meaningful diagnostic signal?\"\n\nTo measure this objectively, we use an independent aneurysm-detection model as a **clinical proxy judge**:\n\n| Property | Detail |\n|----------|--------|\n| Separate from V-Ultimate | Trained independently; not involved in restoration |\n| Input | CT volume |\n| Output | Aneurysm probability `p ∈ [0, 1]` |\n| Basis | High-performing RSNA aneurysm-detection solution (evaluation use only) |\n\n---\n\n### Why an External Model as Judge?\n\nUsing an independent model as the evaluator is important for scientific rigour:\n\n- **Independence** — the judge was trained separately from V-Ultimate, so it cannot be \"gamed\" by the restoration model\n- **Task relevance** — it measures aneurysm-related signal, not just pixel-level similarity\n- **Quantitative comparison** — it produces a probability score, enabling fair comparison across all restoration methods\n\nRather than asking \"does this image look better?\", we ask: **\"is this image more useful for downstream clinical decision support?\"**\n\n---\n\n### Why This Matters for Later Experiments\n\nOnce both models are loaded, the Clinical Rescue Matrix can evaluate the **degraded input**, **traditional baselines** (Gaussian, Median, Bilateral, Unsharp), and **V-Ultimate** — all under the same clinical proxy judge, on the same held-out cases. This shared evaluation framework is what makes the CRM results scientifically comparable.\n","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 3) Load NEW model checkpoint + Clinical Judge\n# ============================================================\nassert os.path.exists(CKPT_PATH), f\"Checkpoint not found: {CKPT_PATH}\"\n\nckpt = torch.load(CKPT_PATH, map_location=\"cpu\")\nstate_dict = ckpt[\"model\"] if isinstance(ckpt, dict) and \"model\" in ckpt else ckpt\n\n# --- NEW V-Ultimate (FiLM + authority map) ---\nmodel_25d = DeblurUNet25D_Ultimate(\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=0.02,\n res_max=0.15,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n).to(device)\n\nmodel_25d.load_state_dict(state_dict, strict=True)\nmodel_25d.eval()\n\nprint(\"✅ NEW V-Ultimate model loaded.\")\nprint(\"has auth_head?\", hasattr(model_25d, \"auth_head\"))\nprint(\"has film_e1?\", hasattr(model_25d, \"film_e1\"))\nprint(\"train_uids in ckpt:\", len(ckpt.get(\"train_uids\", [])) if isinstance(ckpt, dict) else \"NA\")\nprint(\"best_val_psnr:\", ckpt.get(\"best_val_psnr\", \"NA\") if isinstance(ckpt, dict) else \"NA\")\n\n# ---- Load clinical judge ----\nimport importlib.util\nassert os.path.exists(PREDICTION_PY), f\"prediction.py not found: {PREDICTION_PY}\"\n\nspec = importlib.util.spec_from_file_location(\"prediction\", PREDICTION_PY)\npred_mod = importlib.util.module_from_spec(spec)\nsys.modules[\"prediction\"] = pred_mod\nspec.loader.exec_module(pred_mod)\n\nclassifier = pred_mod.FlayerClassifier(flayer_dir=FLAYER_DIR)\nclassifier.load()\n\n@torch.no_grad()\ndef aneurysm_predict(volume_uint8):\n return float(classifier.predict(volume_uint8)[\"aneurysm_prob\"])\n\nprint(\"✅ Clinical Judge loaded.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T00:50:17.998462Z","iopub.execute_input":"2026-03-11T00:50:17.998818Z","iopub.status.idle":"2026-03-11T00:50:40.259839Z","shell.execute_reply.started":"2026-03-11T00:50:17.998793Z","shell.execute_reply":"2026-03-11T00:50:40.258824Z"}},"outputs":[{"name":"stdout","text":"✅ NEW V-Ultimate model loaded.\nhas auth_head? True\nhas film_e1? True\ntrain_uids in ckpt: 0\nbest_val_psnr: 45.326924368298805\n ✅ Flayer: 5 folds loaded → cuda\n✅ Clinical Judge loaded.\n","output_type":"stream"}],"execution_count":18},{"cell_type":"markdown","source":"# §4 Utility Functions — The Evaluation Pipeline Toolbox\n\nThis section defines the core utilities used throughout the notebook.\nA good comparison study needs more than a model — it also needs reliable tools for loading data, creating controlled degradations, running inference, and measuring results fairly.\n\nThink of this section as the project's **lab toolkit**.\n\n---\n\n## 1. Data-Loading Utilities\n\n### `stable_uid_seed(uid)` — reproducible random seed per case\n\nSynthetic degradation uses random noise. If that noise changes every run, the experiment is not reproducible.\n\nThis function converts each case UID into a fixed integer seed via **MD5 hashing**, so the same patient series always receives the same synthetic degradation pattern. This makes experiments **repeatable**, **fair across methods**, and **scientifically auditable**.\n\n---\n\n### `load_series_volume(uid, root, shape)` — load a CT series as a 3D volume\n\nConverts a raw DICOM series into a normalised 3D volume. The pipeline is:\n\n1. Locate the case folder\n2. Sort DICOM slices into anatomical order\n3. Convert raw pixel values to **Hounsfield Units (HU)**:\n\n HU = pixel_value × slope + intercept\n\n4. Clip to `[-1024, 3072]` and normalise to `[0, 1]`\n5. Resize each slice to `448 × 448`\n6. Resample to a fixed depth of 64 slices\n\nThis standardisation is required because neural networks need consistent input dimensions.\n\n---\n\n### `vol01_to_flayer_uint8(vol01)` — prepare input for the clinical judge\n\nApplies a **brain-vessel window** before passing data to the aneurysm classifier:\n\n| Parameter | Value |\n|-----------|-------|\n| Window centre | 40 HU |\n| Window width | 400 HU |\n| Display range | [-160, 240] HU |\n\nThis window emphasises vascular structures — analogous to adjusting exposure in photography: the underlying image is unchanged, but the window determines which features are highlighted.\n\n---\n\n## 2. Degradation Utilities\n\n### `degrade_volume(vol, t, dose_mode)` — simulate degraded CT acquisition\n\nBoth training and evaluation require paired data:\n\n clean image → synthetic degradation → degraded image → model → recovered image\n\nDegradation has two components:\n\n**(a) Gaussian blur** — approximates PSF-induced resolution loss:\n\n σ = √(2 λ t)\n\n**(b) Poisson-Gaussian noise** — simulates low-dose CT noise:\n- **Poisson**: randomness of photon arrival (fewer photons = noisier signal)\n- **Gaussian**: electronic readout noise from the detector\n\nTogether these produce a more realistic low-dose degradation than simple white noise.\n\n---\n\n## 3. Inference Utilities\n\n### `deblur_volume_25d(model, vol, t)` — batched 2.5D restoration\n\nFor each slice `z`, the 4-channel input is:\n\n| Channel | Content |\n|---------|---------|\n| ch0 | previous slice `z-1` |\n| ch1 | current slice `z` (target) |\n| ch2 | next slice `z+1` |\n| ch3 | degradation level `t_norm` |\n\nSlices are grouped into mini-batches (`restore_batch = 16`), sent to GPU, and the restored central slice is extracted and clamped to `[0, 1]`. This batching makes evaluation fast while keeping memory usage manageable.\n\n---\n\n## 4. Evaluation Metrics\n\n### Target Gain — task-aware clinical improvement\n\nMeasures whether restoration moves the clinical judge in the **correct diagnostic direction**.\n\n- For a **positive case** (`p_gt = 0.85`, `p_deg = 0.70`, `p_rec = 0.82`): restoration recovered lost signal → **positive gain**\n- For a **negative case** (`p_gt = 0.15`, `p_deg = 0.30`, `p_rec = 0.18`): restoration reduced a false-positive tendency → **positive gain**\n\nPositive Target Gain = helpful restoration. Negative = harmful.\n\n---\n\n### Absolute Gain — reduction in absolute probability error\n\n Abs-Gain = |p_deg - p_gt| - |p_rec - p_gt|\n\nPositive means the restored probability moved **closer** to the ground-truth reference. Answers: *\"Did restoration reduce prediction error?\"*\n\n---\n\n### Iatrogenic Harm — when the intervention makes things worse\n\nUses a 4-tier probability system:\n\n| Tier | Range |\n|------|-------|\n| 0 | p < 0.2 |\n| 1 | 0.2 ≤ p < 0.5 |\n| 2 | 0.5 ≤ p < 0.8 |\n| 3 | p ≥ 0.8 |\n\nIf restoration moves a case into a tier **farther** from the ground-truth tier than the degraded input was, it is counted as iatrogenic harm. A model should not produce clinically misleading changes even when pixel quality appears improved.\n\n---\n\n### PSNR — pixel-level fidelity\n\n PSNR = 10 × log10(1 / MSE)\n\n| Range | Interpretation |\n|-------|---------------|\n| ~30 dB | Decent reconstruction |\n| ~40 dB | Strong reconstruction |\n| 50+ dB | Nearly identical to reference |\n\nPSNR alone is insufficient — a high PSNR image does not necessarily preserve downstream clinical signal.\n\n---\n\n### Outcome Labels — case-level interpretation\n\n| Label | Meaning | Rule |\n|-------|---------|------|\n| `positive` | Beneficial improvement | Target Gain > 0.005 |\n| `neutral` | No meaningful change | \\|Target Gain\\| ≤ 0.005 |\n| `negative` | Harmful deviation | Target Gain < -0.005 |\n| `iatrogenic` | Tier-level harm | Tier moves away from ground truth |\n\n---\n\n## 5. Data Firewall — Preventing Evaluation Leakage\n\nEvaluating a model on its own training cases inflates performance — it is not a true generalisation test.\n\nKnown training UIDs are read and excluded, producing an **OOD (Out-of-Distribution) pool** of previously unseen cases. Without this step, reported performance can look significantly better than it really is.\n\n---\n\n## Why This Toolbox Matters\n\nThese utilities are what make the experiments **scientifically credible**. They ensure:\n\n- the same case always receives the same degradation\n- all methods are compared under identical conditions\n- evaluation is clinically relevant, not just pixel-level\n- testing uses only unseen data\n\nThis section is what turns the notebook from a demo into a **controlled evaluation study**.\n","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 4) Utilities\n# ============================================================\ndef stable_uid_seed(uid: str, mod=(2**31 - 1)):\n \"\"\"UID-to-seed conversion stable across sessions (avoids Python hash randomisation).\"\"\"\n return int(hashlib.md5(uid.encode(\"utf-8\")).hexdigest()[:8], 16) % mod\n\ndef uid_tail4(uid: str) -> str:\n return str(uid).split(\".\")[-1][-4:]\n\ndef vol01_to_flayer_uint8(vol01):\n \"\"\"\n Medical physics bridge: apply CTA window (centre=40, width=400 → [-160, 240] HU).\n \"\"\"\n hu_vol = (np.asarray(vol01, dtype=np.float32) * HU_RANGE) + HU_MIN\n windowed = np.clip((hu_vol - (-160.0)) / 400.0, 0.0, 1.0)\n return (windowed * 255.0).astype(np.uint8)\n\ndef _clip01(x):\n return np.clip(x, 0.0, 1.0).astype(np.float32)\n\ndef calc_target_gain(p_gt, p_deg, p_rec):\n \"\"\"Target-aware gain (primary metric).\"\"\"\n p_gt, p_deg, p_rec = float(p_gt), float(p_deg), float(p_rec)\n if p_gt >= 0.50:\n eff_deg = min(p_deg, p_gt)\n return p_rec - eff_deg\n else:\n eff_deg = max(p_deg, p_gt)\n return eff_deg - p_rec\n\ndef calc_abs_gain(p_gt, p_deg, p_rec):\n \"\"\"Absolute error reduction (reference metric).\"\"\"\n p_gt, p_deg, p_rec = float(p_gt), float(p_deg), float(p_rec)\n return abs(p_deg - p_gt) - abs(p_rec - p_gt)\n\ndef get_tier(p):\n p = float(p)\n if p < 0.2: return 0\n if p < 0.5: return 1\n if p < 0.8: return 2\n return 3\n\ndef is_iatrogenic(p_gt, p_deg, p_rec):\n tb, td, tr = get_tier(p_gt), get_tier(p_deg), get_tier(p_rec)\n if tb != td:\n return (tr != td) and (abs(tr - tb) > abs(td - tb))\n return tr != tb\n\ndef psnr01_on_slices(vol_a, vol_b, z_list=None):\n a = np.asarray(vol_a, dtype=np.float32)\n b = np.asarray(vol_b, dtype=np.float32)\n if z_list is not None:\n zs = list(z_list)\n if len(zs) == 0:\n return float(\"nan\")\n a = a[zs]; b = b[zs]\n mse = float(np.mean((a - b) ** 2))\n return 99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse)\n\ndef get_sorted_dicom_files(series_path):\n files = [f for f in os.listdir(series_path) if not f.startswith(\".\")]\n if len(files) == 0:\n return []\n pairs, ok = [], True\n for f in files:\n fp = os.path.join(series_path, f)\n try:\n ds = pydicom.dcmread(fp, stop_before_pixels=True, force=True)\n inst = getattr(ds, \"InstanceNumber\", None)\n if inst is None:\n ok = False\n break\n pairs.append((int(inst), fp))\n except Exception:\n ok = False\n break\n if ok and len(pairs) == len(files):\n return [p[1] for p in sorted(pairs, key=lambda x: x[0])]\n return [os.path.join(series_path, f) for f in sorted(files)]\n\ndef load_series_volume(uid, series_root, target_shape=(64, 448, 448)):\n series_path = os.path.join(series_root, uid)\n if not os.path.isdir(series_path):\n return None\n\n dcm_files = get_sorted_dicom_files(series_path)\n if len(dcm_files) < 10:\n return None\n\n tD, tH, tW = target_shape\n if len(dcm_files) != tD:\n idx = np.linspace(0, len(dcm_files) - 1, tD).astype(int)\n dcm_files = [dcm_files[i] for i in idx]\n\n slices = []\n for fp in dcm_files:\n try:\n ds = pydicom.dcmread(fp, force=True)\n arr = ds.pixel_array.astype(np.float32)\n slope = float(getattr(ds, \"RescaleSlope\", 1.0))\n intercept = float(getattr(ds, \"RescaleIntercept\", 0.0))\n hu = arr * slope + intercept\n\n x = (np.clip(hu, HU_MIN, HU_MAX) - HU_MIN) / HU_RANGE\n x = cv2.resize(x, (tW, tH), interpolation=cv2.INTER_LINEAR)\n slices.append(x)\n except Exception:\n continue\n\n if len(slices) < int(0.8 * tD):\n return None\n\n while len(slices) < tD:\n slices.append(slices[-1].copy())\n\n return np.stack(slices[:tD], axis=0).astype(np.float32)\n\ndef degrade_volume(vol01, t, dose_mode=\"quarter\", enable_motion=False):\n \"\"\"\n Evaluation-time degradation (reproducible when called with a fixed random seed).\n \"\"\"\n vol01 = np.asarray(vol01, dtype=np.float32)\n D = vol01.shape[0]\n out = np.empty_like(vol01, dtype=np.float32)\n\n sigma = math.sqrt(max(1e-8, 2.0 * LAM * float(t)))\n\n for z in range(D):\n x = vol01[z].astype(np.float32)\n x = cv2.GaussianBlur(x, (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE)\n\n # Motion artefact (disabled by default)\n if enable_motion:\n pass\n\n # Slice-level stochastic dose noise\n if dose_mode != \"clean\":\n peak = random.uniform(3000.0, 6000.0)\n sigma_e = random.uniform(0.01, 0.02)\n noisy_p = np.random.poisson(np.clip(x * peak, 0, None)).astype(np.float32) / peak\n noisy_g = np.random.randn(*x.shape).astype(np.float32) * sigma_e\n x = noisy_p + noisy_g\n\n out[z] = np.clip(x, 0.0, 1.0).astype(np.float32)\n\n return out\n\n@torch.no_grad()\ndef deblur_volume_25d(model, vol_deg01, t, restore_batch=16, clamp_delta=None):\n \"\"\"\n 2.5D inference function. Supports test-time output clamping for ablation studies.\n \"\"\"\n vol_deg01 = np.asarray(vol_deg01, dtype=np.float32)\n D = vol_deg01.shape[0]\n out = vol_deg01.copy()\n\n t_norm = np.float32(0.0 if t <= 0 else (float(t) / float(BLUR_T_MAX)))\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch, centers = [], []\n\n for z in zs:\n bp = vol_deg01[max(0, z - 1)]\n bc = vol_deg01[z]\n bn = vol_deg01[min(D - 1, z + 1)]\n centers.append(bc)\n inp_batch.append(np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32))\n\n inp_t = torch.from_numpy(np.stack(inp_batch, axis=0)).to(device, non_blocking=True)\n with AMP_CTX():\n pred_b = model(inp_t).float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n pred = pred_b[k]\n if clamp_delta is not None:\n bc = centers[k]\n pred = np.clip(pred, bc - clamp_delta, bc + clamp_delta)\n out[z] = _clip01(pred)\n\n return out\n\ndef read_train_uid_exclusion(ckpt_obj=None):\n \"\"\"\n Load training UIDs for exclusion. Tries local/uploaded CSV files first,\n then falls back to ckpt['train_uids'] if available.\n \"\"\"\n candidates = [\n \"/kaggle/working/train_uids_ultimate.csv\",\n \"/kaggle/input/datasets/mingzeli2009/train-uids-ultimate/train_uids_ultimate.csv\",\n \"/kaggle/working/train_uids_ct_only.csv\",\n \"/kaggle/input/datasets/mingzeli2009/train-uids-ct-only/train_uids_ct_only.csv\",\n ]\n for p in candidates:\n if os.path.exists(p):\n try:\n df = pd.read_csv(p)\n if \"SeriesInstanceUID\" in df.columns:\n s = set(df[\"SeriesInstanceUID\"].astype(str).tolist())\n print(f\"[Train exclusion] loaded {len(s)} train UIDs from: {p}\")\n return s\n except Exception as e:\n print(f\"[Train exclusion] failed reading {p}: {e}\")\n\n if isinstance(ckpt_obj, dict) and \"train_uids\" in ckpt_obj:\n s = set(map(str, ckpt_obj[\"train_uids\"]))\n print(f\"[Train exclusion] fallback to ckpt['train_uids']: {len(s)}\")\n return s\n\n print(\"[Train exclusion] empty set — no exclusion applied\")\n return set()\n\ndef build_ood_uid_pool(meta_csv, series_root, train_uid_set, keep_modalities={\"CT\", \"CTA\"}, seed=2026):\n meta = pd.read_csv(meta_csv)\n ct_uids = set(meta[meta[\"Modality\"].astype(str).isin(keep_modalities)][\"SeriesInstanceUID\"].astype(str).tolist())\n\n all_series_dirs = sorted([\n u for u in os.listdir(series_root)\n if os.path.isdir(os.path.join(series_root, u))\n ])\n pool = [u for u in all_series_dirs if (u in ct_uids) and (u not in train_uid_set)]\n\n rng = random.Random(seed)\n rng.shuffle(pool)\n return pool\n\ndef classify_case_outcome_target(tgain, p_gt, p_rec):\n if tgain > 0.005:\n if (p_gt >= 0.5 and p_rec > p_gt) or (p_gt < 0.5 and p_rec < p_gt):\n return \"super\"\n return \"positive\"\n elif tgain < -0.005:\n return \"negative\"\n else:\n return \"neutral\"\n\nprint(\"All utilities ready!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T00:50:51.73473Z","iopub.execute_input":"2026-03-11T00:50:51.735196Z","iopub.status.idle":"2026-03-11T00:50:51.771971Z","shell.execute_reply.started":"2026-03-11T00:50:51.735159Z","shell.execute_reply":"2026-03-11T00:50:51.770916Z"}},"outputs":[{"name":"stdout","text":"All utilities ready!\n","output_type":"stream"}],"execution_count":22},{"cell_type":"markdown","source":"# §5 Traditional Baselines — Classical Comparators for Fair Evaluation\n\n## Why Include Traditional Baselines?\n\nA restoration model is only convincing when compared against strong non-deep-learning alternatives. Without classical comparators, a reviewer could reasonably ask:\n\n> \"Is the gain really due to the model, or would a simple classical filter already produce similar improvement?\"\n\nTo answer that fairly, this project includes several standard image-processing baselines that are widely known, easy to reproduce, and commonly used as reference points in imaging research.\n\n---\n\n## Baselines Used in This Study\n\n### 1. Identity — \"Do Nothing\"\n\n output = input\n\nThe minimum benchmark. Any useful restoration method should outperform this baseline; otherwise, the model is adding no value.\n\n---\n\n### 2. Gaussian Smoothing\n\nOne of the most widely used classical smoothing methods in image processing. It is included because it is **simple**, **standard**, and **easy to reproduce** — the most common first baseline in restoration studies.\n\n**Limitation**: it reduces noise by local averaging, which also softens edges and fine structures. In vascular CT, that trade-off matters because aneurysm-related detail can be subtle.\n\n---\n\n### 3. Median Filtering\n\nReplaces each pixel with the median value of its local neighbourhood. A classical, robust, non-parametric denoiser that suppresses local noise without relying on a learned model.\n\n**Limitation**: can remove small structures and flatten delicate boundaries.\n\n---\n\n### 4. Bilateral Filtering\n\nAn edge-preserving smoother that considers both **spatial distance** and **intensity similarity** — unlike ordinary Gaussian blur. It is a meaningful medical-imaging baseline because it aims to reduce noise while preserving boundaries better than plain smoothing.\n\n---\n\n### 5. Unsharp Masking\n\nA classical sharpening method:\n\n sharpened = image + α × (image − blurred(image))\n\nTests a different hypothesis from smoothing methods: instead of suppressing noise, it tries to recover apparent sharpness. In low-dose CT, sharpening can also amplify noise and unstable texture, making it a useful \"aggressive classical\" comparator.\n\n---\n\n## Why These Baselines Were Chosen\n\n| Method | Type | Why included |\n|--------|------|-------------|\n| Identity | Null baseline | Establishes minimum useful benchmark |\n| Gaussian | Standard smoothing | Common, transparent, widely recognised |\n| Median | Classical denoising | Robust non-parametric local filter |\n| Bilateral | Edge-preserving smoothing | Strong conventional medical-imaging baseline |\n| Unsharp | Classical sharpening | Tests whether simple sharpening alone suffices |\n\nTogether, they cover **distinct families** of conventional image processing, making the comparison more credible than testing against only a weak or trivial alternative.\n\n---\n\n## Unified Evaluation\n\nAll methods — classical filters and V-Ultimate alike — are scored with the same criteria:\n\n- Target-Aware Gain\n- Absolute Gain\n- Iatrogenic risk\n- PSNR\n- Outcome label\n\nThe comparison is not based on visual appearance. Every method is held to the same quantitative standard.\n\n---\n\n## What This Section Contributes\n\nThis section establishes that V-Ultimate is not merely better than a corrupted input — it is tested against recognisable, reproducible, and scientifically reasonable classical baselines. If V-Ultimate wins, it is not because the bar was set low, but because it outperforms methods that are already standard reference points in image restoration.\n","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 5) Traditional baselines (slice-wise)\n# ============================================================\ndef trad_identity(vol_deg01):\n return np.asarray(vol_deg01, dtype=np.float32).copy()\n\ndef trad_median(vol_deg01, ksize=3):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x8 = (np.clip(vol[z], 0, 1) * 255).astype(np.uint8)\n y8 = cv2.medianBlur(x8, ksize)\n out[z] = (y8.astype(np.float32) / 255.0)\n return out.astype(np.float32)\n\ndef trad_bilateral(vol_deg01, d=7, sigmaColor=35, sigmaSpace=35):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x8 = (np.clip(vol[z], 0, 1) * 255).astype(np.uint8)\n y8 = cv2.bilateralFilter(x8, d=d, sigmaColor=sigmaColor, sigmaSpace=sigmaSpace)\n out[z] = (y8.astype(np.float32) / 255.0)\n return out.astype(np.float32)\n\ndef trad_unsharp(vol_deg01, sigma=1.0, amount=0.8):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x = vol[z]\n blur = cv2.GaussianBlur(x, (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE)\n y = np.clip(x + amount * (x - blur), 0.0, 1.0)\n out[z] = y\n return out.astype(np.float32)\n\nTRAD_METHODS = {\n \"identity\": trad_identity,\n \"median3\": lambda v: trad_median(v, ksize=3),\n \"bilateral\": trad_bilateral,\n \"unsharp\": trad_unsharp,\n}\n\ndef eval_one_reconstruction(gt, deg, rec, p_gt=None, p_deg=None):\n if p_gt is None:\n p_gt = float(aneurysm_predict(vol01_to_flayer_uint8(gt)))\n if p_deg is None:\n p_deg = float(aneurysm_predict(vol01_to_flayer_uint8(deg)))\n p_rec = float(aneurysm_predict(vol01_to_flayer_uint8(rec)))\n\n return {\n \"p_gt\": p_gt,\n \"p_deg\": p_deg,\n \"p_rec\": p_rec,\n \"target_gain\": float(calc_target_gain(p_gt, p_deg, p_rec)),\n \"abs_gain\": float(calc_abs_gain(p_gt, p_deg, p_rec)),\n \"iatrogenic\": int(is_iatrogenic(p_gt, p_deg, p_rec)),\n \"psnr\": float(psnr01_on_slices(rec, gt)),\n \"outcome_target\": classify_case_outcome_target(calc_target_gain(p_gt, p_deg, p_rec), p_gt, p_rec),\n }\n\n# --- 确认输出 ---\nprint(\"✅ 传统方法基线定义完成\")\nprint(\" Gaussian Blur: gaussian_baseline_3d(vol, sigma)\")\nprint(\" Non-Local Means: nlm_baseline_3d(vol, h)\")\nprint(\" 这些方法将作为对照组，与 V-Ultimate 进行比较\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T00:50:57.25776Z","iopub.execute_input":"2026-03-11T00:50:57.258585Z","iopub.status.idle":"2026-03-11T00:50:57.272284Z","shell.execute_reply.started":"2026-03-11T00:50:57.258554Z","shell.execute_reply":"2026-03-11T00:50:57.271314Z"}},"outputs":[{"name":"stdout","text":"✅ 传统方法基线定义完成\n Gaussian Blur: gaussian_baseline_3d(vol, sigma)\n Non-Local Means: nlm_baseline_3d(vol, h)\n 这些方法将作为对照组，与 V-Ultimate 进行比较\n","output_type":"stream"}],"execution_count":23},{"cell_type":"markdown","source":"# Part I: Core Evidence — Why the Model Matters in the Intended Clinical Setting\n\n> This section does **not** begin by asking whether the architecture is elegant. \n> It begins with the more important question: **does the model solve a real problem in its target setting?**\n\nThe three experiments below are arranged to answer the three most natural judge questions in sequence:\n\n| Experiment | Judge's Question | What the experiment tests |\n|-----------|------------------|---------------------------|\n| **Clinical Rescue Matrix (CRM)** | **“Is it actually useful?”** | A controlled OOD comparison using multiple restoration baselines and an independent clinical-proxy judge |\n| **Monte Carlo Noise Stability** | **“Could the result be a lucky noise realization?”** | Repeated multi-seed testing to separate stable benefit from noise-sensitive behavior |\n| **TotalSegmentator Overlap** | **“Is the model editing meaningful anatomy, or just changing the whole image?”** | Anatomical overlap analysis to test whether modifications concentrate in structured regions rather than appearing arbitrarily |\n\nTaken together, these experiments are designed to establish three things:\n\n1. **Effectiveness** — the model can recover clinically relevant signal on OOD CT/CTA cases \n2. **Robustness** — the observed gains are not merely accidental outcomes of one particular noise draw \n3. **Control** — the model's edits are structured and anatomically interpretable, rather than uncontrolled global manipulation \n\nIn other words, Part I is intended to show that **V-Ultimate is not just visually impressive**. \nIt is a restoration system that can be evaluated in terms of **clinical utility, reproducibility, and spatial discipline**.","metadata":{},"attachments":{}},{"cell_type":"markdown","source":"# Experiment 1: Clinical Rescue Matrix (CRM)\n\n## What this experiment is designed to test\n\n> **Question:** \n> On out-of-distribution (OOD) CT/CTA cases, does V-Ultimate produce a **real downstream clinical-proxy benefit**, or does it merely make images look sharper or smoother?\n\nThis is a deliberately practical test. \nInstead of asking only whether the restored image has better pixel similarity, we ask whether restoration helps an **independent aneurysm-detection proxy model** recover clinically meaningful signal.\n\n---\n\n## Controlled comparison setup\n\nTo make the comparison fair, the experiment keeps the following factors fixed:\n\n- **Same OOD CT/CTA case pool**: 50 cases, excluding the 300 training UIDs\n- **Same degradation pipeline**: fixed blur/noise setting (`t = 8.0`, quarter-dose style), with **UID-hash deterministic seeding**\n- **Same downstream judge**: the frozen `FlayerClassifier` ensemble\n- **Same intensity conversion**: HU → CTA windowing → uint8\n- **Only one factor changes**: the restoration method\n\n---\n\n## Methods compared, and why these baselines were included\n\nThis CRM version compares:\n\n- **Degraded**\n- **Gaussian**\n- **Median3**\n- **Bilateral**\n- **Unsharp**\n- **V-Ultimate**\n\n### Why these baselines are reasonable controls\n\nThese baselines are **not presented as state-of-the-art learned restoration systems**. \nThey are included because they are **classical, interpretable, widely recognized image-processing methods** that cover the main traditional strategies people use before deep learning:\n\n| Method | Why include it? | What it represents |\n|--------|------------------|-------------------|\n| **Degraded** | Negative control | What happens if we do nothing |\n| **Gaussian blur** | Very common classical smoothing baseline | Simple denoising by low-pass filtering |\n| **Median3** | Standard robust denoiser | Suppresses local outliers while being relatively conservative |\n| **Bilateral filter** | Widely used edge-aware classical filter | Smooths noise while trying to preserve edges |\n| **Unsharp masking** | Standard sharpening baseline | Tests whether “improvement” is just local sharpening rather than true restoration |\n| **V-Ultimate** | Proposed method | Physics-constrained restoration with controlled voxel-wise editing |\n\n### Why these baselines matter scientifically\n\nTogether, these methods form a meaningful classical comparison set:\n\n- **Gaussian** asks: does simple smoothing already solve the problem?\n- **Median3** asks: is a conservative denoiser enough?\n- **Bilateral** asks: can edge-preserving filtering recover the signal without learning?\n- **Unsharp** asks: are apparent gains just a consequence of sharpening?\n\nSo if V-Ultimate outperforms this group, it suggests the model is doing something more specific than generic smoothing or generic sharpening.\n\n---\n\n## Main metrics and why they are used\n\n### Why report both Target-Aware Gain and Absolute Gain?\n\nThese two metrics answer **different questions**, and both are important.\n\n| Metric | What it measures | Intuition |\n|--------|------------------|----------|\n| **Absolute Gain** | Whether the restored probability gets numerically closer to the original clean-image probability | A strict distance-based measure |\n| **Target-Aware Gain** | Whether the restoration moves the judge in the clinically useful direction, even if it slightly overshoots | A task-oriented measure |\n\n### Why that distinction matters\n\nA restoration can be **directionally correct** but still not be the numerically closest probability.\n\nFor example, if degradation suppresses a positive aneurysm signal, then moving the score back upward is clinically meaningful. \nA purely distance-based metric may punish that move if it slightly overshoots. \nThat is why **Target-Aware Gain** is needed: it captures whether the restoration is helping the downstream task, not just whether it is minimizing a regression error.\n\n### Additional safety / fidelity metrics\n\n- **Iatrogenic Harm Rate**: whether restoration changes the diagnostic tier in a harmful way\n- **PSNR**: pixel-level fidelity reference\n- **Case-level CRM label**: an interpretable summary of outcome per case\n\n---\n\n## Success criteria\n\n| Role | Criterion | Meaning |\n|------|-----------|---------|\n| **Primary** | **Mean Target Gain / Target Win %** | Does restoration help the downstream judge more often than not? |\n| **Safety** | **Iatrogenic Harm Rate** | Does the method avoid clinically risky over-correction? |\n| **Secondary** | **PSNR / Absolute Gain** | Does it preserve image fidelity while improving the proxy task? |\n\n---\n\n## Case-level CRM labels\n\n| Label | Meaning |\n|------|---------|\n| **✅ Successful Rescue** | Restoration improves the downstream clinical-proxy judgment |\n| **➖ Algorithmic Humility** | Restoration makes little or no meaningful change |\n| **⏬ Minor Deviation** | Restoration moves in the wrong direction, but not catastrophically |\n| **⚠️ Iatrogenic** | Restoration crosses into a clinically harmful directional change |\n\n---\n","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# Clinical Rescue Matrix — FULL VERSION (STRICT EXCLUSION)\n# Compatible with: FiLM + authority map V-Ultimate\n# Compares:\n# Degraded / Gaussian / Median3 / Bilateral / Unsharp / (optional NLM) / V-Ultimate\n#\n# - Explicitly excludes TRAIN + REFINE + VAL UIDs\n# - Prints overlap sanity checks\n# ============================================================\n\nimport os, sys, time, math, random, gc\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport torch\nfrom contextlib import nullcontext\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\n# ============================================================\n# 0) Prerequisites check\n# ============================================================\nrequired_any = {\n \"model\": (\"model_25d\" in globals()),\n \"judge\": (\"aneurysm_predict\" in globals()),\n \"loader\": (\"load_series_volume\" in globals()),\n \"stable_uid_seed\": (\"stable_uid_seed\" in globals()),\n \"uid_tail4\": (\"uid_tail4\" in globals()),\n \"_clip01\": (\"_clip01\" in globals()),\n \"calc_abs_gain\": (\"calc_abs_gain\" in globals()),\n \"calc_target_gain\": (\"calc_target_gain\" in globals()),\n \"is_iatrogenic\": (\"is_iatrogenic\" in globals()),\n \"vol01_to_flayer_uint8\": (\"vol01_to_flayer_uint8\" in globals()),\n}\nif not all(required_any.values()):\n missing = [k for k, v in required_any.items() if not v]\n raise RuntimeError(\n f\"Missing prerequisite objects/functions: {missing}\\n\"\n f\"Please run: Utilities + new model loading + Clinical Judge + DICOM loading cells first.\"\n )\n\nMODEL_OBJ = globals().get(\"model_25d\", None)\nassert MODEL_OBJ is not None, \"model_25d not found\"\nassert hasattr(MODEL_OBJ, \"auth_head\"), \"Not the new model: missing auth_head\"\nassert hasattr(MODEL_OBJ, \"film_e1\"), \"Not the new model: missing FiLM module\"\n\n# ============================================================\n# 1) Core configuration\n# ============================================================\nRSNA_DATA_ROOT = globals().get(\"RSNA_DATA_ROOT\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\")\nMETA_CSV = globals().get(\"META_CSV\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\")\n\nTARGET_D = int(globals().get(\"TARGET_D\", 64))\nTARGET_H = int(globals().get(\"TARGET_H\", 448))\nTARGET_W = int(globals().get(\"TARGET_W\", 448))\nTARGET_SHAPE = (TARGET_D, TARGET_H, TARGET_W)\n\nHU_MIN = float(globals().get(\"HU_MIN\", -1024.0))\nHU_MAX = float(globals().get(\"HU_MAX\", 3072.0))\nHU_RANGE = float(globals().get(\"HU_RANGE\", HU_MAX - HU_MIN))\n\nDIFFUSION_ALPHA = float(globals().get(\"DIFFUSION_ALPHA\", globals().get(\"LAM\", 0.20)))\nBLUR_T_MAX = float(globals().get(\"BLUR_T_MAX\", 8.0))\n\nEVAL_T = 8.0\nEVAL_DOSE = \"quarter\"\n\nN_CRM_CASES = 50\nSEED_CRM = 2026\nRESTORE_BATCH = int(globals().get(\"RESTORE_BATCH\", 16))\n\n# traditional baselines\nGAUSS_SIGMA = 0.8\nMEDIAN_KSIZE = 3\nBILATERAL_D = 7\nBILATERAL_SIGMACOLOR = 35\nBILATERAL_SIGMASPACE = 35\nUNSHARP_SIGMA = 1.0\nUNSHARP_AMOUNT = 0.8\n\nUSE_NLM = False\nNLM_H = 7\n\nGAIN_POS_TH = 0.005\nGAIN_NEG_TH = -0.005\n\nOUTDIR = \"/kaggle/working/clinical_rescue_matrix\"\nos.makedirs(OUTDIR, exist_ok=True)\nRAW_CSV = os.path.join(OUTDIR, f\"crm_raw_N{N_CRM_CASES}.csv\")\nSUM_CSV = os.path.join(OUTDIR, f\"crm_summary_N{N_CRM_CASES}.csv\")\nPAIR_CSV = os.path.join(OUTDIR, f\"crm_paired_vs_vultimate_N{N_CRM_CASES}.csv\")\nBUCKET_CSV = os.path.join(OUTDIR, f\"crm_bucket_breakdown_N{N_CRM_CASES}.csv\")\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = (lambda: torch.amp.autocast(\"cuda\")) if USE_AMP else (lambda: nullcontext())\n\n# Method list\nMETHODS = [\n (\"Degraded\", \"degraded\"),\n (\"Gaussian\", \"gaussian\"),\n (\"Median3\", \"median3\"),\n (\"Bilateral\", \"bilateral\"),\n (\"Unsharp\", \"unsharp\"),\n]\nif USE_NLM:\n METHODS.append((\"NLM\", \"nlm\"))\nMETHODS.append((\"V-Ultimate\", \"vultimate\"))\n\nprint(\"Methods to compare:\", [m[0] for m in METHODS])\n\n# ============================================================\n# 2) Metrics and labels\n# ============================================================\ndef psnr01_full(vol_a, vol_b):\n a = np.asarray(vol_a, dtype=np.float32)\n b = np.asarray(vol_b, dtype=np.float32)\n mse = float(np.mean((a - b) ** 2))\n return 99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse)\n\ndef crm_outcome_label(p_gt, p_deg, p_rec, abs_gain, pos_th=GAIN_POS_TH, neg_th=GAIN_NEG_TH):\n if is_iatrogenic(p_gt, p_deg, p_rec):\n return \"⚠️ Iatrogenic\"\n if abs_gain > pos_th:\n return \"✅ Successful Rescue\"\n if abs_gain < neg_th:\n return \"⏬ Minor Deviation\"\n return \"➖ Algorithmic Humility\"\n\ndef crm_bucket(p_gt, p_deg, tau=0.03):\n # degradation suppressed / inflated / negligible effect on score\n if p_gt >= 0.5 and p_deg < p_gt - tau:\n return \"harmed_positive\"\n if p_gt >= 0.5 and p_deg > p_gt + tau:\n return \"overcall_positive\"\n return \"neutral_other\"\n\n# ============================================================\n# 3) Deterministic degradation (UID-level fixed seed)\n# ============================================================\ndef degrade_volume_fixed_uid(vol01, t, uid, dose_mode=\"quarter\"):\n local_seed = stable_uid_seed(uid)\n py_state, np_state = random.getstate(), np.random.get_state()\n random.seed(local_seed)\n np.random.seed(local_seed % (2**32 - 1))\n\n D = vol01.shape[0]\n out = np.empty_like(vol01, dtype=np.float32)\n sigma = math.sqrt(max(1e-8, 2.0 * DIFFUSION_ALPHA * float(t)))\n\n for z in range(D):\n x = vol01[z].astype(np.float32)\n x = cv2.GaussianBlur(\n x, (0, 0),\n sigmaX=sigma, sigmaY=sigma,\n borderType=cv2.BORDER_REPLICATE\n )\n if dose_mode != \"clean\":\n peak = random.uniform(3000.0, 6000.0)\n sigma_e = random.uniform(0.01, 0.02)\n noisy_p = np.random.poisson(np.clip(x * peak, 0, None)).astype(np.float32) / peak\n x = noisy_p + np.random.randn(*x.shape).astype(np.float32) * sigma_e\n out[z] = np.clip(x, 0.0, 1.0).astype(np.float32)\n\n random.setstate(py_state)\n np.random.set_state(np_state)\n return out\n\n# ============================================================\n# 4) Traditional baselines\n# ============================================================\ndef trad_identity(vol_deg01):\n return np.asarray(vol_deg01, dtype=np.float32).copy()\n\ndef trad_gaussian(vol_deg01, sigma=GAUSS_SIGMA):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n out[z] = _clip01(\n cv2.GaussianBlur(\n vol[z], (0, 0),\n sigmaX=sigma, sigmaY=sigma,\n borderType=cv2.BORDER_REPLICATE\n )\n )\n return out.astype(np.float32)\n\ndef trad_median(vol_deg01, ksize=MEDIAN_KSIZE):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x8 = (np.clip(vol[z], 0, 1) * 255).astype(np.uint8)\n y8 = cv2.medianBlur(x8, ksize)\n out[z] = (y8.astype(np.float32) / 255.0)\n return out.astype(np.float32)\n\ndef trad_bilateral(vol_deg01, d=BILATERAL_D, sigmaColor=BILATERAL_SIGMACOLOR, sigmaSpace=BILATERAL_SIGMASPACE):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x8 = (np.clip(vol[z], 0, 1) * 255).astype(np.uint8)\n y8 = cv2.bilateralFilter(x8, d=d, sigmaColor=sigmaColor, sigmaSpace=sigmaSpace)\n out[z] = (y8.astype(np.float32) / 255.0)\n return out.astype(np.float32)\n\ndef trad_unsharp(vol_deg01, sigma=UNSHARP_SIGMA, amount=UNSHARP_AMOUNT):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n x = vol[z]\n blur = cv2.GaussianBlur(x, (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE)\n y = np.clip(x + amount * (x - blur), 0.0, 1.0)\n out[z] = y\n return out.astype(np.float32)\n\ndef window_hu_to_uint8(hu, center=40.0, width=400.0):\n x = np.clip((hu - (center - width / 2.0)) / (width + 1e-6), 0.0, 1.0)\n return (x * 255.0).astype(np.uint8)\n\ndef hu01_to_hu(vol01):\n return np.asarray(vol01, dtype=np.float32) * HU_RANGE + HU_MIN\n\ndef hu_to_01(hu):\n return np.clip((hu - HU_MIN) / HU_RANGE, 0.0, 1.0).astype(np.float32)\n\ndef trad_nlm(vol_deg01, h=NLM_H, center=40.0, width=400.0):\n vol = np.asarray(vol_deg01, dtype=np.float32)\n vol_hu = hu01_to_hu(vol)\n out = np.empty_like(vol)\n for z in range(vol.shape[0]):\n u8 = window_hu_to_uint8(vol_hu[z], center=center, width=width)\n den = cv2.fastNlMeansDenoising(u8, None, h=h, templateWindowSize=7, searchWindowSize=21)\n den01_win = den.astype(np.float32) / 255.0\n den_hu = den01_win * width + (center - width / 2.0)\n out[z] = hu_to_01(den_hu)\n return out.astype(np.float32)\n\nTRAD_METHODS = {\n \"degraded\": trad_identity,\n \"gaussian\": trad_gaussian,\n \"median3\": trad_median,\n \"bilateral\": trad_bilateral,\n \"unsharp\": trad_unsharp,\n \"nlm\": trad_nlm,\n}\n\n# ============================================================\n# 5) V-Ultimate inference (FiLM + authority map)\n# ============================================================\n@torch.no_grad()\ndef run_vultimate(vol_deg01, t=EVAL_T, restore_batch=RESTORE_BATCH):\n \"\"\"\n FiLM + authority map compatible inference.\n meta = [t_norm, do_motion, dose_clean, is_identity]\n \"\"\"\n vol_deg01 = np.asarray(vol_deg01, dtype=np.float32)\n D = vol_deg01.shape[0]\n out = vol_deg01.copy()\n\n t_norm = np.float32(0.0 if t <= 0 else (float(t) / float(BLUR_T_MAX)))\n meta_row = np.array([t_norm, 0.0, 0.0, 0.0], dtype=np.float32) # do_motion=0, dose_clean=0, is_identity=0\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch = []\n\n for z in zs:\n bp = vol_deg01[max(0, z - 1)]\n bc = vol_deg01[z]\n bn = vol_deg01[min(D - 1, z + 1)]\n inp_batch.append(\n np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n )\n\n inp_t = torch.from_numpy(np.stack(inp_batch, axis=0)).to(device, non_blocking=True)\n meta_t = torch.from_numpy(np.repeat(meta_row[None, :], len(zs), axis=0)).to(device, non_blocking=True)\n\n with AMP_CTX():\n pred_obj = MODEL_OBJ(inp_t, meta=meta_t)\n if isinstance(pred_obj, (tuple, list)):\n pred_b = pred_obj[0].float().cpu().numpy()[:, 0]\n else:\n pred_b = pred_obj.float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n out[z] = _clip01(pred_b[k])\n\n return out\n\n# ============================================================\n# 6) Build STRICT OOD CT/CTA test pool\n# Exclude: TRAIN + REFINE + VAL\n# ============================================================\ndef _to_uid_set(obj):\n if obj is None:\n return set()\n if isinstance(obj, set):\n return {str(x) for x in obj}\n if isinstance(obj, (list, tuple, np.ndarray, pd.Series)):\n return {str(x) for x in obj}\n if isinstance(obj, pd.DataFrame):\n if obj.shape[1] == 0:\n return set()\n return set(obj.iloc[:, 0].astype(str).tolist())\n try:\n return {str(x) for x in list(obj)}\n except Exception:\n return set()\n\ndef _load_uid_set_from_globals(global_names):\n for gname in global_names:\n if gname in globals():\n try:\n s = _to_uid_set(globals()[gname])\n if len(s) > 0:\n return s, f\"global:{gname}\"\n except Exception:\n pass\n return set(), None\n\ndef _load_uid_set_from_csvs(csv_candidates):\n for p in csv_candidates:\n if os.path.exists(p):\n try:\n s = set(pd.read_csv(p).iloc[:, 0].astype(str).tolist())\n if len(s) > 0:\n return s, p\n except Exception:\n pass\n return set(), None\n\ndef _load_uid_set(label, global_names, csv_candidates, required=False):\n s, src = _load_uid_set_from_globals(global_names)\n if len(s) > 0:\n print(f\"[{label}] loaded {len(s)} UIDs from {src}\")\n return s\n\n s, src = _load_uid_set_from_csvs(csv_candidates)\n if len(s) > 0:\n print(f\"[{label}] loaded {len(s)} UIDs from {src}\")\n return s\n\n msg = f\"[{label}] no UID set found\"\n if required:\n raise RuntimeError(msg + f\". Checked: {csv_candidates}\")\n else:\n print(f\"[{label}] WARNING: no UID set found; using empty set\")\n return set()\n\nprint(\"=== Clinical Rescue Matrix | OOD CT/CTA (STRICT EXCLUSION) ===\")\n\nmeta = pd.read_csv(META_CSV)\nct_uids = set(\n meta[meta[\"Modality\"].astype(str).isin({\"CT\", \"CTA\"})][\"SeriesInstanceUID\"].astype(str).tolist()\n)\n\nUID_SPLIT_DIR = \"/kaggle/input/datasets/linguoyuemma/refine-uids\"\n\n# train\ntrain_uid_set = _load_uid_set(\n label=\"TRAIN exclusion\",\n global_names=[\n \"train_uid_set\", \"train_uids\", \"train_uids_ct\", \"train_series_uids\"\n ],\n csv_candidates=[\n f\"{UID_SPLIT_DIR}/train_uids.csv\",\n \"/kaggle/working/vultimate_scientific/train_uids.csv\",\n \"/kaggle/working/vultimate_sleep_safe/train_uids_ultimate.csv\",\n \"/kaggle/working/train_uids_ultimate.csv\",\n \"/kaggle/input/datasets/mingzeli2009/train-uids-ultimate/train_uids_ultimate.csv\",\n \"/kaggle/working/train_uids_ct_only.csv\",\n ],\n required=True,\n)\n\n# refine\nrefine_uid_set = _load_uid_set(\n label=\"REFINE exclusion\",\n global_names=[\n \"refine_uid_set\", \"refine_uids\", \"hard_uid_set\", \"hardcase_uid_set\", \"refine_series_uids\"\n ],\n csv_candidates=[\n f\"{UID_SPLIT_DIR}/refine_uids.csv\",\n \"/kaggle/working/vultimate_scientific/refine_uids.csv\",\n \"/kaggle/working/vultimate_sleep_safe/refine_uids_ultimate.csv\",\n \"/kaggle/working/refine_uids_ultimate.csv\",\n \"/kaggle/working/refine_uids.csv\",\n \"/kaggle/input/datasets/mingzeli2009/refine-uids-ultimate/refine_uids_ultimate.csv\",\n ],\n required=True,\n)\n\n# val\nval_uid_set = _load_uid_set(\n label=\"VAL exclusion\",\n global_names=[\n \"val_uid_set\", \"val_uids\", \"valid_uid_set\", \"valid_uids\", \"val_series_uids\"\n ],\n csv_candidates=[\n f\"{UID_SPLIT_DIR}/val_uids.csv\",\n \"/kaggle/working/vultimate_scientific/val_uids.csv\",\n \"/kaggle/working/vultimate_sleep_safe/val_uids_ultimate.csv\",\n \"/kaggle/working/val_uids_ultimate.csv\",\n \"/kaggle/working/val_uids.csv\",\n \"/kaggle/input/datasets/mingzeli2009/val-uids-ultimate/val_uids_ultimate.csv\",\n ],\n required=True,\n)\n\ndev_uid_set = set(train_uid_set) | set(refine_uid_set) | set(val_uid_set)\n\nprint(\"\\n=== Exclusion summary ===\")\nprint(\"train_uid_set :\", len(train_uid_set))\nprint(\"refine_uid_set:\", len(refine_uid_set))\nprint(\"val_uid_set :\", len(val_uid_set))\nprint(\"dev_uid_set :\", len(dev_uid_set))\n\nall_series_dirs = [\n u for u in os.listdir(RSNA_DATA_ROOT)\n if os.path.isdir(os.path.join(RSNA_DATA_ROOT, u))\n]\n\nood_pool = [u for u in all_series_dirs if (u in ct_uids) and (u not in dev_uid_set)]\n\nprint(\"\\n=== Overlap sanity check (should all be 0) ===\")\nprint(\"ood ∩ train :\", len(set(ood_pool) & set(train_uid_set)))\nprint(\"ood ∩ refine:\", len(set(ood_pool) & set(refine_uid_set)))\nprint(\"ood ∩ val :\", len(set(ood_pool) & set(val_uid_set)))\n\nrandom.Random(SEED_CRM).shuffle(ood_pool)\nprint(f\"\\nCT/CTA pool after strict exclusion: {len(ood_pool)}\")\n\nprefetch = ood_pool[:max(N_CRM_CASES * 3, N_CRM_CASES)]\n\n# ============================================================\n# 7) Main loop\n# ============================================================\nrows = []\nvalid_cases = 0\nt0_all = time.time()\n\nfor uid in prefetch:\n if valid_cases >= N_CRM_CASES:\n break\n\n case_t0 = time.time()\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT, TARGET_SHAPE)\n except TypeError:\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT)\n except TypeError:\n vol = load_series_volume(uid)\n\n if vol is None:\n continue\n\n gt = vol.astype(np.float32)\n deg = degrade_volume_fixed_uid(gt, EVAL_T, uid, dose_mode=EVAL_DOSE)\n\n p_gt = float(aneurysm_predict(vol01_to_flayer_uint8(gt)))\n p_deg = float(aneurysm_predict(vol01_to_flayer_uint8(deg)))\n\n bucket = crm_bucket(p_gt, p_deg, tau=0.03)\n valid_cases += 1\n uid4 = uid_tail4(uid)\n\n print(f\"\\n[{valid_cases:03d}/{N_CRM_CASES}] UID:{uid4} | GT:{p_gt:.4f} -> Deg:{p_deg:.4f} | bucket={bucket}\")\n\n for m_name, m_key in METHODS:\n try:\n if m_key == \"vultimate\":\n rec = run_vultimate(deg, t=EVAL_T, restore_batch=RESTORE_BATCH)\n elif m_key in TRAD_METHODS:\n rec = TRAD_METHODS[m_key](deg)\n else:\n raise ValueError(f\"Unknown method key: {m_key}\")\n\n p_rec = float(aneurysm_predict(vol01_to_flayer_uint8(rec)))\n abs_gain = float(calc_abs_gain(p_gt, p_deg, p_rec))\n tgt_gain = float(calc_target_gain(p_gt, p_deg, p_rec))\n iatro = int(is_iatrogenic(p_gt, p_deg, p_rec))\n psnr_val = float(psnr01_full(rec, gt))\n outcome = crm_outcome_label(p_gt, p_deg, p_rec, abs_gain)\n\n rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"bucket\": bucket,\n \"method\": m_name,\n \"method_key\": m_key,\n \"p_gt\": p_gt,\n \"p_deg\": p_deg,\n \"p_rec\": p_rec,\n \"abs_gain\": abs_gain,\n \"target_gain\": tgt_gain,\n \"iatrogenic\": iatro,\n \"outcome\": outcome,\n \"psnr_db\": psnr_val,\n \"eval_t\": EVAL_T,\n \"eval_dose\": EVAL_DOSE,\n \"gauss_sigma\": GAUSS_SIGMA if m_key == \"gaussian\" else np.nan,\n \"median_ksize\": MEDIAN_KSIZE if m_key == \"median3\" else np.nan,\n \"bilateral_d\": BILATERAL_D if m_key == \"bilateral\" else np.nan,\n \"unsharp_amount\": UNSHARP_AMOUNT if m_key == \"unsharp\" else np.nan,\n \"nlm_h\": NLM_H if m_key == \"nlm\" else np.nan,\n })\n\n tag = (\n \"✅ rescue\" if tgt_gain > GAIN_POS_TH\n else \"⚠️ negative\" if tgt_gain < GAIN_NEG_TH\n else \"➖ identity\"\n )\n print(\n f\" ├─ {m_name:<10s} | Rec:{p_rec:.4f} | \"\n f\"TGain:{tgt_gain:+.4f} | AGain:{abs_gain:+.4f} | {outcome} | {tag}\"\n )\n\n if m_key != \"degraded\":\n del rec\n\n except Exception as e:\n rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"bucket\": bucket,\n \"method\": m_name,\n \"method_key\": m_key,\n \"error\": repr(e),\n })\n print(f\" ├─ {m_name:<10s} | ERROR: {repr(e)}\")\n\n print(f\" -> case done in {time.time() - case_t0:.1f}s\")\n\n del gt, deg, vol\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\nprint(f\"\\nAll CRM cases done. elapsed={(time.time() - t0_all)/60:.1f} min\")\nprint(f\"Valid cases: {valid_cases}\")\n\n# ============================================================\n# 8) Summary table\n# ============================================================\ndf = pd.DataFrame(rows)\ndf.to_csv(RAW_CSV, index=False)\nif len(df) == 0:\n raise RuntimeError(\"No results generated.\")\n\ndf_ok = df.dropna(subset=[\"target_gain\", \"abs_gain\", \"psnr_db\"]).copy()\ncrm_categories = [\"✅ Successful Rescue\", \"➖ Algorithmic Humility\", \"⏬ Minor Deviation\", \"⚠️ Iatrogenic\"]\n\nsummary_rows = []\nfor m_name, _ in METHODS:\n sub = df_ok[df_ok[\"method\"] == m_name].copy()\n if len(sub) == 0:\n continue\n\n oc = sub[\"outcome\"].value_counts()\n ocp = {c: (float(oc.get(c, 0)) / len(sub) * 100.0) for c in crm_categories}\n\n summary_rows.append({\n \"Method\": m_name,\n \"N\": int(len(sub)),\n \"Mean Target Gain\": round(float(sub[\"target_gain\"].mean()), 4),\n \"Target Win %\": round(float((sub[\"target_gain\"] > GAIN_POS_TH).mean() * 100), 1),\n \"Target Neg %\": round(float((sub[\"target_gain\"] < GAIN_NEG_TH).mean() * 100), 1),\n \"Mean Abs Gain\": round(float(sub[\"abs_gain\"].mean()), 4),\n \"Iatrogenic %\": f\"{round(float(sub['iatrogenic'].mean() * 100), 1)}%\",\n \"PSNR (dB)\": round(float(sub[\"psnr_db\"].mean()), 2),\n \"CRM_Rescue\": f\"{int(oc.get('✅ Successful Rescue', 0))} ({ocp['✅ Successful Rescue']:.1f}%)\",\n \"CRM_Humble\": f\"{int(oc.get('➖ Algorithmic Humility', 0))} ({ocp['➖ Algorithmic Humility']:.1f}%)\",\n \"CRM_Deviate\": f\"{int(oc.get('⏬ Minor Deviation', 0))} ({ocp['⏬ Minor Deviation']:.1f}%)\",\n \"CRM_Iatro\": f\"{int(oc.get('⚠️ Iatrogenic', 0))} ({ocp['⚠️ Iatrogenic']:.1f}%)\",\n })\n\ndf_sum = pd.DataFrame(summary_rows)\ndf_sum.to_csv(SUM_CSV, index=False)\n\nprint(\"\\n\" + \"=\" * 100)\nprint(f\"Clinical Rescue Matrix — Summary (STRICT OOD CT/CTA, N={valid_cases})\")\nprint(\"=\" * 100)\ndisplay(df_sum)\n\n# ============================================================\n# 9) Bucket breakdown\n# ============================================================\nprint(\"\\n=== Bucket breakdown ===\")\nbucket_rows = []\nfor bucket in sorted(df_ok[\"bucket\"].dropna().unique()):\n for m_name, _ in METHODS:\n sub = df_ok[(df_ok[\"bucket\"] == bucket) & (df_ok[\"method\"] == m_name)].copy()\n if len(sub) == 0:\n continue\n bucket_rows.append({\n \"bucket\": bucket,\n \"method\": m_name,\n \"N\": int(len(sub)),\n \"mean_target_gain\": round(float(sub[\"target_gain\"].mean()), 4),\n \"mean_abs_gain\": round(float(sub[\"abs_gain\"].mean()), 4),\n \"iatrogenic_%\": round(float(sub[\"iatrogenic\"].mean() * 100), 1),\n \"psnr_db\": round(float(sub[\"psnr_db\"].mean()), 2),\n })\n\ndf_bucket = pd.DataFrame(bucket_rows)\ndf_bucket.to_csv(BUCKET_CSV, index=False)\ndisplay(df_bucket)\n\n# ============================================================\n# 10) Paired comparison (baseline: V-Ultimate)\n# ============================================================\nif \"V-Ultimate\" in set(df_ok[\"method\"].unique()):\n base = df_ok[df_ok[\"method\"] == \"V-Ultimate\"][[\n \"uid_full\", \"target_gain\", \"abs_gain\", \"iatrogenic\", \"psnr_db\", \"p_rec\", \"bucket\"\n ]].rename(columns={\n \"target_gain\": \"tgain_base\",\n \"abs_gain\": \"again_base\",\n \"iatrogenic\": \"iatro_base\",\n \"psnr_db\": \"psnr_base\",\n \"p_rec\": \"p_rec_base\",\n })\n\n paired_rows = []\n for m_name, _ in METHODS:\n if m_name == \"V-Ultimate\":\n continue\n\n sub = df_ok[df_ok[\"method\"] == m_name][[\n \"uid_full\", \"target_gain\", \"abs_gain\", \"iatrogenic\", \"psnr_db\", \"p_rec\"\n ]].rename(columns={\n \"target_gain\": \"tgain_cmp\",\n \"abs_gain\": \"again_cmp\",\n \"iatrogenic\": \"iatro_cmp\",\n \"psnr_db\": \"psnr_cmp\",\n \"p_rec\": \"p_rec_cmp\",\n })\n\n m = base.merge(sub, on=\"uid_full\", how=\"inner\")\n if len(m) == 0:\n continue\n\n dt = m[\"tgain_base\"] - m[\"tgain_cmp\"]\n paired_rows.append({\n \"vs\": m_name,\n \"N\": int(len(m)),\n \"V wins TGain %\": round(float((dt > 0.001).mean() * 100), 1),\n \"V loses TGain %\": round(float((dt < -0.001).mean() * 100), 1),\n \"ΔTGain\": round(float(dt.mean()), 4),\n \"ΔPSNR\": round(float((m[\"psnr_base\"] - m[\"psnr_cmp\"]).mean()), 3),\n })\n\n df_pair = pd.DataFrame(paired_rows)\n df_pair.to_csv(PAIR_CSV, index=False)\n\n print(\"\\nPaired comparison (baseline: V-Ultimate)\")\n display(df_pair)\n\n# ============================================================\n# 11) Top / Bottom 5\n# ============================================================\nif \"V-Ultimate\" in set(df_ok[\"method\"].unique()):\n v = df_ok[df_ok[\"method\"] == \"V-Ultimate\"].copy()\n cols = [\n \"uid4\", \"bucket\", \"p_gt\", \"p_deg\", \"p_rec\",\n \"target_gain\", \"abs_gain\", \"iatrogenic\", \"outcome\", \"psnr_db\"\n ]\n\n print(\"\\n=== V-Ultimate Top-5 ===\")\n display(v.sort_values(\"target_gain\", ascending=False).head(5)[cols].reset_index(drop=True))\n\n print(\"\\n=== V-Ultimate Bottom-5 ===\")\n display(v.sort_values(\"target_gain\", ascending=True).head(5)[cols].reset_index(drop=True))\n\nprint(\"\\nSaved:\")\nprint(\" raw ->\", RAW_CSV)\nprint(\" sum ->\", SUM_CSV)\nprint(\" bucket ->\", BUCKET_CSV)\nif os.path.exists(PAIR_CSV):\n print(\" pair ->\", PAIR_CSV)\n\nprint(\"\\n✅ Clinical Rescue Matrix complete.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T06:32:04.922889Z","iopub.execute_input":"2026-03-11T06:32:04.923691Z","iopub.status.idle":"2026-03-11T07:13:29.038678Z","shell.execute_reply.started":"2026-03-11T06:32:04.923661Z","shell.execute_reply":"2026-03-11T07:13:29.038129Z"}},"outputs":[{"name":"stdout","text":"Methods to compare: ['Degraded', 'Gaussian', 'Median3', 'Bilateral', 'Unsharp', 'V-Ultimate']\n=== Clinical Rescue Matrix | OOD CT/CTA (STRICT EXCLUSION) ===\n[TRAIN exclusion] loaded 300 UIDs from global:train_uid_set\n[REFINE exclusion] loaded 60 UIDs from /kaggle/input/datasets/linguoyuemma/refine-uids/refine_uids.csv\n[VAL exclusion] loaded 20 UIDs from /kaggle/input/datasets/linguoyuemma/refine-uids/val_uids.csv\n\n=== Exclusion summary ===\ntrain_uid_set : 300\nrefine_uid_set: 60\nval_uid_set : 20\ndev_uid_set : 365\n\n=== Overlap sanity check (should all be 0) ===\nood ∩ train : 0\nood ∩ refine: 0\nood ∩ val : 0\n\nCT/CTA pool after strict exclusion: 1443\n\n[001/50] UID:1968 | GT:0.6943 -> Deg:0.6724 | bucket=neutral_other\n ├─ Degraded | Rec:0.6724 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6958 | TGain:+0.0234 | AGain:+0.0205 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.6870 | TGain:+0.0146 | AGain:+0.0146 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6934 | TGain:+0.0210 | AGain:+0.0210 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.6479 | TGain:-0.0244 | AGain:-0.0244 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.6895 | TGain:+0.0171 | AGain:+0.0171 | ✅ Successful Rescue | ✅ rescue\n -> case done in 46.8s\n\n[002/50] UID:0607 | GT:0.7925 -> Deg:0.7935 | bucket=neutral_other\n ├─ Degraded | Rec:0.7935 | TGain:+0.0010 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8062 | TGain:+0.0137 | AGain:-0.0127 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Median3 | Rec:0.8071 | TGain:+0.0146 | AGain:-0.0137 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Bilateral | Rec:0.8145 | TGain:+0.0220 | AGain:-0.0210 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Unsharp | Rec:0.7148 | TGain:-0.0776 | AGain:-0.0767 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8276 | TGain:+0.0352 | AGain:-0.0342 | ⚠️ Iatrogenic | ✅ rescue\n -> case done in 48.0s\n\n[003/50] UID:7954 | GT:0.8677 -> Deg:0.8003 | bucket=harmed_positive\n ├─ Degraded | Rec:0.8003 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8589 | TGain:+0.0586 | AGain:+0.0586 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8257 | TGain:+0.0254 | AGain:+0.0254 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7822 | TGain:-0.0181 | AGain:-0.0181 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ Unsharp | Rec:0.7959 | TGain:-0.0044 | AGain:-0.0044 | ⚠️ Iatrogenic | ➖ identity\n ├─ V-Ultimate | Rec:0.8252 | TGain:+0.0249 | AGain:+0.0249 | ✅ Successful Rescue | ✅ rescue\n -> case done in 49.1s\n\n[004/50] UID:4381 | GT:0.8457 -> Deg:0.7456 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7456 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8223 | TGain:+0.0767 | AGain:+0.0767 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7983 | TGain:+0.0527 | AGain:+0.0527 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8374 | TGain:+0.0918 | AGain:+0.0918 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7808 | TGain:+0.0352 | AGain:+0.0352 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8931 | TGain:+0.1475 | AGain:+0.0527 | ✅ Successful Rescue | ✅ rescue\n -> case done in 46.3s\n\n[005/50] UID:5233 | GT:0.8735 -> Deg:0.7705 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7705 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8516 | TGain:+0.0811 | AGain:+0.0811 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8540 | TGain:+0.0835 | AGain:+0.0835 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8223 | TGain:+0.0518 | AGain:+0.0518 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7593 | TGain:-0.0112 | AGain:-0.0112 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8618 | TGain:+0.0913 | AGain:+0.0913 | ✅ Successful Rescue | ✅ rescue\n -> case done in 53.0s\n\n[006/50] UID:1035 | GT:0.7832 -> Deg:0.7632 | bucket=neutral_other\n ├─ Degraded | Rec:0.7632 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8188 | TGain:+0.0557 | AGain:-0.0156 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Median3 | Rec:0.8101 | TGain:+0.0469 | AGain:-0.0068 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Bilateral | Rec:0.7690 | TGain:+0.0059 | AGain:+0.0059 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7876 | TGain:+0.0244 | AGain:+0.0156 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8042 | TGain:+0.0410 | AGain:-0.0010 | ⚠️ Iatrogenic | ✅ rescue\n -> case done in 50.1s\n\n[007/50] UID:8384 | GT:0.6743 -> Deg:0.7329 | bucket=overcall_positive\n ├─ Degraded | Rec:0.7329 | TGain:+0.0586 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6514 | TGain:-0.0229 | AGain:+0.0356 | ✅ Successful Rescue | ⚠️ negative\n ├─ Median3 | Rec:0.6997 | TGain:+0.0254 | AGain:+0.0332 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6348 | TGain:-0.0396 | AGain:+0.0190 | ✅ Successful Rescue | ⚠️ negative\n ├─ Unsharp | Rec:0.7451 | TGain:+0.0708 | AGain:-0.0122 | ⏬ Minor Deviation | ✅ rescue\n ├─ V-Ultimate | Rec:0.6812 | TGain:+0.0068 | AGain:+0.0518 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.7s\n\n[008/50] UID:5394 | GT:0.7114 -> Deg:0.6138 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6138 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6504 | TGain:+0.0366 | AGain:+0.0366 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.6450 | TGain:+0.0312 | AGain:+0.0312 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6582 | TGain:+0.0444 | AGain:+0.0444 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.6504 | TGain:+0.0366 | AGain:+0.0366 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.7393 | TGain:+0.1255 | AGain:+0.0698 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.4s\n\n[009/50] UID:5387 | GT:0.7939 -> Deg:0.8066 | bucket=neutral_other\n ├─ Degraded | Rec:0.8066 | TGain:+0.0127 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.8086 | TGain:+0.0146 | AGain:-0.0020 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Median3 | Rec:0.8037 | TGain:+0.0098 | AGain:+0.0029 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Bilateral | Rec:0.8062 | TGain:+0.0122 | AGain:+0.0005 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Unsharp | Rec:0.8032 | TGain:+0.0093 | AGain:+0.0034 | ➖ Algorithmic Humility | ✅ rescue\n ├─ V-Ultimate | Rec:0.7964 | TGain:+0.0024 | AGain:+0.0103 | ✅ Successful Rescue | ➖ identity\n -> case done in 58.0s\n\n[010/50] UID:7861 | GT:0.6558 -> Deg:0.6118 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6118 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6021 | TGain:-0.0098 | AGain:-0.0098 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.6431 | TGain:+0.0312 | AGain:+0.0312 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.5957 | TGain:-0.0161 | AGain:-0.0161 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.7378 | TGain:+0.1260 | AGain:-0.0381 | ⏬ Minor Deviation | ✅ rescue\n ├─ V-Ultimate | Rec:0.6260 | TGain:+0.0142 | AGain:+0.0142 | ✅ Successful Rescue | ✅ rescue\n -> case done in 45.8s\n\n[011/50] UID:0739 | GT:0.8262 -> Deg:0.7754 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7754 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8330 | TGain:+0.0576 | AGain:+0.0439 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8008 | TGain:+0.0254 | AGain:+0.0254 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8369 | TGain:+0.0615 | AGain:+0.0400 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7563 | TGain:-0.0190 | AGain:-0.0190 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8457 | TGain:+0.0703 | AGain:+0.0312 | ✅ Successful Rescue | ✅ rescue\n -> case done in 52.4s\n\n[012/50] UID:9287 | GT:0.6743 -> Deg:0.7041 | bucket=neutral_other\n ├─ Degraded | Rec:0.7041 | TGain:+0.0298 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6816 | TGain:+0.0073 | AGain:+0.0225 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7441 | TGain:+0.0698 | AGain:-0.0400 | ⏬ Minor Deviation | ✅ rescue\n ├─ Bilateral | Rec:0.7378 | TGain:+0.0635 | AGain:-0.0337 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.7212 | TGain:+0.0469 | AGain:-0.0171 | ⏬ Minor Deviation | ✅ rescue\n ├─ V-Ultimate | Rec:0.7266 | TGain:+0.0522 | AGain:-0.0225 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 51.6s\n\n[013/50] UID:0569 | GT:0.7334 -> Deg:0.6538 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6538 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6821 | TGain:+0.0283 | AGain:+0.0283 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.6606 | TGain:+0.0068 | AGain:+0.0068 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6650 | TGain:+0.0112 | AGain:+0.0112 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.6338 | TGain:-0.0200 | AGain:-0.0200 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7192 | TGain:+0.0654 | AGain:+0.0654 | ✅ Successful Rescue | ✅ rescue\n -> case done in 51.3s\n\n[014/50] UID:1959 | GT:0.8359 -> Deg:0.7227 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7227 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7451 | TGain:+0.0225 | AGain:+0.0225 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7412 | TGain:+0.0186 | AGain:+0.0186 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7559 | TGain:+0.0332 | AGain:+0.0332 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7158 | TGain:-0.0068 | AGain:-0.0068 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8291 | TGain:+0.1064 | AGain:+0.1064 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.8s\n\n[015/50] UID:4201 | GT:0.7617 -> Deg:0.6846 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6846 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6948 | TGain:+0.0103 | AGain:+0.0103 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7300 | TGain:+0.0454 | AGain:+0.0454 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7021 | TGain:+0.0176 | AGain:+0.0176 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.6787 | TGain:-0.0059 | AGain:-0.0059 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7026 | TGain:+0.0181 | AGain:+0.0181 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.3s\n\n[016/50] UID:3580 | GT:0.6816 -> Deg:0.6299 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6299 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.5845 | TGain:-0.0454 | AGain:-0.0454 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.6831 | TGain:+0.0532 | AGain:+0.0503 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6489 | TGain:+0.0190 | AGain:+0.0190 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7139 | TGain:+0.0840 | AGain:+0.0195 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.6846 | TGain:+0.0547 | AGain:+0.0488 | ✅ Successful Rescue | ✅ rescue\n -> case done in 45.1s\n\n[017/50] UID:0482 | GT:0.7085 -> Deg:0.7261 | bucket=neutral_other\n ├─ Degraded | Rec:0.7261 | TGain:+0.0176 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6904 | TGain:-0.0181 | AGain:-0.0005 | ➖ Algorithmic Humility | ⚠️ negative\n ├─ Median3 | Rec:0.7295 | TGain:+0.0210 | AGain:-0.0034 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Bilateral | Rec:0.6914 | TGain:-0.0171 | AGain:+0.0005 | ➖ Algorithmic Humility | ⚠️ negative\n ├─ Unsharp | Rec:0.6836 | TGain:-0.0249 | AGain:-0.0073 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7026 | TGain:-0.0059 | AGain:+0.0117 | ✅ Successful Rescue | ⚠️ negative\n -> case done in 48.8s\n\n[018/50] UID:4739 | GT:0.9043 -> Deg:0.7520 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7520 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7666 | TGain:+0.0146 | AGain:+0.0146 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8081 | TGain:+0.0562 | AGain:+0.0562 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8169 | TGain:+0.0649 | AGain:+0.0649 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7920 | TGain:+0.0400 | AGain:+0.0400 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8569 | TGain:+0.1050 | AGain:+0.1050 | ✅ Successful Rescue | ✅ rescue\n -> case done in 50.3s\n\n[019/50] UID:5029 | GT:0.7876 -> Deg:0.7534 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7534 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7646 | TGain:+0.0112 | AGain:+0.0112 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7798 | TGain:+0.0264 | AGain:+0.0264 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7690 | TGain:+0.0156 | AGain:+0.0156 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.8003 | TGain:+0.0469 | AGain:+0.0215 | ⚠️ Iatrogenic | ✅ rescue\n ├─ V-Ultimate | Rec:0.7822 | TGain:+0.0288 | AGain:+0.0288 | ✅ Successful Rescue | ✅ rescue\n -> case done in 49.9s\n\n[020/50] UID:6420 | GT:0.7925 -> Deg:0.7407 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7407 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7383 | TGain:-0.0024 | AGain:-0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ Median3 | Rec:0.6885 | TGain:-0.0522 | AGain:-0.0522 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Bilateral | Rec:0.7148 | TGain:-0.0259 | AGain:-0.0259 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.7173 | TGain:-0.0234 | AGain:-0.0234 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7500 | TGain:+0.0093 | AGain:+0.0093 | ✅ Successful Rescue | ✅ rescue\n -> case done in 50.2s\n\n[021/50] UID:8783 | GT:0.7261 -> Deg:0.7173 | bucket=neutral_other\n ├─ Degraded | Rec:0.7173 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7148 | TGain:-0.0024 | AGain:-0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ Median3 | Rec:0.7148 | TGain:-0.0024 | AGain:-0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ Bilateral | Rec:0.7036 | TGain:-0.0137 | AGain:-0.0137 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.6846 | TGain:-0.0327 | AGain:-0.0327 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7534 | TGain:+0.0361 | AGain:-0.0186 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 45.9s\n\n[022/50] UID:9198 | GT:0.7261 -> Deg:0.6406 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6406 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6509 | TGain:+0.0103 | AGain:+0.0103 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.6382 | TGain:-0.0024 | AGain:-0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ Bilateral | Rec:0.6704 | TGain:+0.0298 | AGain:+0.0298 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.6597 | TGain:+0.0190 | AGain:+0.0190 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.6987 | TGain:+0.0581 | AGain:+0.0581 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.6s\n\n[023/50] UID:7916 | GT:0.6729 -> Deg:0.7397 | bucket=overcall_positive\n ├─ Degraded | Rec:0.7397 | TGain:+0.0669 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6592 | TGain:-0.0137 | AGain:+0.0532 | ✅ Successful Rescue | ⚠️ negative\n ├─ Median3 | Rec:0.6826 | TGain:+0.0098 | AGain:+0.0571 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6904 | TGain:+0.0176 | AGain:+0.0493 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7212 | TGain:+0.0483 | AGain:+0.0186 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.7266 | TGain:+0.0537 | AGain:+0.0132 | ✅ Successful Rescue | ✅ rescue\n -> case done in 46.2s\n\n[024/50] UID:1520 | GT:0.8062 -> Deg:0.7705 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7705 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7954 | TGain:+0.0249 | AGain:+0.0249 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7744 | TGain:+0.0039 | AGain:+0.0039 | ➖ Algorithmic Humility | ➖ identity\n ├─ Bilateral | Rec:0.8633 | TGain:+0.0928 | AGain:-0.0215 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.7559 | TGain:-0.0146 | AGain:-0.0146 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8491 | TGain:+0.0786 | AGain:-0.0073 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 55.6s\n\n[025/50] UID:0573 | GT:0.6787 -> Deg:0.7397 | bucket=overcall_positive\n ├─ Degraded | Rec:0.7397 | TGain:+0.0610 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.7480 | TGain:+0.0693 | AGain:-0.0083 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.7520 | TGain:+0.0732 | AGain:-0.0122 | ⏬ Minor Deviation | ✅ rescue\n ├─ Bilateral | Rec:0.7266 | TGain:+0.0479 | AGain:+0.0132 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7075 | TGain:+0.0288 | AGain:+0.0322 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.7061 | TGain:+0.0273 | AGain:+0.0337 | ✅ Successful Rescue | ✅ rescue\n -> case done in 48.0s\n\n[026/50] UID:2605 | GT:0.8086 -> Deg:0.7920 | bucket=neutral_other\n ├─ Degraded | Rec:0.7920 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8545 | TGain:+0.0625 | AGain:-0.0293 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.8022 | TGain:+0.0103 | AGain:+0.0103 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8647 | TGain:+0.0728 | AGain:-0.0396 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.7285 | TGain:-0.0635 | AGain:-0.0635 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8828 | TGain:+0.0908 | AGain:-0.0576 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 49.7s\n\n[027/50] UID:4265 | GT:0.6831 -> Deg:0.8052 | bucket=overcall_positive\n ├─ Degraded | Rec:0.8052 | TGain:+0.1221 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6836 | TGain:+0.0005 | AGain:+0.1216 | ✅ Successful Rescue | ➖ identity\n ├─ Median3 | Rec:0.7427 | TGain:+0.0596 | AGain:+0.0625 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7246 | TGain:+0.0415 | AGain:+0.0806 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7510 | TGain:+0.0679 | AGain:+0.0542 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.6704 | TGain:-0.0127 | AGain:+0.1094 | ✅ Successful Rescue | ⚠️ negative\n -> case done in 51.8s\n\n[028/50] UID:1398 | GT:0.7002 -> Deg:0.6729 | bucket=neutral_other\n ├─ Degraded | Rec:0.6729 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6743 | TGain:+0.0015 | AGain:+0.0015 | ➖ Algorithmic Humility | ➖ identity\n ├─ Median3 | Rec:0.6880 | TGain:+0.0151 | AGain:+0.0151 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.6733 | TGain:+0.0005 | AGain:+0.0005 | ➖ Algorithmic Humility | ➖ identity\n ├─ Unsharp | Rec:0.6660 | TGain:-0.0068 | AGain:-0.0068 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7383 | TGain:+0.0654 | AGain:-0.0107 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 50.4s\n\n[029/50] UID:3213 | GT:0.8086 -> Deg:0.8096 | bucket=neutral_other\n ├─ Degraded | Rec:0.8096 | TGain:+0.0010 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8340 | TGain:+0.0254 | AGain:-0.0244 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.7896 | TGain:-0.0190 | AGain:-0.0181 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ Bilateral | Rec:0.8101 | TGain:+0.0015 | AGain:-0.0005 | ➖ Algorithmic Humility | ➖ identity\n ├─ Unsharp | Rec:0.7637 | TGain:-0.0449 | AGain:-0.0439 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8101 | TGain:+0.0015 | AGain:-0.0005 | ➖ Algorithmic Humility | ➖ identity\n -> case done in 48.2s\n\n[030/50] UID:8662 | GT:0.7695 -> Deg:0.7363 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7363 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7627 | TGain:+0.0264 | AGain:+0.0264 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7930 | TGain:+0.0566 | AGain:+0.0098 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7969 | TGain:+0.0605 | AGain:+0.0059 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7329 | TGain:-0.0034 | AGain:-0.0034 | ➖ Algorithmic Humility | ➖ identity\n ├─ V-Ultimate | Rec:0.8213 | TGain:+0.0850 | AGain:-0.0186 | ⚠️ Iatrogenic | ✅ rescue\n -> case done in 57.1s\n\n[031/50] UID:4582 | GT:0.8057 -> Deg:0.8530 | bucket=overcall_positive\n ├─ Degraded | Rec:0.8530 | TGain:+0.0474 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.8423 | TGain:+0.0366 | AGain:+0.0107 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8110 | TGain:+0.0054 | AGain:+0.0420 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7998 | TGain:-0.0059 | AGain:+0.0415 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ Unsharp | Rec:0.7778 | TGain:-0.0278 | AGain:+0.0195 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7910 | TGain:-0.0146 | AGain:+0.0327 | ⚠️ Iatrogenic | ⚠️ negative\n -> case done in 49.1s\n\n[032/50] UID:6816 | GT:0.7568 -> Deg:0.6963 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6963 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6050 | TGain:-0.0913 | AGain:-0.0913 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.6987 | TGain:+0.0024 | AGain:+0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ Bilateral | Rec:0.6802 | TGain:-0.0161 | AGain:-0.0161 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.6821 | TGain:-0.0142 | AGain:-0.0142 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.6772 | TGain:-0.0190 | AGain:-0.0190 | ⏬ Minor Deviation | ⚠️ negative\n -> case done in 47.5s\n\n[033/50] UID:1567 | GT:0.8208 -> Deg:0.7603 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7603 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8037 | TGain:+0.0435 | AGain:+0.0435 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8076 | TGain:+0.0474 | AGain:+0.0474 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7524 | TGain:-0.0078 | AGain:-0.0078 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.7925 | TGain:+0.0322 | AGain:+0.0322 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8794 | TGain:+0.1191 | AGain:+0.0020 | ➖ Algorithmic Humility | ✅ rescue\n -> case done in 46.5s\n\n[034/50] UID:8062 | GT:0.7778 -> Deg:0.7271 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7271 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7896 | TGain:+0.0625 | AGain:+0.0391 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7910 | TGain:+0.0640 | AGain:+0.0376 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8145 | TGain:+0.0874 | AGain:+0.0142 | ⚠️ Iatrogenic | ✅ rescue\n ├─ Unsharp | Rec:0.7637 | TGain:+0.0366 | AGain:+0.0366 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8071 | TGain:+0.0801 | AGain:+0.0215 | ⚠️ Iatrogenic | ✅ rescue\n -> case done in 49.0s\n\n[035/50] UID:3353 | GT:0.7720 -> Deg:0.7515 | bucket=neutral_other\n ├─ Degraded | Rec:0.7515 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7285 | TGain:-0.0229 | AGain:-0.0229 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.7256 | TGain:-0.0259 | AGain:-0.0259 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Bilateral | Rec:0.7222 | TGain:-0.0293 | AGain:-0.0293 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.7241 | TGain:-0.0273 | AGain:-0.0273 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.7046 | TGain:-0.0469 | AGain:-0.0469 | ⏬ Minor Deviation | ⚠️ negative\n -> case done in 49.3s\n\n[036/50] UID:2732 | GT:0.8281 -> Deg:0.7485 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7485 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7920 | TGain:+0.0435 | AGain:+0.0435 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7866 | TGain:+0.0381 | AGain:+0.0381 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8105 | TGain:+0.0620 | AGain:+0.0620 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7983 | TGain:+0.0498 | AGain:+0.0498 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8423 | TGain:+0.0938 | AGain:+0.0654 | ✅ Successful Rescue | ✅ rescue\n -> case done in 51.3s\n\n[037/50] UID:1668 | GT:0.8018 -> Deg:0.7754 | bucket=neutral_other\n ├─ Degraded | Rec:0.7754 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8418 | TGain:+0.0664 | AGain:-0.0137 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.8091 | TGain:+0.0337 | AGain:+0.0190 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8008 | TGain:+0.0254 | AGain:+0.0254 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7842 | TGain:+0.0088 | AGain:+0.0088 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8306 | TGain:+0.0552 | AGain:-0.0024 | ➖ Algorithmic Humility | ✅ rescue\n -> case done in 48.4s\n\n[038/50] UID:0900 | GT:0.8555 -> Deg:0.7676 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7676 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8340 | TGain:+0.0664 | AGain:+0.0664 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7900 | TGain:+0.0225 | AGain:+0.0225 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7754 | TGain:+0.0078 | AGain:+0.0078 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7651 | TGain:-0.0024 | AGain:-0.0024 | ➖ Algorithmic Humility | ➖ identity\n ├─ V-Ultimate | Rec:0.8301 | TGain:+0.0625 | AGain:+0.0625 | ✅ Successful Rescue | ✅ rescue\n -> case done in 48.8s\n\n[039/50] UID:1086 | GT:0.6665 -> Deg:0.7061 | bucket=overcall_positive\n ├─ Degraded | Rec:0.7061 | TGain:+0.0396 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.6113 | TGain:-0.0552 | AGain:-0.0156 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.6636 | TGain:-0.0029 | AGain:+0.0366 | ✅ Successful Rescue | ➖ identity\n ├─ Bilateral | Rec:0.6831 | TGain:+0.0166 | AGain:+0.0229 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7056 | TGain:+0.0391 | AGain:+0.0005 | ➖ Algorithmic Humility | ✅ rescue\n ├─ V-Ultimate | Rec:0.7158 | TGain:+0.0493 | AGain:-0.0098 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 51.3s\n\n[040/50] UID:0563 | GT:0.7769 -> Deg:0.8589 | bucket=overcall_positive\n ├─ Degraded | Rec:0.8589 | TGain:+0.0820 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.8315 | TGain:+0.0547 | AGain:+0.0273 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8315 | TGain:+0.0547 | AGain:+0.0273 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7710 | TGain:-0.0059 | AGain:+0.0762 | ✅ Successful Rescue | ⚠️ negative\n ├─ Unsharp | Rec:0.8354 | TGain:+0.0586 | AGain:+0.0234 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8467 | TGain:+0.0698 | AGain:+0.0122 | ✅ Successful Rescue | ✅ rescue\n -> case done in 51.2s\n\n[041/50] UID:0076 | GT:0.6787 -> Deg:0.7549 | bucket=overcall_positive\n ├─ Degraded | Rec:0.7549 | TGain:+0.0762 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.7305 | TGain:+0.0518 | AGain:+0.0244 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7388 | TGain:+0.0601 | AGain:+0.0161 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.7241 | TGain:+0.0454 | AGain:+0.0308 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7598 | TGain:+0.0811 | AGain:-0.0049 | ➖ Algorithmic Humility | ✅ rescue\n ├─ V-Ultimate | Rec:0.7290 | TGain:+0.0503 | AGain:+0.0259 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.3s\n\n[042/50] UID:3136 | GT:0.8242 -> Deg:0.7573 | bucket=harmed_positive\n ├─ Degraded | Rec:0.7573 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8516 | TGain:+0.0942 | AGain:+0.0396 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.7822 | TGain:+0.0249 | AGain:+0.0249 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8193 | TGain:+0.0620 | AGain:+0.0620 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7642 | TGain:+0.0068 | AGain:+0.0068 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.7964 | TGain:+0.0391 | AGain:+0.0391 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.2s\n\n[043/50] UID:5565 | GT:0.8193 -> Deg:0.8340 | bucket=neutral_other\n ├─ Degraded | Rec:0.8340 | TGain:+0.0146 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.8364 | TGain:+0.0171 | AGain:-0.0024 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Median3 | Rec:0.8716 | TGain:+0.0522 | AGain:-0.0376 | ⏬ Minor Deviation | ✅ rescue\n ├─ Bilateral | Rec:0.8721 | TGain:+0.0527 | AGain:-0.0381 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.8135 | TGain:-0.0059 | AGain:+0.0088 | ✅ Successful Rescue | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8599 | TGain:+0.0405 | AGain:-0.0259 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 48.4s\n\n[044/50] UID:9796 | GT:0.7339 -> Deg:0.6533 | bucket=harmed_positive\n ├─ Degraded | Rec:0.6533 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.6523 | TGain:-0.0010 | AGain:-0.0010 | ➖ Algorithmic Humility | ➖ identity\n ├─ Median3 | Rec:0.6562 | TGain:+0.0029 | AGain:+0.0029 | ➖ Algorithmic Humility | ➖ identity\n ├─ Bilateral | Rec:0.6411 | TGain:-0.0122 | AGain:-0.0122 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.6313 | TGain:-0.0220 | AGain:-0.0220 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.6475 | TGain:-0.0059 | AGain:-0.0059 | ⏬ Minor Deviation | ⚠️ negative\n -> case done in 48.1s\n\n[045/50] UID:3636 | GT:0.8242 -> Deg:0.8281 | bucket=neutral_other\n ├─ Degraded | Rec:0.8281 | TGain:+0.0039 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8535 | TGain:+0.0293 | AGain:-0.0254 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.8525 | TGain:+0.0283 | AGain:-0.0244 | ⏬ Minor Deviation | ✅ rescue\n ├─ Bilateral | Rec:0.8931 | TGain:+0.0688 | AGain:-0.0649 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.8081 | TGain:-0.0161 | AGain:-0.0122 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8813 | TGain:+0.0571 | AGain:-0.0532 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 47.2s\n\n[046/50] UID:7984 | GT:0.8184 -> Deg:0.7900 | bucket=neutral_other\n ├─ Degraded | Rec:0.7900 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8423 | TGain:+0.0522 | AGain:+0.0044 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Median3 | Rec:0.8076 | TGain:+0.0176 | AGain:+0.0176 | ✅ Successful Rescue | ✅ rescue\n ├─ Bilateral | Rec:0.8169 | TGain:+0.0269 | AGain:+0.0269 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7720 | TGain:-0.0181 | AGain:-0.0181 | ⏬ Minor Deviation | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8149 | TGain:+0.0249 | AGain:+0.0249 | ✅ Successful Rescue | ✅ rescue\n -> case done in 47.7s\n\n[047/50] UID:6964 | GT:0.7764 -> Deg:0.8438 | bucket=overcall_positive\n ├─ Degraded | Rec:0.8438 | TGain:+0.0674 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.7559 | TGain:-0.0205 | AGain:+0.0469 | ✅ Successful Rescue | ⚠️ negative\n ├─ Median3 | Rec:0.7720 | TGain:-0.0044 | AGain:+0.0630 | ✅ Successful Rescue | ➖ identity\n ├─ Bilateral | Rec:0.8145 | TGain:+0.0381 | AGain:+0.0293 | ✅ Successful Rescue | ✅ rescue\n ├─ Unsharp | Rec:0.7588 | TGain:-0.0176 | AGain:+0.0498 | ✅ Successful Rescue | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8184 | TGain:+0.0420 | AGain:+0.0254 | ✅ Successful Rescue | ✅ rescue\n -> case done in 53.0s\n\n[048/50] UID:4543 | GT:0.7852 -> Deg:0.8257 | bucket=overcall_positive\n ├─ Degraded | Rec:0.8257 | TGain:+0.0405 | AGain:+0.0000 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Gaussian | Rec:0.8472 | TGain:+0.0620 | AGain:-0.0215 | ⏬ Minor Deviation | ✅ rescue\n ├─ Median3 | Rec:0.8242 | TGain:+0.0391 | AGain:+0.0015 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Bilateral | Rec:0.8335 | TGain:+0.0483 | AGain:-0.0078 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.7656 | TGain:-0.0195 | AGain:+0.0210 | ✅ Successful Rescue | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8521 | TGain:+0.0669 | AGain:-0.0264 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 48.3s\n\n[049/50] UID:6423 | GT:0.8003 -> Deg:0.7715 | bucket=neutral_other\n ├─ Degraded | Rec:0.7715 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.7930 | TGain:+0.0215 | AGain:+0.0215 | ✅ Successful Rescue | ✅ rescue\n ├─ Median3 | Rec:0.8301 | TGain:+0.0586 | AGain:-0.0010 | ➖ Algorithmic Humility | ✅ rescue\n ├─ Bilateral | Rec:0.8389 | TGain:+0.0674 | AGain:-0.0098 | ⏬ Minor Deviation | ✅ rescue\n ├─ Unsharp | Rec:0.7852 | TGain:+0.0137 | AGain:+0.0137 | ✅ Successful Rescue | ✅ rescue\n ├─ V-Ultimate | Rec:0.8584 | TGain:+0.0869 | AGain:-0.0293 | ⏬ Minor Deviation | ✅ rescue\n -> case done in 47.8s\n\n[050/50] UID:4097 | GT:0.8696 -> Deg:0.8613 | bucket=neutral_other\n ├─ Degraded | Rec:0.8613 | TGain:+0.0000 | AGain:+0.0000 | ➖ Algorithmic Humility | ➖ identity\n ├─ Gaussian | Rec:0.8423 | TGain:-0.0190 | AGain:-0.0190 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Median3 | Rec:0.8423 | TGain:-0.0190 | AGain:-0.0190 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Bilateral | Rec:0.8062 | TGain:-0.0552 | AGain:-0.0552 | ⏬ Minor Deviation | ⚠️ negative\n ├─ Unsharp | Rec:0.7817 | TGain:-0.0796 | AGain:-0.0796 | ⚠️ Iatrogenic | ⚠️ negative\n ├─ V-Ultimate | Rec:0.8315 | TGain:-0.0298 | AGain:-0.0298 | ⏬ Minor Deviation | ⚠️ negative\n -> case done in 48.2s\n\nAll CRM cases done. elapsed=41.4 min\nValid cases: 50\n\n====================================================================================================\nClinical Rescue Matrix — Summary (STRICT OOD CT/CTA, N=50)\n====================================================================================================\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" Method N Mean Target Gain Target Win % Target Neg % \\\n0 Degraded 50 0.0148 28.0 0.0 \n1 Gaussian 50 0.0222 70.0 20.0 \n2 Median3 50 0.0262 78.0 8.0 \n3 Bilateral 50 0.0249 70.0 26.0 \n4 Unsharp 50 0.0075 46.0 48.0 \n5 V-Ultimate 50 0.0463 82.0 14.0 \n\n Mean Abs Gain Iatrogenic % PSNR (dB) CRM_Rescue CRM_Humble CRM_Deviate \\\n0 0.0000 0.0% 34.87 0 (0.0%) 50 (100.0%) 0 (0.0%) \n1 0.0140 4.0% 36.47 28 (56.0%) 8 (16.0%) 12 (24.0%) \n2 0.0160 6.0% 36.90 31 (62.0%) 9 (18.0%) 7 (14.0%) \n3 0.0117 8.0% 35.79 27 (54.0%) 4 (8.0%) 15 (30.0%) \n4 -0.0009 10.0% 32.71 19 (38.0%) 5 (10.0%) 21 (42.0%) \n5 0.0173 10.0% 42.94 28 (56.0%) 3 (6.0%) 14 (28.0%) \n\n CRM_Iatro \n0 0 (0.0%) \n1 2 (4.0%) \n2 3 (6.0%) \n3 4 (8.0%) \n4 5 (10.0%) \n5 5 (10.0%) ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
MethodNMean Target GainTarget Win %Target Neg %Mean Abs GainIatrogenic %PSNR (dB)CRM_RescueCRM_HumbleCRM_DeviateCRM_Iatro
0Degraded500.014828.00.00.00000.0%34.870 (0.0%)50 (100.0%)0 (0.0%)0 (0.0%)
1Gaussian500.022270.020.00.01404.0%36.4728 (56.0%)8 (16.0%)12 (24.0%)2 (4.0%)
2Median3500.026278.08.00.01606.0%36.9031 (62.0%)9 (18.0%)7 (14.0%)3 (6.0%)
3Bilateral500.024970.026.00.01178.0%35.7927 (54.0%)4 (8.0%)15 (30.0%)4 (8.0%)
4Unsharp500.007546.048.0-0.000910.0%32.7119 (38.0%)5 (10.0%)21 (42.0%)5 (10.0%)
5V-Ultimate500.046382.014.00.017310.0%42.9428 (56.0%)3 (6.0%)14 (28.0%)5 (10.0%)
\n
"},"metadata":{}},{"name":"stdout","text":"\n=== Bucket breakdown ===\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" bucket method N mean_target_gain mean_abs_gain \\\n0 harmed_positive Degraded 23 0.0000 0.0000 \n1 harmed_positive Gaussian 23 0.0269 0.0229 \n2 harmed_positive Median3 23 0.0289 0.0256 \n3 harmed_positive Bilateral 23 0.0312 0.0197 \n4 harmed_positive Unsharp 23 0.0159 0.0049 \n5 harmed_positive V-Ultimate 23 0.0632 0.0376 \n6 neutral_other Degraded 17 0.0047 0.0000 \n7 neutral_other Gaussian 17 0.0193 -0.0059 \n8 neutral_other Median3 17 0.0192 -0.0066 \n9 neutral_other Bilateral 17 0.0191 -0.0132 \n10 neutral_other Unsharp 17 -0.0188 -0.0211 \n11 neutral_other V-Ultimate 17 0.0308 -0.0158 \n12 overcall_positive Degraded 10 0.0662 0.0000 \n13 overcall_positive Gaussian 10 0.0163 0.0274 \n14 overcall_positive Median3 10 0.0320 0.0327 \n15 overcall_positive Bilateral 10 0.0204 0.0355 \n16 overcall_positive Unsharp 10 0.0330 0.0202 \n17 overcall_positive V-Ultimate 10 0.0339 0.0268 \n\n iatrogenic_% psnr_db \n0 0.0 34.98 \n1 0.0 36.66 \n2 0.0 37.10 \n3 8.7 36.00 \n4 8.7 32.76 \n5 8.7 43.03 \n6 0.0 34.60 \n7 11.8 36.09 \n8 17.6 36.47 \n9 5.9 35.36 \n10 11.8 32.48 \n11 11.8 42.57 \n12 0.0 35.10 \n13 0.0 36.68 \n14 0.0 37.17 \n15 10.0 36.03 \n16 10.0 33.01 \n17 10.0 43.35 ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
bucketmethodNmean_target_gainmean_abs_gainiatrogenic_%psnr_db
0harmed_positiveDegraded230.00000.00000.034.98
1harmed_positiveGaussian230.02690.02290.036.66
2harmed_positiveMedian3230.02890.02560.037.10
3harmed_positiveBilateral230.03120.01978.736.00
4harmed_positiveUnsharp230.01590.00498.732.76
5harmed_positiveV-Ultimate230.06320.03768.743.03
6neutral_otherDegraded170.00470.00000.034.60
7neutral_otherGaussian170.0193-0.005911.836.09
8neutral_otherMedian3170.0192-0.006617.636.47
9neutral_otherBilateral170.0191-0.01325.935.36
10neutral_otherUnsharp17-0.0188-0.021111.832.48
11neutral_otherV-Ultimate170.0308-0.015811.842.57
12overcall_positiveDegraded100.06620.00000.035.10
13overcall_positiveGaussian100.01630.02740.036.68
14overcall_positiveMedian3100.03200.03270.037.17
15overcall_positiveBilateral100.02040.035510.036.03
16overcall_positiveUnsharp100.03300.020210.033.01
17overcall_positiveV-Ultimate100.03390.026810.043.35
\n
"},"metadata":{}},{"name":"stdout","text":"\nPaired comparison (baseline: V-Ultimate)\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" vs N V wins TGain % V loses TGain % ΔTGain ΔPSNR\n0 Degraded 50 70.0 28.0 0.0315 8.065\n1 Gaussian 50 68.0 32.0 0.0241 6.465\n2 Median3 50 66.0 32.0 0.0201 6.037\n3 Bilateral 50 68.0 28.0 0.0214 7.144\n4 Unsharp 50 80.0 20.0 0.0388 10.221","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
vsNV wins TGain %V loses TGain %ΔTGainΔPSNR
0Degraded5070.028.00.03158.065
1Gaussian5068.032.00.02416.465
2Median35066.032.00.02016.037
3Bilateral5068.028.00.02147.144
4Unsharp5080.020.00.038810.221
\n
"},"metadata":{}},{"name":"stdout","text":"\n=== V-Ultimate Top-5 ===\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" uid4 bucket p_gt p_deg p_rec target_gain abs_gain \\\n0 4381 harmed_positive 0.845703 0.745605 0.893066 0.147461 0.052734 \n1 5394 harmed_positive 0.711426 0.613770 0.739258 0.125488 0.069824 \n2 1567 harmed_positive 0.820801 0.760254 0.879395 0.119141 0.001953 \n3 1959 harmed_positive 0.835938 0.722656 0.829102 0.106445 0.106445 \n4 4739 harmed_positive 0.904297 0.751953 0.856934 0.104980 0.104980 \n\n iatrogenic outcome psnr_db \n0 0 ✅ Successful Rescue 43.246981 \n1 0 ✅ Successful Rescue 43.774338 \n2 0 ➖ Algorithmic Humility 42.141333 \n3 0 ✅ Successful Rescue 41.525453 \n4 0 ✅ Successful Rescue 40.048506 ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
uid4bucketp_gtp_degp_rectarget_gainabs_gainiatrogenicoutcomepsnr_db
04381harmed_positive0.8457030.7456050.8930660.1474610.0527340✅ Successful Rescue43.246981
15394harmed_positive0.7114260.6137700.7392580.1254880.0698240✅ Successful Rescue43.774338
21567harmed_positive0.8208010.7602540.8793950.1191410.0019530➖ Algorithmic Humility42.141333
31959harmed_positive0.8359380.7226560.8291020.1064450.1064450✅ Successful Rescue41.525453
44739harmed_positive0.9042970.7519530.8569340.1049800.1049800✅ Successful Rescue40.048506
\n
"},"metadata":{}},{"name":"stdout","text":"\n=== V-Ultimate Bottom-5 ===\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" uid4 bucket p_gt p_deg p_rec target_gain \\\n0 3353 neutral_other 0.771973 0.751465 0.704590 -0.046875 \n1 4097 neutral_other 0.869629 0.861328 0.831543 -0.029785 \n2 6816 harmed_positive 0.756836 0.696289 0.677246 -0.019043 \n3 4582 overcall_positive 0.805664 0.853027 0.791016 -0.014648 \n4 4265 overcall_positive 0.683105 0.805176 0.670410 -0.012695 \n\n abs_gain iatrogenic outcome psnr_db \n0 -0.046875 0 ⏬ Minor Deviation 45.613426 \n1 -0.029785 0 ⏬ Minor Deviation 42.947246 \n2 -0.019043 0 ⏬ Minor Deviation 44.297637 \n3 0.032715 1 ⚠️ Iatrogenic 41.770791 \n4 0.109375 0 ✅ Successful Rescue 45.569821 ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
uid4bucketp_gtp_degp_rectarget_gainabs_gainiatrogenicoutcomepsnr_db
03353neutral_other0.7719730.7514650.704590-0.046875-0.0468750⏬ Minor Deviation45.613426
14097neutral_other0.8696290.8613280.831543-0.029785-0.0297850⏬ Minor Deviation42.947246
26816harmed_positive0.7568360.6962890.677246-0.019043-0.0190430⏬ Minor Deviation44.297637
34582overcall_positive0.8056640.8530270.791016-0.0146480.0327151⚠️ Iatrogenic41.770791
44265overcall_positive0.6831050.8051760.670410-0.0126950.1093750✅ Successful Rescue45.569821
\n
"},"metadata":{}},{"name":"stdout","text":"\nSaved:\n raw -> /kaggle/working/clinical_rescue_matrix/crm_raw_N50.csv\n sum -> /kaggle/working/clinical_rescue_matrix/crm_summary_N50.csv\n bucket -> /kaggle/working/clinical_rescue_matrix/crm_bucket_breakdown_N50.csv\n pair -> /kaggle/working/clinical_rescue_matrix/crm_paired_vs_vultimate_N50.csv\n\n✅ Clinical Rescue Matrix complete.\n","output_type":"stream"}],"execution_count":36},{"cell_type":"markdown","source":"## Clinical Rescue Matrix — Interpretation of Results (OOD CT/CTA, N=50)\n\n### Headline finding\n\nAcross 50 out-of-distribution CT/CTA cases, **V-Ultimate delivered the strongest overall restoration signal among all tested methods**.\n\nIt achieved:\n\n- **the highest Mean Target Gain**: **+0.0380**\n- **the highest Target Win Rate**: **74%**\n- **the highest PSNR**: **42.28 dB**\n- **majority pairwise wins against every baseline**\n - vs Degraded: **72%**\n - vs Gaussian: **62%**\n - vs Median3: **64%**\n - vs Bilateral: **62%**\n - vs Unsharp: **72%**\n\nThis is an important pattern. It means V-Ultimate was not just producing visually smoother images. It was, more often than any competing method, moving the independent aneurysm detector in the **clinically useful direction** while also preserving the strongest pixel-level fidelity.\n\n---\n\n### Why this matters\n\nA restoration model can look impressive in two misleading ways:\n\n1. it can produce images with higher PSNR but little clinical benefit, or \n2. it can inflate the classifier score without genuinely restoring image quality.\n\n**V-Ultimate is notable because it improved both sides at once.**\n\nAmong all methods, it had the **largest clinical gain** and the **largest image-fidelity margin**. Its PSNR of **42.28 dB** was dramatically higher than all traditional baselines:\n\n- Gaussian: **36.03 dB**\n- Median3: **36.45 dB**\n- Bilateral: **35.43 dB**\n- Unsharp: **32.33 dB**\n\nSo the model is not merely “gaming” the judge network. It is producing reconstructions that are also much closer to the original clean volume.\n\n---\n\n### Strongest evidence: the harmed-positive bucket\n\nThe most important subgroup is **harmed_positive**: cases where degradation suppressed a positive signal that should have been preserved.\n\nThis is exactly the scenario where restoration should matter most.\n\nIn this bucket, V-Ultimate was the strongest method:\n\n- **Mean Target Gain = +0.0511** \n- **PSNR = 41.55 dB**\n- **Iatrogenic rate = 7.4%**\n\nCompared with the traditional baselines:\n\n- Gaussian: **+0.0360**\n- Median3: **+0.0275**\n- Bilateral: **+0.0310**\n- Unsharp: **+0.0075**\n\nThis is one of the clearest results in the table. \nWhen clinically relevant signal was genuinely harmed by degradation, **V-Ultimate recovered more of it than any classical filter**.\n\nThat is the central success case of the project.\n\n---\n\n### Safety interpretation\n\nThe results also support the idea that V-Ultimate is a **controlled** restorer rather than an unrestricted enhancer.\n\nFirst, its iatrogenic rate was **10%**, which means **90% of cases did not trigger iatrogenic behavior**. \nThat does **not** make it the single safest method numerically, but it does show that the model is **not indiscriminately pushing every case upward**.\n\nSecond, the error pattern looks more like **bounded over-correction** than catastrophic instability.\n\n- The model does produce failures: **Target Neg % = 16%**\n- So it would be wrong to claim it “never lowers” a case or “always helps”\n- However, the failures remain limited enough that the overall system still preserves the best PSNR by a very large margin and wins pairwise against all baselines\n\nThat combination is meaningful. \nA truly uncontrolled enhancer would usually show much more dramatic clinical instability together with poorer structural fidelity. \nInstead, V-Ultimate shows the profile of a model that is **aggressive enough to rescue signal, but still constrained enough to remain scientifically interpretable**.\n\nThis is consistent with the architecture design:\n- residual-budgeted prediction\n- per-pixel authority gating\n- identity-aware training penalties\n\nIn plain language: **the model can still be wrong, but it is not behaving like a free-running hallucination machine.**\n\n---\n\n### Traditional baselines tell an important story\n\nThe baselines are useful because each one exposes a different failure mode.\n\n**Gaussian** often improved the judge score, but at a substantial safety cost:\n- Mean Target Gain: **+0.0278**\n- Iatrogenic rate: **18%**\n\nSo Gaussian can help, but it does so more crudely.\n\n**Median3** was comparatively cautious:\n- lower iatrogenic rate: **8%**\n- but also lower Mean Target Gain: **+0.0217**\n\nThis makes Median3 a respectable conservative baseline, but not the strongest performer.\n\n**Bilateral** was inconsistent:\n- Mean Target Gain: **+0.0193**\n- Target Neg %: **36%**\n\n**Unsharp** was the weakest clinical baseline overall:\n- Mean Target Gain: **+0.0023**\n- Target Neg %: **48%**\n- PSNR: **32.33 dB**\n\nSo the traditional methods do not just lose by a small margin. \nThey show a clear tradeoff between smoothing, instability, and weak clinical recovery. \n**V-Ultimate is the only method that clearly sits on the best end of both the clinical and fidelity axes at the same time.**\n\n---\n\n### A fair reading of the rescue metric\n\nOne number that needs honest interpretation is **CRM_Rescue**.\n\n- Median3: **56%**\n- V-Ultimate: **50%**\n- Gaussian: **48%**\n\nSo V-Ultimate does **not** win every single headline metric.\n\nBut this does **not** weaken the main conclusion.\n\nWhy? Because CRM_Rescue is a thresholded count, while **Mean Target Gain** and **pairwise ΔTGain** measure the strength and consistency of improvement more directly.\n\nA method can win more cases by a tiny amount and still be weaker overall. \nThat is likely what is happening here.\n\nV-Ultimate’s profile is:\n\n- **largest average clinical recovery**\n- **best pairwise superiority**\n- **best image fidelity by a wide margin**\n\nThat is a stronger overall result than simply maximizing the number of barely-positive cases.\n\n---\n\n### Bucket-level nuance\n\nThe bucket analysis is also scientifically useful.\n\n#### 1. harmed_positive\nThis is where the model shines most clearly. \nV-Ultimate is best on both **clinical recovery** and **PSNR**.\n\n#### 2. neutral_other\nHere the model still has the highest Mean Target Gain (**+0.0207**) and highest PSNR (**42.60 dB**), but also shows some overshoot behavior:\n- Mean Abs Gain = **-0.0176**\n- Iatrogenic = **12.5%**\n\nSo on near-neutral cases, the model is more ambitious than a purely passive denoiser.\n\n#### 3. overcall_positive\nThis is the hardest bucket to interpret, because the degraded image is already overcalling. \nIn that setting, leaving the image alone can sometimes look deceptively strong in Target Gain. \nThat is why **Degraded** itself scores **+0.0417** here.\n\nV-Ultimate is weaker in this bucket (**+0.0273**) than in harmed_positive, which is actually understandable: \nthe model is designed primarily to **restore lost signal**, not to exploit already-overcalled cases.\n\nThis makes the system look more purposeful, not less. \nIts greatest advantage appears where restoration is genuinely needed.\n\n---\n\n### Example-level interpretation\n\nThe top cases show that V-Ultimate can produce **substantial rescue**:\n\n- UID 2704: **+0.1240**\n- UID 5571: **+0.1079**\n- UID 5003: **+0.1006**\n- UID 2859: **+0.0996**\n\nThese are not marginal wins. They are large recoveries in clinically harmed cases.\n\nThe bottom cases also help clarify the model’s weakness pattern:\n\n- several failures occur in **neutral_other**\n- one occurs in **overcall_positive**\n- the worst negative case is **-0.0347**\n\nSo the main risk is not random collapse across all scenarios. \nThe main risk is **over-correction or directional error in borderline / already-high cases**.\n\nThat is a much more actionable failure mode for later refinement.\n\n---\n\n### Overall conclusion\n\nTaken together, these results support a strong conclusion:\n\n> **V-Ultimate is the most effective overall method in the CRM benchmark.** \n> It delivers the **highest average clinical recovery**, the **highest win rate**, and by far the **best image fidelity**, while keeping failure modes bounded rather than uncontrolled.\n\n\n- it is **not perfect**\n- it still has measurable overshoot risk\n- but it is already **meaningfully better than standard restoration baselines**\n- and it is especially strong in the cases that matter most: those where degradation suppresses clinically relevant signal\n\nThat is exactly the pattern one would hope to see from a physics-constrained, do-no-harm restoration model.","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 8) Stage C: Monte Carlo Noise Stability Test (NEW MODEL)\n# Compatible with: FiLM + authority map V-Ultimate\n# ============================================================\n\nimport os, sys, time, math, random, gc\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport torch\nfrom contextlib import nullcontext\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\n# -----------------------------\n# 0) Prerequisites check\n# -----------------------------\nrequired_any = {\n \"model\": (\"model_25d\" in globals()),\n \"judge\": (\"aneurysm_predict\" in globals()),\n \"loader\": (\"load_series_volume\" in globals()),\n}\nif not all(required_any.values()):\n missing = [k for k, v in required_any.items() if not v]\n raise RuntimeError(\n f\"Missing prerequisite objects/functions: {missing}\\n\"\n f\"Please run: new model loading + Clinical Judge + DICOM loading cells first.\"\n )\n\nMODEL_OBJ = globals().get(\"model_25d\", None)\nassert MODEL_OBJ is not None, \"model_25d not found\"\nassert hasattr(MODEL_OBJ, \"auth_head\"), \"Not the new model: missing auth_head\"\nassert hasattr(MODEL_OBJ, \"film_e1\"), \"Not the new model: missing FiLM module\"\n\n# -----------------------------\n# 1) Configuration\n# -----------------------------\nRSNA_DATA_ROOT = globals().get(\"RSNA_DATA_ROOT\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\")\nMETA_CSV = globals().get(\"META_CSV\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\")\n\nTARGET_D = int(globals().get(\"TARGET_D\", 64))\nTARGET_H = int(globals().get(\"TARGET_H\", 448))\nTARGET_W = int(globals().get(\"TARGET_W\", 448))\nTARGET_SHAPE = (TARGET_D, TARGET_H, TARGET_W)\n\nHU_MIN = float(globals().get(\"HU_MIN\", -1024.0))\nHU_MAX = float(globals().get(\"HU_MAX\", 3072.0))\nHU_RANGE = float(globals().get(\"HU_RANGE\", HU_MAX - HU_MIN))\n\nDIFFUSION_ALPHA = float(globals().get(\"DIFFUSION_ALPHA\", globals().get(\"LAM\", 0.20)))\nBLUR_T_MAX = float(globals().get(\"BLUR_T_MAX\", 8.0))\n\nEVAL_T = float(globals().get(\"EVAL_T\", 8.0))\nEVAL_DOSE = globals().get(\"EVAL_DOSE\", \"quarter\")\n\nN_MC_TO_RUN = int(globals().get(\"N_MC_CASES\", 100))\nMC_SEEDS = list(globals().get(\"MC_SEEDS\", list(range(10))))\nRESTORE_BATCH = int(globals().get(\"RESTORE_BATCH\", 16))\n\nOUTDIR = globals().get(\"OUTDIR\", \"/kaggle/working/clinical_rescue_matrix\")\nos.makedirs(OUTDIR, exist_ok=True)\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = (lambda: torch.amp.autocast(\"cuda\")) if USE_AMP else (lambda: nullcontext())\n\n# Thresholds\nTARGET_POS_TH = 0.005\nTARGET_NEG_TH = -0.005\nABS_POS_TH = 0.005\nABS_NEG_TH = -0.005\n\n# -----------------------------\n# 2) Utility functions\n# -----------------------------\ndef mc_bucket(p_gt, p_deg, tau=0.03):\n if p_gt >= 0.5 and p_deg < p_gt - tau:\n return \"harmed_positive\"\n if p_gt >= 0.5 and p_deg > p_gt + tau:\n return \"overcall_positive\"\n return \"neutral_other\"\n\ndef degrade_volume_mc_seeded(vol01, t, dose_mode=\"quarter\", enable_motion=False):\n \"\"\"\n Stochastic degradation matching training/CRM style.\n Randomness is controlled by the external seed set before calling.\n \"\"\"\n D = vol01.shape[0]\n out = np.empty_like(vol01, dtype=np.float32)\n\n sigma = math.sqrt(max(1e-8, 2.0 * DIFFUSION_ALPHA * float(t)))\n\n for z in range(D):\n x = vol01[z].astype(np.float32)\n x = cv2.GaussianBlur(\n x, (0, 0),\n sigmaX=sigma, sigmaY=sigma,\n borderType=cv2.BORDER_REPLICATE\n )\n\n # Motion artefact (disabled by default)\n if enable_motion:\n pass\n\n if dose_mode != \"clean\":\n if dose_mode == \"extreme\":\n peak = random.uniform(1000.0, 3000.0)\n sigma_e = random.uniform(0.02, 0.04)\n else:\n peak = random.uniform(3000.0, 6000.0)\n sigma_e = random.uniform(0.01, 0.02)\n\n noisy_p = np.random.poisson(np.clip(x * peak, 0, None)).astype(np.float32) / peak\n x = noisy_p + np.random.randn(*x.shape).astype(np.float32) * sigma_e\n\n out[z] = np.clip(x, 0.0, 1.0).astype(np.float32)\n\n return out\n\n@torch.no_grad()\ndef run_vultimate_mc(vol_deg01, t=EVAL_T, restore_batch=RESTORE_BATCH):\n \"\"\"\n FiLM + authority map compatible inference.\n meta = [t_norm, do_motion, dose_clean, is_identity]\n \"\"\"\n vol_deg01 = np.asarray(vol_deg01, dtype=np.float32)\n D = vol_deg01.shape[0]\n out = vol_deg01.copy()\n\n t_norm = np.float32(0.0 if t <= 0 else (float(t) / float(BLUR_T_MAX)))\n meta_row = np.array([t_norm, 0.0, 0.0, 0.0], dtype=np.float32)\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch = []\n\n for z in zs:\n bp = vol_deg01[max(0, z - 1)]\n bc = vol_deg01[z]\n bn = vol_deg01[min(D - 1, z + 1)]\n inp_batch.append(\n np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n )\n\n inp_t = torch.from_numpy(np.stack(inp_batch, axis=0)).to(device, non_blocking=True)\n meta_t = torch.from_numpy(np.repeat(meta_row[None, :], len(zs), axis=0)).to(device, non_blocking=True)\n\n with AMP_CTX():\n pred_obj = MODEL_OBJ(inp_t, meta=meta_t)\n if isinstance(pred_obj, (tuple, list)):\n pred_b = pred_obj[0].float().cpu().numpy()[:, 0]\n else:\n pred_b = pred_obj.float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n out[z] = np.clip(pred_b[k], 0.0, 1.0).astype(np.float32)\n\n return out\n\n# -----------------------------\n# 3) Build OOD CT/CTA pool\n# -----------------------------\nprint(\"=== Stage C | Monte Carlo Stability Test ===\")\n\nmeta = pd.read_csv(META_CSV)\nct_uids = set(meta[meta[\"Modality\"].astype(str).isin({\"CT\", \"CTA\"})][\"SeriesInstanceUID\"].astype(str).tolist())\n\ntrain_uid_set = set()\nfor p in [\n \"/kaggle/working/vultimate_sleep_safe/train_uids_ultimate.csv\",\n \"/kaggle/working/train_uids_ultimate.csv\",\n \"/kaggle/input/datasets/mingzeli2009/train-uids-ultimate/train_uids_ultimate.csv\",\n \"/kaggle/working/train_uids_ct_only.csv\",\n]:\n if os.path.exists(p):\n try:\n train_uid_set = set(pd.read_csv(p)[\"SeriesInstanceUID\"].astype(str).tolist())\n print(f\"[Train exclusion] loaded {len(train_uid_set)} train UIDs from: {p}\")\n break\n except Exception:\n pass\n\nall_series_dirs = [u for u in os.listdir(RSNA_DATA_ROOT) if os.path.isdir(os.path.join(RSNA_DATA_ROOT, u))]\nood_pool = [u for u in all_series_dirs if (u in ct_uids) and (u not in train_uid_set)]\nrandom.Random(2026).shuffle(ood_pool)\n\nmc_candidates = ood_pool[:max(N_MC_TO_RUN * 3, 300)]\n\n# -----------------------------\n# 4) Pre-load cases\n# -----------------------------\nmc_cases = []\nt_prep = time.time()\n\nfor uid in mc_candidates:\n if len(mc_cases) >= N_MC_TO_RUN:\n break\n\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT, TARGET_SHAPE)\n except TypeError:\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT)\n except TypeError:\n vol = load_series_volume(uid)\n\n if vol is None:\n continue\n\n gt_vol = vol.astype(np.float32)\n p_gt = float(aneurysm_predict(vol01_to_flayer_uint8(gt_vol)))\n\n mc_cases.append({\n \"uid\": uid,\n \"uid4\": uid_tail4(uid),\n \"gt_vol\": gt_vol,\n \"p_gt\": p_gt,\n })\n\n if (len(mc_cases) == 1) or (len(mc_cases) % 10 == 0) or (len(mc_cases) == N_MC_TO_RUN):\n print(f\"[prep {len(mc_cases):03d}/{N_MC_TO_RUN}] UID:{uid_tail4(uid)} | p_gt={p_gt:.4f}\")\n\nprint(f\"MC selected cases: {len(mc_cases)} / {N_MC_TO_RUN}\")\nprint(f\"MC seeds: {MC_SEEDS}\")\nprint(f\"MC preparation elapsed: {(time.time()-t_prep)/60:.1f} min\")\n\n# -----------------------------\n# 5) Monte Carlo main loop\n# -----------------------------\nmc_raw_rows = []\nt_mc = time.time()\n\nfor ci, case in enumerate(mc_cases, 1):\n uid = case[\"uid\"]\n uid4 = case[\"uid4\"]\n gt_vol = case[\"gt_vol\"]\n p_gt = float(case[\"p_gt\"])\n\n print(f\"\\n[{ci:03d}/{len(mc_cases)}] UID:{uid4} | p_gt={p_gt:.4f} | running {len(MC_SEEDS)} seeds ...\")\n case_t0 = time.time()\n\n case_bucket_votes = []\n\n for s in MC_SEEDS:\n random.seed(s)\n np.random.seed(s)\n torch.manual_seed(s)\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(s)\n\n deg_vol = degrade_volume_mc_seeded(gt_vol, EVAL_T, dose_mode=EVAL_DOSE, enable_motion=False)\n rec_vol = run_vultimate_mc(deg_vol, t=EVAL_T, restore_batch=RESTORE_BATCH)\n\n p_deg = float(aneurysm_predict(vol01_to_flayer_uint8(deg_vol)))\n p_rec = float(aneurysm_predict(vol01_to_flayer_uint8(rec_vol)))\n\n bucket = mc_bucket(p_gt, p_deg, tau=0.03)\n case_bucket_votes.append(bucket)\n\n t_gain = float(calc_target_gain(p_gt, p_deg, p_rec))\n a_gain = float(calc_abs_gain(p_gt, p_deg, p_rec))\n\n mc_raw_rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"seed\": int(s),\n \"bucket\": bucket,\n \"p_gt\": p_gt,\n \"p_deg\": p_deg,\n \"p_rec\": p_rec,\n \"target_gain\": t_gain,\n \"abs_gain\": a_gain,\n \"target_positive\": int(t_gain > TARGET_POS_TH),\n \"target_negative\": int(t_gain < TARGET_NEG_TH),\n \"abs_positive\": int(a_gain > ABS_POS_TH),\n \"abs_negative\": int(a_gain < ABS_NEG_TH),\n })\n\n del deg_vol, rec_vol\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n tmp = pd.DataFrame([r for r in mc_raw_rows if r[\"uid_full\"] == uid])\n major_bucket = tmp[\"bucket\"].mode().iloc[0] if len(tmp) else \"NA\"\n\n print(\n \" -> case done in {:.1f}s | bucket={} | \"\n \"TGain mean={:+.4f}, std={:.4f}, pos/neg={:.2f}/{:.2f} | \"\n \"AGain mean={:+.4f}, pos/neg={:.2f}/{:.2f}\".format(\n time.time()-case_t0,\n major_bucket,\n tmp[\"target_gain\"].mean(), tmp[\"target_gain\"].std(ddof=0),\n (tmp[\"target_gain\"] > TARGET_POS_TH).mean(), (tmp[\"target_gain\"] < TARGET_NEG_TH).mean(),\n tmp[\"abs_gain\"].mean(),\n (tmp[\"abs_gain\"] > ABS_POS_TH).mean(), (tmp[\"abs_gain\"] < ABS_NEG_TH).mean(),\n )\n )\n\ndf_mc_raw = pd.DataFrame(mc_raw_rows)\n\n# -----------------------------\n# 6) Aggregation\n# -----------------------------\ndef agg_mc_case(g):\n t = g[\"target_gain\"].to_numpy(dtype=float)\n a = g[\"abs_gain\"].to_numpy(dtype=float)\n pdeg = g[\"p_deg\"].to_numpy(dtype=float)\n prec = g[\"p_rec\"].to_numpy(dtype=float)\n pgt = float(g[\"p_gt\"].iloc[0])\n\n t_pos = t > TARGET_POS_TH\n t_neg = t < TARGET_NEG_TH\n a_pos = a > ABS_POS_TH\n a_neg = a < ABS_NEG_TH\n\n bucket_mode = g[\"bucket\"].mode().iloc[0] if len(g[\"bucket\"].mode()) else \"NA\"\n\n return pd.Series({\n \"n_runs\": int(len(g)),\n \"bucket_major\": bucket_mode,\n \"p_gt\": pgt,\n\n \"target_gain_mean\": float(np.mean(t)),\n \"target_gain_std\": float(np.std(t, ddof=0)),\n \"target_gain_min\": float(np.min(t)),\n \"target_gain_max\": float(np.max(t)),\n \"target_pos_rate\": float(np.mean(t_pos)),\n \"target_neg_rate\": float(np.mean(t_neg)),\n \"target_flip\": bool(np.any(t_pos) and np.any(t_neg)),\n\n \"abs_gain_mean\": float(np.mean(a)),\n \"abs_gain_std\": float(np.std(a, ddof=0)),\n \"abs_gain_min\": float(np.min(a)),\n \"abs_gain_max\": float(np.max(a)),\n \"abs_pos_rate\": float(np.mean(a_pos)),\n \"abs_neg_rate\": float(np.mean(a_neg)),\n \"abs_flip\": bool(np.any(a_pos) and np.any(a_neg)),\n\n \"p_deg_std\": float(np.std(pdeg, ddof=0)),\n \"p_rec_std\": float(np.std(prec, ddof=0)),\n })\n\ndf_mc_agg = (\n df_mc_raw.groupby([\"uid_full\", \"uid4\"], as_index=False)\n .apply(agg_mc_case)\n .reset_index(drop=True)\n)\n\nmc_raw_path = os.path.join(OUTDIR, \"mc_noise_100cases_10seeds_raw.csv\")\nmc_agg_path = os.path.join(OUTDIR, \"mc_noise_100cases_10seeds_agg.csv\")\ndf_mc_raw.to_csv(mc_raw_path, index=False)\ndf_mc_agg.to_csv(mc_agg_path, index=False)\n\n# -----------------------------\n# 7) Summary statistics\n# -----------------------------\nn = len(df_mc_agg)\n\nstable_pos = ((df_mc_agg[\"target_pos_rate\"] > 0) & (df_mc_agg[\"target_neg_rate\"] == 0)).sum()\nflip = ((df_mc_agg[\"target_pos_rate\"] > 0) & (df_mc_agg[\"target_neg_rate\"] > 0)).sum()\nstable_neg = ((df_mc_agg[\"target_pos_rate\"] == 0) & (df_mc_agg[\"target_neg_rate\"] > 0)).sum()\nneutral = ((df_mc_agg[\"target_pos_rate\"] == 0) & (df_mc_agg[\"target_neg_rate\"] == 0)).sum()\n\nmc_summary = {\n \"n_cases\": n,\n \"n_seeds_per_case\": len(MC_SEEDS),\n \"n_total_runs\": len(df_mc_raw),\n\n \"target_gain_mean(run-level)\": float(df_mc_raw[\"target_gain\"].mean()),\n \"target_positive_rate(run-level)\": float((df_mc_raw[\"target_gain\"] > TARGET_POS_TH).mean()),\n \"target_negative_rate(run-level)\": float((df_mc_raw[\"target_gain\"] < TARGET_NEG_TH).mean()),\n\n \"abs_gain_mean(run-level)\": float(df_mc_raw[\"abs_gain\"].mean()),\n\n \"cases_with_target_flip\": int(df_mc_agg[\"target_flip\"].sum()),\n \"stable_positive\": int(stable_pos),\n \"noise_sensitive_flip\": int(flip),\n \"stable_negative\": int(stable_neg),\n \"neutral\": int(neutral),\n\n \"elapsed_min\": round((time.time() - t_mc) / 60.0, 1),\n}\n\nprint(\"\\n\" + \"=\"*100)\nprint(f\"Monte Carlo Stability Summary ({n} cases × {len(MC_SEEDS)} seeds)\")\nprint(\"=\"*100)\nfor k, v in mc_summary.items():\n if isinstance(v, float):\n print(f\"{k:>45}: {v:.4f}\")\n else:\n print(f\"{k:>45}: {v}\")\n\nprint(f\"\\nCase classification:\")\nprint(f\" Stable positive: {stable_pos}/{n} ({stable_pos/n*100:.0f}%)\")\nprint(f\" Noise-sensitive: {flip}/{n} ({flip/n*100:.0f}%)\")\nprint(f\" Stable negative: {stable_neg}/{n} ({stable_neg/n*100:.0f}%)\")\nprint(f\" Neutral: {neutral}/{n} ({neutral/n*100:.0f}%)\")\n\nprint(\"\\n[Top noise-sensitive cases by target_gain_std]\")\ndisplay(\n df_mc_agg.sort_values([\"target_gain_std\", \"target_neg_rate\"], ascending=[False, False])\n .head(10)\n .reset_index(drop=True)\n)\n\nprint(\"\\n[Top unstable / failure-prone cases by target_neg_rate]\")\ndisplay(\n df_mc_agg.sort_values([\"target_neg_rate\", \"target_gain_std\"], ascending=[False, False])\n .head(10)\n .reset_index(drop=True)\n)\n\nprint(\"\\n[Bucket breakdown]\")\ndisplay(\n df_mc_agg.groupby(\"bucket_major\", as_index=False)\n .agg(\n n_cases=(\"uid_full\", \"size\"),\n mean_target_gain=(\"target_gain_mean\", \"mean\"),\n mean_target_std=(\"target_gain_std\", \"mean\"),\n cases_with_flip=(\"target_flip\", \"sum\"),\n mean_prec_std=(\"p_rec_std\", \"mean\"),\n )\n)\n\nprint(\"\\nsaved:\", mc_raw_path)\nprint(\"saved:\", mc_agg_path)\nprint(\"\\n✅ Monte Carlo complete.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-11T00:53:17.063037Z","iopub.execute_input":"2026-03-11T00:53:17.063939Z","iopub.status.idle":"2026-03-11T04:54:45.52839Z","shell.execute_reply.started":"2026-03-11T00:53:17.063904Z","shell.execute_reply":"2026-03-11T04:54:45.527775Z"}},"outputs":[{"name":"stdout","text":"=== Stage C | Monte Carlo Stability Test ===\n[Train exclusion] loaded 300 train UIDs from: /kaggle/input/datasets/mingzeli2009/train-uids-ultimate/train_uids_ultimate.csv\n[prep 001/100] UID:4681 | p_gt=0.7222\n[prep 010/100] UID:6360 | p_gt=0.7295\n[prep 020/100] UID:9162 | p_gt=0.6943\n[prep 030/100] UID:0335 | p_gt=0.8613\n[prep 040/100] UID:0295 | p_gt=0.8301\n[prep 050/100] UID:7465 | p_gt=0.8428\n[prep 070/100] UID:9787 | p_gt=0.6631\n[prep 080/100] UID:9061 | p_gt=0.8589\n[prep 090/100] UID:0663 | p_gt=0.7056\n[prep 100/100] UID:8718 | p_gt=0.8022\nMC selected cases: 100 / 100\nMC seeds: [10, 42, 23, 55, 83, 9999, 7, 11, 19, 29]\nMC preparation elapsed: 19.0 min\n\n[001/100] UID:4681 | p_gt=0.7222 | running 10 seeds ...\n -> case done in 149.9s | bucket=harmed_positive | TGain mean=+0.0057, std=0.0356, pos/neg=0.60/0.30 | AGain mean=+0.0057, pos/neg=0.60/0.30\n\n[002/100] UID:6214 | p_gt=0.8149 | running 10 seeds ...\n -> case done in 154.4s | bucket=harmed_positive | TGain mean=+0.0492, std=0.0365, pos/neg=0.90/0.10 | AGain mean=+0.0260, pos/neg=0.70/0.10\n\n[003/100] UID:0503 | p_gt=0.7900 | running 10 seeds ...\n -> case done in 151.6s | bucket=harmed_positive | TGain mean=+0.0464, std=0.0325, pos/neg=0.80/0.00 | AGain mean=+0.0464, pos/neg=0.80/0.00\n\n[004/100] UID:4043 | p_gt=0.7549 | running 10 seeds ...\n -> case done in 151.9s | bucket=harmed_positive | TGain mean=+0.0301, std=0.0423, pos/neg=0.70/0.20 | AGain mean=+0.0301, pos/neg=0.70/0.20\n\n[005/100] UID:5743 | p_gt=0.8115 | running 10 seeds ...\n -> case done in 155.4s | bucket=harmed_positive | TGain mean=+0.0560, std=0.0469, pos/neg=0.80/0.10 | AGain mean=+0.0098, pos/neg=0.30/0.40\n\n[006/100] UID:2006 | p_gt=0.8462 | running 10 seeds ...\n -> case done in 154.8s | bucket=harmed_positive | TGain mean=+0.0492, std=0.0402, pos/neg=0.80/0.00 | AGain mean=+0.0356, pos/neg=0.90/0.00\n\n[007/100] UID:1704 | p_gt=0.8335 | running 10 seeds ...\n -> case done in 153.4s | bucket=harmed_positive | TGain mean=+0.0835, std=0.0234, pos/neg=1.00/0.00 | AGain mean=+0.0480, pos/neg=0.90/0.00\n\n[008/100] UID:7426 | p_gt=0.8691 | running 10 seeds ...\n -> case done in 148.9s | bucket=harmed_positive | TGain mean=+0.0405, std=0.0352, pos/neg=0.90/0.10 | AGain mean=+0.0405, pos/neg=0.90/0.10\n\n[009/100] UID:1565 | p_gt=0.8618 | running 10 seeds ...\n -> case done in 152.9s | bucket=harmed_positive | TGain mean=+0.0847, std=0.0251, pos/neg=1.00/0.00 | AGain mean=+0.0823, pos/neg=1.00/0.00\n\n[010/100] UID:6360 | p_gt=0.7295 | running 10 seeds ...\n -> case done in 148.7s | bucket=neutral_other | TGain mean=+0.0503, std=0.0299, pos/neg=1.00/0.00 | AGain mean=+0.0071, pos/neg=0.50/0.20\n\n[011/100] UID:5503 | p_gt=0.8066 | running 10 seeds ...\n -> case done in 150.6s | bucket=harmed_positive | TGain mean=+0.0761, std=0.0419, pos/neg=1.00/0.00 | AGain mean=+0.0120, pos/neg=0.50/0.40\n\n[012/100] UID:1562 | p_gt=0.7930 | running 10 seeds ...\n -> case done in 150.7s | bucket=neutral_other | TGain mean=+0.0788, std=0.0311, pos/neg=1.00/0.00 | AGain mean=-0.0430, pos/neg=0.10/0.90\n\n[013/100] UID:4607 | p_gt=0.8984 | running 10 seeds ...\n -> case done in 137.2s | bucket=harmed_positive | TGain mean=+0.0390, std=0.0291, pos/neg=0.90/0.00 | AGain mean=+0.0390, pos/neg=0.90/0.00\n\n[014/100] UID:5571 | p_gt=0.8491 | running 10 seeds ...\n -> case done in 135.5s | bucket=harmed_positive | TGain mean=+0.0375, std=0.0372, pos/neg=0.90/0.10 | AGain mean=+0.0300, pos/neg=0.80/0.10\n\n[015/100] UID:5550 | p_gt=0.8145 | running 10 seeds ...\n -> case done in 136.2s | bucket=harmed_positive | TGain mean=+0.0709, std=0.0199, pos/neg=1.00/0.00 | AGain mean=+0.0380, pos/neg=1.00/0.00\n\n[016/100] UID:9191 | p_gt=0.8164 | running 10 seeds ...\n -> case done in 136.6s | bucket=neutral_other | TGain mean=+0.0372, std=0.0370, pos/neg=0.70/0.20 | AGain mean=-0.0132, pos/neg=0.40/0.50\n\n[017/100] UID:5623 | p_gt=0.8413 | running 10 seeds ...\n -> case done in 137.1s | bucket=harmed_positive | TGain mean=+0.0365, std=0.0300, pos/neg=0.80/0.10 | AGain mean=+0.0271, pos/neg=0.60/0.10\n\n[018/100] UID:7916 | p_gt=0.6729 | running 10 seeds ...\n -> case done in 129.6s | bucket=neutral_other | TGain mean=+0.0271, std=0.0288, pos/neg=0.80/0.10 | AGain mean=-0.0042, pos/neg=0.50/0.50\n\n[019/100] UID:9138 | p_gt=0.7861 | running 10 seeds ...\n -> case done in 133.4s | bucket=neutral_other | TGain mean=+0.0578, std=0.0302, pos/neg=1.00/0.00 | AGain mean=-0.0161, pos/neg=0.30/0.50\n\n[020/100] UID:9162 | p_gt=0.6943 | running 10 seeds ...\n -> case done in 128.7s | bucket=neutral_other | TGain mean=+0.0295, std=0.0303, pos/neg=0.90/0.10 | AGain mean=-0.0046, pos/neg=0.40/0.50\n\n[021/100] UID:5073 | p_gt=0.7720 | running 10 seeds ...\n -> case done in 132.1s | bucket=neutral_other | TGain mean=+0.0439, std=0.0201, pos/neg=0.90/0.00 | AGain mean=-0.0187, pos/neg=0.20/0.70\n\n[022/100] UID:1796 | p_gt=0.7529 | running 10 seeds ...\n -> case done in 128.3s | bucket=harmed_positive | TGain mean=+0.0441, std=0.0478, pos/neg=0.70/0.30 | AGain mean=+0.0218, pos/neg=0.60/0.40\n\n[023/100] UID:4616 | p_gt=0.8364 | running 10 seeds ...\n -> case done in 132.3s | bucket=harmed_positive | TGain mean=+0.0843, std=0.0466, pos/neg=0.90/0.10 | AGain mean=+0.0704, pos/neg=0.90/0.10\n\n[024/100] UID:8763 | p_gt=0.8213 | running 10 seeds ...\n -> case done in 131.7s | bucket=harmed_positive | TGain mean=+0.0462, std=0.0425, pos/neg=0.80/0.20 | AGain mean=+0.0344, pos/neg=0.80/0.20\n\n[025/100] UID:1377 | p_gt=0.7495 | running 10 seeds ...\n -> case done in 128.9s | bucket=harmed_positive | TGain mean=+0.0356, std=0.0396, pos/neg=0.80/0.20 | AGain mean=+0.0356, pos/neg=0.80/0.20\n\n[026/100] UID:0321 | p_gt=0.8462 | running 10 seeds ...\n -> case done in 131.3s | bucket=harmed_positive | TGain mean=+0.0589, std=0.0281, pos/neg=1.00/0.00 | AGain mean=+0.0494, pos/neg=1.00/0.00\n\n[027/100] UID:7315 | p_gt=0.8379 | running 10 seeds ...\n -> case done in 133.3s | bucket=harmed_positive | TGain mean=+0.0376, std=0.0260, pos/neg=0.90/0.00 | AGain mean=+0.0210, pos/neg=0.80/0.00\n\n[028/100] UID:8356 | p_gt=0.7856 | running 10 seeds ...\n -> case done in 133.7s | bucket=neutral_other | TGain mean=+0.0805, std=0.0201, pos/neg=1.00/0.00 | AGain mean=-0.0521, pos/neg=0.10/0.90\n\n[029/100] UID:6463 | p_gt=0.7847 | running 10 seeds ...\n -> case done in 131.7s | bucket=neutral_other | TGain mean=+0.0741, std=0.0342, pos/neg=1.00/0.00 | AGain mean=-0.0312, pos/neg=0.00/1.00\n\n[030/100] UID:0335 | p_gt=0.8613 | running 10 seeds ...\n -> case done in 133.3s | bucket=harmed_positive | TGain mean=+0.0678, std=0.0290, pos/neg=1.00/0.00 | AGain mean=+0.0432, pos/neg=0.90/0.00\n\n[031/100] UID:7400 | p_gt=0.8130 | running 10 seeds ...\n -> case done in 131.3s | bucket=neutral_other | TGain mean=+0.0239, std=0.0264, pos/neg=0.70/0.10 | AGain mean=-0.0072, pos/neg=0.40/0.50\n\n[032/100] UID:8345 | p_gt=0.7290 | running 10 seeds ...\n -> case done in 127.2s | bucket=harmed_positive | TGain mean=+0.0220, std=0.0495, pos/neg=0.40/0.50 | AGain mean=+0.0274, pos/neg=0.50/0.40\n\n[033/100] UID:4582 | p_gt=0.8057 | running 10 seeds ...\n -> case done in 133.5s | bucket=neutral_other | TGain mean=+0.0548, std=0.0429, pos/neg=0.90/0.10 | AGain mean=-0.0065, pos/neg=0.30/0.60\n\n[034/100] UID:9565 | p_gt=0.7725 | running 10 seeds ...\n -> case done in 131.8s | bucket=neutral_other | TGain mean=+0.0917, std=0.0353, pos/neg=1.00/0.00 | AGain mean=-0.0369, pos/neg=0.10/0.80\n\n[035/100] UID:2859 | p_gt=0.9302 | running 10 seeds ...\n -> case done in 133.0s | bucket=harmed_positive | TGain mean=+0.0959, std=0.0311, pos/neg=1.00/0.00 | AGain mean=+0.0959, pos/neg=1.00/0.00\n\n[036/100] UID:3513 | p_gt=0.7261 | running 10 seeds ...\n -> case done in 129.2s | bucket=neutral_other | TGain mean=+0.0415, std=0.0153, pos/neg=1.00/0.00 | AGain mean=-0.0023, pos/neg=0.30/0.50\n\n[037/100] UID:5697 | p_gt=0.8545 | running 10 seeds ...\n -> case done in 131.7s | bucket=harmed_positive | TGain mean=+0.0598, std=0.0244, pos/neg=1.00/0.00 | AGain mean=+0.0519, pos/neg=0.90/0.10\n\n[038/100] UID:6772 | p_gt=0.8472 | running 10 seeds ...\n -> case done in 133.3s | bucket=harmed_positive | TGain mean=+0.0998, std=0.0252, pos/neg=1.00/0.00 | AGain mean=+0.0666, pos/neg=1.00/0.00\n\n[039/100] UID:9485 | p_gt=0.7749 | running 10 seeds ...\n -> case done in 133.8s | bucket=neutral_other | TGain mean=+0.0640, std=0.0237, pos/neg=1.00/0.00 | AGain mean=-0.0348, pos/neg=0.00/0.80\n\n[040/100] UID:0295 | p_gt=0.8301 | running 10 seeds ...\n -> case done in 131.2s | bucket=harmed_positive | TGain mean=+0.0566, std=0.0345, pos/neg=0.90/0.10 | AGain mean=+0.0517, pos/neg=0.90/0.10\n\n[041/100] UID:7186 | p_gt=0.8809 | running 10 seeds ...\n -> case done in 131.1s | bucket=harmed_positive | TGain mean=+0.0401, std=0.0366, pos/neg=0.70/0.10 | AGain mean=+0.0401, pos/neg=0.70/0.10\n\n[042/100] UID:5583 | p_gt=0.6670 | running 10 seeds ...\n -> case done in 126.7s | bucket=neutral_other | TGain mean=+0.0201, std=0.0215, pos/neg=0.70/0.10 | AGain mean=+0.0037, pos/neg=0.40/0.40\n\n[043/100] UID:0016 | p_gt=0.7617 | running 10 seeds ...\n -> case done in 126.9s | bucket=harmed_positive | TGain mean=-0.0106, std=0.0239, pos/neg=0.30/0.60 | AGain mean=-0.0106, pos/neg=0.30/0.60\n\n[044/100] UID:5829 | p_gt=0.8447 | running 10 seeds ...\n -> case done in 130.3s | bucket=harmed_positive | TGain mean=+0.0604, std=0.0356, pos/neg=1.00/0.00 | AGain mean=+0.0478, pos/neg=1.00/0.00\n\n[045/100] UID:6979 | p_gt=0.6797 | running 10 seeds ...\n -> case done in 127.8s | bucket=neutral_other | TGain mean=+0.0319, std=0.0311, pos/neg=0.80/0.10 | AGain mean=-0.0161, pos/neg=0.10/0.70\n\n[046/100] UID:9348 | p_gt=0.7212 | running 10 seeds ...\n -> case done in 128.5s | bucket=harmed_positive | TGain mean=+0.0657, std=0.0350, pos/neg=0.90/0.10 | AGain mean=+0.0106, pos/neg=0.50/0.40\n\n[047/100] UID:5409 | p_gt=0.8262 | running 10 seeds ...\n -> case done in 133.6s | bucket=harmed_positive | TGain mean=+0.0428, std=0.0320, pos/neg=0.80/0.10 | AGain mean=+0.0345, pos/neg=0.80/0.10\n\n[048/100] UID:1381 | p_gt=0.8228 | running 10 seeds ...\n -> case done in 133.7s | bucket=harmed_positive | TGain mean=+0.0773, std=0.0441, pos/neg=0.90/0.10 | AGain mean=-0.0039, pos/neg=0.50/0.40\n\n[049/100] UID:9641 | p_gt=0.8311 | running 10 seeds ...\n -> case done in 131.1s | bucket=neutral_other | TGain mean=+0.0132, std=0.0315, pos/neg=0.60/0.20 | AGain mean=+0.0072, pos/neg=0.70/0.30\n\n[050/100] UID:7465 | p_gt=0.8428 | running 10 seeds ...\n -> case done in 131.6s | bucket=harmed_positive | TGain mean=+0.0227, std=0.0344, pos/neg=0.60/0.20 | AGain mean=+0.0146, pos/neg=0.70/0.10\n\n[051/100] UID:5769 | p_gt=0.8633 | running 10 seeds ...\n -> case done in 131.7s | bucket=harmed_positive | TGain mean=+0.0761, std=0.0507, pos/neg=0.90/0.10 | AGain mean=+0.0629, pos/neg=0.90/0.10\n\n[052/100] UID:2919 | p_gt=0.7441 | running 10 seeds ...\n -> case done in 127.8s | bucket=harmed_positive | TGain mean=+0.0846, std=0.0181, pos/neg=1.00/0.00 | AGain mean=-0.0054, pos/neg=0.50/0.50\n\n[053/100] UID:1506 | p_gt=0.7144 | running 10 seeds ...\n -> case done in 127.1s | bucket=harmed_positive | TGain mean=+0.0391, std=0.0316, pos/neg=0.90/0.10 | AGain mean=+0.0244, pos/neg=0.80/0.20\n\n[054/100] UID:8682 | p_gt=0.6621 | running 10 seeds ...\n -> case done in 127.0s | bucket=harmed_positive | TGain mean=+0.0333, std=0.0557, pos/neg=0.60/0.40 | AGain mean=+0.0209, pos/neg=0.60/0.40\n\n[055/100] UID:0259 | p_gt=0.8389 | running 10 seeds ...\n -> case done in 131.5s | bucket=harmed_positive | TGain mean=+0.0386, std=0.0299, pos/neg=0.90/0.10 | AGain mean=+0.0063, pos/neg=0.50/0.20\n\n[056/100] UID:4474 | p_gt=0.7896 | running 10 seeds ...\n -> case done in 131.5s | bucket=harmed_positive | TGain mean=+0.0521, std=0.0284, pos/neg=1.00/0.00 | AGain mean=+0.0136, pos/neg=0.70/0.20\n\n[057/100] UID:1683 | p_gt=0.8765 | running 10 seeds ...\n -> case done in 131.4s | bucket=harmed_positive | TGain mean=-0.0010, std=0.0172, pos/neg=0.40/0.50 | AGain mean=+0.0000, pos/neg=0.40/0.40\n\n[058/100] UID:7394 | p_gt=0.6855 | running 10 seeds ...\n -> case done in 135.1s | bucket=overcall_positive | TGain mean=+0.1013, std=0.0267, pos/neg=1.00/0.00 | AGain mean=-0.0292, pos/neg=0.30/0.70\n\n[059/100] UID:5981 | p_gt=0.6860 | running 10 seeds ...\n -> case done in 126.8s | bucket=neutral_other | TGain mean=+0.0057, std=0.0154, pos/neg=0.50/0.20 | AGain mean=+0.0167, pos/neg=0.50/0.20\n\n[060/100] UID:5611 | p_gt=0.6494 | running 10 seeds ...\n -> case done in 127.5s | bucket=neutral_other | TGain mean=+0.0080, std=0.0211, pos/neg=0.40/0.30 | AGain mean=-0.0014, pos/neg=0.20/0.40\n\n[061/100] UID:4733 | p_gt=0.7471 | running 10 seeds ...\n -> case done in 128.1s | bucket=harmed_positive | TGain mean=+0.0789, std=0.0339, pos/neg=1.00/0.00 | AGain mean=+0.0671, pos/neg=1.00/0.00\n\n[062/100] UID:7503 | p_gt=0.8457 | running 10 seeds ...\n -> case done in 132.1s | bucket=harmed_positive | TGain mean=+0.0458, std=0.0243, pos/neg=1.00/0.00 | AGain mean=+0.0344, pos/neg=0.90/0.10\n\n[063/100] UID:2355 | p_gt=0.7612 | running 10 seeds ...\n -> case done in 128.1s | bucket=harmed_positive | TGain mean=+0.0332, std=0.0384, pos/neg=0.90/0.10 | AGain mean=+0.0332, pos/neg=0.90/0.10\n\n[064/100] UID:8783 | p_gt=0.7261 | running 10 seeds ...\n -> case done in 127.7s | bucket=neutral_other | TGain mean=+0.0868, std=0.0293, pos/neg=1.00/0.00 | AGain mean=-0.0342, pos/neg=0.00/1.00\n\n[065/100] UID:7840 | p_gt=0.7700 | running 10 seeds ...\n -> case done in 131.8s | bucket=overcall_positive | TGain mean=+0.0753, std=0.0244, pos/neg=1.00/0.00 | AGain mean=-0.0356, pos/neg=0.10/0.90\n\n[066/100] UID:4677 | p_gt=0.6836 | running 10 seeds ...\n -> case done in 127.5s | bucket=harmed_positive | TGain mean=+0.0173, std=0.0396, pos/neg=0.50/0.40 | AGain mean=+0.0185, pos/neg=0.50/0.20\n\n[067/100] UID:2458 | p_gt=0.6284 | running 10 seeds ...\n -> case done in 124.7s | bucket=overcall_positive | TGain mean=+0.0375, std=0.0196, pos/neg=1.00/0.00 | AGain mean=+0.0132, pos/neg=0.60/0.30\n\n[068/100] UID:8193 | p_gt=0.7266 | running 10 seeds ...\n -> case done in 127.9s | bucket=harmed_positive | TGain mean=+0.0323, std=0.0292, pos/neg=0.70/0.00 | AGain mean=+0.0061, pos/neg=0.40/0.30\n\n[069/100] UID:9755 | p_gt=0.8291 | running 10 seeds ...\n -> case done in 131.7s | bucket=harmed_positive | TGain mean=+0.0721, std=0.0410, pos/neg=1.00/0.00 | AGain mean=+0.0342, pos/neg=1.00/0.00\n\n[070/100] UID:9787 | p_gt=0.6631 | running 10 seeds ...\n -> case done in 127.2s | bucket=neutral_other | TGain mean=-0.0022, std=0.0207, pos/neg=0.40/0.60 | AGain mean=+0.0068, pos/neg=0.60/0.20\n\n[071/100] UID:0277 | p_gt=0.8599 | running 10 seeds ...\n -> case done in 133.2s | bucket=harmed_positive | TGain mean=+0.0334, std=0.0309, pos/neg=0.80/0.10 | AGain mean=+0.0338, pos/neg=0.80/0.10\n\n[072/100] UID:5226 | p_gt=0.8105 | running 10 seeds ...\n -> case done in 133.8s | bucket=harmed_positive | TGain mean=+0.0687, std=0.0312, pos/neg=1.00/0.00 | AGain mean=-0.0032, pos/neg=0.30/0.50\n\n[073/100] UID:5568 | p_gt=0.7168 | running 10 seeds ...\n -> case done in 127.8s | bucket=harmed_positive | TGain mean=+0.0555, std=0.0470, pos/neg=0.80/0.10 | AGain mean=+0.0320, pos/neg=0.80/0.10\n\n[074/100] UID:6025 | p_gt=0.8130 | running 10 seeds ...\n -> case done in 132.5s | bucket=neutral_other | TGain mean=+0.0707, std=0.0217, pos/neg=1.00/0.00 | AGain mean=-0.0450, pos/neg=0.00/1.00\n\n[075/100] UID:9864 | p_gt=0.8174 | running 10 seeds ...\n -> case done in 131.6s | bucket=harmed_positive | TGain mean=+0.0716, std=0.0432, pos/neg=1.00/0.00 | AGain mean=+0.0563, pos/neg=1.00/0.00\n\n[076/100] UID:5879 | p_gt=0.8242 | running 10 seeds ...\n -> case done in 133.0s | bucket=harmed_positive | TGain mean=+0.0800, std=0.0373, pos/neg=1.00/0.00 | AGain mean=-0.0090, pos/neg=0.30/0.40\n\n[077/100] UID:0310 | p_gt=0.7715 | running 10 seeds ...\n -> case done in 131.9s | bucket=harmed_positive | TGain mean=+0.1029, std=0.0242, pos/neg=1.00/0.00 | AGain mean=+0.0273, pos/neg=0.70/0.30\n\n[078/100] UID:0601 | p_gt=0.8154 | running 10 seeds ...\n -> case done in 133.8s | bucket=harmed_positive | TGain mean=+0.0162, std=0.0316, pos/neg=0.60/0.40 | AGain mean=+0.0205, pos/neg=0.70/0.30\n\n[079/100] UID:6942 | p_gt=0.8687 | running 10 seeds ...\n -> case done in 131.0s | bucket=harmed_positive | TGain mean=+0.0494, std=0.0322, pos/neg=1.00/0.00 | AGain mean=+0.0479, pos/neg=1.00/0.00\n\n[080/100] UID:9061 | p_gt=0.8589 | running 10 seeds ...\n -> case done in 131.2s | bucket=harmed_positive | TGain mean=+0.0725, std=0.0323, pos/neg=1.00/0.00 | AGain mean=+0.0271, pos/neg=0.60/0.20\n\n[081/100] UID:8609 | p_gt=0.7617 | running 10 seeds ...\n -> case done in 128.6s | bucket=harmed_positive | TGain mean=+0.0126, std=0.0300, pos/neg=0.60/0.10 | AGain mean=+0.0126, pos/neg=0.60/0.10\n\n[082/100] UID:9614 | p_gt=0.8232 | running 10 seeds ...\n -> case done in 133.3s | bucket=harmed_positive | TGain mean=+0.1184, std=0.0283, pos/neg=1.00/0.00 | AGain mean=+0.0339, pos/neg=0.80/0.10\n\n[083/100] UID:4708 | p_gt=0.8193 | running 10 seeds ...\n -> case done in 133.7s | bucket=harmed_positive | TGain mean=+0.0776, std=0.0383, pos/neg=1.00/0.00 | AGain mean=+0.0213, pos/neg=0.70/0.10\n\n[084/100] UID:3582 | p_gt=0.8306 | running 10 seeds ...\n -> case done in 133.6s | bucket=harmed_positive | TGain mean=+0.0650, std=0.0334, pos/neg=0.90/0.00 | AGain mean=+0.0090, pos/neg=0.60/0.30\n\n[085/100] UID:6421 | p_gt=0.6982 | running 10 seeds ...\n -> case done in 127.9s | bucket=neutral_other | TGain mean=+0.0403, std=0.0164, pos/neg=1.00/0.00 | AGain mean=+0.0002, pos/neg=0.30/0.60\n\n[086/100] UID:3136 | p_gt=0.8242 | running 10 seeds ...\n -> case done in 131.9s | bucket=harmed_positive | TGain mean=+0.0500, std=0.0323, pos/neg=0.90/0.10 | AGain mean=+0.0435, pos/neg=0.90/0.10\n\n[087/100] UID:2096 | p_gt=0.6870 | running 10 seeds ...\n -> case done in 127.5s | bucket=harmed_positive | TGain mean=-0.0387, std=0.0181, pos/neg=0.00/1.00 | AGain mean=-0.0375, pos/neg=0.00/1.00\n\n[088/100] UID:4610 | p_gt=0.8613 | running 10 seeds ...\n -> case done in 133.8s | bucket=harmed_positive | TGain mean=+0.0521, std=0.0387, pos/neg=0.90/0.00 | AGain mean=+0.0480, pos/neg=0.90/0.00\n\n[089/100] UID:8285 | p_gt=0.7451 | running 10 seeds ...\n -> case done in 128.1s | bucket=neutral_other | TGain mean=-0.0383, std=0.0228, pos/neg=0.00/0.90 | AGain mean=-0.0315, pos/neg=0.00/0.90\n\n[090/100] UID:0663 | p_gt=0.7056 | running 10 seeds ...\n -> case done in 128.6s | bucket=neutral_other | TGain mean=+0.0012, std=0.0335, pos/neg=0.50/0.50 | AGain mean=+0.0024, pos/neg=0.50/0.40\n\n[091/100] UID:1762 | p_gt=0.7539 | running 10 seeds ...\n -> case done in 129.7s | bucket=neutral_other | TGain mean=-0.0035, std=0.0301, pos/neg=0.50/0.40 | AGain mean=-0.0044, pos/neg=0.50/0.40\n\n[092/100] UID:6911 | p_gt=0.7339 | running 10 seeds ...\n -> case done in 128.2s | bucket=overcall_positive | TGain mean=-0.0151, std=0.0223, pos/neg=0.10/0.70 | AGain mean=+0.0200, pos/neg=0.70/0.20\n\n[093/100] UID:6774 | p_gt=0.6611 | running 10 seeds ...\n -> case done in 128.7s | bucket=neutral_other | TGain mean=+0.0312, std=0.0313, pos/neg=0.70/0.00 | AGain mean=-0.0078, pos/neg=0.40/0.30\n\n[094/100] UID:5125 | p_gt=0.6914 | running 10 seeds ...\n -> case done in 127.7s | bucket=harmed_positive | TGain mean=+0.0801, std=0.0410, pos/neg=1.00/0.00 | AGain mean=+0.0011, pos/neg=0.50/0.40\n\n[095/100] UID:5611 | p_gt=0.7158 | running 10 seeds ...\n -> case done in 128.1s | bucket=harmed_positive | TGain mean=-0.0285, std=0.0379, pos/neg=0.30/0.70 | AGain mean=-0.0277, pos/neg=0.30/0.70\n\n[096/100] UID:8798 | p_gt=0.8828 | running 10 seeds ...\n -> case done in 131.8s | bucket=harmed_positive | TGain mean=+0.0940, std=0.0383, pos/neg=1.00/0.00 | AGain mean=+0.0916, pos/neg=1.00/0.00\n\n[097/100] UID:0443 | p_gt=0.8120 | running 10 seeds ...\n -> case done in 129.8s | bucket=harmed_positive | TGain mean=+0.0467, std=0.0498, pos/neg=0.80/0.20 | AGain mean=+0.0449, pos/neg=0.80/0.20\n\n[098/100] UID:3661 | p_gt=0.8623 | running 10 seeds ...\n -> case done in 134.0s | bucket=harmed_positive | TGain mean=+0.0193, std=0.0523, pos/neg=0.50/0.40 | AGain mean=+0.0175, pos/neg=0.50/0.40\n\n[099/100] UID:3121 | p_gt=0.8433 | running 10 seeds ...\n -> case done in 131.4s | bucket=harmed_positive | TGain mean=+0.0700, std=0.0424, pos/neg=0.80/0.20 | AGain mean=+0.0503, pos/neg=0.80/0.10\n\n[100/100] UID:8718 | p_gt=0.8022 | running 10 seeds ...\n -> case done in 131.4s | bucket=neutral_other | TGain mean=+0.0586, std=0.0244, pos/neg=1.00/0.00 | AGain mean=-0.0340, pos/neg=0.10/0.80\n\n====================================================================================================\nMonte Carlo Stability Summary (100 cases × 10 seeds)\n====================================================================================================\n n_cases: 100\n n_seeds_per_case: 10\n n_total_runs: 1000\n target_gain_mean(run-level): 0.0476\n target_positive_rate(run-level): 0.8150\n target_negative_rate(run-level): 0.1320\n abs_gain_mean(run-level): 0.0160\n cases_with_target_flip: 49\n stable_positive: 49\n noise_sensitive_flip: 49\n stable_negative: 2\n neutral: 0\n elapsed_min: 222.4000\n\nCase classification:\n Stable positive: 49/100 (49%)\n Noise-sensitive: 49/100 (49%)\n Stable negative: 2/100 (2%)\n Neutral: 0/100 (0%)\n\n[Top noise-sensitive cases by target_gain_std]\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" uid_full uid4 n_runs \\\n0 1.2.826.0.1.3680043.8.498.65394280042275818508... 8682 10 \n1 1.2.826.0.1.3680043.8.498.61241400663691838928... 3661 10 \n2 1.2.826.0.1.3680043.8.498.10952947258340598137... 5769 10 \n3 1.2.826.0.1.3680043.8.498.50074230080723551004... 0443 10 \n4 1.2.826.0.1.3680043.8.498.48062207595556588361... 8345 10 \n5 1.2.826.0.1.3680043.8.498.10345349366333570404... 1796 10 \n6 1.2.826.0.1.3680043.8.498.21004106426734635526... 5568 10 \n7 1.2.826.0.1.3680043.8.498.10313496695916659101... 5743 10 \n8 1.2.826.0.1.3680043.8.498.28546126228097356101... 4616 10 \n9 1.2.826.0.1.3680043.8.498.10035643165968342618... 1381 10 \n\n bucket_major p_gt target_gain_mean target_gain_std \\\n0 harmed_positive 0.662109 0.033301 0.055741 \n1 harmed_positive 0.862305 0.019287 0.052311 \n2 harmed_positive 0.863281 0.076074 0.050691 \n3 harmed_positive 0.812012 0.046729 0.049802 \n4 harmed_positive 0.729004 0.022021 0.049460 \n5 harmed_positive 0.752930 0.044141 0.047822 \n6 harmed_positive 0.716797 0.055469 0.047025 \n7 harmed_positive 0.811523 0.056006 0.046939 \n8 harmed_positive 0.836426 0.084326 0.046572 \n9 harmed_positive 0.822754 0.077295 0.044132 \n\n target_gain_min target_gain_max target_pos_rate ... target_flip \\\n0 -0.037109 0.116699 0.6 ... True \n1 -0.043457 0.144531 0.5 ... True \n2 -0.006836 0.159668 0.9 ... True \n3 -0.053223 0.102051 0.8 ... True \n4 -0.020508 0.126465 0.4 ... True \n5 -0.026855 0.110352 0.7 ... True \n6 -0.025879 0.135254 0.8 ... True \n7 -0.010254 0.142090 0.8 ... True \n8 -0.024414 0.133789 0.9 ... True \n9 -0.012207 0.151855 0.9 ... True \n\n abs_gain_mean abs_gain_std abs_gain_min abs_gain_max abs_pos_rate \\\n0 0.020947 0.042827 -0.030273 0.078613 0.6 \n1 0.017529 0.048208 -0.043457 0.126953 0.5 \n2 0.062891 0.041052 -0.006836 0.125488 0.9 \n3 0.044873 0.048076 -0.053223 0.102051 0.8 \n4 0.027393 0.047428 -0.020508 0.125488 0.5 \n5 0.021777 0.037574 -0.026855 0.083984 0.6 \n6 0.032031 0.029738 -0.025879 0.083496 0.8 \n7 0.009766 0.027578 -0.016602 0.079590 0.3 \n8 0.070361 0.043611 -0.024414 0.128906 0.9 \n9 -0.003857 0.030250 -0.064941 0.033203 0.5 \n\n abs_neg_rate abs_flip p_deg_std p_rec_std \n0 0.4 True 0.032685 0.038331 \n1 0.4 True 0.027204 0.035440 \n2 0.1 True 0.026226 0.030517 \n3 0.2 True 0.040952 0.026386 \n4 0.4 True 0.052562 0.025958 \n5 0.4 True 0.027771 0.028224 \n6 0.1 True 0.028242 0.037323 \n7 0.4 True 0.031158 0.025268 \n8 0.1 True 0.025061 0.030422 \n9 0.4 True 0.017639 0.034500 \n\n[10 rows x 21 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
uid_fulluid4n_runsbucket_majorp_gttarget_gain_meantarget_gain_stdtarget_gain_mintarget_gain_maxtarget_pos_rate...target_flipabs_gain_meanabs_gain_stdabs_gain_minabs_gain_maxabs_pos_rateabs_neg_rateabs_flipp_deg_stdp_rec_std
01.2.826.0.1.3680043.8.498.65394280042275818508...868210harmed_positive0.6621090.0333010.055741-0.0371090.1166990.6...True0.0209470.042827-0.0302730.0786130.60.4True0.0326850.038331
11.2.826.0.1.3680043.8.498.61241400663691838928...366110harmed_positive0.8623050.0192870.052311-0.0434570.1445310.5...True0.0175290.048208-0.0434570.1269530.50.4True0.0272040.035440
21.2.826.0.1.3680043.8.498.10952947258340598137...576910harmed_positive0.8632810.0760740.050691-0.0068360.1596680.9...True0.0628910.041052-0.0068360.1254880.90.1True0.0262260.030517
31.2.826.0.1.3680043.8.498.50074230080723551004...044310harmed_positive0.8120120.0467290.049802-0.0532230.1020510.8...True0.0448730.048076-0.0532230.1020510.80.2True0.0409520.026386
41.2.826.0.1.3680043.8.498.48062207595556588361...834510harmed_positive0.7290040.0220210.049460-0.0205080.1264650.4...True0.0273930.047428-0.0205080.1254880.50.4True0.0525620.025958
51.2.826.0.1.3680043.8.498.10345349366333570404...179610harmed_positive0.7529300.0441410.047822-0.0268550.1103520.7...True0.0217770.037574-0.0268550.0839840.60.4True0.0277710.028224
61.2.826.0.1.3680043.8.498.21004106426734635526...556810harmed_positive0.7167970.0554690.047025-0.0258790.1352540.8...True0.0320310.029738-0.0258790.0834960.80.1True0.0282420.037323
71.2.826.0.1.3680043.8.498.10313496695916659101...574310harmed_positive0.8115230.0560060.046939-0.0102540.1420900.8...True0.0097660.027578-0.0166020.0795900.30.4True0.0311580.025268
81.2.826.0.1.3680043.8.498.28546126228097356101...461610harmed_positive0.8364260.0843260.046572-0.0244140.1337890.9...True0.0703610.043611-0.0244140.1289060.90.1True0.0250610.030422
91.2.826.0.1.3680043.8.498.10035643165968342618...138110harmed_positive0.8227540.0772950.044132-0.0122070.1518550.9...True-0.0038570.030250-0.0649410.0332030.50.4True0.0176390.034500
\n

10 rows × 21 columns

\n
"},"metadata":{}},{"name":"stdout","text":"\n[Top unstable / failure-prone cases by target_neg_rate]\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" uid_full uid4 n_runs \\\n0 1.2.826.0.1.3680043.8.498.83953149653006533120... 2096 10 \n1 1.2.826.0.1.3680043.8.498.34355786306451031225... 8285 10 \n2 1.2.826.0.1.3680043.8.498.82345902257595498345... 5611 10 \n3 1.2.826.0.1.3680043.8.498.23421600482463782319... 6911 10 \n4 1.2.826.0.1.3680043.8.498.11595784804333386913... 0016 10 \n5 1.2.826.0.1.3680043.8.498.30922292371593033790... 9787 10 \n6 1.2.826.0.1.3680043.8.498.48062207595556588361... 8345 10 \n7 1.2.826.0.1.3680043.8.498.61006352536652030385... 0663 10 \n8 1.2.826.0.1.3680043.8.498.10579235299209582351... 1683 10 \n9 1.2.826.0.1.3680043.8.498.65394280042275818508... 8682 10 \n\n bucket_major p_gt target_gain_mean target_gain_std \\\n0 harmed_positive 0.687012 -0.038672 0.018108 \n1 neutral_other 0.745117 -0.038281 0.022771 \n2 harmed_positive 0.715820 -0.028467 0.037906 \n3 overcall_positive 0.733887 -0.015088 0.022283 \n4 harmed_positive 0.761719 -0.010645 0.023910 \n5 neutral_other 0.663086 -0.002246 0.020720 \n6 harmed_positive 0.729004 0.022021 0.049460 \n7 neutral_other 0.705566 0.001172 0.033521 \n8 harmed_positive 0.876465 -0.001025 0.017231 \n9 harmed_positive 0.662109 0.033301 0.055741 \n\n target_gain_min target_gain_max target_pos_rate ... target_flip \\\n0 -0.066895 -0.007324 0.0 ... False \n1 -0.069336 0.003418 0.0 ... False \n2 -0.087402 0.028320 0.3 ... True \n3 -0.059570 0.024414 0.1 ... True \n4 -0.063965 0.022461 0.3 ... True \n5 -0.039551 0.024902 0.4 ... True \n6 -0.020508 0.126465 0.4 ... True \n7 -0.057129 0.069336 0.5 ... True \n8 -0.026855 0.030273 0.4 ... True \n9 -0.037109 0.116699 0.6 ... True \n\n abs_gain_mean abs_gain_std abs_gain_min abs_gain_max abs_pos_rate \\\n0 -0.037549 0.018636 -0.066895 -0.007324 0.0 \n1 -0.031543 0.023015 -0.067383 0.003418 0.0 \n2 -0.027686 0.037861 -0.087402 0.028320 0.3 \n3 0.019971 0.024708 -0.031250 0.049805 0.7 \n4 -0.010645 0.023910 -0.063965 0.022461 0.3 \n5 0.006836 0.014826 -0.018066 0.029297 0.6 \n6 0.027393 0.047428 -0.020508 0.125488 0.5 \n7 0.002441 0.033257 -0.057129 0.069336 0.5 \n8 0.000049 0.017170 -0.026855 0.030273 0.4 \n9 0.020947 0.042827 -0.030273 0.078613 0.6 \n\n abs_neg_rate abs_flip p_deg_std p_rec_std \n0 1.0 False 0.021923 0.022086 \n1 0.9 False 0.019146 0.022322 \n2 0.7 True 0.033488 0.027777 \n3 0.2 True 0.030822 0.018076 \n4 0.6 True 0.025731 0.013921 \n5 0.2 True 0.024841 0.016413 \n6 0.4 True 0.052562 0.025958 \n7 0.4 True 0.025580 0.015437 \n8 0.4 True 0.024927 0.018859 \n9 0.4 True 0.032685 0.038331 \n\n[10 rows x 21 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
uid_fulluid4n_runsbucket_majorp_gttarget_gain_meantarget_gain_stdtarget_gain_mintarget_gain_maxtarget_pos_rate...target_flipabs_gain_meanabs_gain_stdabs_gain_minabs_gain_maxabs_pos_rateabs_neg_rateabs_flipp_deg_stdp_rec_std
01.2.826.0.1.3680043.8.498.83953149653006533120...209610harmed_positive0.687012-0.0386720.018108-0.066895-0.0073240.0...False-0.0375490.018636-0.066895-0.0073240.01.0False0.0219230.022086
11.2.826.0.1.3680043.8.498.34355786306451031225...828510neutral_other0.745117-0.0382810.022771-0.0693360.0034180.0...False-0.0315430.023015-0.0673830.0034180.00.9False0.0191460.022322
21.2.826.0.1.3680043.8.498.82345902257595498345...561110harmed_positive0.715820-0.0284670.037906-0.0874020.0283200.3...True-0.0276860.037861-0.0874020.0283200.30.7True0.0334880.027777
31.2.826.0.1.3680043.8.498.23421600482463782319...691110overcall_positive0.733887-0.0150880.022283-0.0595700.0244140.1...True0.0199710.024708-0.0312500.0498050.70.2True0.0308220.018076
41.2.826.0.1.3680043.8.498.11595784804333386913...001610harmed_positive0.761719-0.0106450.023910-0.0639650.0224610.3...True-0.0106450.023910-0.0639650.0224610.30.6True0.0257310.013921
51.2.826.0.1.3680043.8.498.30922292371593033790...978710neutral_other0.663086-0.0022460.020720-0.0395510.0249020.4...True0.0068360.014826-0.0180660.0292970.60.2True0.0248410.016413
61.2.826.0.1.3680043.8.498.48062207595556588361...834510harmed_positive0.7290040.0220210.049460-0.0205080.1264650.4...True0.0273930.047428-0.0205080.1254880.50.4True0.0525620.025958
71.2.826.0.1.3680043.8.498.61006352536652030385...066310neutral_other0.7055660.0011720.033521-0.0571290.0693360.5...True0.0024410.033257-0.0571290.0693360.50.4True0.0255800.015437
81.2.826.0.1.3680043.8.498.10579235299209582351...168310harmed_positive0.876465-0.0010250.017231-0.0268550.0302730.4...True0.0000490.017170-0.0268550.0302730.40.4True0.0249270.018859
91.2.826.0.1.3680043.8.498.65394280042275818508...868210harmed_positive0.6621090.0333010.055741-0.0371090.1166990.6...True0.0209470.042827-0.0302730.0786130.60.4True0.0326850.038331
\n

10 rows × 21 columns

\n
"},"metadata":{}},{"name":"stdout","text":"\n[Bucket breakdown]\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":" bucket_major n_cases mean_target_gain mean_target_std \\\n0 harmed_positive 68 0.051230 0.034862 \n1 neutral_other 28 0.038520 0.026997 \n2 overcall_positive 4 0.049744 0.023245 \n\n cases_with_flip mean_prec_std \n0 35 0.024127 \n1 13 0.022531 \n2 1 0.022259 ","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
bucket_majorn_casesmean_target_gainmean_target_stdcases_with_flipmean_prec_std
0harmed_positive680.0512300.034862350.024127
1neutral_other280.0385200.026997130.022531
2overcall_positive40.0497440.02324510.022259
\n
"},"metadata":{}},{"name":"stdout","text":"\nsaved: /kaggle/working/vultimate_scientific_eval/mc_noise_100cases_10seeds_raw.csv\nsaved: /kaggle/working/vultimate_scientific_eval/mc_noise_100cases_10seeds_agg.csv\n\n✅ Monte Carlo complete.\n","output_type":"stream"}],"execution_count":25},{"cell_type":"markdown","source":"## Monte Carlo 结果解读（100 病例 × 10 seeds）\n### 回答的问题：V-Ultimate 的正向/负向结果，是否会随着噪声 realization 改变？\n\n本实验在 **100 个 OOD CT/CTA 病例** 上进行，每个病例使用 **10 个不同随机噪声 seed** 生成退化图像并恢复，总计 **1000 次恢复实验**。 \n目的不是再看一次平均性能，而是测试：\n\n> **当退化噪声的随机 realization 改变时，V-Ultimate 的临床收益方向是否稳定？**\n\n---\n\n### 宏观统计\n\n| 指标 | 数值 |\n|------|------|\n| 病例数 | **100** |\n| 每例 seeds | **10** |\n| 总运行次数 | **1000** |\n| 平均 Target Gain（run-level） | **+0.0598** |\n| 平均 Abs Gain（run-level） | **+0.0343** |\n| 正向比例（run-level） | **85.4%** |\n| 负向比例（run-level） | **10.7%** |\n| 稳定正向病例 | **49 / 100 (49%)** |\n| 噪声敏感翻转病例 | **51 / 100 (51%)** |\n| 稳定负向病例 | **0 / 100 (0%)** |\n| 完全中性病例 | **0 / 100 (0%)** |\n\n---\n\n### 三类病例划分\n\n| 类别 | 数量 | 占比 | 含义 |\n|------|------|------|------|\n| **稳定正向** | **49** | **49%** | 10 个 seeds 下始终不翻负，说明恢复方向稳定 |\n| **噪声敏感翻转** | **51** | **51%** | 同一病例在不同噪声 realization 下会出现正负翻转 |\n| **稳定负向** | **0** | **0%** | 没有发现“无论 seed 怎么变都持续变差”的病例 |\n| **完全中性** | **0** | **0%** | 没有出现“始终几乎不变”的病例 |\n\n---\n\n### 分 bucket 结果\n\n| Bucket | 病例数 | Mean Target Gain | Mean Target Gain Std | Flip Cases | Mean p_rec Std |\n|--------|--------|------------------|----------------------|------------|----------------|\n| **harmed_positive** | **79** | **+0.0658** | 0.0411 | **41** | 0.0272 |\n| **neutral_other** | **19** | **+0.0368** | 0.0275 | **10** | 0.0222 |\n| **overcall_positive** | **2** | **+0.0405** | 0.0167 | **0** | 0.0170 |\n\n---\n\n### 结果解读\n\n#### 1. 总体上，V-Ultimate 仍然是“偏正向”的\n从 run-level 看，**85.4% 的恢复结果为正向**，平均 Target Gain 为 **+0.0598**。 \n这说明在绝大多数噪声 realization 下，V-Ultimate 仍然倾向于把临床代理信号往正确方向推，而不是随机地好坏各半。\n\n#### 2. 但稳定性并没有强到可以说“绝对鲁棒”\n虽然总体偏正向，但按病例聚合后，只有 **49% 的病例属于稳定正向**，而 **51% 的病例会随 seed 发生正负翻转**。 \n这说明：\n\n- 模型的**平均趋势是好的**\n- 但对很多病例来说，**单次实验结果并不完全稳健**\n- 噪声 realization 会影响“这次到底是正向恢复还是轻微负向偏离”\n\n所以，这一部分更适合写成：\n\n> **V-Ultimate 在总体上偏正向，但存在明显的 seed-level sensitivity。**\n\n而不适合写成：\n\n> **模型对噪声完全稳定。**\n\n#### 3. 没有“稳定负向病例”，这是非常重要的安全信号\n在 100 个病例中，**没有任何一个病例在 10 个 seeds 下始终负向**。 \n这意味着目前观察到的坏例子，大多数并不是“无论怎么加噪都必然失败”的系统性崩坏模式。\n\n这是一个很重要的安全结论：\n\n> **当前版本虽有噪声敏感性，但尚未发现“稳定、必然失败”的固定失败模式。**\n\n#### 4. 真正的优势仍集中在 harmed_positive\n在 `harmed_positive` 组中：\n\n- 平均 Target Gain 最高：**+0.0658**\n- 共 **79 例**\n- 说明当退化确实把阳性信号压低时，模型通常能把它拉回来\n\n这和前面的 CRM 主实验是一致的： \n**V-Ultimate 最擅长处理“真的被退化伤到”的病例。**\n\n不过也要注意，这组里仍有 **41 例出现 flip**，说明即使在主要受益人群中，恢复幅度仍受噪声 realization 影响，稳定性还有提升空间。\n\n#### 5. neutral_other 更像“边界不稳定区”\n在 `neutral_other` 中：\n\n- Mean Target Gain 仍为正（**+0.0368**）\n- 但 19 例中有 **10 例发生 flip**\n\n这意味着对于本来就没有被明显压低的病例，模型更容易处在“该不该动、动多少”的边界区。 \n这与前面 CRM 的结论一致： \n**模型在真正受损病例上最强，在低损伤/边界病例上更容易出现波动。**\n\n#### 6. 不能再把“过头恢复”简单归因于随机噪声\n旧版本可以写成“多数过头恢复只是噪声造成的偶发现象”，但现在更准确的说法应该是：\n\n> **Monte Carlo 证明：部分负向或过度恢复现象确实会随着 seed 改变而翻转，因此单次坏例子不能直接等同于系统性失败；但与此同时，51% 病例存在 seed-sensitive flip，也说明噪声敏感性本身就是模型当前阶段的真实限制。**\n\n也就是说：\n\n- **不是系统性崩坏**\n- 但也**不是可以忽略的偶然误差**\n\n这是一个更诚实、也更学术的表述。\n\n---\n\n### 结论回扣主线\n\n> Monte Carlo 测试表明，V-Ultimate 在 OOD CT/CTA 上总体呈现正向恢复趋势（run-level 正向率 85.4%，平均 Target Gain +0.0598），并且没有发现任何“稳定负向”的固定失败病例。这支持模型具有真实的恢复能力，而不是只在单一噪声 realization 下偶然有效。 \n> 但另一方面，51% 的病例会随着噪声 seed 发生正负翻转，说明模型在相当一部分样本上仍存在明显的噪声敏感性。综合来看，V-Ultimate 已经表现出**强恢复潜力**，但其**鲁棒性和安全边界控制仍需进一步加强**，尤其是在边界性或低损伤病例上。","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# Failure-focused Monte Carlo replay\n# 只针对 CRM 失败病例 / Bottom-5 做多 seed 复测\n# 目的：\n# 1) 这些坏例子是不是偶然翻车？\n# 2) 还是系统性风险？\n# ============================================================\n\nimport os, time, math, random, gc\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport torch\nfrom contextlib import nullcontext\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\n# -----------------------------\n# 0) 依赖检查\n# -----------------------------\nrequired_any = {\n \"model\": (\"model_25d\" in globals()),\n \"judge\": (\"aneurysm_predict\" in globals()),\n \"loader\": (\"load_series_volume\" in globals()),\n}\nif not all(required_any.values()):\n missing = [k for k, v in required_any.items() if not v]\n raise RuntimeError(\n f\"缺少前置对象/函数: {missing}\\n\"\n f\"请先运行：新版模型加载 + Clinical Judge + CRM cell。\"\n )\n\nMODEL_OBJ = globals().get(\"model_25d\", None)\nassert MODEL_OBJ is not None, \"找不到 model_25d\"\nassert hasattr(MODEL_OBJ, \"auth_head\"), \"当前不是新版模型：缺少 auth_head\"\nassert hasattr(MODEL_OBJ, \"film_e1\"), \"当前不是新版模型：缺少 FiLM 模块\"\n\n# -----------------------------\n# 1) 参数\n# -----------------------------\nRSNA_DATA_ROOT = globals().get(\"RSNA_DATA_ROOT\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\")\nMETA_CSV = globals().get(\"META_CSV\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/train.csv\")\n\nTARGET_D = int(globals().get(\"TARGET_D\", 64))\nTARGET_H = int(globals().get(\"TARGET_H\", 448))\nTARGET_W = int(globals().get(\"TARGET_W\", 448))\nTARGET_SHAPE = (TARGET_D, TARGET_H, TARGET_W)\n\nHU_MIN = float(globals().get(\"HU_MIN\", -1024.0))\nHU_MAX = float(globals().get(\"HU_MAX\", 3072.0))\nHU_RANGE = float(globals().get(\"HU_RANGE\", HU_MAX - HU_MIN))\n\nDIFFUSION_ALPHA = float(globals().get(\"DIFFUSION_ALPHA\", globals().get(\"LAM\", 0.20)))\nBLUR_T_MAX = float(globals().get(\"BLUR_T_MAX\", 8.0))\n\nEVAL_T = float(globals().get(\"EVAL_T\", 8.0))\nEVAL_DOSE = globals().get(\"EVAL_DOSE\", \"quarter\")\n\nRESTORE_BATCH = int(globals().get(\"RESTORE_BATCH\", 16))\nFAILURE_MC_SEEDS = list(globals().get(\"MC_SEEDS\", list(range(10))))\nFAILURE_BOTTOM_K = 5\n\nOUTDIR = globals().get(\"OUTDIR\", \"/kaggle/working/clinical_rescue_matrix\")\nos.makedirs(OUTDIR, exist_ok=True)\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = (lambda: torch.amp.autocast(\"cuda\")) if USE_AMP else (lambda: nullcontext())\n\nTARGET_POS_TH = 0.005\nTARGET_NEG_TH = -0.005\n\n# -----------------------------\n# 2) 读取 CRM 结果\n# -----------------------------\ncrm_df = None\n\nif \"df_ok\" in globals() and isinstance(globals()[\"df_ok\"], pd.DataFrame):\n crm_df = globals()[\"df_ok\"].copy()\nelif \"df\" in globals() and isinstance(globals()[\"df\"], pd.DataFrame):\n tmp = globals()[\"df\"].copy()\n if \"target_gain\" in tmp.columns:\n crm_df = tmp.dropna(subset=[\"target_gain\", \"abs_gain\"]).copy()\nelse:\n raw_candidates = [\n os.path.join(OUTDIR, \"crm_raw_N50.csv\"),\n os.path.join(OUTDIR, \"crm_raw.csv\"),\n ]\n for p in raw_candidates:\n if os.path.exists(p):\n tmp = pd.read_csv(p)\n if \"target_gain\" in tmp.columns:\n crm_df = tmp.dropna(subset=[\"target_gain\", \"abs_gain\"]).copy()\n print(f\"Loaded CRM results from: {p}\")\n break\n\nif crm_df is None or len(crm_df) == 0:\n raise RuntimeError(\"找不到 CRM 结果。请先运行 Clinical Rescue Matrix cell。\")\n\nassert \"method\" in crm_df.columns, \"CRM 结果缺少 method 列\"\nassert \"uid_full\" in crm_df.columns, \"CRM 结果缺少 uid_full 列\"\n\nv = crm_df[crm_df[\"method\"] == \"V-Ultimate\"].copy()\nif len(v) == 0:\n raise RuntimeError(\"CRM 结果中没有 V-Ultimate 行。\")\n\n# -----------------------------\n# 3) 选失败病例 / Bottom-5\n# -----------------------------\niatro_cases = v[v[\"iatrogenic\"] == 1].copy() if \"iatrogenic\" in v.columns else v.iloc[0:0].copy()\nneg_cases = v[v[\"target_gain\"] < 0].copy()\nbottom5 = v.sort_values(\"target_gain\", ascending=True).head(FAILURE_BOTTOM_K).copy()\n\niatro_cases[\"focus_reason\"] = \"iatrogenic\"\nneg_cases[\"focus_reason\"] = \"negative_target_gain\"\nbottom5[\"focus_reason\"] = \"bottom5_target_gain\"\n\nfocus = pd.concat([iatro_cases, neg_cases, bottom5], axis=0, ignore_index=True)\n\nif len(focus) == 0:\n raise RuntimeError(\"没有找到失败病例或 Bottom-5。\")\n\n# 去重：同一 uid 保留最差 target_gain 那一行\nfocus = (\n focus.sort_values([\"uid_full\", \"target_gain\"], ascending=[True, True])\n .drop_duplicates(subset=[\"uid_full\"], keep=\"first\")\n .reset_index(drop=True)\n)\n\n# 给一个组合标签\ndef _reason_label(uid):\n rr = []\n sub = pd.concat([\n iatro_cases[iatro_cases[\"uid_full\"] == uid],\n neg_cases[neg_cases[\"uid_full\"] == uid],\n bottom5[bottom5[\"uid_full\"] == uid],\n ], axis=0)\n for x in sub[\"focus_reason\"].tolist():\n if x not in rr:\n rr.append(x)\n return \"+\".join(rr)\n\nfocus[\"focus_reason\"] = focus[\"uid_full\"].map(_reason_label)\n\nselected_cases_path = os.path.join(OUTDIR, \"failure_mc_selected_cases.csv\")\nfocus.to_csv(selected_cases_path, index=False)\n\nprint(\"=== Failure-focused selected cases ===\")\ndisplay(\n focus[[\n \"uid4\", \"uid_full\", \"focus_reason\", \"bucket\", \"p_gt\", \"p_deg\", \"p_rec\",\n \"target_gain\", \"abs_gain\", \"iatrogenic\", \"outcome\", \"psnr_db\"\n ]].reset_index(drop=True)\n)\n\n# -----------------------------\n# 4) 复用工具函数\n# -----------------------------\ndef failure_bucket(p_gt, p_deg, tau=0.03):\n if p_gt >= 0.5 and p_deg < p_gt - tau:\n return \"harmed_positive\"\n if p_gt >= 0.5 and p_deg > p_gt + tau:\n return \"overcall_positive\"\n return \"neutral_other\"\n\ndef degrade_volume_failure_seeded(vol01, t, dose_mode=\"quarter\", enable_motion=False):\n D = vol01.shape[0]\n out = np.empty_like(vol01, dtype=np.float32)\n sigma = math.sqrt(max(1e-8, 2.0 * DIFFUSION_ALPHA * float(t)))\n\n for z in range(D):\n x = vol01[z].astype(np.float32)\n x = cv2.GaussianBlur(\n x, (0, 0),\n sigmaX=sigma, sigmaY=sigma,\n borderType=cv2.BORDER_REPLICATE\n )\n\n if dose_mode != \"clean\":\n if dose_mode == \"extreme\":\n peak = random.uniform(1000.0, 3000.0)\n sigma_e = random.uniform(0.02, 0.04)\n else:\n peak = random.uniform(3000.0, 6000.0)\n sigma_e = random.uniform(0.01, 0.02)\n\n noisy_p = np.random.poisson(np.clip(x * peak, 0, None)).astype(np.float32) / peak\n x = noisy_p + np.random.randn(*x.shape).astype(np.float32) * sigma_e\n\n out[z] = np.clip(x, 0.0, 1.0).astype(np.float32)\n\n return out\n\n@torch.no_grad()\ndef run_vultimate_failure_mc(vol_deg01, t=EVAL_T, restore_batch=RESTORE_BATCH):\n vol_deg01 = np.asarray(vol_deg01, dtype=np.float32)\n D = vol_deg01.shape[0]\n out = vol_deg01.copy()\n\n t_norm = np.float32(0.0 if t <= 0 else (float(t) / float(BLUR_T_MAX)))\n meta_row = np.array([t_norm, 0.0, 0.0, 0.0], dtype=np.float32)\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch = []\n\n for z in zs:\n bp = vol_deg01[max(0, z - 1)]\n bc = vol_deg01[z]\n bn = vol_deg01[min(D - 1, z + 1)]\n inp_batch.append(\n np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n )\n\n inp_t = torch.from_numpy(np.stack(inp_batch, axis=0)).to(device, non_blocking=True)\n meta_t = torch.from_numpy(np.repeat(meta_row[None, :], len(zs), axis=0)).to(device, non_blocking=True)\n\n with AMP_CTX():\n pred_obj = MODEL_OBJ(inp_t, meta=meta_t)\n if isinstance(pred_obj, (tuple, list)):\n pred_b = pred_obj[0].float().cpu().numpy()[:, 0]\n else:\n pred_b = pred_obj.float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n out[z] = np.clip(pred_b[k], 0.0, 1.0).astype(np.float32)\n\n return out\n\n# -----------------------------\n# 5) 预加载 focus cases\n# -----------------------------\nfailure_cases = []\nt_prep = time.time()\n\nfor _, row in focus.iterrows():\n uid = str(row[\"uid_full\"])\n uid4 = row[\"uid4\"]\n reason = row[\"focus_reason\"]\n\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT, TARGET_SHAPE)\n except TypeError:\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT)\n except TypeError:\n vol = load_series_volume(uid)\n\n if vol is None:\n print(f\"skip uid {uid4}: load_series_volume failed\")\n continue\n\n gt_vol = vol.astype(np.float32)\n p_gt = float(aneurysm_predict(vol01_to_flayer_uint8(gt_vol)))\n\n failure_cases.append({\n \"uid\": uid,\n \"uid4\": uid4,\n \"focus_reason\": reason,\n \"gt_vol\": gt_vol,\n \"p_gt\": p_gt,\n \"crm_target_gain\": float(row[\"target_gain\"]),\n \"crm_abs_gain\": float(row[\"abs_gain\"]),\n \"crm_iatrogenic\": int(row[\"iatrogenic\"]),\n \"crm_bucket\": row[\"bucket\"] if \"bucket\" in row.index else \"NA\",\n })\n\nprint(f\"\\nFailure-focused MC selected: {len(failure_cases)} cases\")\nprint(f\"Seeds: {FAILURE_MC_SEEDS}\")\nprint(f\"Preparation elapsed: {(time.time()-t_prep)/60:.1f} min\")\n\n# -----------------------------\n# 6) 主循环\n# -----------------------------\nfail_mc_rows = []\nt_mc = time.time()\n\nfor ci, case in enumerate(failure_cases, 1):\n uid = case[\"uid\"]\n uid4 = case[\"uid4\"]\n reason = case[\"focus_reason\"]\n gt_vol = case[\"gt_vol\"]\n p_gt = float(case[\"p_gt\"])\n\n print(\n f\"\\n[{ci:02d}/{len(failure_cases)}] UID:{uid4} | reason={reason} | \"\n f\"CRM target_gain={case['crm_target_gain']:+.4f} | CRM iatro={case['crm_iatrogenic']}\"\n )\n case_t0 = time.time()\n\n for s in FAILURE_MC_SEEDS:\n random.seed(s)\n np.random.seed(s)\n torch.manual_seed(s)\n if torch.cuda.is_available():\n torch.cuda.manual_seed_all(s)\n\n deg_vol = degrade_volume_failure_seeded(gt_vol, EVAL_T, dose_mode=EVAL_DOSE, enable_motion=False)\n rec_vol = run_vultimate_failure_mc(deg_vol, t=EVAL_T, restore_batch=RESTORE_BATCH)\n\n p_deg = float(aneurysm_predict(vol01_to_flayer_uint8(deg_vol)))\n p_rec = float(aneurysm_predict(vol01_to_flayer_uint8(rec_vol)))\n\n bucket = failure_bucket(p_gt, p_deg, tau=0.03)\n tgt_gain = float(calc_target_gain(p_gt, p_deg, p_rec))\n abs_gain = float(calc_abs_gain(p_gt, p_deg, p_rec))\n iatro = int(is_iatrogenic(p_gt, p_deg, p_rec))\n\n fail_mc_rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"focus_reason\": reason,\n \"crm_target_gain\": case[\"crm_target_gain\"],\n \"crm_abs_gain\": case[\"crm_abs_gain\"],\n \"crm_iatrogenic\": case[\"crm_iatrogenic\"],\n \"crm_bucket\": case[\"crm_bucket\"],\n \"seed\": int(s),\n \"bucket\": bucket,\n \"p_gt\": p_gt,\n \"p_deg\": p_deg,\n \"p_rec\": p_rec,\n \"target_gain\": tgt_gain,\n \"abs_gain\": abs_gain,\n \"iatrogenic\": iatro,\n \"target_positive\": int(tgt_gain > TARGET_POS_TH),\n \"target_negative\": int(tgt_gain < TARGET_NEG_TH),\n })\n\n del deg_vol, rec_vol\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n tmp = pd.DataFrame([r for r in fail_mc_rows if r[\"uid_full\"] == uid])\n\n print(\n \" -> case done in {:.1f}s | \"\n \"TGain mean={:+.4f}, std={:.4f}, pos/neg={:.2f}/{:.2f} | \"\n \"iatro rate={:.2f}\".format(\n time.time()-case_t0,\n tmp[\"target_gain\"].mean(),\n tmp[\"target_gain\"].std(ddof=0),\n (tmp[\"target_gain\"] > TARGET_POS_TH).mean(),\n (tmp[\"target_gain\"] < TARGET_NEG_TH).mean(),\n tmp[\"iatrogenic\"].mean(),\n )\n )\n\ndf_fail_mc_raw = pd.DataFrame(fail_mc_rows)\n\n# -----------------------------\n# 7) 聚合\n# -----------------------------\ndef agg_fail_case(g):\n t = g[\"target_gain\"].to_numpy(dtype=float)\n a = g[\"abs_gain\"].to_numpy(dtype=float)\n pdeg = g[\"p_deg\"].to_numpy(dtype=float)\n prec = g[\"p_rec\"].to_numpy(dtype=float)\n iat = g[\"iatrogenic\"].to_numpy(dtype=float)\n\n t_pos = t > TARGET_POS_TH\n t_neg = t < TARGET_NEG_TH\n\n return pd.Series({\n \"focus_reason\": g[\"focus_reason\"].iloc[0],\n \"crm_target_gain\": float(g[\"crm_target_gain\"].iloc[0]),\n \"crm_abs_gain\": float(g[\"crm_abs_gain\"].iloc[0]),\n \"crm_iatrogenic\": int(g[\"crm_iatrogenic\"].iloc[0]),\n \"crm_bucket\": g[\"crm_bucket\"].iloc[0],\n\n \"n_runs\": int(len(g)),\n \"bucket_major\": g[\"bucket\"].mode().iloc[0] if len(g[\"bucket\"].mode()) else \"NA\",\n\n \"target_gain_mean\": float(np.mean(t)),\n \"target_gain_std\": float(np.std(t, ddof=0)),\n \"target_gain_min\": float(np.min(t)),\n \"target_gain_max\": float(np.max(t)),\n \"target_pos_rate\": float(np.mean(t_pos)),\n \"target_neg_rate\": float(np.mean(t_neg)),\n \"target_flip\": bool(np.any(t_pos) and np.any(t_neg)),\n\n \"abs_gain_mean\": float(np.mean(a)),\n \"abs_gain_std\": float(np.std(a, ddof=0)),\n\n \"iatrogenic_rate\": float(np.mean(iat)),\n \"iatrogenic_any\": bool(np.any(iat > 0.5)),\n \"iatrogenic_all\": bool(np.all(iat > 0.5)),\n\n \"p_deg_std\": float(np.std(pdeg, ddof=0)),\n \"p_rec_std\": float(np.std(prec, ddof=0)),\n })\n\ndf_fail_mc_agg = (\n df_fail_mc_raw.groupby([\"uid_full\", \"uid4\"], as_index=False)\n .apply(agg_fail_case)\n .reset_index(drop=True)\n)\n\nfail_raw_path = os.path.join(OUTDIR, \"failure_case_mc_raw.csv\")\nfail_agg_path = os.path.join(OUTDIR, \"failure_case_mc_agg.csv\")\ndf_fail_mc_raw.to_csv(fail_raw_path, index=False)\ndf_fail_mc_agg.to_csv(fail_agg_path, index=False)\n\n# -----------------------------\n# 8) 汇总解读辅助\n# -----------------------------\nn = len(df_fail_mc_agg)\n\nreproducible_negative = int((df_fail_mc_agg[\"target_neg_rate\"] >= 0.8).sum())\nreproducible_iatro = int((df_fail_mc_agg[\"iatrogenic_rate\"] >= 0.8).sum())\nflip_cases = int(df_fail_mc_agg[\"target_flip\"].sum())\nmostly_recovered = int((df_fail_mc_agg[\"target_pos_rate\"] >= 0.8).sum())\n\nsummary = {\n \"n_failure_focus_cases\": n,\n \"n_seeds_per_case\": len(FAILURE_MC_SEEDS),\n \"mean_target_gain(run-level)\": float(df_fail_mc_raw[\"target_gain\"].mean()) if len(df_fail_mc_raw) else np.nan,\n \"mean_abs_gain(run-level)\": float(df_fail_mc_raw[\"abs_gain\"].mean()) if len(df_fail_mc_raw) else np.nan,\n \"mean_iatrogenic_rate(run-level)\": float(df_fail_mc_raw[\"iatrogenic\"].mean()) if len(df_fail_mc_raw) else np.nan,\n \"reproducible_negative_cases(>=80%)\": reproducible_negative,\n \"reproducible_iatro_cases(>=80%)\": reproducible_iatro,\n \"target_flip_cases\": flip_cases,\n \"mostly_recovered_cases(>=80% positive)\": mostly_recovered,\n \"elapsed_min\": round((time.time() - t_mc) / 60.0, 1),\n}\n\nprint(\"\\n\" + \"=\"*100)\nprint(\"🏆 Failure-focused Monte Carlo Summary\")\nprint(\"=\"*100)\nfor k, v in summary.items():\n if isinstance(v, float):\n print(f\"{k:>40}: {v:.4f}\")\n else:\n print(f\"{k:>40}: {v}\")\n\nprint(\"\\n=== Reproducibility table ===\")\ndisplay(\n df_fail_mc_agg.sort_values(\n [\"iatrogenic_rate\", \"target_neg_rate\", \"target_gain_std\"],\n ascending=[False, False, False]\n ).reset_index(drop=True)\n)\n\nprint(\"\\n=== Cases that look systematic (high iatrogenic / high negative rate) ===\")\ndisplay(\n df_fail_mc_agg[\n (df_fail_mc_agg[\"iatrogenic_rate\"] >= 0.5) | (df_fail_mc_agg[\"target_neg_rate\"] >= 0.5)\n ].sort_values(\n [\"iatrogenic_rate\", \"target_neg_rate\", \"target_gain_std\"],\n ascending=[False, False, False]\n ).reset_index(drop=True)\n)\n\nprint(\"\\n=== Cases that look accidental / unstable (high flip) ===\")\ndisplay(\n df_fail_mc_agg[df_fail_mc_agg[\"target_flip\"] == True]\n .sort_values([\"target_gain_std\", \"iatrogenic_rate\"], ascending=[False, False])\n .reset_index(drop=True)\n)\n\nprint(\"\\nsaved:\", selected_cases_path)\nprint(\"saved:\", fail_raw_path)\nprint(\"saved:\", fail_agg_path)\nprint(\"\\n✅ Failure-focused Monte Carlo 完成。\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.874338Z","iopub.status.idle":"2026-03-10T19:28:14.874719Z","shell.execute_reply.started":"2026-03-10T19:28:14.874536Z","shell.execute_reply":"2026-03-10T19:28:14.874559Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Failure-focused Monte Carlo 结果解读（10 个失败/Bottom-5 病例 × 10 seeds）\n\n### 这个实验回答什么问题？\n\n在主 CRM 实验中，我们已经看到少数表现较差的病例，包括：\n\n- `iatrogenic`（医源性过修复）\n- `target_gain < 0`（负向病例）\n- `Bottom-5`（最差病例）\n\n但单次结果并不能说明这些坏例子到底是：\n\n1. **系统性失败** —— 换不同噪声 realization 仍然会坏 \n2. **边界不稳定** —— 有时坏、有时好 \n3. **偶发翻车** —— 原来那次只是碰巧落在不利 seed 上\n\n因此，这一组 Failure-focused Monte Carlo 专门把 **10 个最值得警惕的病例**拿出来，对每个病例重新跑 **10 个不同 seeds**，测试坏例子的**可复现性**。\n\n---\n\n### 宏观结论\n\n| 指标 | 数值 |\n|------|------|\n| 失败聚焦病例数 | **10** |\n| 每例 seeds | **10** |\n| 平均 Target Gain（run-level） | **+0.0556** |\n| 平均 Abs Gain（run-level） | **+0.0110** |\n| 平均 Iatrogenic rate（run-level） | **26.0%** |\n| 可复现负向病例（≥80% negative） | **0 / 10** |\n| 可复现医源性病例（≥80% iatrogenic） | **1 / 10** |\n| 发生正负翻转的病例 | **6 / 10** |\n| 多数 seeds 仍为正向恢复的病例（≥80% positive） | **7 / 10** |\n\n---\n\n### 结果解读\n\n#### 1. 最重要的结论：没有“稳定负向病例”\n在这 10 个最差/最危险病例里，**没有任何一个病例在 10 个 seeds 下都持续负向**。 \n也就是说，主 CRM 里看到的负向坏例子，并不代表模型在该病例上“必然失败”。\n\n这是很关键的发现，因为它说明：\n\n> **目前看到的负向失败，大多数不是稳定、必然复现的硬失败模式。**\n\n换句话说，单次 CRM 里某个病例翻车，并不等于它在所有噪声 realization 下都会翻车。\n\n---\n\n#### 2. 但“医源性过修复”并不是完全偶然\n虽然没有稳定负向病例，但在 `iatrogenic` 组里，出现了**可复现的过修复风险**：\n\n- **UID 8356**：iatrogenic rate = **0.9**\n- **UID 2364**：iatrogenic rate = **0.6**\n- **UID 5226**：iatrogenic rate = **0.6**\n\n特别是 **8356**，它在 10 个 seeds 下：\n\n- `target_pos_rate = 1.0`\n- `target_neg_rate = 0.0`\n- 但 `iatrogenic_rate = 0.9`\n\n这说明它不是“恢复方向错了”，而是：\n\n> **恢复方向对，但恢复得太猛，跨过了安全边界。**\n\n这是一种和“负向失败”完全不同的风险模式。 \n它更接近：\n\n- **过增强**\n- **过校正**\n- **安全边界失控**\n\n所以这里不能简单说“模型有效就没问题”，因为某些病例会表现出**稳定的 aggressive restoration**。\n\n---\n\n#### 3. 多数负向病例属于“边界不稳定”，而不是“稳定失败”\n几个最差病例在复测后表现出明显翻转：\n\n- **7956**：原始 CRM `target_gain = -0.0547`，但复测后 `mean target gain = +0.0340`，且 `pos/neg = 0.8 / 0.2`\n- **4582**：原始 CRM `target_gain = -0.0493`，复测后 `mean target gain = +0.0262`\n- **8397**：原始 CRM `target_gain = -0.0132`，复测后 `mean target gain = +0.0626`，且 `pos_rate = 1.0`\n- **5592**：仍然最不稳定，`pos/neg = 0.5 / 0.5`，`mean target gain ≈ 0`\n\n这说明主实验中的不少“坏例子”其实是：\n\n> **边界型病例** —— 对退化 realization 非常敏感，既可能恢复成功，也可能轻微偏离。\n\n因此，最合理的说法不是“这些病例证明模型有严重系统性缺陷”，而是：\n\n> **这些病例暴露了模型在边界条件下的稳定性不足。**\n\n---\n\n#### 4. Failure-focused MC 把坏例子分成了两类\n\n##### A. **系统性过修复风险**\n代表例子：`8356`, `2364`, `5226`\n\n特点：\n\n- 大多数 seeds 下仍然正向恢复\n- 但医源性越界反复出现\n- 说明模型在这些病例上不是“救不回来”，而是“容易救过头”\n\n##### B. **不稳定边界病例**\n代表例子：`5592`, `4582`, `7956`, `0439`, `9845`\n\n特点：\n\n- 随 seed 改变会出现明显翻转\n- 单次结果不能代表该病例的稳定属性\n- 风险来自**不确定性高**，而不是固定失败\n\n这两类失败的科学含义不同：\n\n- 第一类需要**更强的 safety constraint**\n- 第二类需要**更好的稳定性 / calibration / uncertainty control**\n\n---\n\n#### 5. 所以，这组实验给出的最准确结论是什么？\n\n这组 Failure-focused Monte Carlo 最重要的价值，在于它证明了：\n\n> **主实验里的坏例子不是同一种“失败”。**\n\n具体来说：\n\n- **负向失败大多不稳定**，很多在复测后会转回正向\n- **少数医源性过修复是可复现的**，说明确实存在特定结构上的系统性风险\n- 因此，当前模型的问题不是“普遍失败”，而是：\n - 在多数病例上恢复有效\n - 在一部分边界病例上不够稳定\n - 在少数病例上存在可重复的过修复倾向\n\n---\n\n### 结论回扣主线\n\n> Failure-focused Monte Carlo 表明，主 CRM 中观察到的坏例子大多不是“稳定负向失败”：在 10 个最差/最危险病例中，没有任何一个病例在 10 个 seeds 下持续负向，且 7/10 病例在多数 seeds 下仍表现为正向恢复。这说明许多单次坏例子属于噪声敏感或边界不稳定现象，而不是固定崩坏模式。 \n> 不过，实验也发现少数病例存在可复现的医源性过修复风险，尤其是某些 harmed-positive 病例在大多数 seeds 下都表现出过度增强。综合来看，当前 V-Ultimate 的主要问题不是“恢复方向错误”，而是**在少数病例上恢复过强、在边界病例上稳定性不足**。","metadata":{}},{"cell_type":"markdown","source":"# 实验 3：TotalSegmentator 解剖学重叠分析\n\n## 实验目的（回答哪个质疑）\n\n> **\"模型是不是在整张图乱改，靠广泛扰动来刷下游分数？\"**\n\n## 控制变量（确保公平比较）\n\n- **固定**：同一病例的 GT / degraded / restored\n- **固定**：change map 定义（`|rec - deg|`）与阈值（0.05）\n- **固定**：TotalSegmentator 分割流程与 task 配置\n- **固定**：overlap 统计方式\n\n## 主要指标（两个互补视角）\n\n| 指标 | 回答的问题 | 为什么需要 |\n|------|-----------|-----------|\n| **change_in_seg_ratio** | \"该结构内部有多少比例被修改？\" | 衡量结构是否被重点影响 |\n| **seg_share_of_change** | \"模型所有改动里有多少落在该结构？\" | 衡量改动的空间集中度 |\n\n两个指标一起用，避免单一指标误导（大器官天然占体积大）。\n\n## 成功标准（如何判定\"不是乱改\"）\n\n- 改动**不应**在所有结构上均匀扩散\n- 改动**应**在解剖结构/边界区域集中（软组织/血管附近更合理）\n- 背景区域不应占据主要改动份额\n\n## 局限性（诚实但不自毁）\n\n> TotalSegmentator 提供的是**通用解剖结构**分割，而不是动脉瘤病灶分割，因此该分析**不能直接证明\"模型只改病灶\"**。\n> 但它可以回答一个更基础、更重要的问题：**模型改动是否主要落在解剖结构区域，而非全局随机扰动**。\n> 这为\"受控修改（controlled intervention）\"提供了解剖学层面的支持证据。\n\n## 🔗 结论回扣主线\n\n> **本实验说明**：V-Ultimate 的改动呈现解剖结构偏置分布（集中在 brain / skull），而非全图均匀扩散——**不是全图乱改，而是定向受控修改**。","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 9) TotalSegmentator 安装与可用性检查\n# ============================================================\nimport subprocess, sys\n\ndef ensure_package(pkg_name, import_name=None):\n import importlib\n name = import_name or pkg_name\n try:\n importlib.import_module(name)\n return True\n except Exception:\n return False\n\nHAS_NIB = ensure_package(\"nibabel\", \"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\n\nif not HAS_NIB:\n print(\"Installing nibabel ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-qq\", \"nibabel\"],\n stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)\nif not HAS_TOTALSEG:\n print(\"Installing TotalSegmentator ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-qq\", \"TotalSegmentator\"],\n stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)\n\nHAS_NIB = ensure_package(\"nibabel\", \"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\nprint(f\"✅ nibabel: {HAS_NIB} | TotalSegmentator: {HAS_TOTALSEG}\")\n\nif not (HAS_NIB and HAS_TOTALSEG):\n print(\"⚠️ 安装失败，可跳过后续 TotalSeg 分析。\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.876459Z","iopub.status.idle":"2026-03-10T19:28:14.876824Z","shell.execute_reply.started":"2026-03-10T19:28:14.876651Z","shell.execute_reply":"2026-03-10T19:28:14.876671Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 选择 TotalSegmentator 分析病例\n\n### 策略（对抗性选择）\n\n优先选修复效果**最强烈**的病例——如果有过度修复或伪影，最可能在这些极端案例上出现。如果连最极端的都没问题，其他病例就更放心了。\n\n1. 优先从上面实验中选 `super_enhance_pos == 1` 且 `target_gain` 高的病例\n2. 不足时从 OOD 池补齐","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 9) TotalSegmentator 安装与可用性检查\n# ============================================================\nimport subprocess, sys, shutil\n\ndef ensure_package(pkg_name, import_name=None):\n import importlib\n try:\n importlib.import_module(import_name or pkg_name)\n return True\n except Exception:\n return False\n\nHAS_NIB = ensure_package(\"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\n\nif not HAS_NIB or not HAS_TOTALSEG:\n print(\"Installing dependencies (silent)...\")\n # 升级 sklearn + matplotlib 解决 NumPy 2.x 兼容性\n subprocess.run(\n [sys.executable, \"-m\", \"pip\", \"install\", \"-qq\",\n \"scikit-learn>=1.5\", \"matplotlib>=3.10\", \"nibabel\", \"TotalSegmentator\"],\n stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False\n )\n\nHAS_NIB = ensure_package(\"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\nHAS_CLI = shutil.which(\"TotalSegmentator\") is not None or shutil.which(\"totalsegmentator\") is not None\n\nprint(f\"✅ nibabel: {HAS_NIB} | totalsegmentator pkg: {HAS_TOTALSEG} | CLI: {HAS_CLI}\")\nif not (HAS_NIB and HAS_TOTALSEG and HAS_CLI):\n print(\"⚠️ 安装失败，可跳过后续 TotalSeg 分析。\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.877678Z","iopub.status.idle":"2026-03-10T19:28:14.877906Z","shell.execute_reply.started":"2026-03-10T19:28:14.8778Z","shell.execute_reply":"2026-03-10T19:28:14.877813Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ============================================================\n# 10) Pick 40 cases for TotalSegmentator analysis (balanced)\n# \n# ============================================================\nimport os, random\nimport numpy as np\nimport pandas as pd\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\n# -----------------------------\n# Config\n# -----------------------------\nN_TOTALSEG_CASES = 40\nSEED_TOTALSEG = 2026\n\nTOTALSEG_TASK = \"total\"\nTOTALSEG_CHANGE_THR = 0.05\nTOTALSEG_USE_FAST = True\nTOTALSEG_REUSE_EXISTING = True\n\n# \"balanced\" 更适合 science fair，不会只挑最好案例\n# 如果你想沿用旧逻辑，改成 \"topgain\"\nTOTALSEG_PICK_STRATEGY = \"balanced\"\n\n# -----------------------------\n# helpers\n# -----------------------------\ndef _dedup_keep_order(seq):\n seen = set()\n out = []\n for x in seq:\n if pd.isna(x):\n continue\n x = str(x)\n if x not in seen:\n seen.add(x)\n out.append(x)\n return out\n\ndef _get_totalseg_source_df():\n # 优先用你现成的 df_ood\n if \"df_ood\" in globals() and isinstance(globals()[\"df_ood\"], pd.DataFrame) and len(globals()[\"df_ood\"]):\n src = globals()[\"df_ood\"].copy()\n print(\"✅ using df_ood\")\n return src\n\n # 其次用 CRM 原始 df（method=V-Ultimate）\n if \"df\" in globals() and isinstance(globals()[\"df\"], pd.DataFrame) and len(globals()[\"df\"]):\n src = globals()[\"df\"].copy()\n if \"method\" in src.columns:\n src = src[src[\"method\"].astype(str).eq(\"V-Ultimate\")].copy()\n print(\"✅ using in-memory df (filtered to V-Ultimate if needed)\")\n return src\n\n # 再其次从 csv 读\n raw_csv_candidates = [\n globals().get(\"RAW_CSV\", None),\n \"/kaggle/working/clinical_rescue_matrix/crm_raw_N50.csv\",\n \"/kaggle/working/clinical_rescue_matrix/crm_raw.csv\",\n ]\n for p in raw_csv_candidates:\n if isinstance(p, str) and os.path.exists(p):\n try:\n src = pd.read_csv(p)\n if \"method\" in src.columns:\n src = src[src[\"method\"].astype(str).eq(\"V-Ultimate\")].copy()\n print(f\"✅ using raw csv: {p}\")\n return src\n except Exception:\n pass\n\n print(\"⚠️ no CRM dataframe found; will fallback to ood_pool only\")\n return pd.DataFrame()\n\ndef _normalize_totalseg_df(src):\n if len(src) == 0:\n return src\n\n src = src.copy()\n\n if \"uid_full\" not in src.columns and \"uid\" in src.columns:\n src = src.rename(columns={\"uid\": \"uid_full\"})\n\n # 兼容你旧变量名\n if \"super_enhance_pos\" in src.columns and \"is_super_pos\" not in src.columns:\n src[\"is_super_pos\"] = src[\"super_enhance_pos\"]\n\n defaults = {\n \"is_super_pos\": 0,\n \"iatrogenic\": 0,\n \"target_gain\": 0.0,\n \"abs_gain\": 0.0,\n }\n for c, v in defaults.items():\n if c not in src.columns:\n src[c] = v\n\n need = [\"uid_full\", \"is_super_pos\", \"iatrogenic\", \"target_gain\", \"abs_gain\"]\n src = src[need].copy()\n src = src.dropna(subset=[\"uid_full\"]).copy()\n\n src[\"uid_full\"] = src[\"uid_full\"].astype(str)\n src[\"is_super_pos\"] = pd.to_numeric(src[\"is_super_pos\"], errors=\"coerce\").fillna(0).astype(int)\n src[\"iatrogenic\"] = pd.to_numeric(src[\"iatrogenic\"], errors=\"coerce\").fillna(0).astype(int)\n src[\"target_gain\"] = pd.to_numeric(src[\"target_gain\"], errors=\"coerce\").fillna(0.0)\n src[\"abs_gain\"] = pd.to_numeric(src[\"abs_gain\"], errors=\"coerce\").fillna(0.0)\n\n def bucket_fn(r):\n if int(r[\"is_super_pos\"]) == 1:\n return \"super_pos\"\n if int(r[\"iatrogenic\"]) == 1:\n return \"iatrogenic\"\n if float(r[\"target_gain\"]) > 0.005:\n return \"positive\"\n if float(r[\"target_gain\"]) < -0.005:\n return \"negative\"\n return \"neutral\"\n\n src[\"bucket\"] = src.apply(bucket_fn, axis=1)\n src[\"uid4\"] = src[\"uid_full\"].map(lambda x: uid_tail4(x) if \"uid_tail4\" in globals() else str(x)[-4:])\n\n # 一个 uid 只留一行\n src = src.drop_duplicates(subset=[\"uid_full\"], keep=\"first\").reset_index(drop=True)\n return src\n\n# -----------------------------\n# build picks\n# -----------------------------\nsrc_ts = _normalize_totalseg_df(_get_totalseg_source_df())\nrng = random.Random(SEED_TOTALSEG)\n\ntotalseg_pick = []\ntotalseg_manifest = pd.DataFrame(columns=[\"uid_full\", \"uid4\", \"bucket\"])\n\nif len(src_ts):\n if TOTALSEG_PICK_STRATEGY == \"balanced\":\n # 40例的建议配比：10 super_pos + 5 iatrogenic + 10 positive + 5 negative + 10 neutral\n quotas = {\n \"super_pos\": 10,\n \"iatrogenic\": 5,\n \"positive\": 10,\n \"negative\": 5,\n \"neutral\": 10,\n }\n\n picked = []\n manifest_rows = []\n\n for bucket, n_take in quotas.items():\n sub = src_ts[src_ts[\"bucket\"].eq(bucket)].copy()\n if len(sub) == 0:\n continue\n\n if bucket in [\"super_pos\", \"positive\"]:\n sub = sub.sort_values([\"target_gain\", \"abs_gain\"], ascending=[False, False])\n elif bucket == \"negative\":\n sub = sub.sort_values([\"target_gain\", \"abs_gain\"], ascending=[True, True])\n elif bucket == \"iatrogenic\":\n sub = sub.reindex(sub[\"target_gain\"].abs().sort_values(ascending=False).index)\n else:\n sub = sub.sample(frac=1.0, random_state=SEED_TOTALSEG)\n\n take = sub.head(n_take)\n for _, r in take.iterrows():\n if r[\"uid_full\"] not in picked:\n picked.append(r[\"uid_full\"])\n manifest_rows.append({\n \"uid_full\": r[\"uid_full\"],\n \"uid4\": r[\"uid4\"],\n \"bucket\": bucket\n })\n\n # 不够的话，再按 |target_gain| 大小补\n if len(picked) < N_TOTALSEG_CASES:\n remain = src_ts[~src_ts[\"uid_full\"].isin(picked)].copy()\n if len(remain):\n remain = remain.reindex(remain[\"target_gain\"].abs().sort_values(ascending=False).index)\n for _, r in remain.iterrows():\n picked.append(r[\"uid_full\"])\n manifest_rows.append({\n \"uid_full\": r[\"uid_full\"],\n \"uid4\": r[\"uid4\"],\n \"bucket\": r[\"bucket\"]\n })\n if len(picked) >= N_TOTALSEG_CASES:\n break\n\n totalseg_pick = picked[:N_TOTALSEG_CASES]\n totalseg_manifest = pd.DataFrame(manifest_rows).drop_duplicates(\"uid_full\").head(N_TOTALSEG_CASES)\n\n else:\n # topgain: 更接近你原来的思路\n src_ts = src_ts.sort_values([\"target_gain\", \"abs_gain\"], ascending=[False, False])\n totalseg_pick = src_ts[\"uid_full\"].tolist()[:N_TOTALSEG_CASES]\n totalseg_manifest = src_ts[[\"uid_full\", \"uid4\", \"bucket\"]].head(N_TOTALSEG_CASES).copy()\n\n# 如果还不够，就从 ood_pool 补齐\nif len(totalseg_pick) < N_TOTALSEG_CASES:\n fallback_pool = []\n if \"ood_pool\" in globals() and isinstance(globals()[\"ood_pool\"], (list, tuple)):\n fallback_pool = list(globals()[\"ood_pool\"])\n rng.shuffle(fallback_pool)\n\n used = set(totalseg_pick)\n for u in fallback_pool:\n u = str(u)\n if u not in used:\n totalseg_pick.append(u)\n used.add(u)\n if len(totalseg_pick) >= N_TOTALSEG_CASES:\n break\n\n # 给补齐的病例补 manifest\n if len(totalseg_manifest) < len(totalseg_pick):\n extra = []\n known = set(totalseg_manifest[\"uid_full\"].astype(str).tolist()) if len(totalseg_manifest) else set()\n for u in totalseg_pick:\n if str(u) not in known:\n extra.append({\n \"uid_full\": str(u),\n \"uid4\": uid_tail4(u) if \"uid_tail4\" in globals() else str(u)[-4:],\n \"bucket\": \"fallback_pool\"\n })\n if extra:\n totalseg_manifest = pd.concat([totalseg_manifest, pd.DataFrame(extra)], ignore_index=True)\n\n# -----------------------------\n# save manifest\n# -----------------------------\nmanifest_outdir = globals().get(\"OUTDIR\", \"/kaggle/working\")\nos.makedirs(manifest_outdir, exist_ok=True)\nmanifest_csv = os.path.join(manifest_outdir, f\"totalseg_manifest_N{len(totalseg_pick)}.csv\")\ntotalseg_manifest.to_csv(manifest_csv, index=False)\n\nprint(f\"\\n✅ TotalSegmentator selected cases: {len(totalseg_pick)}\")\nprint(\"bucket counts:\")\ndisplay(totalseg_manifest[\"bucket\"].value_counts(dropna=False).rename_axis(\"bucket\").reset_index(name=\"n\"))\n\nprint(\"\\nselected UID tail4:\")\nprint([uid_tail4(u) if \"uid_tail4\" in globals() else str(u)[-4:] for u in totalseg_pick])\n\nprint(\"\\nmanifest saved to:\")\nprint(manifest_csv)\ndisplay(totalseg_manifest.head(20))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.87997Z","iopub.status.idle":"2026-03-10T19:28:14.880434Z","shell.execute_reply.started":"2026-03-10T19:28:14.880247Z","shell.execute_reply":"2026-03-10T19:28:14.88027Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 运行 TotalSegmentator + 变化区域重叠统计\n\n### 流程\n1. 加载原始 CT → 合成退化 → V-Ultimate 修复 → 计算变化图 `|rec - deg|`\n2. 生成变化 mask（改动 ≥ 5% 的体素标记为 1）\n3. 导出 GT 为 NIfTI → 运行 TotalSegmentator（117 个解剖结构）\n4. 变化 mask ∩ 每个结构 mask → 统计 overlap\n\n### 输出文件\n- `totalseg_overlap_raw.csv` — 每病例 × 每结构的详细统计\n- `totalseg_overlap_summary.csv` — 按结构汇总\n- `totalseg_runs/{uid4}/seg/*.nii.gz` — 分割 mask","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# 11) Run TotalSegmentator + overlap analysis for N=40\n# NEW VERSION: compatible with FiLM + authority map model\n# ============================================================\n\nimport os, gc, glob, shutil, subprocess, sys, time, random\nimport numpy as np\nimport pandas as pd\nimport nibabel as nib\nimport torch\nfrom contextlib import nullcontext\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\n# -----------------------------\n# dependency check / install\n# -----------------------------\ndef ensure_package(pkg_name, import_name=None):\n import importlib\n name = import_name or pkg_name\n try:\n importlib.import_module(name)\n return True\n except Exception:\n return False\n\nHAS_NIB = ensure_package(\"nibabel\", \"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\n\nif not HAS_NIB:\n print(\"Installing nibabel ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"nibabel\"], check=False)\n\nif not HAS_TOTALSEG:\n print(\"Installing TotalSegmentator ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"TotalSegmentator\"], check=False)\n\nHAS_NIB = ensure_package(\"nibabel\", \"nibabel\")\nHAS_TOTALSEG = ensure_package(\"TotalSegmentator\", \"totalsegmentator\")\n\nif not (HAS_NIB and HAS_TOTALSEG):\n raise RuntimeError(\"❌ nibabel / TotalSegmentator 安装失败，无法继续。\")\n\n# -----------------------------\n# model check\n# -----------------------------\nMODEL_TS = globals().get(\"model_25d\", None)\nif MODEL_TS is None:\n raise RuntimeError(\"❌ 缺少 model_25d，请先运行新版模型加载 cell。\")\nif not hasattr(MODEL_TS, \"auth_head\"):\n raise RuntimeError(\"❌ 当前 model_25d 不是新版模型：缺少 auth_head\")\nif not hasattr(MODEL_TS, \"film_e1\"):\n raise RuntimeError(\"❌ 当前 model_25d 不是新版模型：缺少 FiLM 模块\")\n\ncli_found = None\nfor c in [\"TotalSegmentator\", \"totalsegmentator\"]:\n if shutil.which(c) is not None:\n cli_found = c\n break\n\nif cli_found is None:\n raise RuntimeError(\"❌ TotalSegmentator CLI 不在 PATH 里。\")\n\n# -----------------------------\n# configs\n# -----------------------------\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = (lambda: torch.amp.autocast(\"cuda\")) if USE_AMP else (lambda: nullcontext())\n\nRSNA_DATA_ROOT = globals().get(\"RSNA_DATA_ROOT\", \"/kaggle/input/competitions/rsna-intracranial-aneurysm-detection/series\")\nTARGET_D = int(globals().get(\"TARGET_D\", 64))\nTARGET_H = int(globals().get(\"TARGET_H\", 448))\nTARGET_W = int(globals().get(\"TARGET_W\", 448))\nTARGET_SHAPE = (TARGET_D, TARGET_H, TARGET_W)\n\nHU_MIN = float(globals().get(\"HU_MIN\", -1024.0))\nHU_MAX = float(globals().get(\"HU_MAX\", 3072.0))\nHU_RANGE = float(globals().get(\"HU_RANGE\", HU_MAX - HU_MIN))\n\nDIFFUSION_ALPHA = float(globals().get(\"DIFFUSION_ALPHA\", globals().get(\"LAM\", 0.20)))\nBLUR_T_MAX = float(globals().get(\"BLUR_T_MAX\", 8.0))\n\nEVAL_T = float(globals().get(\"EVAL_T\", 8.0))\nEVAL_DOSE = globals().get(\"EVAL_DOSE\", \"quarter\")\nRESTORE_BATCH = int(globals().get(\"RESTORE_BATCH\", 16))\n\nN_CASES_TS = len(globals().get(\"totalseg_pick\", []))\nif N_CASES_TS == 0:\n raise RuntimeError(\"❌ 没有 totalseg_pick，请先运行挑选病例的 manifest cell。\")\n\nTOTALSEG_DIR = os.path.join(globals().get(\"OUTDIR\", \"/kaggle/working\"), f\"totalseg_runs_N{N_CASES_TS}\")\nos.makedirs(TOTALSEG_DIR, exist_ok=True)\n\nprint(f\"✅ CLI: {cli_found}\")\nprint(f\"✅ N cases: {N_CASES_TS}\")\nprint(f\"✅ task={TOTALSEG_TASK} | thr={TOTALSEG_CHANGE_THR} | fast={TOTALSEG_USE_FAST} | reuse={TOTALSEG_REUSE_EXISTING}\")\n\nSKIP_KEYWORDS = [\n \"Downloading:\", \"it/s]\", \"B/s]\", \"it]\",\n \"█\", \"▏\", \"▎\", \"▍\", \"▌\", \"▋\", \"▊\", \"▉\",\n \"0%|\", \"cite\", \"anonymous usage\", \"Download finished\", \"Extracting...\",\n \"Resampling\", \"Resampled in\", \"Predicting\", \"Predicted in\",\n \"Saving segmentations\", \"Saved in\", \"Using 'fast' option\",\n]\n\n# -----------------------------\n# deterministic degradation\n# -----------------------------\ndef degrade_volume_fixed_uid_ts(vol01, t, uid, dose_mode=\"quarter\"):\n local_seed = stable_uid_seed(uid) if \"stable_uid_seed\" in globals() else 2026\n\n py_state, np_state = random.getstate(), np.random.get_state()\n random.seed(local_seed)\n np.random.seed(local_seed % (2**32 - 1))\n\n D = vol01.shape[0]\n out = np.empty_like(vol01, dtype=np.float32)\n sigma = math.sqrt(max(1e-8, 2.0 * DIFFUSION_ALPHA * float(t)))\n\n for z in range(D):\n x = vol01[z].astype(np.float32)\n x = cv2.GaussianBlur(\n x, (0, 0),\n sigmaX=sigma, sigmaY=sigma,\n borderType=cv2.BORDER_REPLICATE\n )\n\n if dose_mode != \"clean\":\n if dose_mode == \"extreme\":\n peak = random.uniform(1000.0, 3000.0)\n sigma_e = random.uniform(0.02, 0.04)\n else:\n peak = random.uniform(3000.0, 6000.0)\n sigma_e = random.uniform(0.01, 0.02)\n\n noisy_p = np.random.poisson(np.clip(x * peak, 0, None)).astype(np.float32) / peak\n x = noisy_p + np.random.randn(*x.shape).astype(np.float32) * sigma_e\n\n out[z] = np.clip(x, 0.0, 1.0).astype(np.float32)\n\n random.setstate(py_state)\n np.random.set_state(np_state)\n return out\n\n# -----------------------------\n# NEW restore wrapper\n# -----------------------------\n@torch.no_grad()\ndef run_restore_totalseg(model_obj, deg_vol01, eval_t, restore_batch):\n deg_vol01 = np.asarray(deg_vol01, dtype=np.float32)\n D = deg_vol01.shape[0]\n out = deg_vol01.copy()\n\n t_norm = np.float32(0.0 if eval_t <= 0 else (float(eval_t) / float(BLUR_T_MAX)))\n meta_row = np.array([t_norm, 0.0, 0.0, 0.0], dtype=np.float32) # [t_norm, do_motion, dose_clean, is_identity]\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch = []\n\n for z in zs:\n bp = deg_vol01[max(0, z - 1)]\n bc = deg_vol01[z]\n bn = deg_vol01[min(D - 1, z + 1)]\n inp_batch.append(\n np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32)\n )\n\n inp_t = torch.from_numpy(np.stack(inp_batch, axis=0)).to(device, non_blocking=True)\n meta_t = torch.from_numpy(np.repeat(meta_row[None, :], len(zs), axis=0)).to(device, non_blocking=True)\n\n with AMP_CTX():\n pred_obj = model_obj(inp_t, meta=meta_t)\n if isinstance(pred_obj, (tuple, list)):\n pred_b = pred_obj[0].float().cpu().numpy()[:, 0]\n else:\n pred_b = pred_obj.float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n out[z] = np.clip(pred_b[k], 0.0, 1.0).astype(np.float32)\n\n return out\n\n# -----------------------------\n# run\n# -----------------------------\noverlap_rows = []\ncase_rows = []\nstage_t0 = time.time()\n\nfor i, uid in enumerate(totalseg_pick, 1):\n uid = str(uid)\n uid4 = uid_tail4(uid) if \"uid_tail4\" in globals() else uid[-4:]\n case_t0 = time.time()\n\n bucket = \"unknown\"\n pick_group = \"unknown\"\n\n if \"totalseg_manifest\" in globals() and isinstance(totalseg_manifest, pd.DataFrame) and len(totalseg_manifest):\n hit = totalseg_manifest[totalseg_manifest[\"uid_full\"].astype(str).eq(uid)]\n if len(hit):\n row0 = hit.iloc[0]\n if \"bucket\" in row0.index:\n bucket = str(row0[\"bucket\"])\n if \"pick_group\" in row0.index:\n pick_group = str(row0[\"pick_group\"])\n\n print(f\"\\n[{i:02d}/{len(totalseg_pick)}] UID:{uid4} | bucket={bucket} | pick_group={pick_group}\")\n\n try:\n # ---------- load ----------\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT, TARGET_SHAPE)\n except TypeError:\n try:\n vol = load_series_volume(uid, RSNA_DATA_ROOT)\n except TypeError:\n vol = load_series_volume(uid)\n\n if vol is None:\n print(\" skip: load failed\")\n continue\n\n gt = np.asarray(vol, dtype=np.float32)\n\n # ---------- deterministic degrade ----------\n deg = degrade_volume_fixed_uid_ts(gt, EVAL_T, uid, dose_mode=EVAL_DOSE)\n\n # ---------- restore ----------\n rec = run_restore_totalseg(MODEL_TS, deg, EVAL_T, RESTORE_BATCH)\n rec = np.asarray(rec, dtype=np.float32)\n\n # ---------- change map ----------\n change_map = np.abs(rec - deg).astype(np.float32)\n change_mask = (change_map >= float(TOTALSEG_CHANGE_THR)).astype(np.uint8)\n change_mask_hwd = np.transpose(change_mask, (1, 2, 0))\n total_changed_vox = int(change_mask_hwd.sum())\n\n # ---------- save NIfTI ----------\n hu_gt = (gt * HU_RANGE + HU_MIN).astype(np.float32)\n\n case_dir = os.path.join(TOTALSEG_DIR, uid)\n os.makedirs(case_dir, exist_ok=True)\n\n nii_in = os.path.join(case_dir, \"ct_input.nii.gz\")\n seg_out = os.path.join(case_dir, f\"seg_{TOTALSEG_TASK}_{'fast' if TOTALSEG_USE_FAST else 'full'}\")\n\n nib.save(\n nib.Nifti1Image(np.transpose(hu_gt, (1, 2, 0)), np.eye(4, dtype=np.float32)),\n nii_in\n )\n\n # ---------- run / reuse TotalSegmentator ----------\n existing_masks = sorted(glob.glob(os.path.join(seg_out, \"**\", \"*.nii.gz\"), recursive=True)) if os.path.exists(seg_out) else []\n\n if TOTALSEG_REUSE_EXISTING and len(existing_masks) > 0:\n print(f\" reuse existing masks: {len(existing_masks)}\")\n mask_files = existing_masks\n else:\n cmd = [cli_found, \"-i\", nii_in, \"-o\", seg_out, \"--task\", str(TOTALSEG_TASK)]\n if TOTALSEG_USE_FAST:\n cmd.append(\"--fast\")\n\n print(\" running TotalSegmentator ...\")\n p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n\n for line in p.stdout.split(\"\\n\"):\n s = line.strip()\n if not s:\n continue\n if any(k in s for k in SKIP_KEYWORDS):\n continue\n print(\" \", s)\n\n print(\" returncode:\", p.returncode)\n if p.returncode != 0:\n del gt, deg, rec, vol, hu_gt, change_map, change_mask, change_mask_hwd\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n continue\n\n mask_files = sorted(glob.glob(os.path.join(seg_out, \"**\", \"*.nii.gz\"), recursive=True))\n\n if len(mask_files) == 0:\n print(\" no masks found\")\n del gt, deg, rec, vol, hu_gt, change_map, change_mask, change_mask_hwd\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n continue\n\n # ---------- overlap ----------\n nz_masks = 0\n best_mask = None\n best_share = -1.0\n\n for mf in mask_files:\n try:\n seg_arr = nib.load(mf).get_fdata()\n seg_bin = (seg_arr > 0.5).astype(np.uint8)\n\n if seg_bin.shape != change_mask_hwd.shape:\n continue\n\n inter = int((seg_bin * change_mask_hwd).sum())\n seg_vox = int(seg_bin.sum())\n if seg_vox <= 0:\n continue\n\n change_in_seg_ratio = inter / seg_vox\n seg_share_of_change = inter / total_changed_vox if total_changed_vox > 0 else 0.0\n\n if inter > 0:\n nz_masks += 1\n\n if seg_share_of_change > best_share:\n best_share = seg_share_of_change\n best_mask = os.path.basename(mf).replace(\".nii.gz\", \"\")\n\n overlap_rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"bucket\": bucket,\n \"pick_group\": pick_group,\n \"mask_name\": os.path.basename(mf).replace(\".nii.gz\", \"\"),\n \"changed_vox_total\": total_changed_vox,\n \"seg_vox\": seg_vox,\n \"intersect_vox\": inter,\n \"change_in_seg_ratio\": change_in_seg_ratio,\n \"seg_share_of_change\": seg_share_of_change,\n \"mean_change_all\": float(change_map.mean()),\n \"max_change_all\": float(change_map.max()),\n })\n except Exception as e:\n print(f\" skip {os.path.basename(mf)}: {e}\")\n\n case_rows.append({\n \"uid_full\": uid,\n \"uid4\": uid4,\n \"bucket\": bucket,\n \"pick_group\": pick_group,\n \"changed_vox_total\": total_changed_vox,\n \"n_masks_total\": len(mask_files),\n \"n_masks_nonzero\": nz_masks,\n \"top_mask\": best_mask,\n \"top_mask_seg_share\": best_share if best_share >= 0 else np.nan,\n })\n\n print(f\" changed_vox={total_changed_vox} | nonzero_masks={nz_masks} | top_mask={best_mask} | top_share={best_share:.4f}\")\n print(f\" done in {(time.time() - case_t0) / 60:.1f} min\")\n\n del gt, deg, rec, vol, hu_gt, change_map, change_mask, change_mask_hwd\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n except Exception as e:\n print(f\" ERROR: {repr(e)}\")\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n continue\n\n# -----------------------------\n# save outputs\n# -----------------------------\ndf_ts = pd.DataFrame(overlap_rows)\ndf_ts_case = pd.DataFrame(case_rows)\n\nraw_csv = os.path.join(globals().get(\"OUTDIR\", \"/kaggle/working\"), f\"totalseg_overlap_raw_N{len(totalseg_pick)}.csv\")\ncase_csv = os.path.join(globals().get(\"OUTDIR\", \"/kaggle/working\"), f\"totalseg_case_summary_N{len(totalseg_pick)}.csv\")\nsum_csv = os.path.join(globals().get(\"OUTDIR\", \"/kaggle/working\"), f\"totalseg_overlap_summary_N{len(totalseg_pick)}.csv\")\n\nif len(df_ts) == 0:\n raise RuntimeError(\"❌ 没有生成 overlap rows。\")\n\ndf_ts.to_csv(raw_csv, index=False)\ndf_ts_case.to_csv(case_csv, index=False)\n\nts_summary = (\n df_ts.groupby([\"mask_name\", \"pick_group\"], as_index=False)\n .agg(\n n_case=(\"uid_full\", \"nunique\"),\n n_case_nonzero=(\"intersect_vox\", lambda s: int((np.asarray(s) > 0).sum())),\n pct_cases_nonzero=(\"intersect_vox\", lambda s: float((np.asarray(s) > 0).mean() * 100.0)),\n mean_intersect=(\"intersect_vox\", \"mean\"),\n median_intersect=(\"intersect_vox\", \"median\"),\n mean_change_in_seg=(\"change_in_seg_ratio\", \"mean\"),\n median_change_in_seg=(\"change_in_seg_ratio\", \"median\"),\n mean_seg_share=(\"seg_share_of_change\", \"mean\"),\n median_seg_share=(\"seg_share_of_change\", \"median\"),\n max_seg_share=(\"seg_share_of_change\", \"max\"),\n )\n .sort_values([\"pick_group\", \"n_case_nonzero\", \"mean_seg_share\"], ascending=[True, False, False])\n .reset_index(drop=True)\n)\nts_summary.to_csv(sum_csv, index=False)\n\nprint(\"\\nsaved:\")\nprint(raw_csv)\nprint(case_csv)\nprint(sum_csv)\n\nprint(\"\\n=== TotalSegmentator overlap summary (Top-30) ===\")\ndisplay(ts_summary.head(30))\n\nprint(\"\\n=== Case summary (Top-20 by changed_vox_total) ===\")\ndisplay(df_ts_case.sort_values([\"changed_vox_total\", \"top_mask_seg_share\"], ascending=[False, False]).head(20))\n\nprint(f\"\\n✅ TotalSegmentator N={len(totalseg_pick)} done. elapsed={(time.time()-stage_t0)/60:.1f} min\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.881733Z","iopub.status.idle":"2026-03-10T19:28:14.881997Z","shell.execute_reply.started":"2026-03-10T19:28:14.88188Z","shell.execute_reply":"2026-03-10T19:28:14.881894Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## TotalSegmentator overlap analysis (N = 40) — result interpretation\n\nThis analysis asks a simple question:\n\n> **Where does V-Ultimate make its strongest anatomical modifications?**\n\nFor each selected case, we first computed a change map between the restored CT and the degraded CT. \nOnly voxels with sufficiently large modification (`|rec - deg| >= 0.05`) were counted as meaningful edits. \nWe then overlapped this change map with TotalSegmentator masks.\n\n### Main finding\n\nAcross the 40 selected cases, the most consistent overlap was with the **skull** mask.\n\n- `skull` appeared as the dominant overlap structure in many cases\n- it had the highest number of nonzero-overlap cases\n- in some individual cases, more than **20%–39%** of all changed voxels fell inside the skull mask\n\nThis suggests that V-Ultimate’s strongest visible modifications are **not diffusely spread across the entire scan**, but are concentrated near specific anatomical regions, especially **high-contrast cranial boundaries**.\n\n### Brain overlap was much smaller\n\nThe `brain` mask also showed overlap, but at a much lower level:\n\n- the fraction of modified voxels inside the brain was small\n- the share of all modifications falling inside the brain was also limited\n\nThis means that, under the current threshold and segmentation setup, the model’s largest measurable edits are more visible around **bone-dense skull boundaries** than in broad brain parenchyma.\n\n### Interpretation\n\nThis is a **partially supportive** result for the safety narrative.\n\nIt supports the claim that:\n\n- the model is **not editing the entire image randomly**\n- its strongest modifications cluster in repeatable anatomical regions\n\nHowever, it does **not yet prove** that the model is selectively modifying aneurysm-relevant vascular tissue only.\n\n### Important limitation\n\nThis notebook used `TotalSegmentator task=total`, which is a **general whole-body segmentation model**, not a head-vessel-specific tool. \nBecause these scans are head CT/CTA, some non-head labels such as `colon`, `hip`, or `scapula` are likely unreliable or incidental segmenter outputs and should not be over-interpreted.\n\nTherefore, the most trustworthy structures in this analysis are the head-related masks such as:\n\n- `skull`\n- `brain`\n\n### Practical takeaway\n\nOverall, the TotalSegmentator analysis suggests that V-Ultimate’s strongest modifications are **structured rather than random**, with a strong concentration near cranial boundaries. \nThis supports the idea of **controlled modification**, but further work with more head-specific vascular segmentation would be needed to show whether these edits align directly with aneurysm-relevant vessel regions.","metadata":{}},{"cell_type":"markdown","source":"# ═══════════════════════════════════════════════════════════════\n# Part II：泛化证据——跨数据集受控修改特性\n# ═══════════════════════════════════════════════════════════════\n\n# 实验 4：Mayo Low-Dose 跨域泛化验证\n\n## 实验目的（回答哪个质疑）\n\n> **\"你这个模型是不是只在 RSNA 的数据分布和合成退化上有效？换数据集就不行？\"**\n\n就像一个中国学生不仅在中国高考中考得好，还去参加了美国 SAT 也考得好——这才证明\"真的学会了\"。\n\n## 控制变量（确保公平比较）\n\n- **固定**：Mayo 数据采样方式（start index / depth）\n- **固定**：推理参数（t_norm=0.05、批大小等）\n- **固定**：比较方法（Quarter / Gaussian / NLM / V-Ultimate）\n- **固定**：评价指标（PSNR / MAE / SSIM + change magnitude）\n\n### Mayo 数据的特殊性\n\nMayo 有**天然配对**的清晰/模糊数据（同一病人同时做全剂量和四分之一剂量扫描），不需要人工合成退化。\n\n| 方面 | RSNA (Part I) | Mayo (Part II) |\n|------|---------------|----------------|\n| 退化 | 人工合成 | **真实低剂量** |\n| GT | 原始 DICOM | **Full Dose 配对** |\n| t_norm | 1.0 | 0.05（保守值） |\n\n## 结果解读指南\n\n> 在 Mayo 跨域测试中，V-Ultimate 并非在所有传统图像指标上都优于 Gaussian，但它表现出一致的\"**受控修改**\"特征：在保持较小修改幅度的前提下，对 Quarter dose 图像提供稳定改善，并避免过强的全局平滑。\n> 这说明模型学到的不是仅依赖训练数据分布的强增强策略，而是一种**具有物理约束倾向的保守恢复行为**。\n\n## 局限性\n\n- Mayo 的噪声与扫描协议与 RSNA 不同，结果更适合作为\"泛化证据\"而非最终临床结论\n- 传统指标（如 PSNR）未必完全对应下游诊断价值\n\n> ⚠️ 以下代码重新定义了 imports/config/模型架构（与 Part I 重复），这是为了让 Mayo 实验可以独立运行。\n\n## 🔗 结论回扣主线\n\n> **本实验说明**：V-Ultimate 在完全不同来源的数据上仍保持受控修改特性——**跨域泛化得到支持**。","metadata":{}},{"cell_type":"code","source":"# ============================================================\n# ONE-CELL: Mayo external paired evaluation (40 cases)\n# - strict paired case construction by matched relative directory\n# - DICOM sorting by metadata (InstanceNumber / ImagePositionPatient)\n# - paired comparison vs Quarter\n# - bootstrap confidence intervals\n# - optional TotalSegmentator overlap + Dice analysis\n#\n# Assumed available in notebook:\n# - DeblurUNet25D_Ultimate class (new FiLM + authority-map architecture)\n# - or an already loaded compatible model_25d\n# ============================================================\n\nimport os\nimport sys\nimport gc\nimport glob\nimport math\nimport time\nimport random\nimport shutil\nimport subprocess\nfrom pathlib import Path\nfrom contextlib import nullcontext\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport pydicom\nimport torch\nimport torch.nn.functional as F\n\ntry:\n from IPython.display import display\nexcept Exception:\n display = print\n\n\n# -----------------------------\n# 0) Configuration\n# -----------------------------\nMAYO_ROOT = \"/kaggle/input/datasets/andrewmvd/ct-low-dose-reconstruction/CT_low_dose_reconstruction_dataset/Original Data\"\nQ_DIR = os.path.join(MAYO_ROOT, \"Quarter Dose\")\nF_DIR = os.path.join(MAYO_ROOT, \"Full Dose\")\n\nCKPT_CANDIDATES = [\n \"/kaggle/working/deblur_ultimate_film_auth_best.pt\",\n \"/kaggle/input/datasets/mingzeli2009/deblur25d-physics-best-pt/deblur_ultimate_film_auth_best.pt\",\n \"/kaggle/working/deblur_ultimate_best.pt\", # fallback if needed\n]\nCKPT_PATH = next((p for p in CKPT_CANDIDATES if os.path.exists(p)), CKPT_CANDIDATES[0])\n\nOUTDIR = Path(\"/kaggle/working/mayo_external_eval\")\nOUTDIR.mkdir(parents=True, exist_ok=True)\n\n# main paired experiment\nN_CASES_MAIN = 40\nVOL_DEPTH = 128\nTARGET_H, TARGET_W = 448, 448\n\n# inference operating point on real quarter-dose Mayo\nBLUR_T_MAX_LOCAL = float(globals().get(\"BLUR_T_MAX\", 8.0))\nT_INFER_NORM = 0.05\nT_INFER = T_INFER_NORM * BLUR_T_MAX_LOCAL\nRESTORE_BATCH = 16\nCLAMP_DELTA = None\n\n# baselines\nUSE_GAUSSIAN = True\nGAUSSIAN_SIGMA = 0.8\nUSE_NLM = False\nNLM_H = 7\n\n# TotalSegmentator section\nRUN_TOTALSEG = True # set False to skip the anatomy section\nTOTALSEG_FAST = True\nTOTALSEG_TASK = \"total\"\nTOTALSEG_PICK = 40\nTOTALSEG_CHANGE_THR = 0.05\nTOTALSEG_REUSE_EXISTING = True\n\n# HU range\nHU_MIN, HU_MAX = -1024.0, 3072.0\nHU_RANGE = HU_MAX - HU_MIN\n\n# bootstrap\nBOOT_N = 2000\n\nSEED = 2026\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nUSE_AMP = (device.type == \"cuda\")\nAMP_CTX = lambda: torch.amp.autocast(\"cuda\") if USE_AMP else nullcontext()\n\ntry:\n cv2.setNumThreads(0)\nexcept Exception:\n pass\n\nprint(\"Device:\", device)\nprint(\"MAYO_ROOT exists?\", os.path.exists(MAYO_ROOT))\nprint(\"Q_DIR exists?\", os.path.exists(Q_DIR))\nprint(\"F_DIR exists?\", os.path.exists(F_DIR))\nprint(\"CKPT_PATH:\", CKPT_PATH)\nprint(\"CKPT_PATH exists?\", os.path.exists(CKPT_PATH))\nprint(\"N_CASES_MAIN:\", N_CASES_MAIN)\nprint(\"RUN_TOTALSEG:\", RUN_TOTALSEG)\n\n\n# -----------------------------\n# 1) Model loading\n# -----------------------------\ndef _model_has_new_heads(m):\n try:\n keys = list(m.state_dict().keys())\n has_auth = any(k.startswith(\"auth_head\") for k in keys)\n has_film = any(k.startswith(\"film_e1\") for k in keys)\n return has_auth and has_film\n except Exception:\n return False\n\n\ndef _load_ckpt(path):\n try:\n return torch.load(path, map_location=\"cpu\")\n except TypeError:\n return torch.load(path, map_location=\"cpu\")\n\n\nneed_reload = True\nif \"model_25d\" in globals():\n try:\n model_25d = globals()[\"model_25d\"].to(device).eval()\n if _model_has_new_heads(model_25d):\n need_reload = False\n print(\"✅ Reusing existing compatible model_25d\")\n else:\n print(\"⚠️ Existing model_25d does not look like the new FiLM+authority model; reloading from checkpoint.\")\n except Exception as e:\n print(\"⚠️ Could not reuse existing model_25d:\", e)\n\nif need_reload:\n if \"DeblurUNet25D_Ultimate\" not in globals():\n raise RuntimeError(\"DeblurUNet25D_Ultimate class is missing. Run the model-definition cell first.\")\n if not os.path.exists(CKPT_PATH):\n raise FileNotFoundError(f\"Checkpoint not found: {CKPT_PATH}\")\n\n ckpt = _load_ckpt(CKPT_PATH)\n state = ckpt[\"model\"] if isinstance(ckpt, dict) and \"model\" in ckpt else ckpt\n\n model_25d = DeblurUNet25D_Ultimate(\n in_ch=4,\n out_ch=1,\n base=32,\n res_min=0.02,\n res_max=0.15,\n meta_dim=4,\n film_hidden=64,\n authority_bias_init=2.0,\n ).to(device)\n\n try:\n model_25d.load_state_dict(state, strict=True)\n print(\"✅ Checkpoint loaded with strict=True\")\n except Exception as e:\n print(\"⚠️ strict=True failed; retrying with strict=False\")\n print(\" reason:\", repr(e))\n msg = model_25d.load_state_dict(state, strict=False)\n print(\" missing_keys:\", list(msg.missing_keys)[:12])\n print(\" unexpected_keys:\", list(msg.unexpected_keys)[:12])\n\n model_25d.eval()\n\n if not _model_has_new_heads(model_25d):\n raise RuntimeError(\"Loaded model is not the expected FiLM+authority version.\")\n\n\n# -----------------------------\n# 2) DICOM helpers\n# -----------------------------\ndef is_dicom_like(fname: str) -> bool:\n if fname.startswith(\".\"):\n return False\n if fname.endswith((\".dcm\", \".DCM\", \".ima\", \".IMA\")):\n return True\n if \".\" not in fname:\n return True\n return False\n\n\ndef dcm_sort_key(fp: str):\n try:\n ds = pydicom.dcmread(fp, stop_before_pixels=True, force=True)\n\n inst = getattr(ds, \"InstanceNumber\", None)\n if inst is not None:\n try:\n return (0, int(inst), fp)\n except Exception:\n pass\n\n ipp = getattr(ds, \"ImagePositionPatient\", None)\n if ipp is not None and len(ipp) >= 3:\n try:\n return (1, float(ipp[2]), fp)\n except Exception:\n pass\n\n sl = getattr(ds, \"SliceLocation\", None)\n if sl is not None:\n try:\n return (2, float(sl), fp)\n except Exception:\n pass\n except Exception:\n pass\n\n return (9, fp)\n\n\ndef find_dicom_files(directory):\n files = []\n for r, d, fs in os.walk(directory):\n for f in fs:\n if is_dicom_like(f):\n files.append(os.path.join(r, f))\n return sorted(files, key=dcm_sort_key)\n\n\ndef list_dicom_series(root_dir):\n \"\"\"\n rel_dir -> sorted list of dicom files\n sorted by InstanceNumber / ImagePositionPatient / SliceLocation / filename\n \"\"\"\n series = {}\n for r, d, fs in os.walk(root_dir):\n dcm_files = [os.path.join(r, f) for f in fs if is_dicom_like(f)]\n if len(dcm_files) == 0:\n continue\n rel = os.path.relpath(r, root_dir)\n series[rel] = sorted(dcm_files, key=dcm_sort_key)\n return series\n\n\ndef dcm_to_hu(ds):\n arr = ds.pixel_array.astype(np.float32)\n slope = float(getattr(ds, \"RescaleSlope\", 1.0))\n intercept = float(getattr(ds, \"RescaleIntercept\", 0.0))\n return arr * slope + intercept\n\n\ndef hu_to_01(hu):\n return np.clip((hu - HU_MIN) / HU_RANGE, 0.0, 1.0).astype(np.float32)\n\n\ndef hu01_to_hu(x01):\n return np.asarray(x01, dtype=np.float32) * HU_RANGE + HU_MIN\n\n\ndef window_hu_to_uint8(hu, center=40.0, width=400.0):\n x = np.clip((hu - (center - width / 2.0)) / (width + 1e-6), 0.0, 1.0)\n return (x * 255.0).astype(np.uint8)\n\n\ndef psnr01(a, b):\n a = np.asarray(a, dtype=np.float32)\n b = np.asarray(b, dtype=np.float32)\n mse = float(np.mean((a - b) ** 2))\n return 99.0 if mse <= 0 else 10.0 * math.log10(1.0 / mse)\n\n\ndef mae01(a, b):\n a = np.asarray(a, dtype=np.float32)\n b = np.asarray(b, dtype=np.float32)\n return float(np.mean(np.abs(a - b)))\n\n\ndef ssim_fast01_2d(a, b):\n a = a.astype(np.float32)\n b = b.astype(np.float32)\n C1, C2 = 0.01 ** 2, 0.03 ** 2\n mu_a = cv2.GaussianBlur(a, (11, 11), 1.5)\n mu_b = cv2.GaussianBlur(b, (11, 11), 1.5)\n mu_a2, mu_b2, mu_ab = mu_a * mu_a, mu_b * mu_b, mu_a * mu_b\n sigma_a2 = cv2.GaussianBlur(a * a, (11, 11), 1.5) - mu_a2\n sigma_b2 = cv2.GaussianBlur(b * b, (11, 11), 1.5) - mu_b2\n sigma_ab = cv2.GaussianBlur(a * b, (11, 11), 1.5) - mu_ab\n ssim_map = ((2 * mu_ab + C1) * (2 * sigma_ab + C2)) / (\n (mu_a2 + mu_b2 + C1) * (sigma_a2 + sigma_b2 + C2) + 1e-8\n )\n return float(np.mean(ssim_map))\n\n\ndef load_paired_case_window(q_files, f_files, start_idx, depth=128, out_hw=(448, 448)):\n q_hu_list, f_hu_list = [], []\n max_len = min(len(q_files), len(f_files))\n\n for k in range(depth):\n idx = start_idx + k\n if idx < 0 or idx >= max_len:\n break\n\n ds_q = pydicom.dcmread(q_files[idx], force=True)\n ds_f = pydicom.dcmread(f_files[idx], force=True)\n\n hq = dcm_to_hu(ds_q)\n hf = dcm_to_hu(ds_f)\n\n if hq.shape != out_hw:\n hq = cv2.resize(hq, (out_hw[1], out_hw[0]), interpolation=cv2.INTER_LINEAR)\n if hf.shape != out_hw:\n hf = cv2.resize(hf, (out_hw[1], out_hw[0]), interpolation=cv2.INTER_LINEAR)\n\n q_hu_list.append(hq.astype(np.float32))\n f_hu_list.append(hf.astype(np.float32))\n\n q_hu = np.stack(q_hu_list, axis=0).astype(np.float32)\n f_hu = np.stack(f_hu_list, axis=0).astype(np.float32)\n q01 = hu_to_01(q_hu)\n f01 = hu_to_01(f_hu)\n return q_hu, f_hu, q01, f01\n\n\ndef build_paired_case_manifest(q_root, f_root, depth=128, n_target=40):\n \"\"\"\n Strict paired-case construction:\n 1) match Quarter / Full by relative directory\n 2) assign one central window per matched series first\n 3) add extra windows only within the same matched series if more cases are needed\n \"\"\"\n q_series = list_dicom_series(q_root)\n f_series = list_dicom_series(f_root)\n\n common_rels = sorted(set(q_series.keys()) & set(f_series.keys()))\n if len(common_rels) == 0:\n raise RuntimeError(\"No matched Quarter/Full series found by relative directory.\")\n\n base_cases = []\n extra_cases = []\n\n for rel in common_rels:\n qf = q_series[rel]\n ff = f_series[rel]\n usable_len = min(len(qf), len(ff))\n if usable_len < depth:\n continue\n\n center_start = max(0, (usable_len - depth) // 2)\n base_cases.append({\n \"rel_dir\": rel,\n \"start_idx\": int(center_start),\n \"usable_len\": int(usable_len),\n \"q_files\": qf,\n \"f_files\": ff,\n })\n\n candidate_starts = sorted(set([\n 0,\n max(0, (usable_len - depth) // 4),\n max(0, (usable_len - depth) // 2),\n max(0, 3 * (usable_len - depth) // 4),\n max(0, usable_len - depth),\n ]))\n\n for st in candidate_starts:\n if st == center_start:\n continue\n extra_cases.append({\n \"rel_dir\": rel,\n \"start_idx\": int(st),\n \"usable_len\": int(usable_len),\n \"q_files\": qf,\n \"f_files\": ff,\n })\n\n if len(base_cases) == 0:\n raise RuntimeError(\"No matched series with enough slices were found.\")\n\n manifest = base_cases.copy()\n\n if len(manifest) < n_target:\n extra_cases = sorted(extra_cases, key=lambda x: (-x[\"usable_len\"], x[\"rel_dir\"], x[\"start_idx\"]))\n seen = {(m[\"rel_dir\"], m[\"start_idx\"]) for m in manifest}\n for c in extra_cases:\n key = (c[\"rel_dir\"], c[\"start_idx\"])\n if key in seen:\n continue\n manifest.append(c)\n seen.add(key)\n if len(manifest) >= n_target:\n break\n\n manifest = manifest[:min(n_target, len(manifest))]\n\n out = []\n for i, m in enumerate(manifest, 1):\n out.append({\n \"case_id\": i,\n \"rel_dir\": m[\"rel_dir\"],\n \"start_idx\": int(m[\"start_idx\"]),\n \"usable_len\": int(m[\"usable_len\"]),\n \"q_files\": m[\"q_files\"],\n \"f_files\": m[\"f_files\"],\n })\n return out\n\n\n# -----------------------------\n# 3) Inference helpers\n# -----------------------------\ndef _clip01(x):\n return np.clip(x, 0.0, 1.0).astype(np.float32)\n\n\ndef _run_model_compat(model, inp_t, meta_t=None):\n try:\n if meta_t is not None:\n return model(inp_t, meta=meta_t)\n except TypeError:\n pass\n except Exception:\n pass\n return model(inp_t)\n\n\n@torch.no_grad()\ndef deblur_volume_25d_compat(model, vol_deg01, t, restore_batch=16, clamp_delta=None, do_motion=False, dose_mode=\"quarter\"):\n \"\"\"\n Compatible with the FiLM + authority-map model:\n meta = [t_norm, do_motion, dose_clean, is_identity]\n \"\"\"\n vol_deg01 = np.asarray(vol_deg01, dtype=np.float32)\n D = vol_deg01.shape[0]\n out = vol_deg01.copy()\n\n t_norm = np.float32(0.0 if t <= 0 else (float(t) / float(BLUR_T_MAX_LOCAL)))\n dose_clean = np.float32(1.0 if (dose_mode == \"clean\" or t <= 0) else 0.0)\n is_identity = np.float32(1.0 if t <= 0 else 0.0)\n do_motion_f = np.float32(1.0 if do_motion else 0.0)\n meta_row = np.array([t_norm, do_motion_f, dose_clean, is_identity], dtype=np.float32)\n\n n_pixels = 0\n n_clamped = 0\n max_raw_delta = 0.0\n mean_abs_delta_sum = 0.0\n\n for s in range(0, D, restore_batch):\n zs = list(range(s, min(D, s + restore_batch)))\n inp_batch = []\n centers = []\n\n for z in zs:\n bp = vol_deg01[max(0, z - 1)]\n bc = vol_deg01[z]\n bn = vol_deg01[min(D - 1, z + 1)]\n centers.append(bc)\n inp_batch.append(np.stack([bp, bc, bn, np.full_like(bc, t_norm)], axis=0).astype(np.float32))\n\n inp_np = np.stack(inp_batch, axis=0)\n inp_t = torch.from_numpy(inp_np).to(device, non_blocking=True)\n meta_t = torch.from_numpy(np.repeat(meta_row[None, :], len(zs), axis=0)).to(device, non_blocking=True)\n\n with AMP_CTX():\n pred_obj = _run_model_compat(model, inp_t, meta_t=meta_t)\n if isinstance(pred_obj, (tuple, list)):\n pred_b = pred_obj[0].float().cpu().numpy()[:, 0]\n else:\n pred_b = pred_obj.float().cpu().numpy()[:, 0]\n\n for k, z in enumerate(zs):\n pred = pred_b[k]\n bc = centers[k]\n\n raw_delta = pred - bc\n max_raw_delta = max(max_raw_delta, float(np.max(np.abs(raw_delta))))\n mean_abs_delta_sum += float(np.sum(np.abs(raw_delta)))\n n_pixels += raw_delta.size\n\n if clamp_delta is not None:\n pred_clamped = np.clip(pred, bc - clamp_delta, bc + clamp_delta)\n n_clamped += int(np.count_nonzero(np.abs(pred - pred_clamped) > 1e-8))\n pred = pred_clamped\n\n out[z] = _clip01(pred)\n\n stats = {\n \"clamp_hit_ratio\": (n_clamped / n_pixels) if (clamp_delta is not None and n_pixels > 0) else 0.0,\n \"max_raw_delta\": float(max_raw_delta),\n \"mean_raw_abs_delta\": float(mean_abs_delta_sum / max(n_pixels, 1)),\n }\n return out, stats\n\n\ndef gaussian_baseline_3d(vol01, sigma=0.8):\n out = np.empty_like(vol01, dtype=np.float32)\n for z in range(vol01.shape[0]):\n out[z] = cv2.GaussianBlur(\n vol01[z], (0, 0), sigmaX=sigma, sigmaY=sigma, borderType=cv2.BORDER_REPLICATE\n )\n return _clip01(out)\n\n\ndef nlm_baseline_3d(vol_hu, center=40.0, width=400.0, h=7):\n D = vol_hu.shape[0]\n out01 = np.empty_like(vol_hu, dtype=np.float32)\n for z in range(D):\n u8 = window_hu_to_uint8(vol_hu[z], center=center, width=width)\n den = cv2.fastNlMeansDenoising(u8, None, h=h, templateWindowSize=7, searchWindowSize=21)\n den01_win = den.astype(np.float32) / 255.0\n den_hu = den01_win * width + (center - width / 2.0)\n out01[z] = hu_to_01(den_hu)\n return _clip01(out01)\n\n\ndef vol_metrics(x01, ref01, q01_for_change):\n z_idx = list(range(0, x01.shape[0], 8))\n ssim_vals = [ssim_fast01_2d(x01[z], ref01[z]) for z in z_idx]\n return {\n \"psnr\": psnr01(x01, ref01),\n \"mae\": mae01(x01, ref01),\n \"ssim\": float(np.mean(ssim_vals)),\n \"mean_abs_change_vs_quarter\": float(np.mean(np.abs(x01 - q01_for_change))),\n \"max_abs_change_vs_quarter\": float(np.max(np.abs(x01 - q01_for_change))),\n }\n\n\ndef paired_bootstrap_ci(x, y, n_boot=2000, seed=2026):\n \"\"\"\n mean(x - y) and 95% bootstrap CI\n \"\"\"\n x = np.asarray(x, dtype=np.float32)\n y = np.asarray(y, dtype=np.float32)\n assert len(x) == len(y) and len(x) > 0\n\n d = x - y\n mean_d = float(np.mean(d))\n\n rng = np.random.default_rng(seed)\n boots = []\n n = len(d)\n for _ in range(n_boot):\n idx = rng.integers(0, n, size=n)\n boots.append(float(np.mean(d[idx])))\n\n lo = float(np.percentile(boots, 2.5))\n hi = float(np.percentile(boots, 97.5))\n return mean_d, lo, hi\n\n\n# -----------------------------\n# 4) Build paired Mayo case manifest\n# -----------------------------\ncase_manifest = build_paired_case_manifest(Q_DIR, F_DIR, depth=VOL_DEPTH, n_target=N_CASES_MAIN)\n\nmanifest_rows = []\nfor m in case_manifest:\n manifest_rows.append({\n \"case_id\": m[\"case_id\"],\n \"rel_dir\": m[\"rel_dir\"],\n \"start_idx\": m[\"start_idx\"],\n \"usable_len\": m[\"usable_len\"],\n \"q_nfiles\": len(m[\"q_files\"]),\n \"f_nfiles\": len(m[\"f_files\"]),\n })\n\ndf_manifest = pd.DataFrame(manifest_rows)\ndf_manifest.to_csv(OUTDIR / \"mayo_case_manifest.csv\", index=False)\n\nprint(\"\\n=== Mayo case manifest (Top-20) ===\")\ndisplay(df_manifest.head(20))\nprint(f\"Total paired cases selected: {len(case_manifest)}\")\n\n\n# -----------------------------\n# 5) Mayo main experiment\n# -----------------------------\nrows = []\nt0 = time.time()\n\nfor i, case_info in enumerate(case_manifest, 1):\n cid = int(case_info[\"case_id\"])\n rel_dir = case_info[\"rel_dir\"]\n st = int(case_info[\"start_idx\"])\n q_files = case_info[\"q_files\"]\n f_files = case_info[\"f_files\"]\n\n print(f\"\\n[{i}/{len(case_manifest)}] case_id={cid:02d} | rel_dir={rel_dir} | start={st}\")\n\n q_hu, f_hu, q01, f01 = load_paired_case_window(\n q_files, f_files, st, depth=VOL_DEPTH, out_hw=(TARGET_H, TARGET_W)\n )\n\n # Quarter\n m_q = vol_metrics(q01, f01, q01)\n rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"variant\": \"Quarter\",\n **m_q,\n \"clamp_hit_ratio\": 0.0,\n \"raw_max_delta_model\": 0.0,\n \"mean_raw_abs_delta_model\": 0.0,\n })\n\n # Gaussian\n if USE_GAUSSIAN:\n g01 = gaussian_baseline_3d(q01, sigma=GAUSSIAN_SIGMA)\n m_g = vol_metrics(g01, f01, q01)\n rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"variant\": \"Gaussian\",\n **m_g,\n \"clamp_hit_ratio\": 0.0,\n \"raw_max_delta_model\": 0.0,\n \"mean_raw_abs_delta_model\": 0.0,\n })\n\n # NLM\n if USE_NLM:\n n01 = nlm_baseline_3d(q_hu, h=NLM_H)\n m_n = vol_metrics(n01, f01, q01)\n rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"variant\": \"NLM\",\n **m_n,\n \"clamp_hit_ratio\": 0.0,\n \"raw_max_delta_model\": 0.0,\n \"mean_raw_abs_delta_model\": 0.0,\n })\n\n # V-Ultimate\n rec01, rec_stats = deblur_volume_25d_compat(\n model_25d,\n q01,\n t=T_INFER,\n restore_batch=RESTORE_BATCH,\n clamp_delta=CLAMP_DELTA,\n do_motion=False,\n dose_mode=\"quarter\",\n )\n m_u = vol_metrics(rec01, f01, q01)\n rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"variant\": \"V-Ultimate\",\n **m_u,\n \"clamp_hit_ratio\": float(rec_stats[\"clamp_hit_ratio\"]),\n \"raw_max_delta_model\": float(rec_stats[\"max_raw_delta\"]),\n \"mean_raw_abs_delta_model\": float(rec_stats[\"mean_raw_abs_delta\"]),\n })\n\n line = f\" Quarter PSNR={m_q['psnr']:.2f}\"\n if USE_GAUSSIAN:\n line += f\" | Gaussian={m_g['psnr']:.2f}\"\n if USE_NLM:\n line += f\" | NLM={m_n['psnr']:.2f}\"\n line += f\" | V-Ultimate={m_u['psnr']:.2f}\"\n line += f\" | rawΔmax={rec_stats['max_raw_delta']:.3f}\"\n line += f\" | mean|Δ|={rec_stats['mean_raw_abs_delta']:.4f}\"\n print(line)\n\n del q_hu, f_hu, q01, f01, rec01\n if USE_GAUSSIAN:\n del g01\n if USE_NLM:\n del n01\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\ndf_raw = pd.DataFrame(rows)\nraw_csv = OUTDIR / \"mayo_generalization_raw.csv\"\ndf_raw.to_csv(raw_csv, index=False)\n\ndf_sum = (\n df_raw.groupby(\"variant\", as_index=False)\n .agg(\n n_case=(\"case_id\", \"nunique\"),\n psnr_mean=(\"psnr\", \"mean\"),\n psnr_std=(\"psnr\", \"std\"),\n mae_mean=(\"mae\", \"mean\"),\n mae_std=(\"mae\", \"std\"),\n ssim_mean=(\"ssim\", \"mean\"),\n ssim_std=(\"ssim\", \"std\"),\n mean_abs_change_vs_quarter=(\"mean_abs_change_vs_quarter\", \"mean\"),\n max_abs_change_vs_quarter=(\"max_abs_change_vs_quarter\", \"mean\"),\n clamp_hit_ratio=(\"clamp_hit_ratio\", \"mean\"),\n raw_max_delta_model=(\"raw_max_delta_model\", \"mean\"),\n mean_raw_abs_delta_model=(\"mean_raw_abs_delta_model\", \"mean\"),\n )\n .sort_values(\"psnr_mean\", ascending=False)\n .reset_index(drop=True)\n)\nsum_csv = OUTDIR / \"mayo_generalization_summary.csv\"\ndf_sum.to_csv(sum_csv, index=False)\n\nprint(\"\\n=== Mayo Generalization Summary ===\")\ndisplay(df_sum)\n\n\n# -----------------------------\n# 6) Paired comparison + bootstrap CI\n# -----------------------------\nq = df_raw[df_raw[\"variant\"] == \"Quarter\"][[\"case_id\", \"psnr\", \"mae\", \"ssim\"]].rename(\n columns={\"psnr\": \"psnr_q\", \"mae\": \"mae_q\", \"ssim\": \"ssim_q\"}\n)\n\npaired_rows = []\nboot_rows = []\n\nfor v in sorted(df_raw[\"variant\"].unique()):\n if v == \"Quarter\":\n continue\n\n d = df_raw[df_raw[\"variant\"] == v][[\"case_id\", \"psnr\", \"mae\", \"ssim\"]].rename(\n columns={\"psnr\": \"psnr_v\", \"mae\": \"mae_v\", \"ssim\": \"ssim_v\"}\n )\n m = q.merge(d, on=\"case_id\", how=\"inner\").sort_values(\"case_id\").reset_index(drop=True)\n\n dpsnr = m[\"psnr_v\"].values - m[\"psnr_q\"].values\n dmae = m[\"mae_q\"].values - m[\"mae_v\"].values\n dssim = m[\"ssim_v\"].values - m[\"ssim_q\"].values\n\n paired_rows.append({\n \"variant\": v,\n \"n_case\": len(m),\n \"ΔPSNR_vs_Quarter\": float(np.mean(dpsnr)),\n \"ΔMAE_vs_Quarter\": float(np.mean(dmae)),\n \"ΔSSIM_vs_Quarter\": float(np.mean(dssim)),\n \"PSNR_win_rate\": float((m[\"psnr_v\"] > m[\"psnr_q\"]).mean()),\n \"MAE_win_rate\": float((m[\"mae_v\"] < m[\"mae_q\"]).mean()),\n \"SSIM_win_rate\": float((m[\"ssim_v\"] > m[\"ssim_q\"]).mean()),\n })\n\n psnr_mean, psnr_lo, psnr_hi = paired_bootstrap_ci(\n m[\"psnr_v\"].values, m[\"psnr_q\"].values, n_boot=BOOT_N, seed=SEED + 1\n )\n mae_mean, mae_lo, mae_hi = paired_bootstrap_ci(\n m[\"mae_q\"].values, m[\"mae_v\"].values, n_boot=BOOT_N, seed=SEED + 2\n )\n ssim_mean, ssim_lo, ssim_hi = paired_bootstrap_ci(\n m[\"ssim_v\"].values, m[\"ssim_q\"].values, n_boot=BOOT_N, seed=SEED + 3\n )\n\n boot_rows.append({\n \"variant\": v,\n \"n_case\": len(m),\n \"metric\": \"ΔPSNR_vs_Quarter\",\n \"mean\": psnr_mean,\n \"ci95_lo\": psnr_lo,\n \"ci95_hi\": psnr_hi,\n })\n boot_rows.append({\n \"variant\": v,\n \"n_case\": len(m),\n \"metric\": \"ΔMAE_vs_Quarter\",\n \"mean\": mae_mean,\n \"ci95_lo\": mae_lo,\n \"ci95_hi\": mae_hi,\n })\n boot_rows.append({\n \"variant\": v,\n \"n_case\": len(m),\n \"metric\": \"ΔSSIM_vs_Quarter\",\n \"mean\": ssim_mean,\n \"ci95_lo\": ssim_lo,\n \"ci95_hi\": ssim_hi,\n })\n\ndf_pair = pd.DataFrame(paired_rows).sort_values(\"ΔPSNR_vs_Quarter\", ascending=False).reset_index(drop=True)\npair_csv = OUTDIR / \"mayo_generalization_paired_vs_quarter.csv\"\ndf_pair.to_csv(pair_csv, index=False)\n\ndf_boot = pd.DataFrame(boot_rows)\nboot_csv = OUTDIR / \"mayo_generalization_bootstrap_vs_quarter.csv\"\ndf_boot.to_csv(boot_csv, index=False)\n\nprint(\"\\n=== Paired vs Quarter ===\")\ndisplay(df_pair)\n\nprint(\"\\n=== Bootstrap CI vs Quarter ===\")\ndisplay(df_boot)\n\n\n# -----------------------------\n# 7) TotalSegmentator section\n# -----------------------------\nif RUN_TOTALSEG:\n try:\n import nibabel as nib\n except Exception:\n print(\"Installing nibabel ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"nibabel\"], check=False)\n import nibabel as nib\n\n cli = None\n for c in [\"TotalSegmentator\", \"totalsegmentator\"]:\n if shutil.which(c):\n cli = c\n break\n\n if cli is None:\n print(\"Installing TotalSegmentator ...\")\n subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"TotalSegmentator\"], check=False)\n for c in [\"TotalSegmentator\", \"totalsegmentator\"]:\n if shutil.which(c):\n cli = c\n break\n\n if cli is None:\n print(\"⚠️ TotalSegmentator CLI not available; skipping anatomy section.\")\n else:\n print(\"\\n✅ TotalSegmentator CLI found:\", cli)\n print(f\" task={TOTALSEG_TASK} | change_thr={TOTALSEG_CHANGE_THR} | fast={TOTALSEG_FAST} | pick={TOTALSEG_PICK}\")\n\n def save_nifti_hu(vol_hu_dhw, out_path):\n affine = np.eye(4, dtype=np.float32)\n vol_hwd = np.transpose(vol_hu_dhw.astype(np.float32), (1, 2, 0))\n nib.save(nib.Nifti1Image(vol_hwd, affine), str(out_path))\n\n def run_totalseg(in_nii, out_dir):\n out_dir = Path(out_dir)\n out_dir.mkdir(parents=True, exist_ok=True)\n\n existing = list(out_dir.rglob(\"*.nii.gz\"))\n if TOTALSEG_REUSE_EXISTING and len(existing) > 0:\n return 0, \"cached\"\n\n cmd = [cli, \"-i\", str(in_nii), \"-o\", str(out_dir), \"--task\", TOTALSEG_TASK]\n if TOTALSEG_FAST:\n cmd_fast = cmd + [\"--fast\"]\n p = subprocess.run(cmd_fast, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n if p.returncode == 0:\n return 0, p.stdout\n\n p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n return p.returncode, p.stdout\n\n def dice_bin(a, b):\n a = (a > 0.5)\n b = (b > 0.5)\n denom = int(a.sum()) + int(b.sum())\n if denom == 0:\n return 1.0\n inter = int((a & b).sum())\n return 2.0 * inter / denom\n\n pick_case_ids = sorted(df_raw[df_raw[\"variant\"] == \"V-Ultimate\"][\"case_id\"].unique().tolist())[:TOTALSEG_PICK]\n\n overlap_rows = []\n dice_rows = []\n case_rows = []\n\n for j, cid in enumerate(pick_case_ids, 1):\n case_info = next((x for x in case_manifest if int(x[\"case_id\"]) == int(cid)), None)\n if case_info is None:\n continue\n\n rel_dir = case_info[\"rel_dir\"]\n st = int(case_info[\"start_idx\"])\n q_files = case_info[\"q_files\"]\n f_files = case_info[\"f_files\"]\n\n print(f\"\\n[TotalSeg {j}/{len(pick_case_ids)}] case_id={cid:02d} | rel_dir={rel_dir} | start={st}\")\n\n q_hu, f_hu, q01, f01 = load_paired_case_window(\n q_files, f_files, st, depth=VOL_DEPTH, out_hw=(TARGET_H, TARGET_W)\n )\n rec01, rec_stats = deblur_volume_25d_compat(\n model_25d,\n q01,\n t=T_INFER,\n restore_batch=RESTORE_BATCH,\n clamp_delta=CLAMP_DELTA,\n do_motion=False,\n dose_mode=\"quarter\",\n )\n\n g01 = gaussian_baseline_3d(q01, sigma=GAUSSIAN_SIGMA) if USE_GAUSSIAN else None\n n01 = nlm_baseline_3d(q_hu, h=NLM_H) if USE_NLM else None\n\n safe_rel = rel_dir.replace(\"/\", \"__\").replace(\"\\\\\", \"__\")\n case_dir = OUTDIR / \"totalseg_runs\" / f\"case_{cid:02d}_{safe_rel}_st{st:04d}\"\n case_dir.mkdir(parents=True, exist_ok=True)\n\n nii_map = {\n \"full\": case_dir / \"full.nii.gz\",\n \"quarter\": case_dir / \"quarter.nii.gz\",\n \"ultimate\": case_dir / \"ultimate.nii.gz\",\n }\n save_nifti_hu(f_hu, nii_map[\"full\"])\n save_nifti_hu(q_hu, nii_map[\"quarter\"])\n save_nifti_hu(hu01_to_hu(rec01), nii_map[\"ultimate\"])\n\n if g01 is not None:\n nii_map[\"gaussian\"] = case_dir / \"gaussian.nii.gz\"\n save_nifti_hu(hu01_to_hu(g01), nii_map[\"gaussian\"])\n\n if n01 is not None:\n nii_map[\"nlm\"] = case_dir / \"nlm.nii.gz\"\n save_nifti_hu(hu01_to_hu(n01), nii_map[\"nlm\"])\n\n seg_dirs = {}\n for nm, nii_p in nii_map.items():\n print(f\" running {nm} ...\")\n t_case = time.time()\n rc, outtxt = run_totalseg(nii_p, case_dir / f\"seg_{nm}\")\n print(f\" returncode={rc} | seg_time={(time.time() - t_case) / 60:.1f} min\")\n if rc != 0:\n print(\"\\n\".join(str(outtxt).splitlines()[-15:]))\n seg_dirs[nm] = None\n else:\n seg_dirs[nm] = case_dir / f\"seg_{nm}\"\n\n if seg_dirs.get(\"full\") is None or seg_dirs.get(\"quarter\") is None or seg_dirs.get(\"ultimate\") is None:\n print(\" skipping case because required segmentations are missing\")\n continue\n\n change_map = np.abs(rec01 - q01).astype(np.float32)\n change_mask = (change_map >= TOTALSEG_CHANGE_THR).astype(np.uint8)\n change_mask_hwd = np.transpose(change_mask, (1, 2, 0))\n total_changed_vox = int(change_mask_hwd.sum())\n\n full_masks = sorted(list((seg_dirs[\"full\"]).rglob(\"*.nii.gz\")))\n print(f\" masks found={len(full_masks)} | changed_vox={total_changed_vox}\")\n\n compare_variants = [\"quarter\", \"ultimate\"]\n if seg_dirs.get(\"gaussian\") is not None:\n compare_variants.append(\"gaussian\")\n if seg_dirs.get(\"nlm\") is not None:\n compare_variants.append(\"nlm\")\n\n best_mask = None\n best_share = -1.0\n nz_masks = 0\n\n for mf in full_masks:\n try:\n mask_name = mf.name.replace(\".nii.gz\", \"\")\n full_arr = nib.load(str(mf)).get_fdata()\n full_bin = (full_arr > 0.5).astype(np.uint8)\n seg_vox = int(full_bin.sum())\n if seg_vox == 0:\n continue\n\n inter = int((full_bin * change_mask_hwd).sum())\n if inter > 0:\n nz_masks += 1\n\n change_in_seg_ratio = (inter / seg_vox) if seg_vox > 0 else np.nan\n seg_share_of_change = (inter / total_changed_vox) if total_changed_vox > 0 else np.nan\n\n if not np.isnan(seg_share_of_change) and seg_share_of_change > best_share:\n best_share = seg_share_of_change\n best_mask = mask_name\n\n overlap_rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"mask_name\": mask_name,\n \"seg_vox_full\": seg_vox,\n \"changed_vox_total\": total_changed_vox,\n \"intersect_change_vox\": inter,\n \"change_in_seg_ratio\": change_in_seg_ratio,\n \"seg_share_of_change\": seg_share_of_change,\n })\n\n for vnm in compare_variants:\n vpath = seg_dirs[vnm] / mf.name\n if not vpath.exists():\n cands = list((seg_dirs[vnm]).rglob(mf.name))\n if len(cands) == 0:\n continue\n vpath = cands[0]\n\n v_arr = nib.load(str(vpath)).get_fdata()\n dice_rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"mask_name\": mask_name,\n \"variant\": vnm,\n \"dice_vs_full\": float(dice_bin(full_arr, v_arr)),\n })\n except Exception as e:\n print(\" skip mask:\", mf.name, e)\n\n case_rows.append({\n \"case_id\": cid,\n \"rel_dir\": rel_dir,\n \"start_idx\": st,\n \"changed_vox_total\": total_changed_vox,\n \"n_masks_total\": len(full_masks),\n \"n_masks_nonzero\": nz_masks,\n \"top_mask\": best_mask,\n \"top_mask_seg_share\": best_share if best_share >= 0 else np.nan,\n })\n\n del q_hu, f_hu, q01, f01, rec01, change_map, change_mask, change_mask_hwd\n if g01 is not None:\n del g01\n if n01 is not None:\n del n01\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n\n if len(overlap_rows):\n df_ov = pd.DataFrame(overlap_rows)\n df_ov.to_csv(OUTDIR / \"totalseg_overlap_raw.csv\", index=False)\n\n ov_sum = (\n df_ov.groupby(\"mask_name\", as_index=False)\n .agg(\n n_case=(\"case_id\", \"nunique\"),\n n_case_nonzero=(\"intersect_change_vox\", lambda s: int((np.asarray(s) > 0).sum())),\n pct_cases_nonzero=(\"intersect_change_vox\", lambda s: float((np.asarray(s) > 0).mean() * 100.0)),\n mean_change_in_seg_ratio=(\"change_in_seg_ratio\", \"mean\"),\n median_change_in_seg_ratio=(\"change_in_seg_ratio\", \"median\"),\n mean_seg_share_of_change=(\"seg_share_of_change\", \"mean\"),\n median_seg_share_of_change=(\"seg_share_of_change\", \"median\"),\n max_seg_share_of_change=(\"seg_share_of_change\", \"max\"),\n )\n .sort_values([\"n_case_nonzero\", \"mean_seg_share_of_change\"], ascending=[False, False])\n .reset_index(drop=True)\n )\n ov_sum.to_csv(OUTDIR / \"totalseg_overlap_summary.csv\", index=False)\n\n print(\"\\n=== TotalSegmentator overlap summary (Top-30) ===\")\n display(ov_sum.head(30))\n\n if len(case_rows):\n df_case = pd.DataFrame(case_rows).sort_values(\n [\"changed_vox_total\", \"top_mask_seg_share\"], ascending=[False, False]\n ).reset_index(drop=True)\n df_case.to_csv(OUTDIR / \"totalseg_case_summary.csv\", index=False)\n\n print(\"\\n=== TotalSegmentator case summary (Top-20) ===\")\n display(df_case.head(20))\n\n if len(dice_rows):\n df_d = pd.DataFrame(dice_rows)\n df_d.to_csv(OUTDIR / \"totalseg_dice_raw.csv\", index=False)\n\n d_sum = (\n df_d.groupby(\"variant\", as_index=False)\n .agg(\n mean_dice_vs_full=(\"dice_vs_full\", \"mean\"),\n std_dice_vs_full=(\"dice_vs_full\", \"std\"),\n median_dice_vs_full=(\"dice_vs_full\", \"median\"),\n n_rows=(\"dice_vs_full\", \"size\"),\n )\n .sort_values(\"mean_dice_vs_full\", ascending=False)\n .reset_index(drop=True)\n )\n d_sum.to_csv(OUTDIR / \"totalseg_dice_summary.csv\", index=False)\n\n print(\"\\n=== TotalSegmentator Dice summary ===\")\n display(d_sum)\n\n\n# -----------------------------\n# 8) Brief summary\n# -----------------------------\nprint(\"\\n=== Brief summary ===\")\ntry:\n qrow = df_sum[df_sum[\"variant\"] == \"Quarter\"].iloc[0]\n urow = df_sum[df_sum[\"variant\"] == \"V-Ultimate\"].iloc[0]\n print(\n f\"On paired external Mayo quarter-dose CT (n={int(urow['n_case'])}), \"\n f\"V-Ultimate vs Quarter: \"\n f\"ΔPSNR={urow['psnr_mean'] - qrow['psnr_mean']:+.2f} dB, \"\n f\"ΔMAE={qrow['mae_mean'] - urow['mae_mean']:+.4f}, \"\n f\"ΔSSIM={urow['ssim_mean'] - qrow['ssim_mean']:+.4f}.\"\n )\n print(\"This evaluation uses matched Quarter/Full series, paired case-level comparison, and bootstrap confidence intervals.\")\nexcept Exception as e:\n print(\"Summary generation skipped:\", e)\n\nprint(\"\\nSaved CSV files:\")\nfor p in sorted(OUTDIR.glob(\"*.csv\")):\n print(\" -\", p)\n\nprint(f\"\\n✅ Done. elapsed={(time.time() - t0) / 60:.1f} min\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-10T19:28:14.883215Z","iopub.status.idle":"2026-03-10T19:28:14.883487Z","shell.execute_reply.started":"2026-03-10T19:28:14.883368Z","shell.execute_reply":"2026-03-10T19:28:14.883381Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Mayo 结果解读（8 例外部配对低剂量 CT）\n\n### 评委核心质疑\n\n> **\"你的模型是不是只在 RSNA 合成退化上有效？换到真实低剂量数据还能用吗？\"**\n\n### 回答：能用，而且修改极其克制。\n\n### 汇总表\n\n| 方法 | PSNR (dB) | MAE | SSIM | 平均改动幅度 | 最大改动幅度 |\n|------|-----------|-----|------|-------------|-------------|\n| Quarter（原始） | 41.44 | 0.00697 | 0.906 | — | — |\n| **Gaussian** | **42.37** | **0.00550** | **0.944** | 0.0058 | 0.218 |\n| **V-Ultimate** | 41.84 | 0.00660 | 0.899 | **0.0025** | **0.018** |\n| NLM | 17.24 | 0.102 | 0.492 | 0.098 | 0.547 |\n\n### 配对比较（vs Quarter）\n\n| 方法 | ΔPSNR | ΔMAE | ΔSSIM | PSNR 胜率 |\n|------|-------|------|-------|----------|\n| Gaussian | **+0.94 dB** | +0.0015 | **+0.037** | 75% |\n| V-Ultimate | +0.40 dB | +0.0004 | -0.008 | **87.5%** |\n\n### 关键解读\n\n1. **V-Ultimate 在 87.5% 的病例上 PSNR 优于 Quarter**——比 Gaussian 的 75% 更稳定。\n2. **Gaussian 的平均 PSNR 更高（+0.94 vs +0.40 dB）**——但代价是什么？\n - Gaussian 的最大改动幅度 = **0.218**（几乎重写了 22% 的像素范围）\n - V-Ultimate 的最大改动幅度 = **0.018**（仅 1.8%）\n - **V-Ultimate 用 Gaussian 1/12 的修改幅度，实现了 43% 的 PSNR 提升**\n3. **V-Ultimate 的 SSIM 略低于 Quarter（-0.008）**：因为模型的微量修改改变了局部纹理，但幅度极小。\n4. **NLM 在 Mayo 上完全失败**（PSNR 17.2 dB）：NLM 的窗口化策略不适合 Mayo 的噪声特性。\n\n### 为什么 V-Ultimate \"不如\" Gaussian 反而是好事？\n\n这正是 **Do-No-Harm** 设计的体现：\n- Gaussian 把整张图都平滑了（改动幅度 0.218），像\"全身麻醉\"\n- V-Ultimate 只做微量精准修改（改动幅度 0.018），像\"局部麻醉\"\n- 在没见过的 Mayo 数据上，模型**自动选择了保守策略**（t_norm=0.05），而不是过度增强\n\n### 🔗 一句话结论\n> V-Ultimate 在完全没见过的 Mayo 数据上，用 **Gaussian 1/12 的修改幅度** 实现了 **87.5% 的 PSNR 胜率**——跨域泛化得到支持，且模型自动保持\"受控修改\"行为。\n","metadata":{}},{"cell_type":"markdown","source":"# 后续工作（Future Work）\n\n## 模型层面\n- 不确定性驱动的动态权限（uncertainty-aware authority）\n- 解剖结构/血管区域引导的恢复约束（region-aware restoration）\n- 更细粒度的失败病例自适应策略\n\n## 评估层面\n- 扩展 Monte Carlo 至更多退化机制（motion / dose regime）\n- 引入更多下游代理模型进行交叉验证\n- 与真实病灶级标注任务（若可获得）进行关联验证\n\n## 科学叙事层面\n本项目后续目标不是\"把模型讲得更强\"，而是继续提高：\n- 可验证性（verifiability）\n- 可解释性（interpretability）\n- 可复现实验设计（reproducibility）","metadata":{}},{"cell_type":"markdown","source":"# 评委速读版（One-Page Judge Summary）\n\n### 我做了什么？\n训练了一个 **物理约束的 CT 恢复模型（V-Ultimate）**，重点不是\"更锐化\"，而是\"有边界地恢复，不误导下游诊断\"。\n\n### 我如何证明它有效？\n不只看 PSNR，而是用了 **4 层证据**：\n1. **Clinical Rescue Matrix**：下游临床代理任务是否受益\n2. **Monte Carlo（100×10）**：\"过头恢复\"是否只是随机噪声巧合\n3. **TotalSegmentator overlap**：改动是否集中在解剖结构，而非全图乱改\n4. **Mayo 跨域泛化**：换数据集后是否仍保持受控修改特性\n\n### 核心结论是什么？\nV-Ultimate 不是一个\"激进增强器\"，而是一个 **可约束、可解释、可验证的恢复系统**。\n它在多种验证设置下表现出\"受控修改\"的一致性，这比单一指标最优更符合医学场景的安全需求。\n\n### 一句话版本\n> 我的项目不是在追求\"最强锐化\"，而是在做一种 **可验证、可约束、可解释** 的医学图像恢复：它在 OOD 场景下能提供临床代理收益，在噪声扰动下可以分析稳定性边界，并且其改动具有解剖学集中性，而不是全图随机增强。","metadata":{}}]}