{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[],"toc_visible":true},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":99552,"databundleVersionId":13441085,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# DICOM Rescale Normalizer\n\nThis notebook standardizes CT DICOM intensity metadata step-by-step.  \n- **Audit** how `RescaleSlope`/`RescaleIntercept` are used across your dataset.  \n- **Retag (fast)**: set `RescaleIntercept=0` and `RescaleSlope=1` (tags only; pixels unchanged).  \n- **Bake (optional)**: convert stored values to HU, then set tags to identity (pixels changed).  \nAll results are written to `/kaggle/working` (Kaggle inputs are read-only).\n\n---\n\n## Why this exists\n\nQuality volumetric scans found in medical imaging undergo a defacing process to remove pixel-specific PHI related to facial recognition. Different scanners/vendors also tend to store CT intensities differently. Some series are **already HU-baked**--that is, in \"Hounsfield Units\" (pixels ≈ HU, tags 1/0); others are **raw** and need the Modality LUT (`HU = SV * slope + intercept`). This notebook provides:\n- A **fast audit** of slope/intercept usage.\n- A safe **retag** mode for pipelines that require identity tags.\n- A conditional **bake** path for tools that need persisted HU.\n\n> **Recommendation:** For modeling, prefer **read-time normalization** (`apply_modality_lut`) instead of rewriting files. Use retag/bake only when you must standardize files for a downstream tool.\n\n---\n\n## What this notebook does\n\n1. **Audit (default):** scans `.dcm` files with metadata-only reads and reports `(Modality, RescaleSlope, RescaleIntercept)` values.  \n2. **Retag mode:** edits only the rescale tags (keeps Transfer Syntax and PixelData for quick processing).  \n3. **Bake mode (optional):** applies current slope/intercept to pixels (producing HU), updates pixel type, and sets tags to identity. Skips compressed images unless `--decompress` is enabled.\n\n---\n\n## Inputs & Outputs\n\n- **Input:**  A DICOM tree under `/kaggle/input/.../` (e.g., `.../train/images/.../*.dcm`).\n- **Output directory:** `/kaggle/working/rescale_out/...` mirroring the input folder structure.\n- **Report CSV (optional):** `/kaggle/working/rescale_report.csv` with per-file status.\n\n**Report columns**\n- `src` — source file path  \n- `dst` — destination file path  \n- `status` — `changed | unchanged | skipped | failed`  \n- `reason` — reason for skip/fail (e.g., `modality`, `compressed`, `exception`)  \n- `note` — extra details (e.g., exception text or Transfer Syntax)\n","metadata":{"id":"SUDLYc-cYfHU"}},{"cell_type":"markdown","source":"# Code","metadata":{"id":"BV26RZXTascM"}},{"cell_type":"code","source":"# Notebook cell: run-this-as-is in Kaggle\n\nfrom __future__ import annotations\nimport os, csv, shutil, logging\nfrom typing import List, Dict, Any, Tuple, Optional\n\nimport numpy as np\nimport pydicom\nfrom pydicom.tag import Tag\nfrom pydicom.uid import UID\nfrom pydicom.filewriter import dcmwrite\nfrom pydicom.valuerep import DSfloat\nfrom pydicom.pixel_data_handlers.util import apply_modality_lut\n\ntry:\n    from tqdm import tqdm\nexcept Exception:\n    def tqdm(x, **kwargs): return x  # fallback\n\n# -------------------- CONFIG (edit here) --------------------\nMODE = \"audit\"                 # \"audit\" | \"retag\" | \"bake\"\nINPUT_DIR = \"/kaggle/input/rsna-intracranial-aneurysm-detection/series\"      # leave \"\" to auto-detect\nOUTPUT_DIR = \"/kaggle/working/dcm_rescale_out\"\nREPORT_CSV = \"/kaggle/working/dcm_rescale_report.csv\"\n\nDESIRED_INTERCEPT = 0.0        # for retag/bake\nDESIRED_SLOPE = 1.0            # for retag/bake\nMODALITY_FILTER = \"ANY\"         # \"CT\" or \"ANY\"\nCOPY_UNCHANGED = True          # mirror unchanged files in retag/bake\nCONDITIONAL_BAKE = True        # bake only when not already baked\nDECOMPRESS = False             # bake compressed TS (needs gdcm/pylibjpeg)\nVERBOSE = True                 # more logs\n# ------------------------------------------------------------\n\n# ---- logging (why: reproducible, quiet by default) ----\ndef setup_logger(verbose: bool) -> logging.Logger:\n    level = logging.DEBUG if verbose else logging.INFO\n    logging.basicConfig(level=level,\n                        format='[%(asctime)s] %(levelname)s - %(message)s',\n                        datefmt='%H:%M:%S',\n                        force=True)\n    return logging.getLogger(\"dcm_rescaler\")\n\nlogger = setup_logger(VERBOSE)\n\n# ---- tags ----\nMODALITY              = Tag(0x0008, 0x0060)\nRESCALE_INTERCEPT     = Tag(0x0028, 0x1052)\nRESCALE_SLOPE         = Tag(0x0028, 0x1053)\nPIXEL_DATA            = Tag(0x7FE0, 0x0010)\nBITS_ALLOCATED        = Tag(0x0028, 0x0100)\nBITS_STORED           = Tag(0x0028, 0x0101)\nHIGH_BIT              = Tag(0x0028, 0x0102)\nPIXEL_REPRESENTATION  = Tag(0x0028, 0x0103)\nSMALLEST_PIXEL_VALUE  = Tag(0x0028, 0x0106)\nLARGEST_PIXEL_VALUE   = Tag(0x0028, 0x0107)\n\n# ---- helpers ----\ndef auto_detect_input_dir(root=\"/kaggle/input\") -> str:\n    \"\"\"Why: zero-config on Kaggle; picks the first folder containing .dcm.\"\"\"\n    for dirpath, _, files in os.walk(root):\n        if any(f.lower().endswith(\".dcm\") for f in files):\n            return dirpath\n    raise FileNotFoundError(f\"No DICOM files found under {root} (add a dataset as Input).\")\n\ndef gather_dicoms(root: str) -> List[str]:\n    out: List[str] = []\n    for d, _, files in os.walk(root):\n        for f in files:\n            if f.lower().endswith(\".dcm\"):\n                out.append(os.path.join(d, f))\n    return out\n\ndef modality_ok(ds: pydicom.Dataset, wanted: str) -> bool:\n    if wanted.upper() == \"ANY\": return True\n    return str(ds.get(MODALITY, \"\")).upper() == wanted.upper()\n\ndef is_compressed(ds: pydicom.Dataset) -> bool:\n    ts: UID = ds.file_meta.TransferSyntaxUID\n    return ts.is_compressed\n\ndef ensure_parent(path: str) -> None:\n    os.makedirs(os.path.dirname(path), exist_ok=True)\n\ndef rel_dst(src_root: str, dst_root: str, path: str) -> str:\n    return os.path.join(dst_root, os.path.relpath(path, src_root))\n\ndef set_rescale_tags(ds: pydicom.Dataset, intercept: float, slope: float) -> bool:\n    \"\"\"Why: DSfloat preserves VR precision.\"\"\"\n    changed = False\n    cur_i = float(ds.get(RESCALE_INTERCEPT, 0.0))\n    cur_s = float(ds.get(RESCALE_SLOPE, 1.0))\n    if cur_i != intercept:\n        ds[RESCALE_INTERCEPT] = pydicom.DataElement(RESCALE_INTERCEPT, \"DS\", DSfloat(intercept))\n        changed = True\n    if cur_s != slope:\n        ds[RESCALE_SLOPE] = pydicom.DataElement(RESCALE_SLOPE, \"DS\", DSfloat(slope))\n        changed = True\n    return changed\n\ndef is_already_baked(ds: pydicom.Dataset, mean_tol: float = 1e-5, max_tol: float = 1.0) -> bool:\n    \"\"\"Why: avoid rebaking HU; tolerate tiny rounding.\"\"\"\n    sv = ds.pixel_array\n    hu = apply_modality_lut(sv, ds)\n    if sv.shape != hu.shape: return False\n    diff = np.asarray(hu, np.float32) - np.asarray(sv, np.float32)\n    return (float(np.nanmean(np.abs(diff))) <= mean_tol) and (float(np.nanmax(np.abs(diff))) <= max_tol)\n\ndef to_int16_signed(arr: np.ndarray) -> Tuple[np.ndarray, bool]:\n    a = arr.astype(np.float32)\n    neg = bool(np.nanmin(a) < 0)\n    a = np.nan_to_num(a, nan=0.0)\n    a = np.clip(a, np.iinfo(np.int16).min, np.iinfo(np.int16).max)\n    return a.astype(np.int16), neg\n\n# ---- modes ----\ndef run_audit(input_dir: str, report_csv: str) -> None:\n    files = gather_dicoms(input_dir)\n    rows: List[Dict[str, Any]] = []\n    counts: Dict[Tuple[str, float, float], int] = {}\n    for p in tqdm(files, desc=\"Audit\"):\n        try:\n            ds = pydicom.dcmread(p, stop_before_pixels=True, force=True)\n            if not modality_ok(ds, MODALITY_FILTER): continue\n            mod = str(ds.get(MODALITY, \"\"))\n            slope = float(ds.get(RESCALE_SLOPE, 1.0))\n            inter = float(ds.get(RESCALE_INTERCEPT, 0.0))\n            rows.append({\"path\": p, \"modality\": mod, \"slope\": slope, \"intercept\": inter})\n            counts[(mod, slope, inter)] = counts.get((mod, slope, inter), 0) + 1\n        except Exception as e:\n            rows.append({\"path\": p, \"modality\": \"\", \"slope\": \"\", \"intercept\": \"\", \"error\": str(e)})\n\n    ensure_parent(report_csv)\n    keys = sorted({k for r in rows for k in r.keys()})\n    with open(report_csv, \"w\", newline=\"\") as f:\n        w = csv.DictWriter(f, fieldnames=keys)\n        w.writeheader(); w.writerows(rows)\n    logger.info(\"Audit written: %s (rows=%d)\", report_csv, len(rows))\n    for (mod, s, i), n in sorted(counts.items(), key=lambda x: (-x[1], x[0])):\n        print(f\"{mod:>3} slope={s:<10} intercept={i:<10} count={n}\")\n\ndef run_retag(input_dir: str, output_dir: str, report_csv: str) -> None:\n    files = gather_dicoms(input_dir)\n    rows: List[Dict[str, Any]] = []\n    changed = unchanged = skipped = failed = 0\n\n    for src in tqdm(files, desc=\"Retag\"):\n        dst = rel_dst(input_dir, output_dir, src)\n        try:\n            ds = pydicom.dcmread(src, force=True)  # keep PixelData intact; avoid pixel_array\n            if not modality_ok(ds, MODALITY_FILTER):\n                rows.append({\"src\": src, \"dst\": dst, \"status\":\"skipped\", \"reason\":\"modality\", \"note\":str(ds.get(MODALITY,\"\"))})\n                skipped += 1; continue\n\n            did = set_rescale_tags(ds, DESIRED_INTERCEPT, DESIRED_SLOPE)\n            ensure_parent(dst)\n            dcmwrite(dst, ds, write_like_original=True)  # preserves raw encoding\n            rows.append({\"src\": src, \"dst\": dst, \"status\":\"changed\" if did else \"unchanged\", \"reason\":\"\", \"note\":\"\"})\n            changed += int(did); unchanged += int(not did)\n\n            if (not did) and COPY_UNCHANGED:\n                try: shutil.copy2(src, dst)\n                except Exception: pass\n        except Exception as e:\n            rows.append({\"src\": src, \"dst\": dst, \"status\":\"failed\", \"reason\":\"exception\", \"note\":str(e)})\n            failed += 1\n\n    ensure_parent(report_csv)\n    with open(report_csv, \"w\", newline=\"\") as f:\n        w = csv.DictWriter(f, fieldnames=[\"src\",\"dst\",\"status\",\"reason\",\"note\"])\n        w.writeheader(); w.writerows(rows)\n    logger.info(\"Retag done. changed=%d, unchanged=%d, skipped=%d, failed=%d\", changed, unchanged, skipped, failed)\n    logger.info(\"Report: %s (rows=%d)\", report_csv, len(rows))\n\ndef run_bake(input_dir: str, output_dir: str, report_csv: str) -> None:\n    files = gather_dicoms(input_dir)\n    rows: List[Dict[str, Any]] = []\n    changed = unchanged = skipped = failed = 0\n\n    for src in tqdm(files, desc=\"Bake\"):\n        dst = rel_dst(input_dir, output_dir, src)\n        try:\n            ds = pydicom.dcmread(src, force=True)\n            if not modality_ok(ds, MODALITY_FILTER):\n                rows.append({\"src\": src, \"dst\": dst, \"status\":\"skipped\", \"reason\":\"modality\", \"note\":str(ds.get(MODALITY,\"\"))})\n                skipped += 1; continue\n\n            if CONDITIONAL_BAKE:\n                try:\n                    if is_already_baked(ds) and float(ds.get(RESCALE_SLOPE,1.0))==1.0 and float(ds.get(RESCALE_INTERCEPT,0.0))==0.0:\n                        ensure_parent(dst)\n                        if COPY_UNCHANGED: shutil.copy2(src, dst)\n                        rows.append({\"src\": src, \"dst\": dst, \"status\":\"unchanged\", \"reason\":\"conditional\", \"note\":\"already_baked\"})\n                        unchanged += 1; continue\n                except Exception:\n                    pass\n\n            if is_compressed(ds):\n                if not DECOMPRESS:\n                    rows.append({\"src\": src, \"dst\": dst, \"status\":\"skipped\", \"reason\":\"compressed\", \"note\":str(ds.file_meta.TransferSyntaxUID)})\n                    skipped += 1; continue\n                try:\n                    ds.decompress()\n                except Exception as de:\n                    rows.append({\"src\": src, \"dst\": dst, \"status\":\"failed\", \"reason\":\"decompress_error\", \"note\":str(de)})\n                    failed += 1; continue\n\n            sv = ds.pixel_array\n            hu = apply_modality_lut(sv, ds).astype(np.float32)\n            hu_i16, has_neg = to_int16_signed(hu)\n\n            ds[PIXEL_DATA].value = hu_i16.tobytes()\n            ds.Rows, ds.Columns = int(hu_i16.shape[-2]), int(hu_i16.shape[-1])\n            ds[BITS_ALLOCATED] = pydicom.DataElement(BITS_ALLOCATED, \"US\", 16)\n            ds[BITS_STORED]    = pydicom.DataElement(BITS_STORED,    \"US\", 16)\n            ds[HIGH_BIT]       = pydicom.DataElement(HIGH_BIT,       \"US\", 15)\n            ds[PIXEL_REPRESENTATION] = pydicom.DataElement(PIXEL_REPRESENTATION, \"US\", 1 if has_neg else 0)\n            ds[SMALLEST_PIXEL_VALUE] = pydicom.DataElement(SMALLEST_PIXEL_VALUE, \"SS\" if has_neg else \"US\", int(np.min(hu_i16)))\n            ds[LARGEST_PIXEL_VALUE]  = pydicom.DataElement(LARGEST_PIXEL_VALUE,  \"SS\" if has_neg else \"US\", int(np.max(hu_i16)))\n\n            set_rescale_tags(ds, DESIRED_INTERCEPT, DESIRED_SLOPE)\n\n            ensure_parent(dst)\n            dcmwrite(dst, ds, write_like_original=False)\n            rows.append({\"src\": src, \"dst\": dst, \"status\":\"changed\", \"reason\":\"\", \"note\":\"\"})\n            changed += 1\n        except Exception as e:\n            rows.append({\"src\": src, \"dst\": dst, \"status\":\"failed\", \"reason\":\"exception\", \"note\":str(e)})\n            failed += 1\n\n    ensure_parent(report_csv)\n    with open(report_csv, \"w\", newline=\"\") as f:\n        w = csv.DictWriter(f, fieldnames=[\"src\",\"dst\",\"status\",\"reason\",\"note\"])\n        w.writeheader(); w.writerows(rows)\n    logger.info(\"Bake done. changed=%d, unchanged=%d, skipped=%d, failed=%d\", changed, unchanged, skipped, failed)\n    logger.info(\"Report: %s (rows=%d)\", report_csv, len(rows))\n\n# ---- runner ----\nif not INPUT_DIR:\n    INPUT_DIR = auto_detect_input_dir(\"/kaggle/input\")\nlogger.info(f\"Mode={MODE} | Input={INPUT_DIR} | Output={OUTPUT_DIR}\")\nif MODE == \"audit\":\n    run_audit(INPUT_DIR, REPORT_CSV)\nelif MODE == \"retag\":\n    run_retag(INPUT_DIR, OUTPUT_DIR, REPORT_CSV)\nelif MODE == \"bake\":\n    run_bake(INPUT_DIR, OUTPUT_DIR, REPORT_CSV)\nelse:\n    raise ValueError(\"MODE must be one of: 'audit', 'retag', 'bake'\")\n","metadata":{"id":"Yz-fICg_Yd03","trusted":true,"execution":{"iopub.status.busy":"2025-08-22T13:57:36.27685Z","iopub.execute_input":"2025-08-22T13:57:36.277237Z","iopub.status.idle":"2025-08-22T16:50:08.66173Z","shell.execute_reply.started":"2025-08-22T13:57:36.277205Z","shell.execute_reply":"2025-08-22T16:50:08.660439Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Lightweight local helper script (alternative)","metadata":{}},{"cell_type":"code","source":"# File: update_dicom_rescale.py\n\nimport os\nimport pydicom\n\nRESCALE_INTERCEPT_TAG = (0x0028, 0x1052)  # RescaleIntercept\nRESCALE_SLOPE_TAG = (0x0028, 0x1053)      # RescaleSlope\n\ndef update_rescale_tags(filepath):\n    try:\n        ds = pydicom.dcmread(filepath)\n        changed = False\n\n        # Set or correct RescaleIntercept\n        if RESCALE_INTERCEPT_TAG in ds:\n            if float(ds[RESCALE_INTERCEPT_TAG].value) != 0.0:\n                ds[RESCALE_INTERCEPT_TAG].value = \"0\"\n                changed = True\n        else:\n            ds.add_new(RESCALE_INTERCEPT_TAG, \"DS\", \"0\")\n            changed = True\n\n        # Set or correct RescaleSlope\n        if RESCALE_SLOPE_TAG in ds:\n            if float(ds[RESCALE_SLOPE_TAG].value) != 1.0:\n                ds[RESCALE_SLOPE_TAG].value = \"1\"\n                changed = True\n        else:\n            ds.add_new(RESCALE_SLOPE_TAG, \"DS\", \"1\")\n            changed = True\n\n        if changed:\n            ds.save_as(filepath)\n            print(f\"Updated: {filepath}\")\n        else:\n            print(f\"No change: {filepath}\")\n\n    except Exception as e:\n        print(f\"Failed to process {filepath}: {e}\")\n\ndef process_dicom_directory(directory):\n    for root, _, files in os.walk(directory):\n        for fname in files:\n            if fname.lower().endswith(\".dcm\"):\n                fullpath = os.path.join(root, fname)\n                update_rescale_tags(fullpath)\n\nif __name__ == \"__main__\":\n    import sys\n    if len(sys.argv) != 2:\n        print(\"Usage: python update_dicom_rescale.py <dicom_directory>\")\n    else:\n        process_dicom_directory(sys.argv[1])","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}