My RSNADataset is causing severe training bottlenecks due to real-time DICOM processing in Kaggle GPU environment:
():\n ():\n row = .df.iloc[idx]\n series_path = \n volume = process_dicom_series_safe(series_path, target_shape=(, , ))\n x = torch.from_numpy(volume).()\n x, y\npydicom.dcmread())np.percentile(), resizing to 384×384)scipy.ndimage.zoom)My DICOMPreprocessorKaggle class handles:
.dcm files per series with os.walk()p1, p99 = np.percentile(img, [1, 99])cv2.resize(processed_img, (384, 384))ndimage.zoom() for final (32, 384, 384) shapegc.collect().npy files beforehand to avoid real-time processing?pydicom + scipy.ndimage combination?num_workers in Kaggle? (Currently using 0 due to multiprocessing issues)num_workers=0 (multiprocessing causes issues in Kaggle environment)torch.cuda.empty_cache() and memory cleanupLooking for practical solutions to get batch loading under 1 second in Kaggle GPU environment. Current GPU utilization is poor due to waiting for data loading.
\nAny medical imaging competition veterans with experience optimizing DICOM workflows on Kaggle GPU instances?
\nTags: #rsna2024 #dicom #pytorch #dataloader #performance #kaggle-gpu
My RSNADataset is causing severe training bottlenecks due to real-time DICOM processing in Kaggle GPU environment:
():\n ():\n row = .df.iloc[idx]\n series_path = \n volume = process_dicom_series_safe(series_path, target_shape=(, , ))\n x = torch.from_numpy(volume).()\n x, y\npydicom.dcmread())np.percentile(), resizing to 384×384)scipy.ndimage.zoom)My DICOMPreprocessorKaggle class handles:
.dcm files per series with os.walk()p1, p99 = np.percentile(img, [1, 99])cv2.resize(processed_img, (384, 384))ndimage.zoom() for final (32, 384, 384) shapegc.collect().npy files beforehand to avoid real-time processing?pydicom + scipy.ndimage combination?num_workers in Kaggle? (Currently using 0 due to multiprocessing issues)num_workers=0 (multiprocessing causes issues in Kaggle environment)torch.cuda.empty_cache() and memory cleanupLooking for practical solutions to get batch loading under 1 second in Kaggle GPU environment. Current GPU utilization is poor due to waiting for data loading.
\nAny medical imaging competition veterans with experience optimizing DICOM workflows on Kaggle GPU instances?
\nTags: #rsna2024 #dicom #pytorch #dataloader #performance #kaggle-gpu
Hi, I am struggling with the same issue. I have come across dicomsdl, which at least loads tht pixel-data much faster than pydicom. You can check out this notebook:
\nhttps://www.kaggle.com/code/thalro/test-read-speed
\nIt is still painfully slow. If you load all of the slices it takes around 1.5 hours in a kaggle notebook.
Hi, are you facing this data loading issue while inferencing as well?
", "rawMarkdown": "Hi, are you facing this data loading issue while inferencing as well?" }, { "id": 3295228, "postDate": "2025-09-28T04:35:17.377Z", "content": "You can convert DCM volumes to NIFTI, then use them to train.
", "rawMarkdown": "You can convert DCM volumes to NIFTI, then use them to train. " }, { "id": 3294935, "postDate": "2025-09-27T06:44:48.733Z", "content": "Parallel loading of dicom files is possible. Use that to load images, it's much faster
", "rawMarkdown": "Parallel loading of dicom files is possible. Use that to load images, it's much faster" } ], "comments": [ { "id": 3294471, "author_name": "thomas rost", "author_url": "", "post_date": "2025-09-26T08:25:30.213000", "content": "Hi, I am struggling with the same issue. I have come across dicomsdl, which at least loads tht pixel-data much faster than pydicom. You can check out this notebook:
\nhttps://www.kaggle.com/code/thalro/test-read-speed
\nIt is still painfully slow. If you load all of the slices it takes around 1.5 hours in a kaggle notebook.
Hi, are you facing this data loading issue while inferencing as well?
", "votes": 0, "replies": [] }, { "id": 3295228, "author_name": "Saroj Neupane", "author_url": "", "post_date": "2025-09-28T04:35:17.377000", "content": "You can convert DCM volumes to NIFTI, then use them to train.
", "votes": 0, "replies": [] }, { "id": 3294935, "author_name": "jucious", "author_url": "", "post_date": "2025-09-27T06:44:48.733000", "content": "Parallel loading of dicom files is possible. Use that to load images, it's much faster
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3294219": "### **Problem**\nMy `RSNADataset` is causing severe training bottlenecks due to real-time DICOM processing in Kaggle GPU environment:\n```python\nclass RSNADataset(Dataset):\n def __getitem__(self, idx):\n row = self.df.iloc[idx]\n series_path = f\"{self.series_root}/{row[ID_COL]}\"\n volume = process_dicom_series_safe(series_path, target_shape=(32, 384, 384))\n x = torch.from_numpy(volume).float()\n return x, y\n```\n### **Performance Issues**\n- **Per sample:** 2-8 seconds \n- **Batch size 8:** 16-64 seconds per batch\n- **Dataset scale:** 4,348 series × 200-900 DICOM files each\n- **Training time:** Would take 3-9 hours just for data loading per epoch\n- **Environment:** Kaggle notebook with Tesla T4 GPU (16GB)\n**Bottleneck breakdown (tested on Kaggle):**\n1. **File I/O:** 0.5-2.0s (loading hundreds of DICOM files per series)\n2. **DICOM parsing:** 0.2-0.8s (pydicom processing with `pydicom.dcmread()`)\n3. **Image processing:** 0.3-1.0s (statistical normalization using `np.percentile()`, resizing to 384×384)\n4. **3D volume creation:** 0.1-0.3s (stacking slices with `scipy.ndimage.zoom`)\n### **Current DICOM Processing Pipeline**\nMy `DICOMPreprocessorKaggle` class handles:\n- Loading 200-900 `.dcm` files per series with `os.walk()`\n- Statistical normalization: `p1, p99 = np.percentile(img, [1, 99])`\n- 2D resizing: `cv2.resize(processed_img, (384, 384))`\n- 3D volume creation: `ndimage.zoom()` for final (32, 384, 384) shape\n- Memory cleanup with `gc.collect()`\n### **Questions**\n1. **Pre-processing approach:** Should I convert all DICOM series to `.npy` files beforehand to avoid real-time processing?\n2. **Kaggle GPU optimization:** Any specific strategies for Tesla T4 environment with limited session time?\n3. **Alternative libraries:** Faster alternatives to `pydicom` + `scipy.ndimage` combination?\n4. **DataLoader settings:** Best practices for `num_workers` in Kaggle? (Currently using 0 due to multiprocessing issues)\n### **What I've Tried**\n- `num_workers=0` (multiprocessing causes issues in Kaggle environment)\n- Small batch sizes (8) for GPU memory management\n- `torch.cuda.empty_cache()` and memory cleanup\n- Mixed precision training to optimize GPU usage\n### **Target Goal**\nLooking for practical solutions to get batch loading under 1 second in Kaggle GPU environment. Current GPU utilization is poor due to waiting for data loading.\nAny medical imaging competition veterans with experience optimizing DICOM workflows on Kaggle GPU instances?\n**Tags:** #rsna2024 #dicom #pytorch #dataloader #performance #kaggle-gpu", "3294471": "Hi, I am struggling with the same issue. I have come across `dicomsdl`, which at least loads tht pixel-data much faster than `pydicom`. You can check out this notebook:\nhttps://www.kaggle.com/code/thalro/test-read-speed\nIt is still painfully slow. If you load all of the slices it takes around 1.5 hours in a kaggle notebook.", "3295237": "Hi, are you facing this data loading issue while inferencing as well?", "3295228": "You can convert DCM volumes to NIFTI, then use them to train. ", "3294935": "Parallel loading of dicom files is possible. Use that to load images, it's much faster" } }