![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2051313%2Fbc6ec5cb6e0874dedb25052c599c0973%2Fparallel_normalization.png?generation=1610571528559536&alt=media\")
I'm trying to assemble some of the DICOM header information into a dataset. However, it takes quite a long time to run.
\nHere is the relevant code:
\nall_data = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\nsampled_data = all_data.sample(frac=0.25)\ntrain_dir = '../input/vinbigdata-chest-xray-abnormalities-detection/train'\nsexes = []\nages = []\nweights = []\nsizes = []\nfor index, row in sampled_data.iterrows():\n image_id = row['image_id']\n file_path = train_dir + \"/\" + image_id + '.dicom'\n dcm = pydicom.dcmread(file_path)\n sex = get_patients_sex(dcm)\n sexes.append(sex)\n age = get_patients_age(dcm)\n ages.append(age)\n weight = get_patients_weight(dcm)\n weights.append(weight)\n size = get_patients_size(dcm)\n sizes.append(size)\nsampled_data['sex'] = sexes\nsampled_data['age'] = ages\nsampled_data['weight'] = weights\nsampled_data['size'] = sizes\n\nsampled_data.to_csv(\"sampled_data.csv\")\nIf I sample 1/4 of the data, this will run in an hour, which I consider acceptable. However, if I run it on all the code, the notebook will go to sleep before finishing.
\nIs there a way to make this code more efficient? The complete notebook is here.
", "messages": [ { "id": 1152029, "postDate": "2021-01-13T17:57:55.320Z", "content": "@paulreiners if all you want is the dicom metadata, then you can use the stop_before_pixels=True option in the dcmread function so that it doesn't have to load all the image data into memory. So in your code where you read the file, it would become:
dcm = pydicom.dcmread(file_path, stop_before_pixels=True)\nI do the same thing in one of my notebooks where all I wanted was the metadata:
\n\nIt speeds up reading the dicom files to the point where I can read all the training dicoms in about 300 seconds.
", "rawMarkdown": "@paulreiners if all you want is the dicom metadata, then you can use the `stop_before_pixels=True` option in the `dcmread` function so that it doesn't have to load all the image data into memory. So in your code where you read the file, it would become:\n\n dcm = pydicom.dcmread(file_path, stop_before_pixels=True)\n\nI do the same thing in one of my notebooks where all I wanted was the metadata:\n\n* https://www.kaggle.com/craigmthomas/localization-of-findings\n\nIt speeds up reading the dicom files to the point where I can read all the training dicoms in about 300 seconds.", "votes": 5 }, { "id": 1152137, "postDate": "2021-01-13T21:02:38.393Z", "content": "@paulreiners also, if you want to read the dicoms in a more parallel fashion, you can use the joblib package to help out. The Kaggle VMs have 2 processor cores and 4 threads, meaning that you can get 4 parallel processes working at once. I have an extract function that takes an image_id and then extracts useful information from the dicom file such as down scaling the pixel data, width, height, etc., and returns it as a tuple. Here's what it looks like:
When complete, the variable data will be a list of tuples, with each tuple containing the image_id in position 0, a resized pixel_array in position 1, width in position 2, height in position 3, and other information I extracted from the dicom file. As you can see, it takes about 82 minutes to process all the dicom files. To make things easier to work with, you can convert the resultant list back into a dictionary for easier access by doing:
data_dict = {\n i[0]:\n {\n \"pixel_array\": i[1],\n \"width\": i[2],\n \"height\": i[3]\n ...\n } for i in data\n}\nThen, for example, to get the pixel data for image id 1234567890 it would be data_dict[\"1234567890\"][\"pixel_array\"]. Hope this helps.
I'm trying to assemble some of the DICOM header information into a dataset. However, it takes quite a long time to run.
\nHere is the relevant code:
\nall_data = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\nsampled_data = all_data.sample(frac=0.25)\ntrain_dir = '../input/vinbigdata-chest-xray-abnormalities-detection/train'\nsexes = []\nages = []\nweights = []\nsizes = []\nfor index, row in sampled_data.iterrows():\n image_id = row['image_id']\n file_path = train_dir + \"/\" + image_id + '.dicom'\n dcm = pydicom.dcmread(file_path)\n sex = get_patients_sex(dcm)\n sexes.append(sex)\n age = get_patients_age(dcm)\n ages.append(age)\n weight = get_patients_weight(dcm)\n weights.append(weight)\n size = get_patients_size(dcm)\n sizes.append(size)\nsampled_data['sex'] = sexes\nsampled_data['age'] = ages\nsampled_data['weight'] = weights\nsampled_data['size'] = sizes\n\nsampled_data.to_csv(\"sampled_data.csv\")\nIf I sample 1/4 of the data, this will run in an hour, which I consider acceptable. However, if I run it on all the code, the notebook will go to sleep before finishing.
\nIs there a way to make this code more efficient? The complete notebook is here.
", "rawMarkdown": "I'm trying to assemble some of the DICOM header information into a dataset. However, it takes quite a long time to run.\n\nHere is the relevant code:\n\n```\nall_data = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\nsampled_data = all_data.sample(frac=0.25)\ntrain_dir = '../input/vinbigdata-chest-xray-abnormalities-detection/train'\nsexes = []\nages = []\nweights = []\nsizes = []\nfor index, row in sampled_data.iterrows():\n image_id = row['image_id']\n file_path = train_dir + \"/\" + image_id + '.dicom'\n dcm = pydicom.dcmread(file_path)\n sex = get_patients_sex(dcm)\n sexes.append(sex)\n age = get_patients_age(dcm)\n ages.append(age)\n weight = get_patients_weight(dcm)\n weights.append(weight)\n size = get_patients_size(dcm)\n sizes.append(size)\nsampled_data['sex'] = sexes\nsampled_data['age'] = ages\nsampled_data['weight'] = weights\nsampled_data['size'] = sizes\n\nsampled_data.to_csv(\"sampled_data.csv\")\n```\n\nIf I sample 1/4 of the data, this will run in an hour, which I consider acceptable. However, if I run it on all the code, the notebook will go to sleep before finishing.\n\nIs there a way to make this code more efficient? The complete notebook is [here](https://www.kaggle.com/paulreiners/prepare-dicom-images-for-ml).", "votes": 2 } ], "comments": [ { "id": 1152029, "author_name": "Craig Thomas", "author_url": "", "post_date": "2021-01-13T17:57:55.320000", "content": "@paulreiners if all you want is the dicom metadata, then you can use the stop_before_pixels=True option in the dcmread function so that it doesn't have to load all the image data into memory. So in your code where you read the file, it would become:
dcm = pydicom.dcmread(file_path, stop_before_pixels=True)\nI do the same thing in one of my notebooks where all I wanted was the metadata:
\n\nIt speeds up reading the dicom files to the point where I can read all the training dicoms in about 300 seconds.
", "votes": 5, "replies": [] }, { "id": 1152137, "author_name": "Craig Thomas", "author_url": "", "post_date": "2021-01-13T21:02:38.393000", "content": "@paulreiners also, if you want to read the dicoms in a more parallel fashion, you can use the joblib package to help out. The Kaggle VMs have 2 processor cores and 4 threads, meaning that you can get 4 parallel processes working at once. I have an extract function that takes an image_id and then extracts useful information from the dicom file such as down scaling the pixel data, width, height, etc., and returns it as a tuple. Here's what it looks like:
When complete, the variable data will be a list of tuples, with each tuple containing the image_id in position 0, a resized pixel_array in position 1, width in position 2, height in position 3, and other information I extracted from the dicom file. As you can see, it takes about 82 minutes to process all the dicom files. To make things easier to work with, you can convert the resultant list back into a dictionary for easier access by doing:
data_dict = {\n i[0]:\n {\n \"pixel_array\": i[1],\n \"width\": i[2],\n \"height\": i[3]\n ...\n } for i in data\n}\nThen, for example, to get the pixel data for image id 1234567890 it would be data_dict[\"1234567890\"][\"pixel_array\"]. Hope this helps.