{
  "id": 442410,
  "title": "Possible memory leak in pydicom",
  "url": "/competitions/rsna-2023-abdominal-trauma-detection/discussion/442410",
  "author_name": "Arpit",
  "post_date": "2023-09-22T13:47:49.752000",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>So I've been facing a memory leak issue while using a custom training loop for a machine learning model, After debugging I traced the leak to one part of my code, Below is the simplification of the code which reproduces the issue.</p>\n<pre><code>df = pd.read_csv()\n\npatient_id_data = df[].values\n\nuniversal_path = \n\n\n\n i  (, ):\n    ( + (i))\n    patient_id = patient_id_data[i]\n\n    \n    patient_path = universal_path + (patient_id) + \n    picture_path_array = []\n\n     os.scandir(patient_path)  entries:\n             entry  entries:\n                 entry.is_dir():\n                    picture_path_array.append(entry.name)\n\n    \n     i  picture_path_array:\n        dcm_file_path = patient_path + i + \n\n\n        \n        dcm_array = []\n\n         filename  os.listdir(dcm_file_path):\n                 filename.endswith():\n                    dcm_array.append(filename)\n\n\n        \n        \n         i  ():\n            final_path = dcm_file_path + dcm_array[i]\n            dicom = pydicom.dcmread(final_path)\n\n             dicom\n</code></pre>\n<p>Iterating through 40 different patients seem to cause approximately 200MB of memory leak. One thing I have noticed is that no memory is leaked if it is reading the same <code>.dcm</code> file that it read before. Using <code>del dicom</code> and <code>gc.collect()</code> does not seem to help.</p>\n<p>I have tried using a library called <code>dicomsdl</code> and it has the same memory leak issue although in much less magnitude.</p>\n<p>If you want to recreate the issue then you can copy this code and run it, After it has completed execution change the first for loop's range values to <code>(40, 80)</code> and so on to pile the memory leak.</p>",
  "messages": [
    {
      "id": 2451383,
      "postDate": "2023-09-22T13:47:49.753Z",
      "content": "<p>So I've been facing a memory leak issue while using a custom training loop for a machine learning model, After debugging I traced the leak to one part of my code, Below is the simplification of the code which reproduces the issue.</p>\n<pre><code>df = pd.read_csv()\n\npatient_id_data = df[].values\n\nuniversal_path = \n\n\n\n i  (, ):\n    ( + (i))\n    patient_id = patient_id_data[i]\n\n    \n    patient_path = universal_path + (patient_id) + \n    picture_path_array = []\n\n     os.scandir(patient_path)  entries:\n             entry  entries:\n                 entry.is_dir():\n                    picture_path_array.append(entry.name)\n\n    \n     i  picture_path_array:\n        dcm_file_path = patient_path + i + \n\n\n        \n        dcm_array = []\n\n         filename  os.listdir(dcm_file_path):\n                 filename.endswith():\n                    dcm_array.append(filename)\n\n\n        \n        \n         i  ():\n            final_path = dcm_file_path + dcm_array[i]\n            dicom = pydicom.dcmread(final_path)\n\n             dicom\n</code></pre>\n<p>Iterating through 40 different patients seem to cause approximately 200MB of memory leak. One thing I have noticed is that no memory is leaked if it is reading the same <code>.dcm</code> file that it read before. Using <code>del dicom</code> and <code>gc.collect()</code> does not seem to help.</p>\n<p>I have tried using a library called <code>dicomsdl</code> and it has the same memory leak issue although in much less magnitude.</p>\n<p>If you want to recreate the issue then you can copy this code and run it, After it has completed execution change the first for loop's range values to <code>(40, 80)</code> and so on to pile the memory leak.</p>",
      "rawMarkdown": "So I've been facing a memory leak issue while using a custom training loop for a machine learning model, After debugging I traced the leak to one part of my code, Below is the simplification of the code which reproduces the issue.\n\n```python\ndf = pd.read_csv(\"/kaggle/input/rsna-2023-abdominal-trauma-detection/train.csv\")\n\npatient_id_data = df['patient_id'].values\n\nuniversal_path = \"/kaggle/input/rsna-2023-abdominal-trauma-detection/train_images/\"\n\n\n# Looping through 40 different patients\nfor i in range(0, 40):\n    print(\"On: \" + str(i))\n    patient_id = patient_id_data[i]\n    \n    # Getting folder path, Example: train_images/10004/21057\n    patient_path = universal_path + str(patient_id) + \"/\"\n    picture_path_array = []\n    \n    with os.scandir(patient_path) as entries:\n            for entry in entries:\n                if entry.is_dir():\n                    picture_path_array.append(entry.name)\n    \n    # Looping through each folder path a patient has, For example 10004 has 2 paths, 21057 and 51033\n    for i in picture_path_array:\n        dcm_file_path = patient_path + i + \"/\"\n        \n        \n        # Adding all dcm file names to this list\n        dcm_array = []\n        \n        for filename in os.listdir(dcm_file_path):\n                if filename.endswith(\".dcm\"):\n                    dcm_array.append(filename)\n\n                    \n        # Getting first 40 DCM images from a folder\n        # Note: This may be an inaccurate method to train a model, using first 40 images is just for example and explaining purposes\n        for i in range(40):\n            final_path = dcm_file_path + dcm_array[i]\n            dicom = pydicom.dcmread(final_path)\n            \n            del dicom\n            \n```\n\nIterating through 40 different patients seem to cause approximately 200MB of memory leak. One thing I have noticed is that no memory is leaked if it is reading the same `.dcm` file that it read before. Using `del dicom` and `gc.collect()` does not seem to help.\n\nI have tried using a library called `dicomsdl` and it has the same memory leak issue although in much less magnitude.\n\nIf you want to recreate the issue then you can copy this code and run it, After it has completed execution change the first for loop's range values to `(40, 80)` and so on to pile the memory leak.",
      "votes": 4
    },
    {
      "id": 2463100,
      "postDate": "2023-09-30T23:05:44.240Z",
      "content": "<p>it looks like the array dcm_array never gets removed. Theoretically, it should go away after the loop ends, but maybe not.</p>",
      "rawMarkdown": "it looks like the array dcm_array never gets removed. Theoretically, it should go away after the loop ends, but maybe not.",
      "votes": 1,
      "replies": [
        {
          "id": 2463875,
          "postDate": "2023-10-01T16:28:46.717Z",
          "content": "<p>I highly appreciate the response!</p>\n<p>Adding <code>del dcm_array</code> after the for loop which reads the <code>.dcm</code> file does not seem to help in reducing the memory leak, After testing it is still the same amount as before.</p>",
          "rawMarkdown": "I highly appreciate the response!\n\nAdding `del dcm_array` after the for loop which reads the `.dcm` file does not seem to help in reducing the memory leak, After testing it is still the same amount as before."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2463100,
      "author_name": "David Roberts",
      "author_url": "",
      "post_date": "2023-09-30T23:05:44.240000",
      "content": "<p>it looks like the array dcm_array never gets removed. Theoretically, it should go away after the loop ends, but maybe not.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2463875,
          "author_name": "Arpit",
          "author_url": "",
          "post_date": "2023-10-01T16:28:46.717000",
          "content": "<p>I highly appreciate the response!</p>\n<p>Adding <code>del dcm_array</code> after the for loop which reads the <code>.dcm</code> file does not seem to help in reducing the memory leak, After testing it is still the same amount as before.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2451383": "So I've been facing a memory leak issue while using a custom training loop for a machine learning model, After debugging I traced the leak to one part of my code, Below is the simplification of the code which reproduces the issue.\n\n```python\ndf = pd.read_csv(\"/kaggle/input/rsna-2023-abdominal-trauma-detection/train.csv\")\n\npatient_id_data = df['patient_id'].values\n\nuniversal_path = \"/kaggle/input/rsna-2023-abdominal-trauma-detection/train_images/\"\n\n\n# Looping through 40 different patients\nfor i in range(0, 40):\n    print(\"On: \" + str(i))\n    patient_id = patient_id_data[i]\n    \n    # Getting folder path, Example: train_images/10004/21057\n    patient_path = universal_path + str(patient_id) + \"/\"\n    picture_path_array = []\n    \n    with os.scandir(patient_path) as entries:\n            for entry in entries:\n                if entry.is_dir():\n                    picture_path_array.append(entry.name)\n    \n    # Looping through each folder path a patient has, For example 10004 has 2 paths, 21057 and 51033\n    for i in picture_path_array:\n        dcm_file_path = patient_path + i + \"/\"\n        \n        \n        # Adding all dcm file names to this list\n        dcm_array = []\n        \n        for filename in os.listdir(dcm_file_path):\n                if filename.endswith(\".dcm\"):\n                    dcm_array.append(filename)\n\n                    \n        # Getting first 40 DCM images from a folder\n        # Note: This may be an inaccurate method to train a model, using first 40 images is just for example and explaining purposes\n        for i in range(40):\n            final_path = dcm_file_path + dcm_array[i]\n            dicom = pydicom.dcmread(final_path)\n            \n            del dicom\n            \n```\n\nIterating through 40 different patients seem to cause approximately 200MB of memory leak. One thing I have noticed is that no memory is leaked if it is reading the same `.dcm` file that it read before. Using `del dicom` and `gc.collect()` does not seem to help.\n\nI have tried using a library called `dicomsdl` and it has the same memory leak issue although in much less magnitude.\n\nIf you want to recreate the issue then you can copy this code and run it, After it has completed execution change the first for loop's range values to `(40, 80)` and so on to pile the memory leak.",
    "2463100": "it looks like the array dcm_array never gets removed. Theoretically, it should go away after the loop ends, but maybe not."
  }
}