{
  "id": 207834,
  "title": "Faster way to read DICOM data (?)",
  "url": "/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/207834",
  "author_name": "Ultron",
  "post_date": "2020-12-31T13:31:19.013000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I am using pydicom with joblib to just read the dicom files. Here is the code snippet:</p>\n<pre><code>import pydicom as pdc\nfrom joblib import Parallel, delayed\n\ndef read_dicom_df(fn):\n    _ = pdc.read_file(os.path.join(train_dir, fn))\n    pass\n\nParallel(-1, verbose=1)(delayed(read_dicom_df)(x) for x in train_files)\n</code></pre>\n<p>Here is the output snipped:</p>\n<pre><code>[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:    5.8s\n[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   12.1s\n[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   22.2s\n[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:   36.2s\n[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:   54.9s\n[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed:  1.3min\n[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed:  1.7min\n[Parallel(n_jobs=-1)]: Done 3192 tasks      | elapsed:  2.2min\n[Parallel(n_jobs=-1)]: Done 4042 tasks      | elapsed:  2.8min\n[Parallel(n_jobs=-1)]: Done 4992 tasks      | elapsed:  3.4min\n[Parallel(n_jobs=-1)]: Done 6042 tasks      | elapsed:  4.2min\n[Parallel(n_jobs=-1)]: Done 7192 tasks      | elapsed:  4.9min\n[Parallel(n_jobs=-1)]: Done 9336 tasks      | elapsed:  6.4min\n[Parallel(n_jobs=-1)]: Done 12036 tasks      | elapsed:  8.2min\n[Parallel(n_jobs=-1)]: Done 14936 tasks      | elapsed: 10.2min\n[Parallel(n_jobs=-1)]: Done 14993 out of 15000 | elapsed: 10.2min remaining:    0.3s\n[Parallel(n_jobs=-1)]: Done 15000 out of 15000 | elapsed: 10.3min finished\n</code></pre>\n<p>And it's working in around 1400+-50 tasks/min. Any faster method to accomplish the speed?</p>",
  "messages": [
    {
      "id": 1133737,
      "postDate": "2020-12-31T13:31:19.013Z",
      "content": "<p>I am using pydicom with joblib to just read the dicom files. Here is the code snippet:</p>\n<pre><code>import pydicom as pdc\nfrom joblib import Parallel, delayed\n\ndef read_dicom_df(fn):\n    _ = pdc.read_file(os.path.join(train_dir, fn))\n    pass\n\nParallel(-1, verbose=1)(delayed(read_dicom_df)(x) for x in train_files)\n</code></pre>\n<p>Here is the output snipped:</p>\n<pre><code>[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:    5.8s\n[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   12.1s\n[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   22.2s\n[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:   36.2s\n[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:   54.9s\n[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed:  1.3min\n[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed:  1.7min\n[Parallel(n_jobs=-1)]: Done 3192 tasks      | elapsed:  2.2min\n[Parallel(n_jobs=-1)]: Done 4042 tasks      | elapsed:  2.8min\n[Parallel(n_jobs=-1)]: Done 4992 tasks      | elapsed:  3.4min\n[Parallel(n_jobs=-1)]: Done 6042 tasks      | elapsed:  4.2min\n[Parallel(n_jobs=-1)]: Done 7192 tasks      | elapsed:  4.9min\n[Parallel(n_jobs=-1)]: Done 9336 tasks      | elapsed:  6.4min\n[Parallel(n_jobs=-1)]: Done 12036 tasks      | elapsed:  8.2min\n[Parallel(n_jobs=-1)]: Done 14936 tasks      | elapsed: 10.2min\n[Parallel(n_jobs=-1)]: Done 14993 out of 15000 | elapsed: 10.2min remaining:    0.3s\n[Parallel(n_jobs=-1)]: Done 15000 out of 15000 | elapsed: 10.3min finished\n</code></pre>\n<p>And it's working in around 1400+-50 tasks/min. Any faster method to accomplish the speed?</p>",
      "rawMarkdown": "I am using pydicom with joblib to just read the dicom files. Here is the code snippet:\n\n```\nimport pydicom as pdc\nfrom joblib import Parallel, delayed\n\ndef read_dicom_df(fn):\n    _ = pdc.read_file(os.path.join(train_dir, fn))\n    pass\n\nParallel(-1, verbose=1)(delayed(read_dicom_df)(x) for x in train_files)\n```\nHere is the output snipped:\n```\n[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:    5.8s\n[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   12.1s\n[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   22.2s\n[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:   36.2s\n[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:   54.9s\n[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed:  1.3min\n[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed:  1.7min\n[Parallel(n_jobs=-1)]: Done 3192 tasks      | elapsed:  2.2min\n[Parallel(n_jobs=-1)]: Done 4042 tasks      | elapsed:  2.8min\n[Parallel(n_jobs=-1)]: Done 4992 tasks      | elapsed:  3.4min\n[Parallel(n_jobs=-1)]: Done 6042 tasks      | elapsed:  4.2min\n[Parallel(n_jobs=-1)]: Done 7192 tasks      | elapsed:  4.9min\n[Parallel(n_jobs=-1)]: Done 9336 tasks      | elapsed:  6.4min\n[Parallel(n_jobs=-1)]: Done 12036 tasks      | elapsed:  8.2min\n[Parallel(n_jobs=-1)]: Done 14936 tasks      | elapsed: 10.2min\n[Parallel(n_jobs=-1)]: Done 14993 out of 15000 | elapsed: 10.2min remaining:    0.3s\n[Parallel(n_jobs=-1)]: Done 15000 out of 15000 | elapsed: 10.3min finished\n```\nAnd it's working in around 1400+-50 tasks/min. Any faster method to accomplish the speed?",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1133737": "I am using pydicom with joblib to just read the dicom files. Here is the code snippet:\n\n```\nimport pydicom as pdc\nfrom joblib import Parallel, delayed\n\ndef read_dicom_df(fn):\n    _ = pdc.read_file(os.path.join(train_dir, fn))\n    pass\n\nParallel(-1, verbose=1)(delayed(read_dicom_df)(x) for x in train_files)\n```\nHere is the output snipped:\n```\n[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:    5.8s\n[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   12.1s\n[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:   22.2s\n[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:   36.2s\n[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:   54.9s\n[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed:  1.3min\n[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed:  1.7min\n[Parallel(n_jobs=-1)]: Done 3192 tasks      | elapsed:  2.2min\n[Parallel(n_jobs=-1)]: Done 4042 tasks      | elapsed:  2.8min\n[Parallel(n_jobs=-1)]: Done 4992 tasks      | elapsed:  3.4min\n[Parallel(n_jobs=-1)]: Done 6042 tasks      | elapsed:  4.2min\n[Parallel(n_jobs=-1)]: Done 7192 tasks      | elapsed:  4.9min\n[Parallel(n_jobs=-1)]: Done 9336 tasks      | elapsed:  6.4min\n[Parallel(n_jobs=-1)]: Done 12036 tasks      | elapsed:  8.2min\n[Parallel(n_jobs=-1)]: Done 14936 tasks      | elapsed: 10.2min\n[Parallel(n_jobs=-1)]: Done 14993 out of 15000 | elapsed: 10.2min remaining:    0.3s\n[Parallel(n_jobs=-1)]: Done 15000 out of 15000 | elapsed: 10.3min finished\n```\nAnd it's working in around 1400+-50 tasks/min. Any faster method to accomplish the speed?"
  }
}