{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom glob import glob","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading `DICOM` image"},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = \"/kaggle/input/vinbigdata-chest-xray-abnormalities-detection\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pydicom import dcmread\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\n# I copied and adapted this code from\n# https://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way\n\ndef read_xray(path, voi_lut = True, fix_monochrome = True):\n    dicom = dcmread(path)\n    \n    # VOI LUT (if available by DICOM device) is used to transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n               \n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n        \n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n        \n    return data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = read_xray(BASE_DIR + \"/train/000434271f63a053c4128a0ba6352c7f.dicom\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nplt.imshow(data, 'gray')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = (pd.read_csv(BASE_DIR + \"/train.csv\")\n            .drop_duplicates())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"count_by_class = train_df[\"class_name\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.barh(count_by_class.index, count_by_class.values, linewidth=1, edgecolor=\"black\")\nplt.ylabel(\"Class Name\")\nplt.xlabel(\"Number of images\")\nplt.title(\"Image count by class\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As can be seen from the above graph, the dataset is imbalance in terms of classes. This will be a challenge to train models."},{"metadata":{},"cell_type":"markdown","source":"The dataset were manually annotated by a total of 17 experienced radiologists. So I guess the field `rad_id` is associated with identifiers of 17 radiologists."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The number of rads: \", train_df[\"rad_id\"].unique().shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We explore the number of images annotated per radiologist."},{"metadata":{"trusted":true},"cell_type":"code","source":"count_by_radiologist = train_df[\"rad_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.bar(count_by_radiologist.index, count_by_radiologist.values, \n        linewidth=1, edgecolor=\"black\")\nplt.xlabel(\"ID of radiologist\")\nplt.ylabel(\"Number of images\")\nplt.title(\"Image count by radiologist\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_by_radiologist_with_findings = train_df[train_df[\"class_id\"] != 14][\"rad_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.bar(count_by_radiologist_with_findings.index, count_by_radiologist_with_findings.values, \n        linewidth=1, edgecolor=\"black\")\nplt.xlabel(\"ID of radiologist\")\nplt.ylabel(\"Number of images\")\nplt.title(\"Image (with findings) count by radiologist\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, R9, R10 and R8 are three radiologists who significantly annotates the dataset and also detects most of the abnormalities. I think the labels of the datasets would be biased because R8, R9 and R10 are three dominant annotators here. They can detect some kinds of abnormalities well, but they may miss other kinds of abnormalities. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['rad_id'] == \"R8\"][\"class_name\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making a dummy submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p submissions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all are in no-finding class\npred_str = \"14 1 0 0 1 1\"\n\ntest_image_ids = [file.split(\"/\")[-1][:-6] for file in glob(BASE_DIR + \"/test/*.dicom\")]\n\npredictions = [(image_id, pred_str)\n               for image_id in test_image_ids]\n    \npredictions = pd.DataFrame(predictions, columns=[\"image_id\", \"PredictionString\"])\n\npredictions.to_csv(\"submissions/dummy_submissions.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}