{"cells":[{"metadata":{},"cell_type":"markdown","source":"Quick summary of below kernel - \n1. DICOM files are dead slow while loading, will need to convert them into something faster\n2. Null values are present only for the Bounding boxes.\n3. There is a lot of data loss if we remove all the null values from the dataset.\n4. Out of 15000 patients, we are left with only ~4000 patient ids if we remove the null value rows. \n5. There is a lot of data imbalance, cases with \"No finding\" as ground truth are drastically more than other classes(check below for bar plot)\n6. in context to height and width of images, these calues can range between 3200 - 3400 for max value and 927 - 800 for minimum value. \n7. All the DICOM images are in MONOCHROME1 type(quite obvious... it's an x-ray)\n"},{"metadata":{"id":"GvpEn1gP0PsI","trusted":true},"cell_type":"code","source":"#!pip install pydicom\n#import necessary packages \n\nimport os \nimport numpy as np \nimport pydicom \nimport matplotlib.pyplot as plt \n\n","execution_count":null,"outputs":[]},{"metadata":{"id":"K11D0ncXDgdw"},"cell_type":"markdown","source":"Reading any random file to know about the metadata inside "},{"metadata":{"id":"5h-UON6aAZ9j","outputId":"75e9dbbb-28fa-40d1-c7f7-56f44c83f21f","trusted":true},"cell_type":"code","source":"with pydicom.dcmread(\"../input/vinbigdata-chest-xray-abnormalities-detection/test/004f33259ee4aef671c2b95d54e4be68.dicom\") as ds:\n  print(ds)","execution_count":null,"outputs":[]},{"metadata":{"id":"pPncIH6kFLDK"},"cell_type":"markdown","source":"- Photometric Interpretation - MONOCHROME1, this indicates that the greyscale ranges from bright to dark with ascending pixel value\n\n- These attributes are pretty much self-explanatory, for more detailed information please refer to this site - https://dicom.innolitics.com/ciods .\nI will try to focus only on the reading and interpreting these images here. "},{"metadata":{"id":"_wvIpp63JayO","trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"okay... looks like there are some null values... let's have a look at them "},{"metadata":{"id":"VvQ9q_nQG5t0","trusted":true},"cell_type":"code","source":"def missing_values_table(df):\n        mis_val = df.isnull().sum()\n        mis_val_percent = 100 * df.isnull().sum() / len(df)\n        mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)\n        mis_val_table_ren_columns = mis_val_table.rename(\n        columns = {0 : 'Missing Values', 1 : '% of Total Values'})\n        mis_val_table_ren_columns = mis_val_table_ren_columns[\n            mis_val_table_ren_columns.iloc[:,1] != 0].sort_values(\n        '% of Total Values', ascending=False).round(1)\n        print (\"Your selected dataframe has \" + str(df.shape[1]) + \" columns.\\n\"      \n            \"There are \" + str(mis_val_table_ren_columns.shape[0]) +\n              \" columns that have missing values.\")\n        return mis_val_table_ren_columns \n    \nmissing_values = missing_values_table(train)\nprint(missing_values)","execution_count":null,"outputs":[]},{"metadata":{"id":"RQCvm9kCKTKW"},"cell_type":"markdown","source":"46% of the data is missing, this is ... not good... will have to look at that. \nIn other way we can say that we do not have bounding boxes for 46% of the data present, maybe we can treat this as a semi-supervides learning problem([link to research paper](https://arxiv.org/pdf/2005.07377.pdf))... or...  weakly-supervised learning problem(if that's a thing)\n\n\n\"SOME RANDOM THOUGHTS\"\n\nHow to tackle it? - \nThe ground truth here is the \"class_name\".. which certainly big time depends on the anchor boxes(which are missing big time here... ). \n\nWe can divide this training dataset into two sections, one which has the values for bounding boxes as well as the ground truth, accounts for 37000 images, validate on the remaining 31000 images and try to correct the error.we cannot do normal splitting strategies as we follow in other classification problems, has a hardcore data splitting will be done.\n\n(just a thought, i maybe wrong)"},{"metadata":{},"cell_type":"markdown","source":"If it happend that all the missing values are from the same rows, we can create a new dataframe our of the ones which has allt he information present. For this purpose there is a python module called missingno ([github repo](https://github.com/ResidentMario/missingno)). Go ahead and have a look, cool stuff."},{"metadata":{"id":"h2tIsZIKM8E-","outputId":"5584c912-d7f9-400d-9782-da7a47e92bd6","trusted":true,"collapsed":true},"cell_type":"code","source":"!pip install quilt\n!quilt install ResidentMario/missingno_data\n\n\n#github repo - https://github.com/ResidentMario/missingno","execution_count":null,"outputs":[]},{"metadata":{"id":"apJ58VztNVZi","outputId":"1f5b3a04-063a-4068-97a0-ba5ef4f1c911","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib\nimport missingno as msno\n%matplotlib inline\n\ndf = train\nmissingdata_df = df.columns[df.isnull().any()].tolist()\nmsno.matrix(df[missingdata_df])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well... the missing data is only about the bounding boxes, nothing about the ground truth, we can eliminate this and create a new dataset here... with no null values... but we will be keeping both of these datasets, will see at the time of modeling how to use both of these efficiently. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train['class_id'].unique()))\nprint(len(updated_data['class_id'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"id":"uRupGG-ANn8J","trusted":true},"cell_type":"code","source":"updated_data = train.dropna(how = \"any\", inplace = False)\nupdated_data.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"M-KYuDWCN8LI","outputId":"1d1f8757-1a01-4cc8-a3df-6e541213e27c","trusted":true},"cell_type":"code","source":"missing = missing_values_table(updated_data)\nprint(missing)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cool... no missing values... easy life.. "},{"metadata":{},"cell_type":"markdown","source":"A big issue... How many images are we left with...??\nI mean we did actually drop the missing data rows, and most probably we are missing out some images(not omitting them though, ther are still present in the previous dataframe).. so out of 18000 images... how many are we left with ? "},{"metadata":{"trusted":true},"cell_type":"code","source":"len(updated_data['image_id'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"okay... not good... this is an important thing to take care of, out of 15000 unique patient scans present we are having just 4394 scans that have Anchor boxes in them."},{"metadata":{"id":"8mm_HuT2UJIg","trusted":true},"cell_type":"code","source":"updated_data[['x_min', 'y_min','x_max', 'y_max']].describe(percentiles=[0.25, 0.5, 0.75, .95])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train['image_id'].unique())","execution_count":null,"outputs":[]},{"metadata":{"id":"HadkPK7pUJnK","outputId":"2d782d89-8758-43e5-a06f-f4556fe0b5b9","trusted":true,"collapsed":true},"cell_type":"code","source":"folder = '../input/vinbigdata-chest-xray-abnormalities-detection/train/'\nfiles = os.listdir(folder)\nsex = []\nwidth = []\nheight = []\nfilename = []\nfor i in range(len(files)):\n  print(files[i])\n  with pydicom.dcmread(folder + files[i]) as ds:\n    sex.append(ds.PatientSex)\n    width.append(ds.Columns)\n    height.append(ds.Rows)\n    filename.append(files[i])\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_values = {'sex':sex , 'height':height , 'width':width}\nunique_values_dataframe = pd.DataFrame.from_dict(unique_values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_values_dataframe['sex'].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"let's try to plot this and see the distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.countplot(data=unique_values_dataframe ,y='sex')\nplt.title('Sex distribution ',fontsize=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_values_dataframe[['height', 'width']].describe(percentiles=[0.25, 0.5, 0.75, .95])\n","execution_count":null,"outputs":[]},{"metadata":{"id":"I9pyeJHMUJbZ","outputId":"927cc02d-d9d5-4f3e-b56c-922f7b5e7018","trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(10,5))\nsns.countplot(data=train ,y='class_name')\nplt.title('Counts of the Classes',fontsize=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}