{"cells":[{"metadata":{},"cell_type":"markdown","source":"In my [last kernel](https://www.kaggle.com/bdubreu/investigating-outlier-pixels-dicom-gotchas), I was looking at outliers scans in terms of their pixels values. We saw that +30000 pixel value was a perfectly reasonable value, because sometimes patients have a golden tooth that has a very high density.\n\nWhile doing that, we found a crappy scan. Can we find others ? Let's install some stuff and get started"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install torch torchvision feather-format pyarrow --upgrade   > /dev/null\n!pip install git+https://github.com/fastai/fastai_dev             > /dev/null\n\nfrom fastai2.basics           import *\nfrom fastai2.medical.imaging  import *\n\nnp.set_printoptions(linewidth=120)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"path_inp = Path('../input')\npath = path_inp/'rsna-intracranial-hemorrhage-detection'\npath_trn = path/'stage_1_train_images'\npath_tst = path/'stage_1_test_images'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This is the crappy file we found: \ndcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').show(figsize=(6,6))\n\n# the image is really noisy. That's a 600K+ images dataset we're looking at, so there might be more of those. Can we find them ?","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This pic has two interesting features:\n# - a std of 11269!\n# - a lower quartile of -2000\npd.Series((dcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').pixel_array.flatten())).describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### We'll start by sorting images with their stds"},{"metadata":{"trusted":true},"cell_type":"code","source":"# To understand what's going on here, please refer to https://www.kaggle.com/jhoward/some-dicom-gotchas-to-be-aware-of-fastai\npath_df = path_inp/'creating-a-metadata-dataframe'\n\ndf_lbls = pd.read_feather(path_df/'labels.fth')\ndf_tst = pd.read_feather(path_df/'df_tst.fth')\ndf_trn = pd.read_feather(path_df/'df_trn.fth')\n\ncomb = df_trn.join(df_lbls.set_index('ID'), 'SOPInstanceUID')\nassert not len(comb[comb['any'].isna()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# So, a std of 11269 was indeed a very weird value, considering the 99th percentile is 1340 \n# (that means 99% of picture have a std of 1340 or lower)\ncomb['img_std'].quantile([0.5, 0.7, 0.9, 0.99, 0.999, 0.9999])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Indeed, this is actually the only image with that kind of standard deviation\n# The second largest standard dev (in terms of pixel values) is 1513\ncomb['img_std'].sort_values(ascending=False)[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# other images with a large std (> 1500) show no signs of being corrupt\nf_name = comb[ comb['img_std'] > 1500 ].sample()['fname'].values[0]\nprint(f_name)\ndcmread(f_name).show(figsize=(6,6))\npd.Series(dcmread(f_name).pixel_array.flatten()).describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So looking for image with huge stds in terms of pixel values won't help us find other corrupted images. But it might be only because there aren't any to find. I think the std approach was correct because of the following:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# I used matplotlib to get a sense of which line started to get crappy\nimport matplotlib.pyplot as plt\nplt.imshow(dcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').pixel_array)\n# Around 300-350","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Doing some bisection search, I found the limit:\n# line 332 (indexing starts at 0) has 864 std\ndcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').pixel_array[331].std()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# but then at line 333, std in terms of pixel values starts to skyrocket:\ndcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').pixel_array[333].std()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### So I don't think we'll find another picture like the above in the dataset. I've also looked at the test set, and luckily for us there aren't any images with high std. However, I found interesting stuff while looking at i..."},{"metadata":{},"cell_type":"markdown","source":"# Train vs. Test distribution of pixel_values"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_trn['img_std'].drop(640545).describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_tst['img_std'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mean stardard deviation of the train and tests set are very different. It might start getting confusing here, so let me rephrase: on average, the standard deviation in the train set is 791.9.\nIn the train set, a typical image will see its pixel move away from their mean a lot less than a typical image in the test set. This ought to be investigated.\n\nAlso, what is an image with a std of 0 ? Are all pixels black ?"},{"metadata":{},"cell_type":"markdown","source":"# Images with low stds: corrupt and/or peculiar"},{"metadata":{"trusted":true},"cell_type":"code","source":"# we find back the image noted in this kernel : https://www.kaggle.com/tonyyy/corrupted-pixeldata-in-id-6431af929-dcm\ndf_trn[df_trn['img_std'] == 0]['fname']\n# in fact, we can't plot any of those pics because they are all corrupt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# in fact, we can't plot any of those pics because they are all corrupt\n# although only the ID_6431af929.dcm image will raise a value error\n# the four other raise index Errors, and I don't clearly know why\n# select the three following lines and then CTRL+/ to uncomment them all at once (amazing, isn't it?)\n\n# fname = df_trn[df_trn['img_std'] == 0].sample()['fname'].values[0]\n# print(fname)\n# dcmread(fname).show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How does the other stds look like ? "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Images with std between 0 and 1 are really spherical. I don't know what we should do with those ?! \n# You can press shift-enter several times here to see a bunch of examples\nfname = df_trn[(df_trn['img_std'] > 0) & (df_trn['img_std'] < 1)].sample()['fname'].values[0]\nprint(fname)\ndcmread(fname).show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# These is (only) one similar picture in the test set\nfname = df_tst[(df_tst['img_std'] > 0) & (df_tst['img_std'] < 1)].sample()['fname'].values[0]\nprint(fname)\ndcmread(fname).show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# So what does it mean, for a scan, to have high vs. low std ?\nYou'll see that they hardly look similar. In fact, std less than 300 hardly look like brains to me... Whereas 600+ std images look more \"normal\" ?"},{"metadata":{},"cell_type":"markdown","source":"### Low std images"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 4, figsize=(20,10))\nfor i, img in enumerate(comb[ comb['img_std'] < 300 ].sample(8)['fname'].values):\n    dcmread(img).show(ax=axes[i%2, i//2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Medium std images"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 4, figsize=(20,10))\nfor i, img in enumerate(comb[ comb['img_std'] > 600 ].sample(8)['fname'].values):\n    dcmread(img).show(ax=axes[i%2, i//2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### High std images"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 4, figsize=(20,10))\nfor i, img in enumerate(comb[ comb['img_std'] > 1200 ].sample(8)['fname'].values):\n    dcmread(img).show(ax=axes[i%2, i//2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Perhaps unsurprisingly, the more the std, the more detail in the picture. But I would be wary to pass the first bunch of pictures to a model... since the stds seem to be higher in the test set, maybe we can discard the low-detailed pictures ?"},{"metadata":{},"cell_type":"markdown","source":"1. ### Low std images (test set)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Problem: we have low detail images in the test set as well\nfig, axes = plt.subplots(2, 4, figsize=(20,10))\nfor i, img in enumerate(df_tst[ df_tst['img_std'] < 300 ].sample(8)['fname'].values):\n    dcmread(img).show(ax=axes[i%2, i//2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# temporary conclusions\n1. we shouldn't find other noisy pictures like the one at the top of this notebook\n2. pixel values vary much less in test data than in train data\n3. This is probably a problem, because pictures with very low std are actually very different both in SHAPE and TEXTURE than their high std counterparts. Pics with std between 0 and 1 are basically weird spheres (english grammar here: should I say different than or different to ?)\n4. Are these pictures susceptible to screw up the normalizing of the data ? I don't know... What do you think ?"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}