{"cells":[{"metadata":{},"cell_type":"markdown","source":"This work builds on top of Jeremy Howard's amazing kernel \"Some DICOM gotchas to be aware of (fastai)\". In that kernel, Jeremy notices some range of pixel values are WAY out of what we can usually expect in terms of Hounsfield Units. We could decide to discard those scans entirely, but some similar value appear in the test set as well, and (as we'll see) for good reason. I decided to investigate those in depth, to understand what was going on.\n\nNote: this is the first time in three or four competitions that I dare publish a kernel (because usually I never have anything new to offer). So please forgive me if the following is total nonsense, or something that everybody knew and thought was obvious. We'll start by importing the needed stuff."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install torch torchvision feather-format pyarrow --upgrade   > /dev/null\n!pip install git+https://github.com/arraiyopensource/kornia       > /dev/null\n!pip install git+https://github.com/fastai/fastai_dev             > /dev/null\n\n    \nfrom fastai2.basics           import *\nfrom fastai2.medical.imaging  import *\nfrom tqdm import tqdm_notebook\nnp.set_printoptions(linewidth=120)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"path_inp = Path('../input')\npath = path_inp/'rsna-intracranial-hemorrhage-detection'\npath_trn = path/'stage_1_train_images'\npath_tst = path/'stage_1_test_images'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# HounsField Units: 50 shades of gray"},{"metadata":{},"cell_type":"markdown","source":"From what I've gathered so far, Hounsfield Units of body parts don't seem to be supposed to go under a thousand or above three thousand. See: http://www.startradiology.com/the-basics/x-rayct-technique/"},{"metadata":{},"cell_type":"markdown","source":"<img src='https://i.pinimg.com/originals/88/cc/4e/88cc4ed3b0451aa5bbfad74b0d28aecb.jpg'>"},{"metadata":{},"cell_type":"markdown","source":"<img src='https://pocketdentistry.com/wp-content/uploads/285/F000146f014-005-9780323096331.jpg'>"},{"metadata":{},"cell_type":"markdown","source":"Yet, using Jeremy's kernel, we encounter value that go as high as +/- 30000 !"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# To understand what's going on here, please refer to https://www.kaggle.com/jhoward/some-dicom-gotchas-to-be-aware-of-fastai\npath_df = path_inp/'creating-a-metadata-dataframe'\n\ndf_lbls = pd.read_feather(path_df/'labels.fth')\ndf_tst = pd.read_feather(path_df/'df_tst.fth')\ndf_trn = pd.read_feather(path_df/'df_trn.fth')\n\ncomb = df_trn.join(df_lbls.set_index('ID'), 'SOPInstanceUID')\nassert not len(comb[comb['any'].isna()])\n\n# This is where you find the weird min/max values\nrepr_flds = ['BitsStored','PixelRepresentation']\ncomb.pivot_table(values=['img_mean','img_max','img_min','PatientID','any'], index=repr_flds,\n                   aggfunc={'img_mean':'mean','img_max':'max','img_min':'min','PatientID':'count','any':'mean'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we've got 256 images in the training set with such high values\ncomb[comb['img_max'] > 30000].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#as seen two cells above, we've got at least an image with a max px value of 32767\nmax_val = comb['img_max'].max() #32767\n#Actually, we've got two of them\ncomb[comb['img_max'] == max_val]['fname']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We can use fastaiv2 fantastic functionalities to quickly plot this pic\ndcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_d6a09aba5.dcm').show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Holy cow. This guy's scan is really shiny ! This is obvisouly a metal (golden ?) tooth screwing up the scan here. It's also consistent with the values found in the [wikipedia page for hounsfield units](https://en.wikipedia.org/wiki/Hounsfield_scale#cite_note-BolligerOesterhelweg2009-22): \n\n![image.png](attachment:image.png)\n\nThis why I said at the beginning of this kernel that we couldn't discard these pics. We don't want our models to be completely lost when they encounter this type of situation in the wild, because a guy deserves the same quality of health care no matter if he's an \"edge case\" for a model because of his/her dentition or not. This is the kind of situation where I suppose a Hippocrates Serment for data scientist would start to make sense. \n\nBut I digress... The following cell will let you plot random picture with those high pixel values, so that you can get a sense of what's going on.\n","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA5gAAADMCAYAAAAWPwzBAAAgAElEQVR4nO3dvY7ryJkG4Imc+C50Cw42cOyMN2HA8F4Ab8CRA8MbWsoHk5xggzNJqwfrYJLJvcBYamCBTRzuBcw5CTfQX5EssshmsVWtfgZ4gGkVRfGnDj++ZIn6pgn++/bT5+aXL18hSV8BoFRqFMC6vv30uRn675tuwAQAAIAxkwPmvdMw74O+AkCp1CiAdQmYZKevAFAqNQpgXQIm2ekrAJRKjQJYl4BJdvoKAKVSowDWJWCSnb4CQKnUKIB1CZhkp68AUCo1CmBdAibZ6SsAlEqNAliXgEl2+goApVKjANYlYJKdvgJAqdQogHUJmGQ31lee6k2zqZ/br+/rZrPZNNX2pfX6YVs1m2rXHL48N/WmarbHVy7Tvm42m7p5KmDb3MfC7QfwQBafzxx3TbXZNJtQtWsO12lemm11a+vWtnQ7wPsmYJLdWF+5hcbba0/1pqnruhM8TwU4S+EVMAVMgLPF5zP7br0KncPjtf25qVshMtUO8P4JmGQ32leOu6Zqhb2XZltVzfb43NSt18NQFPn/fXgFuRueTgX72rbtBsz21eNb4D29Xu/DZb393buj2lqGToCNXOG+znfSOgwt45Cxde4EzNFlO6/nwHKNtQG8B1PPZ6Ijbs6vh8fMluOuqbrHxn19O4an2gEegIBJduN9JRLiql1ziIa7WEC6BKlboHuq+yHxdjW4O3336nH7/YdtdWvb101VVdd5PdWX+faX4RS8wuVtn4A81Zte+/R1iN/5velcAb8GyOHtN7hs3bu94d9jbQDvxLKAebooWg1dAIyFxbCepdoBHoCASXapvnILal9PIe78/4dt1fr/9hCi4YA0GoK6r0ULeTDPoPgftlUwdDcMwJFliL42sAyvWYduAO/Ou3PC0g+8I3cbk58dWweA92lZwIwMab1eKB24GBjctUy133vbAOQgYJJdsq8E319pDTUKinR7CNL0gDlcvEeuHrfuGN6G6j7V52Gs12GxY4Et9p3RcNhq/w7m6Dp0HyAx8jCIdhjvzy++vEPL1h6a2w60Y20A78NojTo/dC5m+Jh3O6YLmAACJitI9pVr4It977Junq7fywxff6uAefnsl2ZbBcsYhN90wLyFt/BO7ayAOeP7OPMCZmrZbk5DZ+PftRxrAyjZ0u9g9gUjTAyRBRAwyS/dV84hbh+GtpOnetPU2+7rbzhE9st5aOw2/G7o6aE53e91Dg6RjZ1ALB4iO2LOENnkssX21dATDjM+6RfgjSwKmNHjZXCMjdWY3kN+RtoBHoCASXZT+spTvWk9QOfisK0ir88ImAOPgJ/6kJ/b/DbBw31OD3RoD9kdeUhP9wTk+tCdmevQW6ahu4WpBxt1AubYskXbJ7QV0O8Aplj+kJ/I8bvzkDY/UwJ8ZAIm2U3qK+cQ1/tOS+enQU7mBMyvTfu7gnN+piT8vM7Pk/SG8p5/u3NgHrchpJegF85z7jpMGYoa+ZmSgSG948vWbY89cdb3MIH3a/n5TOc77ANfuxj+7nyqHeB9EzDJ7vH7SuKJsSUw5Aog6vFrFMB9CZhk9/h9pbCA2Rs+67uRAEMev0YB3JeASXaP31cKC5hf+j9tIlwCxD1+jQK4LwGT7PQVAEqlRgGsS8AkO30FgFKpUQDrEjDJTl8BoFRqFMC6BEyy01cAKJUaBbAuAZPs9BUASqVGAaxLwCQ7fQWAUqlRAOsSMMnu20+fAQCAD0rAJCt9BYBSqVEA6xIwyU5fAaBUahTAugRMstNXACiVGgWwLgGT7PQVAEqlRgGsS8AkO30FgFKpUQDrEjDJTl8BoFRqFMC6BEyy01cAKJUaBbAuAZPs9BUASqVGAaxLwCQ7feV1DseXuy8D5NTt0/p4Wfvjo/oINcq+Bu5JwCS7qX3lsK2azWbT1Ptu23NTb6pme8y9bGvNd6mXZlttmmrrhKCI/byvm82mbp7e6/JnW4bYv82pun26+/ea61jC9lvBon4Z2/5L9u895Fvm5eczp2XZXE3dL6f90F+H0+uX+fVrQap9bF+vty+mr397+Tf184rrD5RAwCS7aX3lVKCqqmo21a459Noe8ARxkIBZFAGzETALJGAWEjDPtSs4Xp8ulk4PWe11OL9+DV3d+afaU/t6nf0wZ/2f6nD548eGfOsPlEDAJLtJfeVysnTcNVXvZDA8QYycVIQnWpf3b+v21dF9Hbla2p1v1Wz3u6a6XoUNlyPn544ZuLJ7DJere1Ly0myr8LOrZvtfM5dndP6d9t4FgISx985dr2N/W12nP8/r8vdhe7lYMWc/3/b1tW3bPZHv7KPrOk1dnth2GvvMTkBK7KvLSIDYuo21jTv3/2342d0TyPHtcut330b6eDcEDs1rfPnK3X6pz83RLxPHksH98dzE92/831t8n6T+rY5Nk9jXg8ePzjH5Mt35uDZnX009n2kHo7No0B+/qHFdtqpu6u4dzFgN3Ne3dU+1j/WBTqib/u8r4/pHlr91bMy6/kApBEyym9JXnurLFcjY1da5ATMogpdA1QpqnbARzDcslE91WHhzfm5K+u7CUx0u6/lkISyws7fD2PzD9rlXw8fe+4r16jhsq9Y6VlV1nf+tT83Zz/Fl7C1TcJIVvn/a8sS30bXtekIdC0iJbdY92Qv/HmubuB/D6dt3Kca3y7w7mP0+NiWcv4ftl/o39vp+GT+GTN8fcz+/u0/S/1bj06SWc+Lxo3MxZ+6+WhQwYxLH+8N21/p306sr0ZAd9MOx9oHtPnwHsLvN11//0eXLvv5ACQRMskv2lU4x6g+vmR8w23cGhv6eMd+sn5syIcS1li0y/dLlGTuxnmPue1PrFZs+CHd1XZ9PgMITtxn7Oba8vf0cu1p/nuek5Rleh8tr7X8DiSGeU/fVoiGVsf7SPcEf2S5zAmZ0Od/79kvNb2G/7Jq9P17x+a19MuXC09BxKvHvaXCbX6aLDJGcua/yBsx+gEtN27uDHg1Qpz6cak9u92TfeKv1D+6iBuuTf/2BEgiYZJfqK/2CEQtDM4fIHmPv7b4/d8Cc+rkpQydrQw9SiJx8v2p54vMfv3s0btp7Z6xX9L2n6Z/q8xDD67DYaXewwv04fPIycvW8tb+mLE9kG40OO4sFpPFtFh3mPNqWEluGYL2T22V6wGwPbWyLBZj3sf1Sn7uwX3bN3h8TPn90n0z5tzrxbl2wbLOGlY8M503tq9EaFX6toGPw4TyTwmV8m6weMJN94y3Xf05fFzDhPRMwyW68r3RPuGLfC/noAfO2jVonCdkC5vj81wuYr1iv6Laqmu3xpdlWdfN0CXLHS7Cbt5+XB8wpyxPZRpMDUmqb3ZyGYMa/fzbWNvzv9A0D5oz+9n6239jnvoOAOWH46z0CZnfkQHe6Kfsqyx3M86iReQ+bucMQ2bkBc9X1j2wHQ2ThIQmYZJe+OhwpDPs6OCH44AEzVlBzBszU/NcaIvua9Yo4bKvTw0mqXXMIHiQS/37fykNkJy1PejsMDvFMbrNEX5rc1nXvIbKJPlb69kt+7nscItvfFrMDZpYhslP2xXj74oDZqldzTNwmvYfcjLSn1n2NIbJz1j+6TxPHkkXrD5RAwCS7sb4yfEW0O+yw88TB3tPw1g6YOT83JRIww4J6fYhJe5ssCpij8+8HqOlDoEbe+5r1ijkP4bo9TKdqqtb7luzny52nofbIAzKSy5PY373PHAlf3W0WbZ/QNnE/ph4Ck/MhP/1tOnEIYInbb+a/sfn9Mr5N5j7kJ/n5g/vklQFz1kN+useezjKHyzNzXy0KmIuGZ07ZJt3vmKbaU/8+Mj/kZ/b6p+7a5l5/oAQCJtkN9pXkk/YudyHi4eg67GkfXNFcLWDm/NyxJ4qG634rordhXptgm7Sv3i/5Dub4/L8GJ8T9k9rUuqTeO2u9oiInob39Nmc/h9/fqmb8TMnU5Rleh9ZnRof4pvdVuz325NLhtuH9eF6G8Gc0BobZDW2Xbp9u/93tn515JU9gS99+qc/N0S+75uyPuZ/f3SevDZgT/j0NHj/6yxwGpbF91bXkfGbs+6np4/20bdJ/b6p9YBkX/kxJvvVPfX7e9QfuT8AkO30l4rhr6kcpio+0LqW4x5CvR9qPth8zrF6jPnrf+OjrDwiY5Kev9B229cM88e6R1uUuesM/53w30n60/Vhq7Rr10fvGR19/QMBkBfoKjOsOMzPky/bj7ahRAOsSMMlOXwGgVGoUwLoETLLTVwAolRoFsC4Bk+z0FQBKpUYBrEvAJDt9BYBSqVEA6xIwyU5fAaBUahTAugRMstNXACiVGgWwLgGT7L799BkAAPigBEyy0lcAKJUaBbAuAZPs9BUASqVGAaxLwCQ7fQWAUqlRAOsSMMlOXwGgVGoUwLoETLLTVwAolRoFsC4Bk+z0FQBKpUYBrEvAJDt9BYBSqVEA6xIwyU5fAaBUahTAugRMsnuvfeVwfLn7MgCwrvdaowDeCwGT7Mb7ykuzrTZNtR0Kc89Nvama7fEtlzm1TAA8iuXnM89Nvdk0m6u6eZozff3caT/VoEt7vxal2gHKImCS3bKAeQ8lLhMAa1h2PnMKi2G9OGyrkZB5DofXUNmtN9327vxT7QDlETDJLtsdzOOuqTZVs93W7Su/+3rgSnD7Ku+m2jWHYN6nk4BL++Uuaec9nZOA+LzOy7jfNVVvftOWBYD7mHo+81RH7jbu60iYHBt5cwqE9f722mFb3eZ7qXPHzmdcakaqHaBAAibZ5Q2YQYG/BMtoYe7P97CtbkW4e1LQ+jt1Rfl8onEt6JfhTrf5tdsTywLA3SwKmDGxEHiVuIMZC4vHXVNd6kuqHaBAAibZ5Q6Ytyu/3SvBwd/Rq8qnzxpuH1imaPEOP7t/Rbo1/9SyFLCPAD6qvAGzf0FycF7nES29u5nRAHmqg6n2e29LgBgBk+yyD5E9Rtquf5+KdXv4a9vps9pDVttBb8IV5dY04wEzvSz330cAH9VojQq/gtHRv0A4JVz2vzMZBlcBE3hEAibZ3S1gThyCeruSPDC8NkfANBwWoEhZ7mCeR9gkLxoODnE91x9DZIEHJGCS3T0C5vgQ2NRyvMUQWQBKsDhg7uvIg90GpAJmrN70HvIz0g5QIAGT7O4SMC9DlcKiG54EdEPf6AOCpj3kZzBgppalgH0E8FEtCpizh6cODJHtPBTOz5QAj0TAJLspATP6HZf6uXl9wIzNu30SED5kIfqghVk/UzIWMNPLAsB9LDmfGfuO/aUmPNXdAHh58vjQz1a160U/PKbaAcoiYJKdvgJAqVavUcddUwuBwAcmYJKdvgJAqdauUYdtbcQK8KEJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkp2+AkCp1CiAdQmYZKevAFAqNQpgXQIm2ekrAJRKjQJYl4BJdvoKAKVSowDWJWCS3befPgMAAB+UgElW+goApVKjANYlYJKdvgJAqdQogHUJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkp2+AkCp1CiAdQmYZKevAFAqNQpgXQJmiT5/an79zZ+aX//m780/v/yr+Y/f/On09zd/an79+3+cp4u9fnvtj5/v26nuvg0BIEKNAliXgFmiz5+uQfKff/lbL1T+8fPw6798+dp8/3sBEwBi1CiAdWUKmP/X/PXf/7P51e/i/vDjWivwv80ffvdD89f/WWvemZf9x5+aX/3up+b71HRBwOz651/+1vz2L/8aff19BMznpt5smk2g2r7cbZkB+BjyBcyXZlttmno/Ms1x11Sbqtke0/MZroVL2wHeVtaA+W/f/d/dVyifEgPmP5o/fvMp8v7268UHzOOuqTbdonwOnPVzAfsegEeVJ2DeQt1wwLxMMxYwz9Nca9+pFt5C4tJ2gLf3xgGzc6fz3/+7+bnV9kPz1+9+Ordf7kyOvad7B/MUCq/v/y4MdOdpf/zv5t/CaQbvfp4D5nfh9N1wOLZs48vz/Z//s/nVn/+39Zk/f/fD6bVowPxH88dv/tb8x8/d5ey/XnbAPBXDaPHb151C3L4qu6l2zeE6/XNTb6pmuz+F1dM0dfM0uT01/5dmW1XNdluf21NXoAF4D6YGzKc6ftHzsK3ONaNu6pE7mIdt1VR1PX4HM3aHc1/f6tHSdoA7eMOAeQ5jQaj6/s9hKDu390Jne74/f/dDME0YME9h7jrt/1yCYRgw2yGx/fld/el//u6H4O9p63Nb9s78enczT9P/4cev/YD589+b38bC5cDrRQfMScOFvjb9q7LnYn8tmpchtrfQeCr6l79T7f2ge9hWwfzPn69IAzyU5QFz16oj0YB53DVVtWsOqZoXC4PHXVNdatXSdoA7WP07mO3A170DGA5DjYTU6JDSIIiFAfPHn3phsR0II0NeR4esxobIBq+l1ic27x+7d1SD+YfzCwPmz39vfhsbFjv0+pfCA+a+jtxJjIgWyFNoPBXz8P9f0R5djvBkYeROKwDv1tKAeTMUMM8jYI5fkxdV2xc2z4L3LG2/97YGPqa3u4MZCYDt94XB8eQUEMeC6y1gXoeXdj9zUcDsDqENljexPu07rWedUBoOk20tfxAwv/998FMkZ7/9y78GX7+8590HzOgQnzD0nYfAHl/Xfh3iFHF6/4SHNwDw7qRrVLw2xINk//XDtrpdnBQwgQ+o/IA5OIT1a/PeA+bt7866jzxFdoqiA+Zo4XtpDpfX3yJgjg5/FTABHtGqdzAvQ2PDvw2RBT6YtwuYE4fITg+Al/e/1yGywTp/15nX50+nu5K/+Xvzz1k79PR7mL/+puCAOTb0NHy67F2GyPaXU8AEeCxrBsyx0TGD39Xs1qLeQ3wWtAPcQXEP+WkHusiDf378KXj6a3iXMfFQnVcGzOGHAk18aNG1vT+/X74Ew4C7d1/fsak/U9IOmd2fKZn/kJ957ZGH+LSeYitgAjyi9b+DGUgOV/UzJcDjKexnSmK/O9l9gFD3Z0kSP1PSeuLsK4bIhj9rMjAkdvhnSsL27s+mnJ2fdpv19zYL6FTp6S4BcPqPR/d/pmTT1HX9yvbI/CM/kSJgAjyWPL+D+bV5bcB8qrv1rl2LUrVwfjvA28oUMAsV/Z5kYaJDbd+3t+krsSGwc9oB+Ijufj5z3DW1EAg8sMcJmK2hs1+b6XdV7+vn734ofhnnEjABKNW9z2cO29oTXoGH9jgB80v/Z02KDm7nobHF32F9BQETgFK9h/MZgPfsoQImZdBXACiVGgWwLgGT7PQVAEqlRgGsS8AkO30FgFKpUQDrEjDJTl8BoFRqFMC6BEyy01cAKJUaBbAuAZPs9BUASqVGAaxLwCS7bz99BgAAPigBk6z0FQBKpUYBrEvAJDt9BYBSqVEA6xIwyU5fAaBUahTAugRMstNXACiVGgWwLgGT7PQVAEqlRgGsS8AkO30FgFKpUQDrEjDJTl8BoFRqFMC6BEyy01cAKJUaBbAuAZPs9BUASqVGAaxLwCS7KX3lqd40m02gfg7an5t6UzXbY/f/AWCZfOczL8222jT1PnhtX7drW6A1XWQ+l+mq7UvmdoC3JWCSXaqvPNXdQHkujq3XLgRMAPLJcz5zC3XDwfHkqd40m2rXHMbmc61/z03dColL2wHenoBJduN9ZSAw7utms6mbp940t//vB9Pua+2ruO2C/tJsq6rZbi9Xl4VWgI9o6vlMrOb88uVrc9hW5xpTN3X3DmZXq7ZFHHdN1a1H+/pWv5a2A9yBgEl2431l7G7lxcAQ2V6hPl2pPRX303zDq7aHbRUU2fPnKroAH9rygLk716HIENmWfl3qiYXB466pLrVuaTvAHQiYZJfuK6dgGL/TeGmPfQczDJRf24EzepU4LP4TCj0AD29pwLxJBMzU3csv3QuhZ8FdyaXt997WwMckYJLdvL4ShM3Wd0jiD/kJC374/9chSxGnUJm60gzARzBao2Y9pGesrky7qClgAo9IwCS7V/WVVkEceYrs9Ypw+25mtMi2CJgAvNEdzKkhzxBZ4AEJmGSXvjocK3xhYBz7mZLzdNtOAU0ORRIwAXijgDn1QTuxMNh7iM+CdoA7EDDJbtJDfjrF7zTEdfwpsu1pB37qJJzvvg6eFitgAvA2ATP93vY8/EwJ8EgETLKb0lee6s73W1qBczxgnq7YDn8f5jbf8H0CJgC5fgfza5MKmEMhr9/Wrl399y1tB3hbAibZ6SsAlOruNeq4a2ohEHhgAibZ6SsAlOreNeqwrT3hFXhoAibZ6SsAlEqNAliXgEl2+goApVKjANYlYJKdvgJAqdQogHUJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkp2+AkCp1CiAdQmYZPftp88AAMAHJWCSlb4CQKnUKIB1CZhkp68AUCo1CmBdAibZ6SsAlEqNAliXgEl2+goApVKjANYlYJKdvgJAqdQogHUJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkl2uvnI4vtx9XQB4LM5nANYlYJLdlL7yVG+azSZQPwftL8222jTVNlfAfG7qTdVsj/ffNgDc1/LzmeemDuvXpm6eWu2nGnZpT9ey1PRL2wHeloBJdqm+8lTHA+XtNQETgHUsO585hcuwPh22VRAyu/WsP31bavql7QBvT8Aku/G+MhD29vW5QLevxHZD5/X1atccWvMdaxcwATiZej7Tvxj6NahV4bRBjTnumqpbb/Z1pGadpaZf2g5wBwIm2Y33le7V1uFpuldoe1eMrwU01S5gAnCyKGDGhCEvFu6Ou6bqhdKz1PRL2wHuQMAku3Rf6Xx/ZeBu5DUwRq8Yn6ap9xPaBUwAzvIGzPZF0/bFzbPYXcaz1PRL2++9rYGPScAku3l9JQibA9/BPH2/ZRNVbV+S7QImABejNWpfD9aT0wXLUH9EjoAJIGCyglf1lVZBjATMke+TpNoFTAAustzBPO6aKvYwHUNkAQRM8ktfHY4VvtOdzNMV4ilDZKfMM5y3gAlAhoC5r5vNUE2JhbvkQ35Gpl/aDnAHAibZTXrIT6f4xR7z3nsMe/ieVoFPtQuYAJwsCpjJ4ad+pgRAwCS7KX3lqe58vyUaOEd+pqRX4MfaBUwATpacz4x95//2Hc12PeqGvae6+9r49MvbAd6WgEl2+goApbp7jTrumloIBB6YgEl2+goApbp3jTpsayNqgIcmYJKdvgJAqdQogHUJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkp2+AkCp1CiAdQmYZKevAFAqNQpgXQIm2ekrAJRKjQJYl4BJdt9++gwAAHxQAiZZ6SsAlEqNAliXgEl2+goApVKjANYlYJKdvgJAqdQogHUJmGSnrwBQKjUKYF0CJtnpKwCUSo0CWJeASXb6CgClUqMA1iVgkp2+AkCp1CiAdQmYZKevAFAqNQpgXQIm2T1qXzkcX+6+DAAs86g1Ct6K8yFSBEyym9pXnupNs9mEqmZ7nPNZz0099p593Ww2dfO0eJ1emm21aaptrgNqYrnXNrpdnpt6s2nq/f37EcAalp/PnGrCtXbVz6Pt6dqRmn5pe9xT7VhfhlTdXXLOEL73tfMJ39c9H7rz+QzFEjDJLt1XTgfTXlHe181mVrgRMF9FwAQ+sKXnM091WL+69eEc9q7tp2PqcP1ITb+0Pe6wrSLBmPt4q4CZQ+R86LhrqmrXHO6+HSmJgEl2qb7SLs5th23VbFoHqs6V4lZb98B5Dq6Xu6HbeQHzsK0id1OHrlSPLdfc5U447pqqdac3LETnee3DabrznrNdzoVuG84vnPal2VaXeQSfNbqMQ9s23QaQ26xRNt1addw1VewYdjnGR9p/2deRGjE8v9b0S9ujnpu6dVyPBJzWhchYnem+f6z9azNeEwfqSmw7Xac575t95+9Jn/d1Qc1KbavX1MhU3e2eM6TOP7r7OnIHc9b2vLxv+M79U53zIjyPQMAku/G+Midcda/Mngv+9WCaGrYRK3IDunf1egWjf8U4PJi2g3Gqfc426Bez09DisPC317O9jeZul377qdC2t0U/MI8s49i2Hd3uAPktCpg9nWNsLNwdd001dFxLTb+0PaJ/93JKwByrC9PqRqpmjgalS0C7LPclCF3+bgXtKTX4lTVrUsCcWSMnbd+p69Y1FjCnbs/xzx/sx3xoAibZjfaVROFLTxseqIODXiyYzAkro9NGTiAGrs7W+wntS4eszCl2s7dLbKhO+NrE4cKpZXjNPgLIIE/ADO7mBCfW0ZP92F3GidMvbR9a7vYxfFrAHK4LifZkTZxQV86BqHfXL/Z38vMiptasiQFzVo2ctH1HznVG1208YE7anlMC5r2/+kNxBEyymx0we0NHggPpwNDT08HtdkAbLrJTw0t76Ef7QN0+oLaHzrRV25dk++sOxOEw1/6Vz6FiN3+7xJYtXP9UIYstY3rbxtsA8hutUeEQwZEhlP33pI65pQTMVJjpr1O6Loy3p2tiIvxF1yn+FZl6n67R3fWeV7OmBcx5NXL69p2+brHtFBkiO2F7TguYE/YhH4qASXaLhsiGB703DZg3t6fbxg+o48NR0u2vGSLbKh4z7mC+TcBMLePYtp3WBpBL3iGyX5veCJaih8jGjvFvEDBHa+IKATNZg19bs14TMKfU8RkBc9ZQ1LcNmL6HyYWASXZLHvLTOui95RDZnn6oShWi6Z87I2DGTh4KGCLb+7zRZRzbtnPaAJZbFDCjx7bgGBmrW8mH/IxMv7S957V3MHMPkQ3lDZiTavSra9YrAuZr6visIbJj3MHkPgRMspv8MyWDT3XrPMF1xkN+uo9qf/V3MEcfGBD5Ev++7i/3YPvMgNlbrvlXU6dvl2kPDRr+vMgyjm3b0e0OkN+yO5ipAFH6z5TEAkJ3HpchojUVB1UAAAFvSURBVFMfJjetbqRqZraAmfq8JTUrua1eUSMnbd+p5xddbxUwfQeTNgGT7Kb2ldh3CfoFZs7PfYTT9n+OI/UY7dtQmP6yXJd16GdKBoa3DD/m/Pb3vOWqm6fUlePeFc7x7dJ2XrbwcfORn43p7qfxZRzftmNtALktP5+Z9zNV3eN7/5g/Pv3y9rb4b2CG3w+81IDOHba6HljnVHtkm0Vqd76Amfq8ZTVrfFu9pkam6u7YuU7qayW5A2bsfOirp8jSI2CSXbF95bhr6hKHXpa6XAAP6O416u7H/Ll3m2JDOOe08+j8DiZdAibZldpXDtu6yOEbpS4XwCO6d40q4Zgfv4s5RMBkxHHXVO5e0iFgkp2+AkCp1KivzbyHsgiYjO17372kT8AkO30FgFKpUQDrEjDJTl8BoFRqFMC6BEyy01cAKJUaBbAuAZPs9BUASqVGAaxLwCQ7fQWAUqlRAOsSMMlOXwGgVGoUwLrGAub/AwU9NMdVEPoPAAAAAElFTkSuQmCC"}}},{"metadata":{"trusted":true},"cell_type":"code","source":"# floor_value is the value above which we consider pixel values to be curious (the cases we want to investigate)\n# based on the above table, I put it at 3000, but feel free to experiment\nfloor_value = 3000\nf_name = comb[comb['img_max'] > floor_value].sample()['fname'].values[0]\nprint(f_name)\ndcmread(f_name).show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you shift enter the above cell several times, you'll see a lot of shiny items like in the first example. This poses a problem because it affets pixels values that are even far away from it (because of the shiny rays that emanates from them). If you want to see an incredible example of this, look at this one:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b914af0ba.dcm').show(figsize=(6,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As such, I have no clue how to handle those. Should we \"ceil\" the pixels with torch.clamp_max ? I though offering this kernel would start the discussion.\n\nAs this is my very first public Kernel, I'll be grateful for any idea/tip to improve it (i.e, what you'd like me to investigate next"},{"metadata":{"trusted":true},"cell_type":"code","source":"# PS: not all of them are shiny objects. Some other are just plainly screwed images, like this one: \ndcmread('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b79eed528.dcm').show(figsize=(6,6))\n# I wonder how we could come up with a systematized way to find them out ? I'll keep looking and update this kernel if I come up with something interesting.\n# In the meantime, thank you for you attention !\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Edit: # So I've been trying to find other pictures that were damaged like the one above. I don't think there are any, but I found interesting stuff in the process. You'll find the results in [this kernel](http://https://www.kaggle.com/bdubreu/finding-screwed-images-some-data-statistics?scriptVersionId=22054845)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}