{"cells":[{"metadata":{},"cell_type":"markdown","source":"Let's install the things we will need :)\n\nThe below will pull the latest version of the [fastai v2 library](https://github.com/fastai/fastai_dev). The library is under active development and there can be breaking changes. The notebook runs at the moment but should you encounter any issues in the future, it is best to verify whether the API changed or maybe rerun the notebook within a day or two.\n\nIf you find a genuine bug while getting acquainted with the library, it is best to [report it on github](https://github.com/fastai/fastai_dev/issues)!"},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install git+https://github.com/fastai/fastai_dev","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dir_csv = '../input/rsna-intracranial-hemorrhage-detection'\ndir_train_img = '../input/rsna-train-stage-1-images-png-224x/stage_1_train_png_224x'\ndir_test_img = '../input/rsna-test-stage-1-images-png-224x/stage_1_test_png_224x'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai2.torch_basics import *\nfrom fastai2.test import *\nfrom fastai2.layers import *\nfrom fastai2.data.all import *\nfrom fastai2.optimizer import *\nfrom fastai2.learner import *\nfrom fastai2.metrics import *\nfrom fastai2.vision.all import *\nfrom fastai2.vision.learner import *\nfrom fastai2.vision.models import *\nfrom fastai2.callback.all import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"items = get_image_files(dir_train_img)\nitems = [i for i in items if '(copy)' not in i.name]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Processing the `stage_1_train.csv` into something more parsable."},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p data\n\ndf_train = pd.read_csv(f'{dir_csv}/stage_1_train.csv')\ndf_train['fn'] = df_train.ID.apply(lambda x: '_'.join(x.split('_')[:2]) + '.png')\ndf_train.columns = ['ID', 'probability', 'fn']\ndf_train['label'] = df_train.ID.apply(lambda x: x.split('_')[-1])\ndf_train.drop_duplicates('ID', inplace=True)\npivot = df_train.pivot(index='fn', columns='label', values='probability')\npivot.reset_index(inplace=True)\npivot.to_csv('data/train_pivot.csv', index=False)\n\nfrom collections import defaultdict\n\nd = defaultdict(list)\nfor fn in df_train.fn.unique(): d[fn]\n\nfor tup in df_train.itertuples():\n    if tup.probability: d[tup.fn].append(tup.label)\n        \nks, vs = [], []\n\nfor k, v in d.items():\n    ks.append(k), vs.append(' '.join(v))\n    \npd.DataFrame(data={'fn': ks, 'labels': vs}).to_csv('data/train_labels_as_strings.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Labeller():\n    '''path to label, eg. path -> ['subdural', 'any']'''\n    def __init__(self):\n        self.df = pd.read_csv('data/train_labels_as_strings.csv')\n        self.df.set_index('fn', inplace=True)\n    def __call__(self, path):\n        fn = path.name\n        labels_txt = self.df.loc[fn].labels\n        if isinstance(labels_txt, float) or labels_txt == ' ': return []\n        return labels_txt.split(' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labeler = Labeller()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classes = L(pd.read_csv('data/train_pivot.csv').columns.tolist()[1:])\nclasses","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mcat = MultiCategorize(vocab=classes)\nmcat.o2i = classes.val2idx()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = [PILImage.create, [Labeller(), mcat, OneHotEncode(mcat.vocab)]]\n\nds_img_tfms = [ToTensor()]\ndsrc = DataSource(items, tfms, splits=RandomSplitter()(items))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dsrc[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_paths = get_image_files(dir_test_img)\ntest_tfms = [PILImage.create, [lambda x: np.array([0,0,0,0,0,0])]]\ndsrc_test = DataSource([test_paths[0]] + test_paths, test_tfms, splits=[[0], L(range(len(test_paths))).map(lambda x: x + 1)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dsrc_test[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n\n# means, stds = [], []\n\n# for batch in dbch.train_dl:\n#     reshaped = batch[0].permute(1,0,2,3).reshape((3, -1))\n#     means.append(reshaped.mean(1)), stds.append(reshaped.std(1))\n\n# torch.stack(means).mean(0)\n\n# torch.stack(stds).mean(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have not tried whether normalizing with imagenet stats (stats of the dataset the models were pretrained on) or normalizing with stats specific to this dataset work better.\n\nMy approach to this competition is to cut corners where I feel little value can be had (intuition is often misleading though!). I do want to focus on the aspects of the competition that are interesting and fun to me, and where I feel I can learn the most, even at the cost of a lower overall score."},{"metadata":{"trusted":true},"cell_type":"code","source":"means = [0.1627, 0.1348, 0.1373]\nst_devs = [0.2961, 0.2605, 0.1889]\n\ndataset_stats = (means, st_devs)\ndataset_stats = broadcast_vec(1, 4, *dataset_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_img_tfms = [ToTensor()]\ndl_tfms = [Cuda(), ByteToFloatTensor(), Normalize(*dataset_stats)]\n\ndbch = dsrc.databunch(after_item=ds_img_tfms, after_batch=dl_tfms, bs=128, num_workers=4)\ndbch_test = dsrc_test.databunch(after_item=ds_img_tfms, after_batch=dl_tfms, bs=128, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dbch.show_batch(max_n=9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = create_cnn_model(resnet18, 6, -2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I define how I want to split the model for applying differential learing rates"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_segments = model[0][:6], model[0][6:], model[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def trainable_params_mod(model): return L(trainable_params(segment) for segment in model_segments)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"opt_func = partial(Adam, wd=0.01, eps=1e-3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(\n    dbch,\n    model,\n    loss_func=BCEWithLogitsLossFlat(),\n    metrics=[accuracy_multi],\n    opt_func=opt_func,\n    splitter=trainable_params_mod\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.freeze_to(-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find(start_lr=1e-6, end_lr=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(1, 2e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('phase-1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('phase-1');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.freeze_to(-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find(start_lr=1e-8, end_lr=1e-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(1, [1e-3, 1e-4, 1e-5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('phase-2')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('phase-2');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(1, np.array([1e-3, 1e-4, 1e-5]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save('phase-3')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('phase-3')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.metrics = [PrecisionMulti(), RecallMulti()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.validate()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One on hand, it is amazing that the switching of the databunches is possible. This is what flexibility and modularity looks like :) On the other hand, this is a hack.\n\nThis is only necessary right now because either this functionality in fastai v2 has not been added yet or I do not know how to go about this in a more proper way. I am sure the need for doing this will go away soon!"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.dbunch = dbch_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, targs = learn.get_preds()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = []\nlabels = []\n\nfor path, pred in zip(test_paths, preds):\n    for i, label in enumerate(classes):\n        ids.append(f\"{path.name.split('.')[0]}_{label}\")\n        predicted_probability = '{0:1.10f}'.format(pred[i].item())\n        labels.append(predicted_probability)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({'ID': ids, 'Label': labels}).to_csv(f'submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}