{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Commonly we will use over sample strategy to overcome the non-balance labels issue, but even so, negative sample still matters. So in this notebook, I will show you how to not wast any negtive samples and keep the oversampling positive ones. I will use this simple library https://github.com/louis-she/exhaustive-weighted-random-sampler to do this.","metadata":{}},{"cell_type":"code","source":"!pip install -q exhaustive-weighted-random-sampler","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-06T13:10:02.989479Z","iopub.execute_input":"2023-01-06T13:10:02.989878Z","iopub.status.idle":"2023-01-06T13:10:16.263706Z","shell.execute_reply.started":"2023-01-06T13:10:02.989845Z","shell.execute_reply":"2023-01-06T13:10:16.262371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from exhaustive_weighted_random_sampler import ExhaustiveWeightedRandomSampler\nfrom torch.utils.data import WeightedRandomSampler\nimport pandas as pd\nimport pytest\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:14:59.318548Z","iopub.execute_input":"2023-01-06T13:14:59.319575Z","iopub.status.idle":"2023-01-06T13:14:59.326263Z","shell.execute_reply.started":"2023-01-06T13:14:59.319509Z","shell.execute_reply":"2023-01-06T13:14:59.324906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assign weight\n\nAssign positive sample a weight of `len(negatives) / len(positives)` so that the sampler samples positive and negative with the same posibilities.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ndf[\"weight\"] = 1\ndf.loc[df.cancer == 1, \"weight\"] = len(df.loc[df.cancer == 0]) / len(df.loc[df.cancer == 1])","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:06:24.981441Z","iopub.execute_input":"2023-01-06T13:06:24.981897Z","iopub.status.idle":"2023-01-06T13:06:25.078185Z","shell.execute_reply.started":"2023-01-06T13:06:24.981863Z","shell.execute_reply":"2023-01-06T13:06:25.077271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Comparing the results\n\nWe will create a `WeightedRandomSampler` and `ExhaustiveWeightedRandomSampler` and compare the results.","metadata":{}},{"cell_type":"code","source":"wrs = WeightedRandomSampler(df.weight.tolist(), num_samples=10000)\newrs = ExhaustiveWeightedRandomSampler(df.weight.tolist(), num_samples=10000)","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:06:26.959873Z","iopub.execute_input":"2023-01-06T13:06:26.960281Z","iopub.status.idle":"2023-01-06T13:06:26.97575Z","shell.execute_reply.started":"2023-01-06T13:06:26.960247Z","shell.execute_reply":"2023-01-06T13:06:26.974265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### make sure weight is working\n\nFirst of all, let's make sure the weight is working. `WeightedRandomSampler` and `ExhaustiveWeightedRandomSampler` should give approximately equal positive and negative samples.","metadata":{}},{"cell_type":"code","source":"wrs_df = df.loc[list(wrs)]\newrs_df = df.loc[list(ewrs)]\n\nfig, ax = plt.subplots(1, 2, figsize=(12, 5))\n\nax[0].hist(wrs_df.cancer)\n_ = ax[1].hist(ewrs_df.cancer)","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:11:50.829787Z","iopub.execute_input":"2023-01-06T13:11:50.831053Z","iopub.status.idle":"2023-01-06T13:11:51.302658Z","shell.execute_reply.started":"2023-01-06T13:11:50.830998Z","shell.execute_reply":"2023-01-06T13:11:51.301292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### No duplicate negative samples in one epoch\n\nNow let's pay attention to the negative samples, there should be some duplicated negative index yield by the `WeightedRandomSampler` and no duplicated negative index of `ExhaustiveWeightedRandomSampler`","metadata":{}},{"cell_type":"code","source":"assert not wrs_df[wrs_df.cancer == 0].image_id.is_unique\nassert ewrs_df[ewrs_df.cancer == 0].image_id.is_unique\n\n# let's check the highest duplicate with \ndup_wrs = np.unique(wrs_df[wrs_df.cancer == 0].image_id, return_counts=True)[1].max()\ndup_ewrs = np.unique(ewrs_df[ewrs_df.cancer == 0].image_id, return_counts=True)[1].max()\nprint(f\"Maximum duplicates from WeightedRandomSampler is {dup_wrs}\")\nprint(f\"Maximum duplicates from ExhaustiveWeightedRandomSampler is {dup_ewrs}\")","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:16:25.616223Z","iopub.execute_input":"2023-01-06T13:16:25.61665Z","iopub.status.idle":"2023-01-06T13:16:25.634833Z","shell.execute_reply.started":"2023-01-06T13:16:25.616617Z","shell.execute_reply":"2023-01-06T13:16:25.632701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exhaustive sample over the epochs\n\nOK, there truely are no duplicated negative samples yield from `ExhaustiveWeightedRandomSampler`. Let's continue check the case with multiple epochs.\n\nSince the positive and negative are equally sampled now, one epoch will sample 5000 neg samples, so we need at least `len(negative_samples) // 5000 + 1` epochs to sample all the negative samples.","metadata":{}},{"cell_type":"code","source":"wrs = WeightedRandomSampler(df.weight.tolist(), num_samples=10000)\newrs = ExhaustiveWeightedRandomSampler(df.weight.tolist(), num_samples=10000)\n\nwrs_indes = []\newrs_indes = []\n\nepochs = len(df.loc[df.cancer == 0]) // 5000 + 1\n\nfor _ in range(epochs):\n    wrs_indes += list(wrs)\n    ewrs_indes += list(ewrs)\n\nassert len(wrs_indes) == len(ewrs_indes)\n\nwrs_df = df.loc[wrs_indes]\newrs_df = df.loc[ewrs_indes]\n\nprint(f\"need at least {epochs} epochs to sample all the negative samples\")","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:32:30.690444Z","iopub.execute_input":"2023-01-06T13:32:30.690856Z","iopub.status.idle":"2023-01-06T13:32:31.839118Z","shell.execute_reply.started":"2023-01-06T13:32:30.690822Z","shell.execute_reply":"2023-01-06T13:32:31.837908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# is it really exhaustive? check this by unique image_id\n\nprint(f\"Total number of image_id: {len(df.image_id.unique())}\")\nprint(f\"Number of WeightedRandomSampler sampled image_id {len(wrs_df.image_id.unique())}\")\nprint(f\"Number of ExhaustiveWeightedRandomSampler sampled image_id {len(ewrs_df.image_id.unique())}\")","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:32:31.841768Z","iopub.execute_input":"2023-01-06T13:32:31.842267Z","iopub.status.idle":"2023-01-06T13:32:31.862185Z","shell.execute_reply.started":"2023-01-06T13:32:31.842219Z","shell.execute_reply":"2023-01-06T13:32:31.860794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You will see about 20K negative samples are not been sampled after 11 epochs!\n\nStill lets draw the distribution again. Just make sure the weight is working.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(12, 5))\nax[0].hist(wrs_df.cancer)\n_ = ax[1].hist(ewrs_df.cancer)","metadata":{"execution":{"iopub.status.busy":"2023-01-06T13:32:31.863922Z","iopub.execute_input":"2023-01-06T13:32:31.864493Z","iopub.status.idle":"2023-01-06T13:32:32.185703Z","shell.execute_reply.started":"2023-01-06T13:32:31.864443Z","shell.execute_reply":"2023-01-06T13:32:32.184414Z"},"trusted":true},"execution_count":null,"outputs":[]}]}