{"cells":[{"metadata":{},"cell_type":"markdown","source":"**there was a small bug on the morning of the 24th if you tried to use it then and it wasn't working try it now :)**"},{"metadata":{},"cell_type":"markdown","source":"# **First, thanks to all for checking out my notebook!** Hopefully this code will be of use for anyone using YOLOv5 (TF version), and even thoes who just want the labels separated out in this way:  [[R1][R2][R3]] where R1, R2, and R3 represent radiologist 1, 2, and 3's findings and corresponding box coordinates.\n"},{"metadata":{},"cell_type":"markdown","source":"This program was designed to use train.csv to create inputs to a YOLOv5 architecture network. It creates and saves 3 useful files: a list of the unique image IDs, a list of the findings in each image searated into a list for each radiologist, and a list of the box centers width and heighth. People should be able to use it as a generic data loader for the labels and boxes. It groups the labels by image to save computation and by radiologist to better train the networks. The latter part seems to be of more debate: How can we use the radiologist ID data to help us in the contest? So far, we believe at least grouping them into the three radiologists who viewed each image should be done, but perhaps there is more? The reason for separating the radiologists is so that we are training a network to output any findings as one single radiologist would as opposed to a team of three. The drawback here is that we have to feed the network training data where each input will have 3 ground truths. Because we don't know which radiologist is accurately labeling the image, we will have to treat all 3 as if they are equally right even when they disagree. This certainly seems like it could be confusing for the network depending on how much the radiologists findings overlap, but it may actually be helpful in the sense of adding a bit of noise to the data. I would love to hear your opinions on this!\n\n\n**#NOTE#** I try to break the code into separate blocks below, but if you try this, it doesn't populate the lists for some reason... I am not super familiar working in these kaggle environments, it seems like maybe a variable is getting lost somewhere between cells as it does work right in a single cell. Maybe one of you more experienced kaggler's can tell me where it's going wrong..."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport pickle\n\n\n#Read in the labels from the text file in csv format#\ntrain_df = pd.read_csv(\"../input/vinbigdata-chest-xray-abnormalities-detection/train.csv\")\n\nprint(train_df.head())\n\n#Initialize a list for the unique IDs in all the findings#\nimage_ids = []\n\n#Using the index method to return the number of rows in the label file, in other#\n#words it is the total number of findings in all of the images from all Radiologists)#\nf = train_df.index\n\nprint(len(f))\n\n#To group by image, we first need a list of the unique images names from the findings#\n#We can do this easily using the .unique() method on the image_id column of the labels#\n#file#\nimage_ids = list(train_df['image_id'].unique())\n#We also need a list of image IDs for each finding#\nfindings = list(train_df['image_id'])\n\n#Testing that lists are properly initialized#\nprint(findings[59000])\n\nprint(image_ids[12])\n\n#Initializing lists labels_img_rad will be used to create final_labels and boxes_img_rad will do the#\n#same for the associated final_boxes#\nlabels_img_rad = []\nboxes_img_rad = []\nfinal_labels = []\nfinal_boxes = []\n\n#The structure for final_labels and final_boxes is a list for each image containing three lists for the#\n#findings of the three radiologists. In final_labels each list is number corresponding to the category of#\n#that radiologist's findings, while in final_boxes each list is a list of 4 numbers corresponding to#\n#the bounding box for that radiologist's findings. To get to this final structure, I will use labels_img_rad#\n#and boxes_img_rad to store the needed information.#\n\n#This first loop initializes said lists\nfor img in image_ids:\n   labels_img_rad.append([[],img])\n   boxes_img_rad.append([[],img])\n   final_labels.append([[],[],[]])\n   final_boxes.append([[],[],[]])\ni = 0\n#Next, add the radiologist ID and their findings (labels and box coordinates) into labels_img_rad and boxes_img_rad#\nfor name in findings:\n   idx = image_ids.index(name)\n   split = list(train_df.iloc[i])\n   labels_img_rad[idx][0].append({split[3]:int(split[2])})\n   boxes_img_rad[idx][0].append({split[3]:split[4:]})\n   #print(i)\n   i += 1\n\n#Checking that labels_img_rad and boxes_img_rad are correct#    \nprint(labels_img_rad[:10])\nprint(boxes_img_rad[:10])\n\n#This loop uses label_img_rad to fill final_labels with the sorted labels#\ni = 0\nfor img in labels_img_rad:\n   rad1 = []\n   rad1f = []\n   rad2 = []\n   rad2f = []\n   rad3 = []\n   rad3f = []\n\n   for dic in img[0]:\n\n      if dic.keys() in rad1 or rad1 == []:\n         rad1.append(dic.keys())\n         rad1f.append(list(dic.values())[0])\n\n      elif dic.keys() in rad2 or rad2 == []:\n         rad2.append(dic.keys())\n         rad2f.append(list(dic.values())[0])\n\n      elif dic.keys() in rad3 or rad3 == []:\n         rad3.append(dic.keys())\n         rad3f.append(list(dic.values())[0])\n\n   final_labels[i][0] = rad1f\n   final_labels[i][1] = rad2f\n   final_labels[i][2] = rad3f\n   i += 1\n\n    \n#This loop uses boxes_img_rad to fill final_boxes with the sorted bounding boxes#\ni = 0\nfor img in boxes_img_rad:\n   rad1 = []\n   rad1f = []\n   rad2 = []\n   rad2f = []\n   rad3 = []\n   rad3f = []\n   for dic in img[0]:\n      if dic.keys() in rad1 or rad1 == []:\n         rad1.append(dic.keys())\n         rad1f.append(list(dic.values())[0])\n      elif dic.keys() in rad2 or rad2 == []:\n         rad2.append(dic.keys())\n         rad2f.append(list(dic.values())[0])\n      elif dic.keys() in rad3 or rad3 == []:\n         rad3.append(dic.keys())\n         rad3f.append(list(dic.values())[0])\n\n   final_boxes[i][0] = rad1f\n   final_boxes[i][1] = rad2f\n   final_boxes[i][2] = rad3f\n\n\n   i += 1\nprint('first 5 elements of final boxes: ')\nprint(final_boxes[:5])\nprint('first 5 elements of final labels: ')      \nprint(final_labels[:5])\n\n\n#Finally use the pickle module to save output#\npickle.dump( final_boxes, open( \"./final_boxes.pickle\", \"wb\" ) )\npickle.dump( final_labels, open( \"./final_labels.pickle\", \"wb\" ) )\npickle.dump( image_ids, open( \"./image_ids.pickle\", \"wb\" ) )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Import statements#\nimport pandas as pd\nimport numpy as np\nimport os\nimport pickle\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, we read in the labels from the text file in csv format"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport pickle\n\n\n\ntrain_df = pd.read_csv(\"../input/vinbigdata-chest-xray-abnormalities-detection/train.csv\")\n\nprint(train_df.head())\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initialize a list for the unique IDs in all the findings using the index method to return the number of rows in the label file, in other words it is the total number of findings in all of the images from all Radiologists."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimage_ids = []\n\nf = train_df.index\n\nprint(len(f))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets take a look at the total number of findings, we know they eventually go with the 15000 unique images"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(f))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To group by image, we first need a list of the unique images names from the findings.\nWe can do this easily using the .unique() method on the image_id column of the labels file."},{"metadata":{"trusted":true},"cell_type":"code","source":"image_ids = list(train_df['image_id'].unique())\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just looking at a few image ids to make sure the list were populated."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(findings[59000])\n\nprint(image_ids[12])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also need a list of image IDs for each finding"},{"metadata":{"trusted":true},"cell_type":"code","source":"findings = list(train_df['image_id'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initializing lists labels_img_rad will be used to create final_labels and boxes_img_rad will do the same for the associated final_boxes."},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_img_rad = []\nboxes_img_rad = []\nfinal_labels = []\nfinal_boxes = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The structure for final_labels and final_boxes is a list for each image containing three lists for the\nfindings of the three radiologists. In final_labels each list is number corresponding to the category of\nthat radiologist's findings, while in final_boxes each list is a list of 4 numbers corresponding to\nthe bounding box for that radiologist's findings."},{"metadata":{"trusted":true},"cell_type":"code","source":"for img in image_ids:\n   labels_img_rad.append([[],img])\n   boxes_img_rad.append([[],img])\n   final_labels.append([[],[],[]])\n   final_boxes.append([[],[],[]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **This is where the code stops working in the notebook, as you will see final_labels and final_boxes end up as empty lists just like they started... The program works as a single cell, and final_boxes/final_labels have the correct info. It seems like a variable is getting lost or reset or something inbetween cells, if you have any ideas why this might be happening please leave a comment, I am new to kaggle environment and would appreciate any help! Thank you!**"},{"metadata":{},"cell_type":"markdown","source":"This loop uses label_img_rad to fill final_labels with the sorted labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"i = 0\nfor img in labels_img_rad:\n    rads = []\n    rad1 = []\n    rad1f = []\n    rad2 = []\n    rad2f = []\n    rad3 = []\n    rad3f = []\n\n    for dic in img[0]:\n \n       if dic.keys() in rad1 or rad1 == []:\n          rad1.append(dic.keys())\n          rad1f.append(list(dic.values())[0])\n\n       elif dic.keys() in rad2 or rad2 == []:\n          rad2.append(dic.keys())\n          rad2f.append(list(dic.values())[0])\n\n       elif dic.keys() in rad3 or rad3 == []:\n          rad3.append(dic.keys())\n          rad3f.append(list(dic.values())[0])\n\n    final_labels[i][0] = rad1f\n    final_labels[i][1] = rad2f\n    final_labels[i][2] = rad3f\n    i += 1\nprint(final_labels[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This loop uses boxes_img_rad to fill final_boxes with the sorted bounding boxes."},{"metadata":{"trusted":true},"cell_type":"code","source":"i = 0\nfor img in boxes_img_rad:\n   rad1 = []\n   rad1f = []\n   rad2 = []\n   rad2f = []\n   rad3 = []\n   rad3f = []\n   for dic in img[0]:\n      if dic.keys() in rad1 or rad1 == []:\n         rad1.append(dic.keys())\n         rad1f.append(list(dic.values())[0])\n      elif dic.keys() in rad2 or rad2 == []:\n         rad2.append(dic.keys())\n         rad2f.append(list(dic.values())[0])\n      elif dic.keys() in rad3 or rad3 == []:\n         rad3.append(dic.keys())\n         rad3f.append(list(dic.values())[0])\n\n   final_boxes[i][0] = rad1f\n   final_boxes[i][1] = rad2f\n   final_boxes[i][2] = rad3f\n\n\n   i += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the final cell to see what final_labels and final_boxes look like."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(final_boxes[:10])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}