{"cells":[{"metadata":{},"cell_type":"markdown","source":"# VinBigData Chest X-ray Abnormalities Detection\nAutomatically localize and classify thoracic abnormalities from chest radiographs\n\n### The aim of this notebook is to demonstrate: \n1. Model Selection\n2. Model Configuration\n3. Model Training\n4. Postprocessing and Inference\n5. Model Evaluation\n \nThis notebook follows from the first [notebook](https://www.kaggle.com/bhallaakshit/dicom-wrangling-and-enhancement). The output from the first has been made available as a [dataset](https://www.kaggle.com/bhallaakshit/vinbig-tfrecords-for-object-detection?select=annotations) on Kaggle.\n \n### Please consider giving an <font color=\"red\">UPVOTE</font> if you find my work to be beneficial in any way. :D"},{"metadata":{},"cell_type":"markdown","source":"## Install TF 2 Object Detection API\n1. TF Model Garden\n2. Protobuf\n3. COCO API\n4. Object Detection API "},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!# Download models\n!git clone --depth 1 https://github.com/tensorflow/models\n\n!# Compile proto files \n! # sudo apt install -y protobuf-compiler # Already present\n%cd models/research\n!protoc object_detection/protos/*.proto --python_out=.\n%cd ..\n%cd ..\n\n!# Install cocoapi\n!pip install cython \n!git clone https://github.com/cocodataset/cocoapi.git\n%cd cocoapi/PythonAPI\n!make\n%cd ..\n%cd ..\n!cp -r cocoapi/PythonAPI/pycocotools models/research/\n\n!# Install object detection api\n%cd models/research\n!cp object_detection/packages/tf2/setup.py .\n!python -m pip install .\n%cd ..\n%cd ..","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import libraries\n1. **Pandas**: Data manipulation\n2. **Open-CV:** Computer Vision\n3. **Matplotlib:** Plotting\n4. **TensorFlow:** Deep Learning\n5. **Miscellaneous**"},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport tensorflow as tf\nfrom object_detection.utils import label_map_util as map_util\nfrom object_detection.utils import visualization_utils as viz_util\nfrom object_detection.utils import ops as ops_util\nfrom object_detection.utils import config_util\n\nimport requests\nimport tarfile\nfrom tqdm.notebook import tqdm\nfrom io import BytesIO\nfrom shutil import copy2\nimport random","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's begin. Firstly, we set up our workspace."},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# Creating workspace\nos.makedirs(\"workspace/pretrained_models\", exist_ok = True)\nos.makedirs(\"workspace/models\", exist_ok = True)\nos.makedirs(\"workspace/exported_models\", exist_ok = True)\ncopy2(\"models/research/object_detection/model_main_tf2.py\", \"workspace\")\ncopy2(\"models/research/object_detection/exporter_main_v2.py\", \"workspace\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading TFRecords\nLet's have a quick look at a sample record. Similar examples will be used for training."},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"path_annot = \"../input/vinbig-tfrecords-for-object-detection/annotations\"\nraw_dataset = tf.data.TFRecordDataset(os.path.join(path_annot, \"annotations-00000-of-00025\"))\n\nfor raw_record in raw_dataset.take(1): # Select one shard from the TFRecords dataset\n    example = tf.train.Example()\n    example.ParseFromString(raw_record.numpy())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def GetData(example):\n    xmin = example.features.feature['image/object/bbox/xmin'].float_list.value\n    xmax = example.features.feature['image/object/bbox/xmax'].float_list.value\n    ymin = example.features.feature['image/object/bbox/ymin'].float_list.value\n    ymax = example.features.feature['image/object/bbox/ymax'].float_list.value\n\n    class_name_list = example.features.feature['image/object/class/text'].bytes_list.value\n    class_name_list = [c.decode() for c in class_name_list]\n\n    class_id_list = example.features.feature['image/object/class/label'].int64_list.value\n\n    data = pd.DataFrame(\n        zip(xmin, ymin, xmax, ymax, class_name_list, class_id_list), \n        columns = [\"x_min\", \"y_min\", \"x_max\", \"y_max\", \"class_name\", \"class_id\"]\n    )\n\n    height = example.features.feature['image/height'].int64_list.value[0]\n    width = example.features.feature['image/width'].int64_list.value[0]\n\n    data[[\"x_min\", \"x_max\"]] = (data[[\"x_min\", \"x_max\"]]*width).astype(int)\n    data[[\"y_min\", \"y_max\"]] = (data[[\"y_min\", \"y_max\"]]*height).astype(int)\n\n    LABEL_COLORS = [\n        (230, 25, 75), (60, 180, 75), (255, 225, 25), (0, 130, 200), (245, 130, 48), (145, 30, 180), (70, 240, 240), \n        (240, 50, 230), (210, 245, 60), (250, 190, 212), (0, 128, 128), (220, 190, 255), (170, 110, 40), (255, 250, 200), \n    ]\n    data[\"colors\"] = data[\"class_id\"].apply(lambda x: LABEL_COLORS[x])\n    \n    \n    img_encoded = example.features.feature['image/encoded'].bytes_list.value[0]\n    image = tf.io.decode_jpeg(img_encoded)\n    \n    return data, image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"data, image = GetData(example)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\ndef plot_boxes(image, data, title):    \n    img = cv2.cvtColor(image.numpy(), cv2.COLOR_GRAY2RGB)\n    \n    for i, row in data.iterrows():\n    \n        x1, y1 = row[\"x_min\"], row[\"y_min\"]\n        x2, y2 = row[\"x_max\"], row[\"y_max\"]\n    \n        cv2.rectangle(\n            img,\n            pt1 = (x1, y1),\n            pt2 = (x2, y2),\n            color = row[\"colors\"],\n            thickness = 2\n        )\n    \n        cv2.putText(\n            img, \n            row[\"class_name\"], \n            (x1, y1-5), \n            cv2.FONT_HERSHEY_SIMPLEX, \n            0.5, \n            row[\"colors\"], \n            1\n        )\n\n    plt.figure(figsize = (8, 8))\n    plt.imshow(img) \n    plt.title(title)\n\nplot_boxes(image, data, \"Image extracted from TFRecord\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great. Everything on track. Let's move on now."},{"metadata":{},"cell_type":"markdown","source":"## Model Selection\nThe TF 2 Object Detection API let's us play with SOTA object detection models pretrained on the Microsoft COCO [dataset](https://www.tensorflow.org/datasets/catalog/coco). These models have been made available as a GitHub [repository](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md) called TensorFlow 2 Detection Model Zoo. We can fine-tune these models for our purposes and get great results.\n\nThe model of choice for this notebook is [EfficientDet](https://ai.googleblog.com/2020/04/efficientdet-towards-scalable-and.html). "},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# Download EfficientDet from Model Zoo\nurl = \"http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz\"\npath = \"./workspace/pretrained_models\"\nr = requests.get(url)\n\n# Extract model\nthetarfile = tarfile.open(\n    fileobj = BytesIO(r.content), \n    mode = \"r|gz\"\n)\n\n# Save model\nthetarfile.extractall(path = path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model configuration\nThe TensorFlow Object Detection API allows model configuration via the pipeline.config file that goes along with the pretrained model. The config file has 6 sections:\n1. 'model'\n2. 'train_config' \n3. 'train_input_config' \n4. 'eval_config'\n5. 'eval_input_configs'\n6. 'eval_input_config'\n\nNot to be confused between 'eval_input_configs' and 'eval_input_config'. According to the [documentation](https://github.com/tensorflow/models/blob/c40b46ff63d1af2d32e6457dcb4a70d157648db2/research/object_detection/utils/config_util.py#L79):\n> Keeps eval_input_config only for backwards compatibility. All clients should read eval_input_configs instead.\n\nAccording to the official [tutorial](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html#evaluation-sec), there are some basic changes to make to the config. We shall touch upon them only."},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# Moving pipeline.config file to models directory\nfname = \"pipeline.config\"\nmodel_name = \"efficientdet_d0_coco17_tpu-32\"\n\nsrc = os.path.join(path, model_name, fname)\ndst = src.replace(\"pretrained_\", \"\").replace(fname, \"\")\n\nos.makedirs(dst, exist_ok = True)\n\ncopy2(src, dst)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"path_label = \"../input/vinbig-tfrecords-for-object-detection/LabelMap.pbtxt\" \nLabelMap = map_util.create_category_index_from_labelmap(\n    path_label, \n    use_display_name = True\n)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"annot_dir = os.listdir(path_annot)\nrandom.Random(0).shuffle(annot_dir)\n\ntrain_data = annot_dir[:-4]\ntrain_data = [os.path.join(path_annot, d) for d in train_data]\n\nvalid_data = annot_dir[-4:-2]\nvalid_data = [os.path.join(path_annot, d) for d in valid_data]\n\ntest_data = annot_dir[-2:]\ntest_data = [os.path.join(path_annot, d) for d in test_data]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# Making recommended changes\nfpath = os.path.join(dst, fname)\nconfig_dic = config_util.get_configs_from_pipeline_file(fpath)\n\nconfig_dic[\"model\"].ssd.num_classes = len(LabelMap)\nconfig_dic[\"model\"].ssd.image_resizer.keep_aspect_ratio_resizer.min_dimension = 100\n\nconfig_dic[\"train_config\"].batch_size = 16\nconfig_dic[\"train_config\"].fine_tune_checkpoint = os.path.join(path, model_name, \"checkpoint/ckpt-0\")\nconfig_dic[\"train_config\"].fine_tune_checkpoint_type = \"detection\"\nconfig_dic[\"train_config\"].use_bfloat16 = False # Set to True if training on a TPU\nconfig_dic[\"train_config\"].num_steps = 1_000\n\nconfig_dic[\"train_input_config\"].label_map_path = path_label\nconfig_dic[\"train_input_config\"].tf_record_input_reader.input_path[:] = train_data\n\nconfig_dic[\"eval_input_configs\"][0].label_map_path = path_label\nconfig_dic[\"eval_input_configs\"][0].tf_record_input_reader.input_path[:] = valid_data","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"# Save recommended changes\nconfig = config_util.create_pipeline_proto_from_configs(config_dic)\nconfig_util.save_pipeline_config(config, dst)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fine tuning object detection model (training)"},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"!python workspace/model_main_tf2.py --model_dir=$dst --pipeline_config_path=$fpath","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exporting model"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!python workspace/exporter_main_v2.py --input_type=image_tensor --pipeline_config_path=$fpath --trained_checkpoint_dir=$dst --output_directory=workspace/exported_models/$model_name","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# To export model outside, first compress it\ntar_model_name = model_name + \".tar.gz\"\n!tar -zcvf workspace/exported_models/$tar_model_name workspace/exported_models/$model_name","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Handy method to download compressed file:\n<a href=\"workspace/exported_models\"> Click to Download </a>"},{"metadata":{},"cell_type":"markdown","source":"## Postprocessing and Inference\nLet's look at one sample x-ray. \n\n**IMPORTANT** \n\nTensorFlow expects input in NHWC format, which means: (batch-size, height, width, channels). Since our x-rays are grayscale, we can use OpenCV's cv2.COLOR_GRAY2RGB to solve the problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"for shard in test_data[:1]:\n    raw_dataset = tf.data.TFRecordDataset(shard)\n    \n    for raw_record in raw_dataset.take(1): # Select one shard from the TFRecords dataset\n        example = tf.train.Example()\n        example.ParseFromString(raw_record.numpy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_encoded = example.features.feature['image/encoded'].bytes_list.value[0]\nimg = tf.io.decode_jpeg(img_encoded)\nimg = cv2.cvtColor(img.numpy(), cv2.COLOR_GRAY2RGB)\nimg = img[tf.newaxis, ...]\n\ndetector = tf.saved_model.load(os.path.join(\"workspace/exported_models\", model_name, \"saved_model\"))\nresult = detector(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result = {k:v.numpy() for k, v in result.items()}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"viz_util.visualize_boxes_and_labels_on_image_array(\n    image = img[0], \n    boxes = result['detection_boxes'][0],\n    classes = (result['detection_classes'][0]).astype(int), \n    scores = result['detection_scores'][0],\n    category_index = LabelMap,\n    use_normalized_coordinates = True,\n    min_score_thresh = 0.4,\n    line_thickness = 3,\n    max_boxes_to_draw = 100,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nplt.figure(figsize = (8, 8))\nplt.imshow(img[0])\nplt.title(\"Prediction\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating performance\nLet's look at what the real thoracic abnormalitites for this chest x-ray were."},{"metadata":{"trusted":true},"cell_type":"code","source":"data, image = GetData(example)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_boxes(image, data, \"Ground Truth\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cross Validation and Hyperparameter Tuning\nIt would be amazing had there been a simple way to perform cross validation and hyperparameter tuning. Both involve iterative training consuming a lot of time and GPU (which may not always be available). Fortunately there are simple things that can be done to improve performance. For example, when making prediction, we can experiment with the confidence threshold (affecting classification) and tweak IoU (affecting localization). Impact on performance can be iteratively tested by comparing predictions against ground truth labels and bounding boxes, without retraining. This is a common strategy in ML and not included here."},{"metadata":{},"cell_type":"markdown","source":"### Please consider giving an <font color=\"red\">UPVOTE</font> if you find my work to be beneficial in any way. :D"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}