{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ***Disclaimer:*** \nHello Kagglers! I am a Solution Architect with the Google Cloud Platform. I am not an insider to this competition, so I am allowed to contribute and even compete, although I cannot collect prizes. The focus of my contributions is on helping users to leverage GCP components (GCS, TPUs, BigQueryetc..) in order to solve large problems. My ideas and contributions represent my own opinion, and are not representative of an official recommendation by Google. Also, I try to develop notebooks quickly in order to help users early in competitions. There may be better ways to solving particular problems, I welcome comments and suggestions. Use my contributions at your own risk, I don't garantee that they will help on winning any competition, but I am hoping to learn by collaborating with everyone."},{"metadata":{},"cell_type":"markdown","source":"# **Acknowledgements:**\n*     I would like thank [Laura Fink](https://www.kaggle.com/allunia) for her [excellent notebook](https://www.kaggle.com/allunia/pulmonary-dicom-preprocessing) on Dicom processing. I used some snippets to setup the DiCom environment and produce file path names to read the dataset, as well as to plot images[](http://).\n*     I also thank [Ian Pan](https://www.kaggle.com/vaillant) for the very insightful [discussion posting](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930) explaining the importance of the CT Window and how to implement it. I used the snippet in that post to convert the DiCom images using the PE Specific Window as described in his posting.\n    "},{"metadata":{},"cell_type":"markdown","source":"# **Objective:** \nThe objective of this notebook is to help contributors to the [RSNA competition](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection) to get an jump start in using [TFRecords](https://www.tensorflow.org/tutorials/load_data/tfrecord) in order to manage the huge size of the input dataset (close to 1TB!). The strategy I propose is to use Google Cloud Storage (GCS) to store the dataset in TFRecord format and then use the TF Dataset object to read it from different platforms (CPU, GPU, TPU, etc..). This topic was previously explained using a JPEG dataset in [this notebook](https://www.kaggle.com/paultimothymooney/convert-kaggle-dataset-to-gcs-bucket-of-tfrecords), so I thought that an adaptation to a DiCom dataset would give users a jump start.  \n\nNotice that you will get a huge benefit by managing your datasets in GCS in this competition, due to the large amount of data. It is also very important that your model be able to manage the memory used while ingesting the dataset. This is true if you are training using CPU, GPU or TPUs. The best way to this is to use the TFRecord format, which you can then read it directly into a [tf.dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). The tf.dataset has amazing memory management and support for ingestion pipelines. This will give you \"superpowers\" for training with large datasets, in any computing platform.\n\nUsing the TFRecord format for storing data should be easy, but unfortunately it requires data serialization which complicates it a little bit. This is done using [protocol buffers](https://developers.google.com/protocol-buffers/) and that is a bit of a learning curve. But in ML you only need to understand the [TFExample](https://www.tensorflow.org/api_docs/python/tf/train/Example) format. In this Notebook I provide a little template code for dealing with TFExamples that can be quickly customized for any type of data. This template is explained in detail in [this tutorial](https://www.tensorflow.org/tutorials/load_data/tfrecord); but you don't need to read all this, in this Notebook I provide an example specific for image data that you can quickly customize."},{"metadata":{},"cell_type":"markdown","source":"# **Setup:**\nThis notebook can use either local storage or GCS storage. The path you follow is controlled by the GCP_SDK_ENABLED variable below. I set the default for local storage (GCP_SDK_ENABLED = False) to enable users to quickly run the example in local storage just for learning purposes. But the real value of this Notebook is the capability to load large datasets to GCS storage. This is enabled by setting (GCP_SDK_ENABLED = True) in the cell below. To enable the GCP SDK you must also link this Notebook to a GCP project using \"Add-ons->Google Cloud SDK\" in this Notebook. \n\nYou also need to create a GCS bucket in the linked project and provide the bucket name in the BUCKET_NAME variable below. If you get a \"permission denied\" error when reading or writing it will be because either SDK was not enabled or the bucket name was not correct. If you get a \"NOT FOUND\" error it is probably because the GCS Bucket does not exist or was mispelled. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# After you have linked your project change this flag to True to use GCS and execute the next cell as well to enable access\n# To link your GCP project see Menu \"Add-ons->Google Cloud SDK\"\nGCP_SDK_ENABLED = False\n\n## IMPORTANT\n## YOU MUST MODIFY THE LINE BELOW WITH THE NAME OF THE BUCKET YOU ARE USING IN GCS\nBUCKET_NAME = \"gs://your-unique-bucket-name-here\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After you enabled the GCP SDK as described above, you can then grant Tensorflow permission to read and write to your GCS bucket. This is a great feature, it will also allow a TPU to read your datasets. If you get a permission denied at any point, make sure your project is linked and re-run this cell."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Authorize Tensorflow to write directly to GCS\n# This requires this Notebook to be linked to a GCP project, see Menu \"Add-ons->Google Cloud SDK\"\n# After linking the project, import credentials and authorize Tensorflow as follows\n\nfrom kaggle_secrets import UserSecretsClient\n\nif( GCP_SDK_ENABLED):\n    user_secrets = UserSecretsClient()\n    user_credential = user_secrets.get_gcloud_credential()\n    user_secrets.set_tensorflow_credential(user_credential)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Steps:\n1. [Install gdcm package, import packages](#install)\n2. [Build Input File List from RSNA PE Challenge Dataset](#par2)\n3. [Read DiCom images](#par3) \n4. [Define CT Window Function](#par4)\n5. [Define the TFRecord Format](#par5)\n6. [Write TFRecord to Storage](#par6)\n7. [Reload TFRecord from Storage to Test Correctness](#par7)\n8. [Scale up! Upload a range of studies to GCS](#par8)"},{"metadata":{},"cell_type":"markdown","source":"# 1. Install gcdm Package, import packages <a class=\"anchor\" id=\"install\"></a>"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!conda install -c conda-forge gdcm -y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport pydicom\nimport scipy.ndimage\nimport gdcm\n\nfrom os import listdir, mkdir\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Build Input File List from RSNA PE Challenge Dataset <a class=\"anchor\" id=\"par2\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"listdir(\"../input/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"basepath = \"../input/rsna-str-pulmonary-embolism-detection/\"\nlistdir(basepath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(basepath + \"train.csv\")\ntest_df = pd.read_csv(basepath + \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a list of unique Study Ids\nlist_of_studies = train_df.StudyInstanceUID.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_of_studies.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a list of file directories for each study \ntrain_df[\"dcm_path\"] = basepath + \"train/\" + train_df.StudyInstanceUID + \"/\" + train_df.SeriesInstanceUID\nlist_of_directories = train_df.dcm_path.unique()\nlist_of_directories.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Load DiCom Images for a Given Study <a class=\"anchor\" id=\"par3\"></a>\nThe loading function was borrowed by [this posting](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930). "},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dicom_array(dcm_path):\n    #dicom_files = glob.glob(osp.join(f, '*.dcm'))\n    #dicoms = [pydicom.dcmread(d) for d in dicom_files]\n    dicom_files = listdir(dcm_path)\n    dicoms = [pydicom.dcmread(dcm_path + \"/\" + file) for file in listdir(dcm_path)]\n    M = float(dicoms[0].RescaleSlope)\n    B = float(dicoms[0].RescaleIntercept)\n    # Assume all images are axial\n    z_pos = [float(d.ImagePositionPatient[-1]) for d in dicoms]\n    dicoms = np.asarray([d.pixel_array for d in dicoms])\n    dicoms = dicoms[np.argsort(z_pos)]\n    dicoms = dicoms * M\n    dicoms = dicoms + B\n    return dicoms, np.asarray(dicom_files)[np.argsort(z_pos)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dicom_imgs, img_names = load_dicom_array(list_of_directories[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,3))\nax[0].set_title(\"Original CT-scan\")\nax[0].imshow(dicom_imgs[0], cmap=\"bone\")\nax[1].set_title(\"Pixelarray distribution\");\nsns.distplot(dicom_imgs[0].flatten(), ax=ax[1]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Define CT Window Function <a class=\"anchor\" id=\"par4\"></a>\nThe CT Window function below was suggested [in this posting](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def CT_window(img, WL=50, WW=350):\n    upper, lower = WL+WW//2, WL-WW//2\n    X = np.clip(img.copy(), lower, upper)\n    X = X - np.min(X)\n    X = X / np.max(X)\n    #X = (X*255.0).astype('uint8')\n    return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"windowed_ct = CT_window(dicom_imgs[0], 100, 700)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,3))\nax[0].set_title(\"PE Specific CT-scan\")\nax[0].imshow(windowed_ct, cmap=\"bone\")\nax[1].set_title(\"Pixelarray distribution\");\nsns.distplot(windowed_ct.flatten(), ax=ax[1]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Define TFRecord Format <a class=\"anchor\" id=\"par5\"></a>\nThe code that follows is the TFExample template as explained [here](https://www.tensorflow.org/tutorials/load_data/tfrecord). Here is a quick explanation of how this works. For serialization using TFExample, we have to make any data fit into either one of 3 types:\n* bytes_feature\n* float_feature\n* int_64_feature\n\nImage data and strings are coded as bytes_feature. any other type is either float or int. The image_example will produce recrods with the types needed by a model. The image_raw field will contain the image bytes, the other fields are metadata. The pred_label field will be used to indicate if PE is present in the image.\n\nThis is a very useful snippet that you can customize if you want to define your own TFRecord type with specific fields. This template code is associated with a \"TFRecord Write\" procedure and a \"TFRecord Read\" procedure. So, to deal with TFRecords you need the \"definition\" snippet below, the [write snippet](#par6) and the [read snippet](#par7) -- all provided in this Notebook. You can use these 3 snippets as a template and customize them to add fields, etc... But keep in mind that you keep the data type definitions of the 3 snippets in sync."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the TFExample Data type for training models\n# Our TFRecord format will include the CT Image and metadata of the image, including the prediction label (is PE present)\n\nimport tensorflow as tf\n\n# Utilities serialize data into a TFRecord\ndef _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n\ndef image_example(image, study_id, image_name, pred_label):\n    image_shape = image.shape\n    image_bytes = image.tostring()\n    feature = {\n        'height': _int64_feature(image_shape[0]),\n        'width': _int64_feature(image_shape[1]),\n        'image_raw': _bytes_feature(image_bytes),\n        'study_id': _bytes_feature(study_id.encode()),\n        'img_name': _bytes_feature(image_name.encode()),\n        'pred_label':  _int64_feature(pred_label)\n    }\n    return tf.train.Example(features=tf.train.Features(feature=feature))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Write TFRecord to Storage <a class=\"anchor\" id=\"par6\"></a>"},{"metadata":{},"cell_type":"markdown","source":"This is the snippet that defines a function to write TFRecords to Storage. It is very interesting in this example that the tf.io.TFRecordWriter can write data straight to GCS!!! Note that if you have the GCP_SDK_ENABLED it will use a GCS path to write directly. If you get a permission problem make sure your project is linked to GCP and that you have a bucket in that project with the correct name, check the variables in the first cell of this Notebook. If you get a \"NOT FOUND\" it is likely the bucket name was wrong or does not exist. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define function to write the TFRecord\n# First, process each image into `tf.Example` messages.\n# Then,TFRecordWriter to write to a `.tfrecords` file.\n\n\nPE_WINDOW_LEVEL = 100\nPE_WINDOW_WIDTH = 700\n\ndef create_tfrecord( study_id, study_path, sdk_enabled=False):\n    if(sdk_enabled):\n        storage_file_path = BUCKET_NAME+\"/RSNA_PE/PE_Window_512/train/\"+study_id+\".tfrecords\"\n    else:\n        storage_file_path = '/kaggle/working/'+study_id+'.tfrecords'\n\n    study_images, study_image_file_names = load_dicom_array(study_path)\n    num_records = study_images.__len__()\n\n    total_records = 0\n    with tf.io.TFRecordWriter(storage_file_path) as writer:\n        for index in range(num_records):\n            img_file_name = study_image_file_names[index]\n            img_name = img_file_name.split(\".\")[0]\n            img_data = train_df.loc[train_df[\"SOPInstanceUID\"] == img_name]\n            pred_label = img_data[\"pe_present_on_image\"].values[0]\n            #print(\"pred_label \",pred_label)\n            windowed_image = CT_window(study_images[index], PE_WINDOW_LEVEL, PE_WINDOW_WIDTH)\n            tf_example = image_example(windowed_image, study_id, img_name, pred_label)\n            writer.write(tf_example.SerializeToString())\n            total_records = total_records + 1\n            print(\"*\",end='')\n            #print(\"wrote {}\".format(img_name))\n        writer.close()\n        \n    print(\"wrote {} records\".format(total_records))\n    return total_records","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This will write to local storage, just as a test. You can then refresh the output diretory and see the record file\nnum_records = create_tfrecord( list_of_studies[0], list_of_directories[0], False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# now write to Cloud Storage (if GCP_SDK_ENABLED)\nnum_records = create_tfrecord( list_of_studies[0], list_of_directories[0], GCP_SDK_ENABLED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_records","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At this point you can use the GCS storage browser in your project you should see the TFRecord file just created."},{"metadata":{},"cell_type":"markdown","source":"# 7. Reload TFRecord from Storage to Test Correctness <a class=\"anchor\" id=\"par7\"></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dictionary describing the features.\nimage_feature_description = {\n    'height': tf.io.FixedLenFeature([], tf.int64),\n    'width': tf.io.FixedLenFeature([], tf.int64),\n    'image_raw': tf.io.FixedLenFeature([], tf.string),\n    'study_id': tf.io.FixedLenFeature([], tf.string),\n    'img_name': tf.io.FixedLenFeature([], tf.string),\n    'pred_label': tf.io.FixedLenFeature([], tf.int64)\n}\n\ndef _parse_image_function(example_proto):\n  # Parse the input tf.Example proto using the dictionary above.\n  return tf.io.parse_single_example(example_proto, image_feature_description)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# and now we can reload the dataset again directly from GCS\n\nsample_study_id = list_of_studies[0]\nif(GCP_SDK_ENABLED):\n    storage_file_path = BUCKET_NAME+\"/RSNA_PE/PE_Window_512/train/\"+sample_study_id+\".tfrecords\"\nelse:\n    storage_file_path = '/kaggle/working/'+sample_study_id+'.tfrecords'\n\nencoded_image_dataset = tf.data.TFRecordDataset(storage_file_path)\nparsed_image_dataset = encoded_image_dataset.map(_parse_image_function)\nparsed_image_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# extract a record from the dataset and display the image\ndef load_dataset(dataset) :\n    reloaded_images = []\n    img_mtd = []\n    i=0\n    for image_features in dataset.as_numpy_iterator():\n        i=i+1\n        sample_image = np.frombuffer(image_features['image_raw'], dtype='float64')\n        mtd = dict()\n        mtd['width'] = image_features['width']\n        mtd['height'] = image_features['height']\n        mtd['study_id'] = image_features['study_id'].decode()\n        mtd['img_name'] = image_features['img_name'].decode()\n        mtd['pred_label'] = image_features['pred_label']                                  \n        reloaded_images.append(sample_image.reshape(mtd['width'],mtd['height'])) \n        img_mtd.append(mtd)\n    return reloaded_images, img_mtd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(len(sample_array))\nreloaded_images, img_mtd = load_dataset(parsed_image_dataset)\nprint(reloaded_images[0].shape)  \nprint(img_mtd[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,3))\nax[0].set_title(\"Reloaded CT-scan {}\".format(img_mtd[0]['img_name']))\nax[0].imshow(reloaded_images[0], cmap=\"bone\")\nax[1].set_title(\"Pixelarray distribution\");\nsns.distplot(reloaded_images[0].flatten(), ax=ax[1]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Scale up! Upload a range of studies to GCS <a class=\"anchor\" id=\"par8\"></a>\nWe can load a range of studies using the code below, but I found out that are over 7279 studies and that would take days! I don't think training over the entire 1T dataset is the best strategy here... Maybe it is best to select a subset. The functions I provide here take a list of studies as input, so you can trim it to a specific subset of interest. In the example below a write the 3 first studies."},{"metadata":{"trusted":true},"cell_type":"code","source":"num_studies = list_of_studies.shape[0]\nlower_range = 0\nupper_range = 3\n# you can upload up to num_studies, which is more than 17,000 and will take days. \nfor index in range(lower_range,upper_range):\n    print(\"processing study {} out of {}\".format(index,num_studies))\n    print(\"writing tfrecords for {}\".format(list_of_studies[index]))\n    num_records = create_tfrecord(list_of_studies[index], list_of_directories[index], GCP_SDK_ENABLED)\n    print(\"wrote {} records\".format(num_records))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}