{"cells":[{"metadata":{},"cell_type":"markdown","source":"This Notebook creates Tensorflow Records for the Pulmonary Embolism competition.\n\nIt is heavily derived from [https://www.kaggle.com/cdeotte/how-to-create-tfrecords] Courtesy of Chris Deotte.\n\n\nTFRecords can be used by CPU, GPU and TPU. To use them with a TPU I usually make them public. There are ways to use them when they are private, but there is some configuration involved that I haven't tackled.\n\nThere are two ways to get these TFRecords into a dataset:\n\n1. If you run this notebook in interactive mode, download the output and then upload to a dataset.\n\n2. If you run in Commit mode, you can send the output files directly to a dataset. \n\n\nThis notebook stores Images, targets and metadata from the DICOM files in the TFRecords. They can be used in models that combine Image and Metadata into one model.\n\nFirst, we import our typical libraries (probably more than I need for this notebook).\n\nPaths point to the train dataset and a location to store the TFRec files."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np, pandas as pd, os\nimport matplotlib.pyplot as plt, cv2\nimport tensorflow as tf, re, math\nimport random\nimport csv\nimport pydicom\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.utils import Sequence\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.backend\nimport tensorflow.keras.layers as L\nfrom skimage.transform import resize\n\ntrain_or_test = 'train'\n\nmypath = '../input/rsna-str-pulmonary-embolism-detection'\nimages_folder =    mypath+'/'+train_or_test\ntfrec_folder = ''\n\ncsv_file = mypath + '/' + train_or_test + '.csv'\nmetadata_file = 'metadata_' + train_or_test + '.csv'\n\n# change this prefix _a_ -> _b_, etc for each time you run this routine\nprefix = train_or_test + '_tfrec_version1_a_-'\n\n# for processing, break the Train data into smaller chunks.\n\ndf_start = 0\ndf_size = 20000\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First we will extract the DICOM Metadata.\n\nThis routine reads a DICOM directory and gets the Metadata from all the images.\n\nI use try/except blocks since there is no guarantee that a DICOM Tag exists in every file. Pydicom also lets you explicitly check for a tag before reading it.\n\nI provide default values if the tag doesn't exist.\n\nI write these out to a file, which can be Downloaded. I'll use it as we create TFRecords.\n\nI break out Tags that contain arrays into separate elements. For some, I just take the first element."},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_DICOM_attributes(folder):\n    images = list(os.listdir(os.path.join(folder)))\n    for image in images:\n        image_name = image.split(\".\")[0]\n        dicom_file_path = os.path.join(folder,image)\n        ds = pydicom.read_file(dicom_file_path)\n        myStudyInstanceUID = ds.StudyInstanceUID.strip()\n        mySeriesInstanceUID = ds.SeriesInstanceUID.strip()\n        mySOPInstanceUID = ds.SOPInstanceUID.strip()\n        slicethickness = ds.SliceThickness\n        rows = ds.Rows\n        columns = ds.Columns\n        pixelspacing0 = ds.PixelSpacing[0]\n        pixelspacing1 = ds.PixelSpacing[1]\n        wc = ds.WindowCenter\n        if hasattr(wc, \"__len__\"):\n            wc = wc[0]\n        ww = ds.WindowWidth\n        if hasattr(ww, \"__len__\"):\n            ww = ww[0]\n        ri = ds.RescaleIntercept\n        rs = ds.RescaleSlope\n        try:\n            pp = ds.PatientPosition\n        except:\n            pp = 'FFS'\n        try:\n            ipp0 = ds.ImagePositionPatient[0]\n        except:\n            ipp0 = 0\n        try:\n            ipp1 = ds.ImagePositionPatient[1]\n        except:\n            ipp1 = 0\n        try:\n            ipp2 = ds.ImagePositionPatient[2]\n        except:\n            ipp2 = 0\n        try:\n            io0 = ds.ImageOrientationPatient[0]\n        except:\n            io0 = 0\n        try:\n            io1 = ds.ImageOrientationPatient[1]\n        except:\n            io1 = 0\n        try:\n            io2 = ds.ImageOrientationPatient[2]\n        except:\n            io2 = 0\n        try:\n            io3 = ds.ImageOrientationPatient[3]\n        except:\n            io3 = 0\n        try:\n            io4 = ds.ImageOrientationPatient[4]\n        except:\n            io4 = 0\n        try:\n            io5 = ds.ImageOrientationPatient[5]\n        except:\n            io5 = 0\n        try:\n            inum = ds.InstanceNumber\n        except:\n            inum = 0\n        try:\n            kvp = ds.KVP\n        except:\n            kvp = 0\n        try:\n            tc = ds.XRayTubeCurrent\n        except:\n            tc = 0\n        try:\n            exposure = ds.Exposure\n        except:\n            exposure = 0\n\n        mystring= myStudyInstanceUID + ',' + \\\n            mySeriesInstanceUID + ',' + \\\n            mySOPInstanceUID + ',' + \\\n            str(slicethickness) + ',' + \\\n            str(pixelspacing0) + ',' + \\\n            str(pixelspacing1) + ',' + \\\n            str(wc) + ',' + \\\n            str(ww) + ',' + \\\n            str(ri) + ',' + \\\n            str(rs) + ',' + \\\n            str(rows) + ',' + \\\n            str(columns) + ',' + \\\n            str(pp) + ',' + \\\n            str(ipp0) + ',' + \\\n            str(ipp1) + ',' + \\\n            str(ipp2) + ',' + \\\n            str(io0) + ',' + \\\n            str(io1) + ',' + \\\n            str(io2) + ',' + \\\n            str(io3) + ',' + \\\n            str(io4) + ',' + \\\n            str(io5) + ',' + \\\n            str(inum) + ',' + \\\n            str(kvp) + ',' + \\\n            str(tc) + ',' + \\\n            str(exposure) + '\\n'\n        file1.write(mystring)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To break things up, we specify a start position in the train file list and a number of files to get.\n\nWe sort the train.csv file and train images directory list so they are in the same order.\n\nThen we look through the patients and add their DICOM attributes to a metadata file. I read through the metadata first so that I can do post-processing on the metadata within each patient before I write the TFRecords (not present in this notebook). In practice I captured all the metadata once and just reference a file of the full metadata. In this stand-alone notebook, I build it on the fly. Also, for the private test data, you will need to build it on the fly anyway.\n\nIt would be more efficient to combine those two steps and cache the DICOM data, calculate on the patient level and then write the TFRecords."},{"metadata":{"trusted":true},"cell_type":"code","source":"file1 = open(metadata_file,'w')\nfile1.write('StudyInstanceUID,SeriesInstanceUID,SOPInstanceUID,dcm_slice_thickness,dcm_pixel_spacing0,dcm_pixel_spacing1,dcm_window_center,dcm_window_width,dcm_rescale_intercept,dcm_rescale_slope,dcm_rows,dcm_columns,dcm_patient_position,dcm_image_position_patient0,dcm_image_position_patient1,dcm_image_position_patient2,dcm_image_orientation0,dcm_image_orientation1,dcm_image_orientation2,dcm_image_orientation3,dcm_image_orientation4,dcm_image_orientation5,dcm_instance_number,dcm_kvp,dcm_xray_tube_current,dcm_exposure\\n')\n\nmydirs = os.listdir(images_folder)\nmydirs.sort()\n\nprint('only reading limited records', df_size)\ndf = pd.read_csv(csv_file)\ndf = df.sort_values(by=['StudyInstanceUID','SOPInstanceUID'])\ndf = df[df_start:(df_start+df_size)].reset_index(drop=True)\n\ndf_temp = df\ndf_patients = df_temp['StudyInstanceUID'].unique()\npatient_count = len(df_patients)\nprint('only reading limited patients', patient_count)\n\ncount = 0\nfor mydir in mydirs[0:patient_count]:\n    SeriesUID = os.listdir(images_folder+'/'+mydir)\n    extract_DICOM_attributes(images_folder+'/'+mydir+'/'+SeriesUID[0])\n    count = count + 1\n    if count//100 == count/100:\n        print('patient count processed',count)\nfile1.close()\n\nmd = pd.read_csv(metadata_file)\ndf_count = len(md)\nprint('image count (adjusted to get full exams)',df_count)\nprint('done')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we include standard Tensorflow routines for different datatypes. I think this is straight from the Tensorflow examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we have a custom routine that lists all the variables that we want to store in the Tensorflow records.\n\nNote that the first variable is the image itself. The rest are from the train.csv file or the DICOM files."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def serialize_example(feature0, feature1, feature2, feature3, feature4, feature5, feature6,\\\n    feature7, feature8, feature9, feature10,\\\n    feature11, feature12, feature13, feature14, feature15, feature16, feature17, \\\n    feature18, feature19, feature20, feature21, feature22, feature23, feature24, \\\n    feature25, feature26, feature27, feature28, feature29, feature30, feature31):\n  feature = {\n      'image': _bytes_feature(feature0),\n      'StudyInstanceUID': _bytes_feature(feature1),\n      'SeriesInstanceUID': _bytes_feature(feature2),\n      'SOPInstanceUID': _bytes_feature(feature3),\n      'pe_present_on_image': _int64_feature(feature4),\n      'negative_exam_for_pe': _int64_feature(feature5),\n      'qa_motion':  _int64_feature(feature6),\n      'qa_contrast':  _int64_feature(feature7),\n      'flow_artifact':  _int64_feature(feature8),\n      'rv_lv_ratio_gte_1':  _int64_feature(feature9),\n      'rv_lv_ratio_lt_1':  _int64_feature(feature10),\n      'leftsided_pe': _int64_feature(feature11),\n      'chronic_pe':  _int64_feature(feature12),\n      'true_filling_defect_not_pe':  _int64_feature(feature13),\n      'rightsided_pe': _int64_feature(feature14),\n      'acute_and_chronic_pe': _int64_feature(feature15),\n      'central_pe': _int64_feature(feature16),\n      'indeterminate': _int64_feature(feature17),\n      'dcm_slice_thickness': _float_feature(feature18),\n      'dcm_pixel_spacing0': _float_feature(feature19),\n      'dcm_patient_position': _bytes_feature(feature20),\n      'dcm_image_position_patient2': _float_feature(feature21),\n      'dcm_image_orientation0': _float_feature(feature22),\n      'dcm_image_orientation1': _float_feature(feature23),\n      'dcm_image_orientation2': _float_feature(feature24),\n      'dcm_image_orientation3': _float_feature(feature25),\n      'dcm_image_orientation4': _float_feature(feature26),\n      'dcm_image_orientation5': _float_feature(feature27),\n      'dcm_instance_number': _int64_feature(feature28),\n      'dcm_kvp': _int64_feature(feature29),\n      'dcm_xray_tube_current': _int64_feature(feature30),\n      'dcm_exposure': _int64_feature(feature31)\n  }\n  example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n  return example_proto.SerializeToString()\n\ndef serialize_example_image_only(feature0,feature1):\n  feature = {\n      'image': _bytes_feature(feature0),\n      'SOPInstanceUID': _bytes_feature(feature1),\n  }\n  example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n  return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The window and level function is a pretty standard routine. (I'm note sure who claims original authorship.)\n\nThe DICOM data comes in Hounsfield units, which are being converted to grayscale. The range is roughly -1000 to 3000, so we pick a subset of the range that is clinically appropriate to our task.\n\nI read the Window/Level (also called Window/Center) from the DICOM file. For this Competition, I am actually plugging in a standard W/C rather than using the DICOM entries. But I keep the code in case I want to use the actual DICOM values.\n\nNote that WindowCenter and WindowLevel can be multi-valued. I just get the first entry.\n\nI get the image data from the DICOM file within a \"try:\" expression. Some of the image data requires a library GDCM for decompression. This is tricky to install (see Melanoma contest for a wide range of attempts). At least in the Train Dataset, they are not that common, so I just replace them with a blank image.\n\nI then define a \"crop\" function. CT images have a lot of space on the edges and a pulmonary embolism will never be outside the lungs. How much to crop is based on trial and error."},{"metadata":{"trusted":true},"cell_type":"code","source":"def window_image(img, window_center,window_width, intercept, slope):\n    window_width = abs(window_width)\n    img = (img*slope +intercept)\n    img_min = window_center - window_width//2\n    img_max = window_center + window_width//2\n    img[img<img_min] = img_min\n    img[img>img_max] = img_max\n    img = img - np.min(img)\n    img = img / np.max(img)\n    img = (img*255.0).astype('uint8')\n    return img \n\n\ndef dicom_window(dcm_data):\n    if 'RescaleIntercept' in dcm_data:\n        RI = dcm_data.RescaleIntercept\n    else:\n        RI = -1024\n    if 'RescaleSlope' in dcm_data:\n        RS = dcm_data.RescaleSlope\n    else:\n        RS = 1\n    if 'WindowCenter' in dcm_data:\n        WC = dcm_data.WindowCenter\n        if hasattr(WC, \"__len__\"):\n            WC = WC[0]\n    else:\n        WC = 40\n    if 'WindowWidth' in dcm_data:\n        WW = dcm_data.WindowWidth\n        if hasattr(WW, \"__len__\"):\n            WW = WW[0]\n    else:\n        WW = 400\n# get image from DICOM file. Sometimes error due to missing GDCM library.\n    try:\n        img = dcm_data.pixel_array\n    except:\n        print('image error ',dcm_data.SOPInstanceUID)\n        img = np.zeros(shape=(512,512))\n# Here I replace the above parameters with a standard PE specific Window/Level. Idea courtesy of Ian Pan \n# Comment out these two lines if you want to use the W/L stored in the DICOM files.\n    WC = 100\n    WW = 700\n    img = window_image(img,WC,WW,RI,RS)\n    return(img)\n\ndef crop_center(img,cropx,cropy):\n    y,x = img.shape\n    startx = x//2-(cropx//2)\n    starty = y//2-(cropy//2)    \n    return img[starty:starty+cropy,startx:startx+cropx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Because of the size of the dataset, I break it into manageable chunks. Typically 100,000 images at a time. Not perfect, because it could duplicate patients across TFRecords.\n\nI read train.csv and the new metadata file and merge them."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('only reading limited records', df_size)\ndf = pd.read_csv(csv_file)\ndf = df[df_start:(df_start+df_size)].reset_index(drop=True)\ndf['name'] = df['StudyInstanceUID'] + '/' + df['SeriesInstanceUID'] + '/' + df['SOPInstanceUID'] + '.dcm'\ndf['name2'] = df['StudyInstanceUID'] + '/' + df['SOPInstanceUID']\n\n#add dummy placeholder columns for non-existent test metadata. \nif train_or_test == 'test':\n    df = df.assign(pe_present_on_image = 0)\n    df = df.assign(negative_exam_for_pe = 0)\n    df = df.assign(qa_motion = 0)\n    df = df.assign(qa_contrast = 0)\n    df = df.assign(flow_artifact = 0)\n    df = df.assign(rv_lv_ratio_gte_1 = 0)\n    df = df.assign(rv_lv_ratio_lt_1 = 0)\n    df = df.assign(leftsided_pe = 0)\n    df = df.assign(chronic_pe = 0)\n    df = df.assign(true_filling_defect_not_pe = 0)\n    df = df.assign(rightsided_pe = 0)\n    df = df.assign(acute_and_chronic_pe = 0)\n    df = df.assign(central_pe = 0)\n    df = df.assign(indeterminate = 0)\n\nmd = pd.read_csv(metadata_file)\nmd['name2'] = md['StudyInstanceUID'] + '/' + md['SOPInstanceUID']\ndel md['StudyInstanceUID']\ndel md['SeriesInstanceUID']\ndel md['SOPInstanceUID']\n\ndf = pd.merge(df,md,on='name2')\n\nfilenames = df['name'].copy()\n\nprint(train_or_test, 'count',len(filenames))                        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The main routine to read the DICOM images and train.csv data and write the TFRecords. Written in 10,000 record chunks.\n\nAll the images in this competition seem to be 512x512, but I check anyway and resize if necessary.\n\nThen I crop to 400x400 because there is a lot of \"air\" on the edge of the patients and pulmonary embolism is never in the soft tissues of the chest. Value is just be some trial and error and hasn't been proven to be useful.\n\nThen I resize to 256 x 256 for my final images.\n\nI leave the images as Gray-Scale. Ian Pan (in his separate JPEG Dataset) uses the color channels to store three different Windows/Levels. I'm not sure if that takes up more disk storage. Probably a good idea, but I haven't implemented it yet.\n\nI'll change the Gray-Scale images to RGB during training/testing since I use Models that are pretrained on color images."},{"metadata":{"trusted":true},"cell_type":"code","source":"#optional shuffle for randomization\n#random.shuffle(filenames)\n\nSIZE = 10000\nIMGS = filenames\nCT = len(IMGS)//SIZE + int(len(IMGS)%SIZE!=0)\nfor j in range(CT):\n    print(); print('Writing TFRecord %i of %i...'%(j,CT))\n    CT2 = min(SIZE,len(IMGS)-j*SIZE)\n    with tf.io.TFRecordWriter(tfrec_folder + prefix + '%.2i-%i.tfrec'%(j,CT2)) as writer:\n        for k in range(CT2):\n            thisimg = SIZE*j+k\n            thisfile = IMGS[thisimg]\n            ds = pydicom.read_file(images_folder+'/'+thisfile)\n            img = dicom_window(ds)\n            if img.shape != [512,512]:\n                img = resize(img, (512,512), \\\n                   mode='reflect', anti_aliasing=True,preserve_range=True)\n            img = crop_center(img,400,400)\n            img = resize(img, (256,256), \\\n                   mode='reflect', anti_aliasing=True,preserve_range=True)\n            img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n            name = IMGS[SIZE*j+k].split('.')[0]\n            \n            example = serialize_example(\n                img,\n                str.encode(df.loc[df['name']==thisfile, 'StudyInstanceUID'].iloc[0]),\n                str.encode(df.loc[df['name']==thisfile, 'SeriesInstanceUID'].iloc[0]),\n                str.encode(df.loc[df['name']==thisfile, 'SOPInstanceUID'].iloc[0]),\n                df.loc[df['name']==thisfile, 'pe_present_on_image'].iloc[0],\n                df.loc[df['name']==thisfile, 'negative_exam_for_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'qa_motion'].iloc[0],\n                df.loc[df['name']==thisfile, 'qa_contrast'].iloc[0],\n                df.loc[df['name']==thisfile, 'flow_artifact'].iloc[0],\n                df.loc[df['name']==thisfile, 'rv_lv_ratio_gte_1'].iloc[0],\n                df.loc[df['name']==thisfile, 'rv_lv_ratio_lt_1'].iloc[0],\n                df.loc[df['name']==thisfile, 'leftsided_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'chronic_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'true_filling_defect_not_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'rightsided_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'acute_and_chronic_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'central_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'indeterminate'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_slice_thickness'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_pixel_spacing0'].iloc[0],\n                str.encode(df.loc[df['name']==thisfile, 'dcm_patient_position'].iloc[0]),\n                df.loc[df['name']==thisfile, 'dcm_image_position_patient2'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation0'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation1'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation2'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation3'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation4'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation5'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_instance_number'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_kvp'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_xray_tube_current'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_exposure'].iloc[0]\n            )\n            writer.write(example)\n            if k%1000==0: print(k,', ',end='')\nprint('done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Now we should test our files by reading a sample back.\n\nKeeping some header code (not needed in this notebook), so that this section can be a standalone notebook also.\n\nIf you use this section in a separate notebook, you'll need to update the location of the TFRecords to point to your stored DataSet."},{"metadata":{"trusted":true},"cell_type":"code","source":"# import tensorflow as tf \n# from tensorflow import keras \n# from tensorflow.keras.utils import Sequence \n# import tensorflow.keras.backend as K \n# import tensorflow.keras.backend \n# import tensorflow.keras.layers as L\n\n# # adjust to train or test if broken out to separate notebookd\n# #train_or_test = 'train'\n# #train_or_test = 'test'\n\n# # adjust to your directory if not working directory. Will need \"/\"\n# tfrec_folder = ''\n\n# #if accessing through a Dataset, enter name here:\n# my_GCS_PATH = 'your TFRecs Dataset here'\n\n# DEVICE = \"TPU\"\n\n# if DEVICE == \"TPU\":\n#     print(\"connecting to TPU...\")\n#     try:\n#         tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n#         print('Running on TPU ', tpu.master())\n#     except ValueError:\n#         print(\"Could not connect to TPU\")\n#         tpu = None\n\n# if tpu:\n#     try:\n#         print(\"initializing  TPU ...\")\n#         tf.config.experimental_connect_to_cluster(tpu)\n#         tf.tpu.experimental.initialize_tpu_system(tpu)\n#         strategy = tf.distribute.experimental.TPUStrategy(tpu)\n#         print(\"TPU initialized\")\n#     except _:\n#         print(\"failed to initialize TPU\")\n# else:\n#     DEVICE = \"GPU\"\n# if DEVICE != \"TPU\": \n#     print(\"Using default strategy for CPU and single GPU\") \n#     strategy = tf.distribute.get_strategy()\n\n# if DEVICE == \"GPU\": \n#     print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n\n# AUTO = tf.data.experimental.AUTOTUNE\n# REPLICAS = strategy.num_replicas_in_sync\n\n# CFG = dict(\n#     batch_size        = 32,\n#     read_size         = 256, \n#     crop_size         = 256, \n#     net_size          = 256)\n\n# import os, gc, random, cv2, csv\n# from datetime import datetime \n# import numpy as np\n# from numpy.random import seed \n# import pandas as pd\n# from skimage import measure #scikit-image conda-forge \n# from skimage.transform import resize \n# from sklearn.metrics import roc_auc_score #scikit-learn conda-forge \n# import tensorflow as tf \n# from tensorflow import keras \n# from tensorflow.keras.utils import Sequence \n# from matplotlib import pyplot as plt \n# import matplotlib.pyplot as plt \n# import matplotlib.image as mpimg \n# import pydicom #conda-forge\n# import PIL \n# from tqdm import tqdm\n\n# from kaggle_datasets import KaggleDatasets \n# mypath = '../input/metadata' \n\n# #If using the TPU, you have to use a different path.\n\n# if DEVICE == 'TPU': \n#     GCS_PATH = KaggleDatasets().get_gcs_path(my_GCS_PATH) \n# else: \n#     GCS_PATH = ''\n    \n# tfrec_filter = train_or_test + '*.tfrec'\n# print(tfrec_filter)\n\n# tfrec_files = np.sort(np.array(tf.io.gfile.glob(tfrec_filter)))\n# print('Number of files', len(tfrec_files))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Routines here to \n    1. Load images and metadata.\n    2. Augment the images (rotate, intensity, etc). Not needed here, but might be used in a training model."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord_metadata(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'StudyInstanceUID'             : tf.io.FixedLenFeature([], tf.string),\n        'SeriesInstanceUID'            : tf.io.FixedLenFeature([], tf.string),\n        'SOPInstanceUID'               : tf.io.FixedLenFeature([], tf.string),\n        'pe_present_on_image'          : tf.io.FixedLenFeature([], tf.int64),\n        'negative_exam_for_pe'         : tf.io.FixedLenFeature([], tf.int64),\n        'qa_motion'                    : tf.io.FixedLenFeature([], tf.int64),\n        'qa_contrast'                  : tf.io.FixedLenFeature([], tf.int64),\n        'flow_artifact'                : tf.io.FixedLenFeature([], tf.int64),\n        'rv_lv_ratio_gte_1'            : tf.io.FixedLenFeature([], tf.int64),\n        'rv_lv_ratio_lt_1'             : tf.io.FixedLenFeature([], tf.int64),\n        'leftsided_pe'                 : tf.io.FixedLenFeature([], tf.int64),\n        'chronic_pe'                   : tf.io.FixedLenFeature([], tf.int64),\n        'true_filling_defect_not_pe'   : tf.io.FixedLenFeature([], tf.int64),\n        'rightsided_pe'                : tf.io.FixedLenFeature([], tf.int64),\n        'acute_and_chronic_pe'         : tf.io.FixedLenFeature([], tf.int64),\n        'central_pe'                   : tf.io.FixedLenFeature([], tf.int64),\n        'indeterminate'                : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_slice_thickness'          : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_pixel_spacing0'           : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_patient_position'         : tf.io.FixedLenFeature([], tf.string),\n        'dcm_image_position_patient2'  : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation0'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation1'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation2'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation3'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation4'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation5'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_instance_number'          : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_kvp'                      : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_xray_tube_current'        : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_exposure'                 : tf.io.FixedLenFeature([], tf.int64),\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    #placeholder\n    metadata = [1]\n    target = [example['pe_present_on_image'],\n              example['negative_exam_for_pe'],\n              example['rv_lv_ratio_gte_1'],\n              example['rv_lv_ratio_lt_1'],\n              example['leftsided_pe'],\n              example['chronic_pe'],\n              example['rightsided_pe'],\n              example['acute_and_chronic_pe'],\n              example['central_pe'],\n              example['indeterminate']]\n\n    return example['image'], metadata, target\n\ndef prepare_image(img, cfg=None, augment=True):    \n    img = tf.image.decode_jpeg(img, channels=1)\n#    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    img = tf.cast(img, tf.float32) / 255.0\n    \n#    if augment:\n#        img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n#        img = tf.image.random_flip_left_right(img)\n#        img = tf.image.random_hue(img, 0.01)\n#        img = tf.image.random_saturation(img, 0.7, 1.3)\n#        img = tf.image.random_contrast(img, 0.8, 1.2)\n#        img = tf.image.random_brightness(img, 0.1)\n#    else:\n#        img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n                                   \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.image.grayscale_to_rgb(img, name=None)\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    return img\n\ndef get_dataset_metadata(files, cfg, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord_metadata, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord_metadata(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, metadata, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), metadata, imgname_or_label), \\\n                num_parallel_calls=AUTO)\n    ds = ds.map(lambda img, metadata, imgname_or_label: (tuple([img, metadata]), imgname_or_label), num_parallel_calls=AUTO)\n    ds = ds.batch(cfg['batch_size'] * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds\nprint(\"done\") \n\ndef show_dataset_metadata(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n    for idx, data in enumerate(iter(ds)):\n#        img, metadata, target_or_imgid = data\n        inputs, target_or_imgid = data\n        img = inputs[0]\n        metadata = inputs[1]\n#        print(metadata)\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n    display(mosaic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load a sample of images and display."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# files_sample = np.sort(np.array(tfrec_files))\n\n# ds = get_dataset_metadata(files_sample, CFG, shuffle=True,augment=True, labeled=True).unbatch().take(12*5)   \n# show_dataset_metadata(256, 2, 5, ds)\n# print(\"done\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}