{"cells":[{"metadata":{},"cell_type":"markdown","source":"# How does reading from shelve files in a DataLoader compare to the alternatives?\n\nOne of the things that will slow us down in the competition is how to read the .dicom files fast. One standard alternative is to save the X-ray images in some image format like jpg or png, but could it be faster to save the whole data in a `pkl` type single file that one can access by some index?\n\nIn [an EDA notebook](https://www.kaggle.com/bjoernholzhauer/eda-dicom-reading-vinbigdata-chest-x-ray) I created a prototype for putting the training data into `shelve` (like pickle, but allows parallel reading and access via dictionary keys - a persistent dictionary, really - see [the documentation](https://docs.python.org/3/library/shelve.html)). I resized the images so that the shortest dimension is at least 600 pixels, i.e. the images are probably still larger than you'd use as a model input, but the file we end up saving ends up being small enough for our purposes - primarily through saving the image itself as uint8 (which hopefully does not loose meaningful information).\n\nI did for the moment retaining the bounding boxes and class labels (and the information on which radiologist rad_id) for all radiologists gave this annotation. That way, one can then decide what to do - whether you want to somehow aggregate the annotations (e.g. like in [this notebook](https://www.kaggle.com/sreevishnudamodaran/vinbigdata-fusing-bboxes-coco-dataset)) or use the different annotations as data augmentations e.g. by using different ones in different epochs.\n\nIn this notebook I try to compare the different approaches in terms of speed. I'm going to get 8 batches of data to make sure that parallelized approaches don't get penalized too much for the initial set-up (as they would be, if we e.g. just looked at one batch). It turns out the `shelve` approach is fastest out of the approaches I tried - let me know, if I missed obvious alternatives."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport pandas as pd\nimport numpy as np\nimport torch\nimport albumentations\nimport random\nimport shelve\nfrom PIL import Image\nimport re\nimport itertools\nimport matplotlib.pyplot as plt\nimport fastcore\nfrom fastcore.parallel import ProcessPoolExecutor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\nlist_of_images = np.sort(np.unique(train['image_id'].values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define augmentations\nWe will need some augmentations that we want to apply to the images we load. We'll use some light augmentation. Omitting this might make our comparisons here unrealistic."},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT_SHAPE = 224\n\naugs = albumentations.Compose([\n            #albumentations.SmallestMaxSize(max_size=600), # Not necessary, if we are reading the shelve file (already done there)\n            albumentations.RandomResizedCrop(INPUT_SHAPE, INPUT_SHAPE, scale=(0.9, 1.0)),            \n            albumentations.ShiftScaleRotate(rotate_limit=10, p=0.5),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1,0.1), \n                contrast_limit=(-0.1, 0.1), \n                p=0.5\n            ),\n            albumentations.ISONoise()], \n    bbox_params=albumentations.BboxParams(format='pascal_voc'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define dataset and dataloader\n\nThis illustrates how slow this approach relying on `__getitem__`, skip to the next section for a much better approach."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class DetectionDataset:\n    def __init__(self, image_ids, augmentations=None):\n        self.image_ids, self.augmentations = image_ids, augmentations\n    def __len__(self):\n        return len(self.image_ids)\n    def __getitem__(self, item):        \n        with shelve.open('../input/eda-dicom-reading-vinbigdata-chest-x-ray/training_data.db', flag='r', writeback=False) as myshelf:\n            tmpdict = myshelf[self.image_ids[item]]        \n        rad_id = tmpdict['rad_id']        \n        which_rad_id = random.sample( list(np.unique(rad_id)), k=1)[0]\n        which_indices = [idx for idx, val in enumerate(rad_id) if val==which_rad_id]        \n        \n        image = np.stack([tmpdict['image']]*3).transpose(1,2,0)        \n        bboxes = tmpdict['bboxes'][which_indices]\n        class_labels = tmpdict['class_labels'][which_indices]        \n        transformed = self.augmentations(image=image, bboxes=bboxes, class_labels=class_labels)\n        #{'image': transformed['image'], 'bboxes': transformed['bboxes'], 'class_labels': transformed['class_labels']}\n        \n        return {'image': transformed['image'], 'bboxes': transformed['bboxes'], 'class_labels': transformed['class_labels']}\n\ndetdatset = DetectionDataset(image_ids=list_of_images, augmentations=augs)\nexample_loader = torch.utils.data.DataLoader(detdatset, batch_size=64, shuffle=True, num_workers=0, collate_fn=lambda x: x) #collate_fn=","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How fast is the DataLoader?\nLet's test this by getting 8 batches, repeatedly."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(example_loader, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's not great and a lot worse than reading individual image files (see further down), so we can do better."},{"metadata":{},"cell_type":"markdown","source":"# Define our own DataLoader that loads a whole batch\nOne of the stupid things in what I did above is that I keep opening the `shelve` file repeatedly for reading that's pretty inefficient and we can speed things up a lot by reading a whole batch at once! For the same reason this also results in a speedup vs. loading individual image files.\n\nTo do better, let's use a DataLoader that loads a whole batch in one opening of the file. To keep things modular and easier to modify, I'll first define a batched DataSet class and then use that in my DataLoader. That will makes things easier to modify/tinker with."},{"metadata":{"trusted":true},"cell_type":"code","source":"class BatchedDataSet:\n    \n    def __init__(self, image_ids, augmentations=None):\n        self.image_ids, self.augmentations = image_ids, augmentations\n        \n    def __len__(self):\n        return len(self.image_ids)\n    \n    def getbatch(self, items):\n        what_images = [self.image_ids[item] for item in items]\n        with shelve.open('../input/eda-dicom-reading-vinbigdata-chest-x-ray/training_data.db', flag='r', writeback=False) as myshelf:\n            tmpdict =  { key: myshelf[key] for key in what_images }\n        \n        for what_image in what_images:\n            rad_id = tmpdict[what_image]['rad_id']        \n            which_rad_id = random.sample( list(np.unique(rad_id)), k=1)[0]\n            which_indices = [idx for idx, val in enumerate(rad_id) if val==which_rad_id]        \n\n            image = np.stack([tmpdict[what_image]['image']]*3).transpose(1,2,0)        \n            bboxes = tmpdict[what_image]['bboxes'][which_indices]\n            class_labels = tmpdict[what_image]['class_labels'][which_indices]        \n            transformed = self.augmentations(image=image, bboxes=bboxes, class_labels=class_labels)\n            tmpdict[what_image] = {'image': transformed['image'], \n                                   'bboxes': transformed['bboxes'], \n                                   'class_labels': transformed['class_labels']}        \n        return tmpdict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's get to the actual DataLoader. This is what a very basic PyTorch DataLoader that loads data in batches from the `shelve` file looks like. The class needs to have the following methods:\n* `__init__`: for initializing an instance of the class, when we initialize it, we need to provide basic information such as the list of `image_id` values we will select from, the batch size and so on.\n* `__iter__`: Determines what happens at the start of each epoch, i.e. each time we start over in terms of going through the whole data. The main thing we need to do here is shuffling the batches if requested.\n* `__next__`: Generate a batch of data (or the **next** batch of data), this part is what makes this DataLoader different from your basic PyTorch DataLoader that would call `__getitem__` on the DataSet `batch_size` times per batch. Here, we instead get the whole batch at once, which leads to a major speedup, if (as is the case here) we can load a batch very efficiently all at once from a single file.\n* `__len__`: When we loop over the DataLoader for a training epoch, it's useful to know how many batches it returns."},{"metadata":{"trusted":true},"cell_type":"code","source":"class BatchedDataLoader:\n\n    def __init__(self, dataset, batch_size: int=32, shuffle: bool=False, drop_last: bool=False):\n        self.dataset, self.dataset_len, self.batch_size, self.shuffle, self.drop_last = dataset, len(dataset), batch_size, shuffle, drop_last\n\n        # Calculate # batches\n        n_batches, remainder = divmod(self.dataset_len, self.batch_size)\n        if (remainder > 0) & (drop_last==False):\n            n_batches += 1\n            self.n_items = self.dataset_len\n        else:\n            self.n_items = self.n_batches * self.batch_size\n        self.n_batches = n_batches\n        self.batch_list = [i for i in range(self.n_items)]\n\n    def __iter__(self):\n        if self.shuffle:\n            ridx = torch.randperm(self.dataset_len)\n            self.batch_list = [ridx[i] for i in range(self.n_items)]\n            \n        self.i = 0\n        return self\n\n    def __next__(self):\n        if self.i >= self.n_items:\n            raise StopIteration        \n        batch = self.dataset.getbatch( items = self.batch_list[self.i:min(self.n_items, self.i+self.batch_size)] )\n        self.i += self.batch_size\n        return batch\n\n    def __len__(self):\n        return self.n_batches","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batcheddataset = BatchedDataSet(image_ids=list_of_images, augmentations=augs)\ntrain_batches = BatchedDataLoader(dataset=batcheddataset, batch_size=64, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How fast is our own BatchLoader?"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(train_batches, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great! That's a massive speedup vs. loading each item one-at-a-time using the `__getitem__` of the data set we had defined as part of a standard PyTorch DataLoader. It's also a speedup vs. loading individual images without parallelization (see below), but there's not much in it vs. loading images with 2 or 4 workers. Clearly, we should parallelize this to outperform parallel processed image loading. "},{"metadata":{},"cell_type":"markdown","source":"# What does our BatchLoader return?\n\nLet's get a batch of data and see what the DataLoader produced."},{"metadata":{"trusted":true},"cell_type":"code","source":"example_batch = next(iter(train_batches))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's what the first two entries of a batch read from the shelve file looks like:"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_keys = [key for key in example_batch.keys()]\nfor key in example_keys[0:2]:\n    print(example_batch[key])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(example_batch[example_keys[0]]['image']);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_batch[example_keys[0]] #['bboxes']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_batch[example_keys[0]]['class_labels']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Let's add parallel processing in our DataLoader\n\n\nAs said in the previous Section, one obvious feature to add to our batch-DataLoader is parallel loading (i.e. multiple workers that are each responsible for one batch) and adding a `num_workers: int=0` option for that. The task of getting the data is clearly very parallelizable and by using `shelve` there's no problem with multiple workers trying to read from our file.\n\nNow, we'll use the `ProcessPoolExecutor` from `fastcore.parallel` - as described in Chapter 19 __A fastai Learner from Scratch__ of the great **Deep Learning for Coders with fastai & PyTorch** [book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527) - in order to parallelize our DataLoader."},{"metadata":{"trusted":true},"cell_type":"code","source":"def getbatch(items, augmentations):    \n    with shelve.open('../input/eda-dicom-reading-vinbigdata-chest-x-ray/training_data.db', flag='r', writeback=False) as myshelf:\n        tmpdict =  { key: myshelf[key] for key in items }\n        \n    for what_image in items:\n        rad_id = tmpdict[what_image]['rad_id']        \n        which_rad_id = random.sample( list(np.unique(rad_id)), k=1)[0]\n        which_indices = [idx for idx, val in enumerate(rad_id) if val==which_rad_id]        \n\n        image = np.stack([tmpdict[what_image]['image']]*3).transpose(1,2,0)        \n        bboxes = tmpdict[what_image]['bboxes'][which_indices]\n        class_labels = tmpdict[what_image]['class_labels'][which_indices]        \n        transformed = augmentations(image=image, bboxes=bboxes, class_labels=class_labels)\n        tmpdict[what_image] = {'image': transformed['image'], \n                               'bboxes': transformed['bboxes'], \n                               'class_labels': transformed['class_labels']}        \n    return tmpdict\n    \nclass BatchedParallelDataLoader:\n\n    def __init__(self, image_ids, augmentations, batch_size: int=32, shuffle: bool=False, drop_last: bool=False, n_workers: int=1):\n        self.dataset_len, self.batch_size, self.shuffle, self.drop_last, self.n_workers = len(image_ids), batch_size, shuffle, drop_last, n_workers\n        self.image_ids, self.augmentations = image_ids, augmentations\n\n        # Calculate # batches\n        n_batches, remainder = divmod(self.dataset_len, self.batch_size)\n        if (remainder > 0) & (drop_last==False):\n            n_batches += 1\n            self.n_items = self.dataset_len\n        else:\n            self.n_items = self.n_batches * self.batch_size\n        self.n_batches = n_batches\n        self.batch_list = [i for i in range(self.n_items)]\n\n    def __iter__(self):\n        if self.shuffle:\n            ridx = torch.randperm(self.dataset_len)\n            self.batch_list = [ridx[i] for i in range(self.n_items)]\n            \n        chunks = [ list(self.image_ids[ self.batch_list[i:min(self.n_items, i+self.batch_size)]]) for i in range(0, self.n_items, self.batch_size) ]\n        with ProcessPoolExecutor(self.n_workers) as ex:\n            yield from ex.map(getbatch, chunks, augmentations=self.augmentations)\n\n    def __len__(self):\n        return self.n_batches\n    \nbpdl = BatchedParallelDataLoader(n_workers=4, image_ids=list_of_images, augmentations=augs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How fast is the parallelized version of our DataLoader?"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(bpdl, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Woohoo! That clearly beats loading images in parallel with the standard PyTorch parallel DataLoader!"},{"metadata":{},"cell_type":"markdown","source":"# What if we only have 2 workers?\nThis is obviously an important question, because [currently](https://www.kaggle.com/docs/notebooks) a Kaggle GPU notebook gets 2 CPUs (although TPU notebooks get 4)."},{"metadata":{"trusted":true},"cell_type":"code","source":"bpdl = BatchedParallelDataLoader(n_workers=2, image_ids=list_of_images, augmentations=augs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(bpdl, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow! We did not actually gain that much by having 4 vs. 2 workers before. I would guess though that part of it might be how many batches we generate and maybe if we did a lot more we'd gain something with more workers."},{"metadata":{},"cell_type":"markdown","source":"# How does that compare to reading .dicom files on-the-fly?\n\nHere I'll follow the approach I also took in the [notebook](https://www.kaggle.com/bjoernholzhauer/eda-dicom-reading-vinbigdata-chest-x-ray#7.-Creating-fast-to-read-shelve-file) that created the shelve file."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport fastcore\nfrom fastcore.parallel import parallel\n\n# Using function from another great notebook: https://www.kaggle.com/raddar/convert-dicom-to-np-array-the-correct-way\ndef read_xray(path, voi_lut = True, fix_monochrome = True):\n    dicom = pydicom.read_file(path)\n    \n    # VOI LUT (if available by DICOM device) is used to transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)        \n    else:\n        data = dicom.pixel_array\n               \n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n        \n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    data = np.stack([data]*3).transpose(1,2,0)\n        \n    return data\n\n# This function will read a .dicom file, turn the smallest side to 600 pixels, and then save the additional annotations together with the image into a dictionary\ndef get_and_save(x):\n    \n    idx=x[0] \n    image_id=x[1]\n    \n    transform = albumentations.Compose([albumentations.SmallestMaxSize(max_size=600, always_apply=True)], bbox_params=albumentations.BboxParams(format='pascal_voc')) \n    img = read_xray(path='../input/vinbigdata-chest-xray-abnormalities-detection/train/' + image_id + '.dicom')\n    rad_id = np.array([int(re.findall(r'\\d+', rad_id)[0]) for rad_id in train.loc[train['image_id']==image_id, 'rad_id'].values], dtype=np.int8)\n    class_labels = train.loc[train['image_id']==image_id, 'class_id'].values    \n    bboxes = [list(row) for rowid, row in train.loc[train['image_id']==image_id, ['x_min', 'y_min', 'x_max', 'y_max', 'class_id']].fillna({'x_min':0, 'y_min':0, 'x_max':1, 'y_max':1}).astype(np.int16).iterrows() ]\n    \n    transformed = transform(image=img,\n                            bboxes=bboxes, \n                            class_labels=class_labels)\n    \n    return dict(image_id=image_id,\n                image=transformed['image'][:,:,0],\n                rad_id=rad_id,\n                bboxes=np.array(transformed['bboxes'], dtype=np.float32), \n                class_labels=transformed['class_labels'].astype(np.int8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno in range(8):\n    example_batch3 = parallel(get_and_save, [(idx, image_id) for idx, image_id in enumerate(list_of_images[0:64])], n_workers=4, progress=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see that this is way slower than either reading images or reading shelve files."},{"metadata":{},"cell_type":"markdown","source":"# How does that compare to reading images?"},{"metadata":{},"cell_type":"markdown","source":"This is not quite a perfect comparison, because [the Kaggle dataset I used](https://www.kaggle.com/awsaf49/vinbigdata-512-image-dataset) has 512 by 512 images (instead of 600 by aspect-ratio-preserving dimension) and I could not figure out how to get correct bounding boxes for these images. So, some of the augmentation work gets skipped. Hence, the way we do the comparison makes loading individual images look more favorable than it is."},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT_SHAPE = 224\n\naugs = albumentations.Compose([            \n            albumentations.RandomResizedCrop(INPUT_SHAPE, INPUT_SHAPE, scale=(0.9, 1.0)),            \n            albumentations.ShiftScaleRotate(rotate_limit=10, p=0.5),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1,0.1), \n                contrast_limit=(-0.1, 0.1), \n                p=0.5\n            ),\n            albumentations.ISONoise()] #,bbox_params=albumentations.BboxParams(format='pascal_voc')\n)\n\nclass ImageDataSet:\n    def __init__(self, image_ids, train, augmentations=None):\n        self.image_ids, self.train, self.augmentations = image_ids, train, augmentations\n    def __len__(self):\n        return len(self.image_ids)\n    def __getitem__(self, item):\n        image = np.array(Image.open('../input/vinbigdata-512-image-dataset/vinbigdata/train/' + self.image_ids[item] + '.png').convert('RGB'))\n        \n        rad_id = self.train.loc[train['image_id']==self.image_ids[item], 'rad_id'].values\n        class_labels = train.loc[train['image_id']==self.image_ids[item], 'class_id'].values    \n        bboxes = [list(row) for rowid, row in train.loc[train['image_id']==self.image_ids[item], ['x_min', 'y_min', 'x_max', 'y_max', 'class_id']].fillna({'x_min':0, 'y_min':0, 'x_max':1, 'y_max':1}).astype(np.int16).iterrows() ]\n                \n        which_rad_id = random.sample( list(np.unique(rad_id)), k=1)[0]        \n        which_indices = [idx for idx, val in enumerate(rad_id) if val==which_rad_id]\n        bboxes = np.array(bboxes)[which_indices]\n        class_labels = np.array(class_labels)[which_indices]\n        transformed = self.augmentations(image=image) #, bboxes=bboxes, class_labels=class_labels)\n        \n        #return {'image': np.transpose(transformed['image'], (2,0,1)), 'bboxes': transformed['bboxes'], 'class_labels': transformed['class_labels']}    \n        return np.transpose(transformed['image'], (2,0,1))\n\nimageds = ImageDataSet(image_ids=list_of_images, \n                       train=pd.read_csv('../input/vinbigdata-512-image-dataset/vinbigdata/train.csv'), \n                       augmentations=augs)\nexample_loader2 = torch.utils.data.DataLoader(imageds, batch_size=64, shuffle=True, num_workers=4, collate_fn=lambda x: x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(example_loader2, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_loader2a = torch.utils.data.DataLoader(imageds, batch_size=64, shuffle=True, num_workers=1, collate_fn=lambda x: x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(example_loader2a, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_loader2b = torch.utils.data.DataLoader(imageds, batch_size=64, shuffle=True, num_workers=0, collate_fn=lambda x: x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(example_loader2b, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_loader2c = torch.utils.data.DataLoader(imageds, batch_size=64, shuffle=True, num_workers=2, collate_fn=lambda x: x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nfor batchno, batch in enumerate(itertools.islice(example_loader2c, 8)):\n    len(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How can we use it with a model?"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}