{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"!apt-get update\n!apt-get install libturbojpeg\n!pip install -qU git+git://github.com/lilohuang/PyTurboJPEG.git\n    \n!pip install -q guppy3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Just run through all the data using open. Looks like a memory leak.\n\nNote this is not a tqdm issue. You can check by removing it.\n\nYou can also try any of these with `cv2` instead of `TurboJPEG` and the effect should be the same."},{"metadata":{"trusted":true,"id":"06ajfnboQbrI","outputId":"f950800d-2d63-45ea-d482-0fd9f402ced3"},"cell_type":"code","source":"import gc\n\nimport pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n    # choose whether to try with cv2 or TurboJPEG()\n#     img = cv2.imread(path)\n    with open(path, 'rb') as f:\n        img = JPEG.decode(f.read())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm no expert at memory profiling but what's even more confusing is that I can't see where the memory went. The `Total size` is still around 1.5 GB."},{"metadata":{"trusted":true},"cell_type":"code","source":"h = hpy()\nprint(h.heap())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### BUT, try the same again but this time loading the same file each loop. No leak."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n    # choose whether to try with cv2 or TurboJPEG()\n#     img = cv2.imread(path)\n    with open('../input/pe-train-512x512-fold-1-batch-5/000233653a7b.jpg', 'rb') as f:\n        img = JPEG.decode(f.read())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Going back to loading different files... Okay so what if I try some variations of `del` and `gc.collect()`\n\nPutting in a `del img` doesn't help. And I can't tell if I add a `gc.collect()` on top of that because the loop takes 20x longer. So I settle for `gc.collect()` every 1000 iterations. Seems the problem is still there."},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\n\nimport pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor i, path in enumerate(tqdm(train_df.image_path)):\n    # choose whether to try with cv2 or TurboJPEG()\n#     img = cv2.imread(path)\n    with open(path, 'rb') as f:\n        img = JPEG.decode(f.read())\n    del img\n    if i%1000 == 0:\n        gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Time to get creative? Nope, there's still a memory leak"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nimg = np.zeros(shape=(512, 512, 3)).astype(np.uint8)\n\nfor path in tqdm(train_df.image_path):\n    img *= 0\n    # choose whether to try with cv2 or TurboJPEG()\n#     img += cv2.imread(path)\n    with open(path, 'rb') as f:\n        img += JPEG.decode(f.read())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}