{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Save space with sparse tensor\nSince I'm trying to train the network in my PC and I run out of space, I decided to implement a script that is able to read the .zip downloaded file withouth decompressing it, and extract the images with a sparse representation of the images.\nSince the images are mainly white (255), I decided to use their negatives (so white becomes black, 0) and to extract sparse tensors. Then I save them in a compressed way.\n\nThis script is not going to be run in this environment since data doesn't come as .zip files. Be free to use this implementation in your personal workstation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Steps for compression\nI'm going to open the .zip file and to extract the .tiff train images in the form of sparse tensors.\nIn this way I'm able to spare up to 1/10 of space - since most images are plenty of 255.\n\n* without unzipping the original dataset, I load the information and open only the training .tiff images, the second layer.\n* I calculate the opposite by subtracting 255\n* I calculate the indices, values and size of the original image\n* I cast every tensor to its fitter representation (no indexes above 10K for second layer tiff images)\n* (optional) I reconstruct the image and check if they are equal (checked for 100 images without errors)\n* I save the three tensors for each image with the original name, followed by `_values.pt.gz`, `_indices.pt.gz`, `_size.pt`\n(The .gz is for the gzip compressed files, since still too heavy)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Imports","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from tifffile import TiffFile\nfrom zipfile import ZipFile\nfrom io import BytesIO\nimport os\nimport torch\nfrom tqdm import tqdm\nimport gzip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Actual script","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"path_to_compressed_archive = 'D:\\\\datasets\\\\PANDA_dataset\\\\prostate-cancer-grade-assessment.zip'\npath_to_dest_dataset = '../dataset/train_images_torch'\nwhile False:  # So no annoying printings in the preview version of this notebook\n    with ZipFile(path_to_compressed_archive, 'r') as zipped_files:  # Open zipped file\n        for file in tqdm(zipped_files.infolist()):  # Obtain file list\n            zip_filepath = file.filename  # Extract actual file path\n            if 'train_images' in zip_filepath and zip_filepath.endswith('.tiff'):  # Leave masks\n                # Open tiff image and obtain second layer (in the 'asarray' parameter)\n                img = TiffFile(BytesIO(zipped_files.read(file))).asarray(1)\n                # Transform to torch tensor to do operations\n                t_img = torch.tensor(img, dtype=torch.uint8)\n                # Calculate its opposite to cast the images to sparse representation\n                t_img_negative = 255 - t_img\n                # Calculate sparse tensor\n                t_img_negative_sparse = t_img_negative.to_sparse()\n                # Cast to right data type to spare space\n                indices = t_img_negative_sparse.indices().type(torch.int16)\n                values = t_img_negative_sparse.values().type(torch.uint8)\n                size = t_img_negative_sparse.size()\n                # Create new h5 file path\n                new_filepath = os.path.join(path_to_dest_dataset, file.filename.split('/')[-1].split('.')[0])\n                # (Optional: check if images are reconstructed in the right way)\n                # reconstruct_img = torch.sparse_coo_tensor(indices, values, size, dtype=torch.uint8).to_dense()\n                # assert t_img_negative.equal(reconstruct_img)\n                # torch.save(torch.tensor(img, dtype=torch.uint8), new_filepath + '.pt')\n                # Assign new names for indices, values and size\n                indices_path = new_filepath + '_indices.pt.gz'\n                values_path = new_filepath + '_values.pt.gz'\n                size_path = new_filepath + '_size.pt'\n                # Save the tensors to disk\n                with gzip.GzipFile(indices_path, 'w', compresslevel=1) as f:\n                    torch.save(indices, f)\n                with gzip.GzipFile(values_path, 'w', compresslevel=1) as f:\n                    torch.save(values, f)\n                torch.save(size, size_path)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}