{"cells":[{"metadata":{},"cell_type":"markdown","source":"

VinBigData Chest X-ray Abnormalities Detection

\n

Automatically localize and classify thoracic abnormalities from chest radiographs

\n

Detailed Data Analysis

"},{"metadata":{},"cell_type":"markdown","source":"## Table of contents\n\n1. [Introduction](#Introduction)\n\n2. [The Dataset](#The-Dataset)\n - [DATASET INFORMATION](#DATASET-INFORMATION)\n - [DATA FILES](#DATA-FILES)\n\n3. [Getting DICOM Metadata](#Getting-DICOM-Metadata)\n4. [EDA](#EDA)\n5. [Analysis For Data Cleaning](#Analysis-For-Data-Cleaning)\n - [Redundant Samples](#Redundant-Samples)\n - [Consider Columns That Have Very Few Values](##Consider-Columns-That-Have-Very-Few-Values)\n\n\n## More Coming soon"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib\nimport seaborn as sns\nimport pydicom as dicom\nimport cv2\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/kaggle/input/vinbigdata-chest-xray-abnormalities-detection/'\ntrain_data = pd.read_csv(path+'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_example(idx_list):\n fig, axs = plt.subplots(1, 3, figsize=(15, 10))\n fig.subplots_adjust(hspace = .1, wspace=.1)\n axs = axs.ravel()\n for i in range(3):\n image_id = train_data.loc[idx_list[i], 'image_id']\n data_file = dicom.dcmread(path+'train/'+image_id+'.dicom')\n img = data_file.pixel_array\n axs[i].imshow(img, cmap='gray')\n axs[i].set_title(train_data.loc[idx_list[i], 'class_name'])\n axs[i].set_xticklabels([])\n axs[i].set_yticklabels([])\n if train_data.loc[idx_list[i], 'class_name'] != 'No finding':\n bbox = [train_data.loc[idx_list[i], 'x_min'],\n train_data.loc[idx_list[i], 'y_min'],\n train_data.loc[idx_list[i], 'x_max'],\n train_data.loc[idx_list[i], 'y_max']]\n p = matplotlib.patches.Rectangle((bbox[0], bbox[1]),\n bbox[2]-bbox[0],\n bbox[3]-bbox[1],\n ec='r', fc='none', lw=2.)\n axs[i].add_patch(p)\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n---\n\n## Introduction\n\n[[ go back to the top ]](#Table-of-contents)\n\nIn this kaggle competition, we are going to automatically localize and classify 14 types of thoracic abnormalities from chest radiographs.we will work with a dataset consisting of 18,000 scans that have been annotated by experienced radiologists. we can train your model with 15,000 independently-labeled images and will be evaluated on a test set of 3,000 images.\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n---\n\n\n## The Dataset\n\n[[ go back to the top ]](#Table-of-contents)\n\nIn this competition, we are classifying common thoracic lung diseases and localizing critical findings. This is an object detection and classification problem.\n\nFor each test image, you will be predicting a bounding box and class for all findings. If you predict that there are no findings, you should create a prediction of **`14 1 0 0 1 1`** *(14 is the class ID for no finding, and this provides a one-pixel bounding box with a confidence of 1.0)*\n\nNote that the images are in **DICOM** format, which means they contain additional data that might be useful for visualizing and classifying.\n\n\n\n![Example Radiographs](https://i.imgur.com/QWmbhXx.png)\n\n
\n\n#### DATASET INFORMATION\n\nThe dataset comprises **`18,000`** postero-anterior (PA) CXR scans in DICOM format, which were de-identified to protect patient privacy. \n\nAll images were labeled by a panel of experienced radiologists for the presence of **14** critical radiographic findings as listed below:\n\n- [0. Aortic enlargement](#0.-Aortic-enlargement)\n- [1. Atelectasis](#1.-Atelectasis)\n- [2. Calcification](#2.-Calcification)\n- [3. Cardiomegaly](#3.-Cardiomegaly)\n- [4. Consolidation](#4.-Consolidation)\n- [5. ILD](#5.-ILD)\n- [6. Infiltration](#6.-Infiltration)\n- [7. Lung Opacity](#7.-Lung-Opacity)\n- [8. Nodule/Mass](#8.-Nodule/Mass)\n- [9. Other lesion](#9.-Other-lesion)\n- [10. Pleural effusion](#10.-Pleural-effusion)\n- [11. Pleural thickening](#11.-Pleural-thickening)\n- [12. Pneumothorax](#12.-Pneumothorax)\n- [13. Plumonary fibrosis](#13.-Plumonary-fibrosis)\n- [14. No finding](#14.-No-finding)\n\nNote that a key part of this competition is working with ground truth from multiple radiologists. That means that the same image will have multiple ground-truth labels as annotated by different radiologists.\n\n
\n\n\n#### DATA FILES\n\n> **`train.csv`** - the train set metadata, with one row for each object, including a class and a bounding box (multiple rows per image possible)
\n**`sample_submission.csv`** - a sample submission file in the correct format\n\n
\n\nTRAIN COLUMNS\n> **`image_id`** - unique image identifier
\n**`class_name`** - the name of the class of detected object (or \"No finding\")
\n**`class_id`** - the ID of the class of detected object
\n**`rad_id`** - the ID of the radiologist that made the observation
\n**`x_min`** - minimum X coordinate of the object's bounding box
\n**`y_min`** - minimum Y coordinate of the object's bounding box
\n**`x_max`** - maximum X coordinate of the object's bounding box
\n**`y_max`** - maximum Y coordinate of the object's bounding box"},{"metadata":{},"cell_type":"markdown","source":"## 0. Aortic enlargement\n\n[[Back to top]](#The-Dataset)\n\nAlso known as an aortic aneurysm, this condition can be deadly if left undiagnosed.\n\n![Example Radiographs](https://www.umcvc.org/sites/default/files/styles/large/public/Types%20of%20Aneurysms.jpg?itok=2POtyibt)\n\nThe above Illustration shows three types of aneurysm and the two aneurysm shapes. The three types of aneurysm are Ascending Thoracic aortic Aneurysm (TAA), Descending Thoracic Aortic Aneurysm (TAA), and Abdominal Aortic Aneurysm (AAA). The Fusiform Aneurysm and Saccular Aneurysm show the two types of aneurysm shapes.\n\nThe aorta is your largest artery and it brings oxygenated blood to all parts of the body. If the walls of the aorta become weak, an enlargement can occur, which is known as an aortic aneurysm. Aneurysms can form in any section of the aorta, but are most common in the abdomen (abdominal aortic aneurysm) or the upper body (thoracic aortic aneurysm). \n \n(Please compare the below images to the normal images.)"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==0][9:12].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Atelectasis\n\n[[Back to top]](#The-Dataset)\n\n\n\"drawing\"\n\nYour airways are branching tubes that run throughout each of your lungs. When you breathe, air moves from the main airway in your throat, sometimes called your windpipe, to your lungs. The airways continue branching and get progressively smaller until they end in little sacs called alveoli.\n\nYour alveoli help to exchange the oxygen in the air for carbon dioxide, a waste product from your tissues and organs. In order to do this, your alveoli must fill with air.\n\nWhen some of your alveoli don’t fill with air, it’s called “atelectasis.”\n\nDepending on the underlying cause, atelectasis can involve either small or large portions of your lung."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==1][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Calcification\n\n[[Back to top]](#The-Dataset)\n\n\n\n\"Example\n\nMany diseases or conditions can cause calcification on chest x-ray. Calcium (calcification) may be deposited in areas where previous inflammation of the lungs or pleura has healed. Calcium may be deposited in the aorta due to atherosclerosis. Or calcification may occur in mediastinal lymph nodes.\n\nThe image is characterized by a density similar to that of bone."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==2][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Cardiomegaly\n\n[[Back to top]](#The-Dataset)\n\n\n\nCardiomegaly (sometimes megacardia or megalocardia) is a medical condition in which the heart is enlarged. As such, it is more commonly referred to simply as \"having an enlarged heart\".\n\nCardiomegaly is not a disease, but rather a condition that can result from a host of other diseases such as obesity or coronary artery disease. Cardiomegaly can be serious: depending on what part of the heart is enlarged, the patient can suffer from heart failure. Recent studies suggest that cardiomegaly is associated with a higher risk of sudden cardiac death"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==3][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Consolidation\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nA pulmonary consolidation is a region of normally compressible lung tissue that has filled with liquid instead of air.[1] The condition is marked by induration[2] (swelling or hardening of normally soft tissue) of a normally aerated lung.\n\nIn other words,Lung consolidation occurs when the air that usually fills the small airways in your lungs is replaced with something else. Depending on the cause, the air may be replaced with:\n\n - a fluid, such as pus, blood, or water\n - a solid, such as stomach contents or cells\n \nThe appearance of your lungs on a chest X-ray, and your symptoms, are similar for all these substances. So, you’ll typically need more tests to find out why your lungs are consolidated. With appropriate treatment, the consolidation usually goes away and air returns.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==4][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. ILD\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nILD stands for \"Interstitial Lung Disease.\"\n\nInterstitial lung disease (ILD) is the name for more than 200 lung disorders that affect the interstitium, the tissue and space around the alveoli (air sacs). The interstitium is the tiny, fluid-filled tissue and space around the air sacs in the lungs.\n\nIn healthy people, the interstitium is very thin. In interstitial lung diseases, the interstitium thickens and scars. Over time, the scarring can cause lung stiffness and eventually affect breathing.\n\nPeople with interstitial lung disease find it hard to get enough oxygen into their bloodstream. If you have an interstitial lung disease, other compartments of your lungs can also be affected, including:\n\n - Alveoli\n - Airways (trachea, bronchi, and bronchioles)\n - Blood vessels\n - Pleura (outside lining of the lung)."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==5][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Infiltration\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nA pulmonary infiltrate is a substance denser than air, such as pus, blood, or protein, which lingers within the parenchyma of the lungs. Pulmonary infiltrates are associated with pneumonia, and tuberculosis. Pulmonary infiltrates can be observed on a chest radiograph."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==6][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Lung Opacity\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nPulmonary opacification represents the result of a decrease in the ratio of gas to soft tissue (blood, lung parenchyma and stroma) in the lung.\n\nAs we can see the this classification is generic and to check that who is tagging this classification."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==7][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 8. Nodule/Mass\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nPulmonary nodules are small, rounded opacities within the pulmonary interstitium. Pulmonary nodules are common and, as the spatial resolution of CT scanners has increased, detection of smaller and smaller nodules has occurred, which are more often an incidental finding.\n\nIn other words, a nodule/mass is a round shade (typically less than 3 cm in diameter)."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==8][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9. Other lesion\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nOthers include all abnormalities that do not fall into any other category. This includes bone penetrating images, fractures, and subcutaneous emphysema, etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==9][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 10. Pleural effusion\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nA pleural effusion is excess fluid that accumulates in the pleural cavity, the fluid-filled space that surrounds the lungs. Excess fluid can impair breathing by limiting the expansion of the lungs. Various kinds of pleural effusion, depending on the nature of the fluid and what caused its entry into the pleural space, are hydrothorax (serous fluid), hemothorax (blood), urinothorax (urine), chylothorax (chyle), or pyothorax (pus) commonly known as pleural empyema. In contrast, a pneumothorax is the accumulation of air in the pleural space, and is commonly called a \"collapsed lung\".\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==10][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 11. Pleural thickening\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nPleural thickening refers to a thickening of the lining of the lungs, the pleura, which is a thin layer of membrane that covers the inside of the rib-cage as well as the outside of the lungs. Diffuse pleural thickening (DPT) is diagnosed when the pleura thickens to the extent that it causes breathlessness.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==11][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 12. Pneumothorax\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nA pneumothorax is a condition in which air leaks from the lungs and accumulates in the chest cavity. When air leaks and accumulates in the chest, it cannot expand outward like a balloon due to the ribs' presence. Instead, the lungs are pushed by the air and become smaller. In other words, a pneumothorax is a situation where air leaks from the lungs and the lungs become smaller (collapsed).\n\nIn a chest radiograph of a pneumothorax, the collapsed lung is whiter than normal, and the area where the lung is gone is uniformly black. Besides, the edges of the lung may appear linear.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==12][1:4].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 13. Plumonary fibrosis\n\n[[Back to top]](#The-Dataset)\n\n\n\n\n\nPulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly. As pulmonary fibrosis worsens, you become progressively more short of breath."},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==13][3:6].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 14. No finding\n\n[[Back to top]](#The-Dataset)\n\n\nThis is the normal image."},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"idx_list = train_data[train_data['class_id']==14][0:3].index.values\nplot_example(idx_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n----\n\n# Getting DICOM Metadata\n\n[[ go back to the top ]](#Table-of-contents)\n\n\nThe idea is to get DICOM metadata, combine it and do some analysis. First lets peak into metadata of trains DICOM."},{"metadata":{"trusted":true},"cell_type":"code","source":"filepath = \"../input/vinbigdata-chest-xray-abnormalities-detection/train/50a418190bc3fb1ef1633bf9678929b3.dicom\"\nds = dicom.filereader.dcmread(filepath)\nprint(ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filepath = \"../input/vinbigdata-chest-xray-abnormalities-detection/test/00a2145de1886cb9eb88869c85d74080.dicom\"\nds = dicom.filereader.dcmread(filepath)\nprint(ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#copies from https://www.kaggle.com/hiicao/dicom-metadata-to-pandasframe\n\n#Geting DICOM meta data and savinf in a csv file \ndef extract_DICOM_metadata(folder_path):\n text_file = open(\"dicom_metadata.txt\", \"w\")\n \n cnt = 0\n images = list(os.listdir(folder_path))\n df = pd.DataFrame()\n \n for image in images:\n cnt += 1\n \n image_name = image.split(\".\")[0]\n dicom_file_path = os.path.join(folder_path,image)\n dicom_file_dataset = dicom.read_file(dicom_file_path)\n text_file.write(str(dicom_file_dataset))\n\n \n ElementKeys = dicom_file_dataset.dir()\n \n PatientSex = dicom_file_dataset.PatientSex if 'PatientSex' in ElementKeys else \"\"\n PatientAge = dicom_file_dataset.PatientAge if 'PatientAge' in ElementKeys else \"\"\n PhotometricInterpretation = dicom_file_dataset.PhotometricInterpretation if 'PhotometricInterpretation' in ElementKeys else \"\"\n\n\n df = df.append(pd.DataFrame({'image_id': image_name, \n 'PatientSex': PatientSex, 'PatientAge': PatientAge, 'PhotometricInterpretation': PhotometricInterpretation }, index = [cnt]))\n df.to_csv(r'./dicom_metadata.csv', index=False) \n text_file.close()\n return df\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_DICOM_metadata('../input/vinbigdata-chest-xray-abnormalities-detection/train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#loading DICOM meta data \nmeta_data = pd.read_csv('./dicom_metadata.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/vinbigdata-chest-xray-abnormalities-detection/train.csv')\n\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#combining it with TRAIN data\nall_df = pd.merge(left=meta_data, right=train_df, on=\"image_id\", how=\"inner\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n---\n\n\n# EDA\n\n[[ go back to the top ]](#Table-of-contents)\n\nChecking missing values."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Data dimension: {all_df.shape}\")\nfor col in all_df.columns:\n print(f\"Column: {col:35} | type: {str(all_df[col].dtype):7} \\\n| missing values: {all_df[col].isna().sum():3d}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Peaking into Numerical features"},{"metadata":{"trusted":true},"cell_type":"code","source":"# define numerical features\nnumerical_features = [col for col in all_df.columns if np.issubdtype(all_df[col].dtype, np.number)]\nprint(numerical_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print statistics about the different numerical columns\nall_df[numerical_features].describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distributions of numerical features\nplt.figure(figsize=(10,18))\nfor index, col in enumerate(numerical_features):\n plt.subplot(5, 2, index+1)\n sns.distplot(all_df[col])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, peaking into Categorical Features."},{"metadata":{"trusted":true},"cell_type":"code","source":"# define categorical features\ncategorical_features = [col for col in all_df.columns if pd.api.types.is_string_dtype(all_df[col])]\ndel categorical_features[0]\ndel categorical_features[1]\nprint(categorical_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot distributions of numerical features\nplt.figure(figsize=(25,35))\nfor index, col in enumerate(categorical_features):\n plt.subplot(6, 2, index+1)\n ax = sns.countplot(y=col, data=all_df)\n ax.set_xlabel(\"count\", fontsize=20)\n ax.set_ylabel(col, fontsize=20)\n ax.tick_params(labelsize=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n---\n\n\n# Analysis For Data Cleaning"},{"metadata":{},"cell_type":"markdown","source":"## Redundant Samples\n\n[[Back to top]](#Table-of-contents)\n\nLet's go ahead and find out which duplicate rows in our dataset\n\nIts is possible that we would have two types of data duplication in rows (as there is very slim chances because we dont need any preprocessing KAGGLE datset)."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Complete duplicate rows\ncheck_df_duplication = all_df.groupby(all_df.columns.tolist(),as_index=False).size() \ncheck_df_duplication[check_df_duplication['size'] > 1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Consider Columns That Have Very Few Values\n\n[[Back to top]](#Table-of-contents)\n\nLet's go ahead and find out feature whave have few observation.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#replace NaN values with zeros \nall_df = all_df.replace(np.nan, 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['PatientSex'] = all_df['PatientSex'].replace(0, 'U')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['PatientAge'] = all_df['PatientAge'].replace(0, 'U')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df['PatientAge'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df = all_df.fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from numpy import unique\ndf_array = pd.DataFrame(all_df).to_numpy()\nfor i in range(df_array.shape[1]):\n num = len(unique(df_array[:, i]))\n percentage = float(num) / df_array.shape[0] * 100\n print('%d, %d, %.1f%%' % (i, num, percentage))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Further ANALYSIS COMING sOON"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}