{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<img src=\"https://i.imgur.com/de3OrxO.png\">\n\n<center><h1> Detect Sleep States </h1></center>\n<center><h1>- bringing down the memory usage -</h1></center>\n\n> 📌 **Competition Scope**: detect the occurrence of *onset* (the beginning of sleep) and *wakeup* (the end of sleep) in the accelerometer series.\n\n### Dataframes are very large\n\nThe first thing I tried to do in this competition is look at the data, but the `train_series.parquet` file is HUGE (contains hundreds of millions rows).\n\nI downloaded the data locally and worked to bring down the memory usage as effectively as possible. This notebook contains the code I used, the process and results.\n\n* `train_events` -> **73% reduction**\n* `train_series` -> **86% reduction**\n\n### ○ Libraries","metadata":{}},{"cell_type":"code","source":"# libraries\nimport os\nimport re\nimport gc\nimport wandb\nimport random\nimport math\nfrom glob import glob\nfrom tqdm import tqdm\nfrom time import time\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport warnings\nimport pandas as pd\nimport numpy as np\nimport cudf\n\n# env check\nwarnings.filterwarnings('ignore')\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': '2023_sleep', '_wandb_kernel': 'aot', \"source_type\": \"artifact\"}\n\n# color\nclass clr:\n    S = '\\033[1m' + '\\033[90m'\n    E = '\\033[0m'\n    \nmy_colors = [\"#f79256\", \"#fbd1a2\", \"#7dcfb6\", \"#00b2ca\"]\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:03:20.189436Z","iopub.execute_input":"2023-09-10T13:03:20.190002Z","iopub.status.idle":"2023-09-10T13:03:26.251106Z","shell.execute_reply.started":"2023-09-10T13:03:20.18996Z","shell.execute_reply":"2023-09-10T13:03:26.249887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🐝 W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n🐝**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n<center><img src=\"https://i.imgur.com/fFccmoS.png\" width=500></center>","metadata":{}},{"cell_type":"code","source":"# 🐝 secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"wandb\")\n\n! wandb login $secret_value_0","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:03:26.257248Z","iopub.execute_input":"2023-09-10T13:03:26.257723Z","iopub.status.idle":"2023-09-10T13:03:30.761136Z","shell.execute_reply.started":"2023-09-10T13:03:26.25768Z","shell.execute_reply":"2023-09-10T13:03:30.759897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ○ Helper Functions Below","metadata":{}},{"cell_type":"code","source":"# === data discover ===\ndef get_general_info(df, desc=None):\n    \n    # 🐝 new exp\n    run = wandb.init(project='2023_sleep', name=f'{desc}_data_summary', config=CONFIG)\n\n    print(clr.S+\"--- General Info ---\"+clr.E)\n    print(clr.S+\"Data Shape:\"+clr.E, df.shape)\n    print(clr.S+\"Data Cols:\"+clr.E, df.columns.tolist())\n    print(clr.S+\"Total No. of Cols:\"+clr.E, len(df.columns.tolist()))\n    print(clr.S+\"No. Missing Values:\"+clr.E, df.isna().sum().sum())\n    print(clr.S+\"Columns with missing data:\"+clr.E, \"\\n\",\n          df.isna().sum()[df.isna().sum() != 0], \"\\n\")\n\n    print(clr.S+\"--- [object] columns ---\"+clr.E)\n    str_cols = df.select_dtypes(include=[object]).columns\n    for col in str_cols:\n        print(clr.S+f\"[nunique] {col}:\"+clr.E, \n              df[col].nunique())\n\n    print(\"\\n\")\n\n    print(clr.S+\"--- [numerical] columns ---\"+clr.E)\n    digit_cols = df.select_dtypes(include=[int, float]).columns\n    for col in digit_cols:\n        print(clr.S+f\"[describe] {col}:\"+clr.E, \"\\n\",\n              df[col].describe())\n        \n    # log data\n    wandb.log\n    (\n        {\"data_shape\": len(df),\n         \"missing_values\": df.isna().sum().sum(),\n         \"unique_id\": df.series_id.nunique(),\n        }\n    )\n    wandb.finish()\n    print(\"🐝 Info saved to dashboard.\")\n            \n\ndef get_missing_values_plot(df):\n    '''\n    Plots missing values barchart for a given dataframe.\n    '''\n    \n    # count missing values\n    missing_counts = df.isnull().sum().reset_index()\\\n                            .sort_values(0, ascending=False)\\\n                            .reset_index(drop=True)\n    missing_counts.columns = [\"col_name\", \"missing_count\"]\n\n    # plot\n    plt.figure(figsize=(24, 16))\n    axs = sns.barplot(y=missing_counts.col_name, x=missing_counts.missing_count, \n                      color=my_colors[0])\n    show_values_on_bars(axs, h_v=\"h\", space=0.4)\n    plt.xlabel('no. missing values', size=20, weight=\"bold\")\n    plt.ylabel('column name', size=20, weight=\"bold\")\n    plt.title('Missing Values', size=22, weight=\"bold\")\n    plt.show();\n            \n            \n# === plots ===\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n    '''Plots the value at the end of the a seaborn barplot.\n    axs: the ax of the plot\n    h_v: weather or not the barplot is vertical/ horizontal'''\n    \n    def _show_on_single_plot(ax):\n        if h_v == \"v\":\n            for p in ax.patches:\n                _x = p.get_x() + p.get_width() / 2\n                _y = p.get_y() + p.get_height()\n                value = int(p.get_height())\n                ax.text(_x, _y, format(value, ','), ha=\"center\") \n        elif h_v == \"h\":\n            for p in ax.patches:\n                _x = p.get_x() + p.get_width() + float(space)\n                _y = p.get_y() + p.get_height()\n                value = int(p.get_width())\n                ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n    if isinstance(axs, np.ndarray):\n        for idx, ax in np.ndenumerate(axs):\n            _show_on_single_plot(ax)\n    else:\n        _show_on_single_plot(axs)\n        \n        \n# === 🐝 w&b ===\ndef save_dataset_artifact(run_name, artifact_name, path, data_type=\"dataset\"):\n    '''Saves dataset to W&B Artifactory.\n    run_name: name of the experiment\n    artifact_name: under what name should the dataset be stored\n    path: path to the dataset'''\n    \n    run = wandb.init(project='2023_sleep', \n                     name=run_name, \n                     config=CONFIG)\n    artifact = wandb.Artifact(name=artifact_name, \n                              type=data_type)\n    artifact.add_file(path)\n\n    wandb.log_artifact(artifact)\n    wandb.finish()\n    print(f\"🐝Artifact {artifact_name} has been saved successfully.\")\n    \n    \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n    '''Create and save lineplot/barplot in W&B Environment.\n    x_data & y_data: Pandas Series containing x & y data\n    x_name & y_name: strings containing axis names\n    title: title of the graph\n    log: string containing name of log'''\n    \n    data = [[label, val] for (label, val) in zip(x_data, y_data)]\n    table = wandb.Table(data=data, columns = [x_name, y_name])\n    \n    if plot == \"line\":\n        wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n    elif plot == \"bar\":\n        wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n    elif plot == \"scatter\":\n        wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n        \n        \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n    '''Create and save histogram in W&B Environment.\n    x_data: Pandas Series containing x values\n    x_name: strings containing axis name\n    title: title of the graph\n    log: string containing name of log'''\n    \n    data = [[x] for x in x_data]\n    table = wandb.Table(data=data, columns=[x_name])\n    wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-10T13:03:30.763889Z","iopub.execute_input":"2023-09-10T13:03:30.764336Z","iopub.status.idle":"2023-09-10T13:03:30.794046Z","shell.execute_reply.started":"2023-09-10T13:03:30.76427Z","shell.execute_reply":"2023-09-10T13:03:30.792577Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# events.csv","metadata":{}},{"cell_type":"code","source":"# read data\nevents = cudf.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\nevents.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:03:30.797194Z","iopub.execute_input":"2023-09-10T13:03:30.797526Z","iopub.status.idle":"2023-09-10T13:03:33.280959Z","shell.execute_reply.started":"2023-09-10T13:03:30.797502Z","shell.execute_reply":"2023-09-10T13:03:33.280057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_general_info(events, desc=\"events\")","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:03:33.282199Z","iopub.execute_input":"2023-09-10T13:03:33.282536Z","iopub.status.idle":"2023-09-10T13:04:41.474904Z","shell.execute_reply.started":"2023-09-10T13:03:33.282503Z","shell.execute_reply":"2023-09-10T13:04:41.473928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Initial memory usage","metadata":{}},{"cell_type":"code","source":"# initial memory usage\nevents.memory_usage(deep=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.476389Z","iopub.execute_input":"2023-09-10T13:04:41.476743Z","iopub.status.idle":"2023-09-10T13:04:41.493103Z","shell.execute_reply.started":"2023-09-10T13:04:41.476711Z","shell.execute_reply":"2023-09-10T13:04:41.492099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# total in MB\nevents.memory_usage(deep=True).sum() / (1024 * 1024)","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.4945Z","iopub.execute_input":"2023-09-10T13:04:41.495045Z","iopub.status.idle":"2023-09-10T13:04:41.504754Z","shell.execute_reply.started":"2023-09-10T13:04:41.495Z","shell.execute_reply":"2023-09-10T13:04:41.503813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# these are the initial dtypes\n# for the variables within the dataframe\nevents.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.505943Z","iopub.execute_input":"2023-09-10T13:04:41.506721Z","iopub.status.idle":"2023-09-10T13:04:41.518223Z","shell.execute_reply.started":"2023-09-10T13:04:41.506688Z","shell.execute_reply":"2023-09-10T13:04:41.517344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Decrease memory usage\n\n* `series_id`: from object -> uint16\n    * there are only 277 unique ids\n    * hence I remapped them with an id_map from 0 to 277\n    * easier during development, can easily switch back to original id\n* `night`: from int64 -> uint16\n* `event`: from object -> uint8\n    * relabeled as follows: 'onset':'1', 'wakeup':'2'\n* `step`: from int64 -> uint32\n* `timestamp`: from object -> datetime64","metadata":{}},{"cell_type":"code","source":"new_events = events.copy()\n\n# 277 ids (same id's as in series)\n# new_events.series_id = new_events.series_id.str.filter_characters({'0':'9'}).astype(np.int64)\n# or map it :)\ntrain_id_map = cudf.DataFrame({\"series_id\": new_events.series_id.unique(),\n                               \"id_map\": new_events.series_id.unique().index})\ntrain_id_map.id_map = train_id_map.id_map.astype(np.uint16)\ntrain_id_map.to_parquet(\"./train_id_map.parquet\", index=False)\nnew_events = new_events.merge(right=train_id_map, on=\"series_id\").drop(columns=\"series_id\")\n\n# night\nnew_events.night = new_events.night.astype(np.uint16)\n# event relabeled\nnew_events.event = new_events.event.replace({'onset':'1', 'wakeup':'2'}).astype(np.uint8)\n# step\nnew_events.step = new_events.step.astype(np.uint32)\n# timestamp\nnew_events.timestamp = cudf.to_datetime(new_events.timestamp, format='%Y-%m-%d %H:%M:%S')","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.519471Z","iopub.execute_input":"2023-09-10T13:04:41.520667Z","iopub.status.idle":"2023-09-10T13:04:41.641062Z","shell.execute_reply.started":"2023-09-10T13:04:41.520643Z","shell.execute_reply":"2023-09-10T13:04:41.640259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Updated memory","metadata":{}},{"cell_type":"code","source":"# updated memory usage\nnew_events.memory_usage(deep=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.644956Z","iopub.execute_input":"2023-09-10T13:04:41.645683Z","iopub.status.idle":"2023-09-10T13:04:41.657316Z","shell.execute_reply.started":"2023-09-10T13:04:41.645658Z","shell.execute_reply":"2023-09-10T13:04:41.656043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# total in MB\nnew_events.memory_usage(deep=True).sum() / (1024 * 1024)","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.658778Z","iopub.execute_input":"2023-09-10T13:04:41.659206Z","iopub.status.idle":"2023-09-10T13:04:41.668767Z","shell.execute_reply.started":"2023-09-10T13:04:41.659175Z","shell.execute_reply":"2023-09-10T13:04:41.667824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" new_events.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.67025Z","iopub.execute_input":"2023-09-10T13:04:41.670795Z","iopub.status.idle":"2023-09-10T13:04:41.680454Z","shell.execute_reply.started":"2023-09-10T13:04:41.670762Z","shell.execute_reply":"2023-09-10T13:04:41.679299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparison before & after:\n\n<center><img src=\"https://i.imgur.com/7qVZHBe.jpg\" width=800></center>","metadata":{}},{"cell_type":"markdown","source":"# series.parquet\n\n> 📌 **Note**: I cannot showcase the changes I did for `train_series.parquet`, as I get a lot of OOM error within the Kaggle environment. Plus the conversion of `timestamp` from dtype `object` to dtype `datetime` took **7 hours to run** :).\n\n## 1. Initial memory usage\n\n<center><img src=\"https://i.imgur.com/XRwBiCL.jpg\" width=600></center>","metadata":{}},{"cell_type":"markdown","source":"## 2. Decrease memory usage\n\nThe code I used is almost the same as for events.csv, but used `pandas` for the `progress_apply()` function:\n```\nseries = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\")\n```\n\nUsed `id_map` mapped previously from the events dataset:\n```\ntrain_id_map = pd.read_parquet(\"/kaggle/input/detect-sleep-states-memory-decrease/train_id_map.parquet\")\nnew_series = series.merge(right=train_id_map, on=\"series_id\")\\\n            .drop(columns=\"series_id\")\\\n            .reset_index(drop=True)\n```\n\nStep convert:\n```\nnew_series.step = new_series.step.astype(np.uint32)\n```\n\nTimestamp convert (thank you [@carlmcbrideellis](https://www.kaggle.com/carlmcbrideellis) for help!):\n```\nfrom pandarallel import pandarallel\npandarallel.initialize(progress_bar=True)\n\n# Local Time converter\ndef to_date_time(x):\n    import pandas as pd\n    return pd.to_datetime(x, format='%Y-%m-%d %H:%M:%S') # utc=True\n\ndef to_localize(t):\n    import pandas as pd\n    return t.tz_localize(None)\n\nnew_series[\"timestamp\"] = new_series.timestamp.parallel_apply(to_date_time).parallel_apply(to_localize)\n```","metadata":{}},{"cell_type":"markdown","source":"## 3. Updated memory\n\n<center><img src=\"https://i.imgur.com/DBN7WZY.png\" width=500></center>\n\n### before & after comparison\n\n<center><img src=\"https://i.imgur.com/IhRv8EO.jpg\" width=800></center>","metadata":{}},{"cell_type":"code","source":"# 🐝 train_id_map.parquet\nsave_dataset_artifact(run_name=\"train_id_map\",\n                      artifact_name=\"train_id_map\",\n                      path=\"/kaggle/input/detect-sleep-states-memory-decrease/train_id_map.parquet\", \n                      data_type=\"dataset\")","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:04:41.681849Z","iopub.execute_input":"2023-09-10T13:04:41.68283Z","iopub.status.idle":"2023-09-10T13:05:44.50174Z","shell.execute_reply.started":"2023-09-10T13:04:41.682799Z","shell.execute_reply":"2023-09-10T13:05:44.500711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 🐝 train_events.parquet\nsave_dataset_artifact(run_name=\"train_events_memory\",\n                      artifact_name=\"train_events\",\n                      path=\"/kaggle/input/detect-sleep-states-memory-decrease/train_events.parquet\", \n                      data_type=\"dataset\")","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:05:44.502856Z","iopub.execute_input":"2023-09-10T13:05:44.503634Z","iopub.status.idle":"2023-09-10T13:06:49.254751Z","shell.execute_reply.started":"2023-09-10T13:05:44.503598Z","shell.execute_reply":"2023-09-10T13:06:49.253752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 🐝 train_series.parquet\n# this saves very fast, considering we deal with ~140mil rows\nsave_dataset_artifact(run_name=\"train_series_memory\",\n                      artifact_name=\"train_series\",\n                      path=\"/kaggle/input/detect-sleep-states-memory-decrease/train_series.parquet\", \n                      data_type=\"dataset\")","metadata":{"execution":{"iopub.status.busy":"2023-09-10T13:06:49.256241Z","iopub.execute_input":"2023-09-10T13:06:49.256586Z","iopub.status.idle":"2023-09-10T13:09:01.148836Z","shell.execute_reply.started":"2023-09-10T13:06:49.256554Z","shell.execute_reply":"2023-09-10T13:09:01.147766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# new datasets links:\n\nYou can find the new datasets:\n* **kaggle datasets**: https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease\n* 🐝 **W&B artifacts**: https://wandb.ai/andrada/2023_sleep/artifacts/code/source-2023_sleep-None/v1\n    * easier for storage & versioning :)\n    \n### a moment for pandarallel library\n\nThe `timestamp` column is a pain in the behind. Using `pandas.apply()` to convert it from object to datetime dtype took ~7hrs on my workstation.\n\nBut using `pandarallel.parallel_apply()` sped it up from 7hrs to just 1 hr (I have 14 CPU cores tho). :) But what a change in performance! Gave me a bunch of time to try to debug why in the end the variable still saves as `object` instead of `datetime`.\n\n<center><img src=\"https://i.imgur.com/3au3CjS.png\" width=800></center>","metadata":{}},{"cell_type":"markdown","source":"### 🐝 [my W&B dash](https://wandb.ai/andrada/2023_sleep?workspace=user-andrada)\n    \n<center><img src=\"https://i.imgur.com/DoYO51s.jpg\"></center>\n\n------\n\n<center><img src=\"https://i.imgur.com/knxTRkO.png\"></center>\n\n### My Specs\n\n* 🖥 Z8 G4 Workstation\n* 💾 2 CPUs & 96GB Memory\n* 🎮 2x NVIDIA A6000\n* 💻 Zbook Studio G9 on the go","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}