{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ProteiNet v2 🧬 [Training Notebook]","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://structuralbioinformatician.files.wordpress.com/2013/03/1ece.gif\">","metadata":{}},{"cell_type":"markdown","source":"# 1. Go beyond with Aspects Experts Predictions!","metadata":{}},{"cell_type":"markdown","source":"In the previous episode, we took a look at the competition context and developed a first model for this 5th edition of the CAFA competition named **proteiNet** : https://www.kaggle.com/code/henriupton/proteinet-pytorch-ems2-t5-protbert-embeddings\n\nFollowing this, we implemented various additions to build its big brother: proteiNet v2! This new version actually aims to train not one, not two, but 3 models, all three specialized in predicting a group of GOs of a particular aspect among the three sets presented for CAFA5: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC).\n\nThis notebook is the first section of proteiNet v2 and it is dedicated to the training part of the models. If you want to have a look into the inference section, follow this link : https://www.kaggle.com/code/henriupton/proteinet-aspects-experts-infer\n\nGitHub version of proteiNet v2 is also available : https://github.com/henriupton99/proteinet-cafa5\n\nFeel free to give feedback for improvement, and drop an upvote to support our investment to the project. ☀️☀️\n\nLet's take a look at what's new in this model!","metadata":{}},{"cell_type":"markdown","source":"# 2. New features of proteiNet v2","metadata":{}},{"cell_type":"markdown","source":"Thanks to the great interest shown in the notebook dedicated to ProteiNet, a large number of bugs and defects have been corrected in this new version. On the other hand, my team (M. Sato, F. Lin and myself) have tried to innovate as much as possible and incorporate various topics of discussion from the competition for ProteiNet v2. Here is an exhaustive list of the most important innovations:\n\n- **Rather than training a single model to predict the scores of all GOs for each protein, train three separate models capable of predicting the scores of each protein for GOs of a specific aspect (BPO, MFO, CCO)**. This is why we call these three models \"experts\". This practice has several theoretical virtues, such as the fact that each model aims to perform multilabel classification on a smaller number of classes. In addition, they are trained to work on GO embeddings that are highly likely to be parent/child, as they come from the same aspect. Once the models have been trained, the predictions of each model will be concatenated to form the final submission.\n\n- **The GO classes to be predicted are no longer naively the top K of the most frequent GO classes in the database.** It has been discussed time and again that various other methods can be used to select GO classes more strategically. In addition, as we train experts based on aspect groups, we define as classes for each model the top K most frequent GOs filtered on aspects. The number of classes per aspect is defined on the basis of these observations: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491 \n\n- **We restrict the GO classes to be predicted to those whose evidence code is Inferred from Experiment (EXP) (and its subgroups)**.(https://wiki.geneontology.org/index.php/Inferred_from_Experiment_(EXP)) This choice stems from the desire to be as close as possible to the explanations given in the Background section of the Competition Evaluation page: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation\n\n- **We incorporate the weights given by IA.txt into our Cross Entropy loss function when training the model.** Again in an effort to keep up with the competition, the weights enable us to place greater emphasis on infrequent GOs (at the root of the graph), which are consequently the most important ones. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237)(https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)\n\n- **We have implemented a cross-validation (CV) process to prevent from overfitting.** There has long been a consensus that incorporating this method into the pipeline of one's work reduces the risk of overfitting on the public LB and by consequence get a good result on the private LB. The actual CV is composed of 5 folds (so 5 different classic 80-20 train-test splits). (https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-use-k-fold-cross-validation-with-pytorch.md) Once the models perform well for the 5 folds, we train the final models on the full train set thanks to the config hyperparameter VALIDATION_MODE = False.\n\n- **Once the predictions were formed for the three expert models, we defined a minimal score threshold in order to filter the predictions.** For each prediction row, if it does not exceed the threshold, it is deleted from the final submission. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431652) This hyperparameter can be tuned thanks to the constant PROB_THRESHOLD in the config class. It allows to benefit at maximum from the propagation process of GOs predictions. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)","metadata":{}},{"cell_type":"markdown","source":"# 3. Configuration of the working environment","metadata":{}},{"cell_type":"code","source":"import torch\nclass CONFIG:\n    \n    # CONSTANTS FOR DATA PATHS\n    MAIN_DIR = \"/kaggle/input/cafa-5-protein-function-prediction/\"\n    GO_OBO_FILE = MAIN_DIR + \"Train/go-basic.obo\"\n    TRAIN_SEQUENCES_FASTA = MAIN_DIR  + \"Train/train_sequences.fasta\"\n    TRAIN_LABELS = MAIN_DIR + \"Train/train_terms.tsv\"\n    TRAIN_IDS = \"/kaggle/input/protbert-embeddings-for-cafa5/train_ids.npy\"\n    IA_WEIGHTS = MAIN_DIR + \"IA.txt\"\n    TEST_SEQUENCES_FASTA = MAIN_DIR + \"/Test (Targets)/testsuperset.fasta\"\n    TARGETS_PATH = \"/kaggle/working/train-labels-targets/\"\n    EVIDENCE_CODES = \"/kaggle/input/enhanced-train-terms/propagated_evidenceCode.parquet\"\n    \n    # CONSTANTS FOR ASPECTS :\n    ASPECTS = [\"BPO\", \"CCO\", \"MFO\"]\n    ASPECTS_LABELS = {\"BPO\" : 1100, \"CCO\" : 300, \"MFO\" : 450}\n\n    # CONSTANTS FOR TRAINING : \n    EMBEDDINGS_SOURCE = \"ESM2\"\n    K_FOLDS = 5\n    N_EPOCHS = 10\n    BATCHS_SIZE = 256\n    LEARNING_RATE = 0.001\n    DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n    ASPECTS_HIDDEN_SIZE = {\"BPO\" : 1256, \"CCO\" : 512, \"MFO\" : 850}\n    VALIDATION_MODE = True\n    \n    # CONSTANTS FOR POSTPROCESSING :\n    PROB_THRESHOLD = 0.10","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Preprocess the data : build the targets","metadata":{}},{"cell_type":"code","source":"\"\"\"FUNCTIONS FOR GENERATING THE LABELS TAGRETS FOR ALL ASPECTS\n\"\"\"\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\nimport re\nimport gc\n\ndef extract_go_terms_and_branches(\n    file_path : str\n    ) -> dict:\n    \"\"\"Utilitary function to construct a mapping {GO TERM : ASPECT} for each GO TERM in input OBO file\n\n    Args:\n        file_path (str): file path for input GO terms\n\n    Returns:\n        go_terms_dict (dict): mapping dictionnary\n    \"\"\"\n    with open(file_path, 'r') as file:\n        content = file.read()\n        stanzas = re.findall(r'\\[Term\\][\\s\\S]*?(?=\\n\\[|$)', content)\n    go_terms_dict = {}\n    for stanza in stanzas:\n        go_id = re.search(r'^id: (GO:\\d+)', stanza, re.MULTILINE)\n        if go_id:\n            go_id = go_id.group(1)\n        namespace = re.search(r'^namespace: (\\w+)', stanza, re.MULTILINE)\n        if namespace:\n            namespace = namespace.group(1)\n        if go_id and namespace:\n            branch_abbr = {'biological_process': 'BPO', 'cellular_component': 'CCO', 'molecular_function': 'MFO'}\n            go_terms_dict[go_id] = branch_abbr[namespace]\n\n    return go_terms_dict\n\ndef generate_labels_matrix(\n    ids : np.ndarray,\n    labels_names : list[str],\n    id_labels : dict,\n    go_terms_map : dict\n    ):\n    \"\"\"Utilitary function to generate labels target matrix given :\n    - protein ids, labels_names (GO terms names)\n    - id labels : id of labels\n    - go terms map generated by function extract_go_terms_and_branches\n\n    \"\"\"\n    labels_matrix = np.zeros((len(ids), len(labels_names)))\n    \n    for index, id in tqdm(enumerate(ids)):\n        try :\n            id_gos_list = id_labels[id]\n            temp = [go_terms_map[go] for go in labels_names if go in id_gos_list]\n            labels_matrix[index, temp] = 1\n        except:\n            pass\n        \n    return labels_matrix\n\ndef generate_targets(\n    ids_path : str,\n    labels_path : str,\n    weights_path : str,\n    go_obo_path : str,\n    evidence_codes_path : str,\n    targets_path : str,\n    aspects_list : str,\n    go_terms_per_aspects : dict[int]\n    ):\n    \"\"\"Function to generate labels (targets) for a given aspect (BPO, CCO, or MFO) for each protein id in ids, based on labels dataframe.\n    NB : For memory usage and models precision reasons, we only consider a subset of all GO terms labels\n    We consider to top K most frequent for each aspect (based on go_terms_per_aspects input dictionnary)\n\n    Args:\n        ids_path (str): path to protein ids \n        labels_paths (str): path to labels annotations dataframe for each protein in ids\n        weights_path (str): path to IA weights for GO terms metric computation\n        go_obo_path (str) : path to obo graph file\n        evidence_codes_path (str) : path to evidence codes to filter for EXP GO terms\n        targets_path (str) : path where to save the target labels\n        aspects_list (str): list of aspects to consider\n        go_terms_per_aspects (dict[int]): number of GO term classes to consider per aspect\n    \"\"\"\n    ids = np.load(ids_path)\n    labels = pd.read_csv(labels_path, sep = \"\\t\")\n    colnames = [\"term\", \"weight\"]\n    ia_weights = pd.read_csv(weights_path, sep = \"\\t\", names = colnames, header=None)\n    evidence_codes = pd.read_parquet(evidence_codes_path)\n    evidence_codes = evidence_codes[evidence_codes[\"EvidenceCode\"].notnull()][\"term\"].unique().tolist()\n    \n    for aspect in aspects_list:\n        print(\"=\"*25)\n        print(\"START LOADING FOR ASPECT {}\".format(aspect))\n        aspects_labels = labels[labels[\"aspect\"] == aspect]\n        aspects_labels = aspects_labels[aspects_labels[\"term\"].isin(evidence_codes)]\n        top_terms = aspects_labels.groupby(\"term\")[\"EntryID\"].count().sort_values(ascending=False).to_frame()\n        map_go_terms_aspects = extract_go_terms_and_branches(\n                                file_path=go_obo_path\n                                )\n        top_terms[\"aspect\"] = top_terms.index.map(map_go_terms_aspects)\n        labels_names = top_terms[:go_terms_per_aspects[aspect]]\n        labels_names = labels_names.index.values\n        weights_df = pd.DataFrame(data={\"term\" : labels_names})\n        weights_df = weights_df.merge(ia_weights, on = \"term\", how = \"left\")\n        print(\"NUMBER OF GO TERMS IN {} SUBSET : {}\".format(aspect,str(len(labels_names))))\n        train_labels_sub = labels[(labels.term.isin(labels_names)) & (labels.EntryID.isin(ids))]\n        id_labels = train_labels_sub.groupby('EntryID')['term'].apply(list).to_dict()\n        go_terms_map = {label: i for i, label in enumerate(labels_names)}\n        labels_matrix = generate_labels_matrix(\n            ids=ids,\n            labels_names=labels_names,\n            id_labels=id_labels,\n            go_terms_map=go_terms_map\n        )\n        labels_list = []\n        for l in range(labels_matrix.shape[0]):\n            labels_list.append(labels_matrix[l, :])\n\n        labels_df = pd.DataFrame(data={\"EntryID\":ids, \"labels_vect\":labels_list})\n        labels_df.to_pickle(\"/kaggle/working/train_targets_{}.pkl\".format(aspect))\n        weights_df.to_csv(\"/kaggle/working/weights_{}.csv\".format(aspect),index=None)\n        del aspects_labels, labels_df, weights_df, labels_list, labels_matrix, go_terms_map\n        gc.collect()\n        print(\"GENERATION FINISHED FOR ASPECT {}\".format(aspect))\n    \n    del ids, labels, ia_weights, evidence_codes\n    gc.collect()\n    print(\"GENERATION FINISHED ! :D\")\n    return ","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:03:45.347777Z","iopub.execute_input":"2023-08-21T18:03:45.348196Z","iopub.status.idle":"2023-08-21T18:03:45.373214Z","shell.execute_reply.started":"2023-08-21T18:03:45.348165Z","shell.execute_reply":"2023-08-21T18:03:45.372122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generate_targets(\n    ids_path = CONFIG.TRAIN_IDS,\n    labels_path = CONFIG.TRAIN_LABELS,\n    weights_path = CONFIG.IA_WEIGHTS,\n    go_obo_path = CONFIG.GO_OBO_FILE,\n    evidence_codes_path = CONFIG.EVIDENCE_CODES,\n    targets_path = CONFIG.TARGETS_PATH,\n    aspects_list = CONFIG.ASPECTS,\n    go_terms_per_aspects = CONFIG.ASPECTS_LABELS\n)","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:03:52.50209Z","iopub.execute_input":"2023-08-21T18:03:52.502473Z","iopub.status.idle":"2023-08-21T18:06:41.605836Z","shell.execute_reply.started":"2023-08-21T18:03:52.502443Z","shell.execute_reply":"2023-08-21T18:06:41.604786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Build the Pytorch *Dataset* instance","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom torch.utils.data import Dataset\n\nclass ProteinSequenceDataset(Dataset):\n    \n    embeds_map = {\n    \"T5\" : \"/kaggle/input/t5embeds\",\n    \"ProtBERT\" : \"/kaggle/input/protbert-embeddings-for-cafa5/\",\n    \"ESM2\" : \"/kaggle/input/4637427/\"\n    }\n    \n    def __init__(self, aspect, datatype, embeddings_source):\n        super(ProteinSequenceDataset).__init__()\n        self.datatype = datatype\n        \n        if embeddings_source == \"ProtBERT\":\n            embeds = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_embeddings.npy\")\n            ids = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_ids.npy\")\n        if embeddings_source == \"T5\":\n            embeds = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_embeds.npy\")\n            ids = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_ids.npy\")\n        if embeddings_source == \"ESM2\":\n            embeds = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_embeds_esm2_t36_3B_UR50D.npy\")\n            ids = np.load(ProteinSequenceDataset.embeds_map[embeddings_source]+datatype+\"_ids_esm2_t36_3B_UR50D.npy\")\n                 \n        embeds_list = []\n        for l in range(embeds.shape[0]):\n            embeds_list.append(embeds[l,:])\n        self.df = pd.DataFrame(data={\"EntryID\": ids, \"embed\" : embeds_list})\n        \n        if datatype==\"train\":         \n            df_labels = pd.read_pickle(\n                \"/kaggle/working/train_targets_{}.pkl\".format(aspect)\n            )  \n            self.df = self.df.merge(df_labels, how=\"right\", on=\"EntryID\")\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index):\n        embed = torch.tensor(self.df.iloc[index][\"embed\"] , dtype = torch.float32)\n        if self.datatype==\"train\":\n            targets = torch.tensor(self.df.iloc[index][\"labels_vect\"], dtype = torch.float32)\n            return embed, targets\n        if self.datatype==\"test\":\n            id = self.df.iloc[index][\"EntryID\"]\n            return embed, id","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:06:46.080577Z","iopub.execute_input":"2023-08-21T18:06:46.081275Z","iopub.status.idle":"2023-08-21T18:06:46.095758Z","shell.execute_reply.started":"2023-08-21T18:06:46.081239Z","shell.execute_reply":"2023-08-21T18:06:46.093847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Define the Pytorch *Model* instance","metadata":{}},{"cell_type":"code","source":"from torch import nn\n\nembeds_dim = {\n    \"T5\" : 1024,\n    \"ProtBERT\" : 1024,\n    \"EMS2\" : 2560\n}\n\nclass LinearModel(torch.nn.Module):\n\n    def __init__(self, input_dim, hidden_dim, num_classes):\n        super(LinearModel, self).__init__()\n        self.linear1 = torch.nn.Linear(input_dim, hidden_dim)\n        self.activation1 = torch.nn.ReLU()\n        self.linear2 = torch.nn.Linear(hidden_dim, num_classes)\n\n    def forward(self, x):\n        x = self.linear1(x)\n        x = self.activation1(x)\n        x = self.linear2(x)\n        return x\n    \n\nclass CNN1D(nn.Module):\n    def __init__(self, input_dim, num_classes):\n        super(CNN1D, self).__init__()\n        self.conv1 = nn.Conv1d(in_channels=1, out_channels=3, kernel_size=3, dilation=1, padding=1, stride=1)\n        self.pool1 = nn.MaxPool1d(kernel_size=2, stride=2)\n        self.conv2 = nn.Conv1d(in_channels=3, out_channels=8, kernel_size=3, dilation=1, padding=1, stride=1)\n        self.pool2 = nn.MaxPool1d(kernel_size=2, stride=2)\n        self.fc1 = nn.Linear(in_features=int(8 * input_dim/4), out_features=128)\n        self.fc2 = nn.Linear(in_features=128, out_features=num_classes)\n\n    def forward(self, x):\n        x = x.reshape(x.shape[0], 1, x.shape[1])\n        x = self.pool1(nn.functional.relu(self.conv1(x)))\n        x = self.pool2(nn.functional.relu(self.conv2(x)))\n        x = torch.flatten(x, 1)\n        x = nn.functional.relu(self.fc1(x))\n        x = self.fc2(x)\n        return x\n","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:06:47.114235Z","iopub.execute_input":"2023-08-21T18:06:47.11463Z","iopub.status.idle":"2023-08-21T18:06:47.12837Z","shell.execute_reply.started":"2023-08-21T18:06:47.114598Z","shell.execute_reply":"2023-08-21T18:06:47.126889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Train the models & Cross Validate","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nimport os\n\nfrom sklearn.model_selection import KFold\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torchmetrics.classification import MultilabelF1Score\nfrom torchmetrics.classification import MultilabelAccuracy\n\nembeds_dim = {\n    \"T5\" : 1024,\n    \"ProtBERT\" : 1024,\n    \"ESM2\" : 2560\n}\n\ndef train_model(\n    aspect : str,\n    embeddings_source : str,\n    num_classes : int,\n    hidden_dim : int,\n    k_folds : int,\n    batch_size : int,\n    n_epochs : int,\n    learning_rate : float,\n    device : str,\n    validation_mode : bool\n    ):\n    \"\"\"Function to train an expert model (BPO, CCO, or MFO) based on a given input aspect name.\n    NB : You need to first train with validation_mode=True in order to validate your model on several folds\n    Then run again your model with validation_mode=False to train on all data for the final inference model\n\n    Args:\n        aspect (str): name of the aspect to train a model\n        embeddings_source (str): source of the embeddings to consider (\"T5\",\"ProtBERT\",\"EMS2\")\n        num_classes (int): number of classes for the model\n        hidden_dim (int) : dimension of the hidden layer for the model\n        k_folds (int) : number of folds in the Kfold Cross Validation routine\n        batch_size (int) : batch size of the dataloader \n        n_epochs (int): number of epochs for training\n        learning_rate (float): learning rate for optimizer\n        device (str): device for training (cuda for GPU, or cpu for CPU)\n        validation_mode (bool): option for validation of final training of the model\n    \"\"\"\n    \n    train_dataset = ProteinSequenceDataset(aspect = aspect, datatype=\"train\", embeddings_source = embeddings_source)  \n    models_path = \"/kaggle/working/models/\"    \n    aspects_path = \"/kaggle/working/models/{}/\".format(aspect)\n    history_path = \"/kaggle/working/history/\"\n    for path in [models_path, aspects_path, history_path]:\n        if not os.path.exists(path):\n            os.mkdir(path)\n        \n    losses_history = pd.DataFrame({\"epoch\" : [e for e in range(1,n_epochs+1)]})\n    scores_history = pd.DataFrame({\"epoch\" : [e for e in range(1,n_epochs+1)]})\n    \n    if validation_mode == True:\n        kfold = KFold(n_splits=k_folds, shuffle=True) \n        for fold, (train_ids, val_ids) in enumerate(kfold.split(train_dataset)):\n\n            print(\"=\"*25)\n            print(\"FOLD {}\".format(fold+1))\n            train_subsampler = torch.utils.data.SubsetRandomSampler(train_ids)\n            val_subsampler = torch.utils.data.SubsetRandomSampler(val_ids)\n\n            train_dataloader = torch.utils.data.DataLoader(\n                train_dataset,\n                batch_size=batch_size,\n                sampler=train_subsampler\n                )\n            val_dataloader = torch.utils.data.DataLoader(\n                train_dataset,\n                batch_size=batch_size,\n                sampler=val_subsampler\n                )\n            gc.collect()\n\n            model = LinearModel(input_dim=embeds_dim[embeddings_source],hidden_dim=hidden_dim, num_classes=num_classes).to(device)\n\n            optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)\n            #scheduler = ReduceLROnPlateau(optimizer, factor=0.1, patience=1, threshold=0.001, verbose=True)\n\n            df_weights = pd.read_csv(\n                \"/kaggle/working/weights_{}.csv\".format(aspect)\n            )\n            classes_weights = torch.tensor(df_weights.weight.values, dtype=torch.float64)\n\n            CrossEntropy = torch.nn.CrossEntropyLoss(weight=classes_weights).to(device)\n            f1_score = MultilabelF1Score(num_labels=num_classes).to(device)\n\n            print(\"BEGIN TRAINING...\")\n            train_loss_history=[]\n            val_loss_history=[]\n\n            train_f1score_history=[]\n            val_f1score_history=[]\n\n            for epoch in range(n_epochs):\n                print(\"EPOCH \", epoch+1)\n                ## TRAIN PHASE :\n                losses = []\n                scores = []\n                model.train()\n                for embed, targets in tqdm(train_dataloader):\n                    embed, targets = embed.to(device), targets.to(device)\n                    optimizer.zero_grad()\n                    preds = model(embed)\n                    loss= CrossEntropy(preds, targets)\n                    score=f1_score(preds, targets)\n                    losses.append(loss.item()) \n                    scores.append(score.item())\n                    loss.backward()\n                    optimizer.step()\n                avg_loss = np.mean(losses)\n                avg_score = np.mean(scores)\n                print(\"Running Average TRAIN Loss : \", avg_loss)\n                print(\"Running Average TRAIN F1-Score : \", avg_score)\n                train_loss_history.append(avg_loss)\n                train_f1score_history.append(avg_score)\n\n                ## VALIDATION PHASE : \n                model.eval()\n                losses = []\n                scores = []\n                for embed, targets in val_dataloader:\n                    embed, targets = embed.to(device), targets.to(device)\n                    preds = model(embed)\n                    loss= CrossEntropy(preds, targets)\n                    score=f1_score(preds, targets)\n                    losses.append(loss.item())\n                    scores.append(score.item())\n                avg_loss = np.mean(losses)\n                avg_score = np.mean(scores)\n                print(\"Running Average VAL Loss : \", avg_loss)\n                print(\"Running Average VAL F1-Score : \", avg_score)\n                val_loss_history.append(avg_loss)\n                val_f1score_history.append(avg_score)\n\n                #scheduler.step(avg_loss)\n                print(\"\\n\")\n\n            print(\"TRAINING FINISHED\")\n            print(\"FINAL TRAINING SCORE FOLD {}: \".format(fold+1), train_f1score_history[-1])\n            print(\"FINAL VALIDATION SCORE : {}\".format(fold+1), val_f1score_history[-1])\n\n            losses_history[\"train_fold_{}\".format(fold+1)] = train_loss_history\n            losses_history[\"val_fold_{}\".format(fold+1)] = val_loss_history\n\n            scores_history[\"train_fold_{}\".format(fold+1)] = train_f1score_history\n            scores_history[\"val_fold_{}\".format(fold+1)] = val_f1score_history\n\n            torch.save(model.state_dict(), \"/kaggle/working/models/{}/expert_model_fold_{}.pt\".format(aspect,fold+1))\n            print(\"MODEL SAVED AT /kaggle/working/models/{}/expert_model_fold_{}.pt\".format(aspect,fold+1))\n\n            del model, train_dataloader, val_dataloader\n            gc.collect()\n            print(\"=\"*25)\n\n        losses_history.to_csv(history_path + \"losses_history_{}.csv\".format(aspect),index=None)\n        scores_history.to_csv(history_path + \"scores_history_{}.csv\".format(aspect),index=None)\n        del losses_history, scores_history\n        del train_dataset\n        gc.collect()\n    \n    if validation_mode == False:\n        train_dataloader = torch.utils.data.DataLoader(\n                train_dataset,\n                batch_size=batch_size,\n                shuffle=True\n                )\n        gc.collect()\n\n        model = LinearModel(input_dim=embeds_dim[embeddings_source],hidden_dim=hidden_dim, num_classes=num_classes).to(device)\n\n        optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)\n        #scheduler = ReduceLROnPlateau(optimizer, factor=0.1, patience=1, threshold=0.001, verbose=True)\n\n        df_weights = pd.read_csv(\n            \"/kaggle/working/weights_{}.csv\".format(aspect)\n        )\n        classes_weights = torch.tensor(df_weights.weight.values, dtype=torch.float64)\n\n        CrossEntropy = torch.nn.CrossEntropyLoss(weight=classes_weights).to(device)\n        f1_score = MultilabelF1Score(num_labels=num_classes).to(device)\n\n        print(\"BEGIN TRAINING...\")\n        train_loss_history=[]\n\n        train_f1score_history=[]\n\n        for epoch in range(n_epochs):\n            print(\"EPOCH \", epoch+1)\n            ## TRAIN PHASE :\n            losses = []\n            scores = []\n            model.train()\n            for embed, targets in tqdm(train_dataloader):\n                embed, targets = embed.to(device), targets.to(device)\n                optimizer.zero_grad()\n                preds = model(embed)\n                loss= CrossEntropy(preds, targets)\n                score = f1_score(preds, targets)\n                losses.append(loss.item()) \n                scores.append(score.item())\n                loss.backward()\n                optimizer.step()\n            avg_loss = np.mean(losses)\n            avg_score = np.mean(scores)\n            print(\"Running Average TRAIN Loss : \", avg_loss)\n            print(\"Running Average TRAIN F1-Score : \", avg_score)\n            train_loss_history.append(avg_loss)\n            train_f1score_history.append(avg_score)\n        print(\"TRAINING FINISHED\")\n        print(\"FINAL TRAINING SCORE : \", train_f1score_history[-1])\n\n        losses_history[\"train\"] = train_loss_history\n        scores_history[\"train\"] = train_f1score_history\n\n        torch.save(model.state_dict(), \"/kaggle/working/models/{}/expert_model.pt\".format(aspect))\n        print(\"MODEL SAVED AT /kaggle/working/models/{}/expert_model.pt\".format(aspect))\n\n        del model, train_dataloader\n        gc.collect()\n        print(\"=\"*25)\n\n        losses_history.to_csv(history_path + \"losses_history_{}.csv\".format(aspect),index=None)\n        scores_history.to_csv(history_path + \"scores_history_{}.csv\".format(aspect),index=None)\n        del losses_history, scores_history\n        del train_dataset\n        gc.collect()\n    \n    print(\"TRAINING FINISHED ! :D\")\n    return","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:06:48.010679Z","iopub.execute_input":"2023-08-21T18:06:48.011047Z","iopub.status.idle":"2023-08-21T18:06:48.050173Z","shell.execute_reply.started":"2023-08-21T18:06:48.011017Z","shell.execute_reply":"2023-08-21T18:06:48.049191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from time import time\n\nfor aspect in CONFIG.ASPECTS:\n    print(\"=\"*25)\n    print(\"START TRAINING PHASE FOR ASPECT : \" , aspect)\n    start = time()\n    train_model(\n        aspect=aspect,\n        embeddings_source=CONFIG.EMBEDDINGS_SOURCE,\n        num_classes=CONFIG.ASPECTS_LABELS[aspect],\n        hidden_dim=CONFIG.ASPECTS_HIDDEN_SIZE[aspect],\n        k_folds=CONFIG.K_FOLDS,\n        batch_size=CONFIG.BATCHS_SIZE,\n        n_epochs=CONFIG.N_EPOCHS,\n        learning_rate=CONFIG.LEARNING_RATE,\n        device=CONFIG.DEVICE,\n        validation_mode=CONFIG.VALIDATION_MODE\n    )\n    end = time()\n    print(\"TRAINING FINISHED (time elapsed : \", round(end-start,2), \" seconds)\")","metadata":{"execution":{"iopub.status.busy":"2023-08-21T18:08:24.632649Z","iopub.execute_input":"2023-08-21T18:08:24.633016Z","iopub.status.idle":"2023-08-21T18:13:29.109037Z","shell.execute_reply.started":"2023-08-21T18:08:24.632985Z","shell.execute_reply":"2023-08-21T18:13:29.105272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. Visualisations","metadata":{}},{"cell_type":"code","source":"if CONFIG.VALIDATION_MODE == True:\n    import matplotlib.pyplot as plt\n    plt.style.use(\"ggplot\")\n\n    fig, axes = plt.subplots(len(CONFIG.ASPECTS),2, figsize=(10,15))\n\n    for i, aspect in enumerate(CONFIG.ASPECTS):\n        df = pd.read_csv(\"/kaggle/working/history/scores_history_{}.csv\".format(aspect))\n        for j, datatype in enumerate([\"train\",\"val\"]):\n            for fold in range(1,CONFIG.K_FOLDS+1):\n                axes[i,j].plot(df[\"epoch\"], df[\"{}_fold_{}\".format(datatype,fold)],label=\"fold {}\".format(fold))\n                axes[i,j].legend()\n                axes[i,j].set_ylabel(\"F1 Score\")\n                axes[i,j].set_xlabel(\"Epoch\")\n            axes[i,0].set_title(\"Aspect {}\".format(aspect))\n\n    plt.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-19T21:59:39.551697Z","iopub.execute_input":"2023-08-19T21:59:39.552051Z","iopub.status.idle":"2023-08-19T21:59:39.563645Z","shell.execute_reply.started":"2023-08-19T21:59:39.552022Z","shell.execute_reply":"2023-08-19T21:59:39.562614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CONFIG.VALIDATION_MODE == True:\n    fig, axes = plt.subplots(len(CONFIG.ASPECTS),2, figsize=(10,15))\n    for i, aspect in enumerate(CONFIG.ASPECTS):\n        df = pd.read_csv(\"/kaggle/working/history/losses_history_{}.csv\".format(aspect))\n        for j, datatype in enumerate([\"train\",\"val\"]):\n            for fold in range(1,CONFIG.K_FOLDS+1):\n                axes[i,j].plot(df[\"epoch\"], df[\"{}_fold_{}\".format(datatype,fold)],label=\"fold {}\".format(fold))\n                axes[i,j].legend()\n                axes[i,j].set_ylabel(\"Loss\")\n                axes[i,j].set_xlabel(\"Epoch\")\n            axes[i,0].set_title(\"Aspect {}\".format(aspect))\n\n    plt.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-19T21:59:42.161588Z","iopub.execute_input":"2023-08-19T21:59:42.162117Z","iopub.status.idle":"2023-08-19T21:59:42.174144Z","shell.execute_reply.started":"2023-08-19T21:59:42.162077Z","shell.execute_reply":"2023-08-19T21:59:42.17322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thanks for reading :D dont forget to upvote and give feedback !","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://media.giphy.com/media/10LKovKon8DENq/giphy.gif\">","metadata":{}}]}