{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Objective : To predict protein function (GO term)\n\n### **Data**\n\n- Train Data :\n    1. `train_sequences.fasta` : UniProt, Sequence, Taxonomy\n    2. `taxonomy.tsv` : UniProt, Taxonomy ID\n    3. `terms.tsv` : UniProt, GO term ID\n    \n    \n- Test Data :\n    1. `superset.fasta` : UniProt,  Sequence, Taxonomy ID\n    2. `taxon_list.tsv` : Taxonomy ID, Taxonomy\n    \n    \n- Embedding vector :\n    1. `ProtT5`\n    2. `ProtBert`\n    3. `EMS-2`\n\n\n### **Target**\n- X : UniProt, Sequence, Taxonomy\n- Y : GO term ID","metadata":{}},{"cell_type":"markdown","source":"## Baseline\n\n### Vectorization\n- X, Y values = object type, **vectorization to float type for modeling**\n- However, since the nature of Sequence is not meaningful in itself, but a molecule composed of a combination of individuals, vectorization is required accordingly.\n\n- To this end, sequence, UniProt, and GOTerm are vectorized to **T5 Embedded vector** using the natural language processing model **T5 tokenizer**\n\nReference : https://www.kaggle.com/datasets/sergeifironov/t5embeds\n\n### Convert Fasta file to DataFrame & Matching Species\n- In addition to the data processed above, specifications suitable for each protein are matched to utilize specifications.\n\n### Dummy Encoding\n- Because the species is also an object type, conversion is necessary. To do this, we use Dummy encoding. (It can save memory compared to one-hot encoding)\n\n### Modeling\n- I made two models, concatted them, and then added a new output layer. One is a data set containing embedded vectors, and the other is a data set with Dummy encoding of Species.\n\n### Submission\n- By applying this to the test data set, we intend to achieve the desired goal.\n\nI referred to the notebook in the following link\n\nhttps://www.kaggle.com/code/simonveitner/simple-mlp","metadata":{}},{"cell_type":"markdown","source":"### Import Library","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import Ridge\nfrom sklearn.metrics import roc_auc_score\n\nimport numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm\ntqdm.pandas()\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense\n\n# measure roc auc score metric \nfrom tensorflow.keras.metrics import AUC","metadata":{"execution":{"iopub.status.busy":"2023-06-01T04:59:51.950958Z","iopub.execute_input":"2023-06-01T04:59:51.951403Z","iopub.status.idle":"2023-06-01T05:00:04.513729Z","shell.execute_reply.started":"2023-06-01T04:59:51.95137Z","shell.execute_reply":"2023-06-01T05:00:04.512486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assign Label\n\nThere are a total of 31,466 GOTerms, and working while processing all of them not only increases the computation volume, but also may not help the model. Therefore, it will be selected in the order of high frequency in about 500, 1000, and proceed.\n\nAfter that, the GOTerm ID is 'annoted' and then 'labeled'.","metadata":{}},{"cell_type":"code","source":"DATA_DIR = '/kaggle/input/cafa-5-protein-function-prediction'\nMAX_LABELS = 500","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:04.51595Z","iopub.execute_input":"2023-06-01T05:00:04.517092Z","iopub.status.idle":"2023-06-01T05:00:04.522327Z","shell.execute_reply.started":"2023-06-01T05:00:04.517049Z","shell.execute_reply":"2023-06-01T05:00:04.521389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_terms = pd.read_csv(os.path.join(DATA_DIR, 'Train', 'train_terms.tsv'), sep='\\t')","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:04.524067Z","iopub.execute_input":"2023-06-01T05:00:04.524835Z","iopub.status.idle":"2023-06-01T05:00:08.828384Z","shell.execute_reply.started":"2023-06-01T05:00:04.5248Z","shell.execute_reply":"2023-06-01T05:00:08.82717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"terms = train_terms.groupby(['aspect', 'term'])['term'].count().reset_index(name='frequency')\nfractions = (terms.groupby('aspect')['term'].nunique() / terms['term'].nunique() * MAX_LABELS).apply(round)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:08.831421Z","iopub.execute_input":"2023-06-01T05:00:08.831792Z","iopub.status.idle":"2023-06-01T05:00:11.004405Z","shell.execute_reply.started":"2023-06-01T05:00:08.83176Z","shell.execute_reply":"2023-06-01T05:00:11.002997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selected_terms = set()\nfor aspect, number in fractions.items():\n    selection = terms.loc[(terms.aspect == aspect)]\n    selection = selection.nlargest(number, columns = 'frequency', keep = 'first')\n    selected_terms.update(selection.term.to_list())\n\nprint(len(selected_terms))\nprint()\n# print(selected_terms)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:11.006138Z","iopub.execute_input":"2023-06-01T05:00:11.006492Z","iopub.status.idle":"2023-06-01T05:00:11.051472Z","shell.execute_reply.started":"2023-06-01T05:00:11.006461Z","shell.execute_reply":"2023-06-01T05:00:11.05063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selected_terms_freq = terms[terms['term'].isin(selected_terms)]\nselected_terms_freq = selected_terms_freq.sort_values(by='frequency', ascending=False)\nprint(selected_terms_freq)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:11.052635Z","iopub.execute_input":"2023-06-01T05:00:11.053431Z","iopub.status.idle":"2023-06-01T05:00:11.079981Z","shell.execute_reply.started":"2023-06-01T05:00:11.053399Z","shell.execute_reply":"2023-06-01T05:00:11.079065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def assign_labels(annotations, selected_terms = selected_terms):\n    intersection = selected_terms.intersection(annotations)\n    labels = np.isin(np.array(list(selected_terms)), np.array(list(intersection)))\n    return list(labels.astype('int'))","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:11.081198Z","iopub.execute_input":"2023-06-01T05:00:11.081898Z","iopub.status.idle":"2023-06-01T05:00:11.091817Z","shell.execute_reply.started":"2023-06-01T05:00:11.081865Z","shell.execute_reply":"2023-06-01T05:00:11.09048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"annotations = train_terms.groupby('EntryID')['term'].apply(set)\nlabels = annotations.progress_apply(assign_labels)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:00:11.093402Z","iopub.execute_input":"2023-06-01T05:00:11.093739Z","iopub.status.idle":"2023-06-01T05:01:26.634917Z","shell.execute_reply.started":"2023-06-01T05:00:11.093713Z","shell.execute_reply":"2023-06-01T05:01:26.633719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df = pd.DataFrame(labels)\nlabels_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:26.636812Z","iopub.execute_input":"2023-06-01T05:01:26.637277Z","iopub.status.idle":"2023-06-01T05:01:26.672685Z","shell.execute_reply.started":"2023-06-01T05:01:26.637236Z","shell.execute_reply":"2023-06-01T05:01:26.671464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Vectorization","metadata":{}},{"cell_type":"markdown","source":"### Load Train data\n\nTake T5 embeds and designate the actual x_train, and match the GO term that exists only in the previously assigned GO term label and designate it as y_train.","metadata":{}},{"cell_type":"code","source":"train_ids = np.load('/kaggle/input/t5embeds/train_ids.npy')\n\nx_train = np.load('/kaggle/input/t5embeds/train_embeds.npy')\ny_train = np.array(labels[train_ids].to_list())","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:26.677947Z","iopub.execute_input":"2023-06-01T05:01:26.678337Z","iopub.status.idle":"2023-06-01T05:01:44.882736Z","shell.execute_reply.started":"2023-06-01T05:01:26.678307Z","shell.execute_reply":"2023-06-01T05:01:44.881535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train_df = pd.DataFrame(x_train)\nx_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:44.884533Z","iopub.execute_input":"2023-06-01T05:01:44.885221Z","iopub.status.idle":"2023-06-01T05:01:44.914292Z","shell.execute_reply.started":"2023-06-01T05:01:44.885188Z","shell.execute_reply":"2023-06-01T05:01:44.913323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convert Fasta file to DataFrame (Train) & Matching Species","metadata":{}},{"cell_type":"code","source":"from Bio import SeqIO\nimport re\n\nfasta_file = '/kaggle/input/cafa-5-protein-function-prediction/Train/train_sequences.fasta'\n\n# list set\nids = []\ndescriptions = []\nspecies = []\ngenes = []\nlengths = []\nsequences = []\n\n# Read fasta file using SeqIO.parse() method\nrecords = list(SeqIO.parse(fasta_file, 'fasta'))\n\n# Defining Regular Expression Patterns\npattern = re.compile(r'(?:OS=)([^\\s]+)')\n\n# Extracting values using regular expression patterns\nfor record in records:\n    # id\n    id_ = record.id\n    \n    # description\n    description = record.description\n    \n    # species\n    species_match = pattern.search(description)\n    if species_match:\n        species_ = species_match.group(1)\n    else:\n        species_ = ''\n    \n    # gene\n    gene_match = re.search(r'(?:GN=)([^\\s]+)', description)\n    if gene_match:\n        gene_ = gene_match.group(1)\n    else:\n        gene_ = ''\n    \n    # length\n    length_ = len(record.seq)\n    \n    # sequence\n    sequence = str(record.seq)\n    \n    # Add extracted values to the list\n    ids.append(id_)\n    descriptions.append(description)\n    species.append(species_)\n    genes.append(gene_)\n    lengths.append(length_)\n    sequences.append(sequence)\n\n# Convert to Data Frame\nseq_df = pd.DataFrame({\n    'EntryID': ids,\n    'description': descriptions,\n    'species': species,\n    'gene': genes,\n    'length': lengths,\n    'sequence': sequences\n})\n\nseq_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:44.915947Z","iopub.execute_input":"2023-06-01T05:01:44.916473Z","iopub.status.idle":"2023-06-01T05:01:53.034236Z","shell.execute_reply.started":"2023-06-01T05:01:44.916441Z","shell.execute_reply":"2023-06-01T05:01:53.03288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train_df['species'] = seq_df['species']\nx_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:53.036275Z","iopub.execute_input":"2023-06-01T05:01:53.036728Z","iopub.status.idle":"2023-06-01T05:01:53.071102Z","shell.execute_reply.started":"2023-06-01T05:01:53.036685Z","shell.execute_reply":"2023-06-01T05:01:53.069862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train_df.rename(columns={'species':'1024'}, inplace=True)\nx_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:53.07291Z","iopub.execute_input":"2023-06-01T05:01:53.073367Z","iopub.status.idle":"2023-06-01T05:01:53.104759Z","shell.execute_reply.started":"2023-06-01T05:01:53.073328Z","shell.execute_reply":"2023-06-01T05:01:53.102972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Test data","metadata":{}},{"cell_type":"code","source":"test_ids = np.load('/kaggle/input/t5embeds/test_ids.npy')\nx_test = np.load('/kaggle/input/t5embeds/test_embeds.npy')","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:01:53.106595Z","iopub.execute_input":"2023-06-01T05:01:53.106967Z","iopub.status.idle":"2023-06-01T05:02:04.851394Z","shell.execute_reply.started":"2023-06-01T05:01:53.106935Z","shell.execute_reply":"2023-06-01T05:02:04.850236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test_df = pd.DataFrame(x_test)\nx_test_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:04.852967Z","iopub.execute_input":"2023-06-01T05:02:04.853435Z","iopub.status.idle":"2023-06-01T05:02:04.881762Z","shell.execute_reply.started":"2023-06-01T05:02:04.853379Z","shell.execute_reply":"2023-06-01T05:02:04.880846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convert Fasta file to DataFrame (test) & Matching Species","metadata":{}},{"cell_type":"code","source":"test_taxon = pd.read_csv('/kaggle/input/cafa-5-protein-function-prediction/Test (Targets)/testsuperset-taxon-list.tsv', sep='\\t', encoding='ISO-8859-1')","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:04.882819Z","iopub.execute_input":"2023-06-01T05:02:04.88329Z","iopub.status.idle":"2023-06-01T05:02:04.91296Z","shell.execute_reply.started":"2023-06-01T05:02:04.883259Z","shell.execute_reply":"2023-06-01T05:02:04.91168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_taxon.rename(columns={'ID': 'taxonomyID'}, inplace=True)\ntest_taxon.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:04.914925Z","iopub.execute_input":"2023-06-01T05:02:04.915768Z","iopub.status.idle":"2023-06-01T05:02:04.928411Z","shell.execute_reply.started":"2023-06-01T05:02:04.915722Z","shell.execute_reply":"2023-06-01T05:02:04.926939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fasta_file = '/kaggle/input/cafa-5-protein-function-prediction/Test (Targets)/testsuperset.fasta'\n\ndata = {'EntryID': [], 'taxonomyID': [], 'sequence': []}\ncurrent_entry = None\n\nwith open(fasta_file, 'r') as f:\n    for line in f:\n        if line.startswith('>'):\n            entry_id, taxonomy_id = line.strip().lstrip('>').split('\\t')\n            current_entry = entry_id\n            data['EntryID'].append(entry_id)\n            data['taxonomyID'].append(taxonomy_id)\n            data['sequence'].append('')\n        else:\n            data['sequence'][-1] += line.strip()\n\ntest_seq_df = pd.DataFrame(data)\ntest_seq_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:04.930992Z","iopub.execute_input":"2023-06-01T05:02:04.931698Z","iopub.status.idle":"2023-06-01T05:02:06.772101Z","shell.execute_reply.started":"2023-06-01T05:02:04.931643Z","shell.execute_reply":"2023-06-01T05:02:06.771013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_seq_df['taxonomyID'] = test_seq_df['taxonomyID'].astype(int)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:06.773963Z","iopub.execute_input":"2023-06-01T05:02:06.77441Z","iopub.status.idle":"2023-06-01T05:02:06.806934Z","shell.execute_reply.started":"2023-06-01T05:02:06.774371Z","shell.execute_reply":"2023-06-01T05:02:06.805889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.merge(test_taxon, test_seq_df, on='taxonomyID')\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:06.808617Z","iopub.execute_input":"2023-06-01T05:02:06.809077Z","iopub.status.idle":"2023-06-01T05:02:06.88617Z","shell.execute_reply.started":"2023-06-01T05:02:06.809046Z","shell.execute_reply":"2023-06-01T05:02:06.885084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_entries = pd.DataFrame({'EntryID': test_ids})\ndf_match = test_df[['EntryID', 'Species']].drop_duplicates()\nnew_df = pd.merge(df_entries, df_match, on='EntryID', how='left')\nnew_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:06.887805Z","iopub.execute_input":"2023-06-01T05:02:06.888263Z","iopub.status.idle":"2023-06-01T05:02:07.188006Z","shell.execute_reply.started":"2023-06-01T05:02:06.888225Z","shell.execute_reply":"2023-06-01T05:02:07.186891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df['Species'] = new_df['Species'].str.split(' ', expand=True)[0]\nnew_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:07.191076Z","iopub.execute_input":"2023-06-01T05:02:07.191472Z","iopub.status.idle":"2023-06-01T05:02:09.047748Z","shell.execute_reply.started":"2023-06-01T05:02:07.191441Z","shell.execute_reply":"2023-06-01T05:02:09.046533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df['Species'] = new_df['Species'].str.capitalize()\nnew_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:09.049565Z","iopub.execute_input":"2023-06-01T05:02:09.049904Z","iopub.status.idle":"2023-06-01T05:02:09.133386Z","shell.execute_reply.started":"2023-06-01T05:02:09.049876Z","shell.execute_reply":"2023-06-01T05:02:09.132115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test_df = pd.concat([x_test_df, new_df['Species']], axis=1)\nx_test_df.rename(columns={'Species':'1024'}, inplace=True)\nx_test_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:09.135303Z","iopub.execute_input":"2023-06-01T05:02:09.136052Z","iopub.status.idle":"2023-06-01T05:02:11.181473Z","shell.execute_reply.started":"2023-06-01T05:02:09.135993Z","shell.execute_reply":"2023-06-01T05:02:11.180273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dummy Encoding Species","metadata":{}},{"cell_type":"code","source":"# Select a specific column in the data frame.\nx_train = x_train_df['1024']\n\n# Dummy encoding.\nencoded_data = pd.get_dummies(x_train)\n\n# Concat the encoded data into the new data frame.\nx_train_df = pd.concat([x_train_df, encoded_data], axis=1)\nx_train_df.drop('1024', axis=1, inplace=True)\nx_train_df","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:11.182994Z","iopub.execute_input":"2023-06-01T05:02:11.183336Z","iopub.status.idle":"2023-06-01T05:02:15.462432Z","shell.execute_reply.started":"2023-06-01T05:02:11.183308Z","shell.execute_reply":"2023-06-01T05:02:15.461347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Gets the column names as a list.\ncolumn_names = x_train_df.columns.tolist()\n\n# Converts column names from 1025th to 2497th to numeric.\nfor i in range(1024, 2497):\n    column_names[i] = str(i)\n\n# Rename the column to a number and apply it to the data frame.\nx_train_df.columns = column_names\nx_train_df","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:15.463828Z","iopub.execute_input":"2023-06-01T05:02:15.464193Z","iopub.status.idle":"2023-06-01T05:02:15.508635Z","shell.execute_reply.started":"2023-06-01T05:02:15.464165Z","shell.execute_reply":"2023-06-01T05:02:15.507517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test = x_test_df['1024']\n\nencoded_data = pd.get_dummies(x_test)\n\n# Cocat the encoded data into the new data frame.\nx_test_df = pd.concat([x_test_df, encoded_data], axis=1)\nx_test_df.drop('1024', axis=1, inplace=True)\nx_test_df","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:15.51529Z","iopub.execute_input":"2023-06-01T05:02:15.515645Z","iopub.status.idle":"2023-06-01T05:02:16.521991Z","shell.execute_reply.started":"2023-06-01T05:02:15.515617Z","shell.execute_reply":"2023-06-01T05:02:16.520953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Gets the column names as a list.\ncolumn_names = x_test_df.columns.tolist()\n\n# Converts column names from 1025th to 2497th to numeric.\nfor i in range(1024, 1099):\n    column_names[i] = str(i)\n\n# Rename the column to a number and apply it to the data frame.\nx_test_df.columns = column_names\nx_test_df","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:16.523875Z","iopub.execute_input":"2023-06-01T05:02:16.524338Z","iopub.status.idle":"2023-06-01T05:02:16.568411Z","shell.execute_reply.started":"2023-06-01T05:02:16.524297Z","shell.execute_reply":"2023-06-01T05:02:16.567429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test_df_resized = pd.concat([x_test_df, pd.DataFrame(0, index=x_test_df.index, columns=range(1099, 2497))], axis=1)\nx_test_df_resized","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:16.571884Z","iopub.execute_input":"2023-06-01T05:02:16.572336Z","iopub.status.idle":"2023-06-01T05:02:19.456306Z","shell.execute_reply.started":"2023-06-01T05:02:16.572305Z","shell.execute_reply":"2023-06-01T05:02:19.455111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train_np = x_train_df.to_numpy()\nx_test_np = x_test_df_resized.to_numpy()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:19.457964Z","iopub.execute_input":"2023-06-01T05:02:19.458387Z","iopub.status.idle":"2023-06-01T05:02:21.688522Z","shell.execute_reply.started":"2023-06-01T05:02:19.458357Z","shell.execute_reply":"2023-06-01T05:02:21.687275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeling","metadata":{}},{"cell_type":"code","source":"x_train, x_valid, y_train, y_valid = train_test_split(x_train_np, y_train, shuffle=True, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:21.690162Z","iopub.execute_input":"2023-06-01T05:02:21.690612Z","iopub.status.idle":"2023-06-01T05:02:26.874657Z","shell.execute_reply.started":"2023-06-01T05:02:21.690574Z","shell.execute_reply":"2023-06-01T05:02:26.873323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Dense, concatenate\n\nnfeats = x_train.shape[1]\nnlabels = y_train.shape[1]\n\n# First feature\ninput1 = Input(shape=(1024,))\nx1 = Dense(256, activation='relu')(input1)\nx1 = Dense(128, activation='relu')(x1)\n\n# Second feature\ninput2 = Input(shape=(x_train.shape[1] - 1024,))\nx2 = Dense(256, activation='relu')(input2)\nx2 = Dense(128, activation='relu')(x2)\n\n# Merged\nmerged = concatenate([x1, x2])\n\n# Output layer for final classification\noutput = Dense(nlabels, activation='sigmoid')(merged)\n\n# Creating a Model\nmodel = Model(inputs=[input1, input2], outputs=output)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=[AUC()])\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:26.876009Z","iopub.execute_input":"2023-06-01T05:02:26.876382Z","iopub.status.idle":"2023-06-01T05:02:27.209363Z","shell.execute_reply.started":"2023-06-01T05:02:26.876352Z","shell.execute_reply":"2023-06-01T05:02:27.208075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# First feature data\nx_train_feature1 = x_train[:, :1024]\nx_valid_feature1 = x_valid[:, :1024]\n\n# Second feature data\nx_train_feature2 = x_train[:, 1024:]\nx_valid_feature2 = x_valid[:, 1024:]","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:27.210777Z","iopub.execute_input":"2023-06-01T05:02:27.211165Z","iopub.status.idle":"2023-06-01T05:02:27.217069Z","shell.execute_reply.started":"2023-06-01T05:02:27.211135Z","shell.execute_reply":"2023-06-01T05:02:27.216235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training\nmodel.fit([x_train_feature1, x_train_feature2], y_train, epochs=10, batch_size=32, validation_data=([x_valid_feature1, x_valid_feature2], y_valid))","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:02:27.218044Z","iopub.execute_input":"2023-06-01T05:02:27.218369Z","iopub.status.idle":"2023-06-01T05:08:21.017212Z","shell.execute_reply.started":"2023-06-01T05:02:27.218341Z","shell.execute_reply":"2023-06-01T05:08:21.015827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del x_train, y_train, x_valid, y_valid, labels\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:08:21.019221Z","iopub.execute_input":"2023-06-01T05:08:21.02024Z","iopub.status.idle":"2023-06-01T05:08:22.670328Z","shell.execute_reply.started":"2023-06-01T05:08:21.020193Z","shell.execute_reply":"2023-06-01T05:08:22.669087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"x_test_feature1 = x_test_np[:, :1024]\nx_test_feature2 = x_test_np[:, 1024:]","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:08:22.67184Z","iopub.execute_input":"2023-06-01T05:08:22.672208Z","iopub.status.idle":"2023-06-01T05:08:22.677278Z","shell.execute_reply.started":"2023-06-01T05:08:22.672179Z","shell.execute_reply":"2023-06-01T05:08:22.676271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = model.predict([x_test_feature1, x_test_feature2])\ndel x_test_np\ngc.collect()\n\nchunk_size = 5_000\nchunks = [range(i, min(i + chunk_size, len(predictions))) for i in range(0, len(predictions), chunk_size)]\n\nfinal_sub = pd.DataFrame()  # Create an empty DataFrame to hold the final result\n\nprint(f\"processing {len(chunks)} chunks of {chunk_size} predictions each\")\n\nfor chunk in chunks:\n    print(f\"processing chunk {chunk}\")\n    sub = pd.DataFrame(data=predictions[chunk], columns=list(selected_terms), index=test_ids[chunk])\n    sub = sub.T.unstack().reset_index(name='prediction')\n    sub = sub.loc[sub['prediction'] > 0]\n    final_sub = pd.concat([final_sub, sub])  # Concatenate current chunk DataFrame to the final DataFrame\n\nfinal_sub.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:08:22.67887Z","iopub.execute_input":"2023-06-01T05:08:22.679594Z","iopub.status.idle":"2023-06-01T05:09:45.699655Z","shell.execute_reply.started":"2023-06-01T05:08:22.679563Z","shell.execute_reply":"2023-06-01T05:09:45.698419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_sub.to_csv('submission.tsv', sep='\\t', index=False, header=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-01T05:09:45.701317Z","iopub.execute_input":"2023-06-01T05:09:45.70165Z","iopub.status.idle":"2023-06-01T05:15:05.275543Z","shell.execute_reply.started":"2023-06-01T05:09:45.701622Z","shell.execute_reply":"2023-06-01T05:15:05.274304Z"},"trusted":true},"execution_count":null,"outputs":[]}]}