{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":154281}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## RSNA Knee Abnormality Detection - Competition  & Data Structure\n\n### Competition Overview\nThe knee is the most commonly injured and imaged joint in the body, however, \nthe ways in which radiologists interpret MRI imaging scans differ.  The \ngoal of the competition is to develop ML models that detect clinically\nimportant knee abnormalities, which could provide useful decision support\ntools to radiologists in practice.\n\n### Competition Data\nBased on the competition data descriptions the following is expected:\n\n|File or Folder|Description|Notes|\n|--------------|-----------|-----|\n|`train.csv`|Contains one row per study with the following labels: `StudyInstanceUID`(Study Unique ID), `Report`, `ACL`, `MCL`, `Medial Meniscus`, `Lateral Meniscus`, `Medial OA`, `Lateral_OA`, `PF OA`,`Effusion`,`Synovitis`, `Baker's`, `Contusion`, `Fracture`|All are binary except `StudyInstanceUID`, `PatientSex`, and `Report`. `Report` is the clinicians' report and includes reports in multiple languages. `PatientSex` is described in the docs, but missing|\n|`train_series.csv` | Contains one row per training series (each study containing several series) with the following labels: `StudyInstanceUID` (Study Unique ID), `SeriesInstanceUID` (Series Unique ID), `Fluid_Sensitive`, `Fat_Suppression`, `Anotomical Plane`| `Fluid Sensitive` and `Fat Suppression` are binary and indicate different MRI weightings. Values for `Anatomical Plan` are 'Sagittal', 'Coronal', or 'Axial'|\n|`train_series/`|Folder containing MRI series associated with the training studies, with files in dcm (DICOM) format). Files are organized as  `train_series/<StudyInstanceUID>/<SeriesInstanceUID>/<SOPInstanceUID>.dcm`| Series may have 20–45 slices (median 30), with a long tail out to a few hundred|\n|`test.csv`| Example test file with three study IDs from the public test set. During scoring, this example data will be replaced with the actual test data.|There are about 1300 studies in the test set.|\n|`test_series.csv`|Same schema as `train_series.csv` for the example test studies. |Replaced with the real test-series descriptors during scoring.|\n|`test_series/`|Example test DICOMs, same layout as `train_series/`|Replaced with the real test DICOMs during scoring.|\n\n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom pathlib import Path\nDATA_LOC = Path('/kaggle/input/competitions/rsna-knee-abnormality-detection')\n\ntry:\n    test_df = pd.read_csv(f'{DATA_LOC / \"test.csv\"}')\n    print(f\"'test.csv'loaded successfully:{test_df.shape[0]} rows, {test_df.shape[1]} columns\")\n    test_series_df = pd.read_csv(f'{DATA_LOC / \"test_series.csv\"}')\n    print(f\"'test_series.csv'loaded successfully:{test_series_df.shape[0]} rows, {test_series_df.shape[1]} columns\")\n    train_df = pd.read_csv(f'{DATA_LOC / \"train.csv\"}')\n    print(f\"'train.csv'loaded successfully:{train_df.shape[0]} rows, {train_df.shape[1]} columns\")\n    train_series_df = pd.read_csv(f'{DATA_LOC / \"train_series.csv\"}')\n    print(f\"'train_series.csv'loaded successfully:{train_series_df.shape[0]} rows, {train_series_df.shape[1]} columns\")\nexcept Exception as e:\n    print(f\"Error when loading csv files: {e}\")\n    \n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T18:00:05.512148Z","iopub.execute_input":"2026-08-13T18:00:05.512584Z","iopub.status.idle":"2026-08-13T18:00:05.694627Z","shell.execute_reply.started":"2026-08-13T18:00:05.512534Z","shell.execute_reply":"2026-08-13T18:00:05.693655Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# There should be no duplication of StudyInstanceUIDs in train.csv\nassert (\n    len(train_df['StudyInstanceUID']) == \n    len(train_df['StudyInstanceUID'].unique())\n), 'Duplicate StudyInstanceUIDs in train.csv'\n\n# There should be no duplication of SeriesInstanceUIDs in train_series.csv\nassert (\n    len(train_series_df['SeriesInstanceUID'].unique()) ==\n    len(train_series_df['SeriesInstanceUID'])\n), 'Duplicate SeriesInstanceUIDs in train.csv'\n    \n# The count of unique StudyInstanceUIDs in train_series.csv \n# should be equal to the count of StudyInstanceUIDs in train.csv\nassert (\n    len(train_series_df['StudyInstanceUID'].unique())== \n    len(train_df['StudyInstanceUID']) \n), 'Study ID counts differ between training CSVs'\n\n# Each StudyInstanceUID train_series.csv appears in train.csv               \nassert (\n    len(train_df['StudyInstanceUID']) ==\n    sum(pd.Series(train_series_df['StudyInstanceUID'].unique()).isin(train_df['StudyInstanceUID']))\n), 'Training csv files have different numbers of StudyInstanceUIDs'\n\n# The names of training series folder names in each study folder should\n# match the SeriesInstanceUIDs in train_series.csv\ntrain_study_loc = DATA_LOC / 'train_series'\n\ndisk = {p.name for p in train_study_loc.iterdir() if p.is_dir()}\nextra = disk - set(train_df['StudyInstanceUID'].astype(str))\n\ndisk_series = {\n    (study.name, series.name)\n    for study in train_study_loc.iterdir() if study.is_dir()\n    for series in study.iterdir() if series.is_dir()\n}\ncsv_series = set(zip(\n    train_series_df['StudyInstanceUID'].astype(str),\n    train_series_df['SeriesInstanceUID'].astype(str),\n))\nextra_series = disk_series - csv_series\ncsv_studies = set(train_df['StudyInstanceUID'].astype(str))\n\nassert disk == csv_studies, (\n    f'Study folders absent from train.csv: {sorted(disk - csv_studies)[:5]}\\n'\n    f'train.csv studies without folders: {sorted(csv_studies - disk)[:5]}'\n)\nassert disk_series == csv_series, (\n    f'Series folders absent from CSV: {sorted(disk_series - csv_series)[:5]}\\n'\n    f'CSV series without folders: {sorted(csv_series - disk_series)[:5]}'\n)\n\n# These columns should be present in train.csv: 'StudyInstanceUID', 'Report', \n# 'ACL', 'MCL', 'Medial Meniscus', 'Lateral Meniscus', 'Medial OA', 'Lateral OA', 'PF OA', \n# 'Effusion', 'Synovitis', \"Baker's\", 'Contusion', and 'Fracture'\nexp_study_features = set(['StudyInstanceUID', 'Report', 'ACL', 'MCL', \n    'Medial Meniscus', 'Lateral Meniscus', 'Medial OA', 'Lateral OA', 'PF OA', \n    'Effusion', 'Synovitis', \"Baker's\", 'Contusion', 'Fracture'])\nactual_study_features = set(train_df.columns)\nunmatched_expected =  exp_study_features - actual_study_features\nunmatched_actual = actual_study_features - exp_study_features\nassert exp_study_features == actual_study_features, (\n    f'Unmatched expected features: {unmatched_expected}\\n'\n    f'Unmatched actual features: {unmatched_actual}'\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T18:33:17.760947Z","iopub.execute_input":"2026-08-13T18:33:17.76171Z","iopub.status.idle":"2026-08-13T18:33:45.503063Z","shell.execute_reply.started":"2026-08-13T18:33:17.76163Z","shell.execute_reply":"2026-08-13T18:33:45.501972Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Takeaways\n- Data generally shaped as expected; study UIDs and series UIDs are indeed unique\n- Clean linkage between study UIDs, series UIDs, and actual file folders\n- **Important Note**: `PatientSex` is *not provided* in the training data ('train.csv') despite the competition documentation; however, it may be worth extracting from the notes.","metadata":{}},{"cell_type":"code","source":"train_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T19:19:27.428166Z","iopub.execute_input":"2026-08-13T19:19:27.428532Z","iopub.status.idle":"2026-08-13T19:19:27.441513Z","shell.execute_reply.started":"2026-08-13T19:19:27.4285Z","shell.execute_reply":"2026-08-13T19:19:27.440361Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# All of the data in the knee injury label columns should be binary or NA according to the docs\nlabel_cols = sorted(actual_study_features - {'StudyInstanceUID', 'Report'})\nvals = train_df[label_cols]\nbad = ~(vals.isin([0, 1]) | vals.isna())\nassert not bad.any().any(), (\n    'Unexpected values:\\n'\n    f'{vals[bad].stack().groupby(level=1).unique()}'\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T19:01:42.109506Z","iopub.execute_input":"2026-08-13T19:01:42.109885Z","iopub.status.idle":"2026-08-13T19:01:42.122749Z","shell.execute_reply.started":"2026-08-13T19:01:42.109853Z","shell.execute_reply":"2026-08-13T19:01:42.121715Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Takeaways\n- All studies associated with StudyInstanceUID and report\n- 58 studies with labeled knee defects (gold labeled);\n  only binary values as expected\n- Will need to rely on the report to silver label the rest;\n  can use gold labeled records for prompt eval","metadata":{}},{"cell_type":"code","source":"train_series_df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T18:54:25.687583Z","iopub.execute_input":"2026-08-13T18:54:25.687956Z","iopub.status.idle":"2026-08-13T18:54:25.702637Z","shell.execute_reply.started":"2026-08-13T18:54:25.687888Z","shell.execute_reply":"2026-08-13T18:54:25.701717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"label_cols = sorted(set(train_series_df.columns) - {'StudyInstanceUID', 'SeriesInstanceUID'})\nvals = train_series_df[label_cols]\nbad = ~(vals.isin([0, 1, 'Sagittal', 'Coronal', 'Axial']))\nassert not bad.any().any(), (\n    'Unexpected values:\\n'\n    f'{vals[bad].stack().groupby(level=1).unique()}'\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-08-13T19:15:18.115599Z","iopub.execute_input":"2026-08-13T19:15:18.11596Z","iopub.status.idle":"2026-08-13T19:15:18.126675Z","shell.execute_reply.started":"2026-08-13T19:15:18.115897Z","shell.execute_reply":"2026-08-13T19:15:18.125716Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Takeaways\n- No missing data in train_series\n- Values for the non ID columns are as expected","metadata":{}}]}