{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.14"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84795,"databundleVersionId":10462807,"sourceType":"competition"}],"dockerImageVersionId":30804,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":22.341371,"end_time":"2024-12-11T03:22:13.479076","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-12-11T03:21:51.137705","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.core.display import HTML\n\n\ndef apply_css_styles(\n font_name: str = \"Lato\",\n fallback_font: str = \"Verdana\",\n css_content: str = None,\n verbose: bool = False\n) -> HTML:\n \"\"\"Applies custom CSS styles within a Jupyter notebook cell.\n\n Args:\n font_name (str, optional): \n The primary font to use in the styles.\n fallback_font (str, optional): \n The fallback font to use if the primary font is unavailable.\n css_content (str, optional): \n Custom CSS content to use. \n If None, default styles are used.\n verbose (bool, optional): \n Whether to print the generated CSS for debugging.\n\n Returns:\n IPython.core.display.HTML: \n HTML object with the injected styles.\n \"\"\"\n try:\n # Default CSS content if none is provided\n default_css = '''\np, li, a, b, h1, h2, h3, h4, h5, h6, title, ul, strong, sup, sub, em, i, blockquote, label {\n font-family: Verdana !important;\n}\n\nb, h1 {\n font-weight: 900 !important;\n}\n\nh2, h3, h4 ul {\n font-weight: 700 !important;\n}\n\n.fa, .far, .fas {\n font-family: \"Font Awesome 5 Free\" !important;\n}\n'''\n\n # Generate font import string dynamically based on the provided font name\n font_import = (\n f\"\\n@import url('https://fonts.googleapis.com/css2?family={font_name.replace(' ', '+')}:ital,wght@0,100;0,300;0,400;0,700;0,900;1,100;1,300;1,400;1,700;1,900&display=swap');\\n\"\n )\n\n # Use provided CSS content or fallback to default\n css_to_use = css_content or default_css\n\n # Replace fallback font in the CSS content\n css_to_use = css_to_use.replace(\"Verdana\", font_name)\n\n # Combine the font import and the CSS content into a single HTML style block\n combined_styles = f\"\"\n\n if verbose:\n print(combined_styles) # Print the CSS for debugging if verbose is True\n\n return HTML(combined_styles) # Return the generated styles as an HTML object\n\n except Exception as e:\n raise RuntimeError(f\"An error occurred while applying styles: {str(e)}\")\n\n# Apply styles (example usage)\napply_css_styles(verbose=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T14:59:36.024156Z","iopub.execute_input":"2025-01-08T14:59:36.024553Z","iopub.status.idle":"2025-01-08T14:59:36.066319Z","shell.execute_reply.started":"2025-01-08T14:59:36.024517Z","shell.execute_reply":"2025-01-08T14:59:36.065103Z"},"_kg_hide-input":true,"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n

KPRIZE vs. SWE-BENCH

\n

Can Language Models Resolve Real-World GitHub Issues?

\n \"Swell\n \n
\n\n
\n\n

\n \n LET'S \n LEARN \n TOGETHER !\n

\n\n UNDERSTANDING \n THROUGH\n EXPLORATION\n

\n\n

CREATED BY: DARIEN SCHETTLER

\n\n
\n\n
\n πŸ›‘   WARNING:

THIS IS A WORK IN PROGRESS ---- SUPER–DUPER–WIP ----
\n
\n\n
\n πŸ‘   IF YOU FORK THIS OR FIND THIS HELPFUL   πŸ‘

PLEASE UPVOTE!

This was a lot of work for me, and it may seem silly, but the πŸ”Ό makes me feel appreciated. πŸ˜…\n
\n\n
","metadata":{}},{"cell_type":"markdown","source":"

\n CHANGELOG\n

\n\n","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n TABLE OF CONTENTS\n

\n\n
\n\n

1    INTRODUCTION & JUSTIFICATION

\n\n
\n\n

2    BACKGROUND INFORMATION

\n\n
\n\n

3    IMPORTS

\n\n
\n\n

4    SETUP AND HELPER FUNCTIONS

\n\n
\n\n

5    EXPLORATORY DATA ANALYSIS

\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

1  INTRODUCTION & JUSTIFICATION    ⤒

\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.1 WHAT IS THIS?

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.2 WHY IS THIS?

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.3 WHO IS THIS FOR?

\n
\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.4 HOW WILL THIS WORK?

\n
\n\n\n

I'm going to assemble some markdown cells (like this one) at the beginning of the notebook to go over some concepts/details/etc.

\n\n

Following this, I will likely go through a few examples from the SWE-Lite dataset to better understand the problems and how they are solved

","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2  BACKGROUND INFORMATION    ⤒

\n\n
\n\nYour challenge is to develop an 'agent' that can resolve real-world GitHub issues. \n\nThis competition builds on the SWE-bench benchmark by using Kaggle's forecasting format to ensure that all of the git issues used for the private test set cannot did not exist when the submitted model was trained.\n\nEssentially, this competition emulates SWE-BENCH with some minor – BUT IMPORTANT – modifications. Let's briefly discuss those differences beefore we dive into more generic background information about SWE-BENCH and other related background information.\n\n
\n\nKEY DIFFERENCES BETWEEN KPRIZE AND SWE-BENCH\n
\n\nBesides the obvious key differences of where/how the hosting is done (containerization, raw setup, constraints, etc.), the following are the main differences I wanted to call out.\n\n1. **EVALUATION METRICS**\n * **SWE-Bench:**\n * Primarily measures the percentage of issues resolved by the AI agent, focusing on the agent's ability to generate correct solutions without explicitly penalizing incorrect ones.\n * **KPrize:**\n * The **scoring metric** for the competition is designed to incentivize skipping an issue (if the solution is uncertain) over submitting a bad patch. Here's how the formula works:\n * $$\\text{score} = \\frac{a - b}{a + b + c}$$\n * Where:\n - $a$ is the number of correctly resolved issues.\n - $b$ is the number of failing issues (incorrect solutions).\n - $c$ is the number of skipped issues.\n2. **DATA CONTAMINATION AND PHASING**\n * **SWE-Bench**:\n * The test set is publicly available, which poses a risk of data contamination.\n * Models might inadvertently train on test data, leading to inflated performance metrics.\n * **KPrize**:\n * To mitigate the contamination issue, KPrize utilizes Kaggle's forecasting format to ensure that all of the git issues used for the private test set cannot did not exist when the submitted model was trained.\n * This results in a phased competition:\n > **------** PHASE 1 **------**\n > \n > A model training phase with a leaderboard using only the public test set of historical data. This test set has about 100 instances and will require approximately one hour of your notebook's nine hour runtime limit to execute unit tests. We expect to improve this performance in the future. The public test set labels can be scraped from github. Accordingly, we may make updates mid-competition in order to keep the public leaderboard reasonably useful. With that said, it is infeasible to keep the public leaderboard results entirely leak-free.\n >\n > \n > **------** PHASE 2 **------**\n > \n > A forecasting phase with a leaderboard using only data collected after the submission deadline. You should expect this test set to contain approximately 150-200 instances. The exact count will be provided by the evaluation API. The public test set instances will not be served by the evaluation API during the forecasting phase.\n \n\n3. **THE SWE-BENCH PYTHON LIBRARY**\n * **SWE-Bench**\n * The [SWE-Bench library](https://github.com/swe-bench/SWE-bench) is designed to evaluate the ability of large language models (LLMs) to generate patches that resolve real-world software issues. It provides tasks where models receive an issue description and a codebase, and must generate fixes to address the issue. Below is a high-level overview of its structure:\n * **Root Directory (SWE-bench)**:\n * Contains essential files for the benchmark, such as setup scripts, configuration files, and Docker files for creating reproducible environments.\n * **Subdirectory (`swebench`)**: The core module of SWE-Bench, organized into several submodules:\n * **`collect`**: Handles the collection of evaluation tasks from GitHub repositories, including mirroring repositories and extracting task instances.\n * **`harness`**: Provides the evaluation framework to assess model-generated patches against the benchmark.\n * **`inference`**: Manages running inference for models, facilitating the generation of fixes for issues in repositories.\n * **`versioning`**: Maintains version control and manages installation configurations for repositories in the benchmark.\n * **`utils`**: Includes utility functions shared across the framework to support core functionalities.\n * **KPrize**\n * `kprize_setup/kprize` is a package in the competition dataset that contains the files used for installing this competition's adaptation of the SWE-Bench library. Note that this won't currently work on Windows. It has quite a few differences, but at a high level the structure is (ignoring the root directory):\n * **Package Directory (`kprize`)**:\n * Contains many utility files and other essential files (far more than are found in the original implementation). I think some of the utility files here may have been found previously in the **`utils`** submodule?\n * Contains various submodules that mostly overlap with the original implementation but differ in some regards:\n * **`collection`**: Same as **`collect`**, but I haven't investigated.\n * **`harness`**: This is likely the same as the original, but I haven't investigated.\n * **`bundling`**: This is new. TBD\n * **`evaluation`**: This is new. TBD\n * **`scripts`**: This is new. TBD\n","metadata":{}},{"cell_type":"markdown","source":"

2.1 UNDERSTANDING THE KPRIZE

\n
\n\n
\n\nThe K Prize competition, announced by Andy Konwinski at NeurIPS in December 2024, inspired by SWE-Bench but born from a passion for open-source innovation and competitive programming, this prize challenges teams to build AI models that can effectively solve real-world GitHub issues.\n\nThe competition builds upon SWE-bench, a benchmark that tests AI models against real-world software engineering problems from GitHub repositories. However, it introduces a crucial innovation: a contamination-free evaluation framework that prevents models from being inadvertently trained on test data.\n\nAs Konwinski notes, \"Automating this task will let human software engineers spend lots more time designing new features, reforming abstractions, interfacing with users, and other tasks that are more inherently human (and, for many of us, more fun).\" \n\nBy providing compute resources through Kaggle and implementing a substantial prize structure, the competition aims to catalyze research progress in the same way the Netflix Prize inspired the creation of Apache Spark.\n\n
","metadata":{}},{"cell_type":"markdown","source":"

2.2 COMPETITION OVERVIEW

\n
\n\n
\n\nPRIMARY TASK DESCRIPTION\n
\n
\nDevelop open-source large language models capable of achieving a 90% score on a contamination-free version of SWE-bench, significantly surpassing the current highest-performing model's 55% achievement. The competition specifically targets real-world software engineering problems, with test data collected after the submission deadline to ensure genuine evaluation.\n\n
\n\nCOMPETITION STRUCTURE\n
\n
\nThe competition proceeds in two distinct phases:\n\n**Model Training Phase:**\n* Public test set of ~100 historical instances\n* Approximately one hour runtime per evaluation\n* Public leaderboard based on historical data\n* Labels can be scraped from GitHub\n\n**Forecasting Phase:**\n* 150-200 new instances collected post-submission\n* Completely new, unseen test data\n* Final evaluation on contamination-free dataset\n* Exact instance count provided by evaluation API\n\n
\n\nEVALUATION INNOVATIONS AND THE METRIC\n
\n
\nThe K Prize introduces three key innovations:\n\n**Contamination-Free Testing:**\n* Test set created after model submissions\n* Prevents training on test data\n* Ensures genuine evaluation of AI capabilities\n\n**Focus on Open Source:**\n* Only open-source code and open-weight models eligible\n* Encourages community collaboration\n* Builds on collective innovation\n\n**Support for Independent Developers:**\n* Kaggle provides computing resources\n* Levels playing field for smaller teams\n* Promotes \"small AI\" innovation\n\n
\n\nSubmissions are scored using a metric that rewards quality over quantity:\n\n$$\\text{score} = \\frac{a - b}{a + b + c}$$\n\nWhere:\n- $a$ is the number of correctly resolved issues.\n- $b$ is the number of failing issues (incorrect solutions).\n- $c$ is the number of skipped issues.\n\nThis formula **HEAVILY incentivizes** skipping difficult issues rather than guessing or submitting incorrect patches.\n\n
\n\nPRIZE STRUCTURE\n
\n
\nTotal Prize Fund: **\\$1,225,000**\n\n* **Leaderboard Prizes:**\n * 1st Place: **\\$50,000**\n * 2nd Place: **\\$20,000**\n * 3rd-5th Place: **\\$10,000 each**\n\n* **Threshold Bonuses:**\n * Additional **\\$50,000** pool for each threshold (30%, 40%, 50%, 60%, 70%, 80%, 90%)\n * Distributed proportionally among qualifying top-5 teams\n\n* **Grand Prize:**\n * **\\$775,000** additional for first place if reaching 90%\n * Brings total potential first place prize to **\\$1 million**\n\n
\n\nTECHNICAL FRAMEWORK REQUIREMENTS\n
\n
\n**Environment:**\n* Python 3.11-based evaluation environments\n* Custom adaptation of SWE-bench library\n* Kaggle-provided Docker container recommended for testing\n\n**Data Structure:**\n* Issue metadata including repo, problem statement, and test information\n* Complete repository copies for context\n* Specialized evaluation API for submission handling\n\n**Submission Requirements:**\n* Must use provided Python evaluation API\n* Open source code and open weight models only\n* Full documentation and reproducibility required\n\n
\n\nCOMPETITION HOST(S)/CONTRIBUTOR(S)\n
\n
\nAndy Konwinski is a co-founder of Databricks, Perplexity and Laude Ventures. The prize is personally funded by Konwinski, leveraging his success from Databricks (valued at $62 billion) and Perplexity (approaching double-digit billion-dollar valuation). \n\nThe competition is hosted on Kaggle, working in collaboration with SWE-bench to develop and maintain the evaluation framework.\n\nThe initiative was announced at the Neural Information Processing Systems (NeurIPS) conference in December 2024.\n\nRead more about Andy and bask in his beardliness here\n\n
\n\nKEY DATES\n
\n
\n**Start Date:** December 11, 2024\n\n**Entry Deadline:** March 5, 2025\n\n**Team Merger Deadline:** March 5, 2025\n\n**Final Submission Deadline:** March 12, 2025\n\n**Competition End Date:** June 11, 2025\n","metadata":{}},{"cell_type":"markdown","source":"

2.3 AN EXAMPLE TO ILLUSTRATE (ASTROPY ISSUE 16825)

\n
\n\nRetrieved with:\n\n```python\n# (0) Necessary imports\nimport pandas as pd\n\n# (1) Unzip so we have access to the KPRIZE dataset\n%%capture\n!mkdir -p /kaggle/tmp/konwinski-prize-alt\n!unzip /kaggle/input/konwinski-prize/data.a_zip -d /kaggle/tmp/konwinski-prize-alt/\n\n# (2) Load into a dataframe\ndf = pd.read_parquet('/kaggle/tmp/konwinski-prize-alt/data/data.parquet')\n\n# (3) Get a pd.Series for a demo example\n# --> This is actually the row with the least text in `patch` and `test_patch` combined\nDEMO_EX = df.iloc[4]\n```\n\nAnd then I manually formatted and took photos and hosted the relevant images...\n\n
\n\n`instance_id`\n
\n
\n\n> astropy__astropy-16830\n\n
\n\n`repo`\n
\n
\n\n> astropy/astropy\n\n
\n\n`problem_statement`\n
\n
\n\n> KeyError: \\'version_1_3_or_later\\' when parsing certain VOTables\\n### Description\\n\\nWhen parsing VOTables (for instance) VOTables with empty integer literals in some places (e.g., MIN value=\"\" in a VALUES element), astropy crashes with\\r\\n\\r\\n\\`\\`\\`\\r\\nKeyError: \\'version_1_3_or_later\\'\\r\\n\\`\\`\\`\\n\\n### Expected behavior\\n\\nWell, you *could* argue that at least some such VOTables should be rejected (a NULL value in a MIN, indeed, does not make sense) with a sensible error message; with the patch proposed in the accompanying PR, astropy emits a warning. I notice in passing that the reproducer table passes stilts votlint. I also note in passing that against the warning, a value=\"\" is accepted in a PARAM (as it should, as this is the way to express NULLs).\\r\\n\\r\\nMy fix in the accompanying PR just stops the crashing, leading to a workable table.\\n\\n### How to Reproduce\\n\\nTry\\r\\n\\r\\n\\`\\`\\`python\\r\\nfrom astropy import table\\r\\ntable.Table.read(\"with-empty-min.vot\", format=\"votable\")\\r\\n\\`\\`\\`\\r\\n\\r\\nwith the table from https://docs.g-vo.org/with-empty-min.vot\\n\\n### Versions\\n\\nBoth current HEAD and what\\'s in Debian bookworm.\\r\\n\\nFix votable 1 3 check\\n\\r\\n\\r\\n\\r\\n### Description\\r\\n\\r\\n\\r\\nWhen parsing VOTables (for instance) VOTables with empty integer literals in some places (e.g., MIN value=\"\" in a VALUES element), astropy crashes with\\r\\n\\r\\n\\`\\`\\`\\r\\nKeyError: \\'version_1_3_or_later\\'\\r\\n\\`\\`\\`\\r\\n\\r\\nThis is a simple fix for the problem at hand, doing the key check in analogy to the other key checks of this sort in the few places where an index rather than the get method was used on the config object.\\r\\n\\r\\nOne *might* want to dig deeper, though; I am not exactly sure why the version_1_3_or_later key is missing on the example table from the bug report. But I suspect (well: hope:-) that is not critical for the bug fix.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nFixes #16825.\\n\n\n\n\n
\n\n`patch`\n
\n
\n\n> diff --git a/astropy/io/votable/tree.py b/astropy/io/votable/tree.py\\nindex 036ca8af732..ed85aeffcfd 100644\\n--- a/astropy/io/votable/tree.py\\n+++ b/astropy/io/votable/tree.py\\n@@ -1053,7 +1053,7 @@ def min(self):\\n @min.setter\\n def min(self, min):\\n if hasattr(self._field, \"converter\") and min is not None:\\n- self._min = self._field.converter.parse(min)[0]\\n+ self._min = self._field.converter.parse(min, config=self._config)[0]\\n else:\\n self._min = min\\n \\n@@ -1089,7 +1089,7 @@ def max(self):\\n @max.setter\\n def max(self, max):\\n if hasattr(self._field, \"converter\") and max is not None:\\n- self._max = self._field.converter.parse(max)[0]\\n+ self._max = self._field.converter.parse(max, config=self._config)[0]\\n else:\\n self._max = max\\n \\ndiff --git a/docs/changes/io.votable/16830.bugfix.rst b/docs/changes/io.votable/16830.bugfix.rst\\nnew file mode 100644\\nindex 00000000000..d30a2a9ff96\\n--- /dev/null\\n+++ b/docs/changes/io.votable/16830.bugfix.rst\\n@@ -0,0 +1,1 @@\\n+Fix KeyError when parsing certain VOTables.\\n\n\n\n\n
\n\n`test_patch`\n
\n
\n\n> diff --git a/astropy/io/votable/tests/test_tree.py b/astropy/io/votable/tests/test_tree.py\\nindex 7fd771cef77..224d189a33f 100644\\n--- a/astropy/io/votable/tests/test_tree.py\\n+++ b/astropy/io/votable/tests/test_tree.py\\n@@ -90,6 +90,31 @@ def test_namespace_warning():\\n parse(io.BytesIO(good_namespace_13), verify=\"exception\")\\n \\n \\n+def test_votable_values_empty_min_max():\\n+ \"\"\"Regression test for https://github.com/astropy/astropy/issues/16825\"\"\"\\n+ with_empty_minmax = b\"\"\"\\n+ \\n+ \\n+ \\n+ \\n+ HD number for this object\\n+ \\n+ \\n+ \\n+ \\n+ \\n+ \\n+ \\n+ AAMNIg==\\n+ \\n+ \\n+
\\n+ \\n+ \\n+ \"\"\"\\n+ parse(io.BytesIO(with_empty_minmax), verify=\"exception\")\\n+\\n+\\n def test_version():\\n \"\"\"\\n VOTableFile.__init__ allows versions of \\'1.1\\', \\'1.2\\', \\'1.3\\' and \\'1.4\\'.\\n\n\n\n\n
\n\n`pull_number`\n
\n
\n\n> 16830\n\n
\n\n`base_commit`\n
\n
\n\n> e39f486fec48d87aa3677326167954370d7a7bf9\n\n
\n\n`PASS_TO_PASS`\n
\n
\n\n> ['astropy/io/votable/tests/test_tree.py::test_check_astroyear_fail'\n 'astropy/io/votable/tests/test_tree.py::test_string_fail'\n 'astropy/io/votable/tests/test_tree.py::test_make_Fields'\n 'astropy/io/votable/tests/test_tree.py::test_unit_format'\n 'astropy/io/votable/tests/test_tree.py::test_namespace_warning'\n 'astropy/io/votable/tests/test_tree.py::test_version'\n 'astropy/io/votable/tests/test_tree.py::test_votable_tag'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_constructor'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_readout'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_write'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_write_after_table'\n 'astropy/io/votable/tests/test_tree.py::test_write_no_mivot'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_write_after_resource'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_forbidden_write'\n 'astropy/io/votable/tests/test_tree.py::test_mivot_order']\n\n
\n\n`FAIL_TO_PASS`\n
\n
\n\n> ['astropy/io/votable/tests/test_tree.py::test_votable_values_empty_min_max']\n\n
\n\n`issue_numbers`\n
\n
\n\n> [16825, 16826]","metadata":{}},{"cell_type":"markdown","source":"

2.4 EVALUATION INFORMATION

\n
\n\n
\n\nTo determine what 'correct' means we should review the description in the original SWE-Bench paper.\n> Evaluation is performed by unit test verification using post-PR behavior as the reference solution.\n\nKnowing this we can reexamine the metric provided by the hosts below:\n\n$$\\text{score} = \\frac{a - b}{a + b + c}$$\n\nWhere:\n- $a$ is the number of correctly resolved issues.\n- $b$ is the number of failing issues (incorrect solutions).\n- $c$ is the number of skipped issues.\n\nRemember, this formula **HEAVILY incentivizes** skipping difficult issues rather than guessing or submitting incorrect patches.\n\n
\n\nHOST PROVIDED IMPLEMENTATION\n
\n
\n\n```python\n# TBD\ndef host_implementation(...):\n ...\n```\n\nAs a placeholder here is my overly verbose implementation that would simply take the number of correct, incorrect and skipped examples:\n\n```python\ndef calculate_score(correct: int, incorrect: int, skipped: int) -> float:\n \"\"\"Calculate the SWE-Bench score based on correct, incorrect, and skipped solutions.\n \n Args:\n correct (int): Number of correctly resolved issues (a)\n incorrect (int): Number of failing issues (b)\n skipped (int): Number of skipped issues (c)\n \n Returns:\n float: Score calculated using (a-b)/(a+b+c) formula\n \"\"\"\n # All values must be positive or 0 and all values cannot be 0\n correct, incorrect, skipped = max(0, correct), max(0, incorrect), max(0, skipped)\n if not any([correct, incorrect, skipped])\n return 0.0\n \n # Calculate score using provided formula (no 0 check required due to above)\n return (correct - incorrect) / (correct + incorrect + skipped)\n```","metadata":{}},{"cell_type":"markdown","source":"

2.5 COMPETITION DATA OVERVIEW

\n
\n\n
\n\nDATA FILE DESCRIPTIONS\n\n**`data.a_zip` description:**\n* A renamed `.zip` archive created to work around filename and nested archive constraints\n* Unzipping this file creates the main data folder containing all dataset components\n\n**`data/data.parquet` columns:**\n* **`instance_id`** (string)\n * Unique string identifier for each instance (GitHub issue)\n* **`repo`** (string)\n * The GitHub repository relevant to the issue\n * Also accessible through the evaluation API\n* **`problem_statement`** (string)\n * Textual description of the issue\n * Also accessible through the evaluation API\n* **`patch`** (string)\n * The patch that resolves the issue\n * *Only provided in the train set*\n* **`test_patch`** (string)\n * The patch that resolves the issue\n * *Only provided in the train set*\n* **`pull_number`** (int)\n * The pull request number that resolved the issue\n* **`base_commit`** (string)\n * The commit used as the foundation for the provided repository copy\n* **`issue_numbers`** (int)\n * The original ID number of the GitHub issue\n* **`[PASS_TO_PASS/FAIL_TO_PASS]`** (list)\n * Lists containing unit tests to be executed for this issue\n\n**Directory Structure Details:**\n* **`data/*/`**\n * All other subdirectories are utilized by the evaluation API\n * Used to configure evaluation environments\n * All evaluation environments run Python 3.11\n\n**`kprize_setup/` description:**\n* Contains files for installing the competition's adapted swebench library\n* *Note: Currently not compatible with Windows systems*\n\n**`kaggle_evaluation/` description:**\n* Contains files implementing the evaluation API\n* Implementation details may be useful for offline testing\n* Recommended to start with the demo submission notebook\n* Strong recommendation to run API in Docker container based on Kaggle's image for local execution\n * Helps avoid conflicts with existing Python environments\n* The API will install if needed:\n * Micromamba\n * Several libraries (listed in kprize_setup/pip_packages)\n * Creates new Python environments as required\n\n
\n\nADDITIONAL NOTES:\n\n* Further updates are planned for the evaluation API and kprize library\n * Aimed at improving runtime\n * Will provide additional useful tooling\n * Updates not expected to impact core submission workflow\n* Refer to this forum post for additional details\n* Users are encouraged to source additional codebases for model training\n* Most metadata is only available in the train set\n\n
\n\nPOST-EDA DATA OBSERVATIONS\n\nTBD\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"

2.6 GLOSSARY AND BACKGROUND RESEARCH

\n
\n\n
\n\nI'm going to blow this out more in the future, most of my glossary/research is done by hand in a notebook right now, so this is just some preliminary definitions to get us started...\n\n
\n\n| Term | Definition | Example | Example Explanation |\n|------|------------|---------|-------------------|\n| Diff | A textual representation showing the differences between two versions of a file or set of files, highlighting added, removed, and modified lines. | `@@ -10,7 +10,7 @@\\n def hello():\\n- print 'hello'\\n+ print('hello')\\n` | Shows a change from Python 2.x to 3.x print syntax. The `-` line shows removed code, the `+` line shows added code. |\n| Patch | A file containing a set of changes (diff) that can be applied to a codebase. | `--- a/hello.py\\n+++ b/hello.py\\n@@ -1,3 +1,3 @@\\n def greet(name):\\n- print \"Hi, \" + name\\n+ print(f\"Hi, {name}\")` | A patch file showing the location (`hello.py`) and changes to update string formatting. |\n| Test Patch | A modification specifically made to test files in a codebase. | `--- a/test_calc.py\\n+++ b/test_calc.py\\n@@ -1,2 +1,6 @@\\n def test_add():\\n- assert calc.add(2,2) == 4\\n+ assert calc.add(2,2) == 4\\n+ assert calc.add(-1,1) == 0\\n+ assert calc.add(0,0) == 0` | Adds new test cases to verify edge cases for an addition function. |\n| Unix Patch Apply | A command-line operation that applies a patch file to a codebase. | `$ patch -p1 < bugfix.patch` | Applies changes from `bugfix.patch` to the codebase, where `-p1` strips one directory level from paths. |\n| GitHub Issue | A tracked item in GitHub's issue system. | `Title: \"TypeError when using pandas.read_csv() with custom delimiter\"\\nDescription: \"When using a tab delimiter...\"` | A bug report describing unexpected behavior with specific steps to reproduce. |\n| Pull Request (PR) | A GitHub feature that proposes changes from one branch to another. | `PR #1234: \"Fix TypeError in CSV parser with custom delimiters\"` | A proposed solution to the issue, containing code changes and tests. |\n| Base Commit | The specific commit that serves as the starting point for changes. | `git checkout a1b2c3d` | References commit `a1b2c3d` as the state before any changes are made. |\n| Pass to Pass Tests (P2P) | Tests that passed before and should pass after changes. | `def test_existing():\\n assert len([1,2,3]) == 3` | A test verifying basic list functionality that shouldn't be affected by changes. |\n| Fail to Pass Tests (F2P) | Tests that failed before and should pass after changes. | `def test_fix():\\n assert parser.read_tab('file.txt')` | A test specifically checking if the bug fix works. |\n| SWE-bench Verified | 500 manually verified solvable problems. | `\"Fix pandas DataFrame.fillna() with datetime values\"` | A verified issue with clear reproduction steps and known solution. |\n| SWE-bench Lite | 300 self-contained issues focused on functional bugs. | `\"Fix off-by-one error in list slicing\"` | A focused bug fix that doesn't require extensive codebase knowledge. |\n| Oracle Retrieval | Providing only the files edited in the reference solution. | `{\"files\": [\"pandas/core/frame.py\", \"pandas/tests/frame/test_datetime.py\"]}` | Only retrieves the specific files that need modification. |\n| BM25 Retrieval | Selecting relevant files based on issue description. | `query: \"DataFrame fillna datetime\"\\nreturned: [\"pandas/core/frame.py\", ...]` | Uses text similarity to find potentially relevant files. |\n| Breaking Resolved | Fixes the target issue but breaks existing functionality. | `Fixed: test_new_feature()\\nBroken: test_old_feature()` | The change fixed the bug but introduced a regression. |\n| Docker Evaluation | Containerized testing environment. | `docker run swe-bench eval --task pandas-1234` | Runs evaluation in an isolated, reproducible environment. |\n| Gold Patch | Reference solution from the original PR. | `diff --git a/src/main.py\\n--- a/src/main.py\\n+++ b/src/main.py\\n...` | The accepted solution that properly fixed the issue. |\n| Cross-Context Edit | Changes requiring modifications across multiple files. | `Changed: src/parser.py, src/utils.py, tests/test_parser.py` | A fix that requires coordinated changes in multiple components. |\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

3  IMPORTS    ⤒

\n\n
\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n... PIP INSTALLS STARTING ...\\n\")\n!pip install -q google-genai # For free Gemini access\n!pip install -q /kaggle/input/konwinski-prize/kprize_setup/kprize-1.0.0-py3-none-any.whl --no-index --find-links /kaggle/input/konwinski-prize/kprize_setup/pip_packages\nprint(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n\n# Competition specific imports\nimport kaggle_evaluation.konwinski_prize_inference_server\nfrom datasets import load_dataset\nfrom google.genai import types\nfrom google import genai\nimport unidiff\n\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None)\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\")\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\")\nimport polars as pl; print(f\"\\t\\t– POLARS VERSION: {pl.__version__}\")\n\n# Built-In Imports (mostly don't worry about these)\nfrom typing import Iterable, Any, Literal, Callable, Generator\nfrom kaggle_datasets import KaggleDatasets\nfrom dataclasses import dataclass\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom io import StringIO\nfrom glob import glob\nimport subprocess\nimport tempfile\nimport warnings\nimport requests\nimport textwrap\nimport hashlib\nimport imageio\nimport IPython\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport copy\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport gc\nimport re\nimport os\n\n# Rich\nfrom rich import pretty; pretty.install()\nfrom rich.markdown import Markdown\nfrom rich import print as rprint\nfrom rich.console import Console\nfrom rich.style import Style\nfrom rich.live import Live\nfrom rich.text import Text\nfrom rich import inspect\n\n# Visualization Imports (overkill)\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nfrom IPython.core.display import HTML\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport seaborn as sns\nimport plotly\n\ndef seed_it_all(seed: int = 7, fix_tf_seed: bool = False):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n\n if fix_tf_seed:\n raise NotImplementedError(\"Not importing TF yet...\")\n # tf.random.set_seed(seed)\n \nseed_it_all()\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T14:59:36.068669Z","iopub.execute_input":"2025-01-08T14:59:36.069027Z","iopub.status.idle":"2025-01-08T15:00:06.660268Z","shell.execute_reply.started":"2025-01-08T14:59:36.068994Z","shell.execute_reply":"2025-01-08T15:00:06.658795Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n\n\n\n

4  SETUP AND HELPER FUNCTIONS    ⤒

\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"

4.0 HELPER FUNCTIONS

\n
\n\n
\n","metadata":{}},{"cell_type":"code","source":"\"\"\"Rich visualization utility.\"\"\"\n\nimport pathlib\nfrom rich.filesize import decimal\nfrom rich.markup import escape\nfrom rich.text import Text\nfrom rich.tree import Tree\n\nDEFAULT_IGNORE_LIST: Iterable[str] = \".ipynb_checkpoints\", \".DS_Store\", \".git\", \".idea\", \".coverage\", \".pytest_cache\"\n\ndef get_directory_tree(\n directory: pathlib.Path,\n tree: Tree | None = None,\n show_hidden: bool = False,\n inplace: bool = False,\n ignore_list_extras: Iterable[str] = (),\n) -> Tree | None:\n \"\"\"Recursively build a Tree with directory contents.\n\n Args:\n directory (pathlib.Path): The directory to walk.\n tree (Tree, optional):\n The Tree object to build.\n If not provided, a new Tree is created.\n show_hidden (bool, optional):\n Whether to show hidden files.\n inplace (bool, optional):\n Whether to print the tree in place.\n If False, the tree is returned.\n ignore_list_extras (Iterable[str], optional):\n Additional file extensions to ignore.\n\n Returns:\n If inplace is False, the Tree object with the directory contents.\n Else, None.\n \"\"\"\n # Get the ignore list that includes the default and any extras\n ignore_list = sorted(set(DEFAULT_IGNORE_LIST) | set(ignore_list_extras))\n\n # Create a new Tree if one is not provided\n tree = tree or Tree(label=f\"[bold]{directory!s}[/bold] File Tree\") # type: ignore\n\n # Sort dirs first then by filename\n paths = sorted(\n pathlib.Path(directory).iterdir(),\n key=lambda path: (path.is_file(), path.name.lower()),\n )\n\n # Sort dirs first then by filename\n for path in paths:\n\n # Remove hidden files if show_hidden is False\n if path.name.startswith(\".\") and not show_hidden:\n continue\n\n # Skip files in the ignore list by suffix or name\n if path.is_file() and (path.suffix in ignore_list or path.name in ignore_list):\n continue\n\n # Skip directories only by name (not suffix)\n if path.is_dir() and path.name in ignore_list:\n continue\n\n # Add the directory to the tree\n if path.is_dir():\n\n # Style directories starting with \"__\" differently\n style = \"dim\" if path.name.startswith(\"__\") else \"\"\n\n # Add the directory to the tree\n branch = tree.add(\n f\"[bold magenta]:open_file_folder: [link file://{path}]{escape(path.name)}\",\n style=style,\n guide_style=style,\n )\n get_directory_tree(path, branch)\n\n # Add the file to the tree\n else:\n main_style = \"dim green\" if path.name.startswith(\"_\") else \"green\"\n ext_style = \"dim red\" if path.name.startswith(\"_\") else \"bold red\"\n file_size_style = \"dim blue\" if path.name.startswith(\"_\") else \"blue\"\n text_filename = Text(path.name, main_style)\n text_filename.highlight_regex(r\"\\..*$\", ext_style)\n text_filename.stylize(f\"link file://{path}\")\n text_filename.append(f\" ({decimal(path.stat().st_size)})\", file_size_style)\n if path.suffix == \".py\":\n icon = \"🐍 \"\n elif path.suffix == \".ipynb\":\n icon = \"πŸπŸ““ \"\n elif path.suffix == \".sh\":\n icon = \"πŸ”§ \"\n elif \".env\" in path.name.lower():\n icon = \"πŸ”‘ \"\n elif path.suffix == \".csv\":\n icon = \"πŸ“Š \"\n elif path.suffix in [\".yaml\", \".yml\", \".json\"]:\n icon = \"πŸ“œ \"\n elif path.suffix in [\".txt\", \".md\"]:\n icon = \"πŸ“ \"\n elif path.suffix in [\".png\", \".jpg\", \".jpeg\", \".gif\", \".svg\"]:\n icon = \"πŸ–ΌοΈ \"\n elif path.suffix in [\".zip\", \".tar\", \".gz\", \".7z\"]:\n icon = \"πŸ“¦ \"\n elif path.suffix in [\".pdf\"]:\n icon = \"πŸ“° \"\n elif path.suffix in [\".mp4\", \".avi\", \".mov\", \".mkv\"]:\n icon = \"πŸŽ₯ \"\n elif path.suffix in [\".mp3\", \".wav\", \".flac\"]:\n icon = \"🎡 \"\n elif path.suffix in [\".html\", \".css\", \".js\"]:\n icon = \"🌐 \"\n elif path.suffix in [\".exe\", \".msi\"]:\n icon = \"πŸ› οΈ \"\n elif path.suffix in [\".docx\", \".pptx\", \".xlsx\"]:\n icon = \"πŸ“„ \"\n elif path.suffix in [\".parquet\", \".feather\"]:\n icon = \"🧼 \"\n elif path.suffix in [\".db\", \".sqlite\", \".sql\", \".jsonl\"]:\n icon = \"πŸ—„οΈ \"\n else:\n icon = \"πŸ“„ \"\n\n # Prefix hidden files with a \"🀫\" emoji\n if path.name.startswith(\".\"):\n icon = \"🀫\"+icon\n\n # Add the file to the tree (with icon prefix)\n tree.add(Text(icon) + text_filename)\n\n # If inplace is False, return the tree... otherwise the Tree object is updated in place\n if not inplace:\n return tree\n return None\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:06.663192Z","iopub.execute_input":"2025-01-08T15:00:06.664127Z","iopub.status.idle":"2025-01-08T15:00:06.689222Z","shell.execute_reply.started":"2025-01-08T15:00:06.664069Z","shell.execute_reply":"2025-01-08T15:00:06.688117Z"},"_kg_hide-output":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def stream_with_styling(chunk_generator: Any, return_completion: bool = True) -> None:\n \"\"\"Streams text content with real-time Markdown styling using Rich library.\n \n Processes chunks of text from a generator and displays them with live Markdown \n formatting, continuously updating the display as new chunks arrive.\n \n Args:\n chunk_generator (Any): \n A generator-like object that yields objects containing text attributes.\n Each chunk object must have a 'text' attribute containing the text to display.\n return_completion (bool, optional):\n Whether we return the consolidated text.\n \n Returns:\n str; The text that was returned by the model \n --OR--\n None;\n \"\"\"\n console = Console()\n with Live(console=console, auto_refresh=False) as live:\n current_text = \"\"\n for chunk in chunk_generator:\n current_text += chunk.text\n live.update(Markdown(current_text))\n live.refresh()\n \n # Final render\n live.update(Markdown(current_text))\n live.refresh()\n time.sleep(0.01)\n \n return current_text if return_completion else None\n\nclass MyUserSecretsClient:\n \"\"\"A wrapper around UserSecretsClient enhancing secret management functionality.\n \n This class extends the base UserSecretsClient by adding error handling and \n environment variable integration capabilities.\n \"\"\"\n \n def __init__(self) -> None:\n \"\"\"Initialize the MyUserSecretsClient with a base UserSecretsClient instance.\n \n Attributes:\n _user_secrets_client (UserSecretsClient):\n The original secret management client object we wrap.\n \"\"\"\n self._user_secrets_client = UserSecretsClient()\n\n def get_secret(self, label: str, default: Any = None) -> Any:\n \"\"\"Retrieve a secret by label with fallback to default value.\n \n Attempts to retrieve a secret value, handling potential errors gracefully\n by returning a default value if the secret cannot be accessed.\n \n Args:\n label (str): \n The identifier of the secret to retrieve.\n default (Any, optional): \n Value to return if the secret cannot be retrieved.\n \n Returns:\n The secret value if successfully retrieved (str), \n Otherwise, the default value specified (by default this is None).\n \n Example:\n >>> client = MyUserSecretsClient()\n >>> api_key = client.get_secret(\"API_KEY\", default=\"fallback_key\")\n \"\"\"\n try:\n return self._user_secrets_client.get_secret(label)\n except BackendError:\n # Expected error when secret doesn't exist\n return default\n except Exception as e:\n # Log unexpected errors while still maintaining graceful fallback\n print(\n f\"Unexpected error while retrieving secret '{label}': {str(e)}. \"\n \"This may indicate a problem beyond the secret not existing.\"\n )\n return default\n\n def secret_to_env(\n self, \n label: str, \n env_var_label: str | None = None, \n default: Any = None\n ) -> None:\n \"\"\"Set an environment variable using a secret value.\n \n Retrieves a secret and sets it as an environment variable. If the secret\n cannot be retrieved, uses the provided default value instead.\n\n Note the provided value will be cast as a string when set as an environment variable.\n \n Args:\n label (str): \n The identifier of the secret to retrieve.\n env_var_label (str, optional): \n The name of the environment variable to set.\n If None, uses the secret's label.\n default (Any, optional): \n Value to use if the secret cannot be retrieved.\n \n Returns:\n None; \n An environment variable will be updated/created with the \n appropriate value.\n \n \n Example:\n >>> client = MyUserSecretsClient()\n >>> client.secret_to_env(\"DB_PASSWORD\", \"DATABASE_PASSWORD\", \"default_pass\")\n \"\"\"\n env_var_label = env_var_label if env_var_label is not None else label\n os.environ[env_var_label] = str(self.get_secret(label, default))\n\n\ndef is_valid_patch_format(patch: str | StringIO) -> bool:\n \"\"\"Validates if the input string represents a valid unified diff patch format.\n\n This function checks if the provided input can be parsed as a valid unified diff\n patch using the unidiff library. It verifies both the syntax and presence of\n actual patch content.\n\n Based on: https://www.kaggle.com/code/sohier/patch-validation-snippet\n\n Args:\n patch (str | StringIO): A string or StringIO object containing the potential patch content.\n The patch should be in unified diff format\n - created by diff -u or similar tools\n - or generated by an LLM as in our case.\n\n Returns:\n bool: True if the input is a valid non-empty patch, False otherwise.\n\n Raises:\n unidiff.UnidiffParseError: Parsing exception\n Exception: Any other problems with the string\n \n Examples:\n >>> is_valid_patch_format(\"--- a/file.txt\\n+++ b/file.txt\\n@@ -1,1 +1,1 @@\\n-old\\n+new\")\n True\n >>> is_valid_patch_format(\"invalid content\")\n False\n \"\"\"\n # All patches must be of type StringIO\n if not isinstance(patch, (str, StringIO)):\n return False\n\n try:\n # Convert string to StringIO if needed for consistent handling\n patch_content = StringIO(patch) if isinstance(patch, str) else patch\n \n # Attempt to parse the patch (may trigger an error on fail which results in False)\n patch_set = unidiff.PatchSet(patch_content)\n \n # Verify the patch contains actual changes\n if len(patch_set) == 0:\n print(\"Patch is either not a valid patch or contains no actual changes!\\n\")\n return False\n \n # Additional validation: check if there are any actual changes\n # - Check if file has any hunks, if so we return True.\n # - Otherwise return False\n return any(len(patched_file) > 0 for patched_file in patch_set)\n\n # Log expected unidiff parsing errors and return False\n except unidiff.UnidiffParseError as e:\n print(f\"Unidiff parsing error (returning False)\\n\\tunidiff.UnidiffParseError: {str(e)}\\n\")\n return False\n\n # Log unexpected errors while still returning False\n except Exception as e:\n print(f\"Unexpected error validating patch (returning False)\\n\\tGeneral Exception: {str(e)}\\n\")\n return False\n\n\ndef calculate_kprize_score(correct: int, incorrect: int, skipped: int) -> float:\n \"\"\"Calculate the SWE-Bench score based on correct, incorrect, and skipped solutions.\n \n Args:\n correct (int): Number of correctly resolved issues (a)\n incorrect (int): Number of failing issues (b)\n skipped (int): Number of skipped issues (c)\n \n Returns:\n float: Score calculated using (a-b)/(a+b+c) formula\n \"\"\"\n # All values must be positive or 0 and all values cannot be 0\n correct, incorrect, skipped = max(0, correct), max(0, incorrect), max(0, skipped)\n if not any([correct, incorrect, skipped]):\n return 0.0\n \n # Calculate score using provided formula (no 0 check required due to above)\n return (correct - incorrect) / (correct + incorrect + skipped)\n\ndef calculate_swebench_score(correct: int, incorrect: int) -> float:\n \"\"\"Calculate the SWE-Bench score based on correct and incorrect solutions.\n \n Args:\n correct (int): Number of correctly resolved issues (a)\n incorrect (int): Number of failing issues (b)\n \n Returns:\n float: Score calculated using a/b formula\n \"\"\"\n # All values must be positive or 0 and all values cannot be 0\n correct, incorrect = max(0, correct), max(0, incorrect)\n if not any([correct, incorrect]):\n return 0.0\n \n # Calculate score using provided formula (no 0 check required due to above)\n return correct / incorrect\n\n \ndef flatten(items: Iterable[Any], as_list: bool = True) -> Generator[Any, None, None] | list[Any]:\n \"\"\"Flattens an iterable of items or nested iterables into a single level sequence.\n\n Strings are treated as atomic elements and will not be flattened.\n Dictionaries are treated as lists where the keys are the elements.\n\n Args:\n items (Iterable[Any]): \n An iterable containing either individual elements or nested iterables.\n as_list (bool, optional): \n If True, returns a list. [DEFAULT BEHAVIOUR]\n If False, returns a generator.\n\n Returns:\n If `as_list` is True, returns a flattened list.\n If `as_list` is False, returns a generator yielding flattened elements.\n\n Examples:\n Basic usage returning a list:\n >>> nested = [1, [2, 3], [4, [5, 6]]]\n >>> flatten(nested)\n [1, 2, 3, 4, 5, 6]\n\n Using generator output:\n >>> nested = ['a', ['b', 2], 'c']\n >>> list(flatten(nested, as_list=False))\n ['a', 'b', 2, 'c']\n\n Handles non-nested iterables:\n >>> simple = [1, 2, 3]\n >>> flatten(simple)\n [1, 2, 3]\n\n Raises:\n TypeError: If input is not an iterable.\n \"\"\"\n def _flatten_generator(items: Iterable[Any]) -> Generator[Any, None, None]:\n # Iterate through each item in the input iterable\n for item in items:\n # Check if item is an iterable but not a string/bytes\n # Strings/bytes are treated as atomic elements\n if isinstance(item, Iterable) and not isinstance(item, (str, bytes)):\n # Recursively flatten nested iterables\n yield from _flatten_generator(item)\n else:\n # Yield non-iterable items directly\n yield item\n\n # Return either a generator or list based on as_list parameter\n generator = _flatten_generator(items)\n return list(generator) if as_list else generator\n \n### TEST IT OUT IF YOU WANT ###\n# rprint(Markdown(\"---\"))\n# # Valid patch\n# if is_valid_patch_format(dev_df.patch[0]):\n# rprint(\"[bold green]SUCCESS[/bold green]\")\n# else:\n# rprint(\"[bold red]FAILURE[/bold red]\")\n# rprint(Markdown(\"---\"))\n\n# # Example showing unidiff parsing error\n# if is_valid_patch_format(dev_df.patch[0].replace(\".py\", \"0.py\", 3)):\n# rprint(\"[bold green]SUCCESS[/bold green]\")\n# else:\n# rprint(\"[bold red]FAILURE[/bold red]\")\n# rprint(Markdown(\"---\"))\n\n# # Example showing complete failure (weird dash)\n# if is_valid_patch_format(dev_df.patch[0].replace(\"-\", \"–\")):\n# rprint(\"[bold green]SUCCESS[/bold green]\")\n# else:\n# rprint(\"[bold red]FAILURE[/bold red]\")\n# rprint(Markdown(\"---\"))\n\n# # Example showing complete failure (random stringf)\n# if is_valid_patch_format(\"invalid content\"):\n# rprint(\"[bold green]SUCCESS[/bold green]\")\n# else:\n# rprint(\"[bold red]FAILURE[/bold red]\")\n# rprint(Markdown(\"---\"))\n\n### TRY IT OUT ###\n# flatten(((1,[2,{3: \"hi\"}]), [\"a b c d\".split(), \"abcd\", \"b\", \"5\", 5, [\"hi\", \"there\"]]), as_list=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:06.69095Z","iopub.execute_input":"2025-01-08T15:00:06.69138Z","iopub.status.idle":"2025-01-08T15:00:06.717771Z","shell.execute_reply.started":"2025-01-08T15:00:06.69133Z","shell.execute_reply":"2025-01-08T15:00:06.716472Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

4.1 PATH DEFINITIONS

\n
\n\n
\n","metadata":{}},{"cell_type":"code","source":"# Rudimentary Paths\nBASE_DIR = \"/kaggle\"\nTMP_DIR = os.path.join(BASE_DIR, \"tmp\")\nWORKING_DIR = os.path.join(BASE_DIR, \"working\")\nINPUT_DIR = os.path.join(BASE_DIR, \"input\")\n\n# Basic Competition Paths\nCOMP_DIR = os.path.join(INPUT_DIR, \"konwinski-prize\")\nCOMP_KAGGLE_EVALUATION_DIR = os.path.join(COMP_DIR, \"kaggle_evaluation\")\nCOMP_KPRIZE_SETUP_DIR = os.path.join(COMP_DIR, \"kprize_setup\")\n\n# Dataset Competition Paths\nCOMP_DATA_ZIP_PATH = os.path.join(COMP_DIR, \"data.a_zip\")\nCOMP_TMP_DIR = os.path.join(TMP_DIR, \"konwinski-prize-alt\")\nCOMP_TMP_DATA_DIR = os.path.join(COMP_TMP_DIR, \"data\")\nCOMP_DATA_PARQUET_PATH = os.path.join(COMP_TMP_DATA_DIR, \"data.parquet\")\nCOMP_CONDA_PACKAGES_DIR = os.path.join(COMP_TMP_DATA_DIR, \"conda_packages\")\nCOMP_PIP_PACKAGES_DIR = os.path.join(COMP_TMP_DATA_DIR, \"pip_packages\")\nCOMP_REPO_CONFIGS_DIR = os.path.join(COMP_TMP_DATA_DIR, \"repo_configs\")\nCOMP_REPOS_DIR = os.path.join(COMP_TMP_DATA_DIR, \"repos\")\n\n# SWE Dataset Paths ... https://huggingface.co/datasets/...\nHF_SWE_BENCH_PROVIDER = \"princeton-nlp\"\nHF_SWE_BENCH_PATH = os.path.join(HF_SWE_BENCH_PROVIDER, \"SWE-bench\")\nHF_SWE_BENCH_LITE_PATH = os.path.join(HF_SWE_BENCH_PROVIDER, \"SWE-bench_Lite\")\nHF_SWE_BENCH_VERIFIED_PATH = os.path.join(HF_SWE_BENCH_PROVIDER, \"SWE-bench_Verified\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:06.720445Z","iopub.execute_input":"2025-01-08T15:00:06.720861Z","iopub.status.idle":"2025-01-08T15:00:06.742599Z","shell.execute_reply.started":"2025-01-08T15:00:06.720824Z","shell.execute_reply":"2025-01-08T15:00:06.741309Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

4.2 SECRET/ENVIRONMENT LOADING/SETUP

\n
\n\n
","metadata":{}},{"cell_type":"code","source":"USE_KAGGLE_SECRETS = True\nSECRETS_TO_USE = [\"GEMINI_API_KEY\", \"HUGGINGFACE_TOKEN\", \"OPENAI_API_KEY\"]\n\nif USE_KAGGLE_SECRETS:\n from kaggle_secrets import UserSecretsClient, BackendError\n user_secrets = MyUserSecretsClient()\n for secret in SECRETS_TO_USE: \n user_secrets.secret_to_env(secret)\nelse:\n from dotenv import load_dotenv, find_dotenv\n _ = load_dotenv(find_dotenv())\n\n# Check if env is loaded\nfor secret in SECRETS_TO_USE:\n if os.getenv(secret):\n print(f\"βœ… The {repr(secret)} value HAS been set.\")\n else:\n print(f\"❌ The {repr(secret)} value HAS NOT set.\")\n\nclient = genai.Client(api_key=os.getenv(\"GEMINI_API_KEY\"))\n\nrprint(\"\\n\\n[bold cyan]TESTING OUR GEMINI AUTHENTICATION WITH A MESSAGE TO GEMINI FLASH![/bold cyan]\\n\")\nDEMO_MESSAGE = \"Tell me about Andy Konwinski the Computer scientist\"\n\nstream_with_styling(client.models.generate_content_stream(model=\"gemini-2.0-flash-exp\", contents=DEMO_MESSAGE))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:06.744138Z","iopub.execute_input":"2025-01-08T15:00:06.744497Z","iopub.status.idle":"2025-01-08T15:00:14.314487Z","shell.execute_reply.started":"2025-01-08T15:00:06.744464Z","shell.execute_reply":"2025-01-08T15:00:14.313448Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

4.3 LOAD THE SWE-LITE DATASET

\n
\n\n
\n\nThis dataset is a slim version of the full SWE-Bench dataset and can be found on HuggingFace\n\nThis dataset contains two splits:\n* **DEV**\n * 27 Examples\n* **TEST**\n * 300 Examples","metadata":{}},{"cell_type":"code","source":"# Load the huggingface dataset\nswe_bench_lite_ds = load_dataset(HF_SWE_BENCH_LITE_PATH)\n\n# Create the sub-dataframes\ndev_df = swe_bench_lite_ds[\"dev\"].to_pandas()\ntest_df = swe_bench_lite_ds[\"test\"].to_pandas()\n\n# Display The Two Dataframes\nrprint(\"\\n[bold blue]DEV SWE BENCH LITE DATAFRAME[/bold blue]\")\ndisplay(dev_df.head(3))\n\nrprint(\"\\n[bold green]TEST SWE BENCH LITE DATAFRAME[/bold green]\")\ndisplay(test_df.head(3))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:14.316014Z","iopub.execute_input":"2025-01-08T15:00:14.316491Z","iopub.status.idle":"2025-01-08T15:00:20.841925Z","shell.execute_reply.started":"2025-01-08T15:00:14.316441Z","shell.execute_reply":"2025-01-08T15:00:20.840839Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

4.4 LOAD THE KPRIZE DATASET

\n
\n\n
\n\nThis is the provided 'training' data. Really these are just a few examples to show us the format and structure that we are to expect. It's on us to gather more data probably.","metadata":{}},{"cell_type":"code","source":"# (1) Handle unzipping the compressed dataset (or skip if already done)\n# Check if the zip file has already been uncompressed\nif not os.path.isfile(COMP_DATA_PARQUET_PATH): \n # Make the directory to unzip to\n os.makedirs(COMP_TMP_DIR, exist_ok=True)\n \n # Open and extract the zip file\n with ZipFile(COMP_DATA_ZIP_PATH, 'r') as zip_ref:\n zip_ref.extractall(COMP_TMP_DIR)\n\n# (2) Load into a dataframe\nkprize_df = pd.read_parquet(COMP_DATA_PARQUET_PATH)\ndisplay(kprize_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:20.843208Z","iopub.execute_input":"2025-01-08T15:00:20.843556Z","iopub.status.idle":"2025-01-08T15:00:24.790155Z","shell.execute_reply.started":"2025-01-08T15:00:20.843523Z","shell.execute_reply":"2025-01-08T15:00:24.78899Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"
\n\n\n\n

5  EXPLORATORY DATA ANALYSIS    ⤒

\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"

5.1 KPRIZE DATA

\n
\n\n
\n\nFrom the competition readme and our earlier investigation we know that the dataframe contains the following:\n\n* **`instance_id`** (string)\n * Unique string identifier for each instance (GitHub issue)\n* **`repo`** (string)\n * The GitHub repository relevant to the issue\n * Also accessible through the evaluation API\n* **`problem_statement`** (string)\n * Textual description of the issue\n * Also accessible through the evaluation API\n* **`patch`** (string)\n * The patch that resolves the issue\n * *Only provided in the train set*\n* **`test_patch`** (string)\n * The patch that resolves the issue\n * *Only provided in the train set*\n* **`pull_number`** (int)\n * The pull request number that resolved the issue\n* **`base_commit`** (string)\n * The commit used as the foundation for the provided repository copy\n* **`issue_numbers`** (int)\n * The original ID number of the GitHub issue\n* **`[PASS_TO_PASS/FAIL_TO_PASS]`** (list)\n * Lists containing unit tests to be executed for this issue\n\n
\n πŸ””   NOTE   πŸ””

Since we only have 5 examples, we won’t get extensive statistical insights, but we can still carry out some basic exploratory data analysis (EDA) to understand each feature, spot any quirks in the data, and think through potential next steps.\n
\n\n\n\n","metadata":{}},{"cell_type":"code","source":"rprint(f\"{kprize_df.shape=}\\n\")\n\nrprint(\"\\nAny Missing Values?\\n\")\nrprint(kprize_df.isnull().sum())\n\nrprint(\"\\nDatatypes?\\n\")\nkprize_df.info(show_counts=True)\n\nrprint(\"\\nRepo Distribution?\\n\")\nrprint(kprize_df['repo'].value_counts())\n# display(kprize_df['repo'].value_counts().plot(kind='bar'))\n\n# Fixes can reference more than one GitHub issue.\nrprint(\"\\nNumber of Issues per PR\\n\")\nkprize_df[\"issue_numbers\"].apply(len).value_counts()\n\nkprize_df['problem_statement_length'] = kprize_df['problem_statement'].apply(lambda x: len(x.split()))\nrprint(\"\\nProblem Statement Lengths\\n\")\ndisplay(kprize_df['problem_statement_length'].describe())\n\nrprint(\"\\nPatch Lengths\\n\")\nkprize_df['patch_length'] = kprize_df['patch'].apply(lambda x: len(x))\nkprize_df['test_patch_length'] = kprize_df['test_patch'].apply(lambda x: len(x))\ndisplay(kprize_df[['patch_length', 'test_patch_length']].describe())\n\nrprint(\"\\nTest Counts\\n\")\nkprize_df['PASS_TO_PASS_count'] = kprize_df['PASS_TO_PASS'].apply(len)\nkprize_df['PASS_TO_PASS_count'] = kprize_df['FAIL_TO_PASS'].apply(len)\ndisplay(kprize_df[['PASS_TO_PASS', 'FAIL_TO_PASS', 'PASS_TO_PASS_count', 'PASS_TO_PASS_count']])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-08T15:00:24.791527Z","iopub.execute_input":"2025-01-08T15:00:24.79188Z","iopub.status.idle":"2025-01-08T15:00:24.923819Z","shell.execute_reply.started":"2025-01-08T15:00:24.791846Z","shell.execute_reply":"2025-01-08T15:00:24.9227Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

5.2 SWE-BENCH LITE DATA

\n
\n\n
\n\n\nSWE-bench was designed to provide a diverse set of codebase problems that were verifiable using in-repo unit tests. The full SWE-bench test split comprises 2,294 issue-commit pairs across 12 python repositories.\n
\n
\nSince its release, we've found that for most systems evaluating on SWE-bench, running each instance can take a lot of time and compute. We've also found that SWE-bench can be a particularly difficult benchmark, which is useful for evaluating LMs in the long term, but discouraging for systems trying to make progress in the short term.\n
\n
\nTo remedy these issues, we've released a canonical subset of SWE-bench called SWE-bench Lite. SWE-bench Lite comprises 300 instances from SWE-bench that have been sampled to be more self-contained, with a focus on evaluating functional bug fixes. SWE-bench Lite covers 11 of the original 12 repositories in SWE-bench, with a similar diversity and distribution of repositories as the original. We perform similar filtering on the SWE-bench dev set to provide 23 development instances that can be useful for active development on the SWE-bench task. We recommend future systems evaluating on SWE-bench to report numbers on SWE-bench Lite in lieu of the full SWE-bench set if necessary. You can find the source code for how SWE-bench Lite was created in SWE-bench/swebench/collect/make_lite.\n
\n
\nHere's a list of the general criteria we used to select SWE-bench Lite instances:\n\n\n
\n \nYou can download SWE-bench Lite and its baselines from Hugging Face Datasets:\n\n