{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":28755,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# # This Python 3 environment comes with many helpful analytics libraries installed\n# # It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# # For example, here's several helpful packages to load\n\n# import numpy as np # linear algebra\n# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# # Input data files are available in the read-only \"../input/\" directory\n# # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# # You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n# # Use the kagglehub client library to attach Kaggle resources like competitions, datasets, and models to your session\n# # Learn more about kagglehub: https://github.com/Kaggle/kagglehub/blob/main/README.md\n\n# import kagglehub\n# # kagglehub.dataset_download('<owner>/<dataset-slug>')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-09-29T20:33:56.032059Z","iopub.execute_input":"2026-09-29T20:33:56.032753Z","iopub.status.idle":"2026-09-29T20:33:56.037239Z","shell.execute_reply.started":"2026-09-29T20:33:56.032709Z","shell.execute_reply":"2026-09-29T20:33:56.036267Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# =============================================================================\n#  PIPELINE v13 — MULTI-ARCHITECTURE; NINE FINDINGS, TWO TIERS, THREE INSTITUTIONS, THREE SEEDS\n#\n#  This is the FULL pipeline. It supersedes v8 and embeds preprocessing.py.\n#\n#  ####  WHAT CHANGED FROM v8  ####\n#   1. NINE FINDINGS IN TWO TIERS instead of five.\n#        Tier A (5): atelectasis, cardiomegaly, consolidation, pleural\n#                    effusion, pneumothorax. Labelled by all three sites.\n#        Tier B (4): infiltration, pleural thickening, fibrosis, nodule/mass.\n#                    Labelled compatibly by NIH and VinDr only.\n#      The model has nine output heads. Each site is evaluated only on the\n#      findings it labels: VinDr on nine, CheXpert on five.\n#   2. WHY: Claim 4 (prevalence shift predicts the benefit of the free fix)\n#      was a correlation over only 5 findings, and 3 under the majority-reader\n#      rule. With 9 it has real statistical power, and several tier-B findings\n#      become MORE common at VinDr, so both directions of shift are covered.\n#   3. VinDr is reported twice: \"[9]\" (all findings, for Claims 1 and 4) and\n#      \"[5]\" (tier A only, directly comparable with CheXpert).\n#   4. The annotation-protocol comparison now uses the SAME findings at both\n#      sites. v8 compared VinDr-r2 (3 surviving findings) against CheXpert (5),\n#      which confounded the comparison.\n#   5. NO RE-DECODING. Cached image arrays are re-used; the four new labels\n#      are attached by image ID. Changing the label map never costs another\n#      multi-hour DICOM decode.\n#   6. Two gates: tier A (primary, must pass) and tier B (fibrosis is only\n#      ~1.5% at NIH, so its in-domain estimates are noisier).\n#   7. The crossover budget (Claim 3) is reported PER SEED and never averaged.\n#   8. (v9.1) VinDr and CheXpert both ship train.csv + train/, so datasets are\n#      now identified by their CSV columns, not file names alone.\n#\n#  ####  WHAT CHANGED IN v10  ####\n#   9. Reads CheXpert directly from your Redivis / CheXpert Plus download\n#      notebook's saved output (shard*_*.tar archives, manifest_*.csv files and\n#      the CheXbert label files). Attach that output with \"Add Input\"; several\n#      download runs can be attached at once and are merged automatically.\n#      The original CheXpert release still works too.\n#  10. Labels come from the IMPRESSION section by default, matching the\n#      original CheXpert labeller. Change CHEX_LABELS to \"report\" to use the\n#      whole report instead.\n#  11. CheXpert is split into calibration and evaluation sets BY PATIENT.\n#  12. The saved NIH/VinDr cache is found automatically wherever it is\n#      attached; CACHE_IN is only a first guess.\n#\n#  ####  WHAT CHANGED IN v11 (weak points a reviewer would raise)  ####\n#  13. TRULY LABEL-FREE FIX. The free fix previously took the target disease\n#      rate from labelled images. It is now reported two ways: with the rate\n#      \"known from hospital records\", and with the rate estimated by EM from\n#      the model's predictions alone (Saerens 2002) - zero labels, no records.\n#  14. FAIR COMPETITOR. Temperature scaling can only change sharpness, so it\n#      cannot fix a level error by construction. Platt scaling with labels\n#      (fixes level AND sharpness) is added as the fair label-based rival, in\n#      both the full comparison and the label-budget curves.\n#  15. SIGNIFICANCE. Free vs label-based fixes are compared with a paired\n#      bootstrap over evaluation images. Budget verdicts now also require the\n#      label method to win in at least 75% of draws.\n#  16. ROBUSTNESS. The free fix is re-run with the recorded disease rate off\n#      by -50%, -25%, +25% and +50%.\n#  17. Claim 4 is reported for CheXpert and pooled across both hospitals.\n#  18. FIGURES: level vs sharpness, reliability diagrams, label-budget curves\n#      and the Claim 4 scatter, saved as PNG and PDF in figures/.\n#\n#  ####  WHAT CHANGED IN v13  ####\n#  19. ARCH switch (CONFIG): densenet121 (identical to v11), convnext_tiny,\n#      vit_b_16, raddino_linear (frozen RAD-DINO + linear head, features cached).\n#  20. Image and patient IDs are saved with every prediction array, so later\n#      analyses get patient-level statistics without reconstructing splits.\n#  21. Output files keep the SAME names and keys, so analysis steps 1-8 run\n#      unchanged. Run each architecture in its OWN notebook and attach only that\n#      notebook's output to the analysis notebook.\n#\n#  ####  HOW TO RUN ON KAGGLE  ####\n#      Save Version -> \"Save & Run All (Commit)\" -> Save\n#  Never interactively: idle sessions are killed after ~20 minutes.\n#\n#  REQUIRES: GPU on, and these inputs attached:\n#      nih-chest-xrays/data                              (required)\n#      vinbigdata-chest-xray-abnormalities-detection     (required)\n#      CheXpert small                                    (optional but wanted)\n#      your saved cache dataset                          (optional, saves time)\n#\n#  RUNTIME: with a restored cache, roughly 75 min (3 seeds x ~20 min training\n#  plus analysis). Without a cache, add ~40 min per uncached dataset.\n# =============================================================================\n\nimport os, sys, glob, json, time, shutil, warnings, importlib, base64\nimport numpy as np\nimport pandas as pd\n\nwarnings.filterwarnings(\"ignore\")\nT0 = time.time()\nprint(\"PIPELINE v13 | multi-architecture | 3 hospitals | started\", time.strftime(\"%Y-%m-%d %H:%M\"), flush=True)\nel = lambda: f\"[{(time.time()-T0)/60:6.1f} min]\"\n\n\ndef banner(t):\n    print(\"\\n\" + \"=\" * 94)\n    print(f\"{el()}  {t}\")\n    print(\"=\" * 94, flush=True)\n\n\n# ============================================================ CONFIG\nW, CACHE, LIB = \"/kaggle/working\", \"/kaggle/working/cache\", \"/kaggle/working/lib\"\nCACHE_IN = \"/kaggle/input/datasets/nabeelarshad1/cxr-cache-224-vindr\"\nSEEDS = [42, 43, 44]          # three seeds; calibration is seed-sensitive\nN_NIH = 40000\nN_CHEX = 40000                # uses all downloaded images up to this\nCHEX_LABELS = \"impression\"    # impression | report | findings\nCHEX_IN = \"/kaggle/input/notebooks/nabeelarshad1/chexpertdataset\"  # searched first\n# ---- v13: MODEL ARCHITECTURE -------------------------------------------------\n#   \"densenet121\"    DenseNet-121, ImageNet init. IDENTICAL to v11 (reproduces it).\n#   \"convnext_tiny\"  ConvNeXt-Tiny, ImageNet init, full fine-tune. No pretraining\n#                    overlap with any test hospital: valid at VinDr AND CheXpert.\n#   \"vit_b_16\"       ViT-B/16, ImageNet init, full fine-tune. No overlap either.\n#   \"raddino_linear\" RAD-DINO (microsoft/rad-dino) FROZEN + linear head. Its\n#                    pretraining included NIH and CheXpert (not VinDr): results\n#                    are valid at VinDr; CheXpert must be reported with that caveat.\n#                    Needs Internet ON (downloads the model from Hugging Face).\nARCH = \"convnext_tiny\"\nEPOCHS = 6                    # pilot peaked at epoch 3; best checkpoint kept\nBATCH = 64\nLR = 1e-4\n_ARCH_CFG = {\"densenet121\": dict(lr=1e-4, epochs=6, batch=64),\n             \"convnext_tiny\": dict(lr=1e-4, epochs=6, batch=64),\n             \"vit_b_16\": dict(lr=3e-5, epochs=6, batch=64),\n             \"raddino_linear\": dict(lr=1e-3, epochs=30, batch=256)}\nassert ARCH in _ARCH_CFG, f\"unknown ARCH {ARCH}\"\nLR, EPOCHS, BATCH = _ARCH_CFG[ARCH][\"lr\"], _ARCH_CFG[ARCH][\"epochs\"], _ARCH_CFG[ARCH][\"batch\"]\nFROZEN = ARCH == \"raddino_linear\"      # features computed once, linear head trained per seed\n# ---- v13: SMOKE TEST -----------------------------------------------------------\n# True = quick check that an architecture runs end to end (1 seed, 1 epoch, 2,000\n# training images). Outputs are named logits_SMOKE_s*.npz so the analysis steps\n# never pick them up. Set back to False for the real run.\nSMOKE_TEST = False\nif SMOKE_TEST:\n    SEEDS, EPOCHS = SEEDS[:1], 1\nprint(f\"  ARCH = {ARCH} | lr {LR} | epochs {EPOCHS} | batch {BATCH} | seeds {SEEDS} | smoke test {SMOKE_TEST}\", flush=True)\nCAL_FRAC = 0.35               # external calibration pool; rest is evaluation\nMIN_POS_EVAL = 50\nBUDGETS = [25, 50, 100, 200, 500, 1000, 2000]\nREPEATS = 20\nN_BOOT = 400\nN_BOOT_DIFF = 200             # paired bootstrap, free vs label-based\nPREV_ERRORS = [0.5, 0.75, 1.25, 1.5]  # recorded-rate error tested\nos.makedirs(CACHE, exist_ok=True)\nos.makedirs(LIB, exist_ok=True)\n\n# ============================================================ SECTION 0\nbanner(\"SECTION 0  Write and import the preprocessing module\")\n\n# The preprocessing module is written to disk so ProcessPoolExecutor can pickle\n# its workers, and so the same code is importable outside the notebook.\n# The preprocessing module is embedded as base64 so its own nested worker\n# source cannot collide with this file's quoting. It is decoded to disk at\n# runtime, which is also required for ProcessPoolExecutor to pickle workers.\nPREPROC_B64 = \"\"\"\nIyA9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT09PQojICBQUkVQUk9DRVNTSU5HIE1PRFVMRSDigJQgdGhyZWUt\naW5zdGl0dXRpb24gY2hlc3QgcmFkaW9ncmFwaCBoYXJtb25pc2F0aW9uCiMgIE5JSCBDaGVzdFgt\ncmF5MTQgIHwgIFZpbkRyLUNYUiAgfCAgQ2hlWHBlcnQKIwojICBXSEFUIFRISVMgSVMgRk9SCiMg\nIFRoZSBwaWxvdCBoYW5kbGVkIHR3byBkYXRhc2V0cyB3aXRoIGFkLWhvYyBjb2RlLiBBZGRpbmcg\nQ2hlWHBlcnQgbWVhbnMgdGhyZWUKIyAgbGFiZWwgdm9jYWJ1bGFyaWVzLCB0aHJlZSBmaWxlIGZv\ncm1hdHMgYW5kIHRocmVlIGFubm90YXRpb24gY29udmVudGlvbnMuIFRoaXMKIyAgbW9kdWxlIHB1\ndHMgYWxsIG9mIHRoYXQgaW4gb25lIHBsYWNlIHdpdGggZXhwbGljaXQsIGF1ZGl0YWJsZSBydWxl\ncywgYmVjYXVzZQojICBldmVyeSBsYWJlbC1tYXBwaW5nIGRlY2lzaW9uIGhhcyB0byBiZSBkZWZl\nbnNpYmxlIGluIHRoZSB0aGVzaXMuCiMKIyAgREVTSUdOIFBSSU5DSVBMRVMgKGVhY2ggZXhpc3Rz\nIGJlY2F1c2Ugc29tZXRoaW5nIHdlbnQgd3JvbmcgaW4gdGhlIHBpbG90KQojICAgMS4gUFJFVkFM\nRU5DRSBJUyBTQUNSRUQuIENhbGlicmF0aW9uIGlzIGRlZmluZWQgYWdhaW5zdCB0aGUgYmFzZSBy\nYXRlLCBzbwojICAgICAgYW55IHNhbXBsaW5nIHRoYXQgZGlzdG9ydHMgcHJldmFsZW5jZSBpbnZh\nbGlkYXRlcyB0aGUgc3R1ZHkuIEFuIGVhcmx5CiMgICAgICBwaWxvdCB2ZXJzaW9uIGZpbHRlcmVk\nIHRvIGltYWdlcyBXSVRIIGZpbmRpbmdzIGFuZCBpbmZsYXRlZCBldmVyeQojICAgICAgcHJldmFs\nZW5jZSAzLjh4LiBIYXJkIGFzc2VydGlvbnMgbm93IGJsb2NrIHRoaXMuCiMgICAyLiBMQUJFTCBN\nQVBQSU5HUyBBUkUgREFUQSwgTk9UIENPREUgQ09NTUVOVFMuIEV2ZXJ5IG1hcHBpbmcgaXMgZGVj\nbGFyZWQgaW4KIyAgICAgIExBQkVMX01BUCBiZWxvdyBhbmQgZXhwb3J0ZWQgdG8gQ1NWIGZvciB0\naGUgdGhlc2lzIGFwcGVuZGl4LgojICAgMy4gU1BMSVQgQlkgUEFUSUVOVCwgTkVWRVIgQlkgSU1B\nR0UuIFBhdGllbnRzIGNvbnRyaWJ1dGUgbXVsdGlwbGUgc3R1ZGllczsKIyAgICAgIGltYWdlLWxl\ndmVsIHNwbGl0dGluZyBsZWFrcyBhbmQgaW5mbGF0ZXMgdGhlIGluLWRvbWFpbiByZWZlcmVuY2Uu\nCiMgICA0LiBDQUNIRSBXSVRIIEEgVkVSU0lPTiBUQUcuIEFuIHVudGFnZ2VkIGNhY2hlIG9mIHVu\na25vd24gcHJvdmVuYW5jZSBpcwojICAgICAgcmVidWlsdCByYXRoZXIgdGhhbiB0cnVzdGVkLgoj\nICAgNS4gRkFJTCBMT1VELiBTaWxlbnQgZmFsbGJhY2tzIGNvc3QgdHdvIGZ1bGwgZGF5cyBpbiB0\naGUgcGlsb3QuCiMKIyAgVVNBR0UKIyAgICAgIGltcG9ydCBwcmVwcm9jZXNzaW5nIGFzIHBwCiMg\nICAgICBjZmcgID0gcHAuQ29uZmlnKHNlZWQ9NDIsIG5fbmloPTQwMDAwKQojICAgICAgbmloICA9\nIHBwLmxvYWRfbmloKGNmZyk7ICAgdmluID0gcHAubG9hZF92aW5kcihjZmcpCiMgICAgICBjaGV4\nID0gcHAubG9hZF9jaGV4cGVydChjZmcpCiMgICAgICBwcC5leHBvcnRfbGFiZWxfbWFwX3RhYmxl\nKGYie2NmZy5vdXR9L2xhYmVsX21hcHBpbmcuY3N2IikKIyAgICAgIHBwLnByZXZhbGVuY2VfcmVw\nb3J0KFtuaWgsIHZpbiwgY2hleF0sIGYie2NmZy5vdXR9L3ByZXZhbGVuY2UuY3N2IikKIyAgICAg\nIFgsIGlkeCA9IHBwLmJ1aWxkX2NhY2hlKG5paCwgY2ZnKQojICAgICAgc3BsaXRzID0gcHAucGF0\naWVudF9zcGxpdHMoaWR4LCBjZmcpCiMgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KCmZyb20gX19mdXR1\ncmVfXyBpbXBvcnQgYW5ub3RhdGlvbnMKCmltcG9ydCBvcwppbXBvcnQgcmUKaW1wb3J0IGdsb2IK\naW1wb3J0IGpzb24KaW1wb3J0IHRpbWUKaW1wb3J0IHNodXRpbAppbXBvcnQgdGFyZmlsZQpmcm9t\nIGRhdGFjbGFzc2VzIGltcG9ydCBkYXRhY2xhc3MsIGZpZWxkCmZyb20gY29uY3VycmVudC5mdXR1\ncmVzIGltcG9ydCBQcm9jZXNzUG9vbEV4ZWN1dG9yCgppbXBvcnQgbnVtcHkgYXMgbnAKaW1wb3J0\nIHBhbmRhcyBhcyBwZAoKIyA9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT0gQ09ORklHCgojOiBUaGUgZml2ZSBwYXRob2xvZ2llcyB0aGF0\nIHN1cnZpdmUgdGhlIHNoYXJlZC1sYWJlbCBmaWx0ZXIgYWNyb3NzIGFsbCB0aHJlZQojOiBzb3Vy\nY2VzLiBQbmV1bW9uaWEgaXMgZGVsaWJlcmF0ZWx5IGV4Y2x1ZGVkOiB0b28gZmV3IHBvc2l0aXZl\ncyBpbiB0aGUKIzogZXhwZXJ0LWxhYmVsbGVkIHRhcmdldCBzcGxpdHMgdG8gZXN0aW1hdGUgY2Fs\naWJyYXRpb24gc3RhYmx5LiBUaGUgZXhjbHVzaW9uCiM6IGlzIHJlcG9ydGVkIHJhdGhlciB0aGFu\nIGxlZnQgdW5leHBsYWluZWQuClNIQVJFRCA9IFsiQXRlbGVjdGFzaXMiLCAiQ2FyZGlvbWVnYWx5\nIiwgIkNvbnNvbGlkYXRpb24iLAogICAgICAgICAgIlBsZXVyYWwgZWZmdXNpb24iLCAiUG5ldW1v\ndGhvcmF4Il0KCiM6IFRJRVIgQiDigJQgZm91ciBmdXJ0aGVyIGZpbmRpbmdzIGxhYmVsbGVkIGNv\nbXBhdGlibHkgYnkgTklIIGFuZCBWaW5EciBidXQgTk9UCiM6IGJ5IENoZVhwZXJ0ICh3aGljaCBo\nYXMgbm8gaW5maWx0cmF0aW9uLCBwbGV1cmFsLXRoaWNrZW5pbmcgb3IgZmlicm9zaXMKIzogbGFi\nZWwsIGFuZCBmb2xkcyBub2R1bGVzIGFuZCBtYXNzZXMgaW50byBhIGJyb2FkICJMdW5nIExlc2lv\nbiIpLiBUaGVzZSBhcmUKIzogdXNlZCBPTkxZIGZvciB0aGUgTklIIC0+IFZpbkRyIGNvbXBhcmlz\nb24uIFRoZWlyIHB1cnBvc2UgaXMgc3RhdGlzdGljYWw6CiM6IHRoZXkgcmFpc2UgdGhlIG51bWJl\nciBvZiBmaW5kaW5ncyBhdmFpbGFibGUgZm9yIHRoZSBwcmV2YWxlbmNlLXNoaWZ0CiM6IGFuYWx5\nc2lzIChDbGFpbSA0KSBmcm9tIDUgdG8gOSwgYW5kIHNldmVyYWwgb2YgdGhlbSBiZWNvbWUgTU9S\nRSBjb21tb24gYXQKIzogVmluRHIsIHNvIGJvdGggZGlyZWN0aW9ucyBvZiBwcmV2YWxlbmNlIHNo\naWZ0IGFyZSBiZXR0ZXIgcmVwcmVzZW50ZWQuCkVYVEVOREVEID0gWyJJbmZpbHRyYXRpb24iLCAi\nUGxldXJhbCB0aGlja2VuaW5nIiwgIkZpYnJvc2lzIiwgIk5vZHVsZS9NYXNzIl0KCiM6IFRoZSBt\nb2RlbCBpcyB0cmFpbmVkIG9uIGFsbCBuaW5lLiBFYWNoIGV4dGVybmFsIHNpdGUgaXMgZXZhbHVh\ndGVkIG9uIHRoZQojOiBzdWJzZXQgaXQgbGFiZWxzIGNvbXBhdGlibHk6IFZpbkRyIG9uIGFsbCBu\naW5lLCBDaGVYcGVydCBvbiBTSEFSRUQgb25seS4KQUxMX0ZJTkRJTkdTID0gU0hBUkVEICsgRVhU\nRU5ERUQKU0lURV9GSU5ESU5HUyA9IHsibmloIjogQUxMX0ZJTkRJTkdTLCAidmluZHIiOiBBTExf\nRklORElOR1MsCiAgICAgICAgICAgICAgICAgImNoZXhwZXJ0IjogU0hBUkVEfQoKIzogRVZFUlkg\nbGFiZWwtbWFwcGluZyBkZWNpc2lvbiwgaW4gb25lIGF1ZGl0YWJsZSBwbGFjZS4gRXhwb3J0ZWQg\ndG8gQ1NWIGZvcgojOiB0aGUgdGhlc2lzIGFwcGVuZGl4IHNvIGEgcmV2aWV3ZXIgY2FuIGNoZWNr\nIGVhY2ggb25lLgojOgojOiBBIG5vdGUgb24gd2h5IHRoaXMgbWF0dGVycyBtb3JlIHRoYW4gaXQg\nbG9va3M6IHRoZSB0aHJlZSBzb3VyY2VzIGRvIG5vdCBtZWFuCiM6IHRoZSBzYW1lIHRoaW5nIGJ5\nIHRoZXNlIHdvcmRzLiBOSUgncyBsYWJlbGxlciBmbGFncyBhbnkgbWVudGlvbiBvZiBhIHRlcm0g\naW4KIzogdGhlIHJlcG9ydCB0ZXh0LiBWaW5EciByZXF1aXJlcyByYWRpb2xvZ2lzdHMgdG8gZHJh\ndyBhIGJvdW5kaW5nIGJveCwgd2l0aAojOiBhZ3JlZW1lbnQgdGhyZXNob2xkcyBhcHBsaWVkIGRv\nd25zdHJlYW0uIENoZVhwZXJ0IGFwcGxpZXMgYSBydWxlLWJhc2VkCiM6IGxhYmVsbGVyIHdpdGgg\nZXhwbGljaXQgbmVnYXRpb24gYW5kIHVuY2VydGFpbnR5IGhhbmRsaW5nLiBJZGVudGljYWwgc3Ry\naW5ncwojOiB0aGVyZWZvcmUgZGVub3RlIGRpZmZlcmVudCBjbGluaWNhbCB0aHJlc2hvbGRzLCB3\naGljaCBpcyBwcmVjaXNlbHkgdGhlCiM6IGFubm90YXRpb24tcHJvdG9jb2wgY29uZm91bmQgdGhl\nIHRocmVlLXNpdGUgZGVzaWduIGlzIG1lYW50IHRvIG1lYXN1cmUuCkxBQkVMX01BUCA9IHsKICAg\nICJBdGVsZWN0YXNpcyI6IHsKICAgICAgICAibmloIjogWyJBdGVsZWN0YXNpcyJdLAogICAgICAg\nICJ2aW5kciI6IFsiQXRlbGVjdGFzaXMiXSwKICAgICAgICAiY2hleHBlcnQiOiBbIkF0ZWxlY3Rh\nc2lzIl0sCiAgICAgICAgIm5vdGUiOiAiRGlyZWN0IHRlcm0gbWF0Y2ggaW4gYWxsIHRocmVlIHNv\ndXJjZXMuIiwKICAgIH0sCiAgICAiQ2FyZGlvbWVnYWx5IjogewogICAgICAgICJuaWgiOiBbIkNh\ncmRpb21lZ2FseSJdLAogICAgICAgICJ2aW5kciI6IFsiQ2FyZGlvbWVnYWx5Il0sCiAgICAgICAg\nImNoZXhwZXJ0IjogWyJDYXJkaW9tZWdhbHkiXSwKICAgICAgICAibm90ZSI6ICJEaXJlY3QgbWF0\nY2guIE1vc3Qgb2JqZWN0aXZlbHkgZGVmaW5lZCBvZiB0aGUgZml2ZSAiCiAgICAgICAgICAgICAg\nICAiKGNhcmRpb3Rob3JhY2ljIHJhdGlvKSwgYW5kIGNvcnJlc3BvbmRpbmdseSB0aGUgaGlnaGVz\ndCAiCiAgICAgICAgICAgICAgICAiaW50ZXItcmVhZGVyIGFncmVlbWVudCBpbiBWaW5EciBhdCA1\nNi40IHBlciBjZW50IHVuYW5pbW91cy4iLAogICAgfSwKICAgICJDb25zb2xpZGF0aW9uIjogewog\nICAgICAgICJuaWgiOiBbIkNvbnNvbGlkYXRpb24iXSwKICAgICAgICAidmluZHIiOiBbIkNvbnNv\nbGlkYXRpb24iXSwKICAgICAgICAiY2hleHBlcnQiOiBbIkNvbnNvbGlkYXRpb24iXSwKICAgICAg\nICAibm90ZSI6ICJEaXJlY3QgbWF0Y2guIEhpZ2hseSBjb250ZXN0ZWQgaW4gVmluRHI6IDY1Ljcg\ncGVyIGNlbnQgb2YgIgogICAgICAgICAgICAgICAgInBvc2l0aXZlcyBmbGFnZ2VkIGJ5IGEgc2lu\nZ2xlIHJlYWRlciBvZiB0aHJlZS4iLAogICAgfSwKICAgICJQbGV1cmFsIGVmZnVzaW9uIjogewog\nICAgICAgICJuaWgiOiBbIkVmZnVzaW9uIl0sICAgICAgICAgICMgTklIIG9taXRzIHRoZSAiUGxl\ndXJhbCIgcXVhbGlmaWVyCiAgICAgICAgInZpbmRyIjogWyJQbGV1cmFsIGVmZnVzaW9uIl0sCiAg\nICAgICAgImNoZXhwZXJ0IjogWyJQbGV1cmFsIEVmZnVzaW9uIl0sICAgIyBub3RlIHRoZSBjYXBp\ndGFsIEUKICAgICAgICAibm90ZSI6ICJOSUggdXNlcyB0aGUgYmFyZSB0ZXJtICdFZmZ1c2lvbic7\nIENoZVhwZXJ0IGNhcGl0YWxpc2VzICIKICAgICAgICAgICAgICAgICInRWZmdXNpb24nLiBTYW1l\nIGZpbmRpbmcsIHRocmVlIHNwZWxsaW5ncy4iLAogICAgfSwKICAgICJQbmV1bW90aG9yYXgiOiB7\nCiAgICAgICAgIm5paCI6IFsiUG5ldW1vdGhvcmF4Il0sCiAgICAgICAgInZpbmRyIjogWyJQbmV1\nbW90aG9yYXgiXSwKICAgICAgICAiY2hleHBlcnQiOiBbIlBuZXVtb3Rob3JheCJdLAogICAgICAg\nICJub3RlIjogIkRpcmVjdCBtYXRjaC4gUmFyZXN0IG9mIHRoZSBmaXZlIGF0IHRoZSB0YXJnZXQg\nc2l0ZS4iLAogICAgfSwKICAgICMgLS0tLS0tLS0tLS0tLS0tLSBUSUVSIEI6IE5JSCBhbmQgVmlu\nRHIgb25seSAtLS0tLS0tLS0tLS0tLS0tCiAgICAiSW5maWx0cmF0aW9uIjogewogICAgICAgICJu\naWgiOiBbIkluZmlsdHJhdGlvbiJdLAogICAgICAgICJ2aW5kciI6IFsiSW5maWx0cmF0aW9uIl0s\nCiAgICAgICAgImNoZXhwZXJ0IjogW10sCiAgICAgICAgIm5vdGUiOiAiTklIIC0+IFZpbkRyIG9u\nbHkuIERpcmVjdCB0ZXJtIG1hdGNoLiBDYXV0aW9uOiBOSUgncyAiCiAgICAgICAgICAgICAgICAi\nJ0luZmlsdHJhdGlvbicgaXMgd2lkZWx5IHJlZ2FyZGVkIGFzIGl0cyBsZWFzdCByZWxpYWJsZSAi\nCiAgICAgICAgICAgICAgICAibGFiZWwsIGFuZCBWaW5EcidzIHNlcGFyYXRlICdMdW5nIE9wYWNp\ndHknIGNsYXNzIG92ZXJsYXBzIGl0ICIKICAgICAgICAgICAgICAgICJjbGluaWNhbGx5LiBMdW5n\nIE9wYWNpdHkgaXMgZGVsaWJlcmF0ZWx5IE5PVCBtZXJnZWQgaW4sIHNvICIKICAgICAgICAgICAg\nICAgICJ0aGUgbWFwcGluZyBzdGF5cyBhIGxpa2UtZm9yLWxpa2UgdGVybSBtYXRjaC4iLAogICAg\nfSwKICAgICJQbGV1cmFsIHRoaWNrZW5pbmciOiB7CiAgICAgICAgIm5paCI6IFsiUGxldXJhbF9U\naGlja2VuaW5nIl0sICAgICAgICMgTklIIHVzZXMgYW4gdW5kZXJzY29yZQogICAgICAgICJ2aW5k\nciI6IFsiUGxldXJhbCB0aGlja2VuaW5nIl0sCiAgICAgICAgImNoZXhwZXJ0IjogW10sCiAgICAg\nICAgIm5vdGUiOiAiTklIIC0+IFZpbkRyIG9ubHkuIE5JSCBzcGVsbHMgaXQgJ1BsZXVyYWxfVGhp\nY2tlbmluZycgd2l0aCBhbiAiCiAgICAgICAgICAgICAgICAidW5kZXJzY29yZTsgZXhhY3QtdG9r\nZW4gbWF0Y2hpbmcgcmVxdWlyZXMgdGhlIGV4YWN0IGZvcm0uIiwKICAgIH0sCiAgICAiRmlicm9z\naXMiOiB7CiAgICAgICAgIm5paCI6IFsiRmlicm9zaXMiXSwKICAgICAgICAidmluZHIiOiBbIlB1\nbG1vbmFyeSBmaWJyb3NpcyJdLAogICAgICAgICJjaGV4cGVydCI6IFtdLAogICAgICAgICJub3Rl\nIjogIk5JSCAtPiBWaW5EciBvbmx5LiBWaW5EcidzIHNlcGFyYXRlICdJTEQnIChpbnRlcnN0aXRp\nYWwgbHVuZyAiCiAgICAgICAgICAgICAgICAiZGlzZWFzZSkgY2xhc3MgaXMgcmVsYXRlZCBidXQg\nYnJvYWRlciwgYW5kIGlzIE5PVCBtZXJnZWQgaW4uIiwKICAgIH0sCiAgICAiTm9kdWxlL01hc3Mi\nOiB7CiAgICAgICAgIm5paCI6IFsiTm9kdWxlIiwgIk1hc3MiXSwgICAgICAgICAgICAjIHVuaW9u\nIG9mIHR3byBOSUggbGFiZWxzCiAgICAgICAgInZpbmRyIjogWyJOb2R1bGUvTWFzcyJdLAogICAg\nICAgICJjaGV4cGVydCI6IFtdLAogICAgICAgICJub3RlIjogIk5JSCAtPiBWaW5EciBvbmx5LiBO\nSUggbGFiZWxzIG5vZHVsZXMgKDwzIGNtKSBhbmQgbWFzc2VzICIKICAgICAgICAgICAgICAgICIo\nPjMgY20pIHNlcGFyYXRlbHk7IFZpbkRyIGNvbWJpbmVzIHRoZW0uIFRoZSBOSUggbGFiZWwgaXMg\ndGhlICIKICAgICAgICAgICAgICAgICJ1bmlvbiBvZiBib3RoLCBzbyB0aGUgdHdvIGRlZmluaXRp\nb25zIG1hdGNoLiIsCiAgICB9LAp9CgpFWENMVURFRCA9IHsKICAgICJQbmV1bW9uaWEiOiAiUHJl\nc2VudCBpbiBhbGwgdGhyZWUgc291cmNlcyBidXQgeWllbGRzIHRvbyBmZXcgcG9zaXRpdmVzICIK\nICAgICAgICAgICAgICAgICAiaW4gZXhwZXJ0LWxhYmVsbGVkIHRhcmdldCBzcGxpdHMgZm9yIHN0\nYWJsZSBjYWxpYnJhdGlvbiAiCiAgICAgICAgICAgICAgICAgImVzdGltYXRpb24uIEV4Y2x1ZGVk\nIGEgcHJpb3JpLCBub3QgYWZ0ZXIgc2VlaW5nIHJlc3VsdHMuIiwKfQoKCkBkYXRhY2xhc3MKY2xh\nc3MgQ29uZmlnOgogICAgc2VlZDogaW50ID0gNDIKICAgIGltZ19zaXplOiBpbnQgPSAyMjQKICAg\nIG5fbmloOiBpbnQgPSA0MDAwMCAgICAgICAgICAjIHBsYWluIHJhbmRvbSBzYW1wbGU7IHByZXZh\nbGVuY2UgcHJlc2VydmVkCiAgICBuX2NoZXhwZXJ0OiBpbnQgPSA0MDAwMAogICAgdmluZHJfbWlu\nX3JlYWRlcnM6IGludCA9IDIgICMgbWFqb3JpdHkgb2YgdGhyZWU7IHIxIHZhcmlhbnQgYWxzbyBj\nb21wdXRlZAogICAgY2hleHBlcnRfdW5jZXJ0YWluOiBzdHIgPSAiemVybyIgICAjICd6ZXJvJyB8\nICdvbmVzJyB8ICdkcm9wJwogICAgY2hleHBlcnRfdmlldzogc3RyID0gImZyb250YWwiICAgICAj\nIGZyb250YWwgb25seSwgdG8gbWF0Y2ggTklIIGFuZCBWaW5EcgogICAgIyBXaGljaCByZXBvcnQg\nc2VjdGlvbiB0aGUgQ2hlWHBlcnQgbGFiZWxzIGNvbWUgZnJvbSwgZm9yIHRoZSBSZWRpdmlzIC8K\nICAgICMgQ2hlWHBlcnQgUGx1cyBkb3dubG9hZC4gJ2ltcHJlc3Npb24nIG1hdGNoZXMgdGhlIG9y\naWdpbmFsIENoZVhwZXJ0CiAgICAjIGxhYmVsbGVyLCB3aGljaCBsYWJlbGxlZCB0aGUgaW1wcmVz\nc2lvbiBzZWN0aW9uLgogICAgY2hleHBlcnRfbGFiZWxfc291cmNlOiBzdHIgPSAiaW1wcmVzc2lv\nbiIgICAjIGltcHJlc3Npb258cmVwb3J0fGZpbmRpbmdzCiAgICBjaGV4cGVydF9leHRyYWN0OiBz\ndHIgPSAiIiAgICAgICAgICMgd2hlcmUgVEFSIGltYWdlcyBhcmUgdW5wYWNrZWQ7IGF1dG8KICAg\nIGNoZXhwZXJ0X2luOiBzdHIgPSAiIiAgICAgICAgICAgICAgIyBhdHRhY2hlZCBkb3dubG9hZCBv\ndXRwdXQ7IHNlYXJjaGVkIGZpcnN0CiAgICBjYWNoZTogc3RyID0gIi9rYWdnbGUvd29ya2luZy9j\nYWNoZSIKICAgIG91dDogc3RyID0gIi9rYWdnbGUvd29ya2luZyIKICAgIGxpYjogc3RyID0gIi9r\nYWdnbGUvd29ya2luZy9saWIiCiAgICBjYWNoZV9pbjogc3RyIHwgTm9uZSA9IE5vbmUKICAgIHBy\nZXZhbGVuY2VfdG9sOiBmbG9hdCA9IDAuMTUgICAgICAgIyBzYW1wbGluZyBtdXN0IHByZXNlcnZl\nIHJhdGUgd2l0aGluIDE1JQogICAgdGFnczogZGljdCA9IGZpZWxkKGRlZmF1bHRfZmFjdG9yeT1s\nYW1iZGE6IHsKICAgICAgICAibmloIjogIm5paF92NF9wbGFpbnJhbmRvbSIsICJ2aW5kciI6ICJ2\naW5kcl92MSIsCiAgICAgICAgImNoZXhwZXJ0IjogImNoZXhfdjJfZnJvbnRhbCJ9KQoKCiMgPT09\nPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nIERJU0NPVkVSWQoKZGVmIF9jc3ZfaGVhZGVyKHBhdGg6IHN0cikgLT4gc2V0OgogICAgIiIiQ29s\ndW1uIG5hbWVzIG9mIGEgQ1NWLCByZWFkIGNoZWFwbHkgZnJvbSBpdHMgZmlyc3QgbGluZS4iIiIK\nICAgIHRyeToKICAgICAgICB3aXRoIG9wZW4ocGF0aCwgInIiLCBlcnJvcnM9Imlnbm9yZSIpIGFz\nIGZoOgogICAgICAgICAgICByZXR1cm4ge2Muc3RyaXAoKS5zdHJpcCgnIicpIGZvciBjIGluIGZo\nLnJlYWRsaW5lKCkuc3BsaXQoIiwiKX0KICAgIGV4Y2VwdCBPU0Vycm9yOgogICAgICAgIHJldHVy\nbiBzZXQoKQoKCmRlZiBmaW5kX3Jvb3QobWFya2VyczogbGlzdFtzdHJdLCBsYWJlbDogc3RyLAog\nICAgICAgICAgICAgIG5lZWRfY29sczogdHVwbGUgPSAoKSwgY3N2X25hbWU6IHN0ciA9ICJ0cmFp\nbi5jc3YiKSAtPiBzdHI6CiAgICAiIiJMb2NhdGUgYSBkYXRhc2V0IHJvb3QgYnkgY2hhcmFjdGVy\naXN0aWMgZmlsZXMuIEZhaWxzIGxvdWRseS4KCiAgICBuZWVkX2NvbHMgZGlzYW1iaWd1YXRlcyBk\nYXRhc2V0cyB0aGF0IHNoYXJlIGZpbGUgbmFtZXMuIFZpbkRyLUNYUiBhbmQKICAgIENoZVhwZXJ0\nIEJPVEggc2hpcCBhIHRyYWluLmNzdiBhbmQgYSB0cmFpbi8gZm9sZGVyLCBzbyBmaWxlIG5hbWVz\nIGFsb25lCiAgICBjYW4gbWF0Y2ggdGhlIHdyb25nIGRhdGFzZXQuIENoZWNraW5nIHRoZSBDU1Yn\ncyBjb2x1bW5zIG1ha2VzIHRoZSBtYXRjaAogICAgdW5hbWJpZ3VvdXM6IFZpbkRyIGhhcyBjbGFz\nc19uYW1lIGFuZCByYWRfaWQsIENoZVhwZXJ0IGhhcyBQYXRoLgogICAgIiIiCiAgICBmb3IgYyBp\nbiAoc29ydGVkKGdsb2IuZ2xvYigiL2thZ2dsZS9pbnB1dC8qIikpCiAgICAgICAgICAgICAgKyBz\nb3J0ZWQoZ2xvYi5nbG9iKCIva2FnZ2xlL2lucHV0LyovKiIpKQogICAgICAgICAgICAgICsgc29y\ndGVkKGdsb2IuZ2xvYigiL2thZ2dsZS9pbnB1dC8qLyovKiIpKQogICAgICAgICAgICAgICsgc29y\ndGVkKGdsb2IuZ2xvYigiL2thZ2dsZS9pbnB1dC8qLyovKi8qIikpKToKICAgICAgICBpZiBub3Qg\nKG9zLnBhdGguaXNkaXIoYykgYW5kIGFsbCgKICAgICAgICAgICAgICAgIG9zLnBhdGguZXhpc3Rz\nKG9zLnBhdGguam9pbihjLCBtKSkgZm9yIG0gaW4gbWFya2VycykpOgogICAgICAgICAgICBjb250\naW51ZQogICAgICAgIGlmIG5lZWRfY29sczoKICAgICAgICAgICAgaGRyID0gX2Nzdl9oZWFkZXIo\nb3MucGF0aC5qb2luKGMsIGNzdl9uYW1lKSkKICAgICAgICAgICAgaWYgbm90IGFsbChjb2wgaW4g\naGRyIGZvciBjb2wgaW4gbmVlZF9jb2xzKToKICAgICAgICAgICAgICAgIGNvbnRpbnVlCiAgICAg\nICAgcmV0dXJuIGMKICAgIHJhaXNlIEZpbGVOb3RGb3VuZEVycm9yKAogICAgICAgIGYie2xhYmVs\nfTogbm8gZGlyZWN0b3J5IHVuZGVyIC9rYWdnbGUvaW5wdXQgY29udGFpbnMge21hcmtlcnN9Igog\nICAgICAgICsgKGYiIHdpdGggY29sdW1ucyB7bGlzdChuZWVkX2NvbHMpfSIgaWYgbmVlZF9jb2xz\nIGVsc2UgIiIpCiAgICAgICAgKyAiLiBBdHRhY2ggdGhlIGRhdGFzZXQgYXMgYSBub3RlYm9vayBp\nbnB1dC4iKQoKCmRlZiBfZmluZF9maWxlcyhwYXR0ZXJuX25hbWUsIG1heF9kZXB0aD03KToKICAg\nICIiIkdsb2IgYSBmaWxlIG5hbWUgYXQgaW5jcmVhc2luZyBkZXB0aHMgdW5kZXIgL2thZ2dsZS9p\nbnB1dC4gQ2hlYXBlciB0aGFuCiAgICBhIGZ1bGwgb3Mud2FsayBvdmVyIHRoZSAxMTJrLWZpbGUg\nTklIIHRyZWUuIiIiCiAgICBoaXRzID0gW10KICAgIGZvciBkZXB0aCBpbiByYW5nZSgwLCBtYXhf\nZGVwdGggKyAxKToKICAgICAgICBoaXRzICs9IGdsb2IuZ2xvYigiL2thZ2dsZS9pbnB1dC8iICsg\nIiovIiAqIGRlcHRoICsgcGF0dGVybl9uYW1lKQogICAgcmV0dXJuIHNvcnRlZChzZXQoaGl0cykp\nCgoKZGVmIHJlc3RvcmVfY2FjaGUoY2ZnOiBDb25maWcpIC0+IE5vbmU6CiAgICAiIiJDb3B5IGNh\nY2hlZCBpbWFnZSBhcnJheXMgZnJvbSBhbnkgYXR0YWNoZWQgZGF0YXNldCBpbnRvIHRoZSB3b3Jr\naW5nCiAgICBjYWNoZS4gY2ZnLmNhY2hlX2luIGlzIGNoZWNrZWQgZmlyc3Q7IGV2ZXJ5IG90aGVy\nIGF0dGFjaGVkIGlucHV0IGlzIHRoZW4KICAgIHNlYXJjaGVkIGF1dG9tYXRpY2FsbHksIHNvIGEg\ncmVuYW1lZCBvciByZS1jcmVhdGVkIGNhY2hlIGRhdGFzZXQgaXMgc3RpbGwKICAgIGZvdW5kIHdp\ndGhvdXQgZWRpdGluZyBhbnkgcGF0aC4iIiIKICAgIG9zLm1ha2VkaXJzKGNmZy5jYWNoZSwgZXhp\nc3Rfb2s9VHJ1ZSkKICAgIHdhbnRlZCA9IFtmIntkfV97c30iIGZvciBkIGluICgibmloIiwgInZp\nbmRyIiwgImNoZXhwZXJ0IikKICAgICAgICAgICAgICBmb3IgcyBpbiAoIngubnB5IiwgImlkeC5j\nc3YiLCAidGFnLmpzb24iKV0KICAgIGhpdHMgPSB7fQogICAgaWYgY2ZnLmNhY2hlX2luIGFuZCBv\ncy5wYXRoLmlzZGlyKGNmZy5jYWNoZV9pbik6CiAgICAgICAgZm9yIHJvb3QsIF9kLCBmaWxlcyBp\nbiBvcy53YWxrKGNmZy5jYWNoZV9pbik6CiAgICAgICAgICAgIGZvciBmIGluIGZpbGVzOgogICAg\nICAgICAgICAgICAgaWYgZiBpbiB3YW50ZWQgYW5kIGYgbm90IGluIGhpdHM6CiAgICAgICAgICAg\nICAgICAgICAgaGl0c1tmXSA9IG9zLnBhdGguam9pbihyb290LCBmKQogICAgZm9yIGYgaW4gd2Fu\ndGVkOgogICAgICAgIGlmIGYgbm90IGluIGhpdHM6CiAgICAgICAgICAgIGZvdW5kID0gX2ZpbmRf\nZmlsZXMoZikKICAgICAgICAgICAgaWYgZm91bmQ6CiAgICAgICAgICAgICAgICBoaXRzW2ZdID0g\nZm91bmRbMF0KICAgIGlmIG5vdCBoaXRzOgogICAgICAgIHByaW50KCJbY2FjaGVdIG5vIHNhdmVk\nIGNhY2hlIGF0dGFjaGVkOyBpbWFnZXMgd2lsbCBiZSBkZWNvZGVkICIKICAgICAgICAgICAgICAi\nKFZpbkRyIH4xMC00MCBtaW4sIE5JSCB+MyBtaW4pIikKICAgIGZvciBmLCBzcmMgaW4gc29ydGVk\nKGhpdHMuaXRlbXMoKSk6CiAgICAgICAgZHN0ID0gb3MucGF0aC5qb2luKGNmZy5jYWNoZSwgZikK\nICAgICAgICBpZiBub3Qgb3MucGF0aC5leGlzdHMoZHN0KToKICAgICAgICAgICAgc2h1dGlsLmNv\ncHkoc3JjLCBkc3QpCiAgICAgICAgICAgIHByaW50KGYiW2NhY2hlXSByZXN0b3JlZCB7ZjoxNnN9\nICh7b3MucGF0aC5nZXRzaXplKHNyYykvMWU2OjYuMGZ9IE1CKSIKICAgICAgICAgICAgICAgICAg\nZiIgIGZyb20ge29zLnBhdGguZGlybmFtZShzcmMpfSIpCiAgICAjIFJlZnVzZSBhbiB1bnRhZ2dl\nZCBpbWFnZSBjYWNoZTogcHJvdmVuYW5jZSB1bmtub3duLCBjb3VsZCBiZSB0aGUgb2xkCiAgICAj\nIHByZXZhbGVuY2UtaW5mbGF0ZWQgc2FtcGxlLgogICAgZm9yIGQgaW4gKCJuaWgiLCAiY2hleHBl\ncnQiKToKICAgICAgICBpZiAob3MucGF0aC5leGlzdHMoZiJ7Y2ZnLmNhY2hlfS97ZH1feC5ucHki\nKQogICAgICAgICAgICAgICAgYW5kIG5vdCBvcy5wYXRoLmV4aXN0cyhmIntjZmcuY2FjaGV9L3tk\nfV90YWcuanNvbiIpKToKICAgICAgICAgICAgcHJpbnQoZiJbY2FjaGVdIHtkfSBjYWNoZSBpcyBV\nTlRBR0dFRDsgcmVidWlsZGluZyByYXRoZXIgdGhhbiAiCiAgICAgICAgICAgICAgICAgIGYidHJ1\nc3RpbmcgdW5rbm93biBwcm92ZW5hbmNlIikKICAgICAgICAgICAgZm9yIHN1ZiBpbiAoIngubnB5\nIiwgImlkeC5jc3YiKToKICAgICAgICAgICAgICAgIHB0aCA9IGYie2NmZy5jYWNoZX0ve2R9X3tz\ndWZ9IgogICAgICAgICAgICAgICAgaWYgb3MucGF0aC5leGlzdHMocHRoKToKICAgICAgICAgICAg\nICAgICAgICBvcy5yZW1vdmUocHRoKQoKCiMgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09IExPQURFUlMKCmRlZiBsb2FkX25paChjZmc6\nIENvbmZpZykgLT4gcGQuRGF0YUZyYW1lOgogICAgIiIiTklIIENoZXN0WC1yYXkxNC4gUGlwZS1k\nZWxpbWl0ZWQgJ0ZpbmRpbmcgTGFiZWxzJyBjb2x1bW4sIFBORyBpbWFnZXMuIiIiCiAgICByb290\nID0gZmluZF9yb290KFsiRGF0YV9FbnRyeV8yMDE3LmNzdiJdLCAiTklIIENoZXN0WC1yYXkxNCIp\nCiAgICBjc3YgPSBuZXh0KG9zLnBhdGguam9pbihyb290LCBjKSBmb3IgYyBpbgogICAgICAgICAg\nICAgICAoIkRhdGFfRW50cnlfMjAxNy5jc3YiLCAiRGF0YV9FbnRyeV8yMDE3X3YyMDIwLmNzdiIp\nCiAgICAgICAgICAgICAgIGlmIG9zLnBhdGguZXhpc3RzKG9zLnBhdGguam9pbihyb290LCBjKSkp\nCiAgICBkZiA9IHBkLnJlYWRfY3N2KGNzdikKICAgIGRmLmNvbHVtbnMgPSBbYy5zdHJpcCgpIGZv\nciBjIGluIGRmLmNvbHVtbnNdCgogICAgcGF0aHMgPSAoZ2xvYi5nbG9iKG9zLnBhdGguam9pbihy\nb290LCAiaW1hZ2VzKiIsICJpbWFnZXMiLCAiKi5wbmciKSkKICAgICAgICAgICAgIG9yIGdsb2Iu\nZ2xvYihvcy5wYXRoLmpvaW4ocm9vdCwgIioqIiwgIioucG5nIiksIHJlY3Vyc2l2ZT1UcnVlKSkK\nICAgIGlmIG5vdCBwYXRoczoKICAgICAgICByYWlzZSBGaWxlTm90Rm91bmRFcnJvcihmIk5JSDog\nbm8gUE5HIGZpbGVzIHVuZGVyIHtyb290fSIpCiAgICBkZlsicGF0aCJdID0gZGZbIkltYWdlIElu\nZGV4Il0ubWFwKHtvcy5wYXRoLmJhc2VuYW1lKHApOiBwIGZvciBwIGluIHBhdGhzfSkKICAgIGRm\nID0gZGYuZHJvcG5hKHN1YnNldD1bInBhdGgiXSkucmVzZXRfaW5kZXgoZHJvcD1UcnVlKQoKICAg\nICMgU3BsaXQgdGhlIHBpcGUtZGVsaW1pdGVkIGZpZWxkIGludG8gYSBzZXQgcGVyIHJvdywgdGhl\nbiBtYXRjaCBvbiBleGFjdAogICAgIyB0b2tlbnMuIFN1YnN0cmluZyBtYXRjaGluZyB3b3VsZCBt\nYWtlICdFZmZ1c2lvbicgYWxzbyBmaXJlIG9uIG5vdGhpbmcKICAgICMgaGVyZSwgYnV0IGV4YWN0\nLXRva2VuIG1hdGNoaW5nIGlzIHRoZSBkZWZlbnNpYmxlIHJ1bGUgYW5kIGdlbmVyYWxpc2VzLgog\nICAgdG9rZW5zID0gZGZbIkZpbmRpbmcgTGFiZWxzIl0uc3RyLnNwbGl0KCJ8IikuYXBwbHkoCiAg\nICAgICAgbGFtYmRhIHhzOiB7dC5zdHJpcCgpIGZvciB0IGluIHhzfSkKICAgIGZvciBjYW5vbiBp\nbiBBTExfRklORElOR1M6CiAgICAgICAgd2FudCA9IHNldChMQUJFTF9NQVBbY2Fub25dWyJuaWgi\nXSkKICAgICAgICBkZltjYW5vbl0gPSB0b2tlbnMuYXBwbHkobGFtYmRhIHMsIHc9d2FudDogaW50\nKGJvb2wocyAmIHcpKSkKCiAgICBkZlsicGF0aWVudF9pZCJdID0gZGZbIlBhdGllbnQgSUQiXS5h\nc3R5cGUoc3RyKQogICAgZGZbInNvdXJjZSJdID0gIm5paCIKICAgIG91dCA9IGRmW1siSW1hZ2Ug\nSW5kZXgiLCAicGF0aCIsICJwYXRpZW50X2lkIiwgInNvdXJjZSJdCiAgICAgICAgICAgICArIEFM\nTF9GSU5ESU5HU10uY29weSgpCiAgICBvdXQgPSBvdXQucmVuYW1lKGNvbHVtbnM9eyJJbWFnZSBJ\nbmRleCI6ICJpbWFnZV9pZCJ9KQogICAgcHJpbnQoZiJbbmloXSB7bGVuKG91dCk6LH0gaW1hZ2Vz\nLCB7b3V0LnBhdGllbnRfaWQubnVuaXF1ZSgpOix9IHBhdGllbnRzIikKICAgIHJldHVybiBvdXQK\nCgpkZWYgbG9hZF92aW5kcihjZmc6IENvbmZpZykgLT4gcGQuRGF0YUZyYW1lOgogICAgIiIiVmlu\nRHItQ1hSLiBPbmUgcm93IHBlciBib3VuZGluZyBib3ggcGVyIHJhZGlvbG9naXN0LCBzbyBhbiBp\nbWFnZSB3aXRoCiAgICB0d28gbm9kdWxlcyBhcHBlYXJzIHR3aWNlLiBDb3VudCBESVNUSU5DVCBy\nYWRfaWQgcGVyIChpbWFnZSwgY2xhc3MpIOKAlAogICAgY291bnRpbmcgcm93cyB3b3VsZCBiYWRs\neSBvdmVyY291bnQuIiIiCiAgICByb290ID0gZmluZF9yb290KFsidHJhaW4uY3N2IiwgInRyYWlu\nIl0sICJWaW5Eci1DWFIiLAogICAgICAgICAgICAgICAgICAgICBuZWVkX2NvbHM9KCJpbWFnZV9p\nZCIsICJjbGFzc19uYW1lIiwgInJhZF9pZCIpKQogICAgYW5uID0gcGQucmVhZF9jc3Yob3MucGF0\naC5qb2luKHJvb3QsICJ0cmFpbi5jc3YiKSkKICAgIGlkcyA9IHBkLkluZGV4KGFublsiaW1hZ2Vf\naWQiXS51bmlxdWUoKSwgbmFtZT0iaW1hZ2VfaWQiKQoKICAgIHZvdGVzID0gKGFublthbm5bImNs\nYXNzX25hbWUiXSAhPSAiTm8gZmluZGluZyJdCiAgICAgICAgICAgICAuZ3JvdXBieShbImltYWdl\nX2lkIiwgImNsYXNzX25hbWUiXSlbInJhZF9pZCJdLm51bmlxdWUoKQogICAgICAgICAgICAgLnJl\nc2V0X2luZGV4KG5hbWU9Im5fcmVhZGVycyIpCiAgICAgICAgICAgICAucGl2b3QoaW5kZXg9Imlt\nYWdlX2lkIiwgY29sdW1ucz0iY2xhc3NfbmFtZSIsIHZhbHVlcz0ibl9yZWFkZXJzIikKICAgICAg\nICAgICAgIC5yZWluZGV4KGlkcykuZmlsbG5hKDApKQoKICAgIGRmID0gcGQuRGF0YUZyYW1lKGlu\nZGV4PWlkcykKICAgIGZvciBjYW5vbiBpbiBBTExfRklORElOR1M6CiAgICAgICAgY29scyA9IFtj\nIGZvciBjIGluIExBQkVMX01BUFtjYW5vbl1bInZpbmRyIl0gaWYgYyBpbiB2b3Rlcy5jb2x1bW5z\nXQogICAgICAgIGlmIG5vdCBjb2xzOgogICAgICAgICAgICBwcmludChmIlt2aW5kcl0gV0FSTklO\nRyBubyBWaW5EciBjbGFzcyBmb3Ige2Nhbm9ufTsgIgogICAgICAgICAgICAgICAgICBmImF2YWls\nYWJsZToge3NvcnRlZCh2b3Rlcy5jb2x1bW5zKX0iKQogICAgICAgIG4gPSB2b3Rlc1tjb2xzXS5t\nYXgoYXhpcz0xKSBpZiBjb2xzIGVsc2UgcGQuU2VyaWVzKDAuMCwgaW5kZXg9aWRzKQogICAgICAg\nICMgQm90aCBhZ3JlZW1lbnQgcnVsZXMgcmV0YWluZWQ6IHIxID0gYW55IHJlYWRlciwgcjIgPSBt\nYWpvcml0eSBvZiAzLgogICAgICAgICMgUmVwb3J0aW5nIHVuZGVyIGJvdGggaXMgYSByb2J1c3Ru\nZXNzIGFuYWx5c2lzIHRoYXQgY29zdHMgbm8gcmV0cmFpbmluZy4KICAgICAgICBkZltmIntjYW5v\nbn1fX3IxIl0gPSAobiA+PSAxKS5hc3R5cGUoaW50KQogICAgICAgIGRmW2Yie2Nhbm9ufV9fcjIi\nXSA9IChuID49IGNmZy52aW5kcl9taW5fcmVhZGVycykuYXN0eXBlKGludCkKICAgICAgICBkZltm\nIntjYW5vbn1fX25yZWFkZXJzIl0gPSBuLmFzdHlwZShpbnQpCiAgICBkZiA9IGRmLnJlc2V0X2lu\nZGV4KCkKCiAgICBmaWxlcyA9IChnbG9iLmdsb2Iob3MucGF0aC5qb2luKHJvb3QsICJ0cmFpbiIs\nICIqLmRpY29tIikpCiAgICAgICAgICAgICBvciBnbG9iLmdsb2Iob3MucGF0aC5qb2luKHJvb3Qs\nICJ0cmFpbiIsICIqLmRjbSIpKQogICAgICAgICAgICAgb3IgZ2xvYi5nbG9iKG9zLnBhdGguam9p\nbihyb290LCAiKioiLCAidHJhaW4iLCAiKi5wbmciKSwKICAgICAgICAgICAgICAgICAgICAgICAg\nICByZWN1cnNpdmU9VHJ1ZSkpCiAgICBpZiBub3QgZmlsZXM6CiAgICAgICAgcmFpc2UgRmlsZU5v\ndEZvdW5kRXJyb3IoZiJWaW5Ecjogbm8gaW1hZ2UgZmlsZXMgdW5kZXIge3Jvb3R9L3RyYWluIikK\nICAgIGRmWyJwYXRoIl0gPSBkZlsiaW1hZ2VfaWQiXS5tYXAoCiAgICAgICAge29zLnBhdGguc3Bs\naXRleHQob3MucGF0aC5iYXNlbmFtZShwKSlbMF06IHAgZm9yIHAgaW4gZmlsZXN9KQogICAgZGYg\nPSBkZi5kcm9wbmEoc3Vic2V0PVsicGF0aCJdKS5yZXNldF9pbmRleChkcm9wPVRydWUpCiAgICAj\nIFZpbkRyIHB1Ymxpc2hlcyBubyBwYXRpZW50IGlkZW50aWZpZXI7IGVhY2ggaW1hZ2UgaXMgdHJl\nYXRlZCBhcyBpdHMgb3duCiAgICAjIHBhdGllbnQuIFN0YXRlZCBleHBsaWNpdGx5IGJlY2F1c2Ug\naXQgaXMgYW4gYXNzdW1wdGlvbiwgbm90IGEgZmFjdC4KICAgIGRmWyJwYXRpZW50X2lkIl0gPSBk\nZlsiaW1hZ2VfaWQiXQogICAgZGZbInNvdXJjZSJdID0gInZpbmRyIgogICAgIyBEZWZhdWx0IGxh\nYmVsIHZpZXcgaXMgdGhlIG1ham9yaXR5IHJ1bGU7IHIxIGNvbHVtbnMga2VwdCBhbG9uZ3NpZGUu\nCiAgICBmb3IgY2Fub24gaW4gQUxMX0ZJTkRJTkdTOgogICAgICAgIGRmW2Nhbm9uXSA9IGRmW2Yi\ne2Nhbm9ufV9fcjIiXQogICAgcHJpbnQoZiJbdmluZHJdIHtsZW4oZGYpOix9IGltYWdlcyAobm8g\ncGF0aWVudCBJRHMgcHVibGlzaGVkOyB0cmVhdGVkIGFzICIKICAgICAgICAgIGYib25lIHBhdGll\nbnQgcGVyIGltYWdlKSIpCiAgICByZXR1cm4gZGYKCgpkZWYgX2xvYWRfY2hleHBlcnRfb3JpZ2lu\nYWwoY2ZnOiBDb25maWcpIC0+IHBkLkRhdGFGcmFtZToKICAgICIiIkNoZVhwZXJ0IChvcmlnaW5h\nbCByZWxlYXNlLCB0cmFpbi5jc3YgKyBQYXRoIGNvbHVtbikuIExhYmVscyBhcmUgLTEgdW5jZXJ0\nYWluIC8gMCBuZWdhdGl2ZSAvIDEgcG9zaXRpdmUgLyBibGFuay4KICAgIFVuY2VydGFpbnR5IGhh\nbmRsaW5nIGlzIGEgZG9jdW1lbnRlZCBjaG9pY2UsIG5vdCBhbiBpbXBsZW1lbnRhdGlvbiBkZXRh\naWw6CiAgICAnemVybycgdHJlYXRzIHVuY2VydGFpbiBhcyBuZWdhdGl2ZSwgd2hpY2ggaXMgdGhl\nIGNvbnZlbnRpb24gaW4gdGhlCiAgICBDaGVYcGVydCBiYXNlbGluZSBhbmQgdGhlIGNvbnNlcnZh\ndGl2ZSBvcHRpb24gZm9yIHByZXZhbGVuY2UgZXN0aW1hdGlvbi4iIiIKICAgIHJvb3QgPSBmaW5k\nX3Jvb3QoWyJ0cmFpbi5jc3YiXSwgIkNoZVhwZXJ0IiwgbmVlZF9jb2xzPSgiUGF0aCIsKSkKICAg\nIGRmID0gcGQucmVhZF9jc3Yob3MucGF0aC5qb2luKHJvb3QsICJ0cmFpbi5jc3YiKSkKCiAgICAj\nIEZyb250YWwgb25seSwgdG8gbWF0Y2ggTklIIChmcm9udGFsKSBhbmQgVmluRHIgKFBBKS4KICAg\nIGlmIGNmZy5jaGV4cGVydF92aWV3ID09ICJmcm9udGFsIiBhbmQgIkZyb250YWwvTGF0ZXJhbCIg\naW4gZGYuY29sdW1uczoKICAgICAgICBiZWZvcmUgPSBsZW4oZGYpCiAgICAgICAgZGYgPSBkZltk\nZlsiRnJvbnRhbC9MYXRlcmFsIl0gPT0gIkZyb250YWwiXS5yZXNldF9pbmRleChkcm9wPVRydWUp\nCiAgICAgICAgcHJpbnQoZiJbY2hleHBlcnRdIGZyb250YWwgZmlsdGVyOiB7YmVmb3JlOix9IC0+\nIHtsZW4oZGYpOix9IikKCiAgICBmb3IgY2Fub24gaW4gU0hBUkVEOgogICAgICAgIGNvbHMgPSBb\nYyBmb3IgYyBpbiBMQUJFTF9NQVBbY2Fub25dWyJjaGV4cGVydCJdIGlmIGMgaW4gZGYuY29sdW1u\nc10KICAgICAgICBpZiBub3QgY29sczoKICAgICAgICAgICAgcmFpc2UgS2V5RXJyb3IoZiJDaGVY\ncGVydDogbm8gY29sdW1uIGZvciB7Y2Fub259OyAiCiAgICAgICAgICAgICAgICAgICAgICAgICAg\nIGYiYXZhaWxhYmxlOiB7bGlzdChkZi5jb2x1bW5zKVs6MjBdfSIpCiAgICAgICAgcmF3ID0gZGZb\nY29sc10ubWF4KGF4aXM9MSkKICAgICAgICBpZiBjZmcuY2hleHBlcnRfdW5jZXJ0YWluID09ICJ6\nZXJvIjoKICAgICAgICAgICAgZGZbY2Fub25dID0gKHJhdyA9PSAxKS5hc3R5cGUoaW50KQogICAg\nICAgIGVsaWYgY2ZnLmNoZXhwZXJ0X3VuY2VydGFpbiA9PSAib25lcyI6CiAgICAgICAgICAgIGRm\nW2Nhbm9uXSA9IHJhdy5pc2luKFsxLCAtMV0pLmFzdHlwZShpbnQpCiAgICAgICAgZWxpZiBjZmcu\nY2hleHBlcnRfdW5jZXJ0YWluID09ICJkcm9wIjoKICAgICAgICAgICAgZGYubG9jW3JhdyA9PSAt\nMSwgY2Fub25dID0gbnAubmFuCiAgICAgICAgICAgIGRmW2Nhbm9uXSA9IChyYXcgPT0gMSkuYXN0\neXBlKGZsb2F0KQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHJhaXNlIFZhbHVlRXJyb3IoY2Zn\nLmNoZXhwZXJ0X3VuY2VydGFpbikKCiAgICAjIFBhdGggY29sdW1uIGlzIHJlbGF0aXZlLCBlLmcu\nIENoZVhwZXJ0LXYxLjAtc21hbGwvdHJhaW4vcGF0aWVudDAwMDAxLy4uLgogICAgcGF0aGNvbCA9\nICJQYXRoIiBpZiAiUGF0aCIgaW4gZGYuY29sdW1ucyBlbHNlIGRmLmNvbHVtbnNbMF0KICAgIGRm\nWyJwYXRoIl0gPSBkZltwYXRoY29sXS5hcHBseSgKICAgICAgICBsYW1iZGEgcDogcCBpZiBvcy5w\nYXRoLmlzYWJzKHApIGVsc2Ugb3MucGF0aC5qb2luKAogICAgICAgICAgICByb290LCByZS5zdWIo\nciJeQ2hlWHBlcnQtdjFcLjAoLXNtYWxsKT8vIiwgIiIsIHN0cihwKSkpKQogICAgbWlzc2luZyA9\nICh+ZGZbInBhdGgiXS5hcHBseShvcy5wYXRoLmV4aXN0cykpLnN1bSgpCiAgICBpZiBtaXNzaW5n\nOgogICAgICAgIHByaW50KGYiW2NoZXhwZXJ0XSBXQVJOSU5HIHttaXNzaW5nOix9IHBhdGhzIGRv\nIG5vdCByZXNvbHZlOyAiCiAgICAgICAgICAgICAgZiJjaGVjayB0aGUgYXJjaGl2ZSBsYXlvdXQg\ndW5kZXIge3Jvb3R9IikKICAgICAgICBkZiA9IGRmW2RmWyJwYXRoIl0uYXBwbHkob3MucGF0aC5l\neGlzdHMpXS5yZXNldF9pbmRleChkcm9wPVRydWUpCgogICAgIyBwYXRpZW50Tk5OTk4gYXBwZWFy\ncyBpbiB0aGUgcGF0aDsgZ2VudWluZSBwYXRpZW50IElEcyBtYXR0ZXIgaGVyZSBiZWNhdXNlCiAg\nICAjIENoZVhwZXJ0IGNvbnRhaW5zIG1hbnkgc3R1ZGllcyBwZXIgcGF0aWVudC4KICAgIGRmWyJw\nYXRpZW50X2lkIl0gPSBkZltwYXRoY29sXS5hc3R5cGUoc3RyKS5zdHIuZXh0cmFjdCgKICAgICAg\nICByIihwYXRpZW50XGQrKSIsIGV4cGFuZD1GYWxzZSkuZmlsbG5hKGRmLmluZGV4LnRvX3Nlcmll\ncygpLmFzdHlwZShzdHIpKQogICAgZGZbImltYWdlX2lkIl0gPSBkZltwYXRoY29sXS5hc3R5cGUo\nc3RyKQogICAgZGZbInNvdXJjZSJdID0gImNoZXhwZXJ0IgogICAgb3V0ID0gZGZbWyJpbWFnZV9p\nZCIsICJwYXRoIiwgInBhdGllbnRfaWQiLCAic291cmNlIl0gKyBTSEFSRURdCiAgICBwcmludChm\nIltjaGV4cGVydF0ge2xlbihvdXQpOix9IGZyb250YWwgaW1hZ2VzLCAiCiAgICAgICAgICBmIntv\ndXQucGF0aWVudF9pZC5udW5pcXVlKCk6LH0gcGF0aWVudHMgIgogICAgICAgICAgZiIodW5jZXJ0\nYWluIC0+IHtjZmcuY2hleHBlcnRfdW5jZXJ0YWlufSkiKQogICAgcmV0dXJuIG91dAoKCl9LRVlf\nUkUgPSByZS5jb21waWxlKHIiKHBhdGllbnRcZCspW15hLXowLTldKihzdHVkeVxkKykiCiAgICAg\nICAgICAgICAgICAgICAgIHIiKD86W15hLXowLTldKih2aWV3XGQrX1thLXpdKykpPyIpCgoKZGVm\nIF9jaGV4X2tleSh0ZXh0KToKICAgICIiIk5vcm1hbGlzZSBhbnkgQ2hlWHBlcnQgcGF0aCBvciBm\naWxlbmFtZSB0byAocGF0aWVudCwgc3R1ZHksIHZpZXcpLgogICAgSGFuZGxlcyAndHJhaW4vcGF0\naWVudDAwMDAxL3N0dWR5MS92aWV3MV9mcm9udGFsLmpwZycsCiAgICAncGF0aWVudDAwMDAxL3N0\ndWR5MS92aWV3MV9mcm9udGFsLnBuZycgYW5kIHRoZSBmbGF0dGVuZWQgYXJjaGl2ZSBuYW1lCiAg\nICAncGF0aWVudDAwMDAxX3N0dWR5MV92aWV3MV9mcm9udGFsLmpwZycuIiIiCiAgICBtID0gX0tF\nWV9SRS5zZWFyY2goc3RyKHRleHQpLmxvd2VyKCkpCiAgICBpZiBub3QgbToKICAgICAgICByZXR1\ncm4gTm9uZQogICAgcmV0dXJuIG0uZ3JvdXAoMSksIG0uZ3JvdXAoMiksIG0uZ3JvdXAoMykKCgpk\nZWYgX3JlYWRfbGFiZWxfZmlsZShwYXRoKToKICAgICIiIkNoZVhiZXJ0IGxhYmVsIGZpbGVzIG1h\neSBiZSBKU09OIExpbmVzIG9yIGEgc2luZ2xlIEpTT04gYXJyYXkvb2JqZWN0LgogICAgUmV0dXJu\ncyBhIERhdGFGcmFtZSBlaXRoZXIgd2F5LiIiIgogICAgdHh0ID0gb3BlbihwYXRoLCAiciIsIGVy\ncm9ycz0iaWdub3JlIikucmVhZCgpLnN0cmlwKCkKICAgIGlmIG5vdCB0eHQ6CiAgICAgICAgcmV0\ndXJuIHBkLkRhdGFGcmFtZSgpCiAgICBpZiB0eHRbMF0gPT0gIlsiOgogICAgICAgIHJldHVybiBw\nZC5EYXRhRnJhbWUoanNvbi5sb2Fkcyh0eHQpKQogICAgaWYgdHh0WzBdID09ICJ7IjoKICAgICAg\nICB0cnk6ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIG9uZSBKU09OIG9iamVj\ndD8KICAgICAgICAgICAgb2JqID0ganNvbi5sb2Fkcyh0eHQpCiAgICAgICAgICAgIGlmIGlzaW5z\ndGFuY2Uob2JqLCBkaWN0KToKICAgICAgICAgICAgICAgIHZhbHMgPSBsaXN0KG9iai52YWx1ZXMo\nKSkKICAgICAgICAgICAgICAgIGlmIHZhbHMgYW5kIGlzaW5zdGFuY2UodmFsc1swXSwgZGljdCk6\nCiAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHBkLkRhdGFGcmFtZS5mcm9tX2RpY3Qob2JqLCBv\ncmllbnQ9ImluZGV4IikgXAogICAgICAgICAgICAgICAgICAgICAgICAucmVuYW1lX2F4aXMoInBh\ndGhfdG9faW1hZ2UiKS5yZXNldF9pbmRleCgpCiAgICAgICAgICAgICAgICByZXR1cm4gcGQuRGF0\nYUZyYW1lKG9iaikKICAgICAgICBleGNlcHQganNvbi5KU09ORGVjb2RlRXJyb3I6CiAgICAgICAg\nICAgIHBhc3MKICAgICAgICByb3dzID0gW2pzb24ubG9hZHMobCkgZm9yIGwgaW4gdHh0LnNwbGl0\nbGluZXMoKSBpZiBsLnN0cmlwKCldCiAgICAgICAgcmV0dXJuIHBkLkRhdGFGcmFtZShyb3dzKQog\nICAgcmV0dXJuIHBkLnJlYWRfY3N2KHBhdGgpCgoKZGVmIGRpc2NvdmVyX2NoZXhwZXJ0X3Rhciho\naW50OiBzdHIgPSAiIik6CiAgICAiIiJMb2NhdGUgdGhlIG91dHB1dCBvZiB0aGUgUmVkaXZpcyAv\nIENoZVhwZXJ0IFBsdXMgZG93bmxvYWQgbm90ZWJvb2s6CiAgICBtYW5pZmVzdF8qLmNzdiBmaWxl\ncywgc2hhcmQqXyoudGFyIGFyY2hpdmVzIGFuZCB0aGUgc2F2ZWQgbGFiZWwgZmlsZXMuCiAgICBT\nZXZlcmFsIHNhdmVkIG91dHB1dHMgbWF5IGJlIGF0dGFjaGVkIChvbmUgcGVyIGRvd25sb2FkIHJ1\nbik7IHRoZXkgYXJlCiAgICBtZXJnZWQuIFJldHVybnMgTm9uZSBpZiB0aGUgZm9ybWF0IGlzIG5v\ndCBwcmVzZW50LiIiIgogICAgbWFuaWZlc3RzLCBsYWJlbF9pbmRleGVzID0gW10sIFtdCiAgICBp\nZiBoaW50IGFuZCBvcy5wYXRoLmlzZGlyKGhpbnQpOgogICAgICAgIGZvciByb290LCBfZCwgZmls\nZXMgaW4gb3Mud2FsayhoaW50KToKICAgICAgICAgICAgZm9yIGYgaW4gZmlsZXM6CiAgICAgICAg\nICAgICAgICBpZiBmLnN0YXJ0c3dpdGgoIm1hbmlmZXN0X3NoYXJkIikgYW5kIGYuZW5kc3dpdGgo\nIi5jc3YiKToKICAgICAgICAgICAgICAgICAgICBtYW5pZmVzdHMuYXBwZW5kKG9zLnBhdGguam9p\nbihyb290LCBmKSkKICAgICAgICAgICAgICAgIGVsaWYgZiA9PSAiZmlsZV9pbmRleC5jc3YiOgog\nICAgICAgICAgICAgICAgICAgIGxhYmVsX2luZGV4ZXMuYXBwZW5kKG9zLnBhdGguam9pbihyb290\nLCBmKSkKICAgICAgICBpZiBtYW5pZmVzdHM6CiAgICAgICAgICAgIHByaW50KGYiW2NoZXhwZXJ0\nXSBmb3VuZCBkb3dubG9hZCBvdXRwdXQgYXQge2hpbnR9IikKICAgIGlmIG5vdCBtYW5pZmVzdHM6\nCiAgICAgICAgbWFuaWZlc3RzID0gW20gZm9yIG0gaW4gX2ZpbmRfZmlsZXMoIm1hbmlmZXN0X3No\nYXJkKi5jc3YiKV0KICAgIGlmIG5vdCBtYW5pZmVzdHM6CiAgICAgICAgcmV0dXJuIE5vbmUKICAg\nIHRhcnMgPSB7fQogICAgZm9yIG0gaW4gbWFuaWZlc3RzOgogICAgICAgIGZvciB0IGluIGdsb2Iu\nZ2xvYihvcy5wYXRoLmpvaW4ob3MucGF0aC5kaXJuYW1lKG0pLCAic2hhcmQqXyoudGFyIikpOgog\nICAgICAgICAgICB0YXJzLnNldGRlZmF1bHQob3MucGF0aC5iYXNlbmFtZSh0KSwgdCkKICAgIGxh\nYmVsX2luZGV4ZXMgPSBzb3J0ZWQoc2V0KGxhYmVsX2luZGV4ZXMpIHwgc2V0KF9maW5kX2ZpbGVz\nKCJmaWxlX2luZGV4LmNzdiIpKSkKICAgIHJldHVybiB7Im1hbmlmZXN0cyI6IG1hbmlmZXN0cywg\nInRhcnMiOiB0YXJzLAogICAgICAgICAgICAibGFiZWxfaW5kZXhlcyI6IGxhYmVsX2luZGV4ZXN9\nCgoKZGVmIF9sb2FkX2NoZXhwZXJ0X3RhcihjZmc6IENvbmZpZywgZm91bmQ6IGRpY3QpIC0+IHBk\nLkRhdGFGcmFtZToKICAgICIiIkNoZVhwZXJ0IFBsdXMgaW1hZ2VzIGRvd25sb2FkZWQgZnJvbSBS\nZWRpdmlzIGludG8gVEFSIGFyY2hpdmVzLgoKICAgIEltYWdlcyBhcmUgdW5wYWNrZWQgb25jZSB0\nbyBsb2NhbCBkaXNrIHNvIHRoZSBub3JtYWwgY2FjaGVkLCBwYXJhbGxlbAogICAgZGVjb2RlciBj\nYW4gcmVhZCB0aGVtLiBMYWJlbHMgY29tZSBmcm9tIHRoZSBvZmZpY2lhbCBDaGVYYmVydCBsYWJl\nbCBmaWxlcwogICAgYW5kIGFyZSBtYXRjaGVkIHRvIGltYWdlcyBieSBwYXRpZW50LCBzdHVkeSBh\nbmQgdmlldy4iIiIKICAgICMgLS0tLS0tLS0tLS0tLS0tLSBpbWFnZXMgLS0tLS0tLS0tLS0tLS0t\nLQogICAgbWFuID0gcGQuY29uY2F0KFtwZC5yZWFkX2NzdihtKSBmb3IgbSBpbiBmb3VuZFsibWFu\naWZlc3RzIl1dLAogICAgICAgICAgICAgICAgICAgIGlnbm9yZV9pbmRleD1UcnVlKQogICAgaWYg\nInN0YXR1cyIgaW4gbWFuLmNvbHVtbnM6CiAgICAgICAgbWFuID0gbWFuW21hblsic3RhdHVzIl0u\nYXN0eXBlKHN0cikgPT0gInNhdmVkIl0KICAgIG1hbiA9IG1hbi5kcm9wX2R1cGxpY2F0ZXMoImpw\nZyIpLnJlc2V0X2luZGV4KGRyb3A9VHJ1ZSkKICAgIG1hbiA9IG1hblttYW5bInRhciJdLmlzaW4o\nZm91bmRbInRhcnMiXSldLnJlc2V0X2luZGV4KGRyb3A9VHJ1ZSkKICAgIGlmIG1hbi5lbXB0eToK\nICAgICAgICByYWlzZSBGaWxlTm90Rm91bmRFcnJvcigKICAgICAgICAgICAgIkNoZVhwZXJ0OiBt\nYW5pZmVzdHMgZm91bmQgYnV0IG5vbmUgb2YgdGhlaXIgVEFSIGFyY2hpdmVzIGFyZSAiCiAgICAg\nICAgICAgICJhdHRhY2hlZC4gQXR0YWNoIHRoZSBzYXZlZCBub3RlYm9vayBvdXRwdXQgdGhhdCBo\nb2xkcyB0aGUgLnRhciAiCiAgICAgICAgICAgICJmaWxlcy4iKQogICAgcHJpbnQoZiJbY2hleHBl\ncnRdIHtsZW4oZm91bmRbJ21hbmlmZXN0cyddKX0gbWFuaWZlc3QocyksICIKICAgICAgICAgIGYi\ne21hbi50YXIubnVuaXF1ZSgpfSBUQVIgYXJjaGl2ZShzKSwge2xlbihtYW4pOix9IGltYWdlcyIp\nCgogICAgZXh0cmFjdCA9IGNmZy5jaGV4cGVydF9leHRyYWN0IG9yICgKICAgICAgICAiL2thZ2ds\nZS90ZW1wL2NoZXhwZXJ0X2pwZyIgaWYgb3MucGF0aC5pc2RpcigiL2thZ2dsZS90ZW1wIikKICAg\nICAgICBlbHNlICIvdG1wL2NoZXhwZXJ0X2pwZyIpCiAgICBvcy5tYWtlZGlycyhleHRyYWN0LCBl\neGlzdF9vaz1UcnVlKQogICAgd2FudGVkID0gc2V0KG1hblsianBnIl0pCiAgICBoYXZlID0gc2V0\nKG9zLmxpc3RkaXIoZXh0cmFjdCkpCiAgICB0b2RvID0gc29ydGVkKHNldChtYW4ubG9jW35tYW4u\nanBnLmlzaW4oaGF2ZSksICJ0YXIiXSkpCiAgICB0MCA9IHRpbWUudGltZSgpCiAgICBmb3IgdG5h\nbWUgaW4gdG9kbzoKICAgICAgICB3aXRoIHRhcmZpbGUub3Blbihmb3VuZFsidGFycyJdW3RuYW1l\nXSwgInIiKSBhcyB0ZjoKICAgICAgICAgICAgZm9yIG1lbSBpbiB0Zi5nZXRtZW1iZXJzKCk6CiAg\nICAgICAgICAgICAgICBiYXNlID0gb3MucGF0aC5iYXNlbmFtZShtZW0ubmFtZSkKICAgICAgICAg\nICAgICAgIGlmIG1lbS5pc2ZpbGUoKSBhbmQgYmFzZSBpbiB3YW50ZWQgYW5kIGJhc2Ugbm90IGlu\nIGhhdmU6CiAgICAgICAgICAgICAgICAgICAgd2l0aCBvcGVuKG9zLnBhdGguam9pbihleHRyYWN0\nLCBiYXNlKSwgIndiIikgYXMgZmg6CiAgICAgICAgICAgICAgICAgICAgICAgIGZoLndyaXRlKHRm\nLmV4dHJhY3RmaWxlKG1lbSkucmVhZCgpKQogICAgaWYgdG9kbzoKICAgICAgICBwcmludChmIltj\naGV4cGVydF0gdW5wYWNrZWQge2xlbih0b2RvKX0gYXJjaGl2ZShzKSB0byB7ZXh0cmFjdH0gaW4g\nIgogICAgICAgICAgICAgIGYie3RpbWUudGltZSgpLXQwOi4wZn1zIikKICAgIG1hblsicGF0aCJd\nID0gbWFuWyJqcGciXS5tYXAobGFtYmRhIGo6IG9zLnBhdGguam9pbihleHRyYWN0LCBqKSkKICAg\nIG1hbiA9IG1hblttYW5bInBhdGgiXS5tYXAob3MucGF0aC5leGlzdHMpXS5yZXNldF9pbmRleChk\ncm9wPVRydWUpCgogICAgIyAtLS0tLS0tLS0tLS0tLS0tIGxhYmVscyAtLS0tLS0tLS0tLS0tLS0t\nCiAgICBzcmMgPSBjZmcuY2hleHBlcnRfbGFiZWxfc291cmNlCiAgICBsYWJlbF9wYXRoID0gTm9u\nZQogICAgZm9yIGlkeF9jc3YgaW4gZm91bmRbImxhYmVsX2luZGV4ZXMiXToKICAgICAgICBmaSA9\nIHBkLnJlYWRfY3N2KGlkeF9jc3YpCiAgICAgICAgaWYgbm90IHsiZmlsZV9uYW1lIiwgImxvY2Fs\nX2ZpbGUifS5pc3N1YnNldChmaS5jb2x1bW5zKToKICAgICAgICAgICAgY29udGludWUKICAgICAg\nICBoaXQgPSBmaVtmaVsiZmlsZV9uYW1lIl0uYXN0eXBlKHN0cikuc3RyLmNvbnRhaW5zKHNyYywg\nY2FzZT1GYWxzZSldCiAgICAgICAgaWYgbGVuKGhpdCk6CiAgICAgICAgICAgIGNhbmQgPSBvcy5w\nYXRoLmpvaW4ob3MucGF0aC5kaXJuYW1lKGlkeF9jc3YpLAogICAgICAgICAgICAgICAgICAgICAg\nICAgICAgICAgIHN0cihoaXQuaWxvY1swXVsibG9jYWxfZmlsZSJdKSkKICAgICAgICAgICAgaWYg\nb3MucGF0aC5leGlzdHMoY2FuZCk6CiAgICAgICAgICAgICAgICBsYWJlbF9wYXRoID0gY2FuZAog\nICAgICAgICAgICAgICAgYnJlYWsKICAgIGlmIGxhYmVsX3BhdGggaXMgTm9uZToKICAgICAgICBs\nb29zZSA9IF9maW5kX2ZpbGVzKGYie3NyY31fZml4ZWQuanNvbiIpCiAgICAgICAgbGFiZWxfcGF0\naCA9IGxvb3NlWzBdIGlmIGxvb3NlIGVsc2UgTm9uZQogICAgaWYgbGFiZWxfcGF0aCBpcyBOb25l\nOgogICAgICAgIHJhaXNlIEZpbGVOb3RGb3VuZEVycm9yKAogICAgICAgICAgICBmIkNoZVhwZXJ0\nOiBubyAne3NyY30nIGxhYmVsIGZpbGUgZm91bmQuIEF0dGFjaCB0aGUgc2F2ZWQgb3V0cHV0ICIK\nICAgICAgICAgICAgZiJvZiB0aGUgZG93bmxvYWQgcnVuIHRoYXQgdXNlZCBHRVRfTEFCRUxTPVRy\ndWUuIikKICAgIGxhYiA9IF9yZWFkX2xhYmVsX2ZpbGUobGFiZWxfcGF0aCkKICAgIHBhdGhjb2wg\nPSBuZXh0KChjIGZvciBjIGluIGxhYi5jb2x1bW5zIGlmIGxhYltjXS5hc3R5cGUoc3RyKS5zdHIK\nICAgICAgICAgICAgICAgICAgICAuY29udGFpbnMoInBhdGllbnQiLCBjYXNlPUZhbHNlKS5tZWFu\nKCkgPiAwLjUpLCBOb25lKQogICAgaWYgcGF0aGNvbCBpcyBOb25lOgogICAgICAgIHJhaXNlIEtl\neUVycm9yKGYiQ2hlWHBlcnQ6IGxhYmVsIGZpbGUgaGFzIG5vIHBhdGggY29sdW1uOyBjb2x1bW5z\nICIKICAgICAgICAgICAgICAgICAgICAgICBmImFyZSB7bGlzdChsYWIuY29sdW1ucylbOjIwXX0i\nKQogICAga2V5cyA9IGxhYltwYXRoY29sXS5tYXAoX2NoZXhfa2V5KQogICAgbGFiID0gbGFiLmFz\nc2lnbihfcGF0PWtleXMuc3RyWzBdLCBfc3R1PWtleXMuc3RyWzFdLCBfdmlldz1rZXlzLnN0clsy\nXSkKICAgIGxhYiA9IGxhYi5kcm9wbmEoc3Vic2V0PVsiX3BhdCIsICJfc3R1Il0pCgogICAgaWsg\nPSBtYW5bImpwZyJdLm1hcChfY2hleF9rZXkpCiAgICBtYW4gPSBtYW4uYXNzaWduKF9wYXQ9aWsu\nc3RyWzBdLCBfc3R1PWlrLnN0clsxXSwgX3ZpZXc9aWsuc3RyWzJdKQoKICAgIGNvbHMgPSB7fQog\nICAgZm9yIGNhbm9uIGluIFNIQVJFRDoKICAgICAgICBjID0gbmV4dCgoYyBmb3IgYyBpbiBMQUJF\nTF9NQVBbY2Fub25dWyJjaGV4cGVydCJdCiAgICAgICAgICAgICAgICAgIGlmIGMgaW4gbGFiLmNv\nbHVtbnMpLCBOb25lKQogICAgICAgIGlmIGMgaXMgTm9uZToKICAgICAgICAgICAgcmFpc2UgS2V5\nRXJyb3IoZiJDaGVYcGVydDogbGFiZWwgZmlsZSBsYWNrcyAne2Nhbm9ufSc7IGNvbHVtbnMgIgog\nICAgICAgICAgICAgICAgICAgICAgICAgICBmImFyZSB7bGlzdChsYWIuY29sdW1ucylbOjIwXX0i\nKQogICAgICAgIGNvbHNbY2Fub25dID0gYwoKICAgICMgTWF0Y2ggcGVyIGltYWdlIGlmIHRoZSBs\nYWJlbCBmaWxlIGlzIHBlciBpbWFnZSwgZWxzZSBwZXIgc3R1ZHkuCiAgICBwZXJfaW1hZ2UgPSBs\nYWJbIl92aWV3Il0ubm90bmEoKS5tZWFuKCkgPiAwLjUKICAgIG9uID0gWyJfcGF0IiwgIl9zdHUi\nLCAiX3ZpZXciXSBpZiBwZXJfaW1hZ2UgZWxzZSBbIl9wYXQiLCAiX3N0dSJdCiAgICBMID0gbGFi\nLmRyb3BfZHVwbGljYXRlcyhvbilbb24gKyBsaXN0KGNvbHMudmFsdWVzKCkpXQogICAgZGYgPSBt\nYW4ubWVyZ2UoTCwgb249b24sIGhvdz0ibGVmdCIpCiAgICByYXRlID0gZmxvYXQoZGZbbGlzdChj\nb2xzLnZhbHVlcygpKV0ubm90bmEoKS5hbnkoYXhpcz0xKS5tZWFuKCkpCiAgICBwcmludChmIltj\naGV4cGVydF0gbGFiZWxzIGZyb20gJ3tvcy5wYXRoLmJhc2VuYW1lKGxhYmVsX3BhdGgpfScgIgog\nICAgICAgICAgZiIoe3NyY30gc2VjdGlvbiksIG1hdGNoZWQgeydwZXIgaW1hZ2UnIGlmIHBlcl9p\nbWFnZSBlbHNlICdwZXIgc3R1ZHknfSIKICAgICAgICAgIGYiOiB7cmF0ZSoxMDA6LjFmfSUgb2Yg\naW1hZ2VzIGhhdmUgYXQgbGVhc3Qgb25lIGxhYmVsIHZhbHVlIikKICAgIGlmIHJhdGUgPCAwLjU6\nCiAgICAgICAgcmFpc2UgQXNzZXJ0aW9uRXJyb3IoCiAgICAgICAgICAgICJDaGVYcGVydDogZmV3\nZXIgdGhhbiBoYWxmIHRoZSBpbWFnZXMgbWF0Y2hlZCBhIGxhYmVsIHJvdywgd2hpY2ggIgogICAg\nICAgICAgICAibWVhbnMgdGhlIGltYWdlIGFuZCBsYWJlbCBrZXlzIGRpc2FncmVlLiBFeGFtcGxl\nIGltYWdlIGtleTogIgogICAgICAgICAgICBmInt0dXBsZShtYW5bb25dLmlsb2NbMF0pfTsgZXhh\nbXBsZSBsYWJlbCBrZXk6ICIKICAgICAgICAgICAgZiJ7dHVwbGUobGFiW29uXS5pbG9jWzBdKX0i\nKQoKICAgIGZvciBjYW5vbiwgYyBpbiBjb2xzLml0ZW1zKCk6CiAgICAgICAgcmF3ID0gcGQudG9f\nbnVtZXJpYyhkZltjXSwgZXJyb3JzPSJjb2VyY2UiKSAgICMgTmFOID0gbm90IG1lbnRpb25lZAog\nICAgICAgIGlmIGNmZy5jaGV4cGVydF91bmNlcnRhaW4gPT0gInplcm8iOgogICAgICAgICAgICBk\nZltjYW5vbl0gPSAocmF3ID09IDEpLmFzdHlwZShpbnQpCiAgICAgICAgZWxpZiBjZmcuY2hleHBl\ncnRfdW5jZXJ0YWluID09ICJvbmVzIjoKICAgICAgICAgICAgZGZbY2Fub25dID0gcmF3LmlzaW4o\nWzEsIC0xXSkuYXN0eXBlKGludCkKICAgICAgICBlbHNlOgogICAgICAgICAgICByYWlzZSBWYWx1\nZUVycm9yKCJjaGV4cGVydF91bmNlcnRhaW4gbXVzdCBiZSAnemVybycgb3IgJ29uZXMnICIKICAg\nICAgICAgICAgICAgICAgICAgICAgICAgICAiZm9yIHRoZSBUQVIgZm9ybWF0IikKCiAgICBkZlsi\ncGF0aWVudF9pZCJdID0gZGZbIl9wYXQiXQogICAgZGZbImltYWdlX2lkIl0gPSBkZlsianBnIl0K\nICAgIGRmWyJzb3VyY2UiXSA9ICJjaGV4cGVydCIKICAgIG91dCA9IGRmW1siaW1hZ2VfaWQiLCAi\ncGF0aCIsICJwYXRpZW50X2lkIiwgInNvdXJjZSJdICsgU0hBUkVEXSBcCiAgICAgICAgLnJlc2V0\nX2luZGV4KGRyb3A9VHJ1ZSkKICAgIHByaW50KGYiW2NoZXhwZXJ0XSB7bGVuKG91dCk6LH0gZnJv\nbnRhbCBpbWFnZXMsICIKICAgICAgICAgIGYie291dC5wYXRpZW50X2lkLm51bmlxdWUoKTosfSBw\nYXRpZW50czsgcG9zaXRpdmVzOiAiCiAgICAgICAgICArICIsICIuam9pbihmIntjfSB7aW50KG91\ndFtjXS5zdW0oKSl9IiBmb3IgYyBpbiBTSEFSRUQpKQogICAgcmV0dXJuIG91dAoKCmRlZiBsb2Fk\nX2NoZXhwZXJ0KGNmZzogQ29uZmlnKSAtPiBwZC5EYXRhRnJhbWU6CiAgICAiIiJMb2FkIENoZVhw\nZXJ0IGluIHdoaWNoZXZlciBmb3JtYXQgaXMgYXR0YWNoZWQuCiAgICAgIDEuIFRBUiBhcmNoaXZl\ncyBmcm9tIHRoZSBSZWRpdmlzIC8gQ2hlWHBlcnQgUGx1cyBkb3dubG9hZCBub3RlYm9vawogICAg\nICAyLiB0aGUgb3JpZ2luYWwgQ2hlWHBlcnQgcmVsZWFzZSAodHJhaW4uY3N2IHdpdGggYSBQYXRo\nIGNvbHVtbikKICAgIFJhaXNlcyBGaWxlTm90Rm91bmRFcnJvciBpZiBuZWl0aGVyIGlzIHByZXNl\nbnQuIiIiCiAgICBmb3VuZCA9IGRpc2NvdmVyX2NoZXhwZXJ0X3RhcihjZmcuY2hleHBlcnRfaW4p\nCiAgICBpZiBmb3VuZDoKICAgICAgICByZXR1cm4gX2xvYWRfY2hleHBlcnRfdGFyKGNmZywgZm91\nbmQpCiAgICByZXR1cm4gX2xvYWRfY2hleHBlcnRfb3JpZ2luYWwoY2ZnKQoKCiMgPT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09IEFVRElU\nIEVYUE9SVFMKCmRlZiBleHBvcnRfbGFiZWxfbWFwX3RhYmxlKHBhdGg6IHN0cikgLT4gcGQuRGF0\nYUZyYW1lOgogICAgIiIiV3JpdGUgZXZlcnkgbWFwcGluZyBkZWNpc2lvbiB0byBDU1YgZm9yIHRo\nZSB0aGVzaXMgYXBwZW5kaXguIFJldmlld2VycwogICAgYXNrIGZvciB0aGlzLCBhbmQgY3Jvc3Mt\nZGF0YXNldCBkaWZmZXJlbmNlcyBhcmUgcGFydGx5IGF0dHJpYnV0YWJsZSB0bwogICAgbGFiZWwg\nZGVmaW5pdGlvbnMgcmF0aGVyIHRoYW4gbW9kZWwgYmVoYXZpb3VyLiIiIgogICAgcm93cyA9IFtd\nCiAgICBmb3IgY2Fub24sIG0gaW4gTEFCRUxfTUFQLml0ZW1zKCk6CiAgICAgICAgcm93cy5hcHBl\nbmQoeyJjYW5vbmljYWwiOiBjYW5vbiwKICAgICAgICAgICAgICAgICAgICAgInRpZXIiOiAiQSAo\nMyBzaXRlcykiIGlmIGNhbm9uIGluIFNIQVJFRAogICAgICAgICAgICAgICAgICAgICBlbHNlICJC\nIChOSUgtVmluRHIpIiwKICAgICAgICAgICAgICAgICAgICAgIm5paF90ZXJtcyI6ICIgfCAiLmpv\naW4obVsibmloIl0pLAogICAgICAgICAgICAgICAgICAgICAidmluZHJfdGVybXMiOiAiIHwgIi5q\nb2luKG1bInZpbmRyIl0pLAogICAgICAgICAgICAgICAgICAgICAiY2hleHBlcnRfdGVybXMiOiAo\nIiB8ICIuam9pbihtWyJjaGV4cGVydCJdKQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg\nICAgICAgICAgb3IgIihub3QgbGFiZWxsZWQpIiksCiAgICAgICAgICAgICAgICAgICAgICJzdGF0\ndXMiOiAiaW5jbHVkZWQiLCAibm90ZSI6IG1bIm5vdGUiXX0pCiAgICBmb3IgY2Fub24sIHdoeSBp\nbiBFWENMVURFRC5pdGVtcygpOgogICAgICAgIHJvd3MuYXBwZW5kKHsiY2Fub25pY2FsIjogY2Fu\nb24sICJ0aWVyIjogIi0iLCAibmloX3Rlcm1zIjogY2Fub24sCiAgICAgICAgICAgICAgICAgICAg\nICJ2aW5kcl90ZXJtcyI6IGNhbm9uLCAiY2hleHBlcnRfdGVybXMiOiBjYW5vbiwKICAgICAgICAg\nICAgICAgICAgICAgInN0YXR1cyI6ICJFWENMVURFRCIsICJub3RlIjogd2h5fSkKICAgIG91dCA9\nIHBkLkRhdGFGcmFtZShyb3dzKQogICAgb3V0LnRvX2NzdihwYXRoLCBpbmRleD1GYWxzZSkKICAg\nIHByaW50KGYiW2F1ZGl0XSBsYWJlbCBtYXBwaW5nIC0+IHtwYXRofSIpCiAgICByZXR1cm4gb3V0\nCgoKZGVmIHByZXZhbGVuY2VfcmVwb3J0KGZyYW1lczogbGlzdFtwZC5EYXRhRnJhbWVdLCBwYXRo\nOiBzdHIpIC0+IHBkLkRhdGFGcmFtZToKICAgICIiIlByZXZhbGVuY2UgcGVyIHBhdGhvbG9neSBw\nZXIgc291cmNlLCB3aXRoIHNoaWZ0IHJhdGlvcyBhZ2FpbnN0IHRoZQogICAgdHJhaW5pbmcgc291\ncmNlLiBUaGlzIHRhYmxlIGRyaXZlcyB0aGUgZGVjaXNpb24gcnVsZSwgc28gaXQgaXMgcHJvZHVj\nZWQKICAgIGJlZm9yZSBhbnkgbW9kZWxsaW5nLiIiIgogICAgcm93cyA9IHt9CiAgICBmb3IgZGYg\naW4gZnJhbWVzOgogICAgICAgIHNyYyA9IGRmWyJzb3VyY2UiXS5pbG9jWzBdCiAgICAgICAgcm93\nc1tzcmNdID0ge2M6IChmbG9hdChkZltjXS5tZWFuKCkpIGlmIGMgaW4gZGYuY29sdW1ucwogICAg\nICAgICAgICAgICAgICAgICAgICAgZWxzZSBucC5uYW4pIGZvciBjIGluIEFMTF9GSU5ESU5HU30K\nICAgIG91dCA9IHBkLkRhdGFGcmFtZShyb3dzKQogICAgaWYgIm5paCIgaW4gb3V0LmNvbHVtbnM6\nCiAgICAgICAgZm9yIHNyYyBpbiBvdXQuY29sdW1uczoKICAgICAgICAgICAgaWYgc3JjICE9ICJu\naWgiOgogICAgICAgICAgICAgICAgb3V0W2YicmF0aW9fe3NyY31fdnNfbmloIl0gPSAob3V0W3Ny\nY10gLyBvdXRbIm5paCJdKS5yb3VuZCgzKQogICAgICAgICAgICAgICAgb3V0W2YiYWJzTG9nUmF0\naW9fe3NyY30iXSA9IG5wLmFicygKICAgICAgICAgICAgICAgICAgICBucC5sb2cob3V0W3NyY10u\nY2xpcCgxZS02KSAvIG91dFsibmloIl0uY2xpcCgxZS02KSkKICAgICAgICAgICAgICAgICkucm91\nbmQoMykKICAgIG91dC50b19jc3YocGF0aCkKICAgIHByaW50KGYiW2F1ZGl0XSBwcmV2YWxlbmNl\nIC0+IHtwYXRofSIpCiAgICBwcmludChvdXQucm91bmQoNCkudG9fc3RyaW5nKCkpCiAgICByZXR1\ncm4gb3V0CgoKZGVmIHJlYWRlcl9hZ3JlZW1lbnQodmluZHI6IHBkLkRhdGFGcmFtZSwgcGF0aDog\nc3RyKSAtPiBwZC5EYXRhRnJhbWU6CiAgICAiIiJJbnRlci1yZWFkZXIgYWdyZWVtZW50IHBlciBw\nYXRob2xvZ3kuIEEgc3RhbmRhbG9uZSBkZXNjcmlwdGl2ZSByZXN1bHQ6CiAgICBjb250ZXN0ZWQg\nZmluZGluZ3MgYXJlIGV4YWN0bHkgdGhvc2Ugd2hvc2UgcHJldmFsZW5jZSBjb2xsYXBzZXMgbW9z\ndC4iIiIKICAgIHJvd3MgPSBbXQogICAgZm9yIGNhbm9uIGluIEFMTF9GSU5ESU5HUzoKICAgICAg\nICBpZiBmIntjYW5vbn1fX25yZWFkZXJzIiBub3QgaW4gdmluZHIuY29sdW1uczoKICAgICAgICAg\nICAgY29udGludWUKICAgICAgICBuID0gdmluZHJbZiJ7Y2Fub259X19ucmVhZGVycyJdCiAgICAg\nICAgYzEsIGMyLCBjMyA9IGludCgobiA9PSAxKS5zdW0oKSksIGludCgobiA9PSAyKS5zdW0oKSks\nIGludCgobiA+PSAzKS5zdW0oKSkKICAgICAgICB0b3QgPSBjMSArIGMyICsgYzMKICAgICAgICBy\nb3dzLmFwcGVuZCh7InBhdGhvbG9neSI6IGNhbm9uLCAiYW55X3JlYWRlciI6IHRvdCwKICAgICAg\nICAgICAgICAgICAgICAgImV4YWN0bHlfMSI6IGMxLCAiZXhhY3RseV8yIjogYzIsICJhbGxfMyI6\nIGMzLAogICAgICAgICAgICAgICAgICAgICAic2luZ2xlX3JlYWRlcl9wY3QiOiByb3VuZCgxMDAg\nKiBjMSAvIG1heCh0b3QsIDEpLCAxKSwKICAgICAgICAgICAgICAgICAgICAgInVuYW5pbW91c19w\nY3QiOiByb3VuZCgxMDAgKiBjMyAvIG1heCh0b3QsIDEpLCAxKX0pCiAgICBvdXQgPSBwZC5EYXRh\nRnJhbWUocm93cykuc2V0X2luZGV4KCJwYXRob2xvZ3kiKQogICAgb3V0LnRvX2NzdihwYXRoKQog\nICAgcHJpbnQoZiJbYXVkaXRdIHJlYWRlciBhZ3JlZW1lbnQgLT4ge3BhdGh9IikKICAgIHJldHVy\nbiBvdXQKCgojID09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PSBTQU1QTElORwoKZGVmIHN1YnNhbXBsZShkZjogcGQuRGF0YUZyYW1lLCBu\nOiBpbnQsIGNmZzogQ29uZmlnLAogICAgICAgICAgICAgIGZ1bGxfcmVmZXJlbmNlOiBwZC5EYXRh\nRnJhbWUgfCBOb25lID0gTm9uZSkgLT4gcGQuRGF0YUZyYW1lOgogICAgIiIiUExBSU4gUkFORE9N\nIFNBTVBMRS4gTmV2ZXIgZW5yaWNoLCBuZXZlciBmaWx0ZXIgdG8gcG9zaXRpdmVzLgoKICAgIFRo\naXMgZnVuY3Rpb24gaXMgc2hvcnQgYnV0IGl0IGlzIHRoZSBtb3N0IGltcG9ydGFudCBvbmUgaW4g\ndGhlIG1vZHVsZS4gQW4KICAgIGVhcmx5IHBpbG90IHZlcnNpb24ga2VwdCBvbmx5IGltYWdlcyB3\naXRoIGEgZmluZGluZywgd2hpY2ggaW5mbGF0ZWQgZXZlcnkKICAgIHByZXZhbGVuY2UgMy44eC4g\nQmVjYXVzZSBjYWxpYnJhdGlvbiBpcyBtZWFzdXJlZCBhZ2FpbnN0IHRoZSBiYXNlIHJhdGUsIHRo\nZQogICAgbW9kZWwgd2FzIHRoZW4gdHJhaW5lZCB0byBleHBlY3QgYSBwb3B1bGF0aW9uIHRoYXQg\nZG9lcyBub3QgZXhpc3QgYW5kIHRoZQogICAgaW4tZG9tYWluIHJlZmVyZW5jZSB3YXMgbWVhbmlu\nZ2xlc3MuIFRoZSBhc3NlcnRpb24gYmVsb3cgbWFrZXMgdGhhdCBjbGFzcwogICAgb2YgYnVnIGlt\ncG9zc2libGUgdG8gcmVpbnRyb2R1Y2Ugc2lsZW50bHkuCiAgICAiIiIKICAgIHJlZiA9IGZ1bGxf\ncmVmZXJlbmNlIGlmIGZ1bGxfcmVmZXJlbmNlIGlzIG5vdCBOb25lIGVsc2UgZGYKICAgIGlmIG4g\nPj0gbGVuKGRmKToKICAgICAgICBvdXQgPSBkZi5yZXNldF9pbmRleChkcm9wPVRydWUpCiAgICBl\nbHNlOgogICAgICAgIG91dCA9IGRmLnNhbXBsZShuPW4sIHJhbmRvbV9zdGF0ZT1jZmcuc2VlZCku\ncmVzZXRfaW5kZXgoZHJvcD1UcnVlKQogICAgY29scyA9IFtjIGZvciBjIGluIEFMTF9GSU5ESU5H\nUyBpZiBjIGluIG91dC5jb2x1bW5zXQogICAgcmF0aW8gPSAob3V0W2NvbHNdLm1lYW4oKSAvIHJl\nZltjb2xzXS5tZWFuKCkpCiAgICBiYWQgPSByYXRpb1socmF0aW8gPCAxIC0gY2ZnLnByZXZhbGVu\nY2VfdG9sKSB8CiAgICAgICAgICAgICAgICAocmF0aW8gPiAxICsgY2ZnLnByZXZhbGVuY2VfdG9s\nKV0KICAgIGlmIGxlbihiYWQpOgogICAgICAgIHJhaXNlIEFzc2VydGlvbkVycm9yKAogICAgICAg\nICAgICAiU2FtcGxpbmcgZGlzdG9ydGVkIHByZXZhbGVuY2UgYmV5b25kIHRvbGVyYW5jZTsgdGhl\nIGNhbGlicmF0aW9uICIKICAgICAgICAgICAgZiJzdHVkeSByZXF1aXJlcyBpdCBpbnRhY3QuXG57\nYmFkLnJvdW5kKDMpLnRvX3N0cmluZygpfSIpCiAgICBwcmludChmIltzYW1wbGVdIHtkZlsnc291\ncmNlJ10uaWxvY1swXX06IHtsZW4ob3V0KTosfSBpbWFnZXMsICIKICAgICAgICAgIGYicHJldmFs\nZW5jZSBwcmVzZXJ2ZWQgKG1heCBkZXZpYXRpb24gIgogICAgICAgICAgZiJ7ZmxvYXQoKHJhdGlv\nIC0gMSkuYWJzKCkubWF4KCkpICogMTAwOi4xZn0lKSIpCiAgICByZXR1cm4gb3V0CgoKZGVmIHBh\ndGllbnRfc3BsaXRzKGlkeDogcGQuRGF0YUZyYW1lLCBjZmc6IENvbmZpZywKICAgICAgICAgICAg\nICAgICAgIGZyYWNzPSgwLjcwLCAwLjE1LCAwLjE1KSkgLT4gbnAubmRhcnJheToKICAgICIiIlNw\nbGl0IEJZIFBBVElFTlQuIEltYWdlLWxldmVsIHNwbGl0dGluZyBsZWFrcyBzdHVkaWVzIGZyb20g\ndGhlIHNhbWUKICAgIHBhdGllbnQgYWNyb3NzIHRoZSBib3VuZGFyeSBhbmQgaW5mbGF0ZXMgdGhl\nIGluLWRvbWFpbiByZWZlcmVuY2UsIHdoaWNoIGlzCiAgICB0aGUgcXVhbnRpdHkgZXZlcnkgbGF0\nZXIgY29tcGFyaXNvbiBpcyBtZWFzdXJlZCBhZ2FpbnN0LiIiIgogICAgcGlkcyA9IG5wLmFzYXJy\nYXkoaWR4WyJwYXRpZW50X2lkIl0uYXN0eXBlKHN0cikudW5pcXVlKCksIGR0eXBlPW9iamVjdCkK\nICAgIHJuZyA9IG5wLnJhbmRvbS5SYW5kb21TdGF0ZShjZmcuc2VlZCkKICAgIHJuZy5zaHVmZmxl\nKHBpZHMpCiAgICBhID0gaW50KGZyYWNzWzBdICogbGVuKHBpZHMpKQogICAgYiA9IGludCgoZnJh\nY3NbMF0gKyBmcmFjc1sxXSkgKiBsZW4ocGlkcykpCiAgICB0ciwgdmEgPSBzZXQocGlkc1s6YV0p\nLCBzZXQocGlkc1thOmJdKQogICAgb3V0ID0gaWR4WyJwYXRpZW50X2lkIl0uYXN0eXBlKHN0ciku\nbWFwKAogICAgICAgIGxhbWJkYSBwOiAidHJhaW4iIGlmIHAgaW4gdHIgZWxzZSAoInZhbCIgaWYg\ncCBpbiB2YSBlbHNlICJ0ZXN0IikpLnZhbHVlcwogICAgIyBBc3NlcnQgbm8gcGF0aWVudCBhcHBl\nYXJzIGluIHR3byBzcGxpdHMuCiAgICBmb3IgeCwgeSBpbiAoKCJ0cmFpbiIsICJ2YWwiKSwgKCJ0\ncmFpbiIsICJ0ZXN0IiksICgidmFsIiwgInRlc3QiKSk6CiAgICAgICAgb3ggPSBzZXQoaWR4Lmxv\nY1tvdXQgPT0geCwgInBhdGllbnRfaWQiXS5hc3R5cGUoc3RyKSkKICAgICAgICBveSA9IHNldChp\nZHgubG9jW291dCA9PSB5LCAicGF0aWVudF9pZCJdLmFzdHlwZShzdHIpKQogICAgICAgIGFzc2Vy\ndCBub3QgKG94ICYgb3kpLCBmInBhdGllbnQgbGVha2FnZSBiZXR3ZWVuIHt4fSBhbmQge3l9Igog\nICAgcHJpbnQoZiJbc3BsaXRdICIgKyBzdHIoe2s6IGludCgob3V0ID09IGspLnN1bSgpKQogICAg\nICAgICAgICAgICAgICAgICAgICAgICAgZm9yIGsgaW4gKCJ0cmFpbiIsICJ2YWwiLCAidGVzdCIp\nfSkKICAgICAgICAgICsgIiAgKHBhdGllbnQtbGV2ZWwsIGxlYWthZ2UgY2hlY2tlZCkiKQogICAg\ncmV0dXJuIG91dAoKCiMgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09IElNQUdFIENBQ0hFCgpXT1JLRVJfU1JDID0gcicnJwppbXBvcnQg\nbnVtcHkgYXMgbnAsIHB5ZGljb20KZnJvbSBweWRpY29tLnBpeGVsX2RhdGFfaGFuZGxlcnMudXRp\nbCBpbXBvcnQgYXBwbHlfdm9pX2x1dApmcm9tIFBJTCBpbXBvcnQgSW1hZ2UKSU1HX1NJWkUgPSAy\nMjQKCmRlZiBfd2luZG93KGFyciwgc2l6ZSk6CiAgICAiIiJSb2J1c3QgcGVyY2VudGlsZSB3aW5k\nb3dpbmcuIFBlcmNlbnRpbGVzIGFyZSBjb21wdXRlZCBvbiBhIHN0cmlkZWQKICAgIHN1YnNhbXBs\nZTogc3RhdGlzdGljYWxseSBlcXVpdmFsZW50IGZvciB3aW5kb3dpbmcgYW5kIGFib3V0IDE2eCBj\naGVhcGVyLAogICAgd2hpY2ggaXMgbW9zdCBvZiB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIGEgMy1o\nb3VyIGFuZCBhIDMwLW1pbnV0ZSBkZWNvZGUuIiIiCiAgICBzdWIgPSBhcnJbOjo0LCA6OjRdCiAg\nICBsbywgaGkgPSBucC5wZXJjZW50aWxlKHN1YiwgWzAuNSwgOTkuNV0pCiAgICBpZiBoaSA8PSBs\nbzoKICAgICAgICBsbywgaGkgPSBmbG9hdChhcnIubWluKCkpLCBmbG9hdChhcnIubWF4KCkpCiAg\nICBhcnIgPSBucC5jbGlwKGFyciwgbG8sIGhpKQogICAgYXJyID0gKGFyciAtIGxvKSAvIG1heCho\naSAtIGxvLCAxZS04KSAqIDI1NS4wCiAgICByZXR1cm4gbnAuYXNhcnJheShJbWFnZS5mcm9tYXJy\nYXkoYXJyLmFzdHlwZShucC51aW50OCkpCiAgICAgICAgICAgICAgICAgICAgICAucmVzaXplKChz\naXplLCBzaXplKSwgSW1hZ2UuQklMSU5FQVIpLCBkdHlwZT1ucC51aW50OCkKCmRlZiBsb2FkX2Rp\nY29tKHBhdGgsIHNpemU9SU1HX1NJWkUpOgogICAgZHMgPSBweWRpY29tLmRjbXJlYWQocGF0aCkK\nICAgIGFyciA9IGRzLnBpeGVsX2FycmF5CiAgICB0cnk6IGFyciA9IGFwcGx5X3ZvaV9sdXQoYXJy\nLCBkcykKICAgIGV4Y2VwdCBFeGNlcHRpb246IHBhc3MKICAgIGFyciA9IGFyci5hc3R5cGUobnAu\nZmxvYXQzMikKICAgICMgTU9OT0NIUk9NRTEgbWVhbnMgaGlnaCB2YWx1ZSA9IGJsYWNrOyBpbnZl\ncnQgb3IgdGhlIGltYWdlIGlzIGEgbmVnYXRpdmUuCiAgICBpZiBnZXRhdHRyKGRzLCAiUGhvdG9t\nZXRyaWNJbnRlcnByZXRhdGlvbiIsICIiKSA9PSAiTU9OT0NIUk9NRTEiOgogICAgICAgIGFyciA9\nIGFyci5tYXgoKSAtIGFycgogICAgcmV0dXJuIF93aW5kb3coYXJyLCBzaXplKQoKZGVmIGxvYWRf\naW1hZ2UocGF0aCwgc2l6ZT1JTUdfU0laRSk6CiAgICByZXR1cm4gbnAuYXNhcnJheShJbWFnZS5v\ncGVuKHBhdGgpLmNvbnZlcnQoIkwiKQogICAgICAgICAgICAgICAgICAgICAgLnJlc2l6ZSgoc2l6\nZSwgc2l6ZSksIEltYWdlLkJJTElORUFSKSwgZHR5cGU9bnAudWludDgpCgpkZWYgam9iX2RpY29t\nKGEpOgogICAgaSwgcCA9IGEKICAgIHRyeTogcmV0dXJuIGksIGxvYWRfZGljb20ocCkKICAgIGV4\nY2VwdCBFeGNlcHRpb246IHJldHVybiBpLCBOb25lCgpkZWYgam9iX2ltYWdlKGEpOgogICAgaSwg\ncCA9IGEKICAgIHRyeTogcmV0dXJuIGksIGxvYWRfaW1hZ2UocCkKICAgIGV4Y2VwdCBFeGNlcHRp\nb246IHJldHVybiBpLCBOb25lCicnJwoKCmRlZiBfZW5zdXJlX3dvcmtlcihjZmc6IENvbmZpZyk6\nCiAgICAiIiJQcm9jZXNzUG9vbEV4ZWN1dG9yIGNhbm5vdCBwaWNrbGUgZnVuY3Rpb25zIGRlZmlu\nZWQgaW4gYSBub3RlYm9vayBjZWxsLAogICAgc28gdGhlIGRlY29kZXIgbXVzdCBsaXZlIGluIGEg\ncmVhbCBtb2R1bGUgb24gZGlzay4gcHlkaWNvbSBkZWNvZGluZyBpcwogICAgQ1BVLWJvdW5kIFB5\ndGhvbiwgc28gdGhyZWFkcyBnaXZlIGFsbW9zdCBubyBzcGVlZHVwIHVuZGVyIHRoZSBHSUwuIiIi\nCiAgICBvcy5tYWtlZGlycyhjZmcubGliLCBleGlzdF9vaz1UcnVlKQogICAgb3Blbihvcy5wYXRo\nLmpvaW4oY2ZnLmxpYiwgImN4cmxvYWQucHkiKSwgInciKS53cml0ZShXT1JLRVJfU1JDKQogICAg\naW1wb3J0IHN5cwogICAgaW1wb3J0IGltcG9ydGxpYgogICAgaWYgY2ZnLmxpYiBub3QgaW4gc3lz\nLnBhdGg6CiAgICAgICAgc3lzLnBhdGguaW5zZXJ0KDAsIGNmZy5saWIpCiAgICBpbXBvcnQgY3hy\nbG9hZAogICAgaW1wb3J0bGliLnJlbG9hZChjeHJsb2FkKQogICAgcmV0dXJuIGN4cmxvYWQKCgpk\nZWYgYnVpbGRfY2FjaGUoZGY6IHBkLkRhdGFGcmFtZSwgY2ZnOiBDb25maWcsIGtpbmQ6IHN0ciB8\nIE5vbmUgPSBOb25lKToKICAgICIiIkRlY29kZSBvbmNlIHRvIHVpbnQ4IGFycmF5czsgcmV1c2Ug\ndGhlcmVhZnRlci4gVGFnLXZlcnNpb25lZC4iIiIKICAgIG5hbWUgPSBkZlsic291cmNlIl0uaWxv\nY1swXQogICAgdGFnID0gY2ZnLnRhZ3NbbmFtZV0KICAgIGtpbmQgPSBraW5kIG9yICgiZGljb20i\nIGlmIHN0cihkZlsicGF0aCJdLmlsb2NbMF0pLmxvd2VyKCkKICAgICAgICAgICAgICAgICAgICAu\nZW5kc3dpdGgoKCIuZGljb20iLCAiLmRjbSIpKSBlbHNlICJpbWFnZSIpCiAgICBmeCA9IGYie2Nm\nZy5jYWNoZX0ve25hbWV9X3gubnB5IgogICAgZmkgPSBmIntjZmcuY2FjaGV9L3tuYW1lfV9pZHgu\nY3N2IgogICAgZnQgPSBmIntjZmcuY2FjaGV9L3tuYW1lfV90YWcuanNvbiIKCiAgICBpZiBvcy5w\nYXRoLmV4aXN0cyhmdCkgYW5kIG9zLnBhdGguZXhpc3RzKGZ4KSBhbmQgb3MucGF0aC5leGlzdHMo\nZmkpOgogICAgICAgIGlmIGpzb24ubG9hZChvcGVuKGZ0KSkuZ2V0KCJ0YWciKSA9PSB0YWc6CiAg\nICAgICAgICAgIFggPSBucC5sb2FkKGZ4KQogICAgICAgICAgICBpZHggPSBwZC5yZWFkX2Nzdihm\naSkKICAgICAgICAgICAgYXNzZXJ0IGxlbihYKSA9PSBsZW4oaWR4KSwgZiJ7bmFtZX06IGNhY2hl\nIGltYWdlL2xhYmVsIG1pc21hdGNoIgogICAgICAgICAgICBwcmludChmIltjYWNoZV0ge25hbWV9\nIHZhbGlkICh0YWc9e3RhZ30pLCB7WC5zaGFwZX0iKQogICAgICAgICAgICAjIExhYmVscyBhcmUg\ncmUtYXR0YWNoZWQgZnJvbSB0aGUgZnJlc2hseSBsb2FkZWQgZnJhbWUsIHNvIGFkZGluZwogICAg\nICAgICAgICAjIG9yIHJlbWFwcGluZyBmaW5kaW5ncyBuZXZlciBmb3JjZXMgYSByZS1kZWNvZGUg\nb2YgdGhlIGltYWdlcy4KICAgICAgICAgICAgaWR4ID0gcmVmcmVzaF9sYWJlbHMoaWR4LCBkZiwg\nbmFtZSkKICAgICAgICAgICAgaWR4LnRvX2NzdihmaSwgaW5kZXg9RmFsc2UpCiAgICAgICAgICAg\nIHJldHVybiBYLCBpZHgKCiAgICBtb2QgPSBfZW5zdXJlX3dvcmtlcihjZmcpCiAgICBqb2IgPSBt\nb2Quam9iX2RpY29tIGlmIGtpbmQgPT0gImRpY29tIiBlbHNlIG1vZC5qb2JfaW1hZ2UKICAgIHBh\ndGhzID0gZGZbInBhdGgiXS50b2xpc3QoKQogICAgbiwgdDAsIGRvbmUgPSBsZW4ocGF0aHMpLCB0\naW1lLnRpbWUoKSwgMAogICAgWCA9IG5wLnplcm9zKChuLCBjZmcuaW1nX3NpemUsIGNmZy5pbWdf\nc2l6ZSksIGR0eXBlPW5wLnVpbnQ4KQogICAgb2sgPSBucC56ZXJvcyhuLCBkdHlwZT1ib29sKQog\nICAgd2l0aCBQcm9jZXNzUG9vbEV4ZWN1dG9yKG1heF93b3JrZXJzPW1heCgxLCBvcy5jcHVfY291\nbnQoKSBvciAyKSkgYXMgZXg6CiAgICAgICAgZm9yIGksIGltZyBpbiBleC5tYXAoam9iLCBsaXN0\nKGVudW1lcmF0ZShwYXRocykpLCBjaHVua3NpemU9MTYpOgogICAgICAgICAgICBpZiBpbWcgaXMg\nbm90IE5vbmU6CiAgICAgICAgICAgICAgICBYW2ldID0gaW1nCiAgICAgICAgICAgICAgICBva1tp\nXSA9IFRydWUKICAgICAgICAgICAgZG9uZSArPSAxCiAgICAgICAgICAgIGlmIGRvbmUgJSAxMDAw\nID09IDA6CiAgICAgICAgICAgICAgICBtID0gKHRpbWUudGltZSgpIC0gdDApIC8gNjAKICAgICAg\nICAgICAgICAgIHByaW50KGYiW2NhY2hlXSB7bmFtZX0ge2RvbmU6LH0ve246LH0gIHttOi4xZn0g\nbWluLCAiCiAgICAgICAgICAgICAgICAgICAgICBmIn57bSAvIGRvbmUgKiAobiAtIGRvbmUpOi4x\nZn0gbWluIGxlZnQiLCBmbHVzaD1UcnVlKQogICAgWCwgaWR4ID0gWFtva10sIGRmLmxvY1tva10u\ncmVzZXRfaW5kZXgoZHJvcD1UcnVlKQogICAgbnAuc2F2ZShmeCwgWCkKICAgIGlkeC50b19jc3Yo\nZmksIGluZGV4PUZhbHNlKQogICAganNvbi5kdW1wKHsidGFnIjogdGFnLCAibiI6IGludChsZW4o\nWCkpLCAia2luZCI6IGtpbmR9LCBvcGVuKGZ0LCAidyIpKQogICAgcHJpbnQoZiJbY2FjaGVdIHtu\nYW1lfSBkZWNvZGVkIHtsZW4oWCk6LH0ve246LH0gaW4gIgogICAgICAgICAgZiJ7KHRpbWUudGlt\nZSgpIC0gdDApIC8gNjA6LjFmfSBtaW4gKHtpbnQoKH5vaykuc3VtKCkpfSBmYWlsZWQpIikKICAg\nIHJldHVybiBYLCBpZHgKCgpkZWYgcmVmcmVzaF9sYWJlbHMoaWR4X2NhY2hlZDogcGQuRGF0YUZy\nYW1lLCBkZl9mcmVzaDogcGQuRGF0YUZyYW1lLAogICAgICAgICAgICAgICAgICAgbmFtZTogc3Ry\nKSAtPiBwZC5EYXRhRnJhbWU6CiAgICAiIiJSZS1hdHRhY2ggbGFiZWwgY29sdW1ucyBmcm9tIGEg\nZnJlc2hseSBsb2FkZWQgZnJhbWUgdG8gYSBjYWNoZWQgaW5kZXgsCiAgICBtYXRjaGluZyByb3cg\nYnkgcm93IG9uIHRoZSBpbWFnZSBpZGVudGlmaWVyIGFuZCBQUkVTRVJWSU5HIGNhY2hlIG9yZGVy\nCiAgICAod2hpY2ggaXMgdGhlIG9yZGVyIG9mIHRoZSBpbWFnZSBhcnJheSkuCgogICAgV2h5IHRo\naXMgZXhpc3RzOiB0aGUgaW1hZ2UgYXJyYXlzIGFyZSB0aGUgZXhwZW5zaXZlIHBhcnQgKHRoZSBW\naW5EciBESUNPTQogICAgZGVjb2RlIHRvb2sgdXAgdG8gMjE0IG1pbnV0ZXMpLiBMYWJlbHMgYXJl\nIGNoZWFwIGFuZCBjaGFuZ2Ugd2hlbmV2ZXIgdGhlCiAgICBsYWJlbCBtYXAgY2hhbmdlcy4gS2V5\naW5nIGxhYmVscyB0byBpbWFnZXMgYnkgaWRlbnRpZmllciBtZWFucyB0aGUgbGFiZWwKICAgIG1h\ncCBjYW4gZXZvbHZlIHdpdGhvdXQgZXZlciByZS1kZWNvZGluZyBhIHNpbmdsZSBpbWFnZS4KICAg\nICIiIgogICAga2V5X2NhY2hlZCA9ICgiaW1hZ2VfaWQiIGlmICJpbWFnZV9pZCIgaW4gaWR4X2Nh\nY2hlZC5jb2x1bW5zCiAgICAgICAgICAgICAgICAgIGVsc2UgIkltYWdlIEluZGV4IiBpZiAiSW1h\nZ2UgSW5kZXgiIGluIGlkeF9jYWNoZWQuY29sdW1ucwogICAgICAgICAgICAgICAgICBlbHNlIE5v\nbmUpCiAgICBpZiBrZXlfY2FjaGVkIGlzIE5vbmU6CiAgICAgICAgcmFpc2UgS2V5RXJyb3IoZiJ7\nbmFtZX06IGNhY2hlZCBpbmRleCBoYXMgbm8gaW1hZ2UgaWRlbnRpZmllciBjb2x1bW47ICIKICAg\nICAgICAgICAgICAgICAgICAgICBmImRlbGV0ZSB7bmFtZX1fKiBmcm9tIHRoZSBjYWNoZSBhbmQg\ncmVidWlsZCIpCiAgICBrZXlfZnJlc2ggPSAiaW1hZ2VfaWQiIGlmICJpbWFnZV9pZCIgaW4gZGZf\nZnJlc2guY29sdW1ucyBlbHNlICJJbWFnZSBJbmRleCIKCiAgICBsYWJlbF9jb2xzID0gW2MgZm9y\nIGMgaW4gZGZfZnJlc2guY29sdW1ucwogICAgICAgICAgICAgICAgICBpZiBjIG5vdCBpbiAoInBh\ndGgiLCAicGF0aWVudF9pZCIsICJzb3VyY2UiLCBrZXlfZnJlc2gpXQogICAgZnJlc2ggPSBkZl9m\ncmVzaFtba2V5X2ZyZXNoXSArIGxhYmVsX2NvbHNdLmRyb3BfZHVwbGljYXRlcyhrZXlfZnJlc2gp\nCiAgICBiYXNlID0gaWR4X2NhY2hlZC5kcm9wKGNvbHVtbnM9W2MgZm9yIGMgaW4gbGFiZWxfY29s\ncwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZiBjIGluIGlkeF9jYWNoZWQu\nY29sdW1uc10pCiAgICBiYXNlID0gYmFzZS5yZW5hbWUoY29sdW1ucz17a2V5X2NhY2hlZDogImlt\nYWdlX2lkIn0pCiAgICBmcmVzaCA9IGZyZXNoLnJlbmFtZShjb2x1bW5zPXtrZXlfZnJlc2g6ICJp\nbWFnZV9pZCJ9KQogICAgYmFzZVsiX29yZGVyIl0gPSBucC5hcmFuZ2UobGVuKGJhc2UpKQogICAg\nb3V0ID0gYmFzZS5tZXJnZShmcmVzaCwgb249ImltYWdlX2lkIiwgaG93PSJsZWZ0Iikuc29ydF92\nYWx1ZXMoIl9vcmRlciIpCiAgICBvdXQgPSBvdXQuZHJvcChjb2x1bW5zPSJfb3JkZXIiKS5yZXNl\ndF9pbmRleChkcm9wPVRydWUpCgogICAgbWlzc2luZyA9IG91dFtsYWJlbF9jb2xzXS5pc25hKCku\nYW55KGF4aXM9MSkuc3VtKCkKICAgIGlmIG1pc3Npbmc6CiAgICAgICAgcmFpc2UgQXNzZXJ0aW9u\nRXJyb3IoCiAgICAgICAgICAgIGYie25hbWV9OiB7bWlzc2luZ30gY2FjaGVkIGltYWdlcyBoYXZl\nIG5vIGxhYmVscyBpbiB0aGUgZnJlc2ggIgogICAgICAgICAgICBmImZyYW1lLiBUaGUgY2FjaGUg\nYW5kIHRoZSBkYXRhc2V0IGRpc2FncmVlOyByZWJ1aWxkIHRoZSBjYWNoZS4iKQogICAgZm9yIGMg\naW4gbGFiZWxfY29sczoKICAgICAgICBpZiBvdXRbY10uZHJvcG5hKCkuaXNpbihbMCwgMV0pLmFs\nbCgpOgogICAgICAgICAgICBvdXRbY10gPSBvdXRbY10uYXN0eXBlKGludCkKICAgIGlmICJzb3Vy\nY2UiIG5vdCBpbiBvdXQuY29sdW1uczoKICAgICAgICBvdXRbInNvdXJjZSJdID0gbmFtZQogICAg\nYWRkZWQgPSBbYyBmb3IgYyBpbiBsYWJlbF9jb2xzIGlmIGMgbm90IGluIGlkeF9jYWNoZWQuY29s\ndW1uc10KICAgIGlmIGFkZGVkOgogICAgICAgIHByaW50KGYiW2NhY2hlXSB7bmFtZX06IGF0dGFj\naGVkIHtsZW4oYWRkZWQpfSBuZXcgbGFiZWwgY29sdW1uKHMpICIKICAgICAgICAgICAgICBmIndp\ndGhvdXQgcmUtZGVjb2RpbmcgaW1hZ2VzIikKICAgIHJldHVybiBvdXQKCgojID09PT09PT09PT09\nPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PSBTQU5JVFkg\nQ0hFQ0tTCgpkZWYgaW50ZWdyaXR5X2NoZWNrcyhYOiBucC5uZGFycmF5LCBpZHg6IHBkLkRhdGFG\ncmFtZSwgY2ZnOiBDb25maWcsCiAgICAgICAgICAgICAgICAgICAgIGZ1bGxfcmVmZXJlbmNlOiBw\nZC5EYXRhRnJhbWUgfCBOb25lID0gTm9uZSkgLT4gTm9uZToKICAgICIiIlJ1biBvbiB3aGF0ZXZl\nciBhY3R1YWxseSBsb2FkZWQsIGNhY2hlZCBvciBmcmVzaC4gVGhlIHBpbG90J3Mgb3JpZ2luYWwK\nICAgIHByZXZhbGVuY2UgYXNzZXJ0aW9uIHRlc3RlZCB0aGUgZnJlc2hseSBzYW1wbGVkIGZyYW1l\nLCBzbyBhIHN0YWxlIGNhY2hlCiAgICBsb2FkZWQgZnJvbSBkaXNrIGJ5cGFzc2VkIGl0IGVudGly\nZWx5LiIiIgogICAgbmFtZSA9IGlkeFsic291cmNlIl0uaWxvY1swXQogICAgYXNzZXJ0IGxlbihY\nKSA9PSBsZW4oaWR4KSwgZiJ7bmFtZX06IHtsZW4oWCl9IGltYWdlcyB2cyB7bGVuKGlkeCl9IGxh\nYmVscyIKICAgIGZvciBjIGluIFNJVEVfRklORElOR1MuZ2V0KG5hbWUsIFNIQVJFRCk6CiAgICAg\nICAgYXNzZXJ0IGMgaW4gaWR4LmNvbHVtbnMsIGYie25hbWV9OiBjYWNoZSBtaXNzaW5nIGNvbHVt\nbiB7Y30iCiAgICBhc3NlcnQgWC5kdHlwZSA9PSBucC51aW50OCwgZiJ7bmFtZX06IGV4cGVjdGVk\nIHVpbnQ4LCBnb3Qge1guZHR5cGV9IgogICAgYXNzZXJ0IFguc2hhcGVbMTpdID09IChjZmcuaW1n\nX3NpemUsIGNmZy5pbWdfc2l6ZSksIFwKICAgICAgICBmIntuYW1lfTogZXhwZWN0ZWQge2NmZy5p\nbWdfc2l6ZX1weCwgZ290IHtYLnNoYXBlWzE6XX0iCgogICAgIyBDaHVua2VkOiBYLnJlc2hhcGUo\nbiwtMSkuc3RkKGF4aXM9MSkgcHJvbW90ZXMgdG8gZmxvYXQ2NCBhbmQgd291bGQKICAgICMgYWxs\nb2NhdGUgfjE1IEdCIG9uIGEgNDBrIGNhY2hlLiBTYW1wbGUgYW5kIGNodW5rIGluc3RlYWQuCiAg\nICBibGFuaywgc3RlcCA9IDAsIDIwMDAKICAgIHByb2JlID0gbnAuYXJhbmdlKDAsIGxlbihYKSwg\nbWF4KDEsIGxlbihYKSAvLyAyMDAwMCkgb3IgMSlbOjIwMDAwXQogICAgZm9yIGEgaW4gcmFuZ2Uo\nMCwgbGVuKHByb2JlKSwgc3RlcCk6CiAgICAgICAgY2h1bmsgPSBYW3Byb2JlW2E6YSArIHN0ZXBd\nXS5yZXNoYXBlKC0xLCBjZmcuaW1nX3NpemUgKiBjZmcuaW1nX3NpemUpCiAgICAgICAgYmxhbmsg\nKz0gaW50KChjaHVuay5zdGQoYXhpcz0xLCBkdHlwZT1ucC5mbG9hdDMyKSA8IDEuMCkuc3VtKCkp\nCiAgICBpZiBibGFuazoKICAgICAgICBwcmludChmIltjaGVja10ge25hbWV9OiBXQVJOSU5HIHti\nbGFua30gbmVhci1jb25zdGFudCBpbWFnZXMgaW4gYSAiCiAgICAgICAgICAgICAgZiJ7bGVuKHBy\nb2JlKTosfS1pbWFnZSBwcm9iZSAocG9zc2libGUgZGVjb2RlIGZhaWx1cmVzKSIpCiAgICBpZiBm\ndWxsX3JlZmVyZW5jZSBpcyBub3QgTm9uZToKICAgICAgICBjb2xzID0gW2MgZm9yIGMgaW4gU0lU\nRV9GSU5ESU5HUy5nZXQobmFtZSwgU0hBUkVEKQogICAgICAgICAgICAgICAgaWYgYyBpbiBpZHgu\nY29sdW1ucyBhbmQgYyBpbiBmdWxsX3JlZmVyZW5jZS5jb2x1bW5zXQogICAgICAgIHJhdGlvID0g\nKGlkeFtjb2xzXS5tZWFuKCkgLyBmdWxsX3JlZmVyZW5jZVtjb2xzXS5tZWFuKCkpCiAgICAgICAg\nYmFkID0gcmF0aW9bKHJhdGlvIDwgMSAtIGNmZy5wcmV2YWxlbmNlX3RvbCkgfAogICAgICAgICAg\nICAgICAgICAgIChyYXRpbyA+IDEgKyBjZmcucHJldmFsZW5jZV90b2wpXQogICAgICAgIGFzc2Vy\ndCBub3QgbGVuKGJhZCksICgKICAgICAgICAgICAgZiJ7bmFtZX06IGNhY2hlZCBwcmV2YWxlbmNl\nIGRpc3RvcnRlZCDigJQgYWxtb3N0IGNlcnRhaW5seSBhIHN0YWxlICIKICAgICAgICAgICAgZiJl\nbnJpY2hlZCBjYWNoZS4gRGVsZXRlIHtjZmcuY2FjaGV9L3tuYW1lfV8qIGFuZCByZWJ1aWxkLlxu\nIgogICAgICAgICAgICBmIntiYWQucm91bmQoMykudG9fc3RyaW5nKCl9IikKICAgIG1wID0gZmxv\nYXQoWFtwcm9iZV0ubWVhbigpKSBpZiBsZW4ocHJvYmUpIGVsc2UgZmxvYXQoIm5hbiIpCiAgICBw\ncmludChmIltjaGVja10ge25hbWV9OiB7bGVuKFgpOix9IGltYWdlcywgbWVhbiBwaXhlbCB7bXA6\nLjFmfSAiCiAgICAgICAgICBmIihzYW1wbGVkKSwgcHJldmFsZW5jZSBpbnRhY3QiKQoKCmRlZiBu\nb3JtYWxpc2UoYmF0Y2g6IG5wLm5kYXJyYXksIG1lYW46IGZsb2F0ID0gMC40ODUsCiAgICAgICAg\nICAgICAgc3RkOiBmbG9hdCA9IDAuMjI5KSAtPiBucC5uZGFycmF5OgogICAgIiIidWludDggLT4g\nbm9ybWFsaXNlZCBmbG9hdDMyLiBBcHBsaWVkIGF0IGJhdGNoIHRpbWUsIG5vdCBhdCBjYWNoZSB0\naW1lLAogICAgc28gdGhlIGNhY2hlIHN0YXlzIGNvbXBhY3QgYW5kIGF1Z21lbnRhdGlvbiBjYW4g\nYmUgaW5zZXJ0ZWQgYmV0d2Vlbi4iIiIKICAgIHJldHVybiAoKGJhdGNoLmFzdHlwZShucC5mbG9h\ndDMyKSAvIDI1NS4wKSAtIG1lYW4pIC8gc3RkCg==\n\"\"\"\nPREPROC = base64.b64decode(PREPROC_B64.encode()).decode()\n# Always write the embedded copy. Searching for other copies was slow (it\n# walked every input file) and, on a re-run in the same session, found its\n# own previous copy and crashed copying a file onto itself.\nopen(f\"{LIB}/preprocessing.py\", \"w\").write(PREPROC)\nprint(\"  wrote embedded preprocessing module\")\n\nif LIB not in sys.path:\n    sys.path.insert(0, LIB)\nimport preprocessing as pp\nimportlib.reload(pp)\n\ncfg = pp.Config(seed=SEEDS[0], n_nih=N_NIH, n_chexpert=N_CHEX,\n                cache=CACHE, out=W, lib=LIB, cache_in=CACHE_IN,\n                chexpert_label_source=CHEX_LABELS, chexpert_in=CHEX_IN)\npp.restore_cache(cfg)\nprint(\"  cache dir:\", sorted(os.listdir(CACHE)) or \"(empty)\")\nSHARED = pp.SHARED                 # tier A: 5 findings, all three sites\nEXTENDED = pp.EXTENDED             # tier B: 4 more, NIH -> VinDr only\nFINDINGS = pp.ALL_FINDINGS         # model heads: all 9\nprint(\"  tier A (3 sites):   \", SHARED)\nprint(\"  tier B (NIH-VinDr): \", EXTENDED)\nprint(\"  model outputs:      \", len(FINDINGS), \"findings\")\n\n# ============================================================ SECTION 1\nbanner(\"SECTION 1  Load, harmonise and audit all available sources\")\n\nnih = pp.load_nih(cfg)\nvin = pp.load_vindr(cfg)\n\n# CheXpert is optional: run with two sites if it is not attached.\ntry:\n    chex = pp.load_chexpert(cfg)\n    HAVE_CHEX = True\nexcept (FileNotFoundError, KeyError) as e:\n    chex, HAVE_CHEX = None, False\n    print(\"  To add CheXpert: attach the SAVED OUTPUT of your CheXpert download\")\n    print(\"  notebook via Add Input (it must contain the .tar files, the\")\n    print(\"  manifest_*.csv files and the labels/ folder).\")\n    print(f\"  CheXpert NOT available ({type(e).__name__}): {e}\")\n    print(\"  Running with two\")\n    print(\"  sites. NOTE: without CheXpert the study cannot separate\")\n    print(\"  prevalence shift from annotation shift — add it before writing up.\")\n\npp.export_label_map_table(f\"{W}/label_mapping.csv\")\nframes = [nih, vin] + ([chex] if HAVE_CHEX else [])\nPREV = pp.prevalence_report(frames, f\"{W}/prevalence.csv\")\nAGREE = pp.reader_agreement(vin, f\"{W}/reader_agreement.csv\")\nprint(\"\\n  INTER-READER AGREEMENT (VinDr, 3 readers per image)\")\nprint(AGREE.to_string())\n\n# ============================================================ SECTION 2\nbanner(\"SECTION 2  Image caches\")\n\nnih_s = pp.subsample(nih, N_NIH, cfg, full_reference=nih)\nXn, IDXn = pp.build_cache(nih_s, cfg, kind=\"image\")\npp.integrity_checks(Xn, IDXn, cfg, full_reference=nih)\n\nXv, IDXv = pp.build_cache(vin, cfg, kind=\"dicom\")\npp.integrity_checks(Xv, IDXv, cfg)\n\nif HAVE_CHEX:\n    chex_s = pp.subsample(chex, N_CHEX, cfg, full_reference=chex)\n    Xc, IDXc = pp.build_cache(chex_s, cfg, kind=\"image\")\n    pp.integrity_checks(Xc, IDXc, cfg, full_reference=chex)\n\n# ============================================================ SECTION 3\nbanner(\"SECTION 3  Splits\")\n\nimport torch, torch.nn as nn, torchvision\nfrom torch.utils.data import Dataset, DataLoader\nfrom sklearn.metrics import roc_auc_score, brier_score_loss\nfrom sklearn.linear_model import LogisticRegression\n\nDEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"  device:\", DEV)\n\nsplit = pp.patient_splits(IDXn, cfg)\n\n# External sites split into a calibration pool (the labels a hospital would\n# \"pay\" for) and a disjoint evaluation set.\ndef ext_split(idx, seed, by_patient=False):\n    \"\"\"Split an external site into a calibration pool and an evaluation set.\n    by_patient=True keeps every image of a patient on one side, so labels a\n    hospital \"pays\" for never come from the same people it is evaluated on.\n    CheXpert has several studies per patient, so it needs this. VinDr\n    publishes no patient IDs, so it is split by image.\"\"\"\n    rng = np.random.RandomState(seed)\n    if by_patient:\n        pids = np.asarray(idx[\"patient_id\"].astype(str).unique(), dtype=object)\n        rng.shuffle(pids)\n        cal_p = set(pids[:int(CAL_FRAC * len(pids))])\n        is_cal = idx[\"patient_id\"].astype(str).isin(cal_p).values\n        cal, ev = np.where(is_cal)[0], np.where(~is_cal)[0]\n        assert not (set(idx.patient_id.iloc[cal]) & set(idx.patient_id.iloc[ev]))\n        return cal, ev\n    perm = rng.permutation(len(idx))\n    n = int(CAL_FRAC * len(perm))\n    return perm[:n], perm[n:]\n\nvin_cal, vin_eval = ext_split(IDXv, cfg.seed)\nYv = {\"r1\": IDXv[[f\"{s}__r1\" for s in FINDINGS]].values,\n      \"r2\": IDXv[[f\"{s}__r2\" for s in FINDINGS]].values}\nprint(f\"  VinDr  calib {len(vin_cal):,} / eval {len(vin_eval):,}\")\n\nif HAVE_CHEX:\n    chex_cal, chex_eval = ext_split(IDXc, cfg.seed, by_patient=True)\n    print(f\"  CheXpert split by patient: \"\n          f\"{IDXc.patient_id.iloc[chex_cal].nunique():,} / \"\n          f\"{IDXc.patient_id.iloc[chex_eval].nunique():,} patients\")\n    # CheXpert labels only tier A. Tier-B columns are filled with -1 as an\n    # explicit \"not labelled\" marker; they are never evaluated because each\n    # site is analysed only on its own finding list.\n    Yc_all = np.full((len(IDXc), len(FINDINGS)), -1, dtype=np.int64)\n    for j, f in enumerate(FINDINGS):\n        if f in IDXc.columns:\n            Yc_all[:, j] = IDXc[f].values\n    print(f\"  CheXpert calib {len(chex_cal):,} / eval {len(chex_eval):,}\")\n\nYn = IDXn[FINDINGS].values\nMEAN, STD = 0.485, 0.229\n\n\nclass CXR(Dataset):\n    def __init__(s, X, Y, train=False):\n        s.X, s.Y, s.train = X, Y.astype(np.float32), train\n\n    def __len__(s):\n        return len(s.X)\n\n    def __getitem__(s, i):\n        a = s.X[i].astype(np.float32) / 255.\n        if s.train:\n            if np.random.rand() < .5:\n                a = a[:, ::-1].copy()\n            a = np.clip(a + np.random.uniform(-.08, .08), 0, 1)\n        a = (a - MEAN) / STD\n        return torch.from_numpy(a).unsqueeze(0).repeat(3, 1, 1), \\\n            torch.from_numpy(s.Y[i])\n\n\nmk = lambda X, Y, sh, tr=False: DataLoader(\n    CXR(X, Y, tr), batch_size=BATCH, shuffle=sh, num_workers=2, pin_memory=True)\n\n\n# ============================================================ v13: OFFLINE WEIGHTS\n# Kaggle notebooks without Internet cannot download pretrained weights. Attach the\n# weight file (or, for RAD-DINO, the Hugging Face model folder) as a dataset; the\n# helper below finds it anywhere under /kaggle/input and puts it where torchvision\n# or transformers expect it. With Internet ON nothing needs to be attached.\n_TV_WEIGHTS = {\"densenet121\": \"DenseNet121_Weights\", \"convnext_tiny\": \"ConvNeXt_Tiny_Weights\",\n               \"vit_b_16\": \"ViT_B_16_Weights\"}\n\n\ndef ensure_offline_weights(arch):\n    if arch in _TV_WEIGHTS:\n        url = getattr(torchvision.models, _TV_WEIGHTS[arch]).IMAGENET1K_V1.url\n        fname = os.path.basename(url)\n        hub = os.path.join(torch.hub.get_dir(), \"checkpoints\")\n        dst = os.path.join(hub, fname)\n        if os.path.exists(dst):\n            print(f\"  weights ready: {dst}\"); return\n        found = glob.glob(f\"/kaggle/input/**/{fname}\", recursive=True)\n        if found:\n            os.makedirs(hub, exist_ok=True); shutil.copy(found[0], dst)\n            print(f\"  weights copied from attached input: {found[0]} -> {dst}\")\n        else:\n            print(f\"  weights '{fname}' not attached; will try to download (needs Internet ON).\\n\"\n                  f\"  If Internet is off: download {url}\\n  and attach it as a Kaggle dataset.\")\n        return None\n    if arch == \"raddino_linear\":\n        for cfg in glob.glob(\"/kaggle/input/**/config.json\", recursive=True):\n            d = os.path.dirname(cfg)\n            try:\n                txt = open(cfg).read().lower()\n            except Exception:\n                continue\n            if \"dinov2\" in txt and (glob.glob(f\"{d}/*.safetensors\") or glob.glob(f\"{d}/*.bin\")):\n                print(f\"  RAD-DINO found in attached input: {d}\")\n                return d\n        print(\"  RAD-DINO not attached; will download microsoft/rad-dino (needs Internet ON).\")\n        return \"microsoft/rad-dino\"\n\n\nRADDINO_SOURCE = ensure_offline_weights(ARCH)\n\n# ============================================================ v13: MODELS\nRADDINO_DIM = 768        # overwritten by the real encoder output size when FROZEN\n\n\ndef build_model(arch, n_out):\n    \"\"\"Trainable network for image-input architectures.\"\"\"\n    if arch == \"densenet121\":\n        m = torchvision.models.densenet121(weights=\"IMAGENET1K_V1\")\n        m.classifier = nn.Linear(m.classifier.in_features, n_out)\n    elif arch == \"convnext_tiny\":\n        m = torchvision.models.convnext_tiny(weights=\"IMAGENET1K_V1\")\n        m.classifier[2] = nn.Linear(m.classifier[2].in_features, n_out)\n    elif arch == \"vit_b_16\":\n        m = torchvision.models.vit_b_16(weights=\"IMAGENET1K_V1\")\n        m.heads.head = nn.Linear(m.heads.head.in_features, n_out)\n    elif arch == \"raddino_linear\":\n        m = nn.Linear(RADDINO_DIM, n_out)      # trained on cached frozen features\n    else:\n        raise ValueError(arch)\n    return m\n\n\nclass RadDinoEncoder(nn.Module):\n    \"\"\"Frozen RAD-DINO. Receives images in THIS pipeline's normalisation\n    (grey (x-MEAN)/STD repeated to 3 channels), converts them to RAD-DINO's own\n    normalisation and input size, and returns the CLS embedding.\"\"\"\n\n    def __init__(self, repo=\"microsoft/rad-dino\"):\n        super().__init__()\n        from transformers import AutoModel, AutoImageProcessor\n        proc = AutoImageProcessor.from_pretrained(repo)\n        self.backbone = AutoModel.from_pretrained(repo).eval()\n        for p_ in self.backbone.parameters():\n            p_.requires_grad = False\n        size = getattr(proc, \"crop_size\", None) or getattr(proc, \"size\", None) or {\"height\": 518}\n        self.side = int(size.get(\"height\", size.get(\"shortest_edge\", 518))) if isinstance(size, dict) else int(size)\n        self.register_buffer(\"mean\", torch.tensor(proc.image_mean, dtype=torch.float32).view(1, 3, 1, 1))\n        self.register_buffer(\"std\", torch.tensor(proc.image_std, dtype=torch.float32).view(1, 3, 1, 1))\n\n    @torch.no_grad()\n    def forward(self, x):\n        x = x * STD + MEAN                                   # back to [0, 1]\n        x = torch.nn.functional.interpolate(x, size=(self.side, self.side), mode=\"bilinear\", align_corners=False)\n        x = (x - self.mean) / self.std\n        return self.backbone(pixel_values=x).pooler_output\n\n\n\ndef frozen_features(enc, X, name):\n    \"\"\"CLS features for every image in X (same row order), cached on disk.\"\"\"\n    fp = f\"{CACHE}/{name}_feat_{ARCH}.npy\"\n    if os.path.exists(fp):\n        F = np.load(fp)\n        if len(F) == len(X):\n            print(f\"  [features] {name}: restored {F.shape} from cache\", flush=True)\n            return F\n    dl = DataLoader(CXR(X, np.zeros((len(X), 1)), False), batch_size=64, shuffle=False,\n                    num_workers=2, pin_memory=True)\n    out, t0 = [], time.time()\n    for i, (x, _) in enumerate(dl):\n        with torch.autocast(\"cuda\", enabled=DEV == \"cuda\", dtype=torch.float16):\n            out.append(enc(x.to(DEV, non_blocking=True)).float().cpu().numpy())\n        if i % 50 == 0:\n            print(f\"  [features] {name}: {min((i+1)*64, len(X)):,}/{len(X):,}  \"\n                  f\"{(time.time()-t0)/60:.1f} min\", flush=True)\n    F = np.concatenate(out).astype(np.float32)\n    np.save(fp, F)\n    print(f\"  [features] {name}: {F.shape} in {(time.time()-t0)/60:.1f} min\", flush=True)\n    return F\n\n\nclass FeatDS(Dataset):\n    def __init__(s, F, Y, train=False):\n        s.F, s.Y = torch.from_numpy(np.ascontiguousarray(F, dtype=np.float32)), torch.from_numpy(Y.astype(np.float32))\n\n    def __len__(s):\n        return len(s.F)\n\n    def __getitem__(s, i):\n        return s.F[i], s.Y[i]\n\n\nif FROZEN:\n    banner(\"SECTION 3b  RAD-DINO: extracting frozen features once (cached)\")\n    _enc = RadDinoEncoder(RADDINO_SOURCE or \"microsoft/rad-dino\").to(DEV)\n    with torch.no_grad():\n        _probe = _enc(torch.zeros(1, 3, 224, 224, device=DEV))\n    RADDINO_DIM = int(_probe.shape[-1])\n    print(f\"  RAD-DINO input {_enc.side}px, feature dim {RADDINO_DIM}\")\n    # Replace the image arrays by feature arrays with IDENTICAL row order, so every\n    # later slice (Xn[split == ...], Xv[vin_cal], Xc[chex_eval]) works unchanged.\n    Xn = frozen_features(_enc, Xn, \"nih\")\n    Xv = frozen_features(_enc, Xv, \"vindr\")\n    if HAVE_CHEX:\n        Xc = frozen_features(_enc, Xc, \"chexpert\")\n    del _enc\n    if DEV == \"cuda\":\n        torch.cuda.empty_cache()\n    mk = lambda X, Y, sh, tr=False: DataLoader(FeatDS(X, Y, tr), batch_size=BATCH, shuffle=sh,\n                                                num_workers=0, pin_memory=True)\nprint(f\"  architecture: {ARCH} | lr {LR} | epochs {EPOCHS} | batch {BATCH} | frozen features: {FROZEN}\")\n\n# Report how many evaluation positives each pathology will have per site, so\n# small-n pathologies are visible before any modelling.\nrows = {\"VinDr r1\": Yv[\"r1\"][vin_eval].sum(0),\n        \"VinDr r2\": Yv[\"r2\"][vin_eval].sum(0)}\nif HAVE_CHEX:\n    rows[\"CheXpert\"] = np.where(Yc_all[chex_eval].min(0) < 0, -1,\n                                Yc_all[chex_eval].sum(0))\nev = pd.DataFrame(rows, index=FINDINGS)\nev.insert(0, \"tier\", [\"A\" if f in SHARED else \"B\" for f in FINDINGS])\nprint(\"\\n  Evaluation positives per finding (-1 = not labelled at that site):\")\nprint(ev.to_string())\nprint(f\"  A finding is reported at a site only with >= {MIN_POS_EVAL} \"\n      f\"evaluation positives.\")\n\n# ============================================================ METRICS\nsig = lambda z: 1. / (1. + np.exp(-z))\n\n\ndef nbins(y, per=15, lo=4, hi=15):\n    return int(np.clip(max(int(y.sum()), 1) // per, lo, hi))\n\n\ndef ece(p, y, bins=None):\n    \"\"\"Quantile-binned ECE. Equal-width bins are unstable when positives are\n    scarce. Reported as SECONDARY: ECE is prevalence-weighted and therefore\n    falls as a disease becomes rarer, which would make a model look better\n    abroad purely because the disease is rarer there.\"\"\"\n    n = len(y)\n    if n == 0:\n        return np.nan\n    e = np.unique(np.quantile(p, np.linspace(0, 1, (bins or nbins(y)) + 1)))\n    if len(e) < 3:\n        return abs(y.mean() - p.mean())\n    out = 0.\n    for i in range(len(e) - 1):\n        m = ((p >= e[i]) if i == 0 else (p > e[i])) & (p <= e[i + 1])\n        if m.sum():\n            out += (m.sum() / n) * abs(y[m].mean() - p[m].mean())\n    return out\n\n\ndef boot_ci(fn, p, y, n_boot=N_BOOT, seed=0):\n    r, n, v = np.random.RandomState(seed), len(y), []\n    for _ in range(n_boot):\n        i = r.randint(0, n, n)\n        if y[i].sum() >= 3:\n            v.append(fn(p[i], y[i]))\n    return (float(np.percentile(v, 2.5)), float(np.percentile(v, 97.5))) \\\n        if v else (np.nan, np.nan)\n\n\ndef slope_icpt(p, y):\n    \"\"\"Fit y ~ a + b*logit(p). Perfect calibration is (a=0, b=1).\n    PRIMARY endpoint: prevalence-robust, unlike ECE, and standard in clinical\n    prediction-model reporting (TRIPOD).\"\"\"\n    pc = np.clip(p, 1e-7, 1 - 1e-7)\n    lr = LogisticRegression(C=1e10, solver=\"lbfgs\", max_iter=2000).fit(\n        np.log(pc / (1 - pc)).reshape(-1, 1), y)\n    return float(lr.intercept_[0]), float(lr.coef_[0][0])\n\n\ndef fit_platt(L, Y, min_pos=10):\n    \"\"\"Per-pathology affine map on the logit scale: z -> a + b*z. Corrects both\n    level and sharpness, so it also undoes class weighting if present.\"\"\"\n    A, B = [], []\n    for k in range(L.shape[1]):\n        y = Y[:, k]\n        if y.sum() < min_pos or y.sum() == len(y):\n            A.append(0.); B.append(1.); continue\n        lr = LogisticRegression(C=1e10, solver=\"lbfgs\", max_iter=2000).fit(\n            L[:, k].reshape(-1, 1), y)\n        B.append(float(lr.coef_[0][0])); A.append(float(lr.intercept_[0]))\n    return np.array(A), np.array(B)\n\n\ndef fit_T(L, Y, iters=200):\n    T = []\n    for k in range(L.shape[1]):\n        y = Y[:, k]\n        if y.sum() < 3 or y.sum() == len(y):\n            T.append(1.); continue\n        z = torch.tensor(L[:, k], dtype=torch.float32)\n        t = torch.tensor(y, dtype=torch.float32)\n        lg = torch.zeros(1, requires_grad=True)\n        o = torch.optim.LBFGS([lg], lr=.1, max_iter=iters)\n        lf = nn.BCEWithLogitsLoss()\n\n        def cl():\n            o.zero_grad(); l = lf(z / torch.exp(lg), t); l.backward(); return l\n        o.step(cl)\n        T.append(float(np.clip(np.exp(lg.item()), .05, 50.)))\n    return np.array(T)\n\n\ndef prior_shift(ps, pt):\n    \"\"\"Closed-form logit offset for a change in class prior (Saerens 2002).\n    Requires ONLY the two prevalences — no labelled target images. This is the\n    zero-annotation baseline that label-based methods must beat.\"\"\"\n    ps, pt = np.clip(ps, 1e-6, 1 - 1e-6), np.clip(pt, 1e-6, 1 - 1e-6)\n    return np.log(pt / (1 - pt)) - np.log(ps / (1 - ps))\n\n\ndef em_prior(L, src_prev, iters=200, tol=1e-7):\n    \"\"\"Estimate target prevalence WITHOUT ANY LABELS (Saerens et al. 2002).\n    Uses only the model's predictions on unlabelled target images. It relies\n    on the source model being calibrated, which the in-domain Platt step\n    ensures; calibrated-model + EM is the combination Alexandari et al. (2020)\n    found hard to beat for label shift.\"\"\"\n    P = np.clip(sig(L), 1e-7, 1 - 1e-7)\n    ps = np.clip(src_prev, 1e-6, 1 - 1e-6)\n    pt = ps.copy()\n    for _ in range(iters):\n        w1, w0 = pt / ps, (1 - pt) / (1 - ps)\n        post = w1 * P / (w1 * P + w0 * (1 - P))\n        new = post.mean(0)\n        if np.max(np.abs(new - pt)) < tol:\n            pt = new\n            break\n        pt = new\n    return np.clip(pt, 1e-5, 1 - 1e-5)\n\n\ndef ece_vec(Z, Y):\n    return np.array([ece(sig(Z[:, k]), Y[:, k]) for k in range(Z.shape[1])])\n\n\n# ============================================================ SECTION 4-6\nRES = {}          # seed -> results\n\nfor seed in SEEDS:\n    banner(f\"SECTION 4  Train  (seed {seed} of {SEEDS})\")\n    torch.manual_seed(seed)\n    np.random.seed(seed)\n\n    model = build_model(ARCH, len(FINDINGS)).to(DEV)\n\n    # UNWEIGHTED loss. Inverse-prevalence class weighting raises calibration\n    # error roughly tenfold while leaving AUROC unchanged — a silent failure\n    # documented in the pilot and reported as a side finding.\n    crit = nn.BCEWithLogitsLoss()\n    opt = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay=1e-4)\n    sch = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=EPOCHS)\n    try:\n        from torch.amp import autocast as _ac, GradScaler as _gs\n        amp = lambda: _ac(\"cuda\", enabled=DEV == \"cuda\")\n        scaler = _gs(\"cuda\", enabled=DEV == \"cuda\")\n    except Exception:\n        amp = lambda: torch.cuda.amp.autocast(enabled=DEV == \"cuda\")\n        scaler = torch.cuda.amp.GradScaler(enabled=DEV == \"cuda\")\n\n    _tr = np.where(split == \"train\")[0]\n    if SMOKE_TEST:\n        _tr = _tr[:2000]\n    tr_dl = mk(Xn[_tr], Yn[_tr], True, True)\n    va_dl = mk(Xn[split == \"val\"], Yn[split == \"val\"], False)\n    te_dl = mk(Xn[split == \"test\"], Yn[split == \"test\"], False)\n\n    @torch.no_grad()\n    def infer(dl):\n        \"\"\"Returns raw LOGITS. Calibration cannot use post-sigmoid values.\"\"\"\n        model.eval(); L, Y = [], []\n        for x, y in dl:\n            x = x.to(DEV, non_blocking=True)\n            with amp():\n                L.append(model(x).float().cpu().numpy())\n            Y.append(y.numpy())\n        return np.concatenate(L), np.concatenate(Y)\n\n    best = -1.\n    for ep in range(EPOCHS):\n        model.train(); tot = 0.\n        for x, y in tr_dl:\n            x, y = x.to(DEV, non_blocking=True), y.to(DEV, non_blocking=True)\n            opt.zero_grad(set_to_none=True)\n            with amp():\n                loss = crit(model(x), y)\n            scaler.scale(loss).backward(); scaler.step(opt); scaler.update()\n            tot += loss.item() * len(x)\n        sch.step()\n        lv, yv = infer(va_dl)\n        m = float(np.mean([roc_auc_score(yv[:, k], lv[:, k])\n                           for k in range(len(FINDINGS))\n                           if 0 < yv[:, k].sum() < len(yv)]))\n        print(f\"  {el()} seed {seed} epoch {ep+1}/{EPOCHS}  \"\n              f\"loss {tot/max(len(_tr),1):.4f}  \"\n              f\"val mAUROC {m:.4f}\", flush=True)\n        if m > best:\n            best = m\n            torch.save(model.state_dict(), f\"{W}/best_s{seed}.pt\")\n    model.load_state_dict(torch.load(f\"{W}/best_s{seed}.pt\"))\n    print(f\"  seed {seed}: best val mAUROC {best:.4f}\")\n\n    # -------------------------------------------------- inference\n    L_nv, Y_nv = infer(va_dl)\n    L_nt, Y_nt = infer(te_dl)\n    L_vc, _ = infer(mk(Xv[vin_cal], Yv[\"r2\"][vin_cal], False))\n    L_ve, _ = infer(mk(Xv[vin_eval], Yv[\"r2\"][vin_eval], False))\n    if HAVE_CHEX:\n        L_cc, _ = infer(mk(Xc[chex_cal], np.clip(Yc_all[chex_cal], 0, 1), False))\n        L_ce, _ = infer(mk(Xc[chex_eval], np.clip(Yc_all[chex_eval], 0, 1),\n                           False))\n\n    # -------------------------------------------------- in-domain calibration\n    # A hospital calibrates on its OWN validation data before deployment.\n    # Fit on one half, freeze the threshold on the disjoint half, so the map\n    # and the threshold are never chosen on the same rows.\n    rng = np.random.RandomState(seed)\n    vp = rng.permutation(len(L_nv)); h = len(vp) // 2\n    IFIT, ITHR = vp[:h], vp[h:]\n    CAL_A, CAL_B = fit_platt(L_nv[IFIT], Y_nv[IFIT])\n    cal = lambda L: CAL_A[None, :] + CAL_B[None, :] * L\n    L_nv, L_nt, L_vc, L_ve = map(cal, (L_nv, L_nt, L_vc, L_ve))\n    if HAVE_CHEX:\n        L_cc, L_ce = cal(L_cc), cal(L_ce)\n\n    Pthr, Ythr = sig(L_nv[ITHR]), Y_nv[ITHR]\n    grid, THR = np.linspace(.01, .99, 197), []\n    for k in range(len(FINDINGS)):\n        f1 = []\n        for t in grid:\n            pr = (Pthr[:, k] >= t).astype(int)\n            tp = ((pr == 1) & (Ythr[:, k] == 1)).sum()\n            fp = ((pr == 1) & (Ythr[:, k] == 0)).sum()\n            fn = ((pr == 0) & (Ythr[:, k] == 1)).sum()\n            f1.append(2 * tp / max(2 * tp + fp + fn, 1))\n        THR.append(float(grid[int(np.argmax(f1))]))\n    THR = np.array(THR)\n\n    def per_path(L, Y, min_pos, names):\n        \"\"\"Per-finding metrics. L, Y and names are aligned column-for-column.\"\"\"\n        P, rows = sig(L), []\n        for k, t in enumerate(names):\n            y, p = Y[:, k], P[:, k]\n            if y.min() < 0 or y.sum() < min_pos or y.sum() == len(y):\n                continue\n            thr = THR[FINDINGS.index(t)]\n            pr = (p >= thr).astype(int)\n            tp = ((pr == 1) & (y == 1)).sum(); fn_ = ((pr == 0) & (y == 1)).sum()\n            tn = ((pr == 0) & (y == 0)).sum(); fp_ = ((pr == 1) & (y == 0)).sum()\n            a, b = slope_icpt(p, y)\n            lo, hi = boot_ci(ece, p, y)\n            rows.append({\"finding\": t, \"tier\": \"A\" if t in SHARED else \"B\",\n                         \"n_pos\": int(y.sum()), \"prevalence\": y.mean(),\n                         \"AUROC\": roc_auc_score(y, p),\n                         \"intercept\": a, \"slope\": b,\n                         \"OE\": y.mean() / max(p.mean(), 1e-9),\n                         \"ECE\": ece(p, y), \"ECE_lo\": lo, \"ECE_hi\": hi,\n                         \"Brier\": brier_score_loss(y, p),\n                         \"mean_pred\": p.mean(),\n                         \"sensitivity\": tp / max(tp + fn_, 1),\n                         \"specificity\": tn / max(tn + fp_, 1)})\n        return pd.DataFrame(rows).set_index(\"finding\") if rows \\\n            else pd.DataFrame()\n\n    ref = per_path(L_nt, Y_nt, 20, FINDINGS)\n\n    def gate(df):\n        return bool(((df.intercept.abs() < .6) &\n                     df.slope.between(.7, 1.4)).all()) if len(df) else False\n    gate_A = gate(ref[ref.tier == \"A\"])\n    gate_B = gate(ref[ref.tier == \"B\"])\n    print(f\"\\n  IN-DOMAIN REFERENCE (seed {seed}) — expect intercept~0, slope~1\")\n    print(ref[[\"tier\", \"n_pos\", \"prevalence\", \"AUROC\", \"intercept\", \"slope\",\n               \"OE\", \"ECE\", \"mean_pred\"]].round(3).to_string())\n    print(f\"  GATE tier A (primary): {'PASSED' if gate_A else '*** FAILED ***'}\"\n          f\" | GATE tier B: {'PASSED' if gate_B else 'failed'}\"\n          f\" | mean in-domain ECE {ref.ECE.mean():.4f}\")\n    if not gate_A:\n        print(\"  Tier-A baseline broken: cross-site numbers are not meaningful.\")\n    if gate_A and not gate_B:\n        print(\"  Tier-B findings are rarer at NIH (fibrosis ~1.5%), so their\")\n        print(\"  in-domain calibration estimates are noisier. Interpret tier-B\")\n        print(\"  results with that in mind; tier-A conclusions are unaffected.\")\n\n    # -------------------------------------------------- analysis per site\n    def analyse(site, Lc, Yc_, Le, Ye_, names):\n        cols = [FINDINGS.index(f) for f in names]\n        Lc, Yc_, Le, Ye_ = Lc[:, cols], Yc_[:, cols], Le[:, cols], Ye_[:, cols]\n        vr = per_path(Le, Ye_, MIN_POS_EVAL, names)\n        if vr.empty:\n            print(f\"\\n  {site}: no finding reaches {MIN_POS_EVAL} positives\")\n            return None\n        common = [f for f in names if f in vr.index and f in ref.index]\n        D = pd.DataFrame(index=common)\n        D[\"tier\"] = [\"A\" if f in SHARED else \"B\" for f in common]\n        D[\"prev_src\"] = ref.loc[common, \"prevalence\"]\n        D[\"prev_tgt\"] = vr.loc[common, \"prevalence\"]\n        D[\"prev_ratio\"] = (D.prev_tgt / D.prev_src).round(3)\n        D[\"absLogRatio\"] = np.abs(np.log(D.prev_ratio)).round(3)\n        D[\"AUROC_src\"] = ref.loc[common, \"AUROC\"]\n        D[\"AUROC_tgt\"] = vr.loc[common, \"AUROC\"]\n        D[\"dAUROC\"] = (D.AUROC_tgt - D.AUROC_src).round(4)\n        D[\"intercept\"] = vr.loc[common, \"intercept\"]\n        D[\"slope\"] = vr.loc[common, \"slope\"]\n        D[\"|icpt|\"] = D.intercept.abs()\n        D[\"|slope-1|\"] = (D.slope - 1).abs()\n        D[\"ECE_src\"] = ref.loc[common, \"ECE\"]\n        D[\"ECE_tgt\"] = vr.loc[common, \"ECE\"]\n        D[\"dSens\"] = (vr.loc[common, \"sensitivity\"]\n                      - ref.loc[common, \"sensitivity\"]).round(3)\n        D[\"dSpec\"] = (vr.loc[common, \"specificity\"]\n                      - ref.loc[common, \"specificity\"]).round(3)\n\n        # CLAIM 1: level error vs sharpness error, ranking preserved\n        lvl, shp = D[\"|icpt|\"].mean(), D[\"|slope-1|\"].mean()\n        print(f\"\\n  --- {site} (seed {seed}, {len(common)} findings) ---\")\n        print(D[[\"tier\", \"prev_ratio\", \"AUROC_src\", \"AUROC_tgt\", \"dAUROC\",\n                 \"intercept\", \"slope\", \"ECE_src\", \"ECE_tgt\"]].round(3)\n              .to_string())\n        print(f\"  CLAIM 1  mean |intercept| {lvl:.3f} vs mean |slope-1| \"\n              f\"{shp:.3f}  ->  level error {lvl/max(shp,1e-9):.1f}x sharpness\")\n        print(f\"           mean dAUROC {D.dAUROC.mean():+.4f} \"\n              f\"({int((D.dAUROC>0).sum())}/{len(D)} findings improved)\")\n\n        # ------------------------------------------------------------------\n        # CLAIM 2: corrections.  Two zero-label variants and two label-based.\n        #   2  prior, KNOWN prevalence  (0 labels; rate from hospital records)\n        #   2e prior, EM prevalence     (0 labels, 0 records: fully label-free)\n        #   3  temperature              (labels; sharpness only - cannot fix a\n        #                                level error, so it is NOT the fair rival)\n        #   3p Platt                    (labels; fixes level AND sharpness -\n        #                                the fair label-based competitor)\n        #   4  prior + temperature      (known rate + labels)\n        # ------------------------------------------------------------------\n        keep = [names.index(x) for x in common]\n        n_cal = len(Lc)\n        src_prev = Y_nt[:, cols].mean(0)\n        tgt_prev = Yc_.mean(0)                     # the \"registry\" value\n        em_prev = em_prior(Lc, src_prev)           # predictions only, no labels\n        OFF = prior_shift(src_prev, tgt_prev)\n        OFF_em = prior_shift(src_prev, em_prev)\n        T_par = fit_T(Lc, Yc_)\n        T_seq = fit_T(Lc + OFF[None, :], Yc_)\n        PA, PB = fit_platt(Lc, Yc_, min_pos=3)\n        D[\"prev_em\"] = em_prev[keep]\n        D[\"em_rel_err_%\"] = (100 * (em_prev[keep] - tgt_prev[keep])\n                             / np.maximum(tgt_prev[keep], 1e-9)).round(1)\n        Z = {\"1 uncorrected\": Le,\n             \"2 prior, known rate (0 labels)\": Le + OFF[None, :],\n             \"2e prior, EM rate (0 labels)\": Le + OFF_em[None, :],\n             \"3 temperature (labels)\": Le / T_par[None, :],\n             \"3p Platt (labels)\": PA[None, :] + PB[None, :] * Le,\n             \"4 prior+temp (labels)\": (Le + OFF[None, :]) / T_seq[None, :]}\n        conds = {k: ece_vec(v, Ye_) for k, v in Z.items()}\n        C = pd.DataFrame({k: v[keep] for k, v in conds.items()}, index=common)\n        C.loc[\"MEAN\"] = C.mean()\n        ind = ref.loc[common, \"ECE\"].mean()\n        base = C.loc[\"MEAN\", \"1 uncorrected\"]\n        gapclose = {k: 100 * (base - C.loc[\"MEAN\", k]) / max(base - ind, 1e-9)\n                    for k in list(conds)[1:]}\n        short = {\"2 prior, known rate (0 labels)\": \"free\",\n                 \"2e prior, EM rate (0 labels)\": \"freeEM\",\n                 \"3 temperature (labels)\": \"temp\", \"3p Platt (labels)\": \"platt\",\n                 \"4 prior+temp (labels)\": \"both\"}\n        print(f\"\\n  ECE BY CORRECTION (label-based methods use all {n_cal:,} \"\n              f\"calibration labels)\")\n        print(C.round(4).to_string())\n        print(f\"  gap to in-domain ({ind:.4f}) closed:  \" + \"  \".join(\n            f\"{short[k]}={v:.0f}%\" for k, v in gapclose.items()))\n        print(\"  EM prevalence estimate vs true (relative error %): \" + \", \".join(\n            f\"{f} {D.loc[f, 'em_rel_err_%']:+.0f}\" for f in common))\n\n        # Paired bootstrap over EVALUATION images: is the free fix really\n        # better/worse than each label-based method, or is it noise?\n        rngb = np.random.RandomState(7)\n        pairs = [(\"free\", \"2 prior, known rate (0 labels)\", \"3p Platt (labels)\"),\n                 (\"free\", \"2 prior, known rate (0 labels)\", \"3 temperature (labels)\"),\n                 (\"freeEM\", \"2e prior, EM rate (0 labels)\", \"3p Platt (labels)\")]\n        cmp_rows = []\n        needed = sorted({a for _, a, _ in pairs} | {b for _, _, b in pairs})\n        diffs = {(a, b): [] for _, a, b in pairs}\n        for _ in range(N_BOOT_DIFF):\n            ix = rngb.randint(0, len(Le), len(Le))\n            e = {k: ece_vec(Z[k][ix], Ye_[ix])[keep].mean() for k in needed}\n            for _, a, b in pairs:\n                diffs[(a, b)].append(e[a] - e[b])\n        for lab, a, b in pairs:\n            d = np.array(diffs[(a, b)])\n            lo, hi = np.percentile(d, [2.5, 97.5])\n            verdict = (\"free BETTER\" if hi < 0 else\n                       \"free WORSE\" if lo > 0 else \"no significant difference\")\n            cmp_rows.append({\"comparison\": f\"{lab} vs {short[b]}\",\n                             \"mean_ECE_diff\": d.mean(), \"ci_lo\": lo,\n                             \"ci_hi\": hi, \"verdict\": verdict})\n        CMP = pd.DataFrame(cmp_rows)\n        print(\"\\n  FREE vs LABEL-BASED (paired bootstrap over evaluation images;\"\n              \" negative = free fix better)\")\n        print(CMP.round(4).to_string(index=False))\n\n        # How accurately must the hospital know its disease rate?\n        sens = []\n        for f_ in PREV_ERRORS:\n            pt = np.clip(tgt_prev * f_, 1e-5, .99)\n            e = ece_vec(Le + prior_shift(src_prev, pt)[None, :], Ye_)[keep].mean()\n            sens.append({\"rate_error\": f\"{(f_-1)*100:+.0f}%\", \"mean_ECE\": e,\n                         \"gap_closed_%\": 100 * (base - e) / max(base - ind, 1e-9)})\n        SENS = pd.DataFrame(sens)\n        print(\"\\n  ROBUSTNESS: free fix when the recorded disease rate is wrong\")\n        print(SENS.round(4).to_string(index=False))\n\n        # ------------------------------------------------------------------\n        # CLAIM 3: label budget.  Each method is fitted on n labelled images\n        # and compared with the free fix (known rate, 0 labels).  A verdict\n        # needs BOTH a significant mean difference AND a consistent win rate\n        # across draws, because the draws share one evaluation set.\n        # ------------------------------------------------------------------\n        floor = C.loc[\"MEAN\", \"2 prior, known rate (0 labels)\"]\n        brows = []\n        for n in BUDGETS:\n            if n > n_cal:\n                continue\n            draws = [np.random.RandomState(2000 + r).choice(n_cal, n, replace=False)\n                     for r in range(REPEATS)]\n            for meth in (\"Platt\", \"prior+temp\"):\n                runs = []\n                for i_ in draws:\n                    if meth == \"Platt\":\n                        a_, b_ = fit_platt(Lc[i_], Yc_[i_], min_pos=3)\n                        z_ = a_[None, :] + b_[None, :] * Le\n                    else:\n                        t_ = fit_T(Lc[i_] + OFF[None, :], Yc_[i_])\n                        z_ = (Le + OFF[None, :]) / t_[None, :]\n                    runs.append(ece_vec(z_, Ye_)[keep].mean())\n                runs = np.array(runs)\n                d = runs.mean() - floor\n                se = runs.std(ddof=1) / np.sqrt(len(runs))\n                t = d / se if se > 0 else np.nan\n                win = float((runs < floor).mean())\n                brows.append({\"method\": meth, \"n_labels\": n,\n                              \"mean_ECE\": runs.mean(), \"sd\": runs.std(ddof=1),\n                              \"vs_free_%\": -100 * d / max(floor, 1e-9),\n                              \"win_rate\": win, \"t\": t,\n                              \"verdict\": (\"better\" if (t < -2.09 and win >= .75)\n                                          else \"worse\" if (t > 2.09 and win <= .25)\n                                          else \"no clear difference\")})\n        B = pd.DataFrame(brows)\n        print(f\"\\n  ANNOTATION BUDGET vs the free fix (floor {floor:.4f}); \"\n              f\"win_rate = share of draws beating it\")\n        print(B.round(4).to_string(index=False))\n        cross = {}\n        for meth in (\"Platt\", \"prior+temp\"):\n            bb = B[(B.method == meth) & (B.verdict == \"better\")][\"n_labels\"]\n            cross[meth] = int(bb.min()) if len(bb) else None\n            print(f\"  CLAIM 3  {meth:10s} first beats the free fix at n = \"\n                  f\"{cross[meth] if cross[meth] else 'never (up to ' + str(max(B.n_labels)) + ')'}\")\n\n        # CLAIM 4: does prevalence shift predict benefit? (n = findings)\n        gain = 100 * (C.loc[common, \"1 uncorrected\"]\n                      - C.loc[common, \"2 prior, known rate (0 labels)\"]) \\\n            / C.loc[common, \"1 uncorrected\"]\n        rho = D[\"absLogRatio\"].corr(gain, method=\"spearman\") \\\n            if len(common) >= 3 else np.nan\n        print(f\"  CLAIM 4  Spearman(|log prevalence ratio|, free-fix gain) \"\n              f\"= {rho:+.3f}  (n={len(common)})\")\n\n        tag = site.replace(\" \", \"\").replace(\"[\", \"_\").replace(\"]\", \"\")\n        D.to_csv(f\"{W}/damage_{tag}_s{seed}.csv\")\n        C.to_csv(f\"{W}/corrections_{tag}_s{seed}.csv\")\n        B.to_csv(f\"{W}/budget_{tag}_s{seed}.csv\", index=False)\n        CMP.to_csv(f\"{W}/free_vs_labels_{tag}_s{seed}.csv\", index=False)\n        SENS.to_csv(f\"{W}/rate_robustness_{tag}_s{seed}.csv\", index=False)\n        out = {\"damage\": D, \"gain\": gain, \"corrections\": C, \"budget\": B,\n               \"cmp\": CMP, \"sens\": SENS, \"gapclose\": gapclose,\n               \"rho\": float(rho), \"n\": len(common), \"level\": float(lvl),\n               \"sharp\": float(shp), \"dAUROC\": float(D.dAUROC.mean()),\n               \"cross\": cross, \"findings\": common,\n               \"em_abs_err\": float(np.abs(D[\"em_rel_err_%\"]).mean())}\n        if seed == SEEDS[0]:          # keep predictions for reliability plots\n            out[\"rel\"] = {\"Le\": Le[:, keep], \"Ye\": Ye_[:, keep],\n                          \"OFF\": OFF[keep], \"OFF_em\": OFF_em[keep],\n                          \"names\": common}\n        return out\n\n    # Each site analysed on its own finding list. VinDr twice: on all nine\n    # (tier A+B, for Claim 4 power) and on the five shared (tier A, directly\n    # comparable to CheXpert).\n    JOBS = []\n    for rule in (\"r1\", \"r2\"):\n        JOBS.append((f\"VinDr-{rule} [9]\", L_vc, Yv[rule][vin_cal], L_ve,\n                     Yv[rule][vin_eval], FINDINGS))\n        JOBS.append((f\"VinDr-{rule} [5]\", L_vc, Yv[rule][vin_cal], L_ve,\n                     Yv[rule][vin_eval], SHARED))\n    if HAVE_CHEX:\n        JOBS.append((\"CheXpert [5]\", L_cc, Yc_all[chex_cal], L_ce,\n                     Yc_all[chex_eval], SHARED))\n    site_res = {}\n    for job in JOBS:\n        r = analyse(*job)\n        if r is not None:\n            site_res[job[0]] = r\n\n    np.savez_compressed(f\"{W}/logits_{'SMOKE_' if SMOKE_TEST else ''}s{seed}.npz\",\n                        L_nih_val=L_nv, Y_nih_val=Y_nv,\n                        L_nih_test=L_nt, Y_nih_test=Y_nt,\n                        L_vin_cal=L_vc, L_vin_eval=L_ve,\n                        Yv_r1_cal=Yv[\"r1\"][vin_cal], Yv_r1_eval=Yv[\"r1\"][vin_eval],\n                        Yv_r2_cal=Yv[\"r2\"][vin_cal], Yv_r2_eval=Yv[\"r2\"][vin_eval],\n                        cal_a=CAL_A, cal_b=CAL_B, thr=THR,\n                        findings=np.array(FINDINGS),\n                        arch=np.array(ARCH),\n                        # v13: identifiers, row-aligned with every array above\n                        ids_nih_val=IDXn[\"image_id\"].astype(str).values[split == \"val\"],\n                        pid_nih_val=IDXn[\"patient_id\"].astype(str).values[split == \"val\"],\n                        ids_nih_test=IDXn[\"image_id\"].astype(str).values[split == \"test\"],\n                        pid_nih_test=IDXn[\"patient_id\"].astype(str).values[split == \"test\"],\n                        ids_vin_cal=IDXv[\"image_id\"].astype(str).values[vin_cal],\n                        ids_vin_eval=IDXv[\"image_id\"].astype(str).values[vin_eval],\n                        **({\"ids_chex_cal\": IDXc[\"image_id\"].astype(str).values[chex_cal],\n                            \"pid_chex_cal\": IDXc[\"patient_id\"].astype(str).values[chex_cal],\n                            \"ids_chex_eval\": IDXc[\"image_id\"].astype(str).values[chex_eval],\n                            \"pid_chex_eval\": IDXc[\"patient_id\"].astype(str).values[chex_eval]}\n                           if HAVE_CHEX else {}),\n                        **({\"L_chex_cal\": L_cc, \"L_chex_eval\": L_ce,\n                            \"Y_chex_cal\": Yc_all[chex_cal],\n                            \"Y_chex_eval\": Yc_all[chex_eval]}\n                           if HAVE_CHEX else {}))\n    RES[seed] = {\"best_val\": best, \"gate_A\": gate_A, \"gate_B\": gate_B,\n                 \"ref\": ref, \"sites\": site_res}\n    if seed == SEEDS[0]:\n        RES[seed][\"ref_pred\"] = (L_nt, Y_nt)\n    print(f\"  {el()} seed {seed} complete\")\n\n# ============================================================ SECTION 7\nbanner(\"SECTION 7  Across-seed summary — this is what goes in the paper\")\n\nprint(f\"Every figure is mean [min, max] across {len(SEEDS)} seeds. A claim is\")\nprint(\"asserted only if its range excludes the null value.\\n\")\n\n\ndef across(fn):\n    v = []\n    for s in SEEDS:\n        try:\n            x = fn(RES[s])\n        except (KeyError, TypeError):\n            x = None\n        if x is not None and not (isinstance(x, float) and np.isnan(x)):\n            v.append(x)\n    return (float(np.mean(v)), float(np.min(v)), float(np.max(v)), len(v)) \\\n        if v else (np.nan, np.nan, np.nan, 0)\n\n\ndef fmt(t, d=2, sign=False):\n    f = f\"{{:{'+' if sign else ''}.{d}f}}\"\n    return (f\"{f.format(t[0])} [{f.format(t[1])},{f.format(t[2])}]\"\n            if t[3] else \"n/a\")\n\n\nvm = across(lambda r: r[\"best_val\"])\nprint(f\"val mAUROC (9 findings)  {fmt(vm, 4)}\")\nprint(f\"gate tier A (primary)    \"\n      f\"{sum(RES[s]['gate_A'] for s in SEEDS)}/{len(SEEDS)} seeds passed\")\nprint(f\"gate tier B              \"\n      f\"{sum(RES[s]['gate_B'] for s in SEEDS)}/{len(SEEDS)} seeds passed\")\nprint(f\"in-domain ECE, tier A    \"\n      f\"{fmt(across(lambda r: r['ref'][r['ref'].tier=='A'].ECE.mean()), 4)}\")\nprint(f\"in-domain ECE, tier B    \"\n      f\"{fmt(across(lambda r: r['ref'][r['ref'].tier=='B'].ECE.mean()), 4)}\")\n\nsites = []\nfor s in SEEDS:\n    for k in RES[s][\"sites\"]:\n        if k not in sites:\n            sites.append(k)\n\nrows = []\nfor site in sites:\n    g = lambda key: across(lambda r: r[\"sites\"][site][key])\n    gc = lambda key: across(lambda r: r[\"sites\"][site][\"gapclose\"][key])\n    lvl, shp = g(\"level\"), g(\"sharp\")\n    present = [s for s in SEEDS if site in RES[s][\"sites\"]]\n\n    def verdicts(label):\n        v = [RES[s][\"sites\"][site][\"cmp\"].set_index(\"comparison\")\n             .loc[label, \"verdict\"] for s in present]\n        return (f\"{sum('BETTER' in x for x in v)}B/\"\n                f\"{sum('no significant' in x for x in v)}=\"\n                f\"/{sum('WORSE' in x for x in v)}W\")\n\n    def cr(meth):\n        return \" / \".join(\"never\" if RES[s][\"sites\"][site][\"cross\"][meth]\n                          is None else str(RES[s][\"sites\"][site][\"cross\"][meth])\n                          for s in present)\n    sens = {e: across(lambda r, e=e: r[\"sites\"][site][\"sens\"]\n                      .set_index(\"rate_error\").loc[e, \"gap_closed_%\"])[0]\n            for e in (\"-50%\", \"-25%\", \"+25%\", \"+50%\")}\n    rows.append({\n        \"site\": site, \"n\": f\"{int(g('n')[0])}\",\n        \"level/sharp\": f\"{lvl[0]/max(shp[0],1e-9):.1f}x\",\n        \"dAUROC\": f\"{g('dAUROC')[0]:+.3f}\",\n        \"free%\": f\"{gc('2 prior, known rate (0 labels)')[0]:.0f}\",\n        \"EM%\": f\"{gc('2e prior, EM rate (0 labels)')[0]:.0f}\",\n        \"temp%\": f\"{gc('3 temperature (labels)')[0]:.0f}\",\n        \"Platt%\": f\"{gc('3p Platt (labels)')[0]:.0f}\",\n        \"both%\": f\"{gc('4 prior+temp (labels)')[0]:.0f}\",\n        \"free vs Platt\": verdicts(\"free vs platt\"),\n        \"free vs temp\": verdicts(\"free vs temp\"),\n        \"EM vs Platt\": verdicts(\"freeEM vs platt\"),\n        \"EM err%\": f\"{g('em_abs_err')[0]:.0f}\",\n        \"Platt beats free at n\": cr(\"Platt\"),\n        \"rate +-25%\": f\"{sens['-25%']:.0f}/{sens['+25%']:.0f}\",\n        \"rate +-50%\": f\"{sens['-50%']:.0f}/{sens['+50%']:.0f}\",\n        \"rho\": fmt(g(\"rho\"), 2, sign=True)})\nSUM = pd.DataFrame(rows).set_index(\"site\")\npd.set_option(\"display.width\", 280)\npd.set_option(\"display.max_columns\", 30)\nprint(\"\\n\" + SUM.to_string())\nSUM.to_csv(f\"{W}/summary_across_seeds.csv\")\nprint(\"\"\"\n  HOW TO READ THIS TABLE\n   free%   gap closed by the prior fix with the KNOWN disease rate (0 labels)\n   EM%     same, with the rate estimated from predictions only (0 labels,\n           no records at all). This is the strictly label-free version.\n   temp% / Platt% / both%   label-based fixes using every calibration label.\n           Platt is the fair rival: it can fix level AND sharpness, while\n           temperature can only fix sharpness.\n   free vs Platt etc.   verdicts per seed from a paired bootstrap:\n           B = free significantly better, = no significant difference,\n           W = free significantly worse.\n   EM err%  mean absolute relative error of the EM disease-rate estimate.\n   Platt beats free at n   first label budget at which Platt reliably beats\n           the free fix, per seed. Never average this column.\n   rate +-25% / +-50%   gap closed by the free fix when the recorded disease\n           rate is wrong by that much (under/over).\n\"\"\")\n\n# ---------------- Claim 4 with pooled power ----------------\nfrom scipy.stats import spearmanr\n\n\ndef pooled_claim4(site):\n    parts = [pd.DataFrame({\"absLogRatio\": RES[s][\"sites\"][site][\"damage\"].absLogRatio,\n                           \"gain\": RES[s][\"sites\"][site][\"gain\"]})\n             for s in SEEDS if site in RES[s][\"sites\"]]\n    return pd.concat(parts).groupby(level=0).mean() if parts else None\n\n\nprint(\"\\n--- CLAIM 4: DOES THE SIZE OF THE PREVALENCE SHIFT PREDICT THE BENEFIT? ---\")\nC4 = {}\nfor site in (\"VinDr-r1 [9]\", \"VinDr-r2 [9]\", \"CheXpert [5]\"):\n    pl = pooled_claim4(site)\n    if pl is None or len(pl) < 3:\n        continue\n    r_, p_ = spearmanr(pl.absLogRatio, pl.gain)\n    C4[site] = pl\n    print(f\"  {site:14s} n = {len(pl)} findings | seed-averaged Spearman \"\n          f\"rho = {r_:+.3f}, p = {p_:.3f}\")\n    pl.to_csv(f\"{W}/claim4_{site.replace(' ','').replace('[','_').replace(']','')}.csv\")\nif \"VinDr-r1 [9]\" in C4 and \"CheXpert [5]\" in C4:\n    both = pd.concat([C4[\"VinDr-r1 [9]\"].assign(site=\"VinDr\"),\n                      C4[\"CheXpert [5]\"].assign(site=\"CheXpert\")])\n    r_, p_ = spearmanr(both.absLogRatio, both.gain)\n    print(f\"  BOTH HOSPITALS  n = {len(both)} finding-site pairs | Spearman \"\n          f\"rho = {r_:+.3f}, p = {p_:.4f}\")\n    print(\"  (pairs share one model, so treat this as supporting evidence;\")\n    print(\"   the per-site values above are the primary result)\")\n    both.to_csv(f\"{W}/claim4_both_sites.csv\")\n\n# ---------------- annotation-protocol effect ----------------\nif HAVE_CHEX and any(\"CheXpert [5]\" in RES[s][\"sites\"] for s in SEEDS):\n    print(\"\\n--- ANNOTATION-PROTOCOL EFFECT (the reason for three sites) ---\")\n    print(\"  Compared on the SAME findings at both sites, so the difference\")\n    print(\"  is not driven by which findings happened to survive filtering.\")\n    for rule in (\"r1\", \"r2\"):\n        kv = f\"VinDr-{rule} [5]\"\n        diffs = []\n        for s in SEEDS:\n            sv = RES[s][\"sites\"].get(kv)\n            sc = RES[s][\"sites\"].get(\"CheXpert [5]\")\n            if not sv or not sc:\n                continue\n            common = [f for f in sv[\"findings\"] if f in sc[\"findings\"]]\n            if not common:\n                continue\n            lv = sv[\"damage\"].loc[common, \"|icpt|\"].mean()\n            lc = sc[\"damage\"].loc[common, \"|icpt|\"].mean()\n            diffs.append((lv, lc, lv - lc, len(common)))\n        if diffs:\n            d = np.array(diffs)\n            print(f\"  {kv} vs CheXpert on {int(d[0,3])} shared findings: \"\n                  f\"level error {d[:,0].mean():.2f} vs {d[:,1].mean():.2f}; \"\n                  f\"difference {d[:,2].mean():+.2f} \"\n                  f\"[{d[:,2].min():+.2f}, {d[:,2].max():+.2f}]\")\n    print(\"  NIH->CheXpert shares the source's labelling method, so it isolates\")\n    print(\"  the institutional effect; the difference estimates the annotation\")\n    print(\"  effect.\")\nelse:\n    print(\"\\n  CheXpert absent, so institutional and annotation effects cannot\")\n    print(\"  be separated. Add CheXpert before writing up.\")\n\n# ============================================================ SECTION 8\nbanner(\"SECTION 8  Figures for the thesis and paper\")\nimport matplotlib\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\nFIG = f\"{W}/figures\"\nos.makedirs(FIG, exist_ok=True)\nplt.rcParams.update({\"font.size\": 9, \"axes.spines.top\": False,\n                     \"axes.spines.right\": False, \"figure.dpi\": 110})\n\n\ndef savefig(fig, name):\n    for ext in (\"png\", \"pdf\"):\n        fig.savefig(f\"{FIG}/{name}.{ext}\", dpi=300, bbox_inches=\"tight\")\n    plt.close(fig)\n    print(f\"  saved figures/{name}.png and .pdf\")\n\n\ndef rel_curve(p, y, bins=10):\n    q = np.unique(np.quantile(p, np.linspace(0, 1, bins + 1)))\n    xs, ys = [], []\n    for a, b in zip(q[:-1], q[1:]):\n        m = (p >= a) & (p <= b)\n        if m.sum() >= 20:\n            xs.append(p[m].mean()); ys.append(y[m].mean())\n    return np.array(xs), np.array(ys)\n\n\nPLOT_SITES = [x for x in (\"VinDr-r1 [9]\", \"CheXpert [5]\") if x in sites]\n\ntry:   # Figure 1: level vs sharpness error per finding\n    fig, axes = plt.subplots(1, len(PLOT_SITES), figsize=(5.2 * len(PLOT_SITES), 3.6),\n                             squeeze=False)\n    for ax, site in zip(axes[0], PLOT_SITES):\n        Ds = [RES[s][\"sites\"][site][\"damage\"] for s in SEEDS if site in RES[s][\"sites\"]]\n        lv = pd.concat([d[\"|icpt|\"] for d in Ds], axis=1).mean(1)\n        sh = pd.concat([d[\"|slope-1|\"] for d in Ds], axis=1).mean(1)\n        x = np.arange(len(lv))\n        ax.bar(x - .2, lv.values, .4, label=\"level error |intercept|\")\n        ax.bar(x + .2, sh.values, .4, label=\"sharpness error |slope - 1|\")\n        ax.set_xticks(x); ax.set_xticklabels(lv.index, rotation=45, ha=\"right\")\n        ax.set_title(f\"NIH -> {site}\"); ax.set_ylabel(\"calibration error component\")\n        ax.legend(fontsize=7)\n    savefig(fig, \"fig1_level_vs_sharpness\")\nexcept Exception as e:\n    print(\"  fig1 skipped:\", e)\n\ntry:   # Figure 2: reliability diagrams, first seed\n    for site in PLOT_SITES:\n        rr = RES[SEEDS[0]][\"sites\"][site][\"rel\"]\n        k = len(rr[\"names\"])\n        fig, axes = plt.subplots(1, k, figsize=(2.6 * k, 2.8), squeeze=False)\n        Lr, Yr = RES[SEEDS[0]][\"ref_pred\"]\n        for j_, (ax, f) in enumerate(zip(axes[0], rr[\"names\"])):\n            ax.plot([0, 1], [0, 1], \"k:\", lw=.8)\n            kk = FINDINGS.index(f)\n            for lab, z, y in [(\"NIH (in-domain)\", Lr[:, kk], Yr[:, kk]),\n                              (\"new hospital, raw\", rr[\"Le\"][:, j_], rr[\"Ye\"][:, j_]),\n                              (\"+ free fix\", rr[\"Le\"][:, j_] + rr[\"OFF\"][j_], rr[\"Ye\"][:, j_]),\n                              (\"+ free fix (EM rate)\", rr[\"Le\"][:, j_] + rr[\"OFF_em\"][j_],\n                               rr[\"Ye\"][:, j_])]:\n                xs, ys = rel_curve(sig(z), y)\n                ax.plot(xs, ys, \"o-\", ms=3, lw=1, label=lab)\n            top = max(.05, float(np.nanmax([rr[\"Ye\"][:, j_].mean() * 4, .1])))\n            ax.set_xlim(0, min(1, top)); ax.set_ylim(0, min(1, top))\n            ax.set_title(f, fontsize=8); ax.set_xlabel(\"predicted probability\")\n            if j_ == 0:\n                ax.set_ylabel(\"observed rate\")\n        axes[0][0].legend(fontsize=6, loc=\"upper left\")\n        fig.suptitle(f\"Reliability: NIH -> {site}\", fontsize=9)\n        savefig(fig, \"fig2_reliability_\" + site.split()[0].replace(\"-\", \"_\"))\nexcept Exception as e:\n    print(\"  fig2 skipped:\", e)\n\ntry:   # Figure 3: label budget curves\n    fig, axes = plt.subplots(1, len(PLOT_SITES), figsize=(5 * len(PLOT_SITES), 3.4),\n                             squeeze=False)\n    for ax, site in zip(axes[0], PLOT_SITES):\n        Bs = pd.concat([RES[s][\"sites\"][site][\"budget\"].assign(seed=s)\n                        for s in SEEDS if site in RES[s][\"sites\"]])\n        for meth, mk in ((\"Platt\", \"o\"), (\"prior+temp\", \"s\")):\n            g_ = Bs[Bs.method == meth].groupby(\"n_labels\")[\"mean_ECE\"]\n            ax.errorbar(g_.mean().index, g_.mean().values, yerr=g_.std().fillna(0).values,\n                        marker=mk, ms=4, capsize=2, label=f\"{meth} (labels)\")\n        fl = np.mean([RES[s][\"sites\"][site][\"corrections\"]\n                      .loc[\"MEAN\", \"2 prior, known rate (0 labels)\"]\n                      for s in SEEDS if site in RES[s][\"sites\"]])\n        un = np.mean([RES[s][\"sites\"][site][\"corrections\"]\n                      .loc[\"MEAN\", \"1 uncorrected\"] for s in SEEDS if site in RES[s][\"sites\"]])\n        ax.axhline(fl, color=\"g\", ls=\"--\", label=\"free fix (0 labels)\")\n        ax.axhline(un, color=\"r\", ls=\":\", label=\"uncorrected\")\n        ax.set_xscale(\"log\"); ax.set_xlabel(\"labelled images at the new hospital\")\n        ax.set_ylabel(\"mean calibration error (ECE)\"); ax.set_title(f\"NIH -> {site}\")\n        ax.legend(fontsize=7)\n    savefig(fig, \"fig3_label_budget\")\nexcept Exception as e:\n    print(\"  fig3 skipped:\", e)\n\ntry:   # Figure 4: Claim 4 scatter\n    if C4:\n        fig, ax = plt.subplots(figsize=(4.6, 3.6))\n        for site, mk in zip(C4, (\"o\", \"s\", \"^\")):\n            pl = C4[site]\n            ax.scatter(pl.absLogRatio, pl.gain, marker=mk, label=site)\n            for f, row in pl.iterrows():\n                ax.annotate(f, (row.absLogRatio, row.gain), fontsize=6,\n                            xytext=(3, 2), textcoords=\"offset points\")\n        ax.axhline(0, color=\"k\", lw=.6)\n        ax.set_xlabel(\"|log(prevalence at new / prevalence at training hospital)|\")\n        ax.set_ylabel(\"error removed by free fix (%)\")\n        ax.set_title(\"Bigger prevalence shift -> bigger benefit\"); ax.legend(fontsize=7)\n        savefig(fig, \"fig4_prevalence_shift_vs_benefit\")\nexcept Exception as e:\n    print(\"  fig4 skipped:\", e)\n\njson.dump({\"runtime_min\": round((time.time() - T0) / 60, 1),\n           \"seeds\": SEEDS, \"have_chexpert\": HAVE_CHEX,\n           \"tier_A\": SHARED, \"tier_B\": EXTENDED,\n           \"val_mAUROC\": {str(s): RES[s][\"best_val\"] for s in SEEDS},\n           \"gate_A\": {str(s): RES[s][\"gate_A\"] for s in SEEDS},\n           \"gate_B\": {str(s): RES[s][\"gate_B\"] for s in SEEDS},\n           \"summary\": SUM.to_dict()},\n          open(f\"{W}/summary.json\", \"w\"), indent=2, default=str)\n\nprint(f\"\\n{el()} COMPLETE.  Results in {W}:\")\nfor f in sorted(os.listdir(W)):\n    fp = f\"{W}/{f}\"\n    if os.path.isfile(fp) and f.endswith((\".csv\", \".json\", \".npz\", \".pt\")):\n        print(f\"   {f}  ({os.path.getsize(fp)/1e6:.1f} MB)\")\nprint(f\"\\n  CACHE FILES in {CACHE} (save these as a dataset):\")\ntot = 0\nfor f in sorted(os.listdir(CACHE)):\n    fp = f\"{CACHE}/{f}\"\n    if os.path.isfile(fp):\n        sz = os.path.getsize(fp) / 1e6; tot += sz\n        print(f\"   cache/{f}  ({sz:.0f} MB)\")\nprint(f\"   total {tot:.0f} MB\")\nneed = [\"nih_x.npy\", \"vindr_x.npy\"] + ([\"chexpert_x.npy\"] if HAVE_CHEX else [])\nprint(\"   all image arrays present:\",\n      \"YES\" if all(os.path.exists(f\"{CACHE}/{f}\") for f in need) else \"NO\")\n\nprint(\"\"\"\n================================================================================\nWHAT TO READ\n\n1. Start of log: \"[chexpert] ... 15,000 images\" and \"matched ... 9x%\".\n2. GATE lines in SECTION 4: tier A must pass for every seed.\n3. SECTION 7 table. The four claims:\n   Claim 1  level/sharp            level error dominates sharpness error\n   Claim 2  free% vs Platt%,       the fair test. \"free vs Platt\" gives\n            \"free vs Platt\"        B / = / W counts across seeds\n            EM%, \"EM vs Platt\"     the same with ZERO labels AND no records\n   Claim 3  \"Platt beats free at n\"  labels needed before they help (per seed)\n   Claim 4  rho, and the \"BOTH HOSPITALS\" line\n   Robustness  \"rate +-25%/+-50%\"  how exact the recorded disease rate must be\n4. ANNOTATION-PROTOCOL EFFECT: hospital effect vs labelling effect.\n5. figures/ in the Output tab: fig1-fig4 as PNG (slides) and PDF (paper).\n\nThen save this run's output as your cache source for future runs.\n================================================================================\n\"\"\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-09-29T20:33:56.297264Z","iopub.execute_input":"2026-09-29T20:33:56.298045Z","iopub.status.idle":"2026-09-29T21:43:00.52507Z","shell.execute_reply.started":"2026-09-29T20:33:56.298012Z","shell.execute_reply":"2026-09-29T21:43:00.523861Z"}},"outputs":[],"execution_count":null}]}