{
  "id": 607922,
  "title": "Allowance to re-host dataset",
  "url": "/competitions/rsna-intracranial-aneurysm-detection/discussion/607922",
  "author_name": "NilanE",
  "post_date": "2025-09-16T22:35:22.682000",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I converted the dataset to the WebDataset format and uploaded the shards to HF, but I'm unsure if I'm allowed to make the dataset publicly accessible.</p>\n<p>A 300gb dataset of loose files was a hard pill to swallow, so I (mostly losslessly, due to encoding each pixel as uint16, some contrast would be lost if the range exceeded that) encoded all the volumes using the FFV1 codec alongside the relevant dicom and train metadata into WDS tar files. The total size is around ~132gb, and is streamable through the webdataset library.</p>\n<p>However, after reading through the competition rules, I saw this section:<br>\n'You agree to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition Data. You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition Data to any party not participating in the Competition. You agree to notify Kaggle immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition Data and agree to work with Kaggle to rectify any unauthorized transmission or access.'</p>\n<p>Thus, I had to make the dataset private. I think the difficulty of working with the dataset is likely preventing people from being able to properly take part in the competition. Huggingface does have 'gated datasets', where information can be provided to access them, but only the organizers would be able to set that up.</p>\n<p>In short, would I be able to make the dataset public, and/or would it be possible for the organizers to make a stream-friendly version of it available to participants?</p>",
  "messages": [
    {
      "id": 3289979,
      "postDate": "2025-09-16T22:35:22.683Z",
      "content": "<p>I converted the dataset to the WebDataset format and uploaded the shards to HF, but I'm unsure if I'm allowed to make the dataset publicly accessible.</p>\n<p>A 300gb dataset of loose files was a hard pill to swallow, so I (mostly losslessly, due to encoding each pixel as uint16, some contrast would be lost if the range exceeded that) encoded all the volumes using the FFV1 codec alongside the relevant dicom and train metadata into WDS tar files. The total size is around ~132gb, and is streamable through the webdataset library.</p>\n<p>However, after reading through the competition rules, I saw this section:<br>\n'You agree to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition Data. You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition Data to any party not participating in the Competition. You agree to notify Kaggle immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition Data and agree to work with Kaggle to rectify any unauthorized transmission or access.'</p>\n<p>Thus, I had to make the dataset private. I think the difficulty of working with the dataset is likely preventing people from being able to properly take part in the competition. Huggingface does have 'gated datasets', where information can be provided to access them, but only the organizers would be able to set that up.</p>\n<p>In short, would I be able to make the dataset public, and/or would it be possible for the organizers to make a stream-friendly version of it available to participants?</p>",
      "rawMarkdown": "I converted the dataset to the WebDataset format and uploaded the shards to HF, but I'm unsure if I'm allowed to make the dataset publicly accessible.\n\nA 300gb dataset of loose files was a hard pill to swallow, so I (mostly losslessly, due to encoding each pixel as uint16, some contrast would be lost if the range exceeded that) encoded all the volumes using the FFV1 codec alongside the relevant dicom and train metadata into WDS tar files. The total size is around ~132gb, and is streamable through the webdataset library.\n\nHowever, after reading through the competition rules, I saw this section:\n'You agree to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition Data. You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition Data to any party not participating in the Competition. You agree to notify Kaggle immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition Data and agree to work with Kaggle to rectify any unauthorized transmission or access.'\n\nThus, I had to make the dataset private. I think the difficulty of working with the dataset is likely preventing people from being able to properly take part in the competition. Huggingface does have 'gated datasets', where information can be provided to access them, but only the organizers would be able to set that up.\n\nIn short, would I be able to make the dataset public, and/or would it be possible for the organizers to make a stream-friendly version of it available to participants?",
      "votes": 1
    },
    {
      "id": 3290247,
      "postDate": "2025-09-17T13:01:54.720Z",
      "content": "<p>The short answer is that this is not allowed by the competition rules. Thanks for your efforts and for thinking about other participants.</p>",
      "rawMarkdown": "The short answer is that this is not allowed by the competition rules. Thanks for your efforts and for thinking about other participants."
    }
  ],
  "comments": [
    {
      "id": 3290247,
      "author_name": "Evan Calabrese",
      "author_url": "",
      "post_date": "2025-09-17T13:01:54.720000",
      "content": "<p>The short answer is that this is not allowed by the competition rules. Thanks for your efforts and for thinking about other participants.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3289979": "I converted the dataset to the WebDataset format and uploaded the shards to HF, but I'm unsure if I'm allowed to make the dataset publicly accessible.\n\nA 300gb dataset of loose files was a hard pill to swallow, so I (mostly losslessly, due to encoding each pixel as uint16, some contrast would be lost if the range exceeded that) encoded all the volumes using the FFV1 codec alongside the relevant dicom and train metadata into WDS tar files. The total size is around ~132gb, and is streamable through the webdataset library.\n\nHowever, after reading through the competition rules, I saw this section:\n'You agree to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition Data. You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition Data to any party not participating in the Competition. You agree to notify Kaggle immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition Data and agree to work with Kaggle to rectify any unauthorized transmission or access.'\n\nThus, I had to make the dataset private. I think the difficulty of working with the dataset is likely preventing people from being able to properly take part in the competition. Huggingface does have 'gated datasets', where information can be provided to access them, but only the organizers would be able to set that up.\n\nIn short, would I be able to make the dataset public, and/or would it be possible for the organizers to make a stream-friendly version of it available to participants?",
    "3290247": "The short answer is that this is not allowed by the competition rules. Thanks for your efforts and for thinking about other participants."
  }
}