{
  "id": 384477,
  "title": "Problem with TPU v3.8 and model.fit",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/384477",
  "author_name": "Elias Queiroga Vieira",
  "post_date": "2023-02-08T03:14:56.245000",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am trying to use the dataset provided by <a href=\"https://www.kaggle.com/radek1\" target=\"_blank\">@radek1</a> (<a href=\"https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs\" target=\"_blank\">https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs</a>) to train the model using keras model.fit and it works fine when I run on GPU, TPU VM, but when I run on TPU v3.8, it raises UnimplementedError (File system scheme '[local]' not implemented). I found a few topics mentioning that tfds often doesn't host all the datasets and downloads some from the original source to your local machine, which TPU can't access.<br>\nSo my question is, how to circumvent this problem since I can't have internet access to submit my notebook for the competition?<br>\nThanks in advance to anyone who can shed some light here.</p>",
  "messages": [
    {
      "id": 2134502,
      "postDate": "2023-02-08T03:14:56.247Z",
      "content": "<p>I am trying to use the dataset provided by <a href=\"https://www.kaggle.com/radek1\" target=\"_blank\">@radek1</a> (<a href=\"https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs\" target=\"_blank\">https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs</a>) to train the model using keras model.fit and it works fine when I run on GPU, TPU VM, but when I run on TPU v3.8, it raises UnimplementedError (File system scheme '[local]' not implemented). I found a few topics mentioning that tfds often doesn't host all the datasets and downloads some from the original source to your local machine, which TPU can't access.<br>\nSo my question is, how to circumvent this problem since I can't have internet access to submit my notebook for the competition?<br>\nThanks in advance to anyone who can shed some light here.</p>",
      "rawMarkdown": "I am trying to use the dataset provided by @radek1 (https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs) to train the model using keras model.fit and it works fine when I run on GPU, TPU VM, but when I run on TPU v3.8, it raises UnimplementedError (File system scheme '[local]' not implemented). I found a few topics mentioning that tfds often doesn't host all the datasets and downloads some from the original source to your local machine, which TPU can't access.\nSo my question is, how to circumvent this problem since I can't have internet access to submit my notebook for the competition?\nThanks in advance to anyone who can shed some light here.",
      "votes": 1
    },
    {
      "id": 2134723,
      "postDate": "2023-02-08T08:10:06.117Z",
      "content": "<p>This is because the TPU V3.8 functions as a sort of API where you can send your training/prediction requests to, it is not a piece of hardware attached to your local notebook machine. To use the TPU V3.8 you should make your dataset public and access it through a Google Cloud Storage path, the remote TPU V3.8 can only access data stored there and does not see your local notebook storage.</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('my_dataset').\n\nMY_FILE_PATHS = sorted(tf.io.gfile.glob(f'{GCS_DS_PATH}/*.png'))\nprint(f'Found {len(MY_FILE_PATHS )} PNG Files')\n</code></pre>",
      "rawMarkdown": "This is because the TPU V3.8 functions as a sort of API where you can send your training/prediction requests to, it is not a piece of hardware attached to your local notebook machine. To use the TPU V3.8 you should make your dataset public and access it through a Google Cloud Storage path, the remote TPU V3.8 can only access data stored there and does not see your local notebook storage.\n\n\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('my_dataset').\n\nMY_FILE_PATHS = sorted(tf.io.gfile.glob(f'{GCS_DS_PATH}/*.png'))\nprint(f'Found {len(MY_FILE_PATHS )} PNG Files')\n```",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 2134723,
      "author_name": "Mark Wijkhuizen",
      "author_url": "",
      "post_date": "2023-02-08T08:10:06.117000",
      "content": "<p>This is because the TPU V3.8 functions as a sort of API where you can send your training/prediction requests to, it is not a piece of hardware attached to your local notebook machine. To use the TPU V3.8 you should make your dataset public and access it through a Google Cloud Storage path, the remote TPU V3.8 can only access data stored there and does not see your local notebook storage.</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('my_dataset').\n\nMY_FILE_PATHS = sorted(tf.io.gfile.glob(f'{GCS_DS_PATH}/*.png'))\nprint(f'Found {len(MY_FILE_PATHS )} PNG Files')\n</code></pre>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2134502": "I am trying to use the dataset provided by @radek1 (https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs) to train the model using keras model.fit and it works fine when I run on GPU, TPU VM, but when I run on TPU v3.8, it raises UnimplementedError (File system scheme '[local]' not implemented). I found a few topics mentioning that tfds often doesn't host all the datasets and downloads some from the original source to your local machine, which TPU can't access.\nSo my question is, how to circumvent this problem since I can't have internet access to submit my notebook for the competition?\nThanks in advance to anyone who can shed some light here.",
    "2134723": "This is because the TPU V3.8 functions as a sort of API where you can send your training/prediction requests to, it is not a piece of hardware attached to your local notebook machine. To use the TPU V3.8 you should make your dataset public and access it through a Google Cloud Storage path, the remote TPU V3.8 can only access data stored there and does not see your local notebook storage.\n\n\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('my_dataset').\n\nMY_FILE_PATHS = sorted(tf.io.gfile.glob(f'{GCS_DS_PATH}/*.png'))\nprint(f'Found {len(MY_FILE_PATHS )} PNG Files')\n```"
  }
}