{
  "id": 383333,
  "title": "Influence on seeds ",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/383333",
  "author_name": "Hyunsoo Lee 1010",
  "post_date": "2023-02-03T07:24:50.247000",
  "votes": 0,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I recently tested some baseline to take understand this competition.</p>\n<p>From my understanding, seeds make models output same results, but it didn't in my reference. (different f1 score in epoch every moment)</p>\n<p>Am I missing some seeds that I don't know or my assumption is wrong?</p>\n<hr>\n<p>These are seed lines.</p>\n<p><code>SEED = 43</code></p>\n<pre><code>\n ():\n    os.environ[] = (seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything()\n</code></pre>\n<pre><code>\nTFRECORDS_TRAIN, TFRECORDS_VAL = train_test_split(TFRECORDS_FILE_PATHS, train_size=, random_state=SEED, shuffle=)\n()\n</code></pre>\n<pre><code> ():\n    \n    ()\n    ()\n\n     STRATEGY.scope():\n        \n        seed_everything()\n\n        \n        image = tf.keras.layers.Input(INPUT_SHAPE, name=, dtype=tf.uint8)\n\n        \n        image_norm = normalize(image)\n</code></pre>\n<p>resource)<br>\nTrain : <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed\" target=\"_blank\">https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed</a><br>\nInference : <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385\" target=\"_blank\">https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385</a></p>",
  "messages": [
    {
      "id": 2127818,
      "postDate": "2023-02-03T08:00:32.843Z",
      "content": "<p>Setting the seeds makes the train/validation split and weights initialization in the model consistent and therefore reproducible. However, training takes place on a TPU which makes the training process itself non-reproducible due to the parallel nature of TPU's. The Kaggle TPUs consist of 8 separate compute units, which can be interpreted as 8 \"GPUs\", which all have data loaders running in parallel. The order of the training samples will be nondeterministic making the training process non-reproducible.</p>\n<p>As far as I know, it it at this point not possible to get fully reproducible results on a TPU. On a single GPU however, you should be able to get fully reproducible results.</p>\n<p>In general, making parallel computing deterministic is quite a challenge, let me know when you solved it ;)</p>",
      "rawMarkdown": "Setting the seeds makes the train/validation split and weights initialization in the model consistent and therefore reproducible. However, training takes place on a TPU which makes the training process itself non-reproducible due to the parallel nature of TPU's. The Kaggle TPUs consist of 8 separate compute units, which can be interpreted as 8 \"GPUs\", which all have data loaders running in parallel. The order of the training samples will be nondeterministic making the training process non-reproducible.\n\nAs far as I know, it it at this point not possible to get fully reproducible results on a TPU. On a single GPU however, you should be able to get fully reproducible results.\n\nIn general, making parallel computing deterministic is quite a challenge, let me know when you solved it ;)",
      "votes": 3,
      "replies": [
        {
          "id": 2127830,
          "postDate": "2023-02-03T08:25:40.443Z",
          "content": "<p>It's a great answer! I didn't think about it that way, which is the relationship between a TPU and a GPU for producing the result. </p>\n<p>And this makes me curious about the impact on a different environment, especially tasks of a single GPU and multiple GPUs. It must be a struggle for researchers to deal with the problem in the past and now, but it's worth it.</p>\n<p>Thanks again. I will share the approach If I can deal with the problem through a new method that I didn't find.</p>",
          "rawMarkdown": "It's a great answer! I didn't think about it that way, which is the relationship between a TPU and a GPU for producing the result. \n\nAnd this makes me curious about the impact on a different environment, especially tasks of a single GPU and multiple GPUs. It must be a struggle for researchers to deal with the problem in the past and now, but it's worth it.\n\nThanks again. I will share the approach If I can deal with the problem through a new method that I didn't find."
        }
      ]
    },
    {
      "id": 2128699,
      "postDate": "2023-02-04T00:12:26.583Z",
      "content": "<p>https : // pytorch .org/docs/stable/notes/randomness.html#reproducibility<br>\nhttps : // www .tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism</p>",
      "rawMarkdown": "https : // pytorch .org/docs/stable/notes/randomness.html#reproducibility\nhttps : // www .tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism"
    },
    {
      "id": 2127801,
      "postDate": "2023-02-03T07:24:50.247Z",
      "content": "<p>I recently tested some baseline to take understand this competition.</p>\n<p>From my understanding, seeds make models output same results, but it didn't in my reference. (different f1 score in epoch every moment)</p>\n<p>Am I missing some seeds that I don't know or my assumption is wrong?</p>\n<hr>\n<p>These are seed lines.</p>\n<p><code>SEED = 43</code></p>\n<pre><code>\n ():\n    os.environ[] = (seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything()\n</code></pre>\n<pre><code>\nTFRECORDS_TRAIN, TFRECORDS_VAL = train_test_split(TFRECORDS_FILE_PATHS, train_size=, random_state=SEED, shuffle=)\n()\n</code></pre>\n<pre><code> ():\n    \n    ()\n    ()\n\n     STRATEGY.scope():\n        \n        seed_everything()\n\n        \n        image = tf.keras.layers.Input(INPUT_SHAPE, name=, dtype=tf.uint8)\n\n        \n        image_norm = normalize(image)\n</code></pre>\n<p>resource)<br>\nTrain : <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed\" target=\"_blank\">https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed</a><br>\nInference : <a href=\"https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385\" target=\"_blank\">https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385</a></p>",
      "rawMarkdown": "I recently tested some baseline to take understand this competition.\n\nFrom my understanding, seeds make models output same results, but it didn't in my reference. (different f1 score in epoch every moment)\n\nAm I missing some seeds that I don't know or my assumption is wrong?\n\n\n\n---\n\nThese are seed lines.\n\n`SEED = 43`\n\n```python\n# Seed all random number generators\ndef seed_everything(seed=SEED):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything()\n```\n\n```python\n# Train Test Split\nTFRECORDS_TRAIN, TFRECORDS_VAL = train_test_split(TFRECORDS_FILE_PATHS, train_size=0.80, random_state=SEED, shuffle=True)\nprint(f'# TFRECORDS_TRAIN: {len(TFRECORDS_TRAIN)}, # TFRECORDS_VAL: {len(TFRECORDS_VAL)}')\n```\n\n```python\ndef get_model():\n    # Verify Mixed Policy Settings\n    print(f'Compute dtype: {tf.keras.mixed_precision.global_policy().compute_dtype}')\n    print(f'Variable dtype: {tf.keras.mixed_precision.global_policy().variable_dtype}')\n    \n    with STRATEGY.scope():\n        # Set seed for deterministic weights initialization\n        seed_everything()\n        \n        # Inputs, note the names are equal to the dictionary keys in the dataset\n        image = tf.keras.layers.Input(INPUT_SHAPE, name='image', dtype=tf.uint8)\n        \n        # Normalize Input\n        image_norm = normalize(image)\n```\n\nresource)\nTrain : https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed\nInference : https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385\n\n\n"
    }
  ],
  "comments": [
    {
      "id": 2127818,
      "author_name": "Mark Wijkhuizen",
      "author_url": "",
      "post_date": "2023-02-03T08:00:32.843000",
      "content": "<p>Setting the seeds makes the train/validation split and weights initialization in the model consistent and therefore reproducible. However, training takes place on a TPU which makes the training process itself non-reproducible due to the parallel nature of TPU's. The Kaggle TPUs consist of 8 separate compute units, which can be interpreted as 8 \"GPUs\", which all have data loaders running in parallel. The order of the training samples will be nondeterministic making the training process non-reproducible.</p>\n<p>As far as I know, it it at this point not possible to get fully reproducible results on a TPU. On a single GPU however, you should be able to get fully reproducible results.</p>\n<p>In general, making parallel computing deterministic is quite a challenge, let me know when you solved it ;)</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2127830,
          "author_name": "Hyunsoo Lee 1010",
          "author_url": "",
          "post_date": "2023-02-03T08:25:40.443000",
          "content": "<p>It's a great answer! I didn't think about it that way, which is the relationship between a TPU and a GPU for producing the result. </p>\n<p>And this makes me curious about the impact on a different environment, especially tasks of a single GPU and multiple GPUs. It must be a struggle for researchers to deal with the problem in the past and now, but it's worth it.</p>\n<p>Thanks again. I will share the approach If I can deal with the problem through a new method that I didn't find.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2128699,
      "author_name": "Antti Isosalo",
      "author_url": "",
      "post_date": "2023-02-04T00:12:26.583000",
      "content": "<p>https : // pytorch .org/docs/stable/notes/randomness.html#reproducibility<br>\nhttps : // www .tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2127818": "Setting the seeds makes the train/validation split and weights initialization in the model consistent and therefore reproducible. However, training takes place on a TPU which makes the training process itself non-reproducible due to the parallel nature of TPU's. The Kaggle TPUs consist of 8 separate compute units, which can be interpreted as 8 \"GPUs\", which all have data loaders running in parallel. The order of the training samples will be nondeterministic making the training process non-reproducible.\n\nAs far as I know, it it at this point not possible to get fully reproducible results on a TPU. On a single GPU however, you should be able to get fully reproducible results.\n\nIn general, making parallel computing deterministic is quite a challenge, let me know when you solved it ;)",
    "2128699": "https : // pytorch .org/docs/stable/notes/randomness.html#reproducibility\nhttps : // www .tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism",
    "2127801": "I recently tested some baseline to take understand this competition.\n\nFrom my understanding, seeds make models output same results, but it didn't in my reference. (different f1 score in epoch every moment)\n\nAm I missing some seeds that I don't know or my assumption is wrong?\n\n\n\n---\n\nThese are seed lines.\n\n`SEED = 43`\n\n```python\n# Seed all random number generators\ndef seed_everything(seed=SEED):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_everything()\n```\n\n```python\n# Train Test Split\nTFRECORDS_TRAIN, TFRECORDS_VAL = train_test_split(TFRECORDS_FILE_PATHS, train_size=0.80, random_state=SEED, shuffle=True)\nprint(f'# TFRECORDS_TRAIN: {len(TFRECORDS_TRAIN)}, # TFRECORDS_VAL: {len(TFRECORDS_VAL)}')\n```\n\n```python\ndef get_model():\n    # Verify Mixed Policy Settings\n    print(f'Compute dtype: {tf.keras.mixed_precision.global_policy().compute_dtype}')\n    print(f'Variable dtype: {tf.keras.mixed_precision.global_policy().variable_dtype}')\n    \n    with STRATEGY.scope():\n        # Set seed for deterministic weights initialization\n        seed_everything()\n        \n        # Inputs, note the names are equal to the dictionary keys in the dataset\n        image = tf.keras.layers.Input(INPUT_SHAPE, name='image', dtype=tf.uint8)\n        \n        # Normalize Input\n        image_norm = normalize(image)\n```\n\nresource)\nTrain : https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu#Seed\nInference : https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-inference-tensorflow?scriptVersionId=117353385\n\n\n"
  }
}