{
  "id": 67099,
  "title": "How big is the data? How much processing power is required?",
  "url": "/competitions/quickdraw-doodle-recognition/discussion/67099",
  "author_name": "Mukesh",
  "post_date": "2018-09-28T17:45:59.807000",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi,\nI haven't joined the competition yet, I saw the data size it is 73GB. How much time does it take to train? Can you Post time with your processing power,</p>\n\n<p>Thanks..!!!</p>",
  "messages": [
    {
      "id": 395542,
      "postDate": "2018-09-28T19:24:14.467Z",
      "content": "<p>Mukesh - You're going to want to give a better idea of what you want. Time to train is going to be wildly different based on your approach to the problem. Will you be using a CNN or LSTM? Will you be pre-processing the data? Using the RAW or simplified data? Baseline to train something on a dataset of this size (Over 3m 256x256 images) would be days for a complex model</p>",
      "rawMarkdown": "Mukesh - You're going to want to give a better idea of what you want. Time to train is going to be wildly different based on your approach to the problem. Will you be using a CNN or LSTM? Will you be pre-processing the data? Using the RAW or simplified data? Baseline to train something on a dataset of this size (Over 3m 256x256 images) would be days for a complex model",
      "votes": 1,
      "replies": [
        {
          "id": 395567,
          "postDate": "2018-09-28T20:51:18.130Z",
          "content": "<p>Thanks for your reply!!! With my current GPU (1050TI) , i guess I will be using CNN.</p>",
          "rawMarkdown": "Thanks for your reply!!! With my current GPU (1050TI) , i guess I will be using CNN."
        },
        {
          "id": 395897,
          "postDate": "2018-09-29T14:50:57.123Z",
          "content": "<p>Can you suggest some pre processing methods using the train_simplified? I want to feed it to a CNN</p>",
          "rawMarkdown": "Can you suggest some pre processing methods using the train_simplified? I want to feed it to a CNN"
        },
        {
          "id": 395924,
          "postDate": "2018-09-29T15:37:20.333Z",
          "content": "<p>Check out my kernel on turning the point data into lines, upvote it if it's useful </p>",
          "rawMarkdown": "Check out my kernel on turning the point data into lines, upvote it if it's useful "
        },
        {
          "id": 405189,
          "postDate": "2018-10-17T03:16:17.220Z",
          "content": "<p>You should rent a machine in the cloud if you want to tackle this challenge seriously, apart from that, you can always use kaggle to run it, pretty sure it will be faster than a 1050 (i think).</p>\n\n<p>As for the doodle to image you can use what i used in my kernel:\ndrawing is the input from the csv, shape is the final size of the image, in my case it returns a gray scale image</p>\n\n<pre><code>def drawing_to_np(drawing, shape=(64, 64)):\ndrawing = eval(drawing)\nfig, ax = plt.subplots()\nfor x,y in drawing:\n    ax.plot(x, y, marker='.')\n    ax.axis('off')\nfig.canvas.draw()\n# Convert images to numpy arrat\nnp_drawing = np.array(fig.canvas.renderer._renderer)\nplt.close(fig)\nimg = cv2.resize(np_drawing, shape)\nimg_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\nimg_expanded = img_gray[:, :, np.newaxis]\nreturn img_expanded\n</code></pre>",
          "rawMarkdown": "You should rent a machine in the cloud if you want to tackle this challenge seriously, apart from that, you can always use kaggle to run it, pretty sure it will be faster than a 1050 (i think).\n\nAs for the doodle to image you can use what i used in my kernel:\ndrawing is the input from the csv, shape is the final size of the image, in my case it returns a gray scale image\n\n    def drawing_to_np(drawing, shape=(64, 64)):\n    drawing = eval(drawing)\n    fig, ax = plt.subplots()\n    for x,y in drawing:\n        ax.plot(x, y, marker='.')\n        ax.axis('off')\n    fig.canvas.draw()\n    # Convert images to numpy arrat\n    np_drawing = np.array(fig.canvas.renderer._renderer)\n    plt.close(fig)\n    img = cv2.resize(np_drawing, shape)\n    img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    img_expanded = img_gray[:, :, np.newaxis]\n    return img_expanded"
        }
      ]
    },
    {
      "id": 395505,
      "postDate": "2018-09-28T17:45:59.807Z",
      "content": "<p>Hi,\nI haven't joined the competition yet, I saw the data size it is 73GB. How much time does it take to train? Can you Post time with your processing power,</p>\n\n<p>Thanks..!!!</p>",
      "rawMarkdown": "Hi,\nI haven't joined the competition yet, I saw the data size it is 73GB. How much time does it take to train? Can you Post time with your processing power,\n\nThanks..!!!",
      "votes": 1
    },
    {
      "id": 395820,
      "postDate": "2018-09-29T13:09:16.617Z",
      "rawMarkdown": "",
      "votes": -4,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 395542,
      "author_name": "Simmo",
      "author_url": "",
      "post_date": "2018-09-28T19:24:14.467000",
      "content": "<p>Mukesh - You're going to want to give a better idea of what you want. Time to train is going to be wildly different based on your approach to the problem. Will you be using a CNN or LSTM? Will you be pre-processing the data? Using the RAW or simplified data? Baseline to train something on a dataset of this size (Over 3m 256x256 images) would be days for a complex model</p>",
      "votes": 1,
      "replies": [
        {
          "id": 395567,
          "author_name": "Mukesh",
          "author_url": "",
          "post_date": "2018-09-28T20:51:18.130000",
          "content": "<p>Thanks for your reply!!! With my current GPU (1050TI) , i guess I will be using CNN.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 395897,
          "author_name": "Hasib Zunair",
          "author_url": "",
          "post_date": "2018-09-29T14:50:57.123000",
          "content": "<p>Can you suggest some pre processing methods using the train_simplified? I want to feed it to a CNN</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 395924,
          "author_name": "Simmo",
          "author_url": "",
          "post_date": "2018-09-29T15:37:20.333000",
          "content": "<p>Check out my kernel on turning the point data into lines, upvote it if it's useful </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 405189,
          "author_name": "André Neves",
          "author_url": "",
          "post_date": "2018-10-17T03:16:17.220000",
          "content": "<p>You should rent a machine in the cloud if you want to tackle this challenge seriously, apart from that, you can always use kaggle to run it, pretty sure it will be faster than a 1050 (i think).</p>\n\n<p>As for the doodle to image you can use what i used in my kernel:\ndrawing is the input from the csv, shape is the final size of the image, in my case it returns a gray scale image</p>\n\n<pre><code>def drawing_to_np(drawing, shape=(64, 64)):\ndrawing = eval(drawing)\nfig, ax = plt.subplots()\nfor x,y in drawing:\n    ax.plot(x, y, marker='.')\n    ax.axis('off')\nfig.canvas.draw()\n# Convert images to numpy arrat\nnp_drawing = np.array(fig.canvas.renderer._renderer)\nplt.close(fig)\nimg = cv2.resize(np_drawing, shape)\nimg_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\nimg_expanded = img_gray[:, :, np.newaxis]\nreturn img_expanded\n</code></pre>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 395820,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-09-29T13:09:16.617000",
      "content": "",
      "votes": -4,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "395542": "Mukesh - You're going to want to give a better idea of what you want. Time to train is going to be wildly different based on your approach to the problem. Will you be using a CNN or LSTM? Will you be pre-processing the data? Using the RAW or simplified data? Baseline to train something on a dataset of this size (Over 3m 256x256 images) would be days for a complex model",
    "395505": "Hi,\nI haven't joined the competition yet, I saw the data size it is 73GB. How much time does it take to train? Can you Post time with your processing power,\n\nThanks..!!!",
    "395820": ""
  }
}