{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# | WGAN-GP | GM | QuickDraw | Image Generation |\n\n## WGAN-GP (Wasserstein GAN with Gradient Penalty) and GM (Generative Models) for QuickDraw Image Generation\n\n# <b>1 <span style='color:#78D118'>|</span> Introduction</b>\n\nThis project is an exploration of Generative Models (GM) and its capabilities, focusing on the generation of bicycle images using Wasserstein Generative Adversarial Networks (WGAN-GP) in conjunction with estimators and generators.\n\nWGAN-GP (Wasserstein GAN with Gradient Penalty) is a specific type of generative adversarial network (GAN) utilized for generating realistic data, particularly images.\n\n- **Architecture**:\n   - **WGAN-GP**: WGAN-GP represents a variant of the GAN framework that places significant emphasis on enhancing training stability and the quality of generated outputs. It leverages the concept of Wasserstein divergence, a metric measuring the dissimilarity between two probability distributions. WGAN-GP introduces a gradient penalty mechanism to control the gradient norms of the discriminator, thereby fostering more stable training and smoother gradients.\n\n- **Loss Function**:\n   - **WGAN-GP**: In contrast to traditional GANs, WGAN-GP employs the Wasserstein divergence loss, also referred to as the Earth-Mover (EM) loss. This loss measures the dissimilarity between probability distributions and is considered advantageous for improving the quality of generated data. In addition to this loss, WGAN-GP incorporates a gradient penalty component to enhance training stability further.\n\n- **Training Stability**:\n   - **WGAN-GP**: The primary objective of WGAN-GP is to enhance training stability. By incorporating a gradient penalty, it mitigates issues that can affect some conventional GANs, such as mode collapse, where the generator tends to produce similar-looking samples.\n\nIn summary, WGAN-GP is a specialized GAN variant tailored to enhance the training stability and output quality, particularly when generating images.\n\n## Objectives :\n - Develop and train a powerful WGAN-GP model using the expansive QuickDraw dataset.\n - Cultivate a deep understanding of the cutting-edge WGAN-GP architecture and Generative AI techniques.\n\n## The QuickDraw Dataset:\nThe [Quick Draw dataset](https://quickdraw.withgoogle.com/data) is a treasure trove of approximately 50 million drawings, contributed by real artists. For our endeavor, we have curated a subset consisting of 117,555 meticulously crafted bicycle drawings.\n\n**Access the QuickDraw Dataset:**\n - Dataset Repository: [https://github.com/googlecreativelab/quickdraw-dataset](https://github.com/googlecreativelab/quickdraw-dataset)\n - Numpy Bitmap Files: [https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap](https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap)\n - Bicycle Dataset: [https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/bicycle.npy](https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/bicycle.npy)\n\n## Project Workflow:\n\n- **Setup**: Imports and Parameters.\n-  **Data Exploration**: Discovering bicycle drawings in the Dataset.\n-  **Model Architecture**: Designing a WGAN-GP (Wasserstein Generative Adversarial Network with Gradient Penalty).\n-  **Model Building**: Creating the GAN Model.\n-  **Model Training**: Feed data to the model and watch as it learns to generate bicycles.\n-  **Artistic Analysis**: Delving into the generated Bicycles.","metadata":{}},{"cell_type":"markdown","source":"# <b>2 <span style='color:#78D118'>|</span> Setup</b>\n\n## <b>2.1 <span style='color:#78D118'>|</span> Imports</b>","metadata":{}},{"cell_type":"code","source":"!pip install -q git+https://github.com/YanSteph/SKit.git","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nfrom skimage import io\nimport sys\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.callbacks import TensorBoard\nfrom tensorflow.keras.utils import plot_model\nfrom tensorflow.keras.optimizers.legacy import Adam \nfrom tensorflow.keras.utils import plot_model\n\n#import WGANGP\n#import ImagesCallback\n\nfrom skit.show import show_images, show_history, show_text","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <b>2.1 <span style='color:#78D118'>|</span> Parameters</b>\n\n**Train**\n\n`scale`: With `scale=1`, it takes approximately 5 to 6 minutes on a GPU V100, else exceed 2 hours on a CPU.  \n\n`latent_dim`: This variable represents the dimensionality of the latent space. In many machine learning applications, particularly in generative models like GANs, the latent space is where the model learns to represent complex data in a lower-dimensional space. In this case, `latent_dim` is set to 128, indicating that the latent space has 128 dimensions.\n\n`epochs`: It specifies the number of complete passes through the entire training dataset during the training process. Setting it to 3 means that the model will go through the dataset three times during training.\n\n`n_critic`: This is typically used in the context of training a GAN. It represents the number of times the critic (discriminator) is trained before the generator is updated. Training the critic multiple times before updating the generator is a technique to ensure that the critic provides meaningful feedback to the generator.\n\n`batch_size`: It represents the number of data samples used in each iteration of training. A batch is a subset of the entire dataset, and the model's weights are updated after processing each batch. A batch size of 64 means that 64 data samples are processed together before updating the model's parameters.\n\n**Adam**\n\n`Learning Rate`: This is the learning rate, a crucial parameter during neural network training. It determines the size of the steps the optimizer takes when updating the model's weights at each iteration. A too high learning rate can lead to unstable convergence, while a too low one can make training very slow or converge to a local minimum. It needs to be carefully tuned for good results.\n\n`Beta_1`: This is the attenuation coefficient for the moving average of the first moment (moving average of gradients). It's a number between 0 and 1 that determines the relative importance of recent gradients compared to older ones. A typical value is usually close to 0.9. A value closer to 1 gives more weight to recent gradients, while a value closer to 0 results in a slower-moving average.\n\n`Beta_2`: This is the attenuation coefficient for the moving average of the second moment (moving average of squared gradients). Like beta_1, it's also a number between 0 and 1. A typical value for beta_2 is usually close to 0.999. Beta_2 determines how fast the moving average of the second moment adapts to gradient variations.\n\n**Option**\n\n`fit_verbosity`: Verbosity level during training: 0 = silent, 1 = progress bar, 2 = one line per epoch. \n\n`num_img`: The number of images to visualize during the training.","metadata":{}},{"cell_type":"code","source":"# Train\n# ----\nscale         = 1\nlatent_dim    = 128\nepochs        = 3\nn_critic      = 2\nbatch_size    = 64\n\n# Adam\n# ----\nlearning_rate = 0.0002\nbeta_1        = 0.5\nbeta_2        = 0.9\n\n# Option\n# ----\nnum_img       = 12\nfit_verbosity = 1","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <b>3 <span style='color:#78D118'>|</span> Data Exploration</b>","metadata":{}},{"cell_type":"code","source":"# Load dataset\nx_data = np.load(\"./bike.npy\")\nprint('Original dataset shape : ',x_data.shape)\n\n# Rescale\nn=int(scale*len(x_data))\nx_data = x_data[:n]\nprint('Rescaled dataset shape : ',x_data.shape)\n\n# Normalize, reshape and shuffle\nx_data = x_data/255\nx_data = x_data.reshape(-1,28,28,1)\nnp.random.shuffle(x_data)\nprint('Final dataset shape    : ',x_data.shape)\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These image are drawn by real humans.","metadata":{}},{"cell_type":"code","source":"show_images(\n    x_data, \n    indices      = range(72), \n    columns      = num_img, \n    figure_size  = (2,2), \n    padding      = 0,\n    spines_alpha = 0\n)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <b>4 <span style='color:#78D118'>|</span> Model Architecture</b>\n\n## <b>4.1 <span style='color:#78D118'>|</span> Discriminator</b>\n","metadata":{}},{"cell_type":"code","source":"# Input layer\ninputs = keras.Input(shape=(28, 28, 1))\n\n# Convolutional layers with LeakyReLU activation\nx = layers.Conv2D(64, kernel_size=4, strides=2, padding=\"same\")(inputs)\nx = layers.LeakyReLU(alpha=0.2)(x)\n\nx = layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\")(x)\nx = layers.LeakyReLU(alpha=0.2)(x)\n\nx = layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\")(x)\nx = layers.LeakyReLU(alpha=0.2)(x)\n\n# Flatten the output\nx = layers.Flatten()(x)\n\n# Apply dropout for regularization\nx = layers.Dropout(0.2)(x)\n\n# Output layer with a single unit (binary classification)\nx = layers.Dense(1)(x)\n\n# Create and summary the discriminator model\ndiscriminator = keras.Model(inputs, x, name=\"discriminator\")\ndiscriminator.summary()\nplot_model(discriminator, show_shapes=True, show_layer_names=True)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <b>4.2 <span style='color:#78D118'>|</span> Generator</b>","metadata":{}},{"cell_type":"code","source":"# Define the input layer\ninputs = keras.Input(shape=(latent_dim,))\n\n# Fully connected layer followed by reshaping\nx = layers.Dense(7 * 7 * 64)(inputs)\nx = layers.Reshape((7, 7, 64))(x)\n\n# Upsampling layers\nx = layers.UpSampling2D()(x)\nx = layers.Conv2D(128, kernel_size=3, strides=1, padding='same', activation='relu')(x)\n\nx = layers.UpSampling2D()(x)\nx = layers.Conv2D(256, kernel_size=3, strides=1, padding='same', activation='relu')(x)\n\n# Output layer\noutputs = layers.Conv2D(1, kernel_size=5, strides=1, padding=\"same\", activation='sigmoid')(x)\n\n# Create the generator model\ngenerator = keras.Model(inputs, outputs, name=\"generator\")\n\n# Display model summary\ngenerator.summary()\nplot_model(generator, show_shapes=True, show_layer_names=True)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <b>5 <span style='color:#78D118'>|</span> Model Building</b>","metadata":{}},{"cell_type":"code","source":"gan = WGANGP(\n        discriminator = discriminator, \n        generator     = generator, \n        latent_dim    = latent_dim, \n        n_critic      = n_critic\n    )","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gan.compile(\n    discriminator_optimizer = Adam(learning_rate=learning_rate, beta_1=beta_1, beta_2=beta_2),\n    generator_optimizer     = Adam(learning_rate=learning_rate, beta_1=beta_1, beta_2=beta_2)\n)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <b>6 <span style='color:#78D118'>|</span> Model Training</b>","metadata":{}},{"cell_type":"code","source":"imagesCallback = ImagesCallback(\n    num_img    = num_img, \n    latent_dim = latent_dim, \n    run_dir    = f'./images'\n)\n\nhistory = gan.fit( \n            x_data, \n            epochs     = epochs, \n            batch_size = batch_size, \n            callbacks  = [imagesCallback], \n            verbose    = fit_verbosity \n          ) \n\ngan.save(f'./models/model.h5')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <b>6.1 <span style='color:#78D118'>|</span> History</b>","metadata":{}},{"cell_type":"code","source":"show_history(\n    history,\n    title = \"Loss\",\n    metrics = [\"d_loss\", \"g_loss\"], \n    metric_labels = [\"Discriminator loss\", \"Generator loss\"]\n)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for epoch in range(0,epochs,1):\n    images=[]\n    \n    for i in range(num_img):\n        filename = f'./images/image-{epoch:03d}-{i:02d}.jpg'\n        image    = io.imread(filename)\n        images.append(image)\n        \n    show_text(\"b\", f\"Epoch: {epoch}\", False)\n    show_images(\n        images, \n        None, \n        indices='all', \n        columns=num_img, \n        figure_size=(1,1), \n        interpolation=None, \n        padding=0, \n        spines_alpha=0\n    )","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <b>7 <span style='color:#78D118'>|</span> Artistic Analysis</b>","metadata":{}},{"cell_type":"code","source":"gan.reload(f'./models/model.h5')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generate somes images from latent space :","metadata":{}},{"cell_type":"code","source":"nb_images = num_img*15\n\nz = np.random.normal(size=(nb_images,latent_dim))\nimages = gan.predict(z, verbose=0)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_images(\n    images, \n    None, \n    indices='all', \n    columns=num_img, \n    figure_size=(1,1), \n    interpolation=None, \n    padding=0, \n    spines_alpha=0\n)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## References\n\nThe creation of this document was greatly influenced by the following key sources of information:\n\n1. [Quick Draw dataset](https://quickdraw.withgoogle.com/data) is a treasure trove of approximately 50 million drawings, contributed by real artists.\n2. [Fidle](https://gricad-gitlab.univ-grenoble-alpes.fr/talks/fidle/-/wikis/home) - An informative guide that provides in-depth explanations and examples on various data science topics.","metadata":{}}]}