{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n

\n ๐Ÿ“š๐Ÿ“šWeights & Biases: Visualization using WandB for ML ๐Ÿงพ๐Ÿ“‘\n

\n","metadata":{}},{"cell_type":"markdown","source":"![](https://i.imgur.com/Pell4Oo.png)\n\nFrom [https://i.imgur.com/Pell4Oo.png](https://i.imgur.com/Pell4Oo.png)","metadata":{}},{"cell_type":"markdown","source":"# Very Brief Introduction to Weights & Biases (WandB)\n> Weights and Biases (W&B) is a machine learning tool for experiment tracking, visualization, and collaboration. It helps manage projects, log parameters, visualize results, and support team collaboration.\n\n1. Experiment Tracking\n1. Visualization and Analysis\n1. Model Artifacts and Versioning\n1. Collaboration and Sharing\n1. Integration and Compatibility\n1. Hyperparameter Sweeps (Hyperparameter sweeps in wandb automate hyperparameter tuning to find the best settings for model performance. Define the search space, initialize the sweep, run training jobs with different hyperparameters, log results to the wandb dashboard, and analyze the best configuration.)\n1. Automated Reports","metadata":{}},{"cell_type":"markdown","source":"![](https://docs.wandb.ai/assets/images/intro_what_it_is-8462e8215e06544eaa40dfdfe656d03d.png)","metadata":{}},{"cell_type":"markdown","source":"
\n\n

\n 📥 Some Easy-to-Understand Examples (TensorFlow and Torch)💠\n

\n \n\n
\n","metadata":{}},{"cell_type":"code","source":"# Import PyTorch and PyTorch-related libraries\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\nimport torchvision.transforms as transforms\n\n# Import TensorFlow and TensorFlow-Keras\nimport tensorflow as tf\nfrom tensorflow.keras import layers, models, datasets\n\n# Import Weights & Biases (wandb) library for experiment tracking and visualization\nimport wandb\nfrom wandb.keras import WandbCallback\n\n# Import scikit-learn metrics for evaluation\nfrom sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score\n\n# Import datetime for recording the current date and time\nimport datetime","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:11:54.741697Z","iopub.execute_input":"2023-08-02T04:11:54.742724Z","iopub.status.idle":"2023-08-02T04:11:54.750722Z","shell.execute_reply.started":"2023-08-02T04:11:54.742685Z","shell.execute_reply":"2023-08-02T04:11:54.749561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Codes that are used to help you get your WandB account logged in\n\n'''\nFirst, try importing the UserSecretsClient from kaggle_secrets.\nIf you are running this code on Kaggle, this will help you access the secrets associated with your account.\nThe secrets allow you to securely store sensitive data, such as your WandB API key, without exposing it in the code.\n'''\n\ntry:\n from kaggle_secrets import UserSecretsClient\n\n # Create a UserSecretsClient object to access your secrets\n user_secrets = UserSecretsClient()\n\n # Get the WandB API key from the secrets\n api_key = user_secrets.get_secret(\"WANDB\")\n\n # Log in to WandB using the obtained API key\n wandb.login(key=api_key)\n\n # Set the anonymous variable to None, indicating that you have successfully logged in.\n anonymous = None\n\n# If the import or API key retrieval fails, it means the user hasn't provided the WandB API key.\n# In this case, set the anonymous variable to \"must\" and display instructions on how to provide the API key.\nexcept:\n anonymous = \"must\"\n print('To use your W&B account,\\nGo to Add-ons -> Secrets and provide your W&B access token. Use the Label name as WANDB. \\nGet your W&B access token from here: https://wandb.ai/authorize')\n","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:11:54.754804Z","iopub.execute_input":"2023-08-02T04:11:54.755221Z","iopub.status.idle":"2023-08-02T04:11:55.117443Z","shell.execute_reply.started":"2023-08-02T04:11:54.755194Z","shell.execute_reply":"2023-08-02T04:11:55.116321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nInitialize Weights & Biases (wandb) with specific project and run name\n The wandb.init() function is used to initialize the Weights & Biases experiment.\n\n Arguments:\n - project: A string representing the name of the project where the experiment results will be stored.\n It is used to group multiple runs/experiments under the same project for easier organization.\n For example, if the project is \"cifar10-experiments\", all runs under this project will be grouped together.\n If the project doesn't exist, it will be created automatically.\n\n - name: A string representing the name of the current run/experiment.\n It is used to identify and differentiate individual runs within the same project.\n For example, if the name is \"experiment-2\", it will be associated with this specific run.\n If the name is not provided, WandB will generate a unique run name.\n\"\"\"\n\nwandb.init(project=\"cifar10-experiments\", name=f\"experiment-tensorflow-{str(datetime.datetime.now())[-6:]}\", config={})","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:11:55.118576Z","iopub.execute_input":"2023-08-02T04:11:55.118888Z","iopub.status.idle":"2023-08-02T04:12:26.276122Z","shell.execute_reply.started":"2023-08-02T04:11:55.118836Z","shell.execute_reply":"2023-08-02T04:12:26.275223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the CIFAR-10 dataset using the datasets module from TensorFlow/Keras.\n# The CIFAR-10 dataset is a well-known dataset for image classification tasks, containing 60,000 32x32 color images\n# belonging to 10 different classes, with 6,000 images per class.\n\n# datasets.cifar10.load_data() loads the CIFAR-10 dataset and returns two tuples:\n# - The first tuple (train_images, train_labels) contains the training data, where:\n# - train_images: A 4D NumPy array containing the training images. Shape: (50000, 32, 32, 3)\n# (50000 samples, each 32x32 pixels, with 3 channels for RGB color).\n# - train_labels: A 1D NumPy array containing the labels for the training images. Shape: (50000,)\n# (50000 labels, one for each corresponding image).\n\n# - The second tuple (test_images, test_labels) contains the test data, where:\n# - test_images: A 4D NumPy array containing the test images. Shape: (10000, 32, 32, 3)\n# (10000 samples, each 32x32 pixels, with 3 channels for RGB color).\n# - test_labels: A 1D NumPy array containing the labels for the test images. Shape: (10000,)\n# (10000 labels, one for each corresponding image).\n\n(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()\n\n# Normalize the pixel values of the images to be in the range [0, 1].\n# In deep learning, it's common to normalize the input data to help the model converge faster and avoid large gradients.\n\n# Since the pixel values in the images are in the range [0, 255], we divide all pixel values by 255.0 to scale them\n# down to the range [0, 1].\n\n# train_images and test_images are both 4D NumPy arrays representing the image data.\n# After dividing each pixel value by 255.0, all the values will be between 0 and 1.\n\ntrain_images, test_images = train_images / 255.0, test_images / 255.0","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:12:26.281535Z","iopub.execute_input":"2023-08-02T04:12:26.283806Z","iopub.status.idle":"2023-08-02T04:12:28.017523Z","shell.execute_reply.started":"2023-08-02T04:12:26.283769Z","shell.execute_reply":"2023-08-02T04:12:28.016523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the CNN model using the Sequential API of TensorFlow/Keras.\n# The Sequential API allows us to build a linear stack of layers where each layer has exactly one input tensor and one output tensor.\n\nmodel = models.Sequential([\n # Convolutional layer with 32 filters of size (3, 3) and ReLU activation function.\n # The 'input_shape' parameter specifies the shape of the input images: (height, width, channels).\n # In this case, the input images are 32x32 pixels with 3 color channels (RGB).\n layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),\n \n # MaxPooling layer with pool size (2, 2).\n # The MaxPooling operation reduces the spatial dimensions of the previous layer by taking the maximum value in each pooling region.\n layers.MaxPooling2D((2, 2)),\n \n # Another Convolutional layer with 64 filters of size (3, 3) and ReLU activation function.\n # The number of filters (64) is increased to learn more complex features from the input data.\n layers.Conv2D(64, (3, 3), activation='relu'),\n \n # Another MaxPooling layer with pool size (2, 2).\n layers.MaxPooling2D((2, 2)),\n \n # Another Convolutional layer with 64 filters of size (3, 3) and ReLU activation function.\n # This layer further increases the number of filters for more feature extraction.\n layers.Conv2D(64, (3, 3), activation='relu'),\n \n # Flatten layer to convert the 3D feature maps into a 1D vector.\n # This prepares the data for the fully connected layers.\n layers.Flatten(),\n \n # Dense (fully connected) layer with 64 units and ReLU activation function.\n # Dense layers are used to learn high-level features from the flattened data.\n layers.Dense(64, activation='relu'),\n \n # Final Dense layer with 10 units and no activation function specified.\n # The final layer has 10 units because the CIFAR-10 dataset has 10 classes, and the model needs to output probabilities for each class.\n # The lack of activation function (no activation='...') indicates that this is a linear layer (logits layer) without applying a nonlinearity.\n # During the training process, a softmax activation can be applied later to convert the logits into probabilities for each class.\n layers.Dense(10)\n])\n","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:12:28.02267Z","iopub.execute_input":"2023-08-02T04:12:28.024961Z","iopub.status.idle":"2023-08-02T04:12:28.133518Z","shell.execute_reply.started":"2023-08-02T04:12:28.024922Z","shell.execute_reply":"2023-08-02T04:12:28.132504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compile the CNN model with specified configurations for training.\n\n# Arguments:\n# - optimizer: The optimization algorithm used to update the model's weights during training.\n# 'adam' is a popular adaptive optimization algorithm that performs well in various tasks.\n# It adapts the learning rates of individual model parameters based on their past gradients.\n\n# - loss: The loss function used to measure the difference between predicted values and the true labels during training.\n# 'SparseCategoricalCrossentropy' is suitable for multi-class classification problems like CIFAR-10,\n# where each sample has a single true label (integer) out of multiple classes.\n# 'from_logits=True' indicates that the model's output does not have a softmax activation applied,\n# and the output is treated as logits, which will be later transformed into probabilities using the softmax function.\n\n# - metrics: A list of evaluation metrics to monitor during training.\n# In this case, 'accuracy' is used to track the model's classification accuracy on the training data.\n# It is the fraction of correctly predicted samples out of the total samples.\n\nmodel.compile(optimizer='adam',\n loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:12:28.138335Z","iopub.execute_input":"2023-08-02T04:12:28.140593Z","iopub.status.idle":"2023-08-02T04:12:28.160126Z","shell.execute_reply.started":"2023-08-02T04:12:28.140556Z","shell.execute_reply":"2023-08-02T04:12:28.159204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.config.update(model.get_compile_config())","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:12:28.164338Z","iopub.execute_input":"2023-08-02T04:12:28.166513Z","iopub.status.idle":"2023-08-02T04:12:28.174992Z","shell.execute_reply.started":"2023-08-02T04:12:28.166477Z","shell.execute_reply":"2023-08-02T04:12:28.174119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the CNN model on the provided training data.\n\n# Arguments:\n# - train_images: The 4D NumPy array containing the training images (input data) of shape (num_samples, height, width, channels).\n# - train_labels: The 1D NumPy array containing the labels for the training images (ground truth) of shape (num_samples,).\n\n# - epochs: The number of times the entire training dataset will be passed through the model during training.\n# Each epoch represents a full training cycle.\n\n# - validation_data: A tuple containing the validation data to evaluate the model's performance after each epoch.\n# It consists of two elements:\n# - test_images: The 4D NumPy array containing the test images (input data) of shape (num_samples, height, width, channels).\n# - test_labels: The 1D NumPy array containing the labels for the test images (ground truth) of shape (num_samples,).\n# The model's performance on the validation data will be monitored during training.\n\n# - callbacks: A list of callback functions to apply during training. In this case, a single callback is used:\n# 'WandbCallback()' from the Weights & Biases (wandb) library.\n# This callback logs training metrics and model performance to the WandB dashboard for experiment tracking.\n# wandb.watch(model, log='all', log_graph=True)\nmodel.fit(train_images, train_labels, epochs=4, validation_data=(test_images, test_labels), callbacks=[WandbCallback()])","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:12:28.180292Z","iopub.execute_input":"2023-08-02T04:12:28.183158Z","iopub.status.idle":"2023-08-02T04:13:35.672012Z","shell.execute_reply.started":"2023-08-02T04:12:28.183119Z","shell.execute_reply":"2023-08-02T04:13:35.670923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:13:35.673709Z","iopub.execute_input":"2023-08-02T04:13:35.674395Z","iopub.status.idle":"2023-08-02T04:13:40.206821Z","shell.execute_reply.started":"2023-08-02T04:13:35.674358Z","shell.execute_reply":"2023-08-02T04:13:40.205904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

\n ๐Ÿ“Š๐Ÿ“Š Example with Torch โšพโšพ\n

\n ","metadata":{}},{"cell_type":"markdown","source":"The following codes are inspired by https://theaisummer.com/weights-and-biases-tutorial/","metadata":{}},{"cell_type":"code","source":"# Define a series of image transformations for preprocessing\ntransform = transforms.Compose([\n transforms.ToTensor(), # Convert PIL image to PyTorch tensor\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) # Normalize the image with mean and standard deviation\n])\n\n# Download and load the CIFAR10 dataset for training\ntrainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)\n\n# Download and load the CIFAR10 dataset for testing\ntestset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)\ntestloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)\n\n# Define a custom neural network class\nclass Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n # Define the layers of the neural network\n self.conv1 = nn.Conv2d(3, 6, 5) # 3 input channels, 6 output channels, 5x5 kernel size\n self.pool = nn.MaxPool2d(2, 2) # Max pooling with a 2x2 kernel and stride of 2\n self.conv2 = nn.Conv2d(6, 16, 5) # 6 input channels, 16 output channels, 5x5 kernel size\n self.fc1 = nn.Linear(16 * 5 * 5, 120) # Fully connected layer with 120 output units\n self.fc2 = nn.Linear(120, 84) # Fully connected layer with 84 output units\n self.fc3 = nn.Linear(84, 10) # Fully connected layer with 10 output units (for 10 classes)\n\n def forward(self, x):\n # Define the forward pass of the neural network\n x = self.pool(F.relu(self.conv1(x))) # Convolution, ReLU activation, and max pooling\n x = self.pool(F.relu(self.conv2(x))) # Convolution, ReLU activation, and max pooling\n x = x.view(-1, 16 * 5 * 5) # Flatten the output for the fully connected layers\n x = F.relu(self.fc1(x)) # Fully connected layer with ReLU activation\n x = F.relu(self.fc2(x)) # Fully connected layer with ReLU activation\n x = self.fc3(x) # Final fully connected layer\n return x","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:13:40.210659Z","iopub.execute_input":"2023-08-02T04:13:40.211541Z","iopub.status.idle":"2023-08-02T04:13:41.815318Z","shell.execute_reply.started":"2023-08-02T04:13:40.211503Z","shell.execute_reply":"2023-08-02T04:13:41.814094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# setting device on GPU if available, else CPU\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Using device:', device)","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:13:41.816932Z","iopub.execute_input":"2023-08-02T04:13:41.817305Z","iopub.status.idle":"2023-08-02T04:13:41.823465Z","shell.execute_reply.started":"2023-08-02T04:13:41.81726Z","shell.execute_reply":"2023-08-02T04:13:41.822284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create an instance of the custom neural network 'Net' and move it to the specified device (GPU if available)\nnet = Net().to(device)\n\n# Create an Adam optimizer to optimize the parameters of the neural network during training\noptimizer = torch.optim.Adam(net.parameters())\n\n# Define the CrossEntropyLoss criterion, which is commonly used for multi-class classification tasks\n# It computes the negative log-likelihood loss between the predicted class probabilities and the ground truth labels\ncriterion = nn.CrossEntropyLoss()","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:13:41.825183Z","iopub.execute_input":"2023-08-02T04:13:41.825675Z","iopub.status.idle":"2023-08-02T04:13:41.838693Z","shell.execute_reply.started":"2023-08-02T04:13:41.825568Z","shell.execute_reply":"2023-08-02T04:13:41.837704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.init(project=\"cifar10-experiments\", name=f\"experiment-torch-{str(datetime.datetime.now())[-6:]}\", config={})","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:13:41.840404Z","iopub.execute_input":"2023-08-02T04:13:41.840796Z","iopub.status.idle":"2023-08-02T04:14:13.194008Z","shell.execute_reply.started":"2023-08-02T04:13:41.840762Z","shell.execute_reply":"2023-08-02T04:14:13.193105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set the number of epochs to train the network\nnum_epochs = 3\n\n# Start watching the neural network and criterion for logging purposes using Weights & Biases (wandb)\n# This function call enables automatic logging of gradients, weights, and other metrics during training\n# 'log='all'' indicates that all layers and parameters will be logged\nwandb.watch(net, criterion, log='all')\n\n# Loop over the epochs\nfor epoch in range(num_epochs):\n running_loss = 0.0 # Variable to keep track of the running loss for each epoch\n\n # Loop over the mini-batches in the training data\n for i, data in enumerate(trainloader, 0):\n inputs, labels = data[0].to(device), data[1].to(device) # Move inputs and labels to the device (GPU if available)\n \n optimizer.zero_grad() # Zero the gradients from the previous iteration\n outputs = net(inputs) # Forward pass: compute the output of the network\n loss = criterion(outputs, labels) # Compute the loss between the output and the labels\n loss.backward() # Backpropagation: compute the gradients of the loss with respect to the network's parameters\n optimizer.step() # Update the network's parameters using the computed gradients\n\n running_loss += loss.item() # Accumulate the loss for the current mini-batch\n\n if i % 2000 == 1999: # Print the average loss every 2000 mini-batches\n print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))\n wandb.log({'epoch': epoch + 1, 'loss': running_loss / 2000}) # Log the average loss to Weights & Biases\n running_loss = 0.0 # Reset the running loss for the next 2000 mini-batches\n\nprint('Finished Training') # Training is complete after looping over all epochs","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:14:13.198242Z","iopub.execute_input":"2023-08-02T04:14:13.200448Z","iopub.status.idle":"2023-08-02T04:17:40.139644Z","shell.execute_reply.started":"2023-08-02T04:14:13.200411Z","shell.execute_reply":"2023-08-02T04:17:40.138575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2023-08-02T04:17:40.14161Z","iopub.execute_input":"2023-08-02T04:17:40.142249Z","iopub.status.idle":"2023-08-02T04:17:44.525996Z","shell.execute_reply.started":"2023-08-02T04:17:40.14221Z","shell.execute_reply":"2023-08-02T04:17:44.524976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}