{"cells":[{"metadata":{},"cell_type":"markdown","source":"# What?\nThis is the second notebook for the same audience who knows too much about the **Overview** of RNN but very less about the exact working of it. Another notebook is for [CNNs and it's internal workings](link). I hope you have a basic understanding about RNN or GRU or LSTMs. If not, please read them there as it is just a few minutes of read and I can not cover very basics of RNN but just trying to substantiate my knowledge by writing a notebook. I am just trying to help the people learn the internal workings of different complicated things in a very simplified way.\n\n**[For same in-depth intutive knowledge on CNNs, Visit this link which might look like a Clickbait but isn't ](https://www.kaggle.com/deshwalmahesh/you-ve-been-looking-at-the-cnn-the-wrong-way)**\n\n**PLEASE CORRECT ME IF I AM WRONG**. I'll be more than happy to be corrected and improve than being wrong and .... you know the rest!"},{"metadata":{},"cell_type":"markdown","source":"# What Exactly is RNN\nIf I can say frankly one thing today, it'll be that people are here for the `LSTM` which they mostly use. No one uses `RNN` specially after the launch of `Transformers`. So we'll use RNN = LSTm = GRU until or unless there is a need to specify. There networs are used and useful specially when the data data is related in time. Such as `Time Dependent` data such as stock market data, Text data which are dependent on the previous as well as future words etc. Why do we use RNN? Good question1 Because they are good at remembering things. They can remember something like:\n\n\"World\" has a higher probability after the word \"Hello\" and before the word \"War\". So basically the RNNs try to cram the pattern of how words can appear in different combinations qnd tend to predict. If you pass in the 30 years of Stock Market data to highly optimised RNN, chances are that you can be the next Elon Musk but again, you know RNN have become obsolete and Elon is part of [Paypal Mafia](https://en.wikipedia.org/wiki/PayPal_Mafia) ;)"},{"metadata":{},"cell_type":"markdown","source":"# Misconceptions and Queries\nBefore Telling you about how it exactly works there are a few misconceptions that people have regarding the working of the structure.\n\n## Misconceptions:\n1. RNN works in Parallel\n2. There is just one Weight matrix which is important\n3. No of neurons is \"How many time steps to create\n4. Hidden state is a number\n5. There has to be multiple inputs\n6. Weights of matrix change in RNN\n\n## Queries:\n1. What does it mean to have \"Number of Neurons\" in an RNN\n2. What gets saved when you save weights of a model\n3. What is the output of the model\n4. How does the \"Statte\" change (In Simple Terms)\n5. What are `return_sequence` and `return_state`?"},{"metadata":{},"cell_type":"markdown","source":"# How do they work?\n\nLet's look at the image below and I'lll try to tell you in how many ways you can use RNN and then the structure itself.\n\n\n\nNow look at the Usage of Each type with Example:\n\n1. One-to-one: Typical Neural Network just like a **Dense Network but with extra complexity.**\n2. One-to-many: Story Generation, Music Generation by giving just a single starting data. This one is artist I'm telling you\n3. Many-to-one: This one you are familiar with and use it to the fullest. Yes, Classification. Sentiment classification specially. **Only Last (Final) Hidden State is Used** (Will discuss later).\n4. Many-to-many: A bit tricky but yeah, used for Named Entity Recognition (NER). Here instead of a single result, each \"WORD\" has belongs to a probability distribution (fancy right?). Probability distribution means that we want to classify what the current word belongs to out of (Verb, Noun, Pronoun....) OR (Number, Place, Organization, Date bl bla bla). Each word will have all of the possible values and we choose for each word. `Output Matrix: WhY` is used in this case, per word. \n5. Many-to-many: Used with \"Machine Translation\". English - French conversion, Generating Captions etc. You can use `Seq2Seq` architecture with that. You pass in all the words one by one and then output `Last CELL State` (I repeat CELL). That Last cell state is now used as an Initialize the **NEW RNN** and `Hidden State` to `Generate a new FIRST word`. That word will go into the RNN along with that state and generate a new word again and this happens till you tell the RNN to stop or generate the `` token\n\nI highly recommend you to [go and check this page from Stanford University](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks)\n"},{"metadata":{},"cell_type":"markdown","source":"# The infamous confusing diagram simplified (In the end)\n\nLook at he diagram below (this'll be your second last time, I promise)\n\n\n\nIt confuses lots and lots of people. The left had side `Folded` version is hard to understand for newcomers but the right hand one `UnFolded` version is deceptive enough to get your concepts wrong. This the part where people leave reading in between and think ,\"yeah! okay! It was confusing so made up my mind without reading and what I know now will be correct in future\". No bro! you are in for a show, though!"},{"metadata":{},"cell_type":"markdown","source":"# Training Data Example"},{"metadata":{},"cell_type":"markdown","source":"Let us look at a real time example of how the data looks like and then what is done inside the `RNN(256) / LSTM(256) / GRU(256)` you write.\n\nThe Input data looks something like this:\n\n"},{"metadata":{},"cell_type":"markdown","source":"1. Each **BLUE** column is a word represented as a vector. Either One-Hot Encoded `[0,0,0,0,1,0,0]` etc or `[0.2,0.5,0.8,0.01,0.3]` Embeddings. The length of columns is usually defined by `Embedding_dim`.\n\n2. Each **Grey** column is a `Padding` vector generally denoted by all zeros of same length above `[0,0,0,0,0,0,0]`. It is here to tell you that your text had less number of words than the expected so we gave you all zeros. What a way to support someone! It is defined by `padding='post'`. You can also add zeros in the starting too.\n\n3. The X-axis is the number of words present in a `Text/Line/Sentence/document`. These are defined by `max_len` parameter people use without knowing how it works.\n\n4. Y-axis is how many such Text/ Data Points are there to be processed. It is defined by `Batch_size`.\n\n5. There are columns that are not presented here in the middle row because they have been truncated due to `truncate=\"True\"` so it just trimmed of all the words beyond `max_len`. \n\nSo let us suppose we have `batch_size=4, max_len=3, padding='post', embed_dim=2`, `truncate=True`. And if in a batch of 4, you have data something like:\n\n`\n['hello my name is',\n'what',\n'my name is',\n'slim shady']\n`\n\nSo let us suppose the we have a vocab which is a dictonary and it'll give numbers to the words because machines know numbers only! What a dumb machine!!!\n\nhello=1, my=2, name=3, is=4, what=5, slim=6, shady=7 \npadding=0\n\n`\n[[1,2,3], # max_len = 3 and truncate=True so chipped off everything\n [5,0,0], # padding = 'post', max_len=3\n [2,3,4],\n [6,7,0]]\n`\n\nAgain, these numbers will be converted as `one_hot(numbers)` and then it'll go to the the `Embedding` Layer. Why One-Hot? Because machines think 3>2 and 1<2. It is the case but you'll be saying that `hello` < `my` and `shady`>`slim` which does not make any sense. (when he is the greatest! :/). So we pass it ot embedding Layer and it'll give us **A vector of length `embed_dim=2` for EACH word in EACH sentence**. To know what are embeddings and how can they be used, [Check this Notebook out](https://www.kaggle.com/deshwalmahesh/nlp-beginner-1-rnn-lstm-gru-embeddings-glove).\n\nSo now your data will look something like:\n\n`\n[[[0.7,0.9],[0.32,0.33],[0.01,0.93]], \n [[0.23,0.32],[0.0,0.0],[0.0,0.0]],\n [[0.34,0.23],[0.33,0.34],[0.94,0.23]],\n [[0.97,0.73],[0.67,0.34],[0.0,0.0]]]\n`\n\nNow doesn't it look like the infamous `3-D` input? or the figure above? Go check that figure now and relate.\n1. The inner most list is the length of column or a single Word\n2. Middle list is the collection of words called Sentence\n3. Outer most list is the batch size or how many texts to process/ train at one time\n\nNOTE:These floating numbers given to words are by EMBEDDING Layer. Also, a word can have same vector representation on embedding OR can have a different vector depending on the type of Embedding use such as ELMO. We have given different numbers to show you how EMLO can output a vector based on the context and position of word. It is just a dummy example.."},{"metadata":{},"cell_type":"markdown","source":"# Finally the working\n\n## What Happens\n\n\n\nYup! I know right? Now digest these facts:\n\n1. RNN DO NOT WORK IN PARALLEL (End of discussion): The worst thing that unfolded version can do to people is to make them believe that the words (or data) go in parallel but in fact, one word will go in, change the state matrix `H` and is stored somewhere (just a demonstration like a dictonary) as `{'first_word': H1`}, gives output `Y1` (depends if you want or not) then again second word will go in, change the matrix and it'll be like `{'first_word': H1,'second_word': H2}`, output `Y2` and third will go in, change matrix and so on.\n\n2. There are usually 3 matrices: Yes, there are 3 not 1. Also, 2 extra biases too. Also, all of those matrices are shared across each time stamp (Yup! I said that)\n\n3. Hidden State is not a single number (Start of discussion): `Number of Neurons` is equal to the dimensionality of Hidden State VECTOR `RNN(250)` means that the size of Hidden state VECTOR should be 250. Do not get confused by `Dense(250)`\n\n## How does it happen?\nLet us suppose you initialise a `RNN(256)` with `embed_dim=100`. Now look at that infamous Folded- Unfolded Diagram again (told you that was your second last time) and understand These:\n0. `X`: Word or Word-Embedding of shape `embed_dim x 1` say `100x1` here.\n1. `H`: Hidden State VECTOR of shape `256 x 1` = No. of Nodes in `RNN(256)`. Initialised usually by ZEROS (or randomly) and the numbers inside it change after backpropagation. Value of this matrix is different for different time steps as each incoming word changes its value. \n2. `Wh`: State matrix of shape `256x256`. It is mostly random or zeros in the starting. Responsible for changing the `H`. Its value change ONLY ONCE after back propagation and is same for every time step. So it is one of the shared matrix.\n3. `WhX`: Input weight matrix os shape `256 x 100`. It gets multiplied to EVERY input `X` at each time step. Responsible Indirectly for changing the Hidden State `H`. It is also shared and changed only once during backprop.\n4. `WhY`: Output Matrix of shape `100 x 256`. It gets multiplied to Hidden State VECTOR and is responsible for giving the output for each input word at each time step. \n5. `Bx`: Bias term for Each Input \n6. `By`: bias term for **Each** Input\n7. `Yt`: Output vector for each word of shape `100 x 1`\n\n# Steps\n1. Initially all the matrices have random weights and Hidden State Vector is all zeros when you initialize the `RNN(256)` layer.\n2. First word vector `X0` at time `t0` will go in. It'll get multiplied to the matrix `WhX` and the bias `Bx` will be added to it. Let us say it is stored in `part1`. So `part1 = (WhX * X) + Bx`\n3. In the meantime, the initial Hiddden vector `H0` which will be all zeros mostly will get multiplied to `Wh`. Let us say it is `part2 ` and it'll be `part2 = Wh * H0`\n4. Now both `part1` and `part2` will get multiplied and passed to an activation function usually `tanh` and it'll create new hidden state vector. So `H1 = tanh(part1 + part2)`\n5. Now, the weight matrix `WhY` will get multiplied to `H1` (**PAY ATTENTION to *H1* which was obtained in previous step**) and bias `By` will be added to it and passed thropugh a softmax. So output `Y0` for input `X0` will be computed as: `Y0 = softmax((WhY * H1) + Wb)`\n6. Now the second word `X1` at time `t1` word vector will go in, gets multiplied to the matrix `WhX` (**As it is shared, so it'll be same**), add the bias `Bx`.\n7. Multiply `Wh` (**Again, same weight**) with `H1` (**Only thing changed**) and add to the output of **Step 6** and pass it to `tanH`.\n\nThese process will keep going on for `max_len` times because every word will keep on adding and changing the weight matrix. At the end, after the last word goes and changes hidden vector, it'll be outputted by the `RNN()`."},{"metadata":{},"cell_type":"markdown","source":"# Backprop!\n\n\n\nYes! exactly!! Only place where it all happens. You can multiply billons of parameters but what does that signify? If you can not learn anything? Why are your learning? What exactly are you larning? How are you learning and son... \n\nNewbies sell the sacred Backprop (BPTT in our case) in interviews like fishes in the market. You can smell the quality ;). I'll give you a link later which will in turn give you a brain headache (Backprop in one line).\n\nIt works like a man is costing people for the loss in his company. So he goes to the manager and says you are responsible. You need to give me 10 Lakhs (Total Loss `L` wrt to `X`). Manager calculates it and then says to 3 employees under him that I'll be giving 1 Lakh to boss and all of you 3 should be giving me `[3,4,2]` lakhs depending on how much respoinsible they were. So the each of 3 passes the loss down the layer and this keeps on happening for all those who were responsible.\n\n\nSo we are trying to UPDATE the weights of those 3 matrices. WHY? Because we have to learn something and we do it by updating those numbers inside our matrices because they are only matrcies our results are dependent upon. Let us take a complex example of `NER` or `Named Entity Recognition`. So what it'll do is let us suppose we have a big paragraph and we want to know which words belong to which category?\n\nLet us suppose we have different UNIQUE labels such as: `God, Fictional, Garbage, Stopword, Profession` bla bla bla and we pass a sentence to a very optimised and intelligent model. That model will give a prediction about **EACH** word that it belongs to any of the above given labels. So let us suppose we have our sentence as `Eminem is God and Kanye is a rapper`. so each word is our `X` input and the resulting output will be `Eminem:God, is:stopword, God:fictional, and:stopword, Kanye:garbage, is:stopword, a:stopword, rapper:professsion`. This is ideal scenario and loss is 0. But models don't do that they try to predict and ALWAYS produce some loss no matter the scenario.\n\n## Steps for Backprop\n1. So when each word `Xi` was being fed to the network during training, there was a separate dictonary which was calculating the Loss `Li` for each output `Yi`. Remember it is a sequentially wornking not parallely.\n2. All the words are fed. Now is the portion which takes TOO much time. So what happens here is that all the losses from each words are summed up as a total loss `L` and the machine starts roaring to calculate the derivative of loss. \n3. It'll calculate the loss to adjust the matric of `WhY`. What is `WhY` dependent on? `H`. Now `H` on? `Wh` + `WhX`? You see the pattern? \n4. Now the Loss was dependent on `WhX` as it is also what we were learning right? Same will happen.\n5. That was BPTT. Not the slightest of the hints of BPTT. Below is the real BPTT.\n6. What we call BPTT is the idea that in order to calculate the total loss, we have to calculate `Wh`, `WhX` and `Why`. And for the `L3` or third word's loss, we need its path. And for it's path: We have to calculate `L2` first and in order for `L2`, we need `L1`. and in order to `L1`, we need the loss of `Wh`, `WhX` and `WhY` of that specific node.\n6. That was BPTT for you.\n\nNote: It is not calculating that `Loss L` in loops. It is sitting there are thinking too much. Calculating it for every `Y` at once and all those responsible using chain rule.. \n\n[Just visit this link. If you have problems with maths, well still see it many times to get an understanding](https://www.youtube.com/watch?v=phOVApJHjsU)"},{"metadata":{},"cell_type":"markdown","source":"# Some Popular queries\n\n## What does this `H` vector signifies?\nWhy is this `H` gets changed? Okay! So it works like having a memory for the network. It'll look at the first word and think okay! so this the word (information) I'll remember it. Now the next word will go in and it'll save **SOMETHING** about the the two words. So every time a word goes in, RNN is trying is to remember **Something**. Why **Something**? Because we can not make sense out of the numbers. So it could be **Grammar, sequence of words** or some deeper meaning which we can not put into words (as of now). So this `H` stores **Some Useful** information about all the words it has seen up until now. The moment it'll encounter a new word, it'll store its information too.\n\n## Okay! So why these 3 matrices are used?\nNice one! So these matrices try to control things. By multiplying `H` with `Wh`, we are trying to say that **Do not send whole of the information to the next step but extract some useful and then send it**. Without it, it'll be sending too much information with `H` so `Wh` tries to extract and control the amount of information to be extracted and send to next time step.\n\nWhat about `WhX` and `WhX`?: These try to control the amount of information a word can provide to network. Without`WhX`, a single word can put too much information in the network and it can produce the skewness in the statistics and probability due to that word. `WhY` is used so that the output `Y` of the word `X` is not directly dependent on the Hidden state. It makes sense because without it, there won't be any difference between output `Y` and hidden state vector `H` at any point of time `t`.\n\n## Why bias? \nNow you're asking too much ;)\n\nBias is used to provide `SLOPE` to the line = COntrol the activation threshold. So without bias, we'll be saying something like: Activate all the neurons which have a value greater than 0. What if? What if we do not want to have that threshold? What if all the neurons are above 0? This is where the Bias will come into play. It'll adjust it's value so that we'll be saying that\" Activate all the neurons which are greater than the bias value `b`. So if bias `b` is 2, all the neurons greater than 2 will be activated only instead of default 0.\n\n## Why are weights shared?\nPfff!!! That's hard!! \n\nSo when the RNNs were developed, we did not have enough computational power. So let us suppose that we do it for a sentence with length 50 with `RNN(128)`. So it'll create 50 * 3 = 150 weight matrices, 50 * 2 = 100 biases for of size 128. Calculate how many neurons will be used? It'll blow the memory up.\n\n**BUT**: The main reason is not this. The main reason is that sharing the weights will help the RNN genralise the model for unknown data or words and for different length sentences. So if all of these matrices keep on changing for each word, it won't be able to generalise as it'll learn too much based on each individual word not the sentence collectively.\n\nSo parameter sharing reflects the fact that we are performing the same task at each step, as a result, we don't have to relearn the rules at each point in the sentence. [Visit this ](https://stats.stackexchange.com/questions/221513/why-are-the-weights-of-rnn-lstm-networks-shared-across-time) and [This answer](https://stackoverflow.com/questions/47865034/recurrent-nns-whats-the-point-of-parameter-sharing-doesnt-padding-do-the-tri)"},{"metadata":{},"cell_type":"markdown","source":"## Difference between `dropout` and `recurrent_dropout`?\n1. `dropout` happens when you try to randomly set some % of the neurons of `part1` as 0. So those neurons won't add any information\n2. `recurrent_dropout` is making some % of neurons to 0 of `H` vector.\n\n[Look at the awesome answer here](https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout) and [the second answer here](https://stackoverflow.com/questions/50720670/using-dropout-with-keras-and-lstm-gru-cell)\n\n## What is `return_state` and `return_sequence` and where to use those?\nBy default `Keras` returns the `Hidden State Vector` from the last state after looking at the whole sentence (**Now I've seen Everything**). That vector has the **Collective** information about `What special information each word added`. It is the most used scenario where you use it for Classification problems. That vector has all the necessary information to get classified.\n\nWhen you do `return_sequence`, you are saying that give me Hidden state up until now. So for example, if you return Hidden state for 4th word, it is saying that give an output based on information from all the first 3 words + The information that I just added. You can use these settings for `NER Models`.\n\nFor the `return_state`, You are you are trying to output the `Wh`. Remmeber `Wh` is shared so it does not matter for which time step you return, you'll have same matrix. It is used in scenarion where you have initialise a new RNN based on this values instead of random. Such as in `Seq2Seq` models.\n\nYou can set both the params true at once to return states and sequence too.\n\nLook at these two blogs to know more:\n1. [Blog1](https://www.dlology.com/blog/how-to-use-return_state-or-return_sequences-in-keras/)\n2. [Blog 2](https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/)"},{"metadata":{},"cell_type":"markdown","source":"# A word of Wisdom\n\nYou can use `RNN` (No one uses that now. I know you'll use LSTM) with `Text`, `Stock: TIme Series`, `Audio` as well as `Video` and `Image` data too. I have imported all the 5 different types of data for you to test. We all know `LSTM` for text and time-series but try it with others too. I'm providivg the links for some help:\n\n1. [Video](https://machinelearningmastery.com/how-to-develop-rnn-models-for-human-activity-recognition-time-series-classification/)\n2. [Audio](https://www.kaggle.com/carlolepelaars/bidirectional-lstm-for-audio-labeling-with-keras)\n2. [Image](https://github.com/jiegzhan/image-classification-rnn)"},{"metadata":{},"cell_type":"markdown","source":"# Until next time!\nWait! please make any correction if needed. I don't want the younger ones to be misinformed like me ;("}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}