Deep Learning: Recurrent Neural Networks
in Python
LSTM, GRU, and more RNN machine learning architectures in
Python and Theano
By: The LazyProgrammer (https://lazyprogrammer.me)
Introduction
Chapter 1: The Simple Recurrent Unit
Chapter 2: The Parity Problem
Chapter 3: Recurrent Neural Networks for NLP
Chapter 4: Generating and Classifying Poetry
Chapter 5: Advanced RNN Units - GRU and LSTM
Chapter 6: Learning from Wikipedia Data
Conclusion
Introduction
Recurrent Neural Networks are all about learning sequences.
You may have already learned about Markov models for sequence modeling, which make
use of the Markov assumption, p{ x(t) | x(t-1), …, x(1) } = p{ x(t) | x(t-1) }. In other
words, the current value depends only on the last value. While easy to train, one can
imagine this may not be very realistic. Ex. the previous word in a sentence is “and”,
what’s the next word?
Whereas Markov Models are limited by the Markov assumption, Recurrent Neural
Networks are not. As a result, they are more expressive, and more powerful than anything
we’ve seen on tasks that we haven’t made progress on in decades.
In the first section of the book we are going to add time to our neural networks. I’ll
introduce you to the Simple Recurrent Unit, also known as the Elman unit.
The Simple Recurrent Unit will help you understand the basics of recurrent neural
networks - the types of tasks they can be used for, how to construct the objective functions
for these tasks, and backpropagation through time.
We are going to revisit a classical neural network problem - the XOR problem, but we’re
going to extend it so that it becomes the parity problem - you’ll see that regular
feedforward neural networks will have trouble solving this problem but recurrent networks
will work because the key is to treat the input as a sequence.
In the next section of the course, we are going to revisit one of the most popular
applications of recurrent neural networks - language modeling, which plays
Deep Learning_ Recurrent Neural Networks in Python_LazyProgrammer.pdf