The goal of this course is to give learners basic understanding of modern neural networks and their applications in computer vision and natural language understanding. The course starts with a recap of linear models and discussion of stochastic optimization methods that are crucial for training deep neural networks. Learners will study all popular building blocks of neural networks including fully connected layers, convolutional and recurrent layers. Learners will use these building blocks to define complex modern architectures in TensorFlow and Keras frameworks. In the course project learner will implement deep neural network for the task of image captioning which solves the problem of giving a text description for an input image. The prerequisites for this course are: 1) Basic knowledge of Python. 2) Basic linear algebra and probability. Please note that this is an advanced course and we assume basic knowledge of machine learning. You should understand: 1) Linear regression: mean squared error, analytical solution. 2) Logistic regression: model, cross-entropy loss, class probability estimation. 3) Gradient descent for linear models. Derivatives of MSE and cross-entropy loss functions. 4) The problem of overfitting. 5) Regularization for linear models.
Motivation for recurrent layers
Loading...
来自 National Research University Higher School of Economics 的课程
Introduction to Deep Learning
457 个评分
National Research University Higher School of Economics
457 个评分
课程 1（共 7 门，Specialization Advanced Machine Learning）
从本节课中
Deep learning for sequences
In this week you will learn how to use deep learning for sequences such as texts, video, audio, etc. You will learn about several Recurrent Neural Network (RNN) architectures and how to apply them for different tasks with sequential input/output.
与讲师见面
Evgeny SokolovSenior Lecturer
HSE Faculty of Computer Science
Andrei ZimovnovSenior Lecturer
HSE Faculty of Computer Science
Alexander PaninLecturer
HSE Faculty of Computer Science
Ekaterina LobachevaSenior Lecturer
HSE Faculty of Computer Science
Nikita KazeevResearcher
HSE Faculty of Computer Science
Hi guys.
And welcome back to our Deep Learning Course.
My name is Kate, and this week we'll speak about Deep Learning Architectures for
sequential data.
First, let's understand in which tasks do we really need
to work with sequential data.
I think all of you know at least several examples of such data.
Text, Video, and Audio, you use them everyday, and all of them are sequences.
Text is a sequence of sentences or words or even letters.
Video is a sequence of images.
And audio is a sequence of sounds.
But these are only the most common examples of such data.
There are a lot more of them.
You can find time series in a lot of different areas, in finance,
in industry, in medicine, and so on.
And time series, of course, are also sequences in time.
For example,
in medicine, all of the data from medical sensors are usually sequential.
So as you can see, there are a lot of different tasks with sequential data.
And this week, we'll learn how to use Deep Learning to solve these tasks.
First, I will introduce you to a specific example of simple task with
sequential data called Language model.
In this task we want to train a generative model of natural language.
So, we want to be able to generate text from the probability distribution P
of text.
This probability distribution shows us which text is probable in our language,
and which text is actually not a text, but only a sequence of random words.
Here, we suppose that we generate texts word by word.
So, the probability distribution of our text, the probability of our text
is actually a product of probabilities of the words from this text.
All of these probabilities are conditional probabilities
of current word given all the previous words.
We can actually use Language model in all the different tasks in practice.
For example, we can use it to generate answers in chatbots or
question answering systems.
But of course, here,
we should condition our Language model on the previous conversation,
because the model should know what the question was about to answer it.
Also, Language model can be useful in machine translation or
speech recognition to understand which phrase is more natural to a language.
And actually, we can use Language model in a lot of other tasks about text analysis.
Now, let's try to use a standard multilayer perceptron to construct our Language model.
Let's say that we've already generated i words, and
we want to generate the next word, i + 1.
Then, we can use i words we have as an input on our MLP.
Of course, here, we need to use some numerical features for these words,
for example one-hot encodings or word2vec embeddings.
And as an output of our MLP
we can have a distribution over our vocabulary,
which says which word is more probable to be the next one in our sequence.
Okay, this model will work.
And now, we have i + 1 words, and we want to generate the next word, i + 2.
And what we want to do,
we actually want to give all the words we have as an input to our MLP.
But we can't do it because MLP can work only with fixed number of inputs.
So here we see the main problem of using standard MLP with sequential data.
The problem is that sequences can have different lengths.
So, what should we do with this?
We can use a simple heuristic and give not all of the previous words as an input
to our MLP, but only a fixed number of them.
So, we have a window of fixed size as an input.
This technique works, but it's only a heuristic, and it is not
clear how to choose the size of the window for different tasks in practice.
And in some tasks, we actually need a very wide window to work with.
For example, let's say that we want to generate the story.
And we write it on the first page that we have a character named King Louis, and
now we're on the fifth page.
And we generated word King, and it's clear that the next word should be Louis.
But our model understands this only if the window is more than five pages,
and five pages is very big window, there are a lot of words in them.
So, in this case, our MLP has a lot of parameters.
And that's a problem.
Let's, for example, calculate how many parameters are in the first layer of
our MLP if it contains 100 hidden neurons.
We use window of size 100, and all of our words are described to be of size 100.
Yes, there are more than a million parameters
only in the first layer of our network.
And here, you need to understand that these characteristics are actually not that large.
I mean, the network with 100 hidden neurons, is a very small network.
In practice we need to be able to work with much larger networks.
So that's a problem.
To overcome this problem we can use a recurrent architecture.
In this architecture, we work with a sequence word by word.
So we see only one word in one time step.
Here, our MLP has not only one output which it transfers to the next layer, but
also the second output which it transfers to the next time step.
Therefore, now it's not necessary to our MLP
to use all of the previous words as an input at the same time.
It's enough to take only one current word, but
also to take it's own second additional output from the previous time step.
As a result, our MLP now has the fixed number of parameters.
But still, all the information about the previous words in the sequence.
So now, we can work with the sequences of arbitrary length.
Of course, here,
we need to understand that at first step we need to use some initial vector as
an output from the previous time step, because we don't have previous time step.
Additionally, the dependence between next word that we want to generate and
all of the previous ones is actually independent of the time step.
So we can use this same MLP at each time step.
As a result, all the parameters of our MLP shared between time steps, and
we need much less number of parameters.
Now lets return to our example.
Let's calculate the number of parameters in the first layer of our MLP
if it contains 100 hidden neurons.
All of the words are described with word embeddings of size 100, and
the think that MLP transfer from one step to another
is these exact 100 hidden neurons from the first layer.
Yeah, now we have only 20,000 of parameters.
That's two orders of magnitude less than earlier, so this technique works.
Now, let's summarize what we've learned in this video.
Now we know that there are a lot of different tasks with sequential data, and
we need to know how to work with them.
And we know that feedforward neural networks are not very useful in such
tasks because of arbitrary lengths of sequences, and
also because of large number of parameters.
And here we can use recurrent architecture which is much more useful.
In the next video we will speak about Simple Recurrent Neural Network and
how to train it.
[MUSIC]
