Explore
For Enterprise
Join for Free
Log In

艺术与人文
商务
计算机科学
数据科学
信息技术
健康
数学和逻辑
个人发展
物理科学与工程
社会科学
语言学习
学位
证书

浏览 Coursera 的全部课程

浏览
搜索
企业版
登录
免费加入

Motivation for recurrent layers

Loading...

Introduction to Deep Learning

国立高等经济大学

4.6（1,004 个评分） | 47K 名学生已注册

课程 1（共 7 门，高级机器学习专项课程）

The goal of this course is to give learners basic understanding of modern neural networks and their applications in computer vision and natural language understanding. The course starts with a recap of linear models and discussion of stochastic optimization methods that are crucial for training deep neural networks. Learners will study all popular building blocks of neural networks including fully connected layers, convolutional and recurrent layers. Learners will use these building blocks to define complex modern architectures in TensorFlow and Keras frameworks. In the course project learner will implement deep neural network for the task of image captioning which solves the problem of giving a text description for an input image. The prerequisites for this course are: 1) Basic knowledge of Python. 2) Basic linear algebra and probability. Please note that this is an advanced course and we assume basic knowledge of machine learning. You should understand: 1) Linear regression: mean squared error, analytical solution. 2) Logistic regression: model, cross-entropy loss, class probability estimation. 3) Gradient descent for linear models. Derivatives of MSE and cross-entropy loss functions. 4) The problem of overfitting. 5) Regularization for linear models.

查看授课大纲

您将学习的技能

Recurrent Neural Network, Tensorflow, Convolutional Neural Network, Deep Learning

审阅

4.6（1,004 个评分）

5 stars
741 ratings
4 stars
183 ratings
3 stars
43 ratings
2 stars
18 ratings
1 star
19 ratings

YG

Jan 28, 2018

This is a very hands on Deep Learning class. Like the design of programming assignments a lot. It's very instructive as well as challenging! Great course. I would recommend it!

AS

Mar 26, 2018

Great course! The faculty does an excellent job in explaining some difficult to understand concepts. The discussion forum is very active and the course community is helpful.

从本节课中

Deep learning for sequences

In this week you will learn how to use deep learning for sequences such as texts, video, audio, etc. You will learn about several Recurrent Neural Network (RNN) architectures and how to apply them for different tasks with sequential input/output.

Motivation for recurrent layers7:36

Simple RNN and Backpropagation8:23

教学方

Evgeny Sokolov
Senior Lecturer
Andrei Zimovnov
Senior Lecturer
Alexander Panin
Lecturer
Ekaterina Lobacheva
Senior Lecturer
Nikita Kazeev
Researcher

脚本

Hi guys.

And welcome back to our Deep Learning Course.

My name is Kate, and this week we'll speak about Deep Learning Architectures for

sequential data.

First, let's understand in which tasks do we really need

to work with sequential data.

I think all of you know at least several examples of such data.

Text, Video, and Audio, you use them everyday, and all of them are sequences.

Text is a sequence of sentences or words or even letters.

Video is a sequence of images.

And audio is a sequence of sounds.

But these are only the most common examples of such data.

There are a lot more of them.

You can find time series in a lot of different areas, in finance,

in industry, in medicine, and so on.

And time series, of course, are also sequences in time.

For example,

in medicine, all of the data from medical sensors are usually sequential.

So as you can see, there are a lot of different tasks with sequential data.

And this week, we'll learn how to use Deep Learning to solve these tasks.

First, I will introduce you to a specific example of simple task with

sequential data called Language model.

In this task we want to train a generative model of natural language.

So, we want to be able to generate text from the probability distribution P

of text.

This probability distribution shows us which text is probable in our language,

and which text is actually not a text, but only a sequence of random words.

Here, we suppose that we generate texts word by word.

So, the probability distribution of our text, the probability of our text

is actually a product of probabilities of the words from this text.

All of these probabilities are conditional probabilities

of current word given all the previous words.

We can actually use Language model in all the different tasks in practice.

For example, we can use it to generate answers in chatbots or

question answering systems.

But of course, here,

we should condition our Language model on the previous conversation,

because the model should know what the question was about to answer it.

Also, Language model can be useful in machine translation or

speech recognition to understand which phrase is more natural to a language.

And actually, we can use Language model in a lot of other tasks about text analysis.

Now, let's try to use a standard multilayer perceptron to construct our Language model.

Let's say that we've already generated i words, and

we want to generate the next word, i + 1.

Then, we can use i words we have as an input on our MLP.

Of course, here, we need to use some numerical features for these words,

for example one-hot encodings or word2vec embeddings.

And as an output of our MLP

we can have a distribution over our vocabulary,

which says which word is more probable to be the next one in our sequence.

Okay, this model will work.

And now, we have i + 1 words, and we want to generate the next word, i + 2.

And what we want to do,

we actually want to give all the words we have as an input to our MLP.

But we can't do it because MLP can work only with fixed number of inputs.

So here we see the main problem of using standard MLP with sequential data.

The problem is that sequences can have different lengths.

So, what should we do with this?

We can use a simple heuristic and give not all of the previous words as an input

to our MLP, but only a fixed number of them.

So, we have a window of fixed size as an input.

This technique works, but it's only a heuristic, and it is not

clear how to choose the size of the window for different tasks in practice.

And in some tasks, we actually need a very wide window to work with.

For example, let's say that we want to generate the story.

And we write it on the first page that we have a character named King Louis, and

now we're on the fifth page.

And we generated word King, and it's clear that the next word should be Louis.

But our model understands this only if the window is more than five pages,

and five pages is very big window, there are a lot of words in them.

So, in this case, our MLP has a lot of parameters.

And that's a problem.

Let's, for example, calculate how many parameters are in the first layer of

our MLP if it contains 100 hidden neurons.

We use window of size 100, and all of our words are described to be of size 100.

Yes, there are more than a million parameters

only in the first layer of our network.

And here, you need to understand that these characteristics are actually not that large.

I mean, the network with 100 hidden neurons, is a very small network.

In practice we need to be able to work with much larger networks.

So that's a problem.

To overcome this problem we can use a recurrent architecture.

In this architecture, we work with a sequence word by word.

So we see only one word in one time step.

Here, our MLP has not only one output which it transfers to the next layer, but

also the second output which it transfers to the next time step.

Therefore, now it's not necessary to our MLP

to use all of the previous words as an input at the same time.

It's enough to take only one current word, but

also to take it's own second additional output from the previous time step.

As a result, our MLP now has the fixed number of parameters.

But still, all the information about the previous words in the sequence.

So now, we can work with the sequences of arbitrary length.

Of course, here,

we need to understand that at first step we need to use some initial vector as

an output from the previous time step, because we don't have previous time step.

Additionally, the dependence between next word that we want to generate and

all of the previous ones is actually independent of the time step.

So we can use this same MLP at each time step.

As a result, all the parameters of our MLP shared between time steps, and

we need much less number of parameters.

Now lets return to our example.

Let's calculate the number of parameters in the first layer of our MLP

if it contains 100 hidden neurons.

All of the words are described with word embeddings of size 100, and

the think that MLP transfer from one step to another

is these exact 100 hidden neurons from the first layer.

Yeah, now we have only 20,000 of parameters.

That's two orders of magnitude less than earlier, so this technique works.

Now, let's summarize what we've learned in this video.

Now we know that there are a lot of different tasks with sequential data, and

we need to know how to work with them.

And we know that feedforward neural networks are not very useful in such

tasks because of arbitrary lengths of sequences, and

also because of large number of parameters.

And here we can use recurrent architecture which is much more useful.

In the next video we will speak about Simple Recurrent Neural Network and

how to train it.

[MUSIC]

探索我们的目录

免费加入并获得个性化推荐、更新和优惠。

Coursera 致力于普及全世界最好的教育，它与全球一流大学和机构合作提供在线课程。

© 2019 Coursera Inc. 保留所有权利。

通过 App Store 下载

通过 Google Play 获取

Coursera
关于
管理团队
工作机会
目录
证书
MasterTrack™ 证书
学位
企业版
政府版

社区
Learners
Partners
Developers
Beta Testers
Translators

连接
博客
Facebook
领英
Twitter
Instagram
YouTube
技术博客

更多
条款
隐私
帮助
内容访问
媒体
联系我们
目录
附属公司