Have you ever heard about such technologies as HDFS, MapReduce, Spark? Always wanted to learn these new tools but missed concise starting material? Don’t miss this course either! In this 6-week course you will: - learn some basic technologies of the modern Big Data landscape, namely: HDFS, MapReduce and Spark; - be guided both through systems internals and their applications; - learn about distributed file systems, why they exist and what function they serve; - grasp the MapReduce framework, a workhorse for many modern Big Data applications; - apply the framework to process texts and solve sample business cases; - learn about Spark, the next-generation computational framework; - build a strong understanding of Spark basic concepts; - develop skills to apply these tools to creating solutions in finance, social networks, telecommunications and many other fields. Your learning experience will be as close to real life as possible with the chance to evaluate your practical assignments on a real cluster. No mocking, a friendly considerate atmosphere to make the process of your learning smooth and enjoyable. Get ready to work with real datasets alongside with real masters! Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.
Block and Replica States, Recovery Process 1
Loading...
来自 Yandex 的课程
Big Data Essentials: HDFS, MapReduce and Spark RDD
160 个评分
从本节课中
What are BigData and distributed file systems (e.g. HDFS)?
与讲师见面
Ivan PuzyrevskiyTechnical Team Lead
Emeli Dral
Evgeniy Riabenko
Alexey A. DralHead of Big Data and Machine Learning
Algorithms and Programming Technologies dept. MIPT
Hello. In the last video,
I skimmed through the topic of HDFS recovery process.
If you are going to be a data engineer,
never skim any information through.
Give it an in-depth studying to get a good understanding of the process.
Right now it might be a good idea to re-watch my last video.
Of course, I'm kidding.
To eliminate multiple minutes to be precise and choosy.
Let me introduce a few concepts.
Block and replica.
Replica is a physical data storage on a data node.
There are usually several replicas with the same content on different data nodes.
Block is a meta-information storage on a name node
and provides information about replica's locations and their states.
Both replica and block have their own states.
There are the following data node replica's states.
If replica is in a finalized state
then it means that the content of this replica is cool and icy.
Technically speaking is frozen.
The latter means that meta-information for this block on name node
is aligned with all the corresponding replica's states and data.
For instance you can safely read data from
any data node and you will get exactly the same content.
This property preserves read consistency.
Each block of data has a version number called Generation Stamp or GS for short.
For finalized replicas, you have a guarantee that all of
them have the same GS number which can only increase over time.
It happens during error recovery process or during data appending to a block.
State RBW stands for Replica Being Written to.
It is the state of the last block of
an open file or a file which was reopened for appending.
During this state different data nodes can return to use a different set of bytes.
In short, bytes that are acknowledged by
the downstream data nodes in a pipeline are visible for a reader of this replica.
Moreover, data node on disk data and
name node meta-information may not match during this state.
In case of any failure data node will try to preserve as many bytes as possible.
It is a design goal called data durability.
Replica Waiting to be Recovered or RWR for short,
is a state of all Being Written replicas after data node failure and recovery.
For instance, after a system reboot or after Pacer.sys or BSOD,
which are quite likely from a programming point of view.
RWR replicas will not be in
any data node pipeline and therefore will not receive any new data packets.
So they either become outdated and should be discarded,
or they will participate in
a special recovery process called a lease recovery if the client also dies.
HDFS client requests a lease from a name node to
have an exclusive access to write or append data to a file.
In case of HDFS client lease expiration,
replica transition to a RUR state.
RUR stands for Replica Under Recovery.
Lease expiration usually happens during the client's site failure.
As data grows and different nodes are added or removed from a cluster,
data can become unevenly distributed over the cluster nodes.
A Hadoop administrator can spawn a process of data re-balancing or
a data engineer can request increasing of
the replication factor of data for the sake of durability.
In these cases new generated replicas will be in a state called temporary.
It is pretty much the same state as RBW except
the fact that this data is not visible to user unless finalized.
In case of failure,
the whole chunk of data is removed without any intermediate recovery state.
In addition to the replica transition table,
a name node block has its own collection of states and transitions.
Different from data node replica states,
a block state is stored in memory, it doesn't persist on any disk.
As soon as a user opens a file for writing,
name node creates the corresponding block with the under_construction state.
When a user opens a file for append name node
also transition this block to the state under_construction.
It is always the last block of a file,
it's length and generation stamp are mutable.
Name node block keeps track of right pipeline.
It means that it contains information about all RBW and RWR replicas.
It is quite vindictive and watches every step.
Replicas transitions from RWR to recovery RUR state when the client dies.
Even more generally it happens when a client's lease expires.
Consequently, the corresponding block
transitions from under_construction to under_recovery state.
The under_construction block transitions to a committed state when a client
successfully requests name node to close
a file or to create a new consecutive block of data.
The committed state means that there are already
some finalized replicas but not all of them.
For this reason in order to serve a read request,
the committed block needs to keep track of RBW replicas,
until all the replicas are transitioned to the finalized state
and HDFS client will be able to close the file.
It has to retry it's requests.
Final complete state of a block is a state where all the replicas are in
the finalized state and therefore they have
identical visible length and generation stamps.
Only when all the blocks of a file are complete the file can be closed.
In case of name node restart,
it has to restore the open file state.
All the blocks of the un-closed file are loaded as
complete except the last block which is loaded as under_construction.
Then recovery procedures will start to work.
There are several types of them, replica recovery,
block recovery, lease recovery, and pipeline recovery.
