Hi.

Now I'm going to talk about Hadoop, which is a big data analysis engine.

And to understand this, first we're gonna go over a couple of technical terms

to understand the foundation of where and why Hadoop is an advance.

First, data storage, access, and analysis.

Hard drive storage capacity has tremendously increased, but

the data read and write speeds to and

from the hard disk drives have not significantly improved yet.

Simultaneous parallel read and

write of data with multiple hard disks requires advanced technology.

Here are some of the challenges.

Challenge one, hardware failure.

When using many computers for data storage and analysis,

the probability that one computer will fail Is actually very high.

Challenge number two is cost.

To avoid data loss or computed analysis information loss,

using backup computers and memory is definitely needed.

And this helps the reliability and

the performance but it is actually very expensive.

The next challenge, the third challenge, is combining analyzed data.

Combining the analyzed data is actually very difficult.

If one part of the analyzed data is not ready,

then the overall combining process has to be delayed.

If one part has errors in it, Well, then the overall combined result may

not be that reliable, or it actually may be totally useless.

Hadoop.

Hadoop is a reliable shared storage and analysis system.

Now Hadoop is actually a combination of two major components.

One is HDFS and the other is MapReduce and I wrote alpha

because actually there's a little bit more to make these all work together.

Now HDFS provides data storage.

HDFS stands for Hadoop Distributed FileSystem.

Then there's the MapReduce part.

MapReduce provides data analysis, and MapReduce has two key functions,

the map function, and the reduce function, just like its name.

HDFS first Hadoop Distributed FileSystem.

Now the DFS, the Distributed FileSystem, is designed for

storage management of a network of computers.

HDFS is optimized to store huge files with streaming data access patterns.

It is also designed to run on clusters of general computers.

HDFS was designed to be optimal in performance for a WORM, a write once and

read many times pattern, which is a very efficient data processing pattern.

HDFS was designed considering the time to read the whole data set

to be more important than the time required to read just the first record.

HDFS clusters use two types of nodes.

One is a namenode which is the master node.

The other one is the datanode, which is the worker node.

First, let's look at the namenode.

The namenode manages the filesystem namespace.

It maintains the filesystem tree and

the metadata for all the files and directories in that tree.

It stores on the local disk using two file formats.

One is the namespace image, and the other is the edit log.

Then, datanodes.

Now, these are the workhorse of the filesystem.

They store and retrieve blocks when requested by the client or

the namenode Now, they report back to the namenode periodically

with lists of blocks that were stored in their system.

MapReduce is a program that abstracts the analysis problem from the stored data.

MapReduce transforms the analysis problem into a computational process

that uses a set of keys and values.

The MapReduce system architecture was designed for tasks that will consume

several minutes or hours on a set of dedicated, trusted computers connected

with a broadband high-speed network managed by a single master data center.

MapReduce uses somewhat of a brute-force data analysis approach.

The entire dataset, or

a big part of the dataset at least, is processed for every query.

And this is called Batch Query Process model.

A little bit more on the MapReduce characteristics are,

MapReduce enables the ability to run an ad hoc query against

the whole dataset within a scalable amount of time.

Many distributed systems combine data from multiple resources,

which is very difficult.

But MapReduce does this in a very effective and efficient way.

Now, to talk more about the characteristics and

the features of MapReduce, here are some technical terms.

First, seek time is the delay, the time consumed in finding a file.

Transfer rate is the speed to move a file from one location to another.

Now, if you look at their characteristics, the transfer rate has improved

significantly with new networking techniques and has much faster speeds.

And compared to the improvements in seek time,

still the seek time is relatively slow.

So the transfer rates are showing much better advanced characteristics,

faster characteristics.

So naturally to optimize the system

we will want to use more transfer characteristics than

seeking characteristics in the overall design to maximize our overall speed.

Can we build a big data engine focused on that characteristic?

Now, MapReduce, that is bad.

It gains performance enhancement through optimal balancing of the seeking and

transfer operations.

So reduce seek operations because they consume longer and effectively use

transfer operations to make your MapReduce processing for your big data engine.

In the next lecture we will compare MapReduce with a traditional RDBMS,

the Rational Database Management System.

These are the references that I used for this part of the lecture, and

if you're interested I recommend them to you for further studies.

We'll continue on with more on the next lecture.

Thank you.

HADOOP

大数据、云计算和 CDN 新兴技术

与讲师见面