********* A new, improved version of the Big Data Specialization will become available on June 6! As such, enrollment for this course and all courses in this original Big Data Specialization will close on June 6. The original Big Data Specialization will continue to run until September 2016, when the Capstone will be offered for learners in this version of the Specialization. If you are in the middle of the Specialization and have purchased the entire original Big Data Specialization before June 6, Coursera will reach out to you to offer you the option of staying in the original Specialization or taking the new version. If you are just getting started on this Specialization, we recommend that you wait until June 6 to enroll in the new version. ********* This course is for novice programmers or business people who'd like to understand the core tools used to wrangle and analyze big data. With no prior experience, you'll have the opportunity to walk through hands-on examples with Hadoop and Spark frameworks, two of the most common in the industry. You will be comfortable explaining the specific components and basic processes of the Hadoop architecture, software stack, and execution environment. In the assignments you will be guided in how data scientists apply the important concepts and techniques, such as Map-Reduce that are used to solve fundamental problems in big data. You'll feel empowered to have conversations about big data and the data analysis processes.
Introduction to Apache HIVE
Loading...
来自 University of California San Diego 的课程
Hadoop 平台和应用程序框架
个评分
从本节课中
Introduction to the Hadoop Stack
In this module we will take a detailed look at the Hadoop stack ranging from the basic HDFS components, to application execution frameworks, and languages, services.
与讲师见面
Natasha BalacDirector, Predictive Analytics Center of Excellence (PACE)
San Diego Supercomputer Center
Paul RodriguezResearch Programmer
San Diego Supercomputer Center (SDSC)
Andrea ZoncaHPC Applications Specialist
San Diego Supercomputer Center (SDSC)
Welcome back. So in the first video,
I mean in the second video we saw Apache example.
Now let's take a look at Apache Hive which essentially lets you query and
manage data using something that looks like a SQL interface.
So install HiveQL.
Now, what we are gonna do here is run interactively,
using a program called Beeline.
But there are other run mechanisms that I will list at the end of the video.
So what's Apache Hive?
It's essentially a data warehousing software.
So it lets you structure and query data using an SQL like language.
So you could have the data in HDFS or HBase, but
basically this sits on top of it and lets you do the querying.
The execution environment can be like MapReduce and Yon, or Tez or Spark.
And there are lots of options on the essentially.
And just like we saw in Pig, you can essentially use or
write, Map and Reduce functions, if you need to.
What are the kinds of applications that people use it for?
Hive is useful in data mining, analytics kind of applications, machine learning,
you can do ad hoc analysis of data that's sitting in hdfs Or H-Space.
Again, there's a huge list of Windows and companies using this and
application areas, so you can take a look at the link that I've got on there.
But these are the types of areas people typically use it.
So for our example, what we'll do is go back to our passwords file example from
Pig, and try to do this using hive essentially.
So we, again, start by loading the file into HDFS and,
as I mentioned, the command is hdfs dfs -put /etc/passwd and
least this time we're gonna stick it in the /tmp directory.
Now this /tmp is on the HDFS file system.
And what we are essentially doing is trying to extract the same
information we had from Pig, using, essentially, Hive.
And as I mentioned, we're gonna use beeline to access this data,
and access Hive, and run things interactively.
That's the command to do that.
Remember all these are commands that you're essentially typing in the terminal
window if your [INAUDIBLE] Quickstart.
This is how it looks essentially.
I have a snapshot of the Quickstart window that I used and
you can see there is dfs put command at the first line and
then I 'm using a dfs ls to make sure the password file is indeed in there.
Then I'm just running b line to get interactive access hive.
[COUGH] There's a lot of commands in Hive that are going to be used,
so I'm listing them at the start so that we have some place to copy and
paste it from if you're following along over the video.
So the command that essentially creates a table,
all the import from the password file.
So you kind of look at the password file, see what are the entries and
then you can just create appropriate string entries in this table essentially.
So you have a user name, password, URD, hdir,
a phone line, a home directory, a shell.
So, and they are basically saying that a row format parameter is so
remember in Pig you have to go specify that specifically,
so we are doing it here too.
And it's a text file so it just essentially loading another text file.
So, once the table is set, we essentially load
the data from /tmp/password again, give it time for
part and essentially loading it into the table user info.
And now what we're doing is saying, hey let's select a user name the full name and
the home directory from this table.
And all ordered by user name.
So essentially we're trying to get the same data we got previously from so
how does this look from our snapshot point of view?
So we're just running these commands now.
So first we create the table.
So you have to userinfo table.
And it says no rows affected, that's because we have no data in it.
Then we load the input.
And [COUGH] essentially we are all writing it into the thing and
we are still, you just have the raw data right now.
Now we essentially do the select and do a uname,
fullname, and home directory from user input.
And once we do that, you can see the backend of Hadoop and
my produced jobs are launched.
Essentially one job being gone, you have a number of reduced on sector one because we
just have one task in this case.
You can actually change all these number of producers if you have more, of course.
Then a bunch of Hadoop pass starts off.
And once it's finished you'll see basically there were one map job,
one radials job.
It took about two seconds and it write all the data.
Essentially you get all bytes you want.
You have the username, the full name and the home directory.
And then you're looking at all the values from password files that was just passed.
The only difference being here we're actually solving it with the username, so
you'll see all the a's in the front, and then so on.
So, same example, but using Hive.
And you can see even in this case the high level commands are pretty simple and
then backend MapReduce is handled by framework.
So essentially to summarize we used beeline for the interactive Hive example.
You can also use Hive command line interface,
although this is getting in the newer versions.
There's Hcatalog.
And there's WebHcat.
So there's different options of accessing Hive essentially.
You could order more of the command line there.
So either way you have a lot of options essentially.
You can write your own custom app or use others as I've mentioned.
So you can do quite a bit of data processing using Hive.
So I end for Hive and have a separate module upcoming which will
have a lot more detail and this was a quick overview and
a quick preview of how you can run Hive on a Hadoop cluster.
Thank you.
