Welcome to the Cloud Computing Applications course, the second part of a two-course series designed to give you a comprehensive view on the world of Cloud Computing and Big Data! In this second course we continue Cloud Computing Applications by exploring how the Cloud opens up data analytics of huge volumes of data that are static or streamed at high velocity and represent an enormous variety of information. Cloud applications and data analytics represent a disruptive change in the ways that society is informed by, and uses information. We start the first week by introducing some major systems for data analysis including Spark and the major frameworks and distributions of analytics applications including Hortonworks, Cloudera, and MapR. By the middle of week one we introduce the HDFS distributed and robust file system that is used in many applications like Hadoop and finish week one by exploring the powerful MapReduce programming model and how distributed operating systems like YARN and Mesos support a flexible and scalable environment for Big Data analytics. In week two, our course introduces large scale data storage and the difficulties and problems of consensus in enormous stores that use quantities of processors, memories and disks. We discuss eventual consistency, ACID, and BASE and the consensus algorithms used in data centers including Paxos and Zookeeper. Our course presents Distributed Key-Value Stores and in memory databases like Redis used in data centers for performance. Next we present NOSQL Databases. We visit HBase, the scalable, low latency database that supports database operations in applications that use Hadoop. Then again we show how Spark SQL can program SQL queries on huge data. We finish up week two with a presentation on Distributed Publish/Subscribe systems using Kafka, a distributed log messaging system that is finding wide use in connecting Big Data and streaming applications together to form complex systems. Week three moves to fast data real-time streaming and introduces Storm technology that is used widely in industries such as Yahoo. We continue with Spark Streaming, Lambda and Kappa architectures, and a presentation of the Streaming Ecosystem. Week four focuses on Graph Processing, Machine Learning, and Deep Learning. We introduce the ideas of graph processing and present Pregel, Giraph, and Spark GraphX. Then we move to machine learning with examples from Mahout and Spark. Kmeans, Naive Bayes, and fpm are given as examples. Spark ML and Mllib continue the theme of programmability and application construction. The last topic we cover in week four introduces Deep Learning technologies including Theano, Tensor Flow, CNTK, MXnet, and Caffe on Spark.
2.3.3 Redis Demonstration
Loading...
来自 University of Illinois at Urbana-Champaign 的课程
Cloud Computing Applications, Part 2: Big Data and Applications in the Cloud
198 个评分
从本节课中
Module 2: Large Scale Data Storage
In this module, you will learn about large scale data storage technologies and frameworks. We start by exploring the challenges of storing large data in distributed systems. We then discuss in-memory key/value storage systems, NoSQL distributed databases, and distributed publish/subscribe queues.
与讲师见面
Reza FarivarData Engineering Manager at Capital One, Adjunct Research Assistant Professor of Computer Science
Department of Computer Science
Roy H. CampbellProfessor of Computer Science
Department of Computer Science
[SOUND] Hi, everyone.
Let me show you how we can interact with Redis.
Now, in my video I talked about how great Redis is.
It's a key value store, it's extremely fast.
You kind of need to know what sort of a data model you are interacting with.
But the whole idea is that if you know what keys you have set and
what sort of a data structure you're returning from your keys, things
should be very fast and efficient with Redis.
Here I'm actually running the Redis server here in local mode, but its fine.
And now it's listening on port 6379.
Now any other client that I write,
that connects to this port on this case, my local machines IP address or
whatever IP address that the server is running on, would be able to run commands.
Let me show you how this command kind of works.
In this case I'm going to show you the command line interface.
There are of course programmatic interfaces that if you are writing your
program in Python, in Java, in Scala, pretty much
any language that you can think of, there are libraries that you can load and
then you can kind of use very similar commands to work with Redis.
But let me bring up the Redis command line interface.
For you old-timers, it kind of works,
at least the command line interface, kind of looks like Telnet used to be.
So you'd give it commands and it basically runs the commands.
If you remember from my slides, Redis basically works with p-value pairs, right.
The keys are unique identifiers.
Now the values can be different data structures, they can be strings or
numbers.
And numbers are really the stored strengths and you'll see in a minute.
Your values could be lists, could be sets or could be hash tables.
What is a list?
List is a number of, an ordered set of strengths.
Set is just a bunch of values, their strengths, in Redis.
You can have sorted sets, and we'll look at that in a minute.
Or you can have hash tables, so you use a key to access a hash table.
And once you get that, you can actually give another key and
get a hash table, and get your value out.
Let's start actually by giving it some commands and see how it works.
So the commands in Redis are kind of a very,
are named very accordingly.
So you have set and get and delete right?
So to be able to set a key value pair,
and let me to zoom in a little bit.
I can just set something, so I can say set name to Bob.
And I will just say okay.
By the way, again I want to emphasize that like a single Redis instance
can handle hundreds of thousands of these commands a second.
So you don't definitely need to always interact with it
in a command line interface.
These can be commands that your program sends to it.
Now if I actually just want to access my key value,
I can say what was the name, get name, and it says Bob, right?
So I can now change it.
I can say hey change The name to John.
And now if I get name, it becomes John.
Another command useful is delete.
So I can say delete name, and then if I say get name, now the name,
the key value pair, whose key was name, isn't there anymore.
Same thing with numbers.
You can actually set number to 123,
and I can, of course I can get the number.
See now the number is stored as a string, but once you have numbers,
numbers have some interesting command set you can interact with.
And they're increment and decrement.
So you don't necessarily need to get a number and add it in your program and
send back.
You can just say increments number.
So I can say increment this number, now if I get the number it's 124 now.
Same with decrement number if I get it, it's back to 123.
Of course, I can get the length office rank, that's easy.
So let's say set
myKey as Quick Brown Fox.
Get myKey, Quick Brown Fox.
Now I can say string length of my key.
And it goes and counts and says okay, this is 15 characters.
One interesting thing is I can check to see if a certain key exists or not.
So I can say, exists and let me, let's give it like a bad key.
So, let's type yourKey, it says 0,
that means no, it doesn't exist.
But if I say, exists myKey, it will return 1.
That means that yes, that key value pair exists in the database.
Now, one interesting thing is the server,
again, the server is running separate from this command line.
Different instances of a programs that are interacting with
the server can come and connect to it.
So, if I actually exit my command line, and
if go back here, nothing should be reset.
All my key value pairs should be running because hey,
my server is still running on this other tab, right?
So, If I check mykey exists I can, it says yes.
If I say get mykey the Quick Brown Fox is still there.
That's the whole point of a key value store.
It's typically use in land architectures, for
example in streaming systems for storing the data set.
After working with individual key value pairs,
now let's look at some more advanced usage.
Let's talk about sets.
So I can have a set and you do that with set add.
So let's say I have a set of tickers, for example stock tickers.
And let's say we have A whole bunch of stock tickers here.
So now I'm adding Google, GOOG, to my set.
If I say smembers
It says at the moment I have one set, one, one item in my set.
Now, what if I now add say, IBM, now I have two items in my set.
And you can have whole bunch of other things as well.
I can have NASDAQ, and I can have
DOW, and if I get the set of members now it returns all of them.
I can also remove something from the set, so
I can say remove from the set of tickers IBM.
Now IBM is a member of that set anymore.
So as you can see, Redis can have keys and
key values for values that can be strings that we've seen so far.
It can be a set, and now it could be assorted set.
So, for assorted set, instead of saying set add, I say zadd.
So, let's call it another set, let's say set zadd sports,
and let's give it a rank.
So let's say Football and
let's say zadd sports 2 Hockey,
and then I can interact with my sorted set now and
I can say give me a range from the set sports,
say from one to two, with scores.
So now we can actually give it a range.
And it can then go and give you back the result, sort it, 1, 2, football, hockey.
Even more interesting, use case is using get hash set as the value here.
As you can see, having key values,
actually the whole idea depends on hashing mechanism.
So, I can extend this very powerful idea to values itself,
so a value itself can be a hash set.
So now, instead of getting the string back, I'll get a whole high set back.
Let's take a look at, let's see, if I have a hash get.
And let's give it the users.
So it doesn't have at the moment, users, so
let's actually start giving it some values.
So let me clear.
Now I can say hash set, let's say I'm trying to create
a key value database inside my Redis and call that database users.
So what I'm doing here is I'm saying,
hey let's actually create a hash set called a high set users,
inside it have a key value where the keys name the value is Bob.
And now I can actually now do another thing inside this hash set.
So I can say, now inside this hash set, I'm going to put another key value pair.
Let's say age, and let's say Bob is,
let's say 30 years old, right.
Now, I can actually ask for it to give me the set of keys in that hash set.
So I can say, hey, go find these hash set users,
and tell me what keys I have in there.
Sorry, hkeys.
There we go.
So now it says in the hash set that you define as users,
you'll have two different keys, key name and key age.
Now I can interact with that.
So I can say okay, hget for that user from that users hash set.
Now I need to give it a key, inside the hashset,
so let's say h and it says age is 30.
If I ask it to give me the value for
key name, it goes and finds the value for that and returns me Bob.
Of course I can go and delete individual items,
so I can go and say h delete inside users,
the key value whose key is h,
and it will say fine.
Now if I ask it to show me what keys I have in hash set, it just has name.
So we can think of it as for regular key value pairs,
I use get or set.
For set I can use, for
high sets I was going to use hget or hset.
And similarly for sets,
I used s of course I don't use s set and I use s at and s members.
I think this is good enough.
There are some other interesting things in Redis.
There's lists that you can push things on a list and
get arranged again.
There's the whole idea of kind of published subscribe,
which kind of gets you to some sort of a similar functionality as
another system that we introduced in a different video Kafka does.
So here you can say hey,
I want to subscribe to a certain key, and then if another user comes in and
creates a key value pair with that key, the person will subscribe.
They'll receive that.
And then so another interesting thing that I'm not again going to show,
is the idea of expire.
So you can actually set a key value pair inside Redis, and
then give it a expiration time in seconds.
And then after a certain value, that time is gone.
The key value pair are removed.
So, but I think you get the gist of it, and it's very easy to interact with Redis.
Similar libraries exist for a lot of programming languages.
[MUSIC]
