This course will introduce the learner to the basics of the python programming environment, including fundamental python programming techniques such as lambdas, reading and manipulating csv files, and the numpy library. The course will introduce data manipulation and cleaning techniques using the popular python pandas data science library and introduce the abstraction of the Series and DataFrame as the central data structures for data analysis, along with tutorials on how to use functions such as groupby, merge, and pivot tables effectively. By the end of this course, students will be able to take tabular data, clean it, manipulate it, and run basic inferential statistical analyses. This course should be taken before any of the other Applied Data Science with Python courses: Applied Plotting, Charting & Data Representation in Python, Applied Machine Learning in Python, Applied Text Mining in Python, Applied Social Network Analysis in Python.
Scales
Loading...
来自 University of Michigan 的课程
Introduction to Data Science in Python
7009 个评分
从本节课中
Week 3
In this week you'll deepen your understanding of the python pandas library by learning how to merge DataFrames, generate summary tables, group data into logical pieces, and manipulate dates. We'll also refresh your understanding of scales of data, and discuss issues with creating metrics for analysis. The week ends with a more significant programming assignment.
与讲师见面
Christopher Brooks
Now that we've covered many of the mechanics of Pandas, I want to stop and
talk for a moment about data types and scales.
We've already seen that Pandas supports a number of different computational data
types such as strings, integers and floating point numbers.
What this doesn't capture is what we call the scale of the data.
Let's say that we have a data frame of students and
their academic levels such as being in grade 1 or grade 2 and grade 3.
There's a difference between a student in grade 1 and a student in grade 2, the same
as the difference between a student in grade 8 and a student in grade 9.
Well let's think about the final exam scores these students might get
on assignments.
Is the difference between an A and
an A minus the same as the difference between an A minus and a B plus?
At the University of Michigan at least, the answer is usually no.
We've intuitively seen some different scales, and
as we move through data cleaning and statistical analysis and
machine learning, it's important to clarify our knowledge and terminology.
As a data scientist there four scales that it's worth knowing.
The first is a ratio scale.
In the ratio scale the measurements units are equally spaced and mathematical
operations, such as subtract, division, and multiplication are all valid.
Good examples of ratio scale measurements might be the height and weight.
The next scale is the interval scale.
In the interval scale the measurement units are equally spaced
like the ratio scale.
But there's no clear absence of value.
That is there isn't a true zero, and so operation such as multiplication and
division are not valid.
An example of the interval scale might be the temperatures measured in Celsius or
Fahrenheit.
Since there's never an absence of temperature and
0 degrees is a meaningful value of temperature.
The direction on a campus might be another good example where 0 degrees on the campus
doesn't indicate a lack of direction.
For most of the work you do at data mining, the differences between
the ratio and interval scales might not be clearly apparent or
important than the algorithm you're applying.
But it's important to have this clear in your mind when you're applying advanced
statistical tests.
The next scale is the ordinal scale, and this is also important.
In the ordinal scale the order of values is important but
the differences between the values are not equally spaced.
Here is a grading method that is used in many classes at the University of Michigan
as a great example, where letter grades are given with pluses and minuses.
But when you compare this to percentage values you see that a letter by itself
covers 4% of the available grades and that a letter with a plus or
minus is usually just 4% for the available grades.
Based on this, it would be odd if there were as many students who received
an A plus or A minus as there were who received a straight A.
Ordinal data is very common in machine learning and
can sometimes be a challenge to work with.
The last scale I'll mention is the nominal scale which is
often just called categorical data.
Here the names of teams in a sport might be good example.
There are a limited number of teams but changing their order or
playing mathematical function to them is meaningless.
Categorical values are very common and we generally refer to categories
where there are only two possible values as binary.
So why did I stop talking about Pandas and jump into this discussion of scale.
Well, given how important they are in statistics and
machine learning, Pandas has a number of interesting functions to
deal with converting between measurement scales.
Let's start first with nominal data, which in Pandas is called categorical data.
Panda is actually has a built in type for categorical data and you could
set a column of your data to categorical data by using the as type method.
As type tries to change the underlying type of your data,
in this case to category data.
You can further change this to ordinal data by passing in an ordered flags set to
true and passing in the categories in an ordered fashion.
Here's an example.
Let's create a data frame of letter grades in the descending order.
We can also set an index value of some more course grain measure.
Now when we instruct Pandas to render this as categorical data, we see that the D
type has been set as category and that there are 11 different categories.
If we want to indicate to Pandas that this data is in a logical order,
we pass the ordered equals true flag and
we see those reflected in the category D type using the less than sign.
What can you do with this?
Well, ordinal data has ordering so it can help you with the Boolean masking.
For instance, if we have our list of grades and we compared them with a C.
If we did this lexographically, we would that a C+ and
a C- are both actually greater than a C.
Instead of coding each of these to something which is lexographical,
like a number, we can indicate that there's a clear order to the data.
And then broadcasting will work as we expect.
We can then use a certain set of mathematical operators like minimum,
maximum and others on the ordinal data.
Sometimes it's useful to represent categorical values as each being a column
with a true or a false as to whether that category applies.
This is especially common in feature extraction,
which is a topic in the third course in this specialization.
Variables with a Boolean value are typically called dummy variables.
And pandas has a built-in function called get dummies,
which will convert the values of a single column into multiple columns of 0's and
1's, indicating the presence of a dummy variable.
There's one more function on scales that I'd like to talk about.
And that's on reducing a value which is on the interval or
ratio scale, like a number grade, into one that is categorical like a letter grade.
Now, this might seem a bit counter intuitive to you since you're losing
information about the value.
But it's useful on a couple of places.
First, if you're visualizing the frequencies of categories, and
this can be an extremely useful approach and
histograms are regularly used with converted interval or racial data.
And we'll take a look at histograms in the second course in this specialization.
Second, if you're using a machine learning classification approach on data,
then you need to be using categorical data.
So reducing dimensionality is useful there too.
panda says a function called cut, which takes in argument which is some real
like structure of a column or a data frame or a series.
It also takes a number of bins to be used and all bins are kept at equal spacing.
Let's go back to our census data for an example.
We saw that we could group by state,
then aggregate to get a list of the average county size by state.
If we further apply cut to this with say 10 bins, we can see that
the states listed as categoricals using the average county size.
Here we see the states like Alabama and Alaska fall into the same category, while
California and the District of Colombia fall into a very different category.
Now, cutting is just one way to build categories from your data, and
there's many other methods.
For instance, cut gives you interval data,
where the spacing between each category is equal sized.
But sometimes you want to form categories based on frequency.
You want the number of items in each bin to be the same,
instead of the spacing between bins.
It really depends on the shape of your data and
what you're planning to do with it.
And we'll look at some examples of this in the graphing and charting course for
those of you who are enrolled in the specialization.
That's it for this module of the course.
There's a programming assignment to test your knowledge of this module's content
and I'll see you back in the next module as a wrap up where we'll talk more
about statistics and set up some basics for comparing results of analysis.
