We live in an uncertain and complex world, yet we continually have to make decisions in the present with uncertain future outcomes. Indeed, we should be on the look-out for "black swans" - low-probability high-impact events. To study, or not to study? To invest, or not to invest? To marry, or not to marry? While uncertainty makes decision-making difficult, it does at least make life exciting! If the entire future was known in advance, there would never be an element of surprise. Whether a good future or a bad future, it would be a known future. In this course we consider many useful tools to deal with uncertainty and help us to make informed (and hence better) decisions - essential skills for a lifetime of good decision-making. Key topics include quantifying uncertainty with probability, descriptive statistics, point and interval estimation of means and proportions, the basics of hypothesis testing, and a selection of multivariate applications of key terms and concepts seen throughout the course.
3.2 Data Visualisation
Loading...
来自 University of London 的课程
Probability and Statistics: To p or not to p?
个评分
从本节课中
Describing The World The Statistical Way
与讲师见面
Dr James AbdeyAssistant Professorial Lecturer, LSE
University of London
[MUSIC]
Now many of you have perhaps come across the phrase of big data.
Now while this is not a course in big data per se, well what we're going to consider
now really reflects some of the main challenges of the big data era.
So big data, perhaps in simplistic terms,
is a huge volume of data, hence that word big.
And our goal is to try and simplify, to make sense of it, and
draw insights from it.
So one of the key challenges of big data is what we call data reduction or
data summarization.
Reducing the complexity of our data set, or
we might say reducing the dimensionality of our data set, such that it's easier for
us to digest and draw some insights from it.
Now we'll just consider some fairly small data sets for this purpose.
But nonetheless, it underlies the main goal of trying to reduce the complexity
and see the big picture of our data set.
Now data reduction, we could really consider of doing it in one of two ways.
One is visually, which we'll consider here.
Namely data visualization, where we can produce simple plots
of particular variables and hopefully draw some initial insights from those.
And also we would like to come up with some simple descriptive or
summary statistics whereby we reduce a very complex data set numerically.
So reduce a large number of values into just perhaps one or two figures.
So data visualization, remember we mentioned in the previous session about
the different types of levels of measurement.
We had our categorical and our measurable variables.
And we said at the time that we have statistical techniques to apply to
different types of variables.
Well, similarly with data visualization or data presentation if you prefer,
depending on the types of variables you have, they'll be different
types of graphical display which are appropriate in each situation.
So, for example, if we had some nominal data, for example,
if we considered the list of nations or took a sample of countries in the world.
So different countries, the name is simply a label,
an identifying label, and hence represents a nominal variable.
So let's imagine we had a sample of about 150 or so countries.
And then we may wish to recognize which continent each country belongs to.
So how might we wish to reduce maybe 150 countries
in a more easier to digest format?
Well perhaps a simple tabular form might be appropriate.
We could have a table with the frequency counts representing how frequently
nations in different continents appeared in our sample data set.
But if, let's say, we had 20 appearing as a frequency count.
Is 20 a big value or a small value?
Well it's going to depend on what sort of a context we're dealing with.
20 out of 22, say, is perhaps a large value.
20 out of 1,000, a much smaller value.
So rather than consider simple frequency counts, we may wish to transform
these into percentages which would reflect the relative frequency
with which let's say the different continents occurred in our data set.
Now, perhaps a challenge for you is to go to any sort of mainstream media and
look for examples of tables in various news articles, and
see what kind of variables are being summarized within those tables.
But sometimes we, as human beings, can relate more to pretty pictures and
diagrams than potentially just lots of figures appearing in tabular form.
So if, for example, we had a categorical variable,
such as the continent to which a particular country belongs,
then an appropriate graphical display might be a simple bar chart.
Now, of course, in practice,
we would use computer software to generate diagrams for us.
We'd never really draw these things manually with pen and pencil and paper.
But do be conscious that when computer software produces a diagram,
take great care that the diagram which is generated is not actually
distorting what is trying to be communicated to the wider audience.
So take great care about any axes which are used and
any scales used on those axes.
And make sure that your diagram is not distorting what you're trying to convey.
Equally, if you're looking at a diagram produced by someone else,
particularly by perhaps a politician or an advertizing company,
they may be deliberately trying to distort what the data genuinely is showing
to try get across their particular perspective and point of view.
Now perhaps a very common type of diagram used when
analyzing a single measurable variable might be that of a histogram.
So, for example, we might consider the GDP per capita
across a set of countries within a sample data set.
And a histogram very clearly brings to life the data, and
you get a very strong sense of the distribution of GDP per capita
across the various countries in our data set.
Now that word distribution, that perhaps rings a bell with some work we did in week
two of the course when we constructed some simple probability distributions.
For example, for the score for a fair die, say.
Or perhaps the chances of getting heads and tails when tossing a fair coin.
But, of course, those probability distributions were derived theoretically.
For example, that fair die, hence the six equally likely outcomes.
And we attached a probability of one over six to each of those.
In contrast here, we're looking at the distribution not of a sort of
theoretical probability distribution, but rather a sample distribution.
Ie the distribution of the variable being considered within our sample data set.
So looking at this histogram of GDP per capita,
you can see that it varies a great deal across the countries in our data set.
With the vast majority of countries being quite poor on a GDP per capita basis,
with one or two countries performing very well on the GDP per capita basis.
Of course, we need to look at these things more numerically, and
just because a country has a high GDP per capita
does not necessarily mean everyone within that country is wealthy.
Per capita is simply an example of an average which we're going to be looking at
in the next section whereby we're looking at the total GDP in a country and
imagine, hypothetically, it was split equally among the population.
Just as in countries with very low levels of GDP per capita, of course,
perhaps those at the very elite of society if there was a lot of corruption, say.
Those at the top are perhaps doing very well in life,
with the vast majority of the population struggling a great deal.
So GDP per capital perhaps one fairly simplistic metric
to gain some sense of how wealthy a country is.
But, of course, is not revealing the entire picture.
But nonetheless, it's achieving a simple goal of data reduction.
In a simple histogram we can convey quite a lot of information about how wealth is
spread in different countries around the world.
But so far, things like the bar chart, the histogram is for
displaying a single variable each time.
But what if we wanted to go a bit beyond this and
start to consider perhaps the relationships between two variables?
Well remember, the builder or decorator with the tool kit, different tools for
different jobs.
Similarly, we have different types of graphical display
depending on the types of variables as well as the numbers of variables present.
So continuing with countries for a moment, let's consider GDP per capita.
So some, perhaps an imperfect metric, but
a metric nonetheless of how wealthy countries are.
And perhaps it's also considered a corruption.
Specifically here the control of corruption.
So in this case, if we consider a control of corruption variable,
low scores would indicate that there's not much control of corruption and hence,
there's a large amount of corruption in a particular country.
So control of corruption, GDP per capita, so
these would be two measurable variables.
So if we wanted to display these,
then a scatter diagram would be the appropriate form to use.
So if we consider plotting on the x axis, so the horizontal axis is control
of corruption variable and on the y axis, the GDP per capita variable.
One can instantly see whether or
not there seems to be a relationship between those two variables.
Now this scatter plot tends to demonstrate a positive relationship,
not perfect, but nor is it a perfect world.
And it's very rare in the social sciences, in particular,
to come across perfect relationships.
But there does seem to be a tendency that countries which tend to control corruption
more, and hence have lower levels of corruption,
tend to enjoy higher levels of GDP per capita.
Now, we could get into some semantics about whether or
not this is a linear or non-linear relationship.
But to keep the argument fairly simplistic, let's assume that we could
approximately fit a line through those points, albeit not a perfect fit.
So this would indicate that there is some degree of correlation
between these two variables.
Countries with low levels of corruption, ie, where corruption is very well
controlled, tend to enjoy high levels of GDP per capita.
But, of course, correlation does not necessarily imply causality.
Of course, many politicians and
advertisers would perhaps be wise to remember that fact.
Correlation does not equal causality.
Indeed, particularly with the social sciences,
it can be a great challenge to actually try and infer causality.
Let's consider corruption versus GDP per capita.
You could give some sort of more qualitative arguments for
why one would influence the other.
Perhaps if a country suffers from high levels of corruption,
maybe that is stifling the ability of the economy to grow.
And hence the economy has a low level of GDP and
hence a low level of GDP per capita.
So that argument would suggest that the level of corruption is determining
the level of economic prosperity.
Of course, you could argue it's the other way round.
If an economy is performing very well and everyone is very wealthy,
then there is no need to have corrupt officials, say.
So a correlation can be easily perhaps identified from a scatter plot,
but trying to infer causality is a much more challenging proposition.
But nonetheless, a nice extension from displaying a single variable,
perhaps through a bar chart if it's categorical or
a histogram if it's measurable, to an example where
we can show two measurable variables through a scatter plot.
So again, perhaps another takeaway challenge for you.
Have a look through the mainstream media and try and
find some articles which are showing various scatter diagrams.
And think about what sorts of relationships between the variables is
trying to be conveyed in the report itself.
[MUSIC]
