In this video on visualizing numerical data, we will discuss scatter plots for

paired data and other visualizations for

describing distributions of numerical variables.

The data come from gapminder which pulls this information from a variety of data

sources.

We will be working with two numerical variables.

Income per person, that's in US dollars and life expectancy, in years, for

the year 2012.

Each observation in this data set in a country.

That data set contains data from most but

not all countries, since this information wasn't available for certain countries.

A common tool for

visualizing the relationship between two numerical variables is a scatter plot.

To identify the explanatory variable in a pair of variables, we identify which of

the two is suspected in affecting the other and plan an appropriate analysis.

Since we might suspect that economic wealth of a country might effect

the average life expectancy of it's people, we have set up our analysis with

income as the explanatory and life expectancy as their response variable.

Generally, in a scatter plot, we place the explanatory variable on the x axis and

the response variable on the y axis.

It's very important to note that labeling variables as explanatory and response

does not guarantee that the relationship between the two is actually causal.

Even if an association between the two variables is identified.

We use these labels only to keep track of which variable we suspect

affects the other.

In fact, since these data are observational and

do not come from a randomized controlled experiment, we know that we can only talk

about correlation and not causation between the two variables.

So what is the relationship between these two variables?

The best way to answer this question is to visualize a line or

a curve going through a cloud of the data.

So here I'm drawing a curve that first shows

a positive increase in life expectancy as income increases and

then the relationship levels up such that countries with income levels above

a certain point still have roughly 80 to 85 years of average life expectancy.

The relationship is very strong with not too much scatter around the curve.

In addition,

there are a few countries that stand out from the rest as potential outliers.

We will discuss those in a detail in a moment.

Let's summarize quickly what we've learned about relationships between numerical

variables.

When evaluating the relationship between two numerical variables,

we should make sure to examine the direction of the relationship.

Is it increasing, or decreasing.

The shape of the relationship.

Is it linear, or does it follow some other form?

The strength of the relationship.

Is the relationship strong?

Indicated by little scatter.

Or weak, indicated by lots of scatter.

And any potential outliers.

These can be individual observations or a group of observations.

It's always a good idea to investigate these points carefully

to make sure they're not data entry errors.

Let's take a closer look at the outliers.

Some of them have pretty high income levels.

Luxembourg, a rich country with a small population and

has higher income per person level.

Macao, a special administrative region in China And

Qatar, a country with a small population and lots of oil.

Another potential outlier is Nepal, where the life expectancy is considerable

higher than what would be expected for the low income level compared to others.

These are countries that we would indeed expect to behave differently than

the majority of the countries.

So it's not surprising that they stand out from the rest.

One naive way of dealing with outliers in data analysis is to immediately

exclude them.

But we're calling that approach naive because it's often not the right approach.

This is a good example of when the outliers might be very interesting

in cases.

And handling them with careful consideration of the research question and

other associated variables is important.

Now, let's take a look at the distributions of the variables,

individually.

One good way of visualizing the distribution of a numerical variable

is a histogram.

In a histogram, data are binned into intervals and

height of the bars represent the number of cases that fall into each interval.

In other words a histogram provides a view of the data density,

higher bars represent where data are relatively more common.

For example we can see that majority of the countries have average life

expectancies between 65 to 85 years old.

histograms are also very useful for identifying shapes of distributions.

In this case the distribution of life expectancies

appear to be left skewed which is expected

due to the leveling off of life expectancies we've identified earlier.

There's a physiological limit to how long people live.

And in most countries, people live up to that time but

there are some countries with much lower life expectancies and fewer and

fewer of these countries with lower and lower expectancies.

Resulting in a long left tail.

The distribution of income on the other hand is right skewed.

Incomes can't be negative so we have a natural boundary at zero, but

there is no real upper limit to how high incomes can go.

However, as we go higher and higher we have fewer and fewer countries

with such high levels of personal income resulting in a long right tail.

A shared characteristic between these two distributions

is that they're both unimodel.

Let's focus on these statements on skewness and modality for a bit.

First off, skewness.

Distributions are set to be skewed to the left side of the long tail.

In a left skewed distribution, the longer tail is on the left on the negative end.

If no skewness is apparent, then the distribution is said to be symmetric.

And in a right skewed distribution,

the longer tail is on the right, the positive end.

As you can see, the best way to assess the shape of distributions is to step back and

imagine a smooth curve outlining the distribution,

instead of focusing on the jagged edges of the bars in the histogram.

Another important aspect of shape is modality.

A distribution might be unimodal with one prominent peak,

bimodal with two prominent peaks, or uniform with no prominent peaks.

With more than two prominent peaks a distribution is usually

said to be multimodal.

The distribution that you will most closely work with, and

in an introductory statistics course is unimodal, the normal distribution,

that you may also know as the bell curve.

A bimodal distribution might indicate that there are two distinct groups

in your data.

For example here's a distribution of heights of individuals at a preschool.

The first peak might be the kids and the second might be the teachers.

A uniformed distribution means there's no apparent trend in the data.

That high and low values of the variable are equally likely to occur.

Here's a distribution of the last digits of a random sample of people's social

security numbers.

As expected, the data show no trend as just as likely to have a social security

number that ends with a zero, as a six or a nine.

Assessing modality like shape is also

best done by imagining a smooth curve outlining the distribution.

Here is a trick, think of the bars as the histogram as wooden blocks and

imagine dropping a limp spaghetti over them and try to imagine how the limp

spaghetti would fall over and between the wooden blocks.

Peaks that are further from each other will likely result

in differentiable prominent peaks and

peaks that are close to each other like the ones around zero and two may not.

Identifying the number of modes is not an exact science, and

not one that you should dwell on too much.

Usually all you need to do is to determine whether the distribution is uniform

Unimodal or something else.

We should also note that the chosen bin width of the histogram can

alter the story the histogram is telling.

When the bin width is too wide, we might lose interesting details.

When the bin width is too narrow It might be difficult to get an overall picture of

the distribution.

The ideal bin width depends on the data you're working with.

So you should try playing with it until you're satisfied with the visualization.

Let's go back to the life expectancy data we were working with.

Another technique for visualizing such data is a dotplot.

A dotplot is especially useful when individual values are of interest.

However, as the sample size increases, the dotplot may get too busy.

Yet another visualization technique that is especially useful for

highlighting outliers is a box plot.

A box plot also readily displace the median.

The mid point of the distribution, this is the thick line inside the box, and

the interquartile range, the width of the box.

According to this box plot, the median life expectancy is roughly 73 years, and

the middle 50% of countries have average life expectancies between 65 and

77 years old.

In addition, countries with life expectancies that are below 48 years old

are considered to have unusually low life expectancies.

A box plot of the income distribution

shows the same right skewed distribution we've identified before.

And the outlying countries with unusually high per person income levels stand out

in this visualization as well.

One way of determining the skewness of a distribution from a box plot

is to imagine what the histogram would look like.

The peak of the distribution will be roughly around the median, and

the tails will extend out to the tails in the box plot.

There's one more visualization method that we will discuss in this video.

An intensity map.

For certain types of data, like the one's we've been working with in this video,

it might be useful to view the spatial distribution.

These displays reveal trends in the data, that many of the others did not.

For example, we can see that both income and

life expectancy are lower in Africa, but higher in North America and Europe.

Visualizing Numerical Data

Introduction to Probability and Data

与讲师见面