Step 2-B: Pre-Processing Data

Loading...

来自 University of California San Diego 的课程

Introduction to Big Data

3895 个评分

University of California San Diego

Introduction to Big Data

3895 个评分

课程 1（共 6 门，Specialization Big Data）

Interested in increasing your knowledge of the Big Data landscape? This course is for those new to data science and interested in understanding why the Big Data Era has come to be. It is for those who want to become conversant with the terminology and the core concepts behind big data problems, applications, and systems. It is for those who want to start thinking about how Big Data might be useful in their business or career. It provides an introduction to one of the most common frameworks, Hadoop, that has made big data analysis easier and more accessible -- increasing the potential for data to transform our world! At the end of this course, you will be able to: * Describe the Big Data landscape including examples of real world big data problems including the three key sources of Big Data: people, organizations, and sensors. * Explain the V’s of Big Data (volume, velocity, variety, veracity, valence, and value) and why each impacts data collection, monitoring, storage, analysis and reporting. * Get value out of Big Data by using a 5-step process to structure your analysis. * Identify what are and what are not big data problems and be able to recast big data problems as data science questions. * Provide an explanation of the architectural components and programming models used for scalable big data analysis. * Summarize the features and value of core Hadoop stack components including the YARN resource and job management system, the HDFS file system and the MapReduce programming model. * Install and run a program using Hadoop! This course is for those new to data science. No prior programming experience is needed, although the ability to install applications and utilize a virtual machine is necessary to complete the hands-on assignments. Hardware Requirements: (A) Quad Core Processor (VT-x or AMD-V support recommended), 64-bit; (B) 8 GB RAM; (C) 20 GB disk free. How to find your hardware information: (Windows): Open System by clicking the Start button, right-clicking Computer, and then clicking Properties; (Mac): Open Overview by clicking on the Apple menu and clicking “About This Mac.” Most computers with 8 GB RAM purchased in the last 3 years will meet the minimum requirements.You will need a high speed internet connection because you will be downloading files up to 4 Gb in size. Software Requirements: This course relies on several open-source software tools, including Apache Hadoop. All required software can be downloaded and installed free of charge. Software requirements include: Windows 7+, Mac OS X 10.10+, Ubuntu 14.04+ or CentOS 6+ VirtualBox 5+.

从本节课中

Data Science: Getting Value out of Big Data

We love science and we love computing, don't get us wrong. But the reality is we care about Big Data because it can bring value to our companies, our lives, and the world. In this module we'll introduce a 5 step process for approaching data science problems.

Steps in the Data Science Process3:42

Step 1: Acquiring Data6:21

Step 2-A: Exploring Data4:19

Step 2-B: Pre-Processing Data8:27

Step 3: Analyzing Data8:18

Step 4: Communicating Results4:40

Step 5: Turning Insights into Action2:56

与讲师见面

Ilkay Altintas
Chief Data Science Officer
San Diego Supercomputer Center
Amarnath Gupta
Director, Advanced Query Processing Lab
San Diego Supercomputer Center (SDSC)

Step 2-B: Pre-processing Data

The raw data that you get directly from your sources are never in the format that

you need to perform analysis on.

There are two main goals in the data pre-processing step.

The first is to clean the data to address data quality issues, and

the second is to transform the raw data to make it suitable for analysis.

A very important part of data preparation

is to address quality of issues in your data.

Real-world data is messy.

There are many examples of quality issues with data from real applications including

inconsistent data like a customer with two different addresses, duplicate customer

records, for example, customers address recorded at two different sales locations.

And the two recordings don't agree.

Missing customer agent demographics or studies.

Missing values like missing a customer age in the demographic studies.

invalid data like an invalid zip code for example, a six digit code.

And outliers like a sense of failure causing values to be much higher or

lower than expected for a period of time.

Since we get the data downstream

we usually have little control over how the data is collected.

Preventing data quality problems as the data is being collected is not

often an option.

So we have the data that we get and

we have to address quality issues by detecting and correcting them.

Here are some approaches we can take to address this quality issues.

We can remove data records with missing values.

We can merge duplicate records.

This will require a way to determine how to resolve conflicting values.

Perhaps it makes sense to retain the newer value whenever there's a conflict.

For invalid values, the best estimate for

a reasonable value can be used as a replacement.

For example, for a missing age value for an employee,

a reasonable value can be estimated based on the employee's length of employment.

Outliers can also be removed if they are not important to the task.

In order to address data quality issues effectively,

knowledge about the application, such as how the data was collected,

the user population, and the intended uses of the application is important.

This domain knowledge is essential to making informed

decisions on how to handle incomplete or incorrect data.

The second part of preparing data

is to manipulate the clean data into the format needed for analysis.

The step is known by many names.

Data manipulation, data preprocessing, data wrangling,

and even data munging, some operations for

this type of operation I mean data munging, wrangling, preprocessing,

include, scaling, transformation, feature selection,

dimensionality reduction, and data manipulation.

Scaling involves changing the range of values to be between a specified range.

Such as from zero to one.

This is done to avoid having certain features that large values from

dominating the results.

For example, in analyzing data with height and weight.

To magnitude of weight values is much greater than of the height values.

So scaling all values to be between zero and

one will equalize contributions from both height and weight features.

Various transformations can be performed on the data to reduce noise and

variability.

One such transformation is aggregation.

Aggregate data generally results in data with less variability,

which may help with your analysis.

For example, daily sales figures may have many serious changes.

Aggregating values to weekly or monthly sales figures will result in similar data.

Other filtering techniques can also be used to remove variability in the data.

Of course, this comes at the cost of less detailed data.

So these factors must be weighed for the specific application.

Future selection can involve removing redundant or irrelevant features,

combining features, and creating new features.

During the exploring data step,

you might have discovered that two features are correlated.

In that case one of these features can be removed

without negatively affecting the analysis results.

For example, the purchase price of a product and

the amount of sales tax paid, are likely to be correlated.

Eliminating the sales tax amount, then will be beneficial.

Removing redundant or

irrelevant features will make the subsequent analysis much simpler.

In other cases, you may want to combine features or create new ones.

For example, adding the applicant's education level

as a feature to a loan approval application would make sense.

There are also algorithms to automatically determine the most relevant features,

based on various mathematical properties.

Dimensionality reduction is useful when the data set has a large number of

dimensions.

It involves finding a smaller subset of dimensions that

captures most of the variation in the data.

This reduces the dimensions of the data

while eliminating irrelevant features and makes analysis simpler.

A technique commonly used for

dimensional reduction is called principle component analysis or PCA.

Raw data often has to be manipulated to be in the correct format for analysis.

For example, from samples recording daily changes in stock prices,

we may want the capture price changes for

a particular market segments like real estate or health care.

This would require determining which stocks belong to which market segment.

Grouping them together, and perhaps computing the mean, range,

standard deviation for each group.

In summary,

data preparation is a very important part of the data science process.

In fact, this is where you will spend most of your time on any data science effort.

It can be a tedious process, but it is a crucial step.

Always remember, garbage in, garbage out.

If you don't spend the time and

effort to create good data for the analysis, you will not get good results

no matter how sophisticated the analysis technique you're using is.

探索我们的目录

免费加入并获得个性化推荐、更新和优惠。

Coursera 致力于普及全世界最好的教育，它与全球一流大学和机构合作提供在线课程。

© 2018 Coursera Inc. 保留所有权利。

通过 App Store 下载

通过 Google Play 获取