If you want to break into competitive data science, then this course is for you! Participating in predictive modelling competitions can help you gain practical experience, improve and harness your data modelling skills in various domains such as credit, insurance, marketing, natural language processing, sales’ forecasting and computer vision to name a few. At the same time you get to do it in a competitive context against thousands of participants where each one tries to build the most predictive algorithm. Pushing each other to the limit can result in better performance and smaller prediction errors. Being able to achieve high ranks consistently can help you accelerate your career in data science. In this course, you will learn to analyse and solve competitively such predictive modelling tasks. When you finish this class, you will: - Understand how to solve predictive modelling competitions efficiently and learn which of the skills obtained can be applicable to real-world tasks. - Learn how to preprocess the data and generate new features from various sources such as text and images. - Be taught advanced feature engineering techniques like generating mean-encodings, using aggregated statistical measures or finding nearest neighbors as a means to improve your predictions. - Be able to form reliable cross validation methodologies that help you benchmark your solutions and avoid overfitting or underfitting when tested with unobserved (test) data. - Gain experience of analysing and interpreting the data. You will become aware of inconsistencies, high noise levels, errors and other data-related issues such as leakages and you will learn how to overcome them. - Acquire knowledge of different algorithms and learn how to efficiently tune their hyperparameters and achieve top performance. - Master the art of combining different machine learning models and learn how to ensemble. - Get exposed to past (winning) solutions and codes and learn how to read them. Disclaimer : This is not a machine learning course in the general sense. This course will teach you how to get high-rank solutions against thousands of competitors with focus on practical usage of machine learning methods rather than the theoretical underpinnings behind them. Prerequisites: - Python: work with DataFrames in pandas, plot figures in matplotlib, import and train models from scikit-learn, XGBoost, LightGBM. - Machine Learning: basic understanding of linear models, K-NN, random forest, gradient boosting and neural networks.
Categorical and ordinal features
Loading...
来自 National Research University Higher School of Economics 的课程
How to Win a Data Science Competition: Learn from Top Kagglers
472 个评分
National Research University Higher School of Economics
472 个评分
课程 2（共 7 门，Specialization Advanced Machine Learning）
从本节课中
Feature Preprocessing and Generation with Respect to Models
In this module we will summarize approaches to work with features: preprocessing, generation and extraction. We will see, that the choice of the machine learning model impacts both preprocessing we apply to the features and our approach to generation of new ones. We will also discuss feature extraction from text with Bag Of Words and Word2vec, and feature extraction from images with Convolution Neural Networks.
与讲师见面
Dmitry UlyanovVisiting lecturer
HSE Faculty of Computer Science
Alexander GuschinVisiting lecturer at HSE, Lecturer at MIPT
HSE Faculty of Computer Science
Mikhail TrofimovVisiting lecturer
HSE Faculty of Computer Science
Dmitry AltukhovVisiting lecturer
HSE Faculty of Computer Science
Marios MichailidisResearch Data Scientist
H2O.ai
Hi. In this video,
we will cover categorical and ordinal features.
We will overview methods to work with them.
In particular, what kind of pre-processing will be used for each model type of them?
What is the difference between
categorical and and ordinal features and how we can generate new features from them?
First, let's look at several rows from
the Titanic dataset and find categorical features here.
Their names are: Sex, Cabin and Embarked.
These are usual categorical features but there is one more special, the Pclass feature.
Pclass stands for ticket class,
and has three unique values: one, two, and three.
It is ordinal or,
in other words, order categorical feature.
This basically means that it is ordered in some meaningful way.
For example, if the first class was more expensive than the second,
or the more the first should be more expensive than the third.
We should make an important note here
about differences between ordinal and numeric features.
If Pclass would have been a numeric feature,
we could say that the difference between first,
and the second class is equal to the difference between second and the third class,
but because Pclass is ordinal,
we don't know which difference is bigger.
As these numeric features,
we can't sort and integrate an ordinal feature the other way,
and expect to get similar performance.
Another example for ordinal feature is a driver's license type.
It's either A, B, C,
or D. Or another example,
level of education, kindergarten, school,
undergraduate, bachelor, master, and doctoral.
These categories are sorted in increasingly complex order,
which can prove to be useful.
The simplest way to encode
a categorical feature is to map it's unique values to different numbers.
Usually, people referred to this procedure as label encoding.
This method works fine with two ways because tree-methods can split feature,
and extract most of the useful values in categories on its own.
Non-tree-based-models, on the other side,
usually can't use this feature effectively.
And if you want to train linear model kNN on neural network,
you need to treat a categorical feature differently.
To illustrate this, let's remember example we had in the beginning of this topic.
What if Pclass of one usually leads to the target of one,
Pclass of two leads to zero,
and Pclass of three leads to one.
This dependence is not linear,
and linear model will be confused.
And indeed, here, we can put linear models predictions,
and see they all are around 0.5.
This looks kind of set but three on the other side,
we'll just make two splits select in each unique value and reaching it independently.
Thus, this entries could achieve much better score here using these feature.
Let's take now the categorical feature and again, apply label encoding.
Let this be the feature Embarked.
Although, we didn't have to encode
the previous feature Pclass before using it in the model.
Here, we definitely need to do this with embarked.
It can be achieved in several ways.
First, we can apply encoding in the alphabetical or sorted order.
Unique way to solve of this feature namely S, C, Q.
Thus, can be encoded as two,one, three.
This is called label encoder from sklearn works by default.
The second way is also labeling coding but slightly different.
Here, we encode a categorical feature by order of appearance.
For example, s will change to one because it was meant first in the data.
Second then c, and we will change c to two.
And the last is q, which will be changed to three.
This can make sense if all were sorted in some meaningful way.
This is the default behavior of pandas.factorize function.
The third method that I will tell you about is called frequency encoding.
We can encode this feature via mapping values to their frequencies.
Even 30 percent for us embarked is equal to c and 50 to s and the rest 20 is equal to q.
We can change this values accordingly: c to 0.3, s to 0.
5, and q to 0.2.
This will preserve some information about values distribution,
and can help both linear and three models.
first ones, can find this feature useful
if value frequency is correlated to it's target value.
While the second ones can help with less number of split because of the same reason.
There is another important moment about frequency encoding.
If you have multiple categories with the same frequency,
they won't be distinguishable in this new feature.
We might a apply or run categorization here in order to deal with such ties.
It is possible to do like this.
There are other ways to do label encoding,
and I definitely encourage you to be creative in constructing them.
Okay. We just discussed label encoding,
frequency encoding, and why this works fine for tree-based-methods.
But we also have seen that linear models can struggle with label encoded feature.
The way to identify categorical features to
non-tree-based-models is also quite straightforward.
We need to make new code for each unique value in the future,
and put one in the appropriate place.
Everything else will be zeroes.
This method is called, one-hot encoding.
Let's see how it works on this quick example.
So here, for each unique value of Pclass feature,
we just created a new column.
As I said, this works well for linear methods,
kNN, or neural networks.
Furthermore, one -hot encoding feature is already
scaled because minimum this feature is zero,
and maximum is one.
Note that if you care for a fewer important numeric features,
and hundreds of binary features are used by one-hot encoding,
it could become difficult for tree-methods they use first ones efficiently.
More precisely, tree-methods will slow down,
not always improving their results.
Also, it's easy to imply that if categorical feature has too many unique values,
we will add too many new columns with a few non-zero values.
To store these new array efficiently,
we must know about sparse matrices.
In a nutshell, instead of allocating space in RAM for every element of an array,
we can store only non-zero elements and thus,
save a lot of memory.
Going with sparse matrices makes sense if number of
non-zero values is far less than half of all the values.
Sparse matrices are often useful when they work with categorical features or text data.
Most of the popular libraries can work with these sparse matrices directly namely,
XGBoost, LightGBM, sklearn, and others.
After figuring out how to
pre-processed categorical features for tree based and non-tree based models,
we can take a quick look at feature generation.
One of most useful examples of feature generation
is feature interaction between several categorical features.
This is usually useful for non tree based models namely, linear model, kNN.
For example, let's hypothesize that target depends
on both Pclass feature, and sex feature.
If this is true, linear model could adjust
its predictions for every possible combination of these two features,
and get a better result.
How can we make this happen?
Let's add this interaction by simply concatenating
strings from both columns and one-hot encoding get.
Now linear model can find optimal coefficient for every interaction and improve.
Simple and effective.
More on features interactions will come in the following weeks especially,
in advanced features topic.
Now, let's summarize this features.
First, ordinal is a special case of
categorical feature but with values sorted in some meaningful order.
Second, label encoding, basically replace
this unique values of categorical features with numbers.
Third, frequency encoding in this term,
maps unique values to their frequencies.
Fourth, label encoding and frequency encoding are often used for tree-based methods.
Fifth, One-hot encoding is often used for non-tree-based-methods.
And finally, applying One-hot encoding combination one heart
and chords into combinations of categorical features
allows non-tree- based-models to take into
consideration interactions between features, and improve.
Fine. We just sorted out it feature pre-process for categorical features,
and took a quick look on feature generation.
Now, you will be able to apply these concepts
in your next competition and get better results.
