Skip to main content

Feature Engineering for Data Science


Feature Engineering is one of the most important steps in solving a Data Science problem. This helps us to create a robust and high-performance model. This is the reason why Data Scientists spend time in understanding the data and preprocessing it before diving into modelling.

In this article, I will explain some feature engineering steps that I have learned from the course “How to Win a Data Science Competition: Learn from Top Kagglers”. If you have time, I recommend you to take this course.

Feature Engineering involves mainly feature preprocessing and feature generation. Most of the real-world data is noisy. We cannot expect all the features to be numerical. It may also contain strings, timestamp features etc. it is suggested to convert all the features into numerical and categorical features (some models can handle categorical features too). Using the existing features, we can also create new features based on domain knowledge. They might contribute to predicting the target.

In my previous article, I have discussed the importance of EDA and some preprocessing steps such as removing duplicated data-points, handling highly correlated features, low variance features, imbalance data, treating missing values, encoding categorical features, feature scaling and dimensionality reduction techniques. I recommend reading my previous article before diving into this article.

When we work on tabular data, we come across mainly numeric, continuous and time-related features. Let’s discuss data engineering techniques in detail for each data type.

Data Engineering for Numerical features (continuous):

The continuous features may range from (-inf to inf). It depends on the feature. For example, age, height, salary, population,.etc comes under continuous features. Some preprocessing techniques are:
  1. Feature scaling: Preprocessing continuous features depends on which model we are planning to apply to the data. If it is a tree-based model, we need not worry about scaling. If it is a non-tree-based model, we should scale the features such that they are on the same scale. This is also called as Normalization of features. We try to bring the scale of all the features to [0, 1]. In this case, we can use Min Max Scaler or Standard Scaler to normalize the data.

    • Min Max Scaler: The features are transformed to [0, 1].
      X = (X - X.min()) / (X.max() - X.min())

    • Standard Scaler: The features are standardized by removing the mean and scaled to unit variance.
      X = (X - X.mean()) / X.std()
  2. Remove outliers: Outliers can distract and decrease the performance of the model. We can remove the outliers by clipping the data. You can use Numpy’s clip method to remove the outliers. We can remove the values from the feature if they are less than 1 percentile or greater than 99 percentiles. We can choose minimum and maximum thresholds based on the feature distribution. Tree-based models are robust to outliers.

  3. Feature Transformation: You can use rank transformation, log transformation, etc based on the feature distribution. Rank transformation ranks all the data points in either ascending or descending order. Log transformation helps to transform skewed data into a normal distribution. You can use Scipy’s rankdata method and Numpy’s log1p method for the respective transformations.
Data Engineering for Categorical features:

We can divide categorical features into two types mainly nominal and ordinal. The main difference is that we can order the values in ordinal features. We cannot order or compare the values in nominal features. For example, Sex (M/F) comes under nominal feature and Education (BTech, MTech, PhD) comes under ordinal feature. We can compare the education levels and assign ranks too. Some preprocessing techniques are:
  1. One hot encoding:

    • This is a technique that is used to convert categories into numbers. For example, if we have 5 categories in a feature, then we create 5 features/columns with each category and assign 0 or 1 in the respective column based on the category. OneHotEncoding is widely used when we apply non-tree based methods.

    • If the cardinality of a feature is very high, this technique increases the dimension of the data. In those cases, it is recommended to use BinaryEncoding. This converts each category as binary bit strings (1000100100). For example, we have 1000 categories. If we use OneHotEncoding, it creates 1000 features. Contrary to this, BinaryEncoding creates only 10 features(2^10 = 1024). It represents the same 1000 categories in 10 features.
  2. Label encoding: This technique maps feature to unique integers. Tree-based methods can handle categorical features. So, it is sufficient to use Label Encoding and specify that the features are categorical while initializing the model. It doesn’t increase the dimension of the high cardinality feature too.
Data Engineering for Time-related features:

In addition to numerical and categorical features, date-time features are also generated in some use cases. We can extract a lot of information from date-time features and make use of it. This helps us to capture the seasonality, time,.etc. Some preprocessing techniques are:
  1. From date-time feature, we can extract the day number in a week, month, season, a year from date feature and extract seconds, minutes and the hour from time feature. Some of them are categorical and some are numerical features.

  2. If we have a date-time feature, we can create a new feature “days since” using an anchor date or “days remaining”. It completely depends on the context.

  3. If we have two date-time features, we can create a feature which is the difference between them. The difference can be the number of days, number of months,.etc.
Data generation techniques:

In addition to the above techniques, I will provide some examples on how can we generate new features.
  • We have 2 numerical features “Number of wheels” and “Number of spare wheels”. In this case, you can create a new feature “Total number of wheels” by adding them.

  • There are 2 colours “black” and “blue”. We have the information of 100 people whether they like the colour “black” and “blue” independently. In this case, we can create a new boolean feature “Do they like both colours?” by comparing the features.

  • Most of the techniques that were covered in “time-related features” involves creating new features.
The generation of new features requires domain knowledge and a thorough understanding of the data.

I hope you got some idea on Feature Engineering techniques and how to handle data. If you have any queries, comment in the comments section below. I would be more than happy to answer your queries.

Thank you for reading my blog and supporting me. Stay tuned for my next article. If you want to receive email updates, don’t forget to subscribe to my blog. Keep learning and sharing!!

Follow me here:
GitHub: https://github.com/Abhishekmamidi123
LinkedIn: https://www.linkedin.com/in/abhishekmamidi/
Kaggle: https://www.kaggle.com/abhishekmamidi
If you are looking for any specific blog, please do comment in the comment section below.

Comments

  1. To my mind, the most important aspects are in the first paragraph. If you know '' Feature Scaling '', you will not face any difficulties in understanding the rest of the information.

    ReplyDelete
  2. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. machine learning projects for final year In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.


    The Nodejs Training Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

    ReplyDelete
  3. Master Data Science with R & Python. Also, Learn Machine Learning & Big data. Further More Details Here-9310332343 or Visit Website-http://www.pythontrainingdelhi.com/

    ReplyDelete
  4. The article is so interesting and I really liked it and I would also like to share it with my friends...

    ReplyDelete
  5. It was good experience to read about dangerous punctuation. Informative for everyone looking on the subject.
    data scientist training in hyderabad

    ReplyDelete

Post a Comment

Popular posts from this blog

Google Colab - Increase RAM upto 25GB

Google colab is a free jupyter notebook that is hosted on Google cloud servers. We can use CPU, GPU and TPU for free. It helps you to write and execute your code. You can directly access this through your browser. If you want to use Google Cloud/AWS, it requires hell lot of setup. You have to spin a cluster, create a notebook and then use it. But, Google colab will be readily available for you to use it. You can also install libraries from the notebook itself. These notebooks are very useful for training large models and processing huge datasets. Students and developers can make use of this because it’s very difficult for them to afford GPUs and TPUs. I was trying to run a memory heavy job. The notebook crashed. Then, I came to know how I can increase the RAM. So, I thought of sharing it in my blog. There are some constraints with the notebook. You can run these notebooks for not more than 12 hours and you can use only 12 GB RAM. There is no direct method or button t

Skills required to become a Data Scientist

Data Science is one of the hottest areas in the 21st century. We can solve many complex problems using a huge amount of information. The way electricity has changed the world, information helps us to make our lives easier and comfortable. Every second, an enormous amount of data is being generated. The data may be in the form of text, image, speech or tabular. As there is a lot of growth in the field of Data Science, in recent years, most of the companies have started building their own Data Science teams to get benefited from the information they have. This has created a lot of opportunities and demand for Data Science in different domains. For the next 5+ years, this demand would continue to increase. If we have the right skills, companies are ready to offer salaries more than the market standards. So, this is the right time to explore and gain skills which enables you to enter into this field. We have discussed the importance and demand for data science in the market. Let’s disc

Top 35 frequently asked Data Science interview questions

Interviews are very stressful. We should prepare for the worse. So, we have to plan accordingly in order to crack them. In this blog, you will get to know the type of questions that will be asked during the interview. It also depends on the experience level and the company too. This blog is mainly focused on entry-level Data Science related jobs. If you haven’t read my previous blog-posts, I highly recommend you to go through them: Skills required to become a Data Scientist How to apply for a Data Science job? First of all, you must be thorough with your resume, mainly your Internship experience and academic projects. You will have at least one project discussion round. Take mock interviews and improve your technical and presentation skills, which will surely help in the interviews. Based on my experience, I have curated the topmost 35 frequently asked Data Science questions during the interviews. Explain the Naive Bayes classifier? In case of Regression, how do y

My Data Science Journey and Suggestions - Part 1

I always wanted to share my detailed Data Science journey. So, I have divided the whole journey from BTech first year to final year into 3 parts. I will share everything, without leaving a single detail, starting from my projects, internships to getting a job. You can follow the path that I have followed if you like my journey or create your own path. In 2015, I got a seat in Electronics and Communication Engineering (ECE), IIIT Sri City through IIT JEE Mains. Because of my rank in JEE Mains, I couldn’t get into the Computer Science department. I wanted to shift to Computer Science after my first year, but couldn’t due to some reasons. In our college, we have only two branches, CSE and ECE. For the first three semesters, the syllabus was the same for both the departments except for a few courses. This helped me to explore Computer Science. In the first 3 semesters, I took Computer Programming, Data Structures, Algorithms, Computer Organization, Operation Systems courses, wh

Exploratory Data Analysis and Data Preprocessing steps

Exploratory Data Analysis is the foremost step while solving a Data Science problem. EDA helps us to solve 70% of the problem. We should understand the importance of exploring the data. In general, Data Scientists spend most of their time exploring and preprocessing the data. EDA is the key to building high-performance models. In this article, I will tell you the importance of EDA and preprocessing steps you can do before you dive into modeling. I have divided the article into two parts: Exploratory Data Analysis Data Preprocessing Steps Exploratory Data Analysis Exploratory Data Analysis(EDA) is an art. It’s all about understanding and extracting insights from the data. When you solve a problem using Data Science, it is very important to have domain knowledge. This helps us to get the insights better according to the business problem. We can find the magic features from the data, which boost the performance. We can do the following with EDA. Get comfortable with

SHAP - An approach to explain the output of any ML model (with Python code)

Can we explain the output of complex tree models? We use different algorithms to improve the performance of the model. If you input a new test datapoint into the model, it will produce an output. Did you ever explore which features are causing to produce the output? We can extract the overall feature importance from the model, but can we get which features are responsible for the output? If we use a decision tree, we can at least explain the output by plotting the tree structure. But, it’s not easy to explain the output for advanced tree-based algorithms like XGBoost, LightGBM, CatBoost or other scikit-learn models. To explain the output for the above algorithms, researches have come up with an approach called SHAP. SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate ad

Complete Data Science Pipeline

Data Science is not just modelling. To extract value out from Data Science, it needs to be integrated with business and deploy the product to make it available for the users. To build a Data Science product, it needs to go through several steps. In this article, I will discuss the complete Data Science pipeline. Steps involved in building a Data Science product: Understanding the Business problem Data Collection Data Cleaning Exploratory Data Analysis Modelling Deployment Let us discuss each step in detail. Understanding the business problem: We use Data Science to solve a problem. Without understanding the problem, we can’t apply data science and solve it. Understanding the business is very important in building a data science product. The model which we build completely depends on the problem we are solving. If the requirement is different, we need to adjust our algorithm such that it solves the problem. For example, if we are build

How to apply for a Data Science job?

Job search is one of the painful tasks. We have to invest a lot of time to get placed in one of the best companies, we were dreaming for. The demand for Data Scientists is increasing over the years, and we have to stand out of the crowd to get a job. In this post, I will guide you on “How to apply for a Data Science job?”. I have divided the blog post into the following: What are the skills required for a Data Science job? How to build a good Data Science profile/resume? What are the different ways of applying for a Data Science job? What are the skills required for a Data Science job? I have created a blog-post on “Skills required to become a Data Scientist”, last week. I would suggest going through the previous blog before you go to the next section. How to build a good Data Science profile/resume? After acquiring the necessary skills, it is required to maintain a good Data Science profile. Your presence on the social network makes a difference too. Some tips

A year of experience as a Data Scientist

On June 3rd 2019, I joined ZS Associates as a Data Scientist after graduating from IIIT SriCity. It was my first job and was very happy to get placed as a Data Scientist through lateral hiring. If you haven’t read my Data Science journey, please read it here :) After joining, I had some awesome moments that I never experienced since childhood. I got a chance to stay in a 4 star or 5 star hotel multiple times. I got a chance to travel by flight. I travelled to Pune, Delhi and Bangalore. I saw Vizag, Pune, Delhi and Bangalore airports in less than six months. I loved it. A few office parties, outings during Diwali and New year celebrations. Above are some of the moments that I can never forget in my life. My first job allowed me to experience these first time moments. Enjoying life is more important than anything. If you don’t enjoy your life, you cannot achieve anything big. Okay, let’s go into the main topic in detail. Me (inner voice during BTech):

Building ML Pipelines using Scikit Learn and Hyper Parameter Tuning

Data Scientists often build Machine learning pipelines which involves preprocessing (imputing null values, feature transformation, creating new features), modeling, hyper parameter tuning. There are many transformations that need to be done before modeling in a particular order. Scikit learn provides us with the Pipeline class to perform those transformations in one go. Pipeline serves multiple purposes here (from documentation ): Convenience and encapsulation : You only have to call fit and predict once on your data to fit a whole sequence of estimators. Joint parameter selection : You can grid search over parameters of all estimators in the pipeline at once (hyper-parameter tuning/optimization). Safety : Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors. In this article, I will show you How to build a complete pi