Skip to main content

Complete Data Science Pipeline

Written by Abhishek mamidi, on

Data Science is not just modelling. To extract value out from Data Science, it needs to be integrated with business and deploy the product to make it available for the users. To build a Data Science product, it needs to go through several steps. In this article, I will discuss the complete Data Science pipeline.

Steps involved in building a Data Science product:
  1. Understanding the Business problem
  2. Data Collection
  3. Data Cleaning
  4. Exploratory Data Analysis
  5. Modelling
  6. Deployment
Let us discuss each step in detail.

Understanding the business problem:
We use Data Science to solve a problem. Without understanding the problem, we can’t apply data science and solve it. Understanding the business is very important in building a data science product. The model which we build completely depends on the problem we are solving. If the requirement is different, we need to adjust our algorithm such that it solves the problem.

For example, if we are building a recommendation system to recommend products. Here, the main goal of the problem is to recommend products to new users. But, if we build a recommendation engine that doesn’t care about this constraint, then it’s no use. The recommendations should be completely different for new users and existing users. We have to build a customized algorithm to recommend products to new users. We cannot use the same algorithm for all types of problems. This is the reason why understanding the business problem is very important.

Data Collection:
After understanding the business problem, we should collect data accordingly. If we have more data, we can build robust models. The data should be accurate. If the data itself has more outliers, it’s of no use even if we build a sophisticated model. So, we should be more careful here. Data collection is boring and difficult. It involves a lot of patience and time.

If we consider the above example, we should collect the data from new customers who are using the platform for the first time and track their clicks. We should not use the data of existing customers. This will add noise to the data. So, data collection is a very important step in solving a problem.

Data Cleaning:
After collecting the data, it should be processed, cleaned and stored in a structured form. Here, domain knowledge plays an important role. We should be able to detect the outliers based on the business problem. We should extract the features that make sense from a business perspective. The more noise we remove, the more robust model can be built. The cleaned data will be directly used for analysis and modelling.

For example, if we are scraping data from a website, it will include Html tags. We should remove them because they are not necessary. In some cases, we cannot extract data for all the fields, in that case, we should replace with NA. We should handle null values and errors. If it is image data, we should remove noisy images. So, cleaning data is one of the important phases in the pipeline.

Python, R languages can be used for Data cleaning. If the data is very huge, we can use SQL.

Exploratory Data Analysis:
After creating the data, we should start analyzing it. We should extract insights and hidden information from the data and relate them with business. Visualization tools help us a lot in this step. We should spend more time in understanding the data. Business understanding helps a lot. We can create new features that make sense. We have to make sure that the distribution of train and test (real world) should be the same.

Some visualization libraries are Matplotlib, Seaborn and GGplot2. Pandas library would be very helpful for exploring data.

Refer this article to know more about EDA and Data Preprocessing steps.

Modelling:
We use Machine learning and Deep learning algorithms to solve the problem. This is more exciting and fun to build different models. Think model as a black box that converts the input data into the output. We should try different methods and choose accordingly based on performance. We cannot use the same method for solving all the problems. The business-related features will be more helpful in creating robust models.

We can use Scikit-learn for building Machine learning models and Keras/TensorFlow for Deep learning models. There are many deep learning frameworks out there, compare them in different aspects before using them.

Deployment:
After building a model, it should be accessible to the users and scalable. So, we need to deploy the model using AWS, Google Cloud,..etc. This is the last step of the pipeline. Once we have deployed it, we have successfully built a Data Science product that can be used by the end-user.

In conclusion, we cannot avoid any step in the pipeline. Each stage is important in building a successful product. In this article, you have learned different steps that are involved in the Data Science pipeline.

Thank you for reading my blog and supporting me. Stay tuned for my next article. If you want to receive email updates, don’t forget to subscribe to my blog. If you have queries, please do comment in the comment section below. I will be more than happy to help you. Keep learning and sharing!!

Follow me here:
GitHub: https://github.com/Abhishekmamidi123
LinkedIn: https://www.linkedin.com/in/abhishekmamidi/
Kaggle: https://www.kaggle.com/abhishekmamidi
If you are looking for any specific blog, please do comment in the comment section below.
Labels:

Comments

  1. We offers urgent help to studentsFebruary 6, 2020 at 4:36 PM

    It is very important to know the problem. If you don't know it, you won't find the solution for it. Will you agree with that?

    ReplyDelete
  2. High Technologies SolutionsMarch 6, 2020 at 1:39 PM

    It was great knowledge after reading this. Thanks for sharing such good stuff with us. I am pleased for sharing on this blog its remarkable blog I really fascinated. Otherwise If anyone Want to learn Basic to Adv. MIS Training & Complete Data Science So Contact here-9311002620.

    Certified MIS & Data Science Training Center in Delhi, India

    MIS Training Institute in Noida
    MIS Training Institute in Delhi
    MIS Training Institute in Faridabad

    ReplyDelete
  3. Android Training in JaipurApril 9, 2020 at 3:50 PM

    Awesome post! This is helpful post. Article is very clear and with lots of useful information. Thanks for such nice article
    Visit : https://pythontraining.dzone.co.in/training/data-science-training.html

    ReplyDelete
  4. SERP HouseApril 23, 2020 at 7:13 PM

    Hi there to everybody, it’s my first go to see of this web site; this weblog consists of awesome and in fact good stuff for visitors. Hurrah, that’s what I was exploring for, what stuff! Existing here at this blog, thanks admin of this web site. You can also visit Google News Api for more SERP House related information and knowledge

    ReplyDelete
  5. lead extraxMay 30, 2020 at 10:04 AM

    This is really good, i love this content also visit LinkedIn Lead Extractor. Thanks for sharing.

    ReplyDelete
  6. aman silavatMay 30, 2020 at 10:05 AM

    Lead ExtraxThis is really good, i love this content also visit LinkedIn Lead Extractor. Thanks for sharing.

    ReplyDelete
  7. AkanshyaOctober 9, 2020 at 10:51 AM

    This comment has been removed by a blog administrator.

    ReplyDelete
  8. Laura BushAugust 24, 2021 at 4:08 AM

    I would like to thank you for sharing such an informative post. The post is very informative as it contains some best knowledge. Thanks for posting it. Keep it up. GAS DISTRIBUTION PIPE Brantford.

    ReplyDelete
  9. high technologies solutionsSeptember 21, 2021 at 1:08 PM

    Thanks for sharing this post its very informative post by the way If anyone look for Ms Office training institute in Delhi Contact Here-+91-9311002620 Or Visit our website https://www.htsindia.com/Courses/microsoft-courses/ms-office-course

    ReplyDelete
  10. Dreamsoft ConsultantsFebruary 12, 2022 at 11:22 AM

    I really appreciate you saying it’s an interesting post to read. I learn new information from your blog, you are doing a great job. Thank you for sharing meaningful information

    Get Genuine Experience Certificate with Reasonable Cost
    Want Fill Your Career GAP ! Call Us & Get Genuine Experience Certificate

    ReplyDelete

Post a Comment

Popular posts from this blog

Google Colab - Increase RAM upto 25GB

Written by Abhishek mamidi, on
Google colab is a free jupyter notebook that is hosted on Google cloud servers. We can use CPU, GPU and TPU for free. It helps you to write and execute your code. You can directly access this through your browser. If you want to use Google Cloud/AWS, it requires hell lot of setup. You have to spin a cluster, create a notebook and then use it. But, Google colab will be readily available for you to use it. You can also install libraries from the notebook itself. These notebooks are very useful for training large models and processing huge datasets. Students and developers can make use of this because it’s very difficult for them to afford GPUs and TPUs. I was trying to run a memory heavy job. The notebook crashed. Then, I came to know how I can increase the RAM. So, I thought of sharing it in my blog. There are some constraints with the notebook. You can run these notebooks for not more than 12 hours and you can use only 12 GB RAM. There is no direct method or button t
18 comments
Read more

Skills required to become a Data Scientist

Written by Abhishek mamidi, on
Data Science is one of the hottest areas in the 21st century. We can solve many complex problems using a huge amount of information. The way electricity has changed the world, information helps us to make our lives easier and comfortable. Every second, an enormous amount of data is being generated. The data may be in the form of text, image, speech or tabular. As there is a lot of growth in the field of Data Science, in recent years, most of the companies have started building their own Data Science teams to get benefited from the information they have. This has created a lot of opportunities and demand for Data Science in different domains. For the next 5+ years, this demand would continue to increase. If we have the right skills, companies are ready to offer salaries more than the market standards. So, this is the right time to explore and gain skills which enables you to enter into this field. We have discussed the importance and demand for data science in the market. Let’s disc
20 comments
Read more

Top 35 frequently asked Data Science interview questions

Written by Abhishek mamidi, on
Interviews are very stressful. We should prepare for the worse. So, we have to plan accordingly in order to crack them. In this blog, you will get to know the type of questions that will be asked during the interview. It also depends on the experience level and the company too. This blog is mainly focused on entry-level Data Science related jobs. If you haven’t read my previous blog-posts, I highly recommend you to go through them: Skills required to become a Data Scientist How to apply for a Data Science job? First of all, you must be thorough with your resume, mainly your Internship experience and academic projects. You will have at least one project discussion round. Take mock interviews and improve your technical and presentation skills, which will surely help in the interviews. Based on my experience, I have curated the topmost 35 frequently asked Data Science questions during the interviews. Explain the Naive Bayes classifier? In case of Regression, how do y
17 comments
Read more

My Data Science Journey and Suggestions - Part 1

Written by Abhishek mamidi, on
I always wanted to share my detailed Data Science journey. So, I have divided the whole journey from BTech first year to final year into 3 parts. I will share everything, without leaving a single detail, starting from my projects, internships to getting a job. You can follow the path that I have followed if you like my journey or create your own path. In 2015, I got a seat in Electronics and Communication Engineering (ECE), IIIT Sri City through IIT JEE Mains. Because of my rank in JEE Mains, I couldn’t get into the Computer Science department. I wanted to shift to Computer Science after my first year, but couldn’t due to some reasons. In our college, we have only two branches, CSE and ECE. For the first three semesters, the syllabus was the same for both the departments except for a few courses. This helped me to explore Computer Science. In the first 3 semesters, I took Computer Programming, Data Structures, Algorithms, Computer Organization, Operation Systems courses, wh
11 comments
Read more

Exploratory Data Analysis and Data Preprocessing steps

Written by Abhishek mamidi, on
Exploratory Data Analysis is the foremost step while solving a Data Science problem. EDA helps us to solve 70% of the problem. We should understand the importance of exploring the data. In general, Data Scientists spend most of their time exploring and preprocessing the data. EDA is the key to building high-performance models. In this article, I will tell you the importance of EDA and preprocessing steps you can do before you dive into modeling. I have divided the article into two parts: Exploratory Data Analysis Data Preprocessing Steps Exploratory Data Analysis Exploratory Data Analysis(EDA) is an art. It’s all about understanding and extracting insights from the data. When you solve a problem using Data Science, it is very important to have domain knowledge. This helps us to get the insights better according to the business problem. We can find the magic features from the data, which boost the performance. We can do the following with EDA. Get comfortable with
13 comments
Read more

Building ML Pipelines using Scikit Learn and Hyper Parameter Tuning

Written by Abhishek mamidi, on
Data Scientists often build Machine learning pipelines which involves preprocessing (imputing null values, feature transformation, creating new features), modeling, hyper parameter tuning. There are many transformations that need to be done before modeling in a particular order. Scikit learn provides us with the Pipeline class to perform those transformations in one go. Pipeline serves multiple purposes here (from documentation ): Convenience and encapsulation : You only have to call fit and predict once on your data to fit a whole sequence of estimators. Joint parameter selection : You can grid search over parameters of all estimators in the pipeline at once (hyper-parameter tuning/optimization). Safety : Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors. In this article, I will show you How to build a complete pi
11 comments
Read more

SHAP - An approach to explain the output of any ML model (with Python code)

Written by Abhishek mamidi, on
Can we explain the output of complex tree models? We use different algorithms to improve the performance of the model. If you input a new test datapoint into the model, it will produce an output. Did you ever explore which features are causing to produce the output? We can extract the overall feature importance from the model, but can we get which features are responsible for the output? If we use a decision tree, we can at least explain the output by plotting the tree structure. But, it’s not easy to explain the output for advanced tree-based algorithms like XGBoost, LightGBM, CatBoost or other scikit-learn models. To explain the output for the above algorithms, researches have come up with an approach called SHAP. SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate ad
2 comments
Read more

Latent Dirichlet Allocation - LDA (With Python code)

Written by Abhishek mamidi, on
Latent Dirichlet Allocation, also known as LDA, is one of the most popular methods for topic modelling. Using LDA, we can easily discover the topics that a document is made of. LDA assumes that the documents are a mixture of topics and each topic contain a set of words with certain probabilities. For example, consider the below sentences: Apple and Banana are fruits. I bought a bicycle recently. In less than two years, I will buy a bike. The colour of the apple and bicycle are red. The output of LDA would look like this: Topic 1 : 0.7*apple + 0.3*banana Topic 2 : 0.6*bicycle + 0.4*bike Sentence 1 : [(Topic 1, 1), (Topic 2, 0)] Sentence 2 : [(Topic 1, 0), (Topic 2, 1)] Sentence 3 : [(Topic 1, 0.5), (Topic 2, 0.5)] Please note that the above probabilities are made up numbers for intuition. To extract the topics and probability of words using LDA, we should decide the number of topics (k) beforehand. Based on that, LDA discovers the topic distribution of docum
6 comments
Read more

A year of experience as a Data Scientist

Written by Abhishek mamidi, on
On June 3rd 2019, I joined ZS Associates as a Data Scientist after graduating from IIIT SriCity. It was my first job and was very happy to get placed as a Data Scientist through lateral hiring. If you haven’t read my Data Science journey, please read it here :) After joining, I had some awesome moments that I never experienced since childhood. I got a chance to stay in a 4 star or 5 star hotel multiple times. I got a chance to travel by flight. I travelled to Pune, Delhi and Bangalore. I saw Vizag, Pune, Delhi and Bangalore airports in less than six months. I loved it. A few office parties, outings during Diwali and New year celebrations. Above are some of the moments that I can never forget in my life. My first job allowed me to experience these first time moments. Enjoying life is more important than anything. If you don’t enjoy your life, you cannot achieve anything big. Okay, let’s go into the main topic in detail. Me (inner voice during BTech):
29 comments
Read more