5. Exploratory Data Analysis [ EDA ] _Part 3 [ Identifying missing values ]

Note: Please read previous article : Checking for Duplicate Values for better understanding.

b. Identifying Missing Values in Dependent and Independent Variables

Checking for missing values is a crucial step in the data analysis and preprocessing process for several important reasons:

Data Quality Assurance:

1. • Identifying missing values helps ensure the quality and integrity of the dataset. It allows for a thorough examination of data completeness and accuracy.

2. Avoiding Bias in Analysis:

   • Missing values can introduce bias into statistical analyses and machine learning models. Detecting and addressing these gaps is essential to obtain accurate and unbiased results.

3. Preventing Misleading Conclusions:

   • Ignoring missing values may lead to incorrect conclusions and interpretations. It's important to be aware of the extent of missing data to avoid drawing misleading or inaccurate insights.

4. Ensuring Validity of Results:

   • Many statistical tests and analyses assume the availability of complete data. Checking for missing values ensures that the assumptions of these tests are met, contributing to the validity of the results.

5. Making Informed Decisions:

   • Awareness of missing values allows analysts to make informed decisions on how to handle them. Depending on the nature and extent of missing data, strategies such as imputation or removal of incomplete cases can be applied.

6. Improving Model Performance:

   • Missing values can adversely impact the performance of machine learning models. Detecting and appropriately handling missing data contribute to the robustness and effectiveness of predictive models.

7. Understanding Data Patterns:

   • The presence of missing values may indicate patterns or trends in the data. Understanding these patterns can lead to insights about data collection processes, potential biases, or other systematic issues.

8. Meeting Analytical Requirements:

   • Some analytical techniques and algorithms may require complete data. Detecting missing values allows for the preparation of the data to meet the specific requirements of the chosen analytical approach.

9. Compliance and Reporting:

   • In certain industries or applications, compliance standards or reporting regulations may mandate a thorough examination and documentation of missing data. This is particularly important in fields like finance, healthcare, and research.

10. Enhancing Data Cleaning Strategies:

    • Identifying missing values guides the development of effective data cleaning strategies. It helps in deciding whether to impute missing values, remove incomplete cases, or employ other methods based on the specific characteristics of the data.

In summary, checking for missing values is a fundamental aspect of data exploration and analysis. It contributes to the overall reliability, accuracy, and interpretability of results, ensuring that subsequent analyses and models are based on a solid foundation of complete and trustworthy data.

There are several ways to find missing values in a dataset. Here are a few common methods, demonstrated with an example dataset:

Let's consider a hypothetical dataset:

In Pandas missing data is represented by two value:

• None: None is a Python singleton object that is often used for missing data in Python code.
• NaN : NaN (an acronym for Not a Number), is a special floating-point value recognized by all systems that use the standard IEEE floating-point representation

1.Using .isnull() or .isna() method:

2.Using .info() method:

The output will include information on the number of non-null values in each column, making it easy to identify missing values.

3.Using .isnull().sum() method:

This provides the total number of missing values in each column.

4.Using Heatmap Visualization:

A heatmap provides a visual representation of missing values, where yellow lines indicate missing values.

5.Using .isnull().any():

6.Identifying Missing values in particular column using .isnull(), .sum(), .any(), .all()

7. Identifying Missing values in particular column using .isna(), .sum(), .any(), .all()

8.Identifying Missing values in selected two columns

9.Identifying Columns with Missing values using for-loop

10.Identifying Columns with Missing values using Lambda function

Lambda functions are generally used for concise, one-line operations

💻 Assignment

1. Consider the following Data Set and find out answers to the given questions.

Questions:)

1. Find out missing values using .info() method?

2. Find out missing values using .isnull() and .sum(), .any(), .all() metthods?

3. Find out missing values using .isna() and .sum(), .any(), .all() metthods?

4. Find out missing values using for-loop?

5. Find out missing values using lambda function?

6. Find out missing values using Heat map visualization?

7. Name out which columns have missing values?

For more information contact on : venkatesh.mungi.datascientist@gmail.com