Descriptive statistics is a powerful tool to gain insight into the generalization of a data set. The first thing we’ll do is load data from different sources and then organize it a bit.


Summary of the previous section

Clean data is a key component in the development of data-driven solutions since it helps us to boost productivity, minimize annoyance, and provide higher-quality outcomes.

Duplicated observations

Duplicate data is eliminated only if it is the product of an error in the generation of the data set.

Irrelevant observations

Non-informative variables can slow down processes, sometimes less is more, it is better to exclude redundant and non-informative information.

Structural issues

Homogenization of the records, maximum and minimum values in the records, consistencies in the texts, and type of variable.

Undesirable outliers

Extreme values that could affect subsequent adjustments are identified and eliminated or imputed.

Missing data

If we are working with a time series, missing values can be inconvenient, it is advisable to impute them.

Verify and QA

Do the descriptive statistics after the cleaning process indicate that the data is consistent and of good quality?