Drop Features & Smart Correlated Features

Drop Features

It drops a list of variables indicated by the user. Sometimes, we create new variables combining other variables in the dataset, for example, we obtain the variable age by subtracting date_of_application from date_of_birth. After we obtained our new variable, we do not need the date variables in the dataset any more.

Smart Correlated Features

This transformer finds fields that correlate with each other, then drops fields from each correlated group, based on a specified method. It retains fields with:

• the highest variance
• the highest cardinality
• the least missing data
• the best performing model (based on a single feature)
• the strongest correlation with the target variable

Correlation is calculated with pandas.corr().

Parameters:

• method : can take ‘pearson’, ‘spearman’, ‘kendall’ or callable. It refers to the correlation method to be used to identify the correlated features.
• threshold : The correlation threshold above which a feature will be deemed correlated with another one and removed from the dataset.
• selection_method : takes the values 'missing_values', 'cardinality', 'variance', 'model_performance', and 'corr_with_target'.
• • missing_values : keeps the feature from the correlated group with the least missing observations.
  • cardinality : keeps the feature from the correlated group with the highest cardinality.
  • variance : keeps the feature from the correlated group with the highest variance.
  • model_performance : trains a machine learning model using each of the features in a correlated group and retains the feature with the highest importance.
  • corr_with_target : keeps the feature from the correlated group that has the highest (absolute) correlation with the target variable. The same correlation method defined in the method parameter is used to calculate the correlation between the features and the target.