Practical - 2

Aim: 

Perform following Data Pre-processing (Feature Selection/Elimination) tasks using python

THEORY:

    Feature Selection is one of the core concepts in machine learning which hugely impacts the performance of your model. The data features that you use to train your machine learning models have a huge influence on the performance you can achieve.

Feature engineering is the process of translating the collected data into features that better reflect the problem we are trying to solve to the model, enhancing its efficiency and precision.

• Univariate Selection
• Recursive Feature Elimination
• Principal Component Analysis
• Feature Importance  

The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very high-dimensional datasets. 

    

Reduces Overfitting: Less redundant data means less possibility of making decisions based on redundant data/noise.

Improves Accuracy: Less misleading data means modeling accuracy improves.

Reduces Training Time: Less data means that algorithms train faster.

Univariate Selection:

    Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:

• SelectKBest removes all but the  highest scoring features
• SelectPercentile removes all but a user-specified highest scoring percentage of features
• using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.

Recursive feature elimination (RFE): 

    Unlike the univariate method, RFE starts by fitting a model on the entire set of features and computing an importance score for each predictor. The weakest features are then removed, the model is re-fitted, and importance scores are computed again until the specified number of features are used. Features important score are ranked by the model’s coef_ or feature_importances_attributes, and by recursively eliminating a small number of features per loop.

Feature Importance:

    Methods that use ensembles of decision trees (like Random Forest or Extra Trees) can also compute the relative importance of each attribute. These importance values can be used to inform a feature selection process.

This recipe shows the construction of an Extra Trees ensemble of the iris flowers dataset and the display of the relative feature importance.

Principal Component Analysis:

    An important machine learning method for reduction of dimension is called Principal Component Analysis. And this method is also called Data Reduction Technique. A property of PCA is that you have the choice to select the number of dimensions or principal component in the transformed result.

Code for reference