Question 1

What is a decision tree in Machine Learning?

Accepted Answer

A decision tree is a Machine Learning model that makes predictions by splitting data according to hierarchical decision rules. Each internal node represents a test on a feature, each branch represents the outcome of the test, and each leaf represents a final prediction. This model is intuitive and easily interpretable, making it an excellent choice for understanding factors influencing a decision.

Question 2

What criterion is used by default in scikit-learn to measure the quality of a split in a classification tree?

Accepted Answer

The Gini index is the default criterion in scikit-learn for classification trees. It measures the impurity of a node by calculating the probability that an element would be misclassified if randomly classified according to the class distribution. A Gini of 0 means a pure node (single class), while a higher Gini indicates greater class diversity.

Question 3

What is the main difference between Gini index and entropy as split criteria?

Accepted Answer

Gini index and entropy generally produce very similar trees, but Gini is slightly faster to compute as it doesn't require logarithmic calculation. Entropy, based on information theory, may sometimes create slightly more balanced splits. In practice, the choice between the two rarely has a significant impact on model performance.

Question 4

What is pruning in the context of decision trees?

Accepted Answer

Pruning is a regularization technique that reduces the size of a decision tree to avoid overfitting. Pre-pruning limits tree growth during training (via max_depth, min_samples_split), while post-pruning builds a complete tree then removes branches that don't significantly improve validation. This technique improves model generalization.

Question 5

Which hyperparameter controls the maximum depth of a decision tree in scikit-learn?

Accepted Answer

The max_depth parameter limits the maximum tree depth, meaning the maximum number of levels from root to leaves. A low max_depth value produces a simple tree with less overfitting risk but potentially more bias. A high value allows a more complex tree that can better capture patterns but risks overfitting to data noise.

Question 6

What is a Random Forest?

Accepted Answer

A Random Forest is an ensemble method that combines multiple decision trees trained on different data subsamples (bagging) and with a random subset of features at each split. Predictions are aggregated by majority vote (classification) or averaging (regression). This approach reduces variance and overfitting compared to a single tree while maintaining good predictive capacity.

Question 7

What is bagging (Bootstrap Aggregating) in the context of Random Forests?

Accepted Answer

Bagging consists of creating multiple subsamples of the original dataset by sampling with replacement (bootstrap). Each tree is trained on a different bootstrap sample, introducing diversity between trees. Predictions from all trees are then aggregated. This technique reduces model variance without increasing bias, thus improving robustness and generalization.

Question 8

What is the Out-of-Bag (OOB) score in a Random Forest?

Accepted Answer

The OOB score is an estimate of model performance using samples not selected during bootstrap. On average, about 37% of data is not used to train each tree. These OOB samples serve as internal validation without requiring a separate test set. The OOB score provides an unbiased estimate of generalization error similar to cross-validation.

Question 9

Which hyperparameter controls the number of trees in a Random Forest?

Accepted Answer

The n_estimators parameter defines the number of decision trees in the forest. More trees mean a more stable and robust model, but training time increases proportionally. Generally, increasing n_estimators improves performance up to a plateau, after which gains become marginal. A typical value ranges between 100 and 500 trees depending on dataset size.

Question 10

Why does Random Forest select a random subset of features at each split?

Accepted Answer

Random feature selection (controlled by max_features) at each split increases diversity between trees and reduces their correlation. If all trees used all features, they would often make the same splits on dominant features, producing very similar trees. By limiting available features, trees explore different combinations, improving aggregation and reducing overall variance.

Question 11

What is Gradient Boosting?

Accepted Answer

Gradient Boosting is an ensemble technique that trains trees sequentially, where each new tree corrects errors (residuals) from the previous model. Unlike bagging which parallelizes trees, boosting builds them one by one by optimizing a loss function via gradient descent. This approach is very powerful for improving accuracy but requires careful tuning to avoid overfitting.

Question 12

What is the main difference between bagging (Random Forest) and boosting (Gradient Boosting)?

Accepted Answer

Bagging trains trees independently in parallel on different bootstrap samples and aggregates their predictions to reduce variance. Boosting trains trees sequentially, with each tree learning to correct previous errors, which reduces bias. Bagging is more robust and parallelizable, while boosting can achieve better accuracy but is more sensitive to overfitting and noise.

Question 13

What is the role of the learning_rate parameter in Gradient Boosting?

Accepted Answer

The learning_rate (or shrinkage) controls each tree's contribution to the final model by multiplying its prediction by a factor between 0 and 1. A low learning_rate (e.g., 0.01-0.1) slows learning, requiring more trees but reducing overfitting and often improving generalization. A high learning_rate speeds up training but risks converging to a local optimum or overfitting.

Question 14

What is XGBoost and what does it bring compared to classic Gradient Boosting?

Accepted Answer

XGBoost (Extreme Gradient Boosting) is an optimized implementation of Gradient Boosting. It brings built-in L1/L2 regularization to reduce overfitting, efficient handling of missing values, parallel computation of splits, and optimized memory management. These optimizations make it faster and often more performant, establishing it as the reference algorithm for many Kaggle competitions.

Question 15

What are the regularization parameters available in XGBoost?

Accepted Answer

XGBoost offers several regularization parameters: alpha (L1 regularization) which encourages sparsity by penalizing leaf count, lambda (L2 regularization) which penalizes high leaf weights, gamma (minimum loss reduction) which requires minimum gain for a split, and max_depth which limits tree depth. These combined parameters provide fine control over model complexity and overfitting.

Question 16

What is the difference between XGBoost and LightGBM in terms of tree growth strategy?

Accepted Answer

XGBoost uses level-wise growth by default, where all nodes at the same level are expanded before moving to the next. LightGBM uses leaf-wise growth, where the leaf with the highest gain is expanded first. The leaf-wise strategy is generally faster and can achieve better performance with fewer trees, but is more prone to overfitting on small datasets.

Question 17

How to calculate impurity-based feature importance in a decision tree?

Accepted Answer

Impurity-based feature importance is calculated by summing the total impurity reduction (Gini or entropy) brought by each feature across all splits where it's used, weighted by the number of samples passing through those splits. The more frequently and significantly a feature reduces impurity, the higher its importance. This method is fast but may favor features with many unique values (high cardinality).

Question 18

What is permutation feature importance and why is it preferred in some cases?

Accepted Answer

Permutation feature importance measures each feature's impact by randomly shuffling its values and observing model performance degradation. Unlike impurity-based importance, it's calculated on a validation set, better reflects actual importance for prediction, and isn't biased toward high-cardinality features. It's more computationally expensive but provides a more reliable and model-agnostic estimate.

Question 19

What is the impact of the min_samples_split parameter on a decision tree?

Accepted Answer

The min_samples_split parameter defines the minimum number of samples required to split an internal node. A higher value prevents the tree from splitting nodes with few samples, acting as a form of regularization that reduces overfitting. A low value allows deeper and more complex trees that can capture finer patterns but risk overfitting to data noise.

Question 20

How does early stopping work in Gradient Boosting and XGBoost?

Accepted Answer

Early stopping monitors model performance on a validation set during training. If the validation metric doesn't improve for a specified number of iterations (early_stopping_rounds), training stops to avoid overfitting. The model returns to the best observed state. This technique automatically finds the optimal number of trees without manually defining it via n_estimators.

Question 21

When should you use a Random Forest rather than Gradient Boosting?

Accepted Answer

Random Forest is preferable when training time is critical (parallelizable), when the dataset contains lots of noise, when robustness to hyperparameters matters (less tuning required), or when interpretability is needed. Gradient Boosting excels when maximum accuracy is the main goal, when the dataset is clean with little noise, and when time for hyperparameter tuning is available.

Question 22

How does XGBoost handle missing values by default?

Accepted Answer

XGBoost natively handles missing values by automatically learning the best direction (left or right) to send them at each split. During training, the algorithm tests both directions for missing values and chooses the one that minimizes loss. This approach is more effective than manual imputation as it adapts to each split and can capture patterns where the presence of missing values is itself informative.

Question 23

What strategy should be adopted for hyperparameter tuning of Gradient Boosting or XGBoost?

Accepted Answer

An effective strategy is to start with a relatively high learning_rate (0.1) with early stopping to quickly find the right number of trees, then reduce learning_rate and increase n_estimators proportionally. Then tune regularization parameters (max_depth, min_child_weight, gamma, lambda, alpha) via GridSearchCV or RandomizedSearchCV. Use cross-validation to avoid overfitting on the validation set.

Question 24

What is CatBoost and what is its main advantage over XGBoost and LightGBM?

Accepted Answer

CatBoost is a Gradient Boosting algorithm developed by Yandex that excels at handling categorical features. Unlike XGBoost and LightGBM which often require prior encoding (one-hot, label encoding), CatBoost uses built-in ordered target encoding that avoids target leakage. It's also known for working well out-of-the-box with less tuning and for its resistance to overfitting.

Decision Trees & Ensembles

What is a decision tree in Machine Learning?

Answer

What criterion is used by default in scikit-learn to measure the quality of a split in a classification tree?

Answer

What is the main difference between Gini index and entropy as split criteria?

Answer

What is pruning in the context of decision trees?

Which hyperparameter controls the maximum depth of a decision tree in scikit-learn?

What is a Random Forest?

Master Data Science & ML for your next interview