Posted by Random Forest Vulnerabilities
Random Forests have the advantage of performing well without the need for extensive hyper-parameter tuning, as long as the number of trees is sufficient. However, they have a few drawbacks, one of which is their significant vulnerability to extrapolation.
Extrapolation
Extrapolation is a type of estimation, beyond the original observation range, of the value of a variable on the basis of its relationship with another variable.
Source: Wikipedia
In principle, all machine learning algorithms should be cautious with extrapolation, and Random Forests are particularly susceptible.
The image below shows that Random Forests provide constant values for areas outside the learned feature space.
The reason for this characteristic.
This characteristic arises due to the nature of the Decision Tree algorithm.(This means that not only Random Forests but all Decision Tree-based algorithms share similar issues.) Decision Trees split the feature space optimally, and the finally divided feature spaces have the same value(in regression, this would be the mean).
As known, Random Forests create several decision trees on bootstrapped datasets and then average the results of each tree. Therefore, since there is no data in the external area, it cannot be split, leading to the issue that any tree predicts constant values regardless of the feature changes, indicating a lack of sensitivity to feature changes as you move outward.
Even if it’s not outside the learned feature space, if there’s a region without data between constants in the learned layout, Decision Trees might predict constant values based on feature changes in that region.
Conclusion
To summarize, if you are looking to observe sensitivity to small changes in data, Random Forest is not the suitable algorithm. This is especially true if
- You are dealing with situations that require extrapolation.
- Even if not extrapolating, if the intervals between features in the training data are too large.
In these cases, despite the potentially more complex tuning process, utilizing a Neural Network may be a better direction.