CNN_varIMP_NULL_model.RdCNN NULL variable importance. The NULL importance measured by the decrease in a model score (i.e., Mean Decrease Accuracy (MDA), Mean Decrease in RMSE) when a variable is set as NULL variable.
CNN_varIMP_NULL_model(optmodel, feature_names = NULL, train_y = NULL, train_x = NULL, smaller_is_better = NULL, type = c("difference", "ratio"), nsim = 1, sample_size = NULL, sample_frac = NULL, verbose = FALSE, progress = "none", parallel = FALSE, paropts = NULL, ...)
| optmodel | The optimal model |
|---|---|
| feature_names | The variable names, should be the same as the dim of X |
| train_y | The Y variable (dependent variable) used in regression |
| train_x | The independent variable dataset |
| smaller_is_better | Indicating whether the importance is better for small values |
| type | whether "difference" or "ratio" |
| nsim | number of permutations used to estimate the permutation feature importance |
This function estimates the similar model agnostic feature importance but rather using time consuming permutation importance, it uses the Null model importance. Likewise, the best model is determined and the orignal variable metrics are used as the baseline. Then the NULL variable performance metrics are tested using the best model as the training set. This procedure directly set the target variable as NULL variable (Zero variance), thus the drop in the model score is indicative of how much the model depends on the variable.
Return a list of scores, including CNN model decrease in accuracy, the permutation metrics, and the baseline metrics.
Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Model-agnostic interpretability of machine learning." arXiv preprint arXiv:1606.05386 (2016). Fisher, Aaron, Cynthia Rudin, and Francesca Dominici. “Model Class Reliance: Variable importance measures for any machine learning model class, from the ‘Rashomon’ perspective.” http://arxiv.org/abs/1801.01489 (2018).
qinxinghu@gmail.com