| Dataset | Harrell's C | ISBS |
|---|---|---|
| AK | ||
| CarpenterFdaData | 1 / 30 (3.3%) | — |
| channing | 1 / 30 (3.3%) | 1 / 30 (3.3%) |
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| e1684 | — | 3 / 30 (10%) |
| hdfail | 3 / 3 (100%) | 3 / 3 (100%) |
| lung | — | 8 / 30 (26.7%) |
| uis | — | 2 / 30 (6.7%) |
| veteran | — | 3 / 30 (10%) |
| CIF | ||
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| hdfail | 3 / 3 (100%) | 3 / 3 (100%) |
| Flex | ||
| aids.id | 10 / 30 (33.3%) | — |
| check_times | 3 / 3 (100%) | 3 / 3 (100%) |
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| dataFTR | — | 2 / 30 (6.7%) |
| hdfail | 3 / 3 (100%) | 3 / 3 (100%) |
| lung | — | 9 / 30 (30%) |
| nafld1 | 14 / 15 (93.3%) | 14 / 15 (93.3%) |
| nwtco | 15 / 15 (100%) | 15 / 15 (100%) |
| support | 3 / 3 (100%) | 3 / 3 (100%) |
| wa_churn | 15 / 15 (100%) | 15 / 15 (100%) |
| GLMN | ||
| bladder0 | — | 1 / 30 (3.3%) |
| channing | 1 / 30 (3.3%) | — |
| check_times | — | 2 / 3 (66.7%) |
| cost | — | 12 / 30 (40%) |
| dataSTR | — | 2 / 30 (6.7%) |
| hdfail | 3 / 3 (100%) | — |
| std | — | 6 / 30 (20%) |
| uis | — | 4 / 30 (13.3%) |
| veteran | 14 / 30 (46.7%) | — |
| wbc1 | 4 / 30 (13.3%) | — |
| MBSTAFT | ||
| hdfail | — | 2 / 3 (66.7%) |
| MBSTCox | ||
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| dataSTR | 1 / 30 (3.3%) | — |
| hdfail | 3 / 3 (100%) | 3 / 3 (100%) |
| ORSF | ||
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| cost | — | 1 / 30 (3.3%) |
| gbsg | — | 1 / 15 (6.7%) |
| hdfail | 3 / 3 (100%) | 3 / 3 (100%) |
| nafld1 | 9 / 15 (60%) | 1 / 15 (6.7%) |
| uis | 1 / 30 (3.3%) | — |
| veteran | — | 1 / 30 (3.3%) |
| Pen | ||
| aids.id | 9 / 30 (30%) | 1 / 30 (3.3%) |
| bladder0 | — | 8 / 30 (26.7%) |
| channing | — | 1 / 30 (3.3%) |
| check_times | 3 / 3 (100%) | 3 / 3 (100%) |
| cost | — | 3 / 30 (10%) |
| dataSTR | 3 / 30 (10%) | 11 / 30 (36.7%) |
| hdfail | 2 / 3 (66.7%) | — |
| RAN | ||
| check_times | 2 / 3 (66.7%) | 3 / 3 (100%) |
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| cost | — | 1 / 30 (3.3%) |
| hdfail | 1 / 3 (33.3%) | 1 / 3 (33.3%) |
| mgus | 2 / 30 (6.7%) | — |
| nafld1 | 9 / 15 (60%) | 4 / 15 (26.7%) |
| RFSRC | ||
| check_times | 3 / 3 (100%) | 3 / 3 (100%) |
| child | 3 / 3 (100%) | 3 / 3 (100%) |
| colrec | 2 / 3 (66.7%) | 1 / 3 (33.3%) |
| nafld1 | 1 / 15 (6.7%) | 2 / 15 (13.3%) |
| support | 3 / 3 (100%) | 2 / 3 (66.7%) |
| RRT | ||
| dataFTR | — | 5 / 30 (16.7%) |
| lung | — | 8 / 30 (26.7%) |
| metabric | — | 7 / 15 (46.7%) |
| nwtco | — | 7 / 15 (46.7%) |
| ova | — | 3 / 30 (10%) |
| tumor | — | 3 / 30 (10%) |
| SSVM | ||
| check_times | — | 3 / 3 (100%) |
| child | — | 3 / 3 (100%) |
| colrec | — | 3 / 3 (100%) |
| flchain | — | 11 / 15 (73.3%) |
| hdfail | — | 3 / 3 (100%) |
| nafld1 | — | 15 / 15 (100%) |
| nwtco | — | 8 / 15 (53.3%) |
| ova | — | 3 / 30 (10%) |
| support | — | 3 / 3 (100%) |
| wa_churn | — | 15 / 15 (100%) |
Errors and Elapsed Time Limits
The following table lists the number of errors in the outer resampling iterations per tuning measure (tune_measure). These errors were caused by the learner exceeding the time limit or exceeding memory limitations. We attempted to resubmit failing computational jobs with increased memory limits, yet in some cases the jobs still failed, at which point we considered the learner/task combination to be computationally infeasible.
We note:
- the affected learners were particularly slow or memory intensive for large tasks with many observations or a large number of unique time points, where the latter in particular appeared even more relevant than the number of observations.
- the tasks below are most often those with many observations and unique time points (hdfail, child, check_times).
We therefore consider the errors to be a result of the learners’ complexity and the tasks’ size, given reasonable computational constraints.