Aggregate method and evidence

SRAG study

Public SIVEP-Gripe notifications were normalized in Silver and filtered into the task-specific Gold cohort. The modeled cohort contains 1,282,970 hospitalizations: broad COVID definition, hospitalization and closed death/survival outcomes. The model has 40 inputs, excluding identifiers, the target, ICU and ventilatory support.

Age is in decimal years. Unknown categorical states remain explicit. XGBoost uses native categorical inputs.

Temporal model selection

Fit through 2022 on a uniform sample of 200,000 records. Choose with 24,148 validation records from 2023. Explain on all 16,142 temporal test records from 2024.

Highest validation AUC was the primary selection rule. The technical tie rule did not produce a tie. Validation was not added to the selected model’s training data.

The fitted predictor

XGBoost, fixed during PFI. No class weights or resampling in this reference. The historical test has already been inspected; the explanation is exploratory.

{
  "n_estimators": 800,
  "max_depth": 4,
  "learning_rate": 0.05,
  "min_child_weight": 100,
  "reg_lambda": 5.0,
  "tree_method": "hist",
  "enable_categorical": true,
  "eval_metric": "auc",
  "random_state": 42,
  "n_jobs": -1
}

PFI protocol

Original test Brier: 0.1308. Each input is separately shuffled 50 times across all evaluation rows, seed 42. Every repeat copies the original matrix. The predictor and outcomes stay fixed.

The difference is permuted Brier minus original Brier. The ratio divides permuted Brier by original Brier. Whiskers show one sample standard deviation across shuffles (ddof = 1), conditional on this fitted model and these evaluation records.

Dependent-input diagnostics reuse the same fitted model and baseline. Joint shuffling moves heart disease and diabetes using the same donor row. Subgroup shuffling moves heart disease only within each diabetes category. Neither resolves all other input dependencies.

Generic reproduction algorithm

The browser replays recorded aggregate results. It does not train a model or calculate predictions from the symbolic rows.

# Inputs: a fitted probability predictor and evaluation data with outcomes.
rng = np.random.default_rng(42)
orders = [rng.permutation(len(x_eval)) for _ in range(50)]
p_original = np.asarray(model.predict_proba(x_eval)[:, 1], dtype=np.float64)
e_original = brier_score_loss(y_eval, p_original)
for feature in x_eval.columns:
    errors = []
    for order in orders:
        x_perm = x_eval.copy()
        x_perm[feature] = x_eval[feature].iloc[order].set_axis(x_eval.index)
        p_perm = np.asarray(model.predict_proba(x_perm)[:, 1], dtype=np.float64)
        errors.append(brier_score_loss(y_eval, p_perm))
    delta = np.asarray(errors) - e_original
    importance = delta.mean()
    shuffle_sd = delta.std(ddof=1)

Aggregate evidence

Read the aggregate JSON. It contains the ranking summaries, recorded full-test Age errors and dependence-diagnostic errors across repeated shuffles, and study/model metadata. It contains no patient rows or individual probabilities.

The original source measurements are generated by the course evidence notebook. This presentation contains aggregates and a generic algorithm. The module notebooks validate the exported measurements; no individual demonstration records are shipped with the presentation.

Sources and licenses

Molnar · Permutation Feature Importance guides interpretation and subgroup discussion. Raschka · Feature Selection lecture provides the conceptual sequence and original slides.

Lecture code and documentation: MIT License · William Bendinelli.

Raschka’s original pages are linked as a source and are not reproduced in this package. His course repository uses MIT; its notice is retained here. The course Inter font is distributed under the SIL Open Font License.

Prepared by William Bendinelli · SCC5819 · ICMC-USP.