integrated brier score problem with largest observed time point

[Montrey]

Hello, I am trying to calculate the integrated brier score. Unfortunately, I keep getting the error message
"ValueError: time must be smaller than largest observed time point: 76.0".
The shape of the survival probabilities is (171844, 69). I use the version 0.23.1.

I hope you can help me. Thank you very much. :)

import pandas as pd

> import numpy as np
> from sksurv.ensemble import RandomSurvivalForest
> from sksurv.util import Surv
> from sksurv.metrics import integrated_brier_score
> from sklearn.model_selection import train_test_split
> from sklearn.preprocessing import StandardScaler
> import joblib
> 
> # Zielspalten
> duration_col = 'zyklus'
> event_col = 'ausfall'
> 
> # Survival-Datenstruktur erstellen
> y = Surv.from_dataframe(event=event_col, time=duration_col, data=df_resampled)
> 
> # Definiere die verwendeten Features
> features = [
>     'feature_1', 'feature_2', 
>     'feature_3', 'feature_4', 'feature_5', 
>     'feature_6', 'feature_7', 'feature_8', 'feature_9'
> ]
> 
> # Daten splitten
> X_train, X_test, y_train, y_test = train_test_split(df_resampled[features], y, test_size=0.25, random_state=42)
> 
> # StandardScaler anwenden
> scaler = StandardScaler()
> X_train_scaled = scaler.fit_transform(X_train)
> X_test_scaled = scaler.transform(X_test)
> 
> # Scaler speichern
> scaler_path = '/home/user/test/rsf_scaler.pkl'
> joblib.dump(scaler, scaler_path)
> print(f"Scaler wurde gespeichert unter: {scaler_path}")
> 
> # Random Survival Forest erstellen
> rsf = RandomSurvivalForest(
>     n_estimators=100,
>     min_samples_split=10,
>     min_samples_leaf=5,
>     max_features="sqrt",
>     n_jobs=20,
>     verbose=2
> )
> 
> # Modell trainieren
> rsf.fit(X_train_scaled, y_train)
> 
> # Modell speichern
> model_path = '/home/user/test/rsf_model.pkl'
> joblib.dump(rsf, model_path)
> print(f"Modell wurde gespeichert unter: {model_path}")
> 
> # Median-Überlebenszeiten vorhersagen
> predicted_medians = rsf.predict(X_test_scaled)
> 
> # Ergebnisse anzeigen
> print("Median Überlebenszeiten (Predicted):")
> print(predicted_medians)
> 
> # Modellleistung bewerten (C-Index)
> c_index = rsf.score(X_test_scaled, y_test)
> print(f"C-Index: {c_index}")
> 
> from sksurv.metrics import integrated_brier_score
> survs = rsf.predict_survival_function(X_test_scaled)
> times = np.arange(1, 70)  
> 
> # Survival-Wahrscheinlichkeiten für die definierten Zeitpunkte berechnen
> preds = np.asarray([[fn(t) for t in times] for fn in survs])
> 
> print(f"Shape der Survival-Wahrscheinlichkeiten: {preds.shape}")
> 
> ibs = integrated_brier_score(
>     y_train,  # Trainingsdaten
>     y_test,   # Testdaten
>     preds,    # Vorhergesagte Überlebenswahrscheinlichkeiten
>     times     # Zeitpunkte
> )
> 
> # Ergebnis ausgeben
> print(f"Integrated Brier Score (IBS): {ibs}")

[sebp]

In the sample code you provided, the biggest value in the times array is 69, but the error message says 76. Are you sure this is correct? Providing the full traceback would be helpful.

In general, the error indicates that the observed time points passed as y_train cover a smaller time interval than what times covers. You cannot evaluate predictions beyond the greatest time point in y_train.

[Montrey]

No matter how I try to limit the time, I always get the error that my value is above 76. I am at a loss.

print(f"Definierte Zeitpunkte: {times}")
Times: [ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69]

print(f"Shape der Survival-Wahrscheinlichkeiten: {np.array(preds).shape}")
Shape der Survival-Wahrscheinlichkeiten: (171844, 69)

Here is the full traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 75
     71 preds = np.asarray([[fn(t) for t in times] for fn in survs])
     73 print(f"Shape der Survival-Wahrscheinlichkeiten: {preds.shape}")
---> 75 ibs = integrated_brier_score(
     76     y_train,  # Trainingsdaten
     77     y_test,   # Testdaten
     78     preds,    # Vorhergesagte Überlebenswahrscheinlichkeiten
     79     times     # Zeitpunkte
     80 )
     82 # Ergebnis ausgeben
     83 print(f"Integrated Brier Score (IBS): {ibs}")

File ~/.local/lib/python3.10/site-packages/sksurv/metrics.py:737, in integrated_brier_score(survival_train, survival_test, estimate, times)
    640 """The Integrated Brier Score (IBS) provides an overall calculation of
    641 the model performance at all available times :math:`t_1 \\leq t \\leq t_\\text{max}`.
    642 
   (...)
    734        Statistics in Medicine, vol. 18, no. 17-18, pp. 2529–2545, 1999.
    735 """
    736 # Computing the brier scores
--> 737 times, brier_scores = brier_score(survival_train, survival_test, estimate, times)
    739 if times.shape[0] < 2:
    740     raise ValueError("At least two time points must be given")

File ~/.local/lib/python3.10/site-packages/sksurv/metrics.py:621, in brier_score(survival_train, survival_test, estimate, times)
    619 prob_cens_t[prob_cens_t == 0] = np.inf
    620 # calculate inverse probability of censoring weights at observed time point
--> 621 prob_cens_y = cens.predict_proba(test_time)
    622 prob_cens_y[prob_cens_y == 0] = np.inf
    624 # Calculating the brier scores at each time point

File ~/.local/lib/python3.10/site-packages/sksurv/nonparametric.py:510, in SurvivalFunctionEstimator.predict_proba(self, time, return_conf_int)
    508 extends = time > self.unique_time_[-1]
    509 if self.prob_[-1] > 0 and extends.any():
--> 510     raise ValueError(f"time must be smaller than largest observed time point: {self.unique_time_[-1]}")
    512 # beyond last time point is zero probability
    513 Shat = np.empty(time.shape, dtype=float)

ValueError: time must be smaller than largest observed time point: 76.0

[sebp]

Thanks for the traceback.

I think y_test contains time points that are larger than what y_train contains.

[Montrey]

The problem was the standard scaler, without the standard scaler it worked. The solution with StandardScaler was that I artificially reduced the maximum available time from 76 to 75.

rsf.fit(X_train_scaled, y_train)

y_pred_survival = rsf.predict_survival_function(X_test_scaled)

time_min = y_test["zyklus"].min() 
time_max = y_test["zyklus"].max() 

time_grid = np.linspace(time_min, time_max - 1, 100) 

print(f"Zeitbereich für time_grid: {time_min} bis {time_max - 1e-6}")
print(f"time_grid: {time_grid[:5]} ... {time_grid[-5:]}")

y_pred_survival_array = np.array([
    [fn(t) for t in time_grid] for fn in y_pred_survival
])

ibs = integrated_brier_score(y_train, y_test, y_pred_survival_array, time_grid)
print(f"Integrated Brier Score: {ibs:.4f}")

c_index = rsf.score(X_test_scaled, y_test)
print(f"C-Index: {c_index}")