Can we use as_cumulative_dynamic_auc_scorer for Bayes Search CV ?

[t8rohman]

Does anyone have any experience in modifying the as_cumulative_dynamic_auc_scorer as a custom scorer in BayesSearchCV? I have an alternative of using GridSearchCV, but it takes longer than I expected. I'm curious if BayesSearchCV would converge faster to look for the best hyperparameter. But I don't know how to modify this scorer to be able to be passed to BayesSearchCV. Here's the code that I have so far.

%%time

# define model
est_gbsa = GradientBoostingSurvivalAnalysis(
    loss='coxph', 
    random_state=42, 
    n_iter_no_change=10, 
    validation_fraction=0.2
)

# define search space
search_spaces = {
    "learning_rate": Real(0.01, 0.2, prior='log-uniform'),    
    "subsample": Real(0.8, 1.0),                              
    "max_depth": Integer(3, 7),                               
    "min_samples_split": Integer(2, 10),                      
    "min_samples_leaf": Integer(1, 10),                       
    "dropout_rate": Categorical([0.0, 0.2, 0.5])          
}

# cross-validation strategy
cv = StratifiedGroupKFold(n_splits=3, shuffle=True, random_state=42)

# bayesian search
bayes_cv = BayesSearchCV(
    estimator=est_gbsa,
    search_spaces=search_spaces,
    n_iter=30,             
    cv=cv,
    n_jobs=4,
    verbose=2,
    random_state=42,
    scoring=as_cumulative_dynamic_auc_scorer(est_gbsa, times=y_times)  # here's the part where i'm confused
)

# fit the model
bayes_cv.fit(X_tune.loc[:, X_tune.columns != 'customer_id'], y_tune, groups=X_tune['customer_id'])

Can anyone help? Thanks a lot!

[sebp]

It should work when you pass as_cumulative_dynamic_auc_scorer(est_gbsa, times=y_times) as estimator.

[t8rohman]

Great, now it works! I made changes as you instructed, and now it works! Thank you very much for creating this library!

Here's the code later for anyone's reference.

%%time

# define model
est_gbsa = GradientBoostingSurvivalAnalysis(
    loss='coxph', 
    random_state=42, 
    n_iter_no_change=10, 
    validation_fraction=0.2
)

# define search space
search_spaces = {
    "estimator__learning_rate": Real(0.01, 0.2, prior='log-uniform'),    # Continuous range for learning rate
    "estimator__subsample": Real(0.8, 1.0),                              # Continuous range for subsample
    "estimator__max_depth": Integer(3, 7),                               # Integer range for max depth
    "estimator__min_samples_split": Integer(2, 10),                      # Integer range for min samples split
    "estimator__min_samples_leaf": Integer(1, 10),                       # Integer range for min samples leaf
    "estimator__dropout_rate": Categorical([0.0, 0.2, 0.5])              # Fixed values for dropout
}

# cross-validation strategy
cv = StratifiedGroupKFold(n_splits=3, shuffle=True, random_state=42)

# bayesian search
bayes_cv = BayesSearchCV(
    estimator=as_cumulative_dynamic_auc_scorer(est_gbsa, times=y_times),
    search_spaces=search_spaces,
    n_iter=30,              # Limit the number of iterations
    cv=cv,
    n_jobs=4,
    verbose=2,
    random_state=42,
)

bayes_cv.fit(X_tune.loc[:, X_tune.columns != 'customer_id'], y_tune, groups=X_tune['customer_id'])
print("Best parameters:", bayes_cv.best_params_)