How to use heuristic score and custom model

[dmetivie]

I have a heuristic notion of uncertainty associated with my prediction model. I want to compute the score accordingly. Could you provide e.g. for a regression task how to use a score.
In the doc I did not find an example. In the scr/ code it seems to be possible.
Thanks

[pat-alt]

Hi! I'm currently on vacation, back in two weeks. This is definitely possible, because the heuristic is just passed on as a function. I'll add an example here when I'm back or perhaps @MojiFarmanbar @Rockdeldiablo can already help out in the meantime.

Thanks!

[pasq-cat]

I have a heuristic notion of uncertainty associated with my prediction model. I want to compute the score accordingly. Could you provide e.g. for a regression task how to use a score. In the doc I did not find an example. In the scr/ code it seems to be possible. Thanks

i think you have to implement your own method and change the heuristic function there. you can use https://github.com/JuliaTrustworthyAI/ConformalPrediction.jl/blob/main/src/conformal_models/inductive_regression.jl as an example.
after you have created your method you have to add it in the list of supported methods here.
https://github.com/JuliaTrustworthyAI/ConformalPrediction.jl/blob/main/src/conformal_models/conformal_models.jl

@pat-alt maybe we can change the function conformal_model so that in method we pass directly the structs instead of the dict keys? in this way he would need only to pass SimpleInductiveRegressor[heuristic= fun] instead of having to rewrite the whole thing.

@pat-alt btw i am stuck with a problem with the fitresult function. i will try to solve it tomorrow with @MojiFarmanbar.

[MojiFarmanbar]

@dmetivie , did you get your answer?
apart from what @Rockdeldiablo mentioned, you can define your own heuristics here
https://github.com/JuliaTrustworthyAI/ConformalPrediction.jl/blob/main/src/conformal_models/heuristics.jl