Università degli Studi di Milano
Course: Natural Interaction
Student: Matteo Castagna
Registration number: 27366A
Email: [email protected]
Pyscanpath is a framework to easily load and test few scanpath prediction models and to compare their result to a ground truth through some common metrics for scanpath evaluation.
Models
Four models for scanpath prediction are available:
- Itti-Koch (original in MATLAB, here partially rewritten in python)
- Constrained Levy Exploration (CLE)
- DeepGazeIII
- IOR-ROI Recurrent Mixture Density Network
Metrics
Four metrics to compute distances or similarity between scanpaths are available:
- Euclidean distance
- Mannan distance
- Edit distance
- Time delay embedding
Edit distance and time delay embedding implementations are taken from FixaTons.
Datasets
Pyscanpath let test the models on the MIT1003 dataset, the CAT3000 dataset and the OSIE dataset which can be accessed via the datasets module. This functionality is implemented upon pysaliency which needs to be installed. The models can be tested on images and scanpaths, provided as numpy arrays, singularly too, as can be seen in demo.ipynb.
- Clone or download the folder with the code
- Download data.zip from here, extract the data folder and place it in
pyscanpath\models\Iorroi - IOR-ROI model to work requires Meta SAM. After installing SAM, download vit_h.pth from here, create a folder named checkpoint in
pyscanpath\models\Iorroiand place there the .pth file. - DeepgazeIII in order to work requires pysaliency
Examples of predicted scanpath and ground truth (stimuli from MIT1003)