Merge pull request #13 from funkelab/dev

Documentation update

Author: adjavon

.github/workflows/tests.yaml

@@ -22,4 +22,4 @@ jobs:
         pip install ".[dev]"
     - name: Test with pytest
       run: |
-        pytest tests
+        pytest tests

.gitignore

@@ -4,3 +4,7 @@
 **/wandb
 docs/build
 docs/_build
+
+# pixi environments
+.pixi
+*.egg-info

README.md

@@ -2,7 +2,9 @@
 
 [![tests](https://github.com/funkelab/quac/actions/workflows/tests.yaml/badge.svg)](https://github.com/funkelab/quac/actions/workflows/tests.yaml)
 [![documentation](https://github.com/funkelab/quac/actions/workflows/deploy-docs.yaml/badge.svg)](https://github.com/funkelab/quac/actions/workflows/deploy-docs.yaml)
+[![DOI:10.1101/2024.11.26.625505](http://img.shields.io/badge/DOI-10.1101/2024.11.26.625505-B31B1B.svg)](https://doi.org/10.1101/2024.11.26.625505)
 
+Pre-print can be found on [bioRxiv](https://www.biorxiv.org/content/10.1101/2021.01.08.425840v1)
 Documentation can be found [here](https://funkelab.github.io/quac/).
 
 <img src="docs/source/assets/overview.png" />

docs/source/assets/fictus.png

@@ -35,6 +35,7 @@
     "sphinx_togglebutton",
     "sphinxcontrib.jquery",
     "sphinx.ext.intersphinx",
+    "sphinxcontrib.email",
 ]
 
 templates_path = ["_templates"]
@@ -63,3 +64,6 @@
     "networkx": ("https://networkx.org/documentation/stable/", None),
     "numpy": ("https://numpy.org/doc/stable/", None),
 }
+
+# Email
+email_automode = True

docs/source/assets/synapses.png

@@ -0,0 +1,49 @@
+===============
+Example gallery
+===============
+
+Here are some examples of datasets that have been investigated with QuAC, as well as what was found.
+If you have tried QuAC on your own dataset (even unsuccessfully!) please :email:`let us know <[email protected]>` so we can add it to the gallery.
+
+
+*Fictus Aggregatum* synthetic cells
+====================================
+
+The *Fictus aggregatum* dataset is a synthetic dataset that was created in the `Funke Lab <https://www.janelia.org/lab/funke-lab>`_ specifically to understand how QuAC works on biological data.
+The code to generate this dataset is available `here <https://github.com/funkelab/fictus.aggregatum>`_.
+
+.. figure:: assets/fictus.png
+    :figwidth: 100%
+    :alt: Fictus aggregatum
+    :align: center
+
+    An example of the query (top) and counterfactual (bottom) images, highlighting the differences.
+
+We evaluated, using this *fictus* dataset, whether QuAC was able to retrieve all of the differences between classes when these are known.
+We found that, although this was sometimes done in surprising ways, the changes described by QuAC were generally in line with what was expected from the data.
+
+
+*Drosophila melanogaster* synapses
+==================================
+
+The differences between synapses emitting different neurotransmitters in the fruit fly *Drosophila melanogaster* are so subtle that it was not though possible to tell them apart.
+When `it was found that a deep learning model could do so <https://www.cell.com/cell/fulltext/S0092-8674(24)00307-6>`_, however, this opened up possibilities for gaining insight into the relation between strcuture and function in these synapses.
+
+.. figure:: assets/synapses.png
+    :figwidth: 100%
+    :alt: Synapses in EM
+    :align: center
+
+    A few examples of synapses in the *Drosophila* brain, as seen in electron microscopy, translated from one class to another.
+
+QuAC explanations suggested quite a few new features that could be used to distinguish between these synapse types, the prevalence of which is currently being investigated.
+
+
+Fly identity
+============
+
+.. |simon says| image:: assets/quac.png
+    :width: 50
+
+.. attention::
+    This dataset is still under construction. Come back soon for updates! |simon says|

docs/source/conf.py

@@ -3,10 +3,26 @@
 Quantitative Attributions with Counterfactuals
 ==============================================
 
-Description of QuAC
+.. attention::
+  Check out the new pre-print on biorxiv: `Quantitative Attributions with Counterfactuals <https://doi.org/10.1101/2024.11.26.625505>`_!
+
+
+.. image:: assets/overview.png
+  :width: 800
+  :align: center
+
+QuAC is a tool for understanding the sometimes subtle differences between classes of images, by questioning a classifier's decisions.
+The method generates counterfactual images: making slight changes in an image to change its clasification.
+The counterfactuals in QuAC have localized changes, and are quantitatively scored to determine how well they describe the classifier's decision.
+
+Get started with :doc:`installation </install>`, then check out the :doc:`tutorials <tutorials>` to run each step of the QuAC pipeline on an example dataset.
+
+Unsure whether QuAC is right for your dataset? Check out the :doc:`examples </examples>` to see how it has been used already!
+
 
 .. toctree::
   :maxdepth: 2
 
   install
-  tutorials
+  examples
+  Tutorials <tutorials>

docs/source/examples.rst

@@ -1,13 +1,18 @@
+.. _install:
+
+============
 Installation
 ============
 
 Installing:
+
 1. Clone this repository
 2. Create a `conda` environment with `python, pytorch, torchvision`; I recommend `mamba`
 3. Activate your new environment (`mamba activate ...`)
 4. Change into the directory holding this repository.
 5. `pip install .`
 
 Installing as developper:
-1. - 4. Same as above.
-5. `pip install -e .\[dev\]`
+
+1. Do items 1 through 4 the same as above.
+2. `pip install -e .\[dev\]`

docs/source/index.rst

@@ -1,10 +1,60 @@
-Tutorials
+================================
+So you've decided to use QuAC...
+================================
+
+Great! In these tutorials we'll go through the process of setting up QuAC for your images.
+We built QuAC with biological images in mind, so those will be our analogies here. However, you're very welcome to use QuAC with any kind of image data!
+If you're interested in using it with non-image data, please :email:`contact us <[email protected]>`.
+
+But first, a quick overview of the method.
+
+What even is QuAC?
 ==================
+QuAC, or Quantitative Attributions with Counterfactuals, is a method for generating and scoring visual counterfactual explanations of an image clasifier.
+Let's assume, for instance, that you have images of cells grown in two different conditions.
+To your eye, the phenotypic difference between the two conditions is hidden within the cell-to-cell variability of the dataset, but you know it is there because you've trained a classifier to differentiate the two conditions and it works. So how do you pull out the differences?
+
+We begin by training a generative neural network to convert your images from one class to another. Here, we'll use a StarGAN. This allows us to go from our real, **query** image, to our **generated** image.
+Using information learned from **reference** images, the StarGAN is trained in such a way that the **generated** image will have a different class!
+
+While very powerful, these generative networks *can potentially* make some changes that are not necessary to the classification.
+In our example below, the **generated** image's membrane has been unnecessarily changed.
+We use Discriminative Attribution methods to generate a set of candidate attribution masks.
+Among these, we are looking for the smallest mask that has the greatest change in the classification output.
+By taking only the changes *within* that mask, we create the counterfactual image.
+It is as close as possible to the original image, with only the necessary changes to turn its class!
+
+.. image:: assets/overview.png
+    :width: 800
+    :align: center
+
+Before you begin, download [the data]() and [the pre-trained models]() for an example.
+Then, make sure you've installed QuAC by following the :doc:`Installation guide <install>`.
 
 
+The conversion network
+===============================
+
+You have two options for training the StarGAN, you can either :doc:`define parameters directly in Python <tutorials/train>` or :doc:`train it using a YAML file <tutorials/train_yaml>`.
+We recommend the latter, which will make it easier to keep track of your experiments!
+Once you've trained a decent model, generate a set of images using the :doc:`image generation tutorial <tutorials/generate>` before moving on to the next steps.
+
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
 
     tutorials/train
-    tutorials/attribute
-    tutorials/generate
+    tutorials/train_yaml
+    Generating images <tutorials/generate>
+
+Attribution and evaluation
+==========================
+
+With the generated images in hand, we can now run the attribution and evaluation steps.
+These two steps allow us to overcome the limitations of the generative network to create *truly* minimal counterfactual images, and to score the query-counterfactual pairs based on how well they explain the classifier.
+
+.. toctree::
+    :maxdepth: 1
+
+    Attribution <tutorials/attribute>
+    Evaluation <tutorials/evaluate>
+    Visualizing results <tutorials/visualize>

docs/source/install.rst

@@ -1,93 +1,97 @@
 .. _sec_attribute:
 
-================================================
-Attribution and evaluation given counterfactuals
-================================================
+===============================================
+Discriminative attribution from Counterfactuals
+===============================================
 
-Attribution
-===========
+Now that we have generated counterfactuals, we will refine our **generated** images into **counterfactuals** using discriminative attribution.
+Remember that although the conversion network is trained to keep as much of the image fixed as possible, it is not perfect.
+This means that there may still be regions of the **generated** image that differ from the **query** image *even if they don't need to*.
+Luckily, we have a classifier that can help us identify and keep only the necessary regions of change.
+
+The first thing that we want to do is load the classifier.
 
 .. code-block:: python
     :linenos:
 
-    # Load the classifier
+    classifier_checkpoint = "path/to/classifier/checkpoint"
+
     from quac.generate import load_classifier
     classifier = load_classifier(
-
+        checkpoint_path=classifier_checkpoint
     )
 
+Next, we will define the attribution that we want to use.
+In this tutorial, we will use Discriminative Integrated Gradients, using the classifier as a baseline.
+As a comparison, we will also use Vanilla Integrated Gradients, which uses a black image as a baseline.
+This will allow us to identify the regions of the image that are most important for the classifier to make its decision.
+Later in the :doc:`evaluation <evaluate>` tutorial, we will process these attributions into masks, and finally get our counterfactuals.
+
+
+.. code-block:: python
+    :linenos:
+
+    # Parameters
+    attribution_directory = "path/to/store/attributions"
+
     # Defining attributions
     from quac.attribution import (
-        DDeepLift,
         DIntegratedGradients,
+        VanillaIntegratedGradients,
         AttributionIO
     )
     from torchvision import transforms
 
     attributor = AttributionIO(
         attributions = {
-            "deeplift" : DDeepLift(),
-            "ig" : DIntegratedGradients()
+            "discriminative_ig" : DIntegratedGradients(classifier),
+            "vanilla_ig" : VanillaIntegratedGradients(classifier)
         },
-        output_directory = "my_attributions_directory"
+        output_directory = atttribution_directory
     )
 
+
+Finally, we want to make sure that the images are processed as we would like for the classifier.
+Here, we will simply define a set of `torchvision` transforms to do this, we will pass them to the `attributor` object.
+Keep in mind that if you processed your data in a certain way when training your classfier, you will need to use the same processing here.
+
+.. code-block:: python
+    :linenos:
+
     transform = transforms.Compose(
         [
-            transforms.Resize(224),
-            transforms.CenterCrop(224),
-            transforms.Normalize(...)
+            transforms.ToTensor(),
+            transforms.Grayscale(),
+            transforms.Resize(128),
+            transforms.Normalize(0.5, 0.5),
         ]
     )
 
-    # This will run attributions and store all of the results in the output_directory
-    # Shows a progress bar
-    attributor.run(
-        source_directory="my_source_image_directory",
-        counterfactual_directory="my_counterfactual_image_directory",
-        transform=transform
-    )
-
-Evaluation
-==========
-
-Once you have attributions, you can run evaluations.
-You may want to try different methods for thresholding and smoothing the attributions to get masks.
-
-
-In this example, we evaluate the results from the DeepLift attribution method.
+Finally, let's run the attributions.
 
 .. code-block:: python
     :linenos:
 
-    # Defining processors and evaluators
-    from quac.evaluation import Processor, Evaluator
-    from sklearn.metrics import ConfusionMatrixDisplay
+    data_directory = "path/to/data/directory"
+    counterfactual_directory = "path/to/counterfactual/directory"
 
-    classifier = load_classifier(...)
-
-    evaluator = Evaluator(
-        classifier,
-        source_directory="my_source_image_directory",
-        counterfactual_directory="my_counterfactual_image_directory",
-        attribution_directory="my_attributions_directory/deeplift",
+    # This will run attributions and store all of the results in the output_directory
+    # Shows a progress bar
+    attributor.run(
+        source_directory=data_directory,
+        counterfactual_directory=counterfactual_directory,
         transform=transform
     )
 
+If you look into the `attribution_directory`, you should see a set of attributions.
+They will be organized in the following way:
 
-    cf_confusion_matrix = evaluator.classification_report(
-                            data="counterfactuals",  # this is the default
-                            return_classifications=False,
-                            print_report=True,
-                        )
+.. code-block:: bash
 
-    # Plot the confusion matrix
-    disp = ConfusionMatrixDisplay(
-        confusion_matrix=cf_confusion_matrix,
-    )
-    disp.show()
+    attribution_directory/
+        attribution_method_name/
+            source_class/
+                target_class/
+                    image_name.npy
 
-    # Run QuAC evaluation on your attribution and store a report
-    report = evaluator.quantify(processor=Processor())
-    # The report will be stored based on the processor's name, which is "default" by default
-    report.store("my_attributions_directory/deeplift/reports")
+In the next tutorial, we will use these attributions to generate masks and finally get our counterfactuals.