Implement Exploratory Landscape Analysis (ELA) with Information Conte…

…nt (IC) (#647)

docs/tutorials/qml/ml_tools/data_and_config.md

@@ -54,7 +54,7 @@ The [`TrainConfig`][qadence.ml_tools.config.TrainConfig] tells [`Trainer`][qaden
 It is also possible to provide custom callback functions by instantiating a [`Callback`][qadence.ml_tools.callbacks.Callback]
 with a function `callback`.
 
-For example of how to use the TrainConfig with `Trainer`, please see [Examples in Trainer](/trainer.md)
+For example of how to use the TrainConfig with `Trainer`, please see [Examples in Trainer](../trainer)
 
 
 ### 2.1 Explanation of `TrainConfig` Attributes
@@ -172,7 +172,7 @@ def validation_criterion(val_loss: float, best_val_loss: float, val_epsilon: flo
 #### Custom Callbacks
 
 `TrainConfig` supports custom callbacks that can be triggered at specific stages of training. The `callbacks` attribute accepts a list of callback instances, which allow for custom behaviors like early stopping or additional logging.
-See [Callbacks](/callbacks.md) for more details.
+See [Callbacks](../callbacks) for more details.
 
 - `callbacks` (**list[Callback]**): List of custom callbacks to execute during training.
 

docs/tutorials/qml/ml_tools/trainer.md

@@ -469,7 +469,93 @@ for i in range(n_epochs):
 
 
 
-### 6.4. Custom `train` loop
+### 6.4. Performing pre-training Exploratory Landscape Analysis (ELA) with Information Content (IC)
+
+Before one embarks on training a model, one may wish to analyze the loss landscape to judge the trainability and catch vanishing gradient issues early.
+One way of doing this is made possible via calculating the [Information Content of the loss landscape](https://www.nature.com/articles/s41534-024-00819-8).
+This is done by discretizing the gradient in the loss landscapes and then calculating the information content therein.
+This serves as a measure of flatness or ruggedness of the loss landscape.
+Quantitatively, the information content allows us to get bounds on the average norm of the gradient in the loss landscape.
+
+Using the information content technique, we can get two types of bounds on the average of the norm of the gradient.
+1. The bounds as achieved in the maximum Information Content regime: Gives us a lower and upper bound on the average norm of the gradient in case high Information Content is achieved.
+2. The bounds as achieved in the sensitivity regime: Gives us an upper bound on the average norm of the gradient corresponding to the sensitivity IC achieved.
+
+Thus, we get 3 bounds. The upper and lower bounds for the maximum IC and the upper bound for the sensitivity IC.
+
+The `Trainer` class provides a method to calculate these gradient norms.
+
+```python exec="on" source="material-block" html="1"
+import torch
+from torch.optim.adam import Adam
+
+from qadence.constructors import ObservableConfig
+from qadence.ml_tools.config import AnsatzConfig, FeatureMapConfig, TrainConfig
+from qadence.ml_tools.data import to_dataloader
+from qadence.ml_tools.models import QNN
+from qadence.ml_tools.optimize_step import optimize_step
+from qadence.ml_tools.trainer import Trainer
+from qadence.operations.primitive import Z
+
+fm_config = FeatureMapConfig(num_features=1)
+ansatz_config = AnsatzConfig(depth=4)
+obs_config = ObservableConfig(detuning=Z)
+
+qnn = QNN.from_configs(
+    register=4,
+    obs_config=obs_config,
+    fm_config=fm_config,
+    ansatz_config=ansatz_config,
+)
+
+optimizer = Adam(qnn.parameters(), lr=0.001)
+
+batch_size = 25
+x = torch.linspace(0, 1, 32).reshape(-1, 1)
+y = torch.sin(x)
+train_loader = to_dataloader(x, y, batch_size=batch_size, infinite=True)
+
+train_config = TrainConfig(max_iter=100)
+
+trainer = Trainer(
+    model=qnn,
+    optimizer=optimizer,
+    config=train_config,
+    loss_fn="mse",
+    train_dataloader=train_loader,
+    optimize_step=optimize_step,
+)
+
+# Perform exploratory landscape analysis with Information Content
+ic_sensitivity_threshold = 1e-4
+epsilons = torch.logspace(-2, 2, 10)
+
+max_ic_lower_bound, max_ic_upper_bound, sensitivity_ic_upper_bound = (
+    trainer.get_ic_grad_bounds(
+        eta=ic_sensitivity_threshold,
+        epsilons=epsilons,
+    )
+)
+
+print(
+    f"Using maximum IC, the gradients are bound between {max_ic_lower_bound:.3f} and {max_ic_upper_bound:.3f}\n"
+)
+print(
+    f"Using sensitivity IC, the gradients are bounded above by {sensitivity_ic_upper_bound:.3f}"
+)
+
+# Resume training as usual...
+
+trainer.fit(train_loader)
+```
+
+The `get_ic_grad_bounds` function returns a tuple containing a tuple containing the lower bound as achieved in maximum IC case, upper bound as achieved in maximum IC case, and the upper bound for the sensitivity IC case.
+
+The sensitivity IC bound is guaranteed to appear, while the usually much tighter bounds that we get via the maximum IC case is only meaningful in the case of the maximum achieved information content $H(\epsilon)_{max} \geq log_6(2)$.
+
+
+
+### 6.5. Custom `train` loop
 
 If you need custom training functionality that goes beyond what is available in
 `qadence.ml_tools.Trainer` you can write your own
@@ -546,6 +632,6 @@ def train(
     return model, optimizer
 ```
 
-### 6.5. Gradient-free optimization using `Trainer`
+### 6.6. Gradient-free optimization using `Trainer`
 
 We can achieve gradient free optimization with `Trainer.set_use_grad(False)` or `trainer.disable_grad_opt(ng_optimizer)`. An example solving a QUBO using gradient free optimization based on `Nevergrad` optimizers and `Trainer` is shown in the [analog QUBO Tutorial](../../digital_analog_qc/analog-qubo.md).

qadence/ml_tools/__init__.py

@@ -4,6 +4,7 @@
 from .config import AnsatzConfig, FeatureMapConfig, TrainConfig
 from .constructors import create_ansatz, create_fm_blocks, observable_from_config
 from .data import DictDataLoader, InfiniteTensorDataset, OptimizeResult, to_dataloader
+from .information import InformationContent
 from .models import QNN
 from .optimize_step import optimize_step as default_optimize_step
 from .parameters import get_parameters, num_parameters, set_parameters

qadence/ml_tools/information/__init__.py

@@ -0,0 +1,3 @@
+from __future__ import annotations
+
+from .information_content import InformationContent