|
| 1 | +# Configuring nx-parallel |
| 2 | + |
| 3 | +`nx-parallel` provides flexible parallel computing capabilities, allowing you to control settings like `backend`, `n_jobs`, `verbose`, and more. This can be done through two configuration systems: `joblib` and `NetworkX`. This guide explains how to configure `nx-parallel` using both systems. |
| 4 | + |
| 5 | +## 1. Setting configs using `joblib.parallel_config` |
| 6 | + |
| 7 | +`nx-parallel` relies on [`joblib.Parallel`](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html) for parallel computing. You can adjust its settings through the [`joblib.parallel_config`](https://joblib.readthedocs.io/en/latest/generated/joblib.parallel_config.html) class provided by `joblib`. For more details, check out the official [joblib documentation](https://joblib.readthedocs.io/en/latest/parallel.html). |
| 8 | + |
| 9 | +### 1.1 Usage |
| 10 | + |
| 11 | +```python |
| 12 | +from joblib import parallel_config |
| 13 | + |
| 14 | +# Setting global configs |
| 15 | +parallel_config(n_jobs=3, verbose=50) |
| 16 | +nx.square_clustering(H) |
| 17 | + |
| 18 | +# Setting configs in a context |
| 19 | +with parallel_config(n_jobs=7, verbose=0): |
| 20 | + nx.square_clustering(H) |
| 21 | +``` |
| 22 | + |
| 23 | +Please refer the [official joblib's documentation](https://joblib.readthedocs.io/en/latest/generated/joblib.parallel_config.html) to better understand the config parameters. |
| 24 | + |
| 25 | +Note: Ensure that `nx.config.backends.parallel.active = False` when using `joblib` for configuration, as NetworkX configurations will override `joblib.parallel_config` settings if `active` is `True`. |
| 26 | + |
| 27 | +## 2. Setting configs using `networkx`'s configuration system for backends |
| 28 | + |
| 29 | +To use NetworkX’s configuration system in `nx-parallel`, you must set the `active` flag (in `nx.config.backends.parallel`) to `True`. |
| 30 | + |
| 31 | +### 2.1 Configs in NetworkX for backends |
| 32 | + |
| 33 | +When you import NetworkX, it automatically sets default configurations for all installed backends, including `nx-parallel`. |
| 34 | + |
| 35 | +```python |
| 36 | +import networkx as nx |
| 37 | + |
| 38 | +print(nx.config) |
| 39 | +``` |
| 40 | + |
| 41 | +Output: |
| 42 | + |
| 43 | +``` |
| 44 | +NetworkXConfig( |
| 45 | + backend_priority=[], |
| 46 | + backends=Config( |
| 47 | + parallel=ParallelConfig( |
| 48 | + active=False, |
| 49 | + backend="loky", |
| 50 | + n_jobs=None, |
| 51 | + verbose=0, |
| 52 | + temp_folder=None, |
| 53 | + max_nbytes="1M", |
| 54 | + mmap_mode="r", |
| 55 | + prefer=None, |
| 56 | + require=None, |
| 57 | + inner_max_num_threads=None, |
| 58 | + backend_params={}, |
| 59 | + ) |
| 60 | + ), |
| 61 | + cache_converted_graphs=True, |
| 62 | +) |
| 63 | +``` |
| 64 | + |
| 65 | +As you can see in the above output, by default, `active` is set to `False`. So, to enable NetworkX configurations for `nx-parallel`, set `active` to `True`. Please refer the [NetworkX's official backend and config docs](https://networkx.org/documentation/latest/reference/backends.html) for more on networkx configuration system. |
| 66 | + |
| 67 | +### 2.2 Usage |
| 68 | + |
| 69 | +```python |
| 70 | +# enabling networkx's config for nx-parallel |
| 71 | +nx.config.backends.parallel.active = True |
| 72 | + |
| 73 | +# Setting global configs |
| 74 | +nxp_config = nx.config.backends.parallel |
| 75 | +nxp_config.n_jobs = 3 |
| 76 | +nxp_config.verbose = 50 |
| 77 | + |
| 78 | +nx.square_clustering(H) |
| 79 | + |
| 80 | +# Setting config in a context |
| 81 | +with nxp_config(n_jobs=7, verbose=0): |
| 82 | + nx.square_clustering(H) |
| 83 | +``` |
| 84 | + |
| 85 | +The configuration parameters are the same as `joblib.parallel_config`, so you can refer to the [official joblib's documentation](https://joblib.readthedocs.io/en/latest/generated/joblib.parallel_config.html) to better understand these config parameters. |
| 86 | + |
| 87 | +### 2.3 How Does NetworkX's Configuration Work in nx-parallel? |
| 88 | + |
| 89 | +In `nx-parallel`, there's a `_configure_if_nx_active` decorator applied to all algorithms. This decorator checks the value of `active`(in `nx.config.backends.parallel`) and then accordingly uses the appropriate configuration system (`joblib` or `networkx`). If `active=True`, it extracts the configs from `nx.config.backends.parallel` and passes them in a `joblib.parallel_config` context manager and calls the function in this context. Otherwise, it simply calls the function. |
| 90 | + |
| 91 | +## 3. Comparing NetworkX and Joblib Configuration Systems |
| 92 | + |
| 93 | +### 3.1 Using Both Systems Simultaneously |
| 94 | + |
| 95 | +You can use both NetworkX’s configuration system and `joblib.parallel_config` together in `nx-parallel`. However, it’s important to understand their interaction. |
| 96 | + |
| 97 | +Example: |
| 98 | + |
| 99 | +```py |
| 100 | +# Enable NetworkX configuration |
| 101 | +nx.config.backends.parallel.active = True |
| 102 | +nx.config.backends.parallel.n_jobs = 6 |
| 103 | + |
| 104 | +# Global Joblib configuration |
| 105 | +joblib.parallel_config(backend="threading") |
| 106 | + |
| 107 | +with joblib.parallel_config(n_jobs=4, verbose=55): |
| 108 | + # NetworkX config for nx-parallel |
| 109 | + # backend="loky", n_jobs=6, verbose=0 |
| 110 | + nx.square_clustering(G, backend="parallel") |
| 111 | + |
| 112 | + # Joblib config for other parallel tasks |
| 113 | + # backend="threading", n_jobs=4, verbose=55 |
| 114 | + joblib.Parallel()(joblib.delayed(sqrt)(i**2) for i in range(10)) |
| 115 | +``` |
| 116 | + |
| 117 | +- **NetworkX Configurations for nx-parallel**: When calling functions within `nx-parallel`, NetworkX’s configurations will override those specified by Joblib. For example, the `nx.square_clustering` function will use the `n_jobs=6` setting from `nx.config.backends.parallel`, regardless of any Joblib settings within the same context. |
| 118 | + |
| 119 | +- **Joblib Configurations for Other Code**: For any other parallel code outside of `nx-parallel`, such as a direct call to `joblib.Parallel`, the configurations specified within the Joblib context will be applied. |
| 120 | + |
| 121 | +This behavior ensures that `nx-parallel` functions consistently use NetworkX’s settings when enabled, while still allowing Joblib configurations to apply to non-NetworkX parallel tasks. |
| 122 | + |
| 123 | +**Key Takeaway**: When both systems are used together, NetworkX's configuration (`nx.config.backends.parallel`) takes precedence for `nx-parallel` functions. To avoid unexpected behavior, ensure that the `active` setting aligns with your intended configuration system. |
| 124 | + |
| 125 | +### 3.2 Key Differences |
| 126 | + |
| 127 | +- **Parameter Handling**: The main difference is how `backend_params` are passed. Since, in networkx configurations are stored as a [`@dataclass`](https://docs.python.org/3/library/dataclasses.html), we need to pass them as a dictionary, whereas in `joblib.parallel_config` you can just pass them along with the other configurations, as shown below: |
| 128 | + |
| 129 | + ```py |
| 130 | + nx.config.backends.parallel.backend_params = {"max_nbytes": None} |
| 131 | + joblib.parallel_config(backend="loky", max_nbytes=None) |
| 132 | + ``` |
| 133 | + |
| 134 | +- **Default Behavior**: By default, `nx-parallel` looks for configs in `joblib.parallel_config` unless `nx.config.backends.parallel.active` is set to `True`. |
| 135 | + |
| 136 | +### 3.3 When Should You Use Which System? |
| 137 | + |
| 138 | +When the only networkx backend you're using is `nx-parallel`, then either of the NetworkX or `joblib` configuration systems can be used, depending on your preference. |
| 139 | + |
| 140 | +But, when working with multiple NetworkX backends, it's crucial to ensure compatibility among the backends to avoid conflicts between different configurations. In such cases, using NetworkX's configuration system to configure `nx-parallel` is recommended. This approach helps maintain consistency across backends. For example: |
| 141 | + |
| 142 | +```python |
| 143 | +nx.config.backend_priority = ["another_nx_backend", "parallel"] |
| 144 | +nx.config.backends.another_nx_backend.config_1 = "xyz" |
| 145 | +joblib.parallel_config(n_jobs=7, verbose=50) |
| 146 | + |
| 147 | +nx.square_clustering(G) |
| 148 | +``` |
| 149 | + |
| 150 | +In this example, if `another_nx_backend` also internally utilizes `joblib.Parallel` (without exposing it to the user) within its implementation of the `square_clustering` algorithm, then the `nx-parallel` configurations set by `joblib.parallel_config` will influence the internal `joblib.Parallel` used by `another_nx_backend`. To prevent unexpected behavior, it is advisable to configure these settings through the NetworkX configuration system. |
| 151 | + |
| 152 | +**Future Synchronization:** We are working on synchronizing both configuration systems so that changes in one system automatically reflect in the other. This started with [PR#68](https://github.com/networkx/nx-parallel/pull/68), which introduced a unified context manager for `nx-parallel`. For more details on the challenges of creating a compatibility layer to keep both systems in sync, refer to [Issue#76](https://github.com/networkx/nx-parallel/issues/76). |
| 153 | + |
| 154 | +If you have feedback or suggestions, feel free to open an issue or submit a pull request. |
| 155 | + |
| 156 | +Thank you :) |
0 commit comments