You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To run the experiments you need to update the script paths and install fire, pandas and tqdm
1
+
# Running Experiments Guide
2
+
3
+
To run the experiments you need to update the script paths and install `fire`, `pandas` and `tqdm`
2
4
3
5
## Model Checkpoints
4
6
5
-
Need checkpoints from https://github.com/facebookresearch/segment-anything
7
+
You'll need to obtain model checkpoints from the [facebookresearch/segment-anything](https://github.com/facebookresearch/segment-anything) repository. Use the following commands to download them:
Here's the folder structure you should set up for your experimental data:
26
+
27
+
```plaintext
28
+
experiments_data/
29
+
├── tmp/
30
+
│ ├── sam_coco_mask_center_cache/
31
+
│ ├── sam_eval_masks_out/
32
+
├── datasets/
33
+
│ ├── coco2017/
34
+
│ ├── val2017/
35
+
│ ├── annotations/
36
+
├── checkpoints/
28
37
```
29
-
## Environment details
38
+
39
+
## Environment Details
30
40
31
41
### Hardware
32
-
These experiments were run on an Amazon p4d.24xlarge instance. See the Product details of the EC2 website for the exact details. A few key highlights are
42
+
43
+
These experiments were conducted on an Amazon `p4d.24xlarge` instance with the following specifications:
33
44
34
45
- 8 A100 GPUs with 40960MiB running at 400W
35
46
- 96 vCPUs
36
47
- 1152 GiB of RAM
37
-
- Software
38
48
49
+
### Software Versions
39
50
40
-
### Versions
51
+
-**PyTorch**: Utilize the latest nightly build of PyTorch.
52
+
-**Python**: Version 3.10.
53
+
-**Segment-Anything Repositories**:
54
+
-**Main Repository**: Use the original [facebookresearch/segment-anything](https://github.com/facebookresearch/segment-anything) for foundational code and functionalities.
55
+
-**Fork with Enhancements**: Access the fork at [cpuhrsch/segment-anything](https://github.com/cpuhrsch/segment-anything). This fork includes additional commits essential for replicating the baseline and initial experiments.
56
+
-**Optimized Version**: Explore the optimized code at [pytorch-labs/segment-anything-fast](https://github.com/pytorch-labs/segment-anything-fast). This repository offers performance improvements and is crucial for later experiments.
41
57
42
-
- PyTorch nightly and Python 3.10
43
-
-https://github.com/cpuhrsch/segment-anything fork of https://github.com/facebookresearch/segment-anything with additional commits if you want to reproduce baseline and first few experiments
44
-
- This https://github.com/pytorch-labs/segment-anything-fast
If you plan to run the scripts that run the experiments from segment-anything-fast it is important to install the segment-anything fork in editable mode so that the script can switch between different commits of the fork automatically.
79
+
If you intend to run scripts from segment-anything-fast, install the segment-anything fork in editable mode to allow switching between different commits of the fork automatically.
If at any point you run into issue, please note that you can increase verbosity by adding `--capture_output False` to above command. Also, please don't hesitate to open an issue.
73
88
89
+
If you encounter any issues, add `--capture_output False` to increase verbosity, and feel free to open an issue.
74
90
75
91
### Data
76
-
We are using the COCO2017 Validation (Val images) dataset. We use this dataset to serve as a somewhat realistic distribution of input images and aim to measure a) accuracy and b) performance.
77
-
Measurement
78
-
Accuracy
79
-
Our main goal is to verify that our performance optimizations do not degrade the accuracy of the model. We do not aim to reproduce any paper results or aim to make statements about the accuracy of this model on the dataset. This measurement serves as an additional integration test in conjunction with numerous unit and other separate integration tests.
80
92
81
-
We calculate the center points of the mask annotations using a rudimentary version of https://arxiv.org/pdf/2304.02643.pdf, section D.1.Point Sampling ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/data.py#L10-L120)). These center points serve as annotations per image. Note that the number of masks and thus number of annotations per image vary.
93
+
We utilize the COCO2017 Validation (Val images) dataset for these experiments. It provides a realistic distribution of input images for measuring accuracy and performance.
82
94
83
-
These images and annotations are given to the predict_torch method of an instance of SamPredictor to predict masks. These are then compared to the ground truth masks using the Intersection over Union (IoU) metric ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/metrics.py#L4-L22)). We calculate the mean IoU (mIoU) metric over the entire 5000 images of the validation dataset to track accuracy.
84
-
Performance
85
-
Our goal is to measure the runtime of PyTorch models. We purposefully exclude data movements or calculation of the metrics. Specifically we measure the execution time on the GPU of running the image encoder (e.g. vit_h) and SamPredictor.predict_torch ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/eval_combo.py#L127-L165), [code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/eval_combo.py#L68-L99)).
95
+
### Measurement
86
96
87
-
Each experiment is run in a separate Python process created from scratch. We run three batches of warmup before each experiment. This also implies that we are excluding compilation time from benchmarking.
97
+
#### Accuracy
88
98
89
-
We measure the execution time and calculate the number of images that can be processed per image (img/s). We also measure the maximum amount of memory allocated at the end of the process using torch.cuda.max_memory_allocated.
90
-
Tracing
99
+
Our primary goal is to ensure that performance optimizations do not compromise model accuracy. We do not aim to replicate paper results or make claims about model accuracy on the dataset. This measurement serves as an integration test alongside unit and other integration tests.
100
+
We calculate mask annotation center points using a simplified version of [this method](https://arxiv.org/pdf/2304.02643.pdf), section D.1.Point Sampling ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/data.py#L10-L120)). These points serve as annotations per image, and the number of masks and annotations per image can vary.
101
+
These images and annotations are provided to the `predict_torch` method of a `SamPredictor` instance for mask prediction. The predictions are then compared to ground truth masks using the Intersection over Union (IoU) metric ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/metrics.py#L4-L22)). We calculate the mean IoU (mIoU) metric over the entire 5000 images of the validation dataset to track accuracy.
91
102
92
-
We collect kernel and memory traces using PyTorch native tooling and analyze it with [Perfetto UI](https://perfetto.dev/). When collecting these traces and profiles we typically only limit us to a few batches. Otherwise the files can become very large and difficult to load.
103
+
#### Performance
93
104
94
-
### Kernel traces
105
+
Our objective is to measure the runtime of PyTorch models. We intentionally exclude data movement or metric calculation from measurements. Specifically, we measure the GPU execution time of running the image encoder (e.g., `vit_h`) and `SamPredictor.predict_torch` ([code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/eval_combo.py#L127-L165), [code](https://github.com/pytorch-labs/segment-anything-fast/blob/67d5c894569e99b9fdba55cfcf2f724be9f68994/experiments/eval_combo.py#L68-L99)).
106
+
Each experiment runs in a separate Python process created from scratch. We execute three batches of warm-up before each experiment. This also means that we exclude compilation time from benchmarking.
107
+
We measure the execution time and calculate the number of images processed per second (img/s). We also measure the maximum amount of memory allocated at the end of the process using `torch.cuda.max_memory_allocated`.
95
108
96
-
One can write a simple wrapper that runs a function under the tracer context and writes out the result to a compressed json file. The resulting chrome trace can then be analyzed with Perfetto UI.
109
+
#### Tracing
97
110
98
-
```
99
-
def profiler_runner(path, fn, *args, **kwargs):
100
-
with torch.profiler.profile(
101
-
activities=[torch.profiler.ProfilerActivity.CPU,
102
-
torch.profiler.ProfilerActivity.CUDA],
103
-
record_shapes=True) as prof:
104
-
result = fn(*args, **kwargs)
105
-
prof.export_chrome_trace(path)
106
-
return result
107
-
```
111
+
We collect kernel and memory traces using PyTorch native tooling and analyze them with [Perfetto UI](https://perfetto.dev/). We typically limit the collection to a few batches to avoid generating excessively large files.
108
112
109
-
It can be very useful to annotate certain regions in these traces to map (pieces of) the code to the overall traces. For this we frequently use record_function. Consider the following as an example.
113
+
##### Kernel Traces
110
114
111
-
```
112
-
with torch.autograd.profiler.record_function("timed region"):
113
-
with torch.autograd.profiler.record_function("image encoder"):
with torch.autograd.profiler.record_function("nt predict_torch"):
118
-
predictor.reset_image()
119
-
[...]
115
+
You can create a simple wrapper to run a function under the tracer context and write the result to a compressed JSON file. The resulting Chrome trace can be analyzed with Perfetto UI. Here's an example:
116
+
117
+
```python
118
+
defprofiler_runner(path, fn, *args, **kwargs):
119
+
with torch.profiler.profile(
120
+
activities=[torch.profiler.ProfilerActivity.CPU,
121
+
torch.profiler.ProfilerActivity.CUDA],
122
+
record_shapes=True) as prof:
123
+
result = fn(*args, **kwargs)
124
+
prof.export_chrome_trace(path)
125
+
return result
120
126
```
121
127
122
-
### Memory profiles
128
+
It's useful to annotate specific regions in these traces to map code segments to the overall traces. For this, we frequently use `record_function`. See the example in the provided code.
123
129
124
-
We record the memory history and use memory_viz.py to convert the result into a human readable html file.
130
+
##### Memory Profiles
125
131
126
-
```
132
+
We record memory history and use `memory_viz.py` to convert the result into a human-readable HTML file. Here's an example:
0 commit comments