1. add velocity for nuscenes dataset.

2. config file format changed.
3. add some well-tuned nuscenes config.
4. target assigner changed. now support per-class similarity calc.

Author: traveller59

second/NUSCENES-GUIDE.md

@@ -0,0 +1,46 @@
+# Nuscenes Train and Eval guide
+
+## Generate Tips
+
+* Nuscenes dataset evaluation contains many hard examples, you need to modify nms parameters (decrease score threshold, increase max size). You can use ```v1.0-mini``` to tune them.
+
+* Nuscenes dataset contain sweeps. You need to use 10 sweeps if you want to get good detection scores. Key-frame only can't get good result, so I drop support for that.
+
+* Nuscenes dataset contains 28130 train samples and 6019 validation samples. Use Nuscenes mini train set (my custom split, ~3500 samples) when develop if you don't have 4+ GPUs. See ```NuscenesDatasetD8``` for more details.
+
+* Some data augmentation will harm detection performance such as global rotation if their value is too large.
+
+* Use KITTI pretrain model if possible.
+
+## Config Guide
+
+### Anchor Generator
+
+1. use ```get_all_box_mean``` in nuscenes_dataset.py to get mean values of all boxes for each class.
+
+2. change ```size``` and z-center in ```anchor_ranges``` in ```anchor_generator_range```.
+
+3. choose thresholds, add some print function in target assigner code and train some steps to see if the threshold is too large or too small. Then tune them.
+
+4. add ```region_similarity_calculator```. If your anchors are too sparse, you need to use ```distance_similarity``` instead of ```nearest_iou_similarity``` for small classes such as pedestrian.
+
+5. If you want to train with velocity, add ```custom_values``` to anchor generator. you can add two zeros. After that, anchors' shape will become ```[N, 9]```.
+
+### Preprocess
+
+1. disable all ground-truth noise.
+
+2. ```global_rotation_uniform_noise``` may decrease performance.
+
+3. disable ```database_sampler``` by delete all content in ```database_sampler```.
+
+### Train
+
+Use ```set_train_step``` in utils.config_tool.train if you don't want to calculate them manually.
+
+## Develop Guide
+
+* uncomment vis functions in prep_pointcloud to see assigned anchors and point cloud after data augmentation to ensure no bug in preprocess.
+
+* use code such as code in script_server.py instead of use commands in terminal.
+

second/README.md

@@ -0,0 +1,289 @@
+# SECOND for KITTI/NuScenes object detection
+SECOND detector.
+
+ONLY support python 3.6+, pytorch 1.0.0+. Tested in Ubuntu 16.04/18.04/Windows 10.
+
+If you want to train nuscenes dataset, see [this](NUSCENES-GUIDE.md).
+
+## News
+
+2019-4-1: SECOND V1.6.0alpha released: New Data API, [NuScenes](https://www.nuscenes.org) support, [PointPillars](https://github.com/nutonomy/second.pytorch) support, fp16 and multi-gpu support.
+
+2019-3-21: SECOND V1.5.1 (minor improvement and bug fix) released! 
+
+2019-1-20: SECOND V1.5 released! Sparse convolution-based network.
+
+See [release notes](RELEASE.md) for more details.
+
+_WARNING_: you should rerun info generation after every code update.
+
+### Performance in KITTI validation set (50/50 split)
+
+```car.fhd.config``` + 160 epochs (25 fps in 1080Ti):
+
+```
+Car [email protected], 0.70, 0.70:
+bbox AP:90.77, 89.50, 80.80
+bev  AP:90.28, 87.73, 79.67
+3d   AP:88.84, 78.43, 76.88
+```
+
+```car.fhd.config``` + 50 epochs + super converge (6.5 hours) +  (25 fps in 1080Ti):
+
+```
+Car [email protected], 0.70, 0.70:
+bbox AP:90.78, 89.59, 88.42
+bev  AP:90.12, 87.87, 86.77
+3d   AP:88.62, 78.31, 76.62
+```
+
+```car.fhd.onestage.config``` + 50 epochs + super converge (6.5 hours) +  (25 fps in 1080Ti):
+
+```
+Car [email protected], 0.70, 0.70:
+bbox AP:97.65, 89.59, 88.72
+bev  AP:90.38, 88.20, 86.98
+3d   AP:89.16, 78.78, 77.41
+```
+
+### Performance in NuScenes validation set (NuScenes mini train set)
+
+```
+car Nusc dist [email protected], 1.0, 2.0, 4.0
+62.80, 73.30, 76.85, 78.87
+pedestrian Nusc dist [email protected], 1.0, 2.0, 4.0
+61.09, 62.20, 63.66, 65.89
+```
+
+## Install
+
+### 1. Clone code
+
+```bash
+git clone https://github.com/traveller59/second.pytorch.git
+cd ./second.pytorch/second
+```
+
+### 2. Install dependence python packages
+
+It is recommend to use Anaconda package manager.
+
+```bash
+conda install scikit-image scipy numba pillow matplotlib
+```
+
+```bash
+pip install fire tensorboardX protobuf opencv-python
+```
+
+If you don't have Anaconda:
+
+```bash
+pip install numba scikit-image scipy pillow
+```
+
+Follow instructions in [spconv](https://github.com/traveller59/spconv) to install spconv. 
+
+If you want to train with fp16 mixed precision (train faster in RTX series, Titan V/RTX and Tesla V100, but I only have 1080Ti), you need to install [apex](https://github.com/NVIDIA/apex).
+
+If you want to use NuScenes dataset, you need to install [nuscenes-devkit](https://github.com/nutonomy/nuscenes-devkit), I recommend to copy nuscenes in python-sdk to second/.. folder (equalivent to add it to PYTHONPATH) and manually install its dependencies, use pip to install devkit will install many fixed-version library.
+
+### 3. Setup cuda for numba (will be removed in 1.6.0 release)
+
+you need to add following environment variable for numba.cuda, you can add them to ~/.bashrc:
+
+```bash
+export NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.so
+export NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so
+export NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice
+```
+
+### 4. add second.pytorch/ to PYTHONPATH
+
+## Prepare dataset
+
+* KITTI Dataset preparation
+
+Download KITTI dataset and create some directories first:
+
+```plain
+└── KITTI_DATASET_ROOT
+       ├── training    <-- 7481 train data
+       |   ├── image_2 <-- for visualization
+       |   ├── calib
+       |   ├── label_2
+       |   ├── velodyne
+       |   └── velodyne_reduced <-- empty directory
+       └── testing     <-- 7580 test data
+           ├── image_2 <-- for visualization
+           ├── calib
+           ├── velodyne
+           └── velodyne_reduced <-- empty directory
+```
+
+Then run
+```bash
+python create_data.py kitti_data_prep --data_path=KITTI_DATASET_ROOT
+```
+
+* [NuScenes](https://www.nuscenes.org) Dataset preparation
+
+Download NuScenes dataset:
+```plain
+└── NUSCENES_TRAINVAL_DATASET_ROOT
+       ├── samples       <-- key frames
+       ├── sweeps        <-- frames without annotation
+       ├── maps          <-- unused
+       └── v1.0-trainval <-- metadata and annotations
+└── NUSCENES_TEST_DATASET_ROOT
+       ├── samples       <-- key frames
+       ├── sweeps        <-- frames without annotation
+       ├── maps          <-- unused
+       └── v1.0-test     <-- metadata
+```
+
+Then run
+```bash
+python create_data.py nuscenes_data_prep --data_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --max_sweeps=10
+python create_data.py nuscenes_data_prep --data_path=NUSCENES_TEST_DATASET_ROOT --version="v1.0-test" --max_sweeps=10
+```
+
+* Modify config file
+
+There is some path need to be configured in config file:
+
+```bash
+train_input_reader: {
+  ...
+  database_sampler {
+    database_info_path: "/path/to/dataset_dbinfos_train.pkl"
+    ...
+  }
+  dataset: {
+    dataset_class_name: "DATASET_NAME"
+    kitti_info_path: "/path/to/dataset_infos_train.pkl"
+    kitti_root_path: "DATASET_ROOT"
+  }
+}
+...
+eval_input_reader: {
+  ...
+  dataset: {
+    dataset_class_name: "DATASET_NAME"
+    kitti_info_path: "/path/to/dataset_infos_val.pkl"
+    kitti_root_path: "DATASET_ROOT"
+  }
+}
+```
+
+## Usage
+
+### train
+
+I recommend to use script.py to train and eval. see script.py for more details.
+
+#### train with single GPU
+
+```bash
+python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/path/to/model_dir
+```
+
+#### train with multiple GPU (need test, I only have one GPU)
+
+Assume you have 4 GPUs and want to train with 3 GPUs:
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,3 python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/path/to/model_dir --multi_gpu=True
+```
+
+Note: The batch_size and num_workers in config file is per-GPU, if you use multi-gpu, they will be multiplied by number of GPUs. Don't modify them manually.
+
+You need to modify total step in config file. For example, 50 epochs = 15500 steps for car.lite.config and single GPU, if you use 4 GPUs, you need to divide ```steps``` and ```steps_per_eval``` by 4.
+
+#### train with fp16 (mixed precision)
+
+Modify config file, set enable_mixed_precision to true.
+
+* Make sure "/path/to/model_dir" doesn't exist if you want to train new model. A new directory will be created if the model_dir doesn't exist, otherwise will read checkpoints in it.
+
+* training process use batchsize=6 as default for 1080Ti, you need to reduce batchsize if your GPU has less memory.
+
+* Currently only support single GPU training, but train a model only needs 20 hours (165 epoch) in a single 1080Ti and only needs 50 epoch to reach 78.3 AP with super converge in car moderate 3D in Kitti validation dateset.
+
+### evaluate
+
+```bash
+python ./pytorch/train.py evaluate --config_path=./configs/car.fhd.config --model_dir=/path/to/model_dir --measure_time=True --batch_size=1
+```
+
+* detection result will saved as a result.pkl file in model_dir/eval_results/step_xxx or save as official KITTI label format if you use --pickle_result=False.
+
+### pretrained model
+
+You can download pretrained models in [google drive](https://drive.google.com/open?id=1YOpgRkBgmSAJwMknoXmitEArNitZz63C). The ```car_fhd``` model is corresponding to car.fhd.config.
+
+Note that this pretrained model is trained before a bug of sparse convolution fixed, so the eval result may slightly worse. 
+
+## Docker (Deprecated. I can't push docker due to network problem.)
+
+You can use a prebuilt docker for testing:
+```
+docker pull scrin/second-pytorch 
+```
+Then run:
+```
+nvidia-docker run -it --rm -v /media/yy/960evo/datasets/:/root/data -v $HOME/pretrained_models:/root/model --ipc=host second-pytorch:latest
+python ./pytorch/train.py evaluate --config_path=./configs/car.config --model_dir=/root/model/car
+```
+
+## Try Kitti Viewer Web
+
+### Major step
+
+1. run ```python ./kittiviewer/backend/main.py main --port=xxxx``` in your server/local.
+
+2. run ```cd ./kittiviewer/frontend && python -m http.server``` to launch a local web server.
+
+3. open your browser and enter your frontend url (e.g. http://127.0.0.1:8000, default]).
+
+4. input backend url (e.g. http://127.0.0.1:16666)
+
+5. input root path, info path and det path (optional)
+
+6. click load, loadDet (optional), input image index in center bottom of screen and press Enter.
+
+### Inference step
+
+Firstly the load button must be clicked and load successfully.
+
+1. input checkpointPath and configPath.
+
+2. click buildNet.
+
+3. click inference.
+
+![GuidePic](https://raw.githubusercontent.com/traveller59/second.pytorch/master/images/viewerweb.png)
+
+
+
+## Try Kitti Viewer (Deprecated)
+
+You should use kitti viewer based on pyqt and pyqtgraph to check data before training.
+
+run ```python ./kittiviewer/viewer.py```, check following picture to use kitti viewer:
+![GuidePic](https://raw.githubusercontent.com/traveller59/second.pytorch/master/images/simpleguide.png)
+
+## Concepts
+
+
+* Kitti lidar box
+
+A kitti lidar box is consist of 7 elements: [x, y, z, w, l, h, rz], see figure.
+
+![Kitti Box Image](https://raw.githubusercontent.com/traveller59/second.pytorch/master/images/kittibox.png)
+
+All training and inference code use kitti box format. So we need to convert other format to KITTI format before training.
+
+* Kitti camera box
+
+A kitti camera box is consist of 7 elements: [x, y, z, l, h, w, ry].

second/RELEASE.md

@@ -0,0 +1,49 @@
+# Release 1.6.0alpha
+
+## Major Features and Improvements
+1. New dataset API (unstable during alpha), almost completely remove kitti-specific code. you can add your custom dataset by following steps: 
+(1): implement all Dataset API functions
+(2): use web visualization tool to check whether the box is correct.
+(3): add your dataset to all_dataset.py, change the dataset_class_name in config file.
+
+2. Add [NuScenes](https://www.nuscenes.org) dataset support (incomplete in 1.6.0alpha), I plan to reproduce the NDS score in their paper.
+
+3. Add [pointpillars](https://github.com/nutonomy/second.pytorch) to this repo.
+
+4. Full Tensorboard support.
+
+5. FP16 and multi-gpu (need test, I only have one gpu) support.
+
+## Minor Improvements and Bug fixes
+
+1. Move all data-specific functions to their corresponding dataset file.
+
+2. Improved config file structure, remove some unused item.
+
+3. remove much unused and deprecated code.
+
+4. add two learning rate scheduler: exp decay and manual step
+
+# Release 1.5.1
+
+## Minor Improvements and Bug fixes
+
+1. Better support for custom lidar data. You need to check KittiDataset for more details. (no test yet, I don't have custom data)
+* Change all box to center format. 
+* Change kitti info format, now you need to regenerate kitti infos and gt database.
+* Eval functions now support custom data evaluation. you need to specify z_center and z_axis in eval function.
+2. Better RPN, you can add custom block by inherit RPNBase and implement _make_layer method.
+3. Update pretrained model.
+4. Add a simple inference notebook. everyone should start this project by that notebook.
+5. Add windows support. Training on windows is slow than linux.
+
+# Release 1.5
+
+## Major Features and Improvements
+
+1. New sparse convolution based models. VFE-based old models are deprecated. Now the model looks like this:
+points([N, 4])->voxels([N, 5, 4])->Features([N, 4])->Sparse Convolution Networks->RPN. See [this](https://github.com/traveller59/second.pytorch/blob/master/second/pytorch/models/middle.py) for more details of sparse conv networks.
+2. The [SparseConvNet](https://github.com/facebookresearch/SparseConvNet) is deprecated. New library [spconv](https://github.com/traveller59/spconv) is introduced.
+3. Super converge (from fastai) is implemented. Now all network can converge to a good result with only 50~80 epoch. For example. ```car.fhd.config``` only needs 50 epochs to reach 78.3 AP (car mod 3d).
+4. Target assigner now works correctly when using multi-class.
+