v1.6.0alpha

1. add incomplete NuScenes support
2. New Dataset API
3. lots of code changes
4. config change, incompatible with previous

Author: traveller59

README.md

@@ -1,14 +1,17 @@
-# SECOND-V1.5 for KITTI object detection
-SECOND-V1.5 detector.
+# SECOND for KITTI/NuScenes object detection
+SECOND detector.
 
 ONLY support python 3.6+, pytorch 1.0.0+. Tested in Ubuntu 16.04/18.04/Windows 10.
 
 ## News
 
-2019-3-21: SECOND V1.5.1 (minor improvement and bug fix) released! See [release notes](RELEASE.md) for more details.
+2019-4-1: SECOND V1.6.0alpha released: New Data API, [NuScenes](https://www.nuscenes.org) support, [PointPillars](https://github.com/nutonomy/second.pytorch) support.
 
-2019-1-20: SECOND V1.5 released! See [release notes](RELEASE.md) for more details.
+2019-3-21: SECOND V1.5.1 (minor improvement and bug fix) released! 
 
+2019-1-20: SECOND V1.5 released! Sparse convolution-based network.
+
+See [release notes](RELEASE.md) for more details.
 
 ### Performance in KITTI validation set (50/50 split)
 
@@ -68,6 +71,8 @@ pip install numba scikit-image scipy pillow
 
 Follow instructions in [spconv](https://github.com/traveller59/spconv) to install spconv. 
 
+If you want to use NuScenes dataset, you need to install [nuscenes-devkit](https://github.com/nutonomy/nuscenes-devkit), I recommend to copy nuscenes in python-sdk to second/.. folder (equalivent to add it to PYTHONPATH) and manually install its dependencies, use pip to install devkit will install many fixed-version library.
+
 ### 3. Setup cuda for numba
 
 you need to add following environment variable for numba.cuda, you can add them to ~/.bashrc:
@@ -82,7 +87,7 @@ export NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice
 
 ## Prepare dataset
 
-* Dataset preparation
+* KITTI Dataset preparation
 
 Download KITTI dataset and create some directories first:
 
@@ -101,22 +106,32 @@ Download KITTI dataset and create some directories first:
            └── velodyne_reduced <-- empty directory
 ```
 
-* Create kitti infos:
-
+Then run
 ```bash
-python create_data.py create_kitti_info_file --data_path=KITTI_DATASET_ROOT
+python create_data.py kitti_data_prep --data_path=KITTI_DATASET_ROOT
 ```
 
-* Create reduced point cloud:
-
-```bash
-python create_data.py create_reduced_point_cloud --data_path=KITTI_DATASET_ROOT
-```
-
-* Create groundtruth-database infos:
+* [NuScenes](https://www.nuscenes.org) Dataset preparation
 
+Download NuScenes dataset:
+```plain
+└── NUSCENES_TRAINVAL_DATASET_ROOT
+       ├── samples       <-- key frames
+       ├── sweeps        <-- frames without annotation
+       ├── maps          <-- unused
+       └── v1.0-trainval <-- metadata and annotations
+└── NUSCENES_TEST_DATASET_ROOT
+       ├── samples       <-- key frames
+       ├── sweeps        <-- frames without annotation
+       ├── maps          <-- unused
+       └── v1.0-test     <-- metadata
+```
+Since the dataset is really large, you can download parts of the dataset.
+
+Then run
 ```bash
-python create_data.py create_groundtruth_database --data_path=KITTI_DATASET_ROOT
+python create_data.py nuscenes_data_prep --data_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval"
+python create_data.py nuscenes_data_prep --data_path=NUSCENES_TEST_DATASET_ROOT --version="v1.0-test"
 ```
 
 * Modify config file
@@ -127,24 +142,30 @@ There is some path need to be configured in config file:
 train_input_reader: {
   ...
   database_sampler {
-    database_info_path: "/path/to/kitti_dbinfos_train.pkl"
+    database_info_path: "/path/to/dataset_dbinfos_train.pkl"
     ...
   }
-  kitti_info_path: "/path/to/kitti_infos_train.pkl"
-  kitti_root_path: "KITTI_DATASET_ROOT"
+  dataset: {
+    kitti_info_path: "/path/to/dataset_infos_train.pkl"
+    kitti_root_path: "DATASET_ROOT"
+  }
 }
 ...
 eval_input_reader: {
   ...
-  kitti_info_path: "/path/to/kitti_infos_val.pkl"
-  kitti_root_path: "KITTI_DATASET_ROOT"
+  dataset: {
+    kitti_info_path: "/path/to/dataset_infos_val.pkl"
+    kitti_root_path: "DATASET_ROOT"
+  }
 }
 ```
 
 ## Usage
 
 ### train
 
+I recommend to use script.py to train and eval. see script.py for more details.
+
 ```bash
 python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/path/to/model_dir
 ```
@@ -169,7 +190,7 @@ You can download pretrained models in [google drive](https://drive.google.com/op
 
 Note that this pretrained model is trained before a bug of sparse convolution fixed, so the eval result may slightly worse. 
 
-## Docker (I don't have time to build docker for SECOND-V1.5)
+## Docker (Deprecated. I can't push docker due to network problem.)
 
 You can use a prebuilt docker for testing:
 ```
@@ -185,7 +206,7 @@ python ./pytorch/train.py evaluate --config_path=./configs/car.config --model_di
 
 ### Major step
 
-1. run ```python ./kittiviewer/backend.py main --port=xxxx``` in your server/local.
+1. run ```python ./kittiviewer/backend/main.py main --port=xxxx``` in your server/local.
 
 2. run ```cd ./kittiviewer/frontend && python -m http.server``` to launch a local web server.
 

RELEASE.md

@@ -1,12 +1,24 @@
-# Release 1.5
+# Release 1.6.0alpha
 
 ## Major Features and Improvements
+1. New dataset API (unstable during alpha), almost completely remove kitti-specific code. you can add your custom dataset by following steps: 
+(1): implement all Dataset API functions
+(2): use web visualization tool to check whether the box is correct.
+(3): add your dataset to all_dataset.py, change the dataset_class_name in config file.
 
-1. New sparse convolution based models. VFE-based old models are deprecated. Now the model looks like this:
-points([N, 4])->voxels([N, 5, 4])->Features([N, 4])->Sparse Convolution Networks->RPN. See [this](https://github.com/traveller59/second.pytorch/blob/master/second/pytorch/models/middle.py) for more details of sparse conv networks.
-2. The [SparseConvNet](https://github.com/facebookresearch/SparseConvNet) is deprecated. New library [spconv](https://github.com/traveller59/spconv) is introduced.
-3. Super converge (from fastai) is implemented. Now all network can converge to a good result with only 50~80 epoch. For example. ```car.fhd.config``` only needs 50 epochs to reach 78.3 AP (car mod 3d).
-4. Target assigner now works correctly when using multi-class.
+2. Add [NuScenes](https://www.nuscenes.org) dataset support (incomplete in 1.6.0alpha), I plan to reproduce the NDS score in their paper.
+
+3. Add [pointpillars](https://github.com/nutonomy/second.pytorch) to this repo.
+
+4. Full Tensorboard support.
+
+## Minor Improvements and Bug fixes
+
+1. Move all data-specific functions to their corresponding dataset file.
+
+2. Improved config file structure, remove some unused item.
+
+3. remove much unused and deprecated code.
 
 # Release 1.5.1
 
@@ -19,4 +31,15 @@ points([N, 4])->voxels([N, 5, 4])->Features([N, 4])->Sparse Convolution Networks
 2. Better RPN, you can add custom block by inherit RPNBase and implement _make_layer method.
 3. Update pretrained model.
 4. Add a simple inference notebook. everyone should start this project by that notebook.
-5. Add windows support. Training on windows is slow than linux.
+5. Add windows support. Training on windows is slow than linux.
+
+# Release 1.5
+
+## Major Features and Improvements
+
+1. New sparse convolution based models. VFE-based old models are deprecated. Now the model looks like this:
+points([N, 4])->voxels([N, 5, 4])->Features([N, 4])->Sparse Convolution Networks->RPN. See [this](https://github.com/traveller59/second.pytorch/blob/master/second/pytorch/models/middle.py) for more details of sparse conv networks.
+2. The [SparseConvNet](https://github.com/facebookresearch/SparseConvNet) is deprecated. New library [spconv](https://github.com/traveller59/spconv) is introduced.
+3. Super converge (from fastai) is implemented. Now all network can converge to a good result with only 50~80 epoch. For example. ```car.fhd.config``` only needs 50 epochs to reach 78.3 AP (car mod 3d).
+4. Target assigner now works correctly when using multi-class.
+

second/builder/dataset_builder.py

@@ -1,16 +1,41 @@
+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Input reader builder.
+
+Creates data sources for DetectionModels from an InputReader config. See
+input_reader.proto for options.
+
+Note: If users wishes to also use their own InputReaders with the Object
+Detection configuration framework, they should define their own builder function
+that wraps the build function.
+"""
+
 from second.protos import input_reader_pb2
-from second.data.dataset import KittiDataset
+from second.data.all_dataset import get_dataset_class
 from second.data.preprocess import prep_pointcloud
+from second.core import box_np_ops
 import numpy as np
 from second.builder import dbsampler_builder
 from functools import partial
-from second.utils import config_tool
+from second.utils.config_tool import get_downsample_factor
 
 def build(input_reader_config,
           model_config,
           training,
           voxel_generator,
-          target_assigner=None):
+          target_assigner):
     """Builds a tensor dictionary based on the InputReader config.
 
     Args:
@@ -26,61 +51,74 @@ def build(input_reader_config,
     if not isinstance(input_reader_config, input_reader_pb2.InputReader):
         raise ValueError('input_reader_config not of type '
                          'input_reader_pb2.InputReader.')
-    generate_bev = model_config.use_bev
-    without_reflectivity = model_config.without_reflectivity
+    prep_cfg = input_reader_config.preprocess
+    dataset_cfg = input_reader_config.dataset
     num_point_features = model_config.num_point_features
-    downsample_factor = config_tool.get_downsample_factor(model_config)
+    out_size_factor = get_downsample_factor(model_config)
+    assert out_size_factor > 0
     cfg = input_reader_config
-    db_sampler_cfg = input_reader_config.database_sampler
+    db_sampler_cfg = prep_cfg.database_sampler
     db_sampler = None
     if len(db_sampler_cfg.sample_groups) > 0:  # enable sample
         db_sampler = dbsampler_builder.build(db_sampler_cfg)
-    u_db_sampler_cfg = input_reader_config.unlabeled_database_sampler
-    u_db_sampler = None
-    if len(u_db_sampler_cfg.sample_groups) > 0:  # enable sample
-        u_db_sampler = dbsampler_builder.build(u_db_sampler_cfg)
     grid_size = voxel_generator.grid_size
     # [352, 400]
-    feature_map_size = grid_size[:2] // downsample_factor
+    feature_map_size = grid_size[:2] // out_size_factor
     feature_map_size = [*feature_map_size, 1][::-1]
     print("feature_map_size", feature_map_size)
     assert all([n != '' for n in target_assigner.classes]), "you must specify class_name in anchor_generators."
+    dataset_cls = get_dataset_class(dataset_cfg.dataset_class_name)
+    assert dataset_cls.NumPointFeatures >= 3, "you must set this to correct value"
     prep_func = partial(
         prep_pointcloud,
-        root_path=cfg.kitti_root_path,
-        class_names=target_assigner.classes,
+        root_path=dataset_cfg.kitti_root_path,
         voxel_generator=voxel_generator,
         target_assigner=target_assigner,
         training=training,
-        max_voxels=cfg.max_number_of_voxels,
+        max_voxels=prep_cfg.max_number_of_voxels,
         remove_outside_points=False,
-        remove_unknown=cfg.remove_unknown_examples,
+        remove_unknown=prep_cfg.remove_unknown_examples,
         create_targets=training,
-        shuffle_points=cfg.shuffle_points,
-        gt_rotation_noise=list(cfg.groundtruth_rotation_uniform_noise),
-        gt_loc_noise_std=list(cfg.groundtruth_localization_noise_std),
-        global_rotation_noise=list(cfg.global_rotation_uniform_noise),
-        global_scaling_noise=list(cfg.global_scaling_uniform_noise),
+        shuffle_points=prep_cfg.shuffle_points,
+        gt_rotation_noise=list(prep_cfg.groundtruth_rotation_uniform_noise),
+        gt_loc_noise_std=list(prep_cfg.groundtruth_localization_noise_std),
+        global_rotation_noise=list(prep_cfg.global_rotation_uniform_noise),
+        global_scaling_noise=list(prep_cfg.global_scaling_uniform_noise),
         global_random_rot_range=list(
-            cfg.global_random_rotation_range_per_object),
+            prep_cfg.global_random_rotation_range_per_object),
+        global_translate_noise_std=list(prep_cfg.global_translate_noise_std),
         db_sampler=db_sampler,
-        unlabeled_db_sampler=u_db_sampler,
-        generate_bev=generate_bev,
-        without_reflectivity=without_reflectivity,
-        num_point_features=num_point_features,
-        anchor_area_threshold=cfg.anchor_area_threshold,
-        gt_points_drop=cfg.groundtruth_points_drop_percentage,
-        gt_drop_max_keep=cfg.groundtruth_drop_max_keep_points,
-        remove_points_after_sample=cfg.remove_points_after_sample,
-        remove_environment=cfg.remove_environment,
-        use_group_id=cfg.use_group_id,
-        downsample_factor=downsample_factor)
-    dataset = KittiDataset(
-        info_path=cfg.kitti_info_path,
-        root_path=cfg.kitti_root_path,
-        num_point_features=num_point_features,
-        target_assigner=target_assigner,
-        feature_map_size=feature_map_size,
+        num_point_features=dataset_cls.NumPointFeatures,
+        anchor_area_threshold=prep_cfg.anchor_area_threshold,
+        gt_points_drop=prep_cfg.groundtruth_points_drop_percentage,
+        gt_drop_max_keep=prep_cfg.groundtruth_drop_max_keep_points,
+        remove_points_after_sample=prep_cfg.remove_points_after_sample,
+        remove_environment=prep_cfg.remove_environment,
+        use_group_id=prep_cfg.use_group_id,
+        out_size_factor=out_size_factor)
+
+    ret = target_assigner.generate_anchors(feature_map_size)
+    class_names = target_assigner.classes
+    anchors_dict = target_assigner.generate_anchors_dict(feature_map_size)
+    anchors = ret["anchors"]
+    anchors = anchors.reshape([-1, 7])
+    matched_thresholds = ret["matched_thresholds"]
+    unmatched_thresholds = ret["unmatched_thresholds"]
+    anchors_bv = box_np_ops.rbbox2d_to_near_bbox(
+        anchors[:, [0, 1, 3, 4, 6]])
+    anchor_cache = {
+        "anchors": anchors,
+        "anchors_bv": anchors_bv,
+        "matched_thresholds": matched_thresholds,
+        "unmatched_thresholds": unmatched_thresholds,
+        "anchors_dict": anchors_dict,
+    }
+    prep_func = partial(prep_func, anchor_cache=anchor_cache)
+
+    dataset = dataset_cls(
+        info_path=dataset_cfg.kitti_info_path,
+        root_path=dataset_cfg.kitti_root_path,
+        class_names=class_names,
         prep_func=prep_func)
 
     return dataset

second/builder/target_assigner_builder.py

@@ -38,4 +38,3 @@ def build(target_assigner_config, bv_range, box_coder):
         positive_fraction=positive_fraction,
         sample_size=target_assigner_config.sample_size)
     return target_assigner
-

second/builder/voxel_builder.py

@@ -3,14 +3,6 @@
 from spconv.utils import VoxelGenerator
 from second.protos import voxel_generator_pb2
 
-class _VoxelGenerator(VoxelGenerator):
-    @property
-    def grid_size(self):
-        point_cloud_range = np.array(self.point_cloud_range)
-        voxel_size = np.array(self.voxel_size)
-        g_size = (point_cloud_range[3:] - point_cloud_range[:3]) / voxel_size
-        g_size = np.round(g_size).astype(np.int64)
-        return g_size
 
 def build(voxel_config):
     """Builds a tensor dictionary based on the InputReader config.
@@ -28,7 +20,7 @@ def build(voxel_config):
     if not isinstance(voxel_config, (voxel_generator_pb2.VoxelGenerator)):
         raise ValueError('input_reader_config not of type '
                          'input_reader_pb2.InputReader.')
-    voxel_generator = _VoxelGenerator(
+    voxel_generator = VoxelGenerator(
         voxel_size=list(voxel_config.voxel_size),
         point_cloud_range=list(voxel_config.point_cloud_range),
         max_num_points=voxel_config.max_number_of_points_per_voxel,