SiftGPU/CUDA observation registration issue

[jayyoung]

Hi folks,
I had an issue today getting the 3d mapping stuff running on my own machine in order to do some meta-room scans in simulation.

Using: Ubuntu 14.04, GeForce GTX 950, nVidia driver 352.63, CUDA Driver/Runtime 7.5 (also tried with 5.5 and got similar results), latest version of hydro-devel branch of strands_3d_mapping from strands_project repo.

The issue seems to be with the SiftGPU library and CUDA. When registering a new observation, the following crash happens on my machine:

[additional_view_registration_server-2]: started with pid [20315]
[ INFO] [1462387636.921738562]: additional_view_registration_server started.
[ INFO] [1462387805.297279541]: Received an observation registration request
[ INFO] [1462387805.297325648]: Target /home/jxy/.semanticMap/20160504/patrol_run_4/room_0/room.xml
[ INFO] [1462387805.297336799]: Source /home/jxy/.semanticMap/20160504/patrol_run_5/room_0/room.xml
[ INFO] [1462387805.297363376]: Registering source to target.
[ INFO] [1462387833.161896495]: Using original (odometry) intermediate cloud transforms
FilterH: invalid device function
FilterV: invalid device function
FilterH: invalid device function
FilterV: invalid device function

[then about ~50 more lines of the above repeated]

PyramidCU::GenerateFeatureList: invalid device function
[observation_registration_server-1] process has died [pid 20302, exit code -11, cmd /home/jxy/soma_ws/devel/lib/observation_registration_server/observation_registration_server __name:=observation_registration_server __log:=/home/jxy/.ros/log/834a1f6e-1219-11e6-87ec-1c872c60d569/observation_registration_server-1.log].
log file: /home/jxy/.ros/log/834a1f6e-1219-11e6-87ec-1c872c60d569/observation_registration_server-1*.log
[observation_registration_server-1] restarting process

This is appears to be an issue with SiftGPU/CUDA not working together properly, which may be something unique to my hardware, drivers or a world of other possibilities. It could be that my card just doesn't support CUDA properly. That said, CUDA itself seems to work as it should, and I can run/use the various demos and samples and deviceQuery gives me the following:

./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 950"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 2047 MBytes (2146762752 bytes)
( 6) Multiprocessors, (128) CUDA Cores/MP: 768 CUDA Cores
GPU Max Clock rate: 1190 MHz (1.19 GHz)
Memory Clock rate: 3305 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 1048576 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GTX 950
Result = PASS

So on paper everything should work. I was unable to find any real solution to the problem, but it seems SiftGPU can run in a bunch of different configurations and doesn't necessarily require CUDA. I fixed the issue by modifying the sift_wrapper.cpp files so that they don't use the CUDA version of SiftGPU any more, and instead used GLSL, which was very simple as all I did was change:

const char *argv_sift[] = {"-m", "-fo","-1", "-s", "-v", "0", "-pack", "-cuda", "-maxd", "3840"};
to

const char *argv_sift[] = {"-m", "-fo","-1", "-s", "-v", "0", "-pack", "-glsl", "-maxd", "3840"};

Once I made this change and re-compiled, everything seemed to work perfectly fine and I was able to register views successfully, SIFT features are extracted, sent to CERES, and then I see the CERES output and stuff saying everything is working. No crashes as of yet.

I am not sure if this is something specific to my machine, video card or whatever else, but I thought I would mention it. I guess this all works fine on the robots, but at least for me this is what I need to do to get stuff running on my desktop in simulation. Ideally it'd be nice if there was a flag somewhere in the makefile to switch between CUDA/GLSL, rather than having to mess with the source code directly.

I thought it would bring it up. At the very least it might save someone some time if they run into the same thing!

[RaresAmbrus]

@jayyoung strange issue, I haven't seen it before. A couple of times it happened that CUDA wasn't set up properly, in which case SIFTGPU would just crash directly.

I'll do a couple of tests over here with the glsl version, and if all goes well I'll switch completely to that and remove the CUDA dependency.