Introduction of EMVS Event-based Multi-View Stereo

Introduction

RPG_EMVS is the implementation of EMVS: Event-based Multi-View Stereo. A 3D Reconstruction with an Event Camera in Real-Time.

Installation

Clone the responsity

cd home\yousa\rosCatkinBuild\src

git clone git@github.com:uzh-rpg/rpg_emvs.git

Run sample

cd home\yousa\rosCatkinBuild\src\ros_emvc
mkdir sample && cd sample
source /home/yousa/rosCatkinBuild/devel/setup.zsh
rosrun mapper_emvs run_emvs --bag_filename=/home/yousa/dataset/shapes_6dof.bag --flagfile=/home/yousa/rosCatkinBuild
/src/rpg_emvs/mapper_emvs/cfg/slider_depth.conf

Output in terminal:

~/rosCatkinBuild/src/rpg_emvs/sample master ?1 ❯ rosrun mapper_emvs run_emvs --bag_filename=/home/yousa/dataset/shapes_6dof.bag --flagfile=/home/yousa/rosCatkinBuild/src/rpg_emvs/mapper_emvs/cfg/slider_depth.conf
I0911 14:59:14.897390  3300 data_loading.cpp:62] initial stamp: 1468939993.067416019
I0911 14:59:14.956785  3300 depth_vector.hpp:133] Using linear spacing in inverse depth
I0911 14:59:14.956879  3300 mapper_emvs.cpp:183] Specified DSI FoV < 10 deg. Will use camera FoV instead.
I0911 14:59:14.956899  3300 mapper_emvs.cpp:191] Focal length of virtual camera: 168.629 pixels
I0911 14:59:15.124897  3300 main.cpp:91] Time to evaluate DSI: 112 milliseconds
I0911 14:59:15.124984  3300 main.cpp:92] Number of events processed: 478860 events
I0911 14:59:15.124994  3300 main.cpp:93] Number of events processed per second: 4.27554 Mev/s
I0911 14:59:15.125013  3300 main.cpp:95] Mean square = 634.101
I0911 14:59:15.157297  3300 main.cpp:142] Saved 1062 data points to pointcloud.pcd

In sample:

└── sample
    ├── confidence_map.png
    ├── depth_colored.png
    ├── depth_map.png
    ├── dsi.npy
    ├── pointcloud.pcd
    └── semidense_mask.png

Visualization

Point cloud

Original python script has bug in python3.X, so we use pcl-tools or open3D to visualize point cloud.

Pcl-tools

sudo apt-get install pcl-tools

pcl_viewer /home/yousa/rosCatkinBuild/src/rpg_emvs/sample/pointcloud.pcd

Notice it is upside down.

Open3D

import open3d as o3d
import numpy as np
 
ply = o3d.io.read_point_cloud('/home/yousa/rosCatkinBuild/src/rpg_emvs/sample/pointcloud.pcd') 
 
o3d.visualization.draw_geometries([ply])

Disparity Space Image(DSI)

Install visvis

pip install visvis

Visualize DSI

In visualize_dsi_volume.py in line 22, change t = vv.volshow(vol, renderStyle = 'mip') to t = vv.volshow3(vol, renderStyle = 'mip')

import numpy as np
import argparse
import visvis as vv
app = vv.use()

if __name__ == "__main__":

    parser = argparse.ArgumentParser(description='Plot the disparity space image (DSI) using 3D slices')
    parser.add_argument('-i', '--input', default='dsi.npy', type=str,
                        help='path to the NPY file containing the DSI (default: dsi.npy)')
    args = parser.parse_args()

    a = vv.gca()
    a.daspect = 1, -1, 1
    a.daspectAuto = True
    vol = np.load(args.input)
  
    # Reorder axis so that the Z axis points forward instead of up
    vol = np.swapaxes(vol, 0, 1)
    vol = np.flip(vol, axis=0)
  
    # t = vv.volshow(vol, renderStyle = 'mip')
    t = vv.volshow3(vol, renderStyle = 'mip')
    t.colormap = vv.CM_HOT
  
    app.Run()

python3 /home/yousa/rosCatkinBuild/src/rpg_emvs/mapper_emvs/scripts/visualize_dsi_volume.py -i /home/yousa/rosCatkinBuild/src/rpg_emvs/sample/dsi.npy

python3 /home/yousa/rosCatkinBuild/src/rpg_emvs/mapper_emvs/scripts/visualize_dsi_slices.py -i /home/yousa/rosCatkinBuild/src/rpg_emvs/sample/dsi.npy