Real-time stitching multi-video to one screen

* Introduction

- The solution shows panorama image from multi images. The panorama images is processing by real-time stitching algorithm.

- Each cameras has a limited field of view, but the solution can be monitoring large areas from merged into a panorama image.

- The performance is excellent with the following technical configuration.
 . real time image processing using GPU.
 . Accurate calculation of R, T, K (Rotation, Translation, Camera intrinsic) between each camera with nonlinear optimization
 . Color calibration using the exposure blending

- The solution can be applied efficiently and easy in Military Region, tourist attractions, intersections, ports

* Real-time N to 1 stitching algorithm

- Existing stitching algorithm is modified to separate 2 parts of offline and online processing for more efficient realtime processing.

- The Off-Line processing part is calculated first time or if the matching inaccurate. 

- On-Line processing part is a routine to create the panoramic image by warping (Warping) calculated by the matching, the blending value.

No ordered input images

- Feature extraction and to calculate the homography matrix between each image by evaluating (RANSAC), and set image position through matching rate.
- To get correct R, T using bundle adjustment

- Searching the overlap region, the blending coefficient is determined with respect to the non-overlapping region.

- Obtained R, T, K, and connected by warping the images and blending and complete the panorama finally

* Experiment

- 4 real-time video stitching speed of about 10~20 fps (Intel® core™ i5-3570 cpu 3.40GHz, NVIDIA Geforce GTX 650)

See the result on youtube


  1. Hi, how can you use STITCHER class with gpu? i tried to build openCV 3.1 with cuda. I set "Stitcher stitch = Stitcher::createDefault(true); " but the function "stitch" reported an error

    1. This is not gpu mode. However, I used this class in version 2.4.7
      refer to this url

    2. This comment has been removed by the author.

  2. Hey, what a nice work!
    Can you explain or refer some links for me about the bundle adjustment. I'm stuck in this step, my panorama alignment can't be center, it keep expanding at the end :(
    Thank you.

    1. It is difficult to answer in detail here ..
      This is to get several unknowns values at once(actually, it iterative method).
      In other words ... it use nonlinear optimization techniques to get various parameters of the camera.
      Sorry. I can not help you now...
      There is a lot of information on the internet.

    2. I can tell you some keywrods.
      - Levenberg-Marquardt
      - Newton's method, Gauss-Newton method
      - unconstrained nonlinear optimization