Real-time stitching multi-video to one screen

* Introduction

- The solution shows panorama image from multi images. The panorama images is processing by real-time stitching algorithm.

- Each cameras has a limited field of view, but the solution can be monitoring large areas from merged into a panorama image.

- The performance is excellent with the following technical configuration.
 . real time image processing using GPU.
 . Accurate calculation of R, T, K (Rotation, Translation, Camera intrinsic) between each camera with nonlinear optimization
 . Color calibration using the exposure blending

- The solution can be applied efficiently and easy in Military Region, tourist attractions, intersections, ports



* Real-time N to 1 stitching algorithm

- Existing stitching algorithm is modified to separate 2 parts of offline and online processing for more efficient realtime processing.

- The Off-Line processing part is calculated first time or if the matching inaccurate. 

- On-Line processing part is a routine to create the panoramic image by warping (Warping) calculated by the matching, the blending value.



No ordered input images

- Feature extraction and to calculate the homography matrix between each image by evaluating (RANSAC), and set image position through matching rate.
- To get correct R, T using bundle adjustment

- Searching the overlap region, the blending coefficient is determined with respect to the non-overlapping region.

- Obtained R, T, K, and connected by warping the images and blending and complete the panorama finally


* Experiment


- 4 real-time video stitching speed of about 10~20 fps (Intel® core™ i5-3570 cpu 3.40GHz, NVIDIA Geforce GTX 650)



See the result on youtube



updating 42/06/2018
I have decided to sell source code ^^
If you have interest, go to here, you can buy source code


** 2021.06 updated ** 
realtime stitching SDK: 

Finding largest subset images that is only adjacent(subsequnce) images, (OpenCV, SurfFeaturesFinder, BestOf2NearestMatcher, leaveBiggestComponent funcions example souce code)

The souce code flow is like that...

1.
find features in each images using SurfFeaturesFinder function.
Features value is contained in the ImageFeatures structure.

2.
Matching features.
Matcher(features, pairwise_matches, matching_mask)
in the source code, features is vector.
So the Matcher function get matcing value of each pair images.

3.
leave biggest component,
Using conf_threshold, the function leaves largest correlation images.

Input
Input image is 6 images.
4 images are sequence images, 2 images is another sequence images.

Output
The souce code gives the result that is index of subset images of 4 images component.



////
#include < stdio.h >  
#include < opencv2\opencv.hpp >  
#include < opencv2\features2d\features2d.hpp >
#include < opencv2\nonfree\features2d.hpp >
#include < opencv2\stitching\detail\matchers.hpp >
#include < opencv2\stitching\stitcher.hpp >


#ifdef _DEBUG  
#pragma comment(lib, "opencv_core247d.lib")   
//#pragma comment(lib, "opencv_imgproc247d.lib")   //MAT processing  
//#pragma comment(lib, "opencv_objdetect247d.lib")   
//#pragma comment(lib, "opencv_gpu247d.lib")  
#pragma comment(lib, "opencv_features2d247d.lib")  
#pragma comment(lib, "opencv_highgui247d.lib")  
//#pragma comment(lib, "opencv_ml247d.lib")
#pragma comment(lib, "opencv_stitching247d.lib");
#pragma comment(lib, "opencv_nonfree247d.lib");

#else  
#pragma comment(lib, "opencv_core247.lib")  
//#pragma comment(lib, "opencv_imgproc247.lib")  
//#pragma comment(lib, "opencv_objdetect247.lib")  
//#pragma comment(lib, "opencv_gpu247.lib")  
#pragma comment(lib, "opencv_features2d247.lib")  
#pragma comment(lib, "opencv_highgui247.lib")  
//#pragma comment(lib, "opencv_ml247.lib")  
#pragma comment(lib, "opencv_stitching247.lib");
#pragma comment(lib, "opencv_nonfree247.lib");
#endif  

using namespace cv;  
using namespace std;


void main()  
{
 vector< Mat > vImg;
 Mat rImg;

 vImg.push_back( imread("./m7.jpg") );
 vImg.push_back( imread("./B1.jpg") );
 vImg.push_back( imread("./m9.jpg") );
 vImg.push_back( imread("./m6.jpg") );
 vImg.push_back( imread("./B2.jpg") );
 vImg.push_back( imread("./m8.jpg") );
 

 //feature extract
 detail::SurfFeaturesFinder FeatureFinder;
 vector< detail::ImageFeatures> features;
 
 for(int i=0; i< vImg.size(); ++i)
 {  
  detail::ImageFeatures F;
  FeatureFinder(vImg[i], F);  
  features.push_back(F);
  features[i].img_idx = i;
  printf("Keypoint of [%d] - %d points \n", i, features[i].keypoints.size() );
 }
 FeatureFinder.collectGarbage();

 //match
 vector<  int> indices_;
 double conf_thresh_ = 1.0;
 Mat matching_mask;
 vector<  detail::MatchesInfo> pairwise_matches;
 detail::BestOf2NearestMatcher Matcher;
 Matcher(features, pairwise_matches, matching_mask);
 Matcher.collectGarbage();

 printf("\nBiggest subset is ...\n");
 // Leave only images we are sure are from the same panorama 
 indices_ = detail::leaveBiggestComponent(features, pairwise_matches, (float)conf_thresh_);
 Matcher.collectGarbage();

 for (size_t i = 0; i <  indices_.size(); ++i)
    {
  printf("%d \n", indices_[i] );
 }
 

}

////









OpenCV Stitching example (Stitcher class, Panorama)

Image size of origin is 320*240. 






Processing time is 30.96 second took.

    The result of stitching

The result is pretty good. but, processing time is too much takes.

My computer spec is that.. (This is vmware system. The main system is mac book air 2013, i7 8bg)



The source code is very easy.
I think if we use stitching algorithm in realtime, we should be programing by GPU.

/////
#include < stdio.h >  
#include < opencv2\opencv.hpp >  
#include < opencv2\stitching\stitcher.hpp >

#ifdef _DEBUG  
#pragma comment(lib, "opencv_core246d.lib")   
#pragma comment(lib, "opencv_imgproc246d.lib")   //MAT processing  
#pragma comment(lib, "opencv_highgui246d.lib")  
#pragma comment(lib, "opencv_stitching246d.lib");

#else  
#pragma comment(lib, "opencv_core246.lib")  
#pragma comment(lib, "opencv_imgproc246.lib")  
#pragma comment(lib, "opencv_highgui246.lib")  
#pragma comment(lib, "opencv_stitching246.lib");
#endif  

using namespace cv;  
using namespace std;


void main()  
{
 vector< Mat > vImg;
 Mat rImg;

 vImg.push_back( imread("./stitching_img/S1.jpg") );
 vImg.push_back( imread("./stitching_img/S2.jpg") );
 vImg.push_back( imread("./stitching_img/S3.jpg") );
 vImg.push_back( imread("./stitching_img/S4.jpg") );
 vImg.push_back( imread("./stitching_img/S5.jpg") );
 vImg.push_back( imread("./stitching_img/S6.jpg") );
  

 Stitcher stitcher = Stitcher::createDefault();


 unsigned long AAtime=0, BBtime=0; //check processing time
 AAtime = getTickCount(); //check processing time

 Stitcher::Status status = stitcher.stitch(vImg, rImg);

 BBtime = getTickCount(); //check processing time 
 printf("%.2lf sec \n",  (BBtime - AAtime)/getTickFrequency() ); //check processing time

 if (Stitcher::OK == status) 
  imshow("Stitching Result",rImg);
  else
  printf("Stitching fail.");

 waitKey(0);

}  
/////

github
https://github.com/MareArts/Still-Image-Stitching-Test-Using-OpenCV

N image, realtime stitching.
source code:
http://study.marearts.com/2016/10/real-time-n-camera-stitching-class.html
how to work:
http://study.marearts.com/2015/02/real-time-stitching-multi-video-to-one.html

2 image stitching.
basic principal on vidoe(code and explanation):
http://study.marearts.com/2013/10/two-view-of-cam-to-one-screen-using.html
basic principal on image(code and explanation):
http://study.marearts.com/2011/08/two-image-mosaic-paranoma-based-on-sift.html


I have decided to sell source code ^^
If you have interest, go to here, you can buy source code.
Thank you very much!!



Two Image mosaic (paranoma) based on SIFT / C++ source (OpenCV) / SIFT 특징 추출기를 이용한 두장의 영상을 모자익(파라노마) 영상으로 만들기

Created Date : 2011.2
Language : C/C++
Tool : Microsoft Visual C++ 2010
Library & Utilized : OpenCV 2.2
Reference : Interent Reference
etc. : 2 adjacent images


two adjacent iamges

Feature extraction by Surf(SIFT)

Feature matching

Mosaic (paranoma)

This program is conducted as follow process.
First, the program finds feature point in each image using SURF.
->cvExtractSURF
Second, feature points on each images is matched by similarity.
->FindMatchingPoints
Third, We get the Homography matrix.
->cvFindHomography
Last, we warp the image for attaching into one image.
->cvWarpPerspective

You can download source here.
If you have good idea or advanced opinion, please reply me.
Thank you.

-----------------------------------------------------------------------------

이웃된 두 장의 영상을 입력 받아 하나의 모자이크 영상(파라노마)으로 만든다.
특징 추출 및 비교 방법 : suft ->cvExtractSURF
특징 매칭 방법 : FindMatchingPoints
호모그라피 행렬 구하기 : cvFindHomography
영상 모자이크 방법 : warpping

전체 소스 코드는 여기서 받을 수 있습니다.
https://github.com/MareArts/Two-Image-mosaic-paranoma-based-on-SIFT
개선 사항이나 좋은 의견 있으시면 답변 주세요.
감사합니다.

< gist >

< /gist >