MareArts Computer Vision Study.

Project - Embeded camera R&D for Real-Time fire detection surveillance in tunnel environment

November 11, 2011

Embeded camera R&D for Real-Time fire detection surveillance in tunnel environment

- 2008.07.01~2009.06.30
- High Computing Power Embeded Camera R&D, Fire & Smoke Detection(My Job)

The goal of the project is to develop embeded camera for detect fire and smoke.
My job was an algorithm R&D for detection fire & smoke and the s/w programing.
The fire detection algorithm uses the HMM algorithm.
The features of the fire's sequence is learned in the off-line. And then the learned Markov model is used in the oline using viterbi algorithm.

Below Movies are the result of the project.

*** Demo 1, Fire detection Simple situation :

*** Demo 2, Fire detection in the road, There is similar light of the car with fire color:

***Demo 3, Fire detection test in the real fire situation :

***Demo 4, Embeded Camera Test. The camera detects the fire and then zoom in the region of the fire.

***Demo 5, Smoke detection test.

8 point algorithm (Matlab source code) / The method to get the Fundamental Matrix and the Essential matrix

November 08, 2011

Created Date : 2011.8

Language : Matlab

Tool : Matlab 2010
Library & Utilized : -
Reference : Multiple View Geometry (Hartly and Zisserman)
etc. : Intrinsic Parameter, 2 adjacent images, matching points

This code is 8 point algorithm.
If we know over 8 corresponding points between two images, we can know Rotation and Translation of camera movement using 8 point algorithm.
The 8 point algorithm is well known in the vision major field.
The algorihtm is introduced at the Multiple View Geometry Book and many websites.

Have you ever listened Fundamental matrix song? The song is very cheerful. ^^

You can download 8 point algorithm at the Peter Covesi homepage.
My code is very simple. so I believe my code will be useful to you.
I will upload RANSAC version later.
Thank you.

github url :https://github.com/MareArts/8point-algorithm
(I used 'getCorrectCameraMatrix' function of Isaac Esteban author.)

main M code..

%//////////////////////////////////////////////////////////////////////////
%// Made by J.H.KIM, 2011 / feelmare@daum.net, feelmare@gmail.com        //
%// blog : http://feelmare.blogspot.com                                  //
%// Eight-Point Algorithm
%//////////////////////////////////////////////////////////////////////////

clc; clear all; close all;

% Corresponding points between two images
% sample #1 I11.jpg, I22.jpg
%{
load I11.txt; load I22.txt;
m1 = I11; m2 = I22;
%}

%sample #2 I1.jpg, I2.jpg
load I1.txt; load I2.txt;
m1 = I1; m2 = I2;

s = length(m1);
m1=[m1(:,1) m1(:,2) ones(s,1)];
m2=[m2(:,1) m2(:,2) ones(s,1)];
Width = 800; %image width
Height = 600; %image height

% Intrinsic Matrix
load intrinsic_matrix.txt
K = intrinsic_matrix;

% The matrix for normalization(Centroid)
N=[2/Width 0 -1;
    0 2/Height -1;
    0   0   1];

%%
% Data Centroid
x1=N*m1'; x2=N*m2';
x1=[x1(1,:)' x1(2,:)'];  
x2=[x2(1,:)' x2(2,:)']; 

% Af=0 
A=[x1(:,1).*x2(:,1) x1(:,2).*x2(:,1) x2(:,1) x1(:,1).*x2(:,2) x1(:,2).*x2(:,2) x2(:,2) x1(:,1) x1(:,2), ones(s,1)];

% Get F matrix
[U D V] = svd(A);
F=reshape(V(:,9), 3, 3)';
% make rank 2 
[U D V] = svd(F);
F=U*diag([D(1,1) D(2,2) 0])*V';

% Denormalize
F = N'*F*N;
%Verification
%L1=F*m1'; m2(1,:)*L1(:,1); m2(2,:)*L1(:,2); m2(3,:)*L1(:,3);

%%
%Get E
E=K'*F*K;
% Multiple View Geometry 259page
%Get 4 Possible P matrix 
P4 = get4possibleP(E);
%Get Correct P matrix 
inX = [m1(1,:)' m2(1,:)'];
P1 = [eye(3) zeros(3,1)];
P2 = getCorrectCameraMatrix(P4, K, K, inX)

%%
%Get 3D Data using Direct Linear Transform(Linear Triangular method)
Xw = Triangulation(m1',K*P1, m2',K*P2);
xx=Xw(1,:);
yy=Xw(2,:);
zz=Xw(3,:);

figure(1);
plot3(xx, yy, zz, 'r+');


%{
%This code is also run well instead of Triangulation Function.
nm1=inv(K)*m1';
nm2=inv(K)*m2';
% Direct Linear Transform
for i=1:s
    A=[P1(3,:).*nm1(1,i) - P1(1,:);
    P1(3,:).*nm1(2,i) - P1(2,:);
    P2(3,:).*nm2(1,i) - P2(1,:);
    P2(3,:).*nm2(2,i) - P2(2,:)];

    A(1,:) = A(1,:)./norm(A(1,:));
    A(2,:) = A(2,:)./norm(A(2,:));
    A(3,:) = A(3,:)./norm(A(3,:));
    A(4,:) = A(4,:)./norm(A(4,:));

    [U D V] = svd(A);
    X(:,i) = V(:,4)./V(4,4);
end
%}

Rotation Matrix Converting Matlab Source (Euler Angle, Rotation Matrix, Quanternion)

November 07, 2011

There are many expression to show the rotation value.
(Ex. : Euler, Matrix, Quaternion.. )

This code is the test source to convert each other.
Euler -> Matrix -> Quanternion -> Matrix -> Euler
We can show the first Euler value is same with the last Euler value.

The source code is like below:
--------------------------------------------------------------

% Rotation vector of x,y,z axis.
Rv = [13 20 50];

% 3x3 matrix of R vector (Results of the Rm1 and Rm2 is similar.)
Rm1 = rodrigues(Rv*pi/180)
Rm2 = mRotMat(Rv)

% Quntenion vector of R matrix
Rq1 = matrix2quaternion(Rm1)
Rq2 = matrix2quaternion(Rm2)

% R matrix of Q vector
Rm1_1 = quaternion2matrix(Rq1)
Rm2_2 = quaternion2matrix(Rq1)

% R vector of R matrix
Rv_1 = rodrigues(Rm1_1(1:3,1:3)) * 180/pi
Rv_2 = rodrigues(Rm2_2(1:3,1:3)) * 180/pi

-----------------------------------------------------------------------
<Source Code>

The copyright of "rodrigues' and 'quaternion' functions is reserved by Peter Kovesi.

I wish this source code is useful to you.
Thank you.

A* path planning Algorithm / C++ source code

November 04, 2011

Created Date : 2010.8

Language : C/C++

Tool : Microsoft Visual C++ 6.0
Library & Utilized : STL
Reference : A* internet reference
etc. : map(.txt file)

It is A* algorithm. A* is known as good path plan algorithm in the Game and other fields.
I made the A* algorithm program. I have refered a lot of reference in the internet. You also can get materials easily in the internet.

I made the algorithm to class type. So we can use the A* Class like this;

CAstar Astar;
Astar.MapCopy(xSize, ySize, &map);
Astar.SetSEpoint(sX, sY, eX, eY);
vector< pair<int, int > > path = Astar.FindPath();

map variable type is 'int ** map'.
sX, Sy is start coordinate. eX, ey is end coordinate.
The path coordinate is saved in the path variable(vector type).

You can download this <source code>.
And I wish to leave your valueable comment.

Thank you.

Sift matching C++ source code / using opencv library

October 27, 2011

Created Date : 2011.10

Language : C/C++

Tool : Microsoft Visual C++ 2008
Library & Utilized : OpenCV 2.3
Reference : SIFT reference
etc. : template Image, WebCam

I made SIFT matching program using OpenCV 2.3.
I was wondering how to know the object pose.
In the internet, there are many source about sift, surf. But most of code introduced about only descripter and matching. There is no code to find object pose.
So I made this code and I should disclose this code.

This code uses openCV functions very useful.
cvExtractSURF, cvFindHomography...

I made matching code to the class. Class file name is MareMatchingClass.h/cpp.
You can use my class in the source very easily.

1. Create Matching class
   CMareMatchingClass MMathing;

2.Input PatchImg
   MMathing.ExtractPatchSurf(PatchImg);

3.Find PatchImg in the background img
   MMathing.GetObjectRectAndBestH(BackGroundImg, &rect4pt);

4.Drawing the rect(rect4pt).
5.Repeat, go to the 3.

The class is consist of like below process;
1. Extract Feature -> use cvExtractSURF function
2. Find Matching point
3. Select some feature in the mached feature points, randomly.
4. calculate Homography matrix. This is geometry relationship between patch and background image.
5. transform features in the patch image by Homography matrix.
6. compare the transformed features to the background features.
7. evaluate how much is the homography exact.
7. repeat 4~6 and select best H.

<source code>

I think the source code is not best.
There are still shortage the source code.
It would need futher improvemnet.
so I want to discuss with you. Please leave your valueable opinion.
Thank you.
Have a nice day~. ^^

Oh~ english is very difficult.....

STL vector(with 2 more elements) sorting code / STL에서 2개 이상의 엘리먼트를 가지고 있는 vector 정렬 code

October 21, 2011

The source is simple but I always don't remember the code well.
If you similar to me, refer to my code easily.

#include <iostream>
#include <cmath>
#include <time.h>
#include <vector>
#include <algorithm>
using namespace std;

class element{
public:
    element(float a, int b){
        value = a;
        index = b;
    }
    float value;
    int index;
};

bool compare(const element &a, const element &b )
{
    return a.value < b.value; //오름차순(ascending order)
    //return a.value > b.value; //내림차순(descending order)
}

void main()
{
    srand( (unsigned int)time(NULL));

    vector< element > A;

    // 값 넣기 (input value)/////////////////////////////////////////////
    for(int i=0; i<20000; ++i){
        A.push_back( element(rand()%10, i) );
    }


    //값 출력  (data print)///////////////////////////////////////////////
    printf("정렬 전(Before ordering) \n");
    for(i=0; i<20000; ++i){
        printf("A[%d] -> value %f :: index -> %d\n", i, A[i].value, A[i].index );
    }


    //정렬 (sorting)
    sort(A.begin(),A.end(),compare);


    //값 출력 (Value print)
    printf("정렬 후(After ordering)\n");
    for(i=0; i<20000; ++i){
        printf("A[%d] -> value %f :: index -> %d\n", i, A[i].value, A[i].index );
    }
      
}

Incremental K-means matlab source code

October 19, 2011

Created Date : 2011.10.

Language : Matlab 2010

Tool : -

Library & Utilized :-

Reference :An Incremental K-means algorithm(D.T. Pham, S.S. Dimov and C.D. Nguyen)

Etc. :-

I made Incremental K-means algorithm as matlab source code.
I made the code base on above table which is introduced in the paper.
The incremental K-means is similar to K-means but the different point is number of cluster class is increasing. but we have to set the maximum number.

Below figure is result of clustering. the sample data is normal vector of the 3D point that is acquied by bumblebee.
I think the K-means algorithm is sensitive to error or outlier data. because the algorithm use euclidean distance. and The result of clustering is different every time because the initial position of the class is selected randomly.

I hope my code help to your problem and study.
thank you.
Give me your valuable comments. ^^

<source code>

Clustering result of the sample datas.

enlarge image of the above figure

BumBleBee 2D, 3D data acquisition source code / 범블비 2D, 3D 데이터 획득 소스

October 17, 2011

Created Date : 2009.10.

Language : C++

Tool : Visual Studio C++ 2008

Library & Utilized : Point Grey-FlyCapture, Triclops, OpenCV 2.1

Reference : PointGrey Bumblebee Reference, www.cylod.com, www.ptgrey.com/

Etc. : STL

BumBleBee Stereo Camera Data Acquisition Source code.

This is Stereo Camera. The name is BumBleBee. This is product of PointGrey Company.
This camera is IEEE 1394 capble type.
This camera can obtain 3D cloud data rapidly and continously.

I need 2 library for using this camera(Triclops SDK, FlyCapture).
You can download these libs on the www.ptgrey.com site(support).
You have to use my source after install libs. and you have to set path(To include directory, lib directory). and you also need opencv 2.1 lib.

I made the acquisition code as class. The class name is CSensorStereo2.
You can use this class like below source code.
The sequence is 'Open->GetData->Close'.
I did that 2D data save Iplimage in opencv and 3D depth data save as Txt file.
The source code is very easy to use ^^.
If you have any question, Plz give your comments to me.

Thank you.

source code is shared on Github

https://github.com/MareArts/Bumblebee_GetDataClass

#include <stdio.h>
#include "SensorStereo2.h"
#include <time.h>

void main()
{

int Width = 320;
int Height = 240; 

CSensorStereo2 CSS2;
CSS2.Initial(Width,Height); 
CSS2.Open();

cvNamedWindow("Reference");
char str[1000];

while(1) {

//get 1 frame data(Image, Depth information)
CSS2.GetData();

//Show Image
cvShowImage("Reference",CSS2.ImgReference); 

//Save Depth
sprintf(str,"./DepthData/%d_Depth.txt",time(0));
printf("%s\n", str);

FILE * fp; fp = fopen(str,"w");
for(int i=0; i<Width; ++i)
{ 
  for(int j=0; j<Height; ++j)
  {
    fprintf(fp,"%lf %lf %lf\n", CSS2.pDepth[i][j].x, CSS2.pDepth[i][j].y, CSS2.pDepth[i][j].z );
  } 
}

fclose(fp);

if(cvWaitKey(1) >= 0 )
   break;
}

cvDestroyWindow("Reference");
CSS2.Close();
}

Catmull-Rom Spline interpolation/ C++ source code/ 스플라인 보간

October 13, 2011

Created Date : 2010.11.

Language : C++

Tool : Visual Studio C++ 2008

Library & Utilized : -

Reference :Catmull-Rom Spline reference

Etc. : STL

This program is Catmull-Rom Spline code.

Please search detail contents about Catmull-Rom Spline in the internet. ^^

In the main.cpp, the source code consisted of Input, calculate spline and output.

In the function of Catmull-Rom Spline, parameters means coordinate(x,y), Step, number of interpolation in a section.

You can get interpolation points easily using this function(PathToCurve).

--------------------------------------------------------------------
이 소스 코드는 Catmull-Rom 보간법을 프로그래밍 한 것 입니다.
이론적인 내용은 인터넷을 검색하시면 많은 자료를 찾을 수 있어서 생략합니다.
메인 함수에는 입력, 보간, 출력으로 구성되어 있고, 입력을 원하는 좌표 값을 입력하여 해당 문제에 적용하시면될 거예요.
보간 함수는 좌표, 스텝,한 구간 보간 갯수를 입력하도록 하였습니다.
매트랩을 통하여 보간결과를 확인해보세요.
간단한 매트랩 파일도 첨부합니다.

좋은 답변 바랍니다. ^^

*main

/////////////////////////////////////////////////////////////////////////
//input
vector< pair<double , double > > InputPath;
FILE * fp1;
fp1 = fopen("input2.txt","r");
double x,y;
while( fscanf(fp1,"%lf %lf", &x, &y)!=EOF )
{
//printf("%lf %lf\n", x,y);
InputPath.push_back( make_pair(x,y) );
}
fclose(fp1);


//////////////////////////////////////////////////////////////////////////
//Spline
vector< pair<double , double > > curvePath;
curvePath = PathToCurve( InputPath, 1, 10);



//////////////////////////////////////////////////////////////////////////
//output
FILE* fp2;
fp2 = fopen("out.txt","w");
for(int i=0; i<curvePath.size(); ++i)
{
fprintf(fp2, "%lf %lf\n", curvePath[i].first, curvePath[i].second );
//printf("%lf %lf\n", curvePath[i].first, curvePath[i].second );
}
fclose(fp2);

*input
5 6
3 4
3 2
5 2
7 4

*output
1.000000 1.000000
1.174000 1.201000
1.392000 1.488000
1.648000 1.837000
1.936000 2.224000
2.250000 2.625000
2.584000 3.016000
2.932000 3.373000
3.288000 3.672000
3.646000 3.889000
4.000000 4.000000
4.359500 3.984500
4.736000 3.856000
5.126500 3.641500
5.528000 3.368000
5.937500 3.062500
6.352000 2.752000
6.768500 2.463500
7.184000 2.224000
7.595500 2.060500
8.000000 2.000000
8.400000 2.056000
8.800000 2.208000
9.200000 2.432000
9.600000 2.704000
10.000000 3.000000
10.400000 3.296000
10.800000 3.568000
11.200000 3.792000
11.600000 3.944000
12.000000 4.000000
12.418000 3.953000
12.864000 3.824000
13.326000 3.631000
13.792000 3.392000
14.250000 3.125000
14.688000 2.848000
15.094000 2.579000
15.456000 2.336000
15.762000 2.137000
16.000000 2.000000

plz give me your valuable comment. Thank you.

Rect Corner Detection / C++, OpenCV1.0 / 사각형 모서리 찾기

October 11, 2011

Created Date : 2008.11.

Language : C++

Tool : Visual Studio C++6.0

Library & Utilized :OpenCV 1.0

Reference : -

Etc. : -

This program is for detecting 4 corners of rects(3 rect).
The detection process is shown below.

1. Candidate corner detection using cvGoodFeaturesToTrack function.
2. Blob Coloring - Find 3 biggest blobs
3. Angle inspection tracking outline of blob. - using cvConvexHull2 function.
Corner's angle would be big than other angles. so we select 4 corners.

<Source Code>
---------------------------------------------------------------------

다음과 같은 과정으로 사각형 캘리브레이션 타켓의 모서리 4군데를 찾는다.

캘리브레이션 타켓의 코너 lefttop, righttop, rightbottom, leftbottom을 검출한다.

cvGoodFeaturesToTrack로 코너 검출
CBlobResult로 Blob Coloring하여 가장 큰 덩어리 3개를 찾음 (사각형 3개)
각 Blob에 대해 cvConvexHull2로 외곽 점 검출
외곽점을 따라가면서 점 간에 각을 저장
점 간의 각도 중 가장 각이 큰 4점을 선택하여 모서리로 판단함
판단된 4점에서 코너점과 가장 가까운 4점을 선택함
이렇게 한 사각형에 대한 모서리 점이 선택됨

결과는 세개의 사각형에 대한
Left Top, Right Top, Right Bottom, Left Bottom 의 이미지 좌표를 알려줌
따라서 카메라 캘리브레이션을 위한 월드 좌표 - 이미지 좌표 값을 자동으로 추출 가능함

10th unmanned vehicle contest. The sponsor is Hyundai-Kia motors(Korea)

October 06, 2011

In 2010, I entered 10th unmanned vehicle contest.
The competition was sponsored by by Hyundai-Kia Motors.
In the our team, my part was software.
I had to solve many problems.
For example; Obstacle Detection and avoidance, mapping, path planing, Localization...
During prepare the competition, I programed nearly 30,000 lines. ^^
My team didn't acomplish the win. But our vehicle have finished all course as unmanned.
And I did my best. so I do not regret it.

OBS have broadcasted this competition.

This movie captured while unmaned vehicle driving.

This is architecture of software.

We have used many sensors and equipments.