Real-Time Panoramic Video Construction Using Harris Corners Detector

This paper discusses the possibility of stitching two video streams to create a panoramic view in real-time, The challenge with constructing Panorama Video in real-time is the time-consuming in stitching images. Image alignment didn't need to detect all interest point, only few well distributed point is enough to calculate good projection matrix, and the great similarity between the frames in video stream provides temporal information which can be used to reduce the number of selected points In this paper the stitching images method which using Harris Corners Detector has been customized to be appropriate to meet the requirements of viewing a video in real time, through reducing the number of Interest Points in each stitching and employ the Parallel Processing to separate the calculation of transform matrix from the process of blending. Reducing the number of Interest Points has been done by fragmentation of images into numerous regions and getting a feedback information from the stitching of the previous frame to limitate the search space based on the area of overlapping. Results show that (i) panoramas generated from the proposed algorithm feature a smooth transition in image overlapping areas and satisfy human visual requirements; and (ii) the preview speed of the generated panorama satisfies the real-time requirements that are commonly accepted in video panorama stitching.


INTRODUCTION
anorama means constructing of high resolution images with large field of view.The terminology "Panoramic Video" sometime used to refer to the construction of a static panoramic image from one video stream, but in this work will be used to refer to the stream of panoramic frames constructed from two or more video streams.
Image stitching is based on finding the similarity between adjacent images and determination of overlapping regions.Image stitching (Image Registration or Image Alignment) algorithms can be classified into three different categories: Intensity-based algorithms which need a large amount of computation and in case of do not give good result rotation and scaling [1] [2].(ii) Frequency-domain-based algorithms are faster with small translation, rotation, and scaling, but it fail with small overlapping regions [3] [4].(iii) Feature-based algorithms, which reduced the computational complexity by processed a small amount of information (points, lines, or edges) to align images, and they are robust to changes in image intensity [5] [6] .
Therefore, most of the researches has been focused on feature-based algorithms.one of the most commonly used feature-based algorithms is the one which has been shown in Figure (1), in which Harris Corners Detector [7][8] detect the common points (Interest Points) between the two images, these points (Corners) have a large intensity variations between the directions around.
Then these points are matched by Normalized-Cross-Correlation (NCC) [9] to generate a candidate pair.It found that some of these pairs are wrong and must be eliminated, so the Random Sample Consensus algorithm (RANSAC) [10] has been used, which estimate a mathematical model representing the largest number of these pairs and remove all the pairs that do not fit with this model.After that Projection one of the two images overlaid on top of the other and calculate the Transform Matrix to transform this image, finally the two images blended together [11].
This algorithm is classified as a feature-based algorithm and it success with small overlapping areas but with small difference in rotation, and scaling.In addition, the technique of threading has been used to separate the process of calculating the Homograph Transformation Matrix which takes a few seconds than the process of Transforming the frames and Blending them, to reach a rate acceptable according to the requirements of Display Video in real-time

P
The rest of this paper is organized as: Section 2 "Related Work" briefly discusses related work; Section 3 "Concepts of the Proposed Video Stitching Algorithm" describes three basic concept of the solution; Section 4 "Structure of the Proposed Video Stitching Algorithm" Introduces the Proposed Algorithm; Section 5 "Performance Evaluation and Results Discussion" describes experimental results; and finally Section 6 "Conclusions".

Related Work
When studying the subject of video and image stitching , it is clear that there are many works relating image stitching, on the other side very little works has been done on the video stitching, especially considering the case when the capturing devices are allowed to move freely, as opposed to the case where the cameras are fixed.The authors in [5][6], proposed a video stitching system combine multiple video feeds from ordinary cameras or webcams into a single panoramic video.And to reduce the computation satisfy real-time requirements, the parameters for stitching are calculated only once, during the system initialization.So their system requires an initialization phase that is not real-time and is needed whenever the cameras or webcams positions change.
In [11] the authors presented a new improved algorithm, self-adaptive selection of Harris corners by fragment the image into regions and select corners according to the normalized variance of region grayscales.They apply multiple constraints, e.g., their midpoints, distances, and slopes, on every two candidate pairs to remove incorrectly matched pairs.Their algorithm, solved the video stitching problem on a frame-byframe basis, and does not fully exploit the correlation between video frames, and requires to identify which of the two images is on the right and which is on the left, and matching only the right-third of the left image with the left-third of the right image.finally they fixed the total number of points that are matching in each stitching.
The authors in [12] proposed a system for stitching videos streamed previously recorded by freely moving mobile phones cameras.However, their proposed system treats the video stitching problem essentially as an image of stitching problem on individual frames independent of each other.
In [13] the authors exploits temporal information to avoid solving the stitching problem fully on a frame by frame basis.The employ of the previous frames stitching results such as tracking interest points using optical flow and using areas of overlap is to limit the search space for interest points.
In all these related works, it was found that the most time-consuming step is the computation of detection and matching large number of Interest Points, it takes more than 85% of the time in stitching two frames.While only four well distributed Interest Points are enough to calculate the Homography Transform Matrix.Hence, most of the study approaches in this paper are aiming at detection and matching less number of Interest Point.

Concepts of the Proposed Video Stitching Algorithm
The performance of the proposed video stitching algorithm is based on three basic concepts:

Fragmentation
With Harris corners detector, the corners tend to cluster around regions with richer texture, whereas fewer corners will be selected in regions with less texture information.That mean , selected corners are not evenly distributed.Therefore, each frame will be fragmented into (n×n) regions where n is the number of fragments in the one side of frame, and select one corner from each region by sorted all candidate corners in their values, The number of all selected corners will not be more than the number of fragments (n×n), this number of fragments will be modified depending on the feedback from stitching the previous frame, (Eq.( 1)) where i is the current frame stitching.
If the number of detected corners in the previous stitching are few, then the fragments number will be increased and reduced its size, and vice versa.This fragmentation will help to reduce the time of matching and correlation , while maintaining the distribution and avoid clustering of the corners in one side of the frame.

Exploit the Area of Overlap from Previous Frame
The area of overlap from frames (i) can potentially limit the search space for Interest Points in frames (i+1).The stitching algorithm performs the same steps as in the case of the first frame pairs but instead of trying to detect Interest Points in the whole frame (i+1), they are detected only in the area of overlap which found in frames (i), plus some buffer region in the frames (i+1) (best value experimentally determined to be a 7 pixel band).The efficiency of the buffer based approach performance is inversely proportional to the size of the overlap area between frames (n-1) and (n).

Parallel Processing
Parallel processing is the simultaneous use of more than one CPU or processor core to execute a program or multiple computational threads.Theoretically, parallel processing makes programs run faster because there are more (CPUs or cores) running it.In practice, it is often difficult to divide a program in such a way that separate CPUs or cores can execute different portions without interfering with each other, all new CPU models have multi-core processor chips with Hyper-threading Technology.
This work needs one thread for each camera to capture the video stream, and two threads for two spirited loops for stitching operations, and one more or two threads for monitoring the result and record the output video.Chart (1) shows the time line of the parallel processing in T 0 CPU(1,2) begin capturing frames from the two cameras, in T 1 finishes capturing first frames (A 1 ,B 1 ) and begins calculating Transform Matrixes, in T 2 finishes calculating first Matrix (M 1 ) and CPU(4) begin blending A n with B n *M 1 , in T 3 finishes calculating Matrix (M 2 ) and begins blending A 2n with B 2n *M 2 .

Structure of the Proposed Video Stitching Algorithm
As previously mentioned the proposed algorithm have two spirited loops for stitching operations, First loop contains three steps, Detection , Matching and Estimation & Projecting, which are the first three main steps in image stitching algorithms, and the second loop contains Blending step, which is the forth main steps in image stitching algorithms.As in the Figure (2).Inability of the algorithm to stitch frame pairs can be due to either: (i) there is no enough overlap between frames (ii) enough correlation pairs between frames, can't be found (iii) the algorithm computes an incorrect geometric alignment.
In the Blending step the degradation method has been used, by linear gradient Transparency from the center of one image to the other

Performance Evaluation and Results Discussion
Experiments were conducted on an Intel ® Core™2 Duo CPU T6500 @ 2.10GHZ, with 4.0 GB RAM, NVIDIA GeForce G 105M video card and two Logitech C905 webcam HD720p , USB 2.0, Ultra-smooth, light correction, Auto Focus with Carl Zeiss optics and 2-megapixel resolution for video and photos up to 8-megapixels.The application was built using C#.Net framework 4.0 and Accord.NET 2013 library.The details (dimension H × W, size of two image, overlapping between two image, distribution of corners, horizontal line, point of view and scene depth) of images pairs that have been used to test the algorithm are shown in Table (1) and Figure ( 3).Lake:

Table (1): Details of the testing images pairs
House: School: Nature: To be more specific, with increasing the value of n, the number of corners will be increased and the result will be closer to that of the traditional Harris detector.On the other hand, if the value of n is small, the number of corners will be decreased, knowing that the execution time has not changed.Table (2) content the results of Applying the Fragmentation on the four images pairs with different value of n (order by the number of corners), and Chart ( 2) shows the number of corners VS the number of Fragment.

Chart (2): Details of Fragmentation VS number of corners
It is clear, that the benefit of fragmentation increases with the large-size and highresolution images.

Evaluation of Limitation the Search Space
The result of limitation the search space based on the area of overlap in the previous frame on the image pair (House) (without using the Fragmentation) is shown in Figure (5).Where the number of the selected corners becomes 249×295 corners instead of 351×361 corners in traditional Harris detector.As well as the time of the search became 0.34 seconds instead of 0.82 seconds.

Figure (5). Limitation the search space on House image
Table (3) and Chart (3) illustrate the change in the selected corners limitation the search space on the four images pairs, and how the usefulness of limitation is directly proportional to the ratio of overlap between images  It is clear that the very little time required for the blending operation, makes the use of parallel processing provides the possibility of merging video frames in real time (camera frame rate), but with a delay in responding to any movement of the scene or camera.This delay has been reduced significantly when using the proposed algorithm, and the results will show in detail in the final Result.

Final Results
The implementation of both fragmentation and limitation on the pair (House) will decrease the total execution time of all stitching operations from 0.95sec.to 0.36sec., without any change in the quality of the resulting panorama.Figure (6) shows the four stitching operations on the image pair (House) with using the proposed algorithm.Table (4) shows details of outputs and execution time of each stitching operations in proposed algorithm compared with the outputs and execution time in the traditional image stitching.

DISCUSSION
The comparing of the percentage of reducing time between the execution time of the traditional algorithm and the execution time of the proposed algorithm with the percentage of overlapping area in each image pairs, shows that the percentage of reducing time (i) is directly proportioning to the size and resolution of the image.In other words, percentage of reducing time increases with larger and high resolution images, because these image have a lot of interest points that can be discarded, (ii) is an inverse proportional to the area of the overlapping , In other words, percentage of reducing time increases with small overlap area, because the search in a smaller space needs less time.

CONCLUSIONS
A new algorithm has been presented to handle challenges in real-time video panoramic Construction, which is time-consuming of stitching processes.the contribution can be summarized by improving images stitching algorithm to reduce the number of Interest Points in each stitching process through: (i) Employ Fragmentation to avoid the points clustering as much as possible, (ii) Employ Feedback to update the Fragmentation and limit the search space by exploiting the previous overlapping area.(iii) Employ Parallel Processing which has been resulted in more efficient panorama stitching process.Experimental results shows that (i) panoramas generated from the proposed algorithm feature a smooth transition in image overlapping areas and satisfy human visual requirements; and (ii) the preview speed of the generated panorama satisfies the real-time requirements that are commonly accepted in video panorama stitching.

Figure ( 1
Figure (1): The structure of the traditional images stitching system

Figure ( 2 )
Figure (2): The structure of the proposed image stitching system

Figure 3 .
Figure (4).Fragmentation on House image.To be more specific, with increasing the value of n, the number of corners will be increased and the result will be closer to that of the traditional Harris detector.On the other hand, if the value of n is small, the number of corners will be decreased, knowing that the execution time has not changed.Table(2) content the results of Applying the Fragmentation on the four images pairs with different value of n (order by the number of corners), and Chart(2) shows the number of corners VS the number of Fragment.Table (2): Details of testing Fragmentation on four images pairs ‫اﻻھﺘﻤﺎم‬ ‫ﻧﻘﺎط‬ ‫ﻋﺪد‬ ‫ﺗﻘﻠﯿﻞ‬ ‫ﺧﻼل‬ ‫ﻣﻦ‬ ‫وذﻟﻚ‬ ‫اﻟﺤﻘﯿﻘﻲ،‬ ‫اﻟﻮﻗﺖ‬ ‫ﻓﻲ‬ ‫ﻓﯿﺪﯾﻮ‬ ‫ﺷﺮﯾﻂ‬ ‫ﻋﺮض‬ ‫ﻣﺘﻄﻠﺒﺎت‬ ‫ﻟﺘﻠﺒﯿﺔ‬ ‫اﻟﻤﺘﺴﻠﺴﻠﺔ‬ ‫اﻟﻤﻌﺎﻟﺠﺔ‬ ‫ﺑﺪل‬ ‫اﻟﻤﺘﻮازﯾﺔ‬ ‫اﻟﻤﻌﺎﻟﺠﺔ‬ ‫واﺳﺘﺨﺪام‬ ‫دﻣﺞ‬ ‫ﻋﻤﻠﯿﺔ‬ ‫ﻋﻦ‬ ‫اﻟﺘﺤﻮﯾﻞ‬ ‫ﻣﺼﻔﻮﻓﺔ‬ ‫ﺣﺴﺎب‬ ‫ﻋﻤﻠﯿﺔ‬ ‫ﻟﻔﺼﻞ‬ ‫اﻟﺪﻣﺞ‬ .‫ﺧﻼ‬ ‫ﻣﻦ‬ ‫ﯾﺘﻢ‬ ‫اﻻھﺘﻤﺎم‬ ‫ﻧﻘﺎط‬ ‫ﻋﺪد‬ ‫ﺗﻘﻠﯿﻞ‬ ‫إن‬ ‫ﻋﻠﻰ‬ ‫اﻟﺤﺼﻮل‬ ‫ﺧﻼل‬ ‫وﻣﻦ‬ ‫اﻟﻤﻨﺎﻃﻖ‬ ‫ﻣﻦ‬ ‫ﻋﺪد‬ ‫إﻟﻰ‬ ‫اﻟﺼﻮر‬ ‫ﺗﺠﺰﺋﺔ‬ ‫ل‬ ‫ﺑﯿﻦ‬ ‫اﻟﺘﺪاﺧﻞ‬ ‫ﻣﺴﺎﺣﺔ‬ ‫إﻟﻰ‬ ً ‫اﺳﺘﻨﺎدا‬ ‫اﻟﺒﺤﺚ‬ ‫ﻣﺴﺎﺣﺔ‬ ‫ﻟﺘﻘﻠﯿﺺ‬ ‫اﻟﺴﺎﺑﻖ‬ ‫اﻹﻃﺎر‬ ‫ﺧﯿﺎﻃﺔ‬ ‫ﻣﻦ‬ ‫اﻟﺮاﺟﻌﺔ‬ ‫اﻟﺘﻐﺬﯾﺔ‬ ‫ﻣﻌﻠﻮﻣﺎت‬ ‫اﻟﺼﻮر‬ .