Multi-camera calibration using semantic features

Abstract

The demand for multi-camera setups has been steadily growing for modern computer vision applications. This necessitates the need for precise multi-camera calibration, whether it is for autonomous vehicles, immersive virtual environments, or surveillance networks. Feature matching plays a crucial role in determining the accuracy of multi-camera calibration. Feature-based methods, such as SIFT (Scale-Invariant Transform) and SURF (Speeded-Up Robust Features), are commonly employed to identify and match distinctive visual features across multiple camera views. These feature-matched correspondences are useful for establishing the relation between the respective cameras and finding their relative pose. While such feature-based methods are effective, they fail under certain circumstances - large viewpoints or inconsistent scale. This paper attempts to address this challenge by using semantic segmentation as a pre-processing step. The experiments showed that using semantic-masked images for the feature-matching process resulted in greater relevant matches in the same semantic class and reduced erroneous matches between dissimilar semantic regions. The results demonstrated significant performance improvement using this approach without compromising on the overall average accuracy.