Automatic 3D Reconstruction from Multi-Date Satellite Images

Dauer der Arbeit: 4 Monate
Abschluss: April 2019
Betreuer: M.Sc. Ke Gong
Prüfer: Prof. Dr.-Ing. Norbert Haala

In recent years, there are plenty of earth observation satellites (such as Pléiades, WorldView-1,2,3,4, QuickBird, IKONOS and so on) that orbit the earth several times per year and can supply us with high-resolution satellite images of the same site with different dates, which makes multiple view stereo reconstruction possible.

Contributions in the field of computer vision are indispensable factors for promoting development of multi-view stereo 3D reconstruction, like SIFT (Distinctive Image Features from Scale-Invariant Keypoints, 2004) for point matching, image rectification (Loop, et al., 1999), SGM algorithm (Hirschmüller, 2008) for stereo matching and triangulation. Besides, the Rational Polynomial Coefficient (Jacek Grodecki, et al., 2003) is also an important contribution for satellites imagery.

A program Satellite Stereo Pipeline, abbreviated as s2p, which is developed by a group from École normale supérieure Paris-Saclay (Facciolo, et al., 2017) (Carlo De Franchis, 2014), will be run and discussed in this thesis.

The program s2p doesn’t apply a true multiple view method, it processes many pairs of images independently and then fuses the results. S2p is written in Python and it mainly consists of the following modules:

• Initialization
• Pointing error
• Rectification
• Stereo matching
• Triangulation
• Dsm-rasterization

In initialization part the input images will be cut into smaller tiles, so that they are rectifiable. This is because the used images are taken with the satellite World-View-3, which uses a pushbroom sensor. The epipolarity of linear pushbroom images is different from that of perspective images (Kim, 2000) and pushbroom images are not able to be rectified. But within a certain small range, pushbroom images’ epipolar curves can be well approximated by straight lines and they can be seemed as rectifiable images.

Next, the pointing error will be corrected, which is caused by limited precision of camera calibration and accuracy of georeferencing of satellite images. After correcting the pointing error, the image pair will be rectified. The corresponding points of rectified pairs lie on the same horizontal line, which makes 2D point searching problem into a 1D problem. The corresponding points will be found by MGM algorithm (Facciolo, 2015), which is developed based on SGM algorithm, and a disparity map will be calculated in stereo matching step. In triangulation module, the heights of the matched points will be determined by RPC and inverse RPC. A point cloud of each tile can be produced by all the points with known height in this tile. In the last step, local and global rasterized dsm will be determined by point cloud.

The s2p program doesn’t provide a module to concatenate the point clouds of each tiles. One could either use CloudCompare to merge all the tiles together, or convert the global rasterized dsm of whole range into point cloud using MATLAB. To save the computing time, the point clouds will be merged by CloudCompare. These point clouds will be compared and analyzed with lidar data (Bosch, et al., 2016). The point clouds with best results will be fused by software SURE (Rothermel, et al., 2012).

For the thesis, 42 images taken by WorldView-3 (0.31m GSD) will be tested. These images are numbered from 1 to 42. Table 1 shows the best 10 pairs.

Completeness is the main criterion to evaluate the point cloud and it is measured by the percentage of points in the MVS point cloud with less than 1m error compared to lidar data (Bosch, et al., 2016).

Figure 1 shows point cloud fused by the best 10 pairs. The fusion step is done by using standalone module of software SURE. Its completeness is . The edges (blue part) are not well triangulated, because the heights of the edges are very different from the heights of other points.

Figure 2 shows the point cloud provided by Institute für Photogrammetrie of university Stuttgart. The same dataset will be processed and this point cloud is also fused by the best 10 pairs. Its completeness is . In figure 2 the edges have better performance than in figure 1.

Figure 3 shows the cloud to cloud distance between IFP and s2p point cloud. The height difference varies from about 0 m to 21.6 m. The big differences mostly occur around high building.

Image pairs, whose incident angles are not larger than 35° and intersection angle between 5° and 40°, yield good result.

• Advantage of s2p

s2p uses open source code and the used algorithms in modules can be replaced by other different algorithms. s2p applies RPC, which makes computation easier. The MGM algorithm improves the quality of stereo matching.

• Disadvantage of s2p

The program is not fully automatic, because the produced point clouds for small tiles are not merged and there is no point cloud fusion module.

Bosch, Marc, et al. 2016. A Multiple View Stereo Benchmark for satellite imagery . 2016.

Carlo De Franchis, Enric Meinhardt-Llopis, Julien Michel, Jean-Michel Morel, Gabriele Facciolo. 2014. an automatic and modular stereo pipeline for pushbroom images. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Sep 2014, Zürich, Switzerland. 10.5194/isprsannals-II-3-49-2014, 2014, Vols. hal-01109526.

Distinctive Image Features from Scale-Invariant Keypoints. Lowe, David G. 2004. 2, Vancouver : s.n., 2004, Vol. 60. 91-110.

Facciolo, Gabriele & de Franchis, Carlo & Meinhardt, Enric. 2015. MGM: A Significantly More Global Matching for Stereovision. 10.5244/C.29.90. , 2015.

Facciolo, Gabriele, Franchis, Carlo de and Meinhardt-Llopis, Enric. 2017. Automatic 3D Reconstruction from Multi-Date Satellite Images. 2017.

Hirschmüller, Heiko. 2008. Stereo Processing by Semi-Global Matching and Mutual Information. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. 2008.

Jacek Grodecki and Gene Dial. 2003. Block Adjustment of High-Resolution Satellite Images Described by Rational Polynomials. 2003.

Kim, Taejung. 2000. A Study on the Epipolarity of Linear Pushbroom Images . American Society for Photogrammetry and remote sensing. 2000.

Loop, Charles and Zhang, Zhengyou. 1999. Computing Rectifying Homographies for Stereo Vision. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1999.

Rothermel, Mathias, et al. 2012. SURE: PHOTOGRAMMETRIC SURFACE RECONSTRUCTION FROM IMAGERY. 2012.