Home
zur Startseite

EuroSDR

EuroSDR Logo  

  EuroSDR project

Benchmark on Image Matching


Background:

Recent innovations in matching algorithms in combination with the increasing quality of digital airborne cameras considerably improved the quality of elevation data generated automatically from aerial images. This development motivated the launch of the “EuroSDR project Benchmarking of Image Matching Approaches for DSM Computation”. It offers a test bed for software developers, distributors and users of dense matching software to evaluate the state-of-the-art and potential of ongoing developments in image based DSM generation.

Scope of the project

As a joint test data set subsets of aerial image blocks are provided. To limit the costs and effort of data processing, the test is currently limited to two representative data sets with different landuse and block geometry. Vaihingen/Enz is from an aerial image flight collected in a semi-rural area. Both ground sampling distance and image overlap are rather moderate, thus this data set is representative for statewide data collection. In contrast, the test data set München has a higher overlap and resolution and is more typical for applications in densely built-up urban area.

Basic scope is the evaluation of elevation data from aerial images produced by different participants with different software systems for image matching. To compare the respective results, it is absolutely essential that all processing is based on the same image orientation. Participants therefore should use the provided image orientations, which are given as Inpho project file and individual orientation files with no modification. Corresponding image and ground coordinates are additional available for each project. Thus, the proper use of the transformations can be controlled by the participants on their own.

The performance of the implemented algorithms is currently evaluated by analysis of a generated DSM raster. This DSM has to be produced by the respective participants in a predefined format. After deliverance to the project team, the difference to a reference DSM will be computed and made available again to the participant.

Access to the data is provided by a ftp-server hosted by the Institute for Photogrammetry. The password will be provided upon request after the participant has provided a signed user agreement from the point of contact

EuroSDR Project Group Dense Image Matching
c/o Prof. Dr.-Ing. Norbert Haala
Institute for Photogrammetry
Geschwister-Scholl-Str. 24D
70174 Stuttgart
Phone: +49-711-685-83383
Fax: +49-711-685-83297
Norbert.Haala@ifp.uni-stuttgart.de

The user agreement can be found here

Test data sets

The data set Vaihingen/Enz is a subset from a flight collected during the project on Digital Airborne Camera Evaluation of the German Society of Photogrammetry, Remote Sensing and Geoinformation (DGPF).

Figure 1: Ortho image and reference DSM of test area Vaihingen/Enz with image footprints and camera stations of UltraCam-X 20cm GSD flight

The aerial images were captured by a UltraCam-X at height above ground of 2900m and a ground sampling distance (GSD) of 20 cm. Figure 1 shows the ortho image and the DSM of the test area. Image footprints and camera stations are additionally overlaid. The overall block consists of 3 strips with 12 images. Participants are required to generate a DSM for the central part with a size of 7.5kmx3.0km at a grid with of 0.2m. Details on the exact definition of the test area and the requested processing are given below.

Figure 2: Color-coded image overlap (maximum nine-folded) for test flight Vaihingen/Enz.

Figure 2 represents the block configuration by a color coded map, with image footprints and camera stations again overlaid. As it is visible, the overlap of 63% in flight and 62% cross flight results in image overlap conditions from one-folded areas (red) to nine-folded overlap (dark green). For the the test area Vaihingen/Enz PAN images are made available as Tiled Tiff uncompressed 8 bit/pix with 9420x14430 pixel at a data volume of 180 Mbyte/image. On demand RGB images can additionally be made available as Tiled Tiff uncompressed 24 bit with 9420x14430 pixel at 537 Mbyte/image.

The data set München is provided by the Landesamt für Vermessung und Geoinformation Bayern (LVG) with support from the City of Munich and the Company Astec. While Vaihingen/Enz was selected as an example for nation-wide DSM generation at areas with varying landuse, the data set München is representative for densely built-up urban areas with complex 3D structures. For such applications, images are usually collected at higher resolution and overlap. As it is visible figure 4, the image sub-block to be processed consists of 3 image strips with 5 images each. This results in 80% in flight and 80% cross flight overlap, which provides up to fifteen-folded object points. The München data set covers a central part of the city and was captured by DMC II 230 at a GSD of 10cm. Each image has a size of 15552x14144 pixel at 16 bit/pix. In addition to the PAN images for image matching, again RGB images can be made available on demand i.e. for visualization purposes. The area to be processed has a size of 1.5x1.7km.

Figure 3: München: Color-coded image overlap maximum fifteen-folded) with image footprints and camera stations(left) and DSM (right).
Figure 4: 3D Visualisation of test area München.

A DSM of the test area is shown in figure 3 (right), figure 4 additionally shows a 3D visualization the data set. As it is visible, the area is densely covered by buildings with heights of up to 50m. These buildings result in occlusions especially for surface parts close to the facades. Thus, visibility can be limited for such regions, which will potentially aggravate the matching processes during DSM generation.

Processing parameters and deliverables

General focus of the benchmark is the comparison of DSM quality as produced by different participants and/or software systems. To allow for comparable results of dense image matching, it is absolutely essential that all the processing is based on the same image orientation. Participants therefore should use the provided orientation, which is given as Inpho project file and individual orientation files with no modification. For each project corresponding image and ground coordinates are provided a file so the proper use of the transformations can be controlled by the participants on their own.

The camera orientation to be used as a common input is available here

In order to simplify the comparison of the results, the DSM generated from the test participants should be provided in the same geometry i.e. they should feature the same resolution and area as well as the same convention for pixel origin.

 

Area to be processed, pixel center

Size of area

Resolution

Vaihingen

Left: 491800

Right: 499300

Bottom: 5418800

Top: 5421800

37500x15000 pixel

7.5x3.0km

0.2 m

München

Left: 4468200

Right: 4469700

Bottom: 5332500

Top: 5334200

15000x17000 pixel

1.5x1.7km

0.1 m

The results shall be provided by a raw Tiff image encoding with float values corresponding to the elevation in meters. For the pixel origin convention, we propose to adopt the so-called "pixel-center" convention, i.e. the pixel of coordinate (100,90) corresponds to the square [99.5,100.5] x [89.5,90.5] of continuous space. The predefined areas of interest for the respective data sets are given in the above table.

In addition to the DSM data the participants also should provide additional information on used hardware environment and time effort for processing of the benchmark data based on the delivered questionnaire.

Preliminary results

After the benchmarks kick-off February 2013, first results were available from a number of participants in May 2013. These results were presented and discussed during the 2nd EuroSDR workshop on 'High Density Image Matching for DSM Computation' June 13th to 14th at the Federal Office of Metrology and Surveying (BEV) in Vienna. This workshop brought together developers, distributors and users of dense matching software to provide a suitable platform for the first review of the respective results. Software systems with respective DSM results, which were presented at the workshop were

  • SocetSet 5.6 (NGATE) from BAE Systems
  • Microsoft’s UltraMap V3.1.
  • Match-T DSM 5.5 from Trimble/inpho
  • ImageStation ISAE-Ext from GeoSystems
  • Pixel Factory from Astrium
  • DSM Tool from the Royal Military Academy (RMA) of Brussels
  • Remote Sensing software package from Joanneum Research
  • MicMac developed at IGN
  • SURE from the Institute for Photogrammetry (IfP), University of Stuttgart
  • FPGA implementation of SGM from the German Aerospace Center (DLR).

The workshop presentations additionally included information on hardware environment and processing strategies. While the heterogeneous hardware prevents a clear ranking with respect to processing times, the general capacity of image matching algorithms for area covering DSM collection was clearly demonstrated. Based on the outcome of the workshop, a preliminary report on the “Landscape of Dense Image Matching Algorithms” was compiled and presented at the Photogrammetric Week 2013 in Stuttgart (see paper and presentation slides).

Figure 5 Vaihingen/Enz: Ortho image and median DSM with RMS differences between the available solutions

As a common reference surface, the results provided by the participants were used to generate a median DSM. Of course this median does not provide an independent ground truth. However, it can be used very well to illustrate differences between the respective solutions and thus give hints to situations which are potentially difficult for image matching.

Figure 5 shows a an ortho image and a shaded representation of this median for a part of the Vaihingen/Enz test area. A color map representing the root mean square differences of all DSMs with respect to this median is additionally overlaid. As it is visible, larger differences occur in the shaded area of the quarry, in the river area, in the vicinity of patches of trees and in the area of the vineyards. However, while comparing the individual results it becomes obvious that a number of software systems still provide consistent accuracies even for such situations.

Figure 6 München: Ortho image and median DSM with exemplary differences for one solution

Similarly, results for the test area München are presented in figure 6. For this example a color map representing the differences to the median DSM was generated from a single solution. Larger differences are mainly visible at small details and steep edges which occur in the vicinity of building borders. Furthermore, cast shadows seem to result in larger differences and increasing noise for the reconstructed surfaces.

While a considerable amount of results were already made available to the project team and have been used to compile these preliminary results, the benchmark data is still made available and can be applied to generate new outcome or improve already existing results.

Further reading

2nd EuroSDR workshop on 'High Density Image Matching for DSM Computation' Program and presentations available at

http://geo.tuwien.ac.at/news/2nd-workshop-on-high-density-image-matching-for-dsm-computation-successfully-completed-2013-06-19/

Haala, N. (2013) The Landscape of Dense Image Matching Algorithms, Photogrammetric Week 2013, Wichmann Verlag, Berlin/Offenbach, pp. 271-284. paper and presentation slides

Principals:

Prof. Dr. Norbert Haala,

Institute for Photogrammetry Universität Stuttgart

Wolfgang Stoessel

Landesamt für Vermessung und Geoinformation Bayern (LVG)

Dr. Michael Gruber

Microsoft Photogrammetry

Prof. Dr. Norbert Pfeifer,

Department of Geodesy and Geoinformation, TU Wien

Michael Franzen, Federal Office for Metrology and Surveying, Wien