Multi-sensor data fusion for resolution enhancement of optical Earth observation data

The “SparsEO” project team headed by Prof. Xiaoxiang Zhu (Professor for Signal Processing in Earth Observation) has developed data fusion algorithms to reliably reconstruct high-resolution details in originally low-resolution optical remote sensing imagery. This image shows an example in which low-resolution multispectral (color-) imagery of originally 7.5 meters spatial resolution is “pan-sharpened” by a factor of 10 to a final resolution of 0.75 meters. The original data set was acquired over Munich in 2012 by the HySpex instrument. Sample magnifications of the Olympic Park (indicated by red circles) and the Leonrodplatz (indicated by yellow circles) reveal the super-resolution capacity of the developed fusion framework.

Most optical Earth observation satellites such as IKONOS, QuickBird, GeoEye, and WorldView-2 through 4 provide two separate products with complementary spatial and spectral resolutions: a single broadband channel panchromatic (Pan) image of high spatial resolution and a multispectral image consisting of multiple channels (typically three to eight) at a lower spatial resolution. While the high-resolution Pan image allows for accurate geometric analysis, the spectral channels provide the information necessary for thematic interpretation. As a special branch of image fusion, pan-sharpening aims at fusing a highresolution Pan image with a corresponding low-resolution multispectral image to meet the demands of remote sensing applications, such as feature detection, change monitoring, and land cover classification. Those and many other applications require both high spatial resolution and multispectral information.

As a solution to this pan-sharpening task, the SparsEO team has developed a sophisticated algorithm, named “Jointly Sparse Fusion of Images” (J-SparseFI), which reconstructs the high-resolution multispectral image patch by patch. In particular, JSparseFI uniquely exploits the possible signal structure correlations between multispectral channels by introducing a joint sparsity model and sharpening the highly correlated adjacent multispectral channels simultaneously. This is done by exploiting the recent theory of distributed compressive sensing which restricts the solution of an underdetermined system while considering an ensemble of signals being jointly sparse.

In the growing field of satellite-based hyperspectral remote sensing, there is an even higher demand for resolution enhancement. One of the most advanced upcoming hyperspectral instruments in space will be the spectrometer of the Environmental Mapping and Analysis Program (EnMAP) mission. EnMAP will provide rich spectral data in over 200 channels between 0.4 and 2.5 micrometers at a spatial resolution of 30 meters. Globally acquired hyperspectral data is essential in many applications in which materials or minerals on the ground need to be reliably identified or discriminated. This task has many applications especially in urban areas, where the variety and diversity of materials is much higher than in rural areas. On the other hand, the very high level of spatial details in cities, such as sharp edges and small objects, makes the analysis of 30-meters resolution EnMAP data tremendously challenging as there will be many materials mixed in single pixels.

Based on the J-SparseFI pan-sharpening method, the SparsEO team has developed a multi-sensor data fusion algorithm, named “Jointly Sparse Fusion of Hyperspectral and Multispectral Data” (J-SparseFI-HM). J-SparseFI-HM is capable of producing highresolution hyperspectral data from original EnMAP- or other low-resolution hyperspectral data by fusing it with panchromatic or multispectral imagery taken over the same area.

The computational burden entailed by the necessity of solving hundreds to thousands of optimization problems for each Earth observation data set is mitigated by a highly efficient and fully parallel implementation of a J-SparseFI/J-SparseFI-HM software suite, which is capable of processes even large Earth observation scenes in a few minutes.

This project is a perfect example of how advanced signal processing techniques and software are used to overcome the physical limitations of sensors and hardware. In the particular problems addressed, the fusion of multi-sensor data combines the advantages of satellite-based observations (global coverage and quick access to basically any place on the Earth) and the high resolution that is physically attainable only via airborne measurements.

Resolution enhancement of optical Earth observation data has already greatly facilitated many applications in the past. We are now looking forward to sharpening hyperspectral data acquired by next-generation space-based spectrometers to generate high-resolution high-quality data and, therefore, enable detailed analyses and monitoring of urban areas and other types of complex ground.