Algorithm Performance Contest

The International Association for Pattern Recognition (IAPR) Technical Committee 7 is organizing an algorithm performance contest in conjunction with the 5th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS 2008). The contest involves the running and evaluation of pattern recognition techniques on two different remote sensing data sets with known ground truth.

The contest is open not only to Technical Committee 7 members but to all researchers. The results of the contest will be presented in the workshop in Tampa, Florida, USA on 7 December 2008, and will be summarized in a paper that will appear in the workshop proceedings in IEEE Xplore.

The contest involves two recognition tasks described below. The data for both tasks are provided by the European Commission, Joint Research Center, Institute for the Protection and Security of the Citizen, Information Support for Effective and Rapid External Action (ISFEREA). The quantitative evaluation criteria include the computation of misdetection and false alarm rates as well as other geometric and thematic performance measures suitable for each problem.

The objective of this task is to delineate the buildings in a Quickbird image of Legaspi, Philippines. The data consist of a panchromatic band with 0.6m spatial resolution and 1668×1668 pixels, and four multispectral bands with 2.4m spatial resolution and 418×418 pixels. The output must be an image where the pixels corresponding to each detected building are labeled with a unique integer value.

Figure 1: Manual delineation of buildings will be used as reference data for evaluation.

Evaluation: The following measures will be used in the evaluation of this task:

1. Correct detection, over-detection, under-detection, missed detection, false alarm rates
2. Maximum-weight bipartite graph matching
3. Normalized Hamming distance
4. Rand index, Fowlkes and Mallows index, Jaccard index

You can check the following references for their definitions:

• A. Hoover, G. Jean-Baptiste, X. Jiang, P. J. Flynn, H. Bunke, D. B. Goldgof, K. Bowyer, D. W. Eggert, A. Fitzgibbon, R. B. Fisher, “An Experimental Comparison of Range Image Segmentation Algorithms,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 7, pp. 673-689, July 1996.
• X. Jiang, C. Marti, C. Irniger, H. Bunke, “Distance Measures for Image Segmentation Evaluation,” EURASIP Journal on Applied Signal Processing, vol. 2006, Article ID 35909, pp. 1-10, 2006
• Q. Huang, B. Dom, “Quantitative Methods of Evaluating Image Segmentation,” in Proceedings of International Conference on Image Processing, vol. 3, pp. 53-56, Washington, DC, USA, October 1995.
• A. Ortiz, G. Oliver, “On the Use of the Overlapping Area Matrix for Image Segmentation Evaluation: A Survey and New Performance Measures,” Pattern Recognition Letters, vol. 27, no. 16, pp. 1916-1926, December 2006.

The objective of this task is to extract a digital surface model for buildings from stereo Ikonos data of Graz, Austria. The data consist of a pair of stereo images where each image has a panchromatic band with 1m spatial resolution and 2974×2918 pixels, and four multispectral bands with 4m spatial resolution and 792×749 pixels. The output must be an image where each pixel has an integer value for the estimated digital surface model.

Figure 2: Example of a reference DSM. (Kindly made available by Dr. Karlheinz Gutjahr, Joanneum Research, Institute of Digital Image Processing, A - 8010 Graz, Austria, Wastiangasse 6, for internal use only.)

Evaluation: The following measures that are based on the residuals (difference between the reference DSM and the output DSM) will be used in the evaluation of this task:

1. Bias (mean and skewness of the residuals)
2. Precision (Root-mean-square, outliers in the residuals)
3. Consistency (Correlation between reference and output DSM)

Each participant can submit results for one or both of the tasks. Please follow the submission instructions on the web form. You can use the same password sent to you for downloading the data to access the submission form. In addition to the results, each submission must also include a maximum two-page description of the algorithms used for obtaining the corresponding results.

• Dr. Selim Aksoy
  • Address: Department of Computer Engineering, Bilkent University, Ankara, Turkey
  • Phone: +90 (312) 290-3405
  • Email: saksoy [at] cs [dot] bilkent [dot] edu [dot] tr
  • URL: http://www.cs.bilkent.edu.tr/~saksoy

• Dr. Martino Pesaresi
  • Address: European Commission, Joint Research Centre, Institute for the Protection and Security of the Citizen, Ispra, Italy
  • Phone: +39 (332) 789524
  • Email: martino [dot] pesaresi [at] jrc [dot] it
  • URL: http://isferea.jrc.it

• Deadline for submission of the results: November 15, 2008
• Workshop: December 7, 2008

Participation to the PRRS 2008 Algorithm Performance Contest requires authorization by the data provider. Please fill in the application form and send it to the attention of Dr. Selim Aksoy both via fax at +90 (312) 266-4047 and via email at saksoy@cs.bilkent.edu.tr as soon as possible. Once your participation is approved, an email about how you can download the data will be sent to the email address that you provided.