WG1: Mobile Multi-Sensor Systems - Final Report (1995 – 1999)
SC4 WG1: Mobile Multi-Sensor Systems
Chairman: Dr. Naser El-Sheimy,
Department of Geomatics Engineering,
The University of Calgary
Calgary, Alberta, Canada,
E-mail : firstname.lastname@example.org
July 12, 1999
Comments and context:
To fullfill the need for up-to-date inventory and geometric data along transportation routes (roads, railways, rivers, pipelines) mobile inventory systems are being operated. In general motion of a vehicle in three-dimensional space can be described by six parameters: they are normally chosen as three position and three orientation parameters. Frequently GPS integrated with other sensors is proposed to fullfil these trajectory requirements. In addition inventory and geometric data can be collected with sensors, such as: CCD cameras, extensometers, tiltmeters, laser scanners, etc. Normally the multi-sensor systems need highly efficient software tools to adjust and store data.
Dr. Naser El-Sheimy (Canada)
H. Döller (Austria), G. Presle (Austria), G. Retscher (Austria), D. Cosandier (Canada), A. Hasan (Canada), W. Benning (Germany), W. Caspary (Germany), W. Niemeier (Germany), I. Colomina (Spain), H. Sternberg, (Germany), J. B. Bullock (USA), R. Li (USA), H. Kinawi (Canada), V. Tao (Canada), Martin J Smith, (United Kingdom), Sabry El-Hakim, (Canada).
Introduction: The Role of Mobile Multi-Sensor Systems in Engineering Applications
Mobile Multi-sensor Systems (MMS) have become an emerging trend in mapping applications because they allow a task-oriented implementation of geodetic concepts at the measurement level. Examples of such systems can be found in airborne remote sensing, airborne gravimetry, airborne laser scanning, and mobile mapping vans and. All of these systems have a common feature in that the sensors necessary to solve a specific problem are mounted on a common platform. By synchronizing the data streams accurately, the solution of a specific problem is possible by using data from one integrated measurement process only. The post-mission integration of results from a number of disjoint measurement processes and the unavoidable errors inherent in such a process are avoided. This results in greater conceptual clarity, task-oriented system design and data flow optimization, and also offers in most cases the potential for real-time solution, which is becoming more important in many applications.
The trend towards multi-sensor systems in geomatics is fuelled by the demand for fast and cost-effective data acquisition and by technological developments which satisfies this demand. Two developments are especially important in this context: Digital imaging and precise navigation. Digital imaging sensors considerably reduce the data processing effort by eliminating the digitizing step. They also opens the way towards new and flexible designs of the processing chain, making ample use of mathematical software tools readily available. In the form of digital frame cameras, they are inexpensive enough to make redundancy a major design tool. In the form of pushbroom scanners, they provide additional layers of information, not available from optical cameras.
Precise navigation has developed to a point where it can provide the solution of the exterior orientation problem without the use of GCPs or block adjustment procedures. Since results are available in a digital form, data fusion with the imaging data is easy and real-time applications are possible in principle. Operational flexibility is greatly enhanced in all cases where a block structure is not needed. Costs are considerably reduced, especially in areas where little or no ground control is available. Current accuracy is sufficient for many mapping applications and the potential to solve even high-accuracy cadastral applications certainly exists.
Combining these two developments, the concept of the georeferenced image as the basic photogrammetric unit emerges. This means that each image is stamped with its georeferencing parameters, namely three positions and three orientations, and can be combined with any other georeferenced image of the same scene by using geometric constraints, such as epipolar geometry or object-space matching. This is a qualitatively new step because the georeferencing parameters for each image are obtained in a direct way by independent measurement. This is conceptually different from the notion that a block of connected images and sufficient ground control is needed to solve the georeferencing problem. The direct method, in contrast, does not require connectivity information within a block of images to solve the georeferencing problem, and thus offers much greater flexibility. It is especially intriguing to consider its use for mapping applications which use either digital frame cameras, pushbroom scanners, or laser scanners as imaging components.
In general, the conceptual layout and data flow of a multi-sensor system for mapping applications is shown in Figure 1. The selection of sensors for such a system obviously depends on system requirements, such as accuracy, reliability, operational flexibility, and range of applications. The data acquisition module has, therefore, to be designed keeping both the carrier vehicle and the intended applications in mind. The data acquisition module contains navigation sensors and imaging sensors. Navigation sensors are used to solve the georeferencing problem. Although a number of different systems are used in general navigation, the rather stringent requirements in terms of accuracy and environment make the integration of an INS with GPS receivers the core of any sensor combination for an accurate mobile mapping system for short range applications. This combination also offers considerable redundancy and makes the use of additional sensors for reliability purposes usually unnecessary. However, the addition of an odometer type device, such as the ABS, for close-range applications may be useful for operational reasons, as for instance keeping a fixed distance between camera exposures.
The selected sensor configuration requires a certain data processing sequence. Part of the processing will have to be performed in real time, such as data compression for the imaging data and initial quality control processing for the navigation data. Most of the data, however, will immediately be stored for post-mission use. In post-mission, the data processing hierarchy is determined by the fact that all mapping data have to be georeferenced first before they can be used in the integration process. The first step is, therefore, the georeferencing of all mapping data and their storage in a multimedia data base. Real-time georeferencing is possible, in principle, because the interior orientation can be done before the run and the exterior orientation parameters can be computed in real time. However, it is not advisable in applications where frequent loss of lock to GPS satellites occurs. In such cases, real-time georeferencing will be much poorer in accuracy and reliability than post-mission processing.
The accuracy of georeferencing is dependent on the accuracy with which this can be achieved. The synchronization accuracy needed is dependent on the required system performance and on the speed with which the survey vehicle moves. It is, therefore, much more critical for airborne applications than for marine and land vehicle applications. Fortunately, GPS provides a well-defined time signal to which the other sensors can be slaved. Still, the implementation of sensor synchronization is not a trivial process and should be done with care. See the WG literature for more details about MMS and their applications in mapping and engineering applications.
Activities of the WG:
- Sponsor of the mobile mapping systems session at KIS 97, The International Symposium on Kinematic Systems in Geodesy, Geomatics and Navigation, KIS97, Banff, Canada, June 3-6, 1997.
- Contribute to the Mobile Multi-Sensor Sessions at The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt, Vienna, Austria, April 20 to April 23, 1998.
- Co-Sponsor of the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999. Organized by the ISPRS, IAG, and FIG.
Highlight from the International Workshop on Mobile Mapping Technology
The International Workshop on Mobile Mapping Technology, was successfully held in Bangkok, Thailand, April 21-23, 1999. It was co-organized by ISPRS WG II/1 Real-time Mapping Technologies, IAG SC4/WG1 Mobile Multi-Sensor Systems, FIG C5/WG3 Real-time Precise Mapping, ISPRS WG V/1 Close-Range Imaging and Metrology, ISPRS IC WG V/III Image Sequence Analysis. The sponsoring organizations included International Society for Photogrammetry and Remote Sensing (ISPRS), Asian Institute of Technology (AIT), National Research Council of Thailand (NRCT), International Association of Geodesy (IAG), International Federation of Surveyors (FIG), Association for Real-time Imaging and Dynamic Analysis (ARIDA), and Japan Association of Remote Sensing (JARS).
It was very clear from panel discussion at the end of the workshop (see Figure 2) that building a mobile mapping system by integrating off-the-shelf hardware and software components is getting easier, but it requires significant courage, investment and efforts. We have seen development activities by many universities and companies on almost all continents in recent years. Land-based systems have demonstrated the power promised at the early time of the development, for example in road and railway survey, utility survey and others. The takeover of the part of such traditional surveying markets is believed to be only a start. Meanwhile, the very same concept has been transferred to airborne and satellite-borne platforms where positional and orientational sensors are integrated with imaging sensors to approach real-time mapping that is not restricted to where only land vehicles can reach. The "dream" is to achieve the same level of ground position accuracy as traditional aerial triangulation. Of course, an integration of the sensor-based orientation data with aerial triangulation would provide much better results. We trust that with the rapid development in mobile mapping, automatic triangulation, and automatic feature/image matching, real-time systems will be a reality in the near future.
Today, mobile mapping is supported by a series of advanced technologies, including navigation sensors of GPS and INS, imaging sensors of high-resolution CCD, SAR, multispactral and hyperspectral sensors, computers and high intelligent processing/automation algorithms. A question we may want to ask ourselves is "have the current mobile mapping systems reached full potential?" The answer is a definite NO. We still see that mobile mapping takes only a small percentage of the overall surveying market where it ought to do a better job. Four aspects need our attentions: a) prices of mobile mapping systems are high, partly contributed by high cost components such as INS and very large CCD chips, b) better tools for efficient and automatic extraction of useful information from massive mobile mapping data are needed, c) accuracy should be increased for applications where higher accuracy is desirable, and d) efforts should be made to make the spatial data community more aware of the existence and potential of the technology. This workshop served as a forum for researchers, developers and users of mobile mapping systems to summarize the achievements, find out the current problems, and map out the future development. The 410 page proceedings of International Workshop on Mobile Mapping Technology, edited by R. Li and S. Murai (ISSN 0256-1840), can be ordered from RICS Books, Surveyor Court, Westwood Way, Coventry CV4 8JE, United Kingdom, Fax: +44-171-334-3800.
The next International Conference on Mobile Mapping Technology will be held in Luxor, Egypt, January 4-6, 2001. For further information please contact Dr. Naser El-Sheimy, Department of Geomatics Engineering, The University of Calgary, 2500 University Dr. N.W., Calgary, Alberta, Canada T2N 1N4, Tel : (403) 220 7587, Fax : (403) 284 1980, E-mail: email@example.com
Mobile Multi-Sensor Systems: Future Trends
The continual development of Mobile Multi-Sensor Systems (MMS) is stimulating the development of intelligent processing techniques and new areas of application. This section discusses future trends in the development of Mobile Multi Sensor Systems; new areas of MMS applications; and emerging intelligent processing techniques. These areas of research will be the bases for the future of this working group.
New Developments in Mobile Multi-Sensor Systems
The emergence of micro-electromechanical systems (MEMS) technology promises substantial developments for MEMS. Since INS technology is capable of working in all environments where GPS is inadequate, MEMS inertial technology is seen as both a possible complement of GPS technology and a potential alternative to GPS given sufficient market demand. With MEMS being an enabling technology with wide application, the global market of MEMS is estimated to grow to US$ 140 billion in 2002. Consequently, the concept of a low-cost 'IMU on a chip' with unit costs as low as GPS module cost is discussed in some quarters. It is envisioned that the costs of MMS can be greatly reduced to a commercially acceptable level.
The integration of GPS with low-cost IMUs has already reached the product stage. Such systems can be used as highly reliable navigation systems, giving position, velocity, and altitude with high accuracy, or as a geo-referencing systems for various imaging sensors (optical or digital cameras, laser scanners, multi-spectral scanners).
The accuracy and resolution of sensors is constantly increasing. Integration of low-cost sensors has also made considerable advances. The use of vision-based systems in conjunction with low-cost position and attitude sensors may become a viable alternative in cities if current advances in enabling technologies continue. Specifically the development of digital cameras on a C-MOS chip, the availability of inexpensive storage devices of Gigabyte capacity, and the increase in microprocessor speeds up to 1000 MHz will contribute to the emergence of low-cost, high accuracy vision-based systems
New Applications for Mobile Multi-Sensor Systems
MMS land applications are currently being extended to other applications. Typical examples are the Airborne Integrated Mapping System developed at The Ohio State University Center for Mapping and the Dual-Camera Backpack System developed by the University of Calgary. All these systems use DGPS and INS to georeferenced the digital CCD images. These systems will greatly reduce or eventually eliminate the need for ground control and aerial triangulation.
Emerging Processing Techniques for Mobile Multi-Sensor Systems
Advanced techniques such as neural networks and snake models are currently under development to automate the measuring procedures and automatic object recognition from mobile mapping data. The two unique advantages of MMS-generated data:
- images have unknown exterior orientation parameters, and
- the image sequences are along a known path
These two advantages makes the automation of object recognition and measuring procedures more efficient and robust. The science of multiple image based matching has found its application in the mobile mapping processing. Bayesian networks have been actively researched and promise great potential for feature extraction. Real-time automation, in our opinion, remains to be the future of processing techniques for MMS.
I am indebted to all colleagues and members of the Working Group for their tremendous cooperation and contribution to the work of the working group. A special thanks goes to Profs. K.P. Schwarz and H. Kahmen for giving me the chance to chair this working group and Dr. H. Kinawi for critiquing this report.
Ash, T., Silva, R., Brown, A., and LaMance, J. (1995), "GPS/Inertial Mapping (GIM) System for Real Time Mapping of Roadways Using WADGPS", Proceedings of the Institute of Navigation, ION GPS-95, Palm Springs, CA, USA, September 12-15, pp. 1825-1830.
Baraniak, D. W. (1994), "Creating High Accuracy Digital Maps Using GPS and Stereo Digital Imaging Technologies", Wisconsin Professional Engineer, Published by The Wisconsin Society of Professional Engineers, May/June 1994, Vol. 35, No. 4, pp. 12-14.
Benning, W. Mobile mapping by a car driven survey system (CdSS), The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt , Vienna, Austria, April 20 to April 23, 1998.
Blaho, G. and Toth, C. (1995), "Field Experience With a Fully Digital Mobile Stereo Image Acquisition System", The Mobile Mapping Symposium, Columbus, OH, USA, May 24-26, 1995, pp. 97-104.
Bossler J. D, Goad, C. and Novak, K. (1993), "Mobile Mapping Systems; New Tools for the Fast Collection of GIS Information", GIS'93, Ottawa, Canada, pp. 306-315, March 23-35.
Cramer, M., J. Skaloud, K.P. Schwarz (1996). Exterior Orientation by Direct Measurement of Camera Position and Attitude. XVIII. ISPRS Congress, Vienna, July 9-19, 1996, Int. Archives of Photogrammetry and Remote Sensing, vol. 31, part B3, 125-130.
Dorota A. Grejner-Brzezinska and Charles Toth, Direct Platform Orientation in Aerial Surveying Practice, the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999
El-Sheimy N. and Schwarz K. P., "A Unified Approach to Multi-sensor Integration in Photogrammetry", Invited paper, ISPRS Joint Workshop on Integrated Acquisition and Interpretation of Photogrammetric Data, November 8-10, 1995, Stuttgart, Germany.
El-Sheimy, N., "The Development of VISAT - A Mobile Survey System for GIS Applications", Ph.D. Thesis, Department of Geomatics Engineering, The University of Calgary, UCGE Report No. 20101.
El-Sheimy, N., Schwarz, K.P. , Lavigne, M., "Expert Knowledge System for Mobile Mapping Applications", The International Symposium on Kinematic Systems in Geodesy, Geomatics and Navigation, KIS97, Banff, Canada, June, 1997.
El-Sheimy, N. and M. Lavigne, "3-D GIS Data Acquisition Using GPS/INS/Video Mobile Mapping System", The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt , Vienna, Austria, April 20 to April 23, 1998.
El-Sheimy, N. and Schwarz K.P., " Navigating Urban Areas by VISAT - A Mobile Mapping System Integrating GPS/INS/Digital Cameras for GIS Applications", the USA Institute of Navigation Journal, 1998.
El-Sheimy, Naser, Mike Chapman, and V. Tao, "An Intelligent Mobile Mapping System", the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999
Lohr , U. , TopoSys laserscanner system, The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt , Vienna, Austria, April 20 to April 23, 1998.
Mostafa, M.M.R., K-P Schwarz, and P. Gong (1997). A Fully Digital System For Airborne Mapping. Proceedings of The International Symposium on Kinematic Systems In Geodesy, Geomatics, and Navigation, Banff, Canada, June 3-6, 463-471.
Mostafa, M.M.R., K-P Schwarz, and M.A. Chapman (1998). Development and Testing of an Airborne Remote Sensing Multi-Sensor System. ISPRS-Commission II, Symposium on Data Integration: Systems and Techniques, Cambridge, UK, July 13-17, 1998.
Schwarz K.P. and N. El-Sheimy," Georeferenced Video Images For GIS Applications in Urban Centers", Proceedings of the Workshop "Integrated Sensor Orientation, Theory, Algorithms, and Systems", WICHMAN (ISBN 3-97907-279-05), Ed. Colomina/Navarro, 1995, pp. 78-90.
Schwarz, K.P (1995) , "Integrated Airborne Navigation Systems for Photogrammetry" The Wichmann Verlag, ISBN 3-87907-277-9, Photogrammetric week’95, Stuttgart, Oberkochen, Germany, pp. 139-153.
Schwarz, K.P. and El-Sheimy, N., (1996), "Kinematic Multi-sensor Systems For Close Range Digital Mapping" , ISPRS 1996, Commission V, Working Group III, Vienna, Austria, July 9-19, 1996, Invited Paper.
Schwarz, K.P. and J. Sun (1998). Development and Testing of a Real-Time DGPS/INS Integrated System. ISPRS-Commission II, Symposium on Data Integration: Systems and Techniques, Cambridge, UK, July 13-17, 1998 (in print).
Skaloud, J., Y.C. Li and K.P. Schwarz (1997). Airborne Testing of a C-MIGITS II - Low Cost Integrated GPS/INS. International Symposium on Kinematic Systems in Geodesy, Geomatics and Navigation - KIS97, Banff, Alberta, Canada, June 3-6.
Skaloud, J. and K.P. Schwarz (1998). Accurate Orientation for Airborne Mapping Systems. ISPRS-Commission II, Symposium on Data Integration: Systems and Techniques, Cambridge, UK, July 13-17, 1998.
Sternberg, H., W. Caspary, H. Heister Determination of the trajectory surveyed by the mobile surveying system KISS, The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt , Vienna, Austria, April 20 to April 23, 1998.
Sternberg H., Filter Algorithms for Optical Determination of Position and Attitude of the Mobile Mapping System Kiss, the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999
Tao, C. , M. A. Chapman, N. El-Sheimy, and B. Chaplin, Towards Automated Processing of Mobile Mapping Image Sequences, the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999.
Ron, Li, Concept of Processing Multiplatform and Multisensor Data, the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999.
Retscher, G., Modifications and adaptations of an on-line rail-track evaluation concept, The International Symposium on Kinematic Systems In Geodesy, Geomatics and Navigation, Eisenstadt , Vienna, Austria, April 20 to April 23, 1998.
Retscher, Guenther and Heribert Kahmen Precise 3-D Navigation of Building Machine Platforms, the International Workshop on Mobile Mapping Technology, Bangkok, Thailand, April 21-23, 1999.
Reid, D.B. and E. Lithopoulos (1998), High Precision Pointing System for Airborne Sensors. Proc. IEEE PLANS Symposium, Palm Springs, USA, April 20-23, 1998, 303-308.