Madhavan / Tunstel / Messina Performance Evaluation and Benchmarking of Intelligent Systems

1. Auflage 2009
ISBN: 978-1-4419-0492-8
Verlag: Springer-Verlag
Format: PDF
Kopierschutz: Wasserzeichen (»Systemvoraussetzungen)

E-Book, Englisch, 338 Seiten

To design and develop capable, dependable, and affordable intelligent systems, their performance must be measurable. Scienti?c methodologies for standardization and benchmarking are crucial for quantitatively evaluating the performance of eme- ing robotic and intelligent systems' technologies. There is currently no accepted standard for quantitatively measuring the performance of these systems against user-de?ned requirements; and furthermore, there is no consensus on what obj- tive evaluation procedures need to be followed to understand the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communic- ing results, inter-comparing system performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. Currently, this lack of cohesion in the community hinders progress in many domains, such as m- ufacturing, service, healthcare, and security. By providing the research community with access to standardized tools, reference data sets, and open source libraries of solutions, researchers and consumers will be able to evaluate the cost and be- ?ts associated with intelligent systems and associated technologies. In this vein, the edited book volume addresses performance evaluation and metrics for intel- gent systems, in general, while emphasizing the need and solutions for standardized methods. To the knowledge of the editors, there is not a single book on the market that is solely dedicated to the subject of performance evaluation and benchmarking of intelligent systems.

Autoren/Hrsg.

Madhavan, Raj

Tunstel, Edward

Messina, Elena

Inhaltsverzeichnis

1;Preface;5
2;Contents;12
3;Contributors;14
4;1 Metrics for Multiagent Systems;17
4.1;1.1 Introduction;17
4.2;1.2 Background on Multiagent Systems and Metrics;18
4.2.1;1.2.1 Anatomy of a Multiagent System;19
4.2.2;1.2.2 Types of Metrics;20
4.2.2.1;1.2.2.1 Effectiveness vs Performance;21
4.2.2.2;1.2.2.2 Data Classification;21
4.3;1.3 Metrics;22
4.3.1;1.3.1 Agents and Frameworks;22
4.3.2;1.3.2 Platform;23
4.3.2.1;1.3.2.1 Distributed Systems;23
4.3.2.2;1.3.2.2 Networking;23
4.3.3;1.3.3 Environment/Host Metrics;24
4.3.4;1.3.4 System;25
4.4;1.4 Analysis Framework for Multiagent Systems;26
4.4.1;1.4.1 Selection;27
4.4.2;1.4.2 Collection;28
4.4.3;1.4.3 Application;28
4.5;1.5 Case Study: DCOP Algorithms and DCOPolis;28
4.5.1;1.5.1 Experimental Setup;30
4.5.2;1.5.2 Results and Analysis;31
4.6;1.6 Summary;31
5;References;32
6;2 Evaluation Criteria for Human-Automation PerformanceMetrics;36
6.1;2.1 Introduction;36
6.2;2.2 Generalizable Metric Classes;37
6.3;2.3 Metric Evaluation Criteria;39
6.3.1;2.3.1 Experimental Constraints;40
6.3.2;2.3.2 Comprehensive Understanding;41
6.3.3;2.3.3 Construct Validity;42
6.3.4;2.3.4 Statistical Efficiency;43
6.3.5;2.3.5 Measurement Technique Efficiency;44
6.4;2.4 Metric Costs vs. Benefits;44
6.4.1;2.4.1 Example 1: Mental Workload Measures;46
6.4.1.1;2.4.1.1 Performance Measures;47
6.4.1.2;2.4.1.2 Subjective Measures;49
6.4.1.3;2.4.1.3 Physiological Measures;49
6.4.2;2.4.2 Example 2: Attention Allocation Efficiency Measures;50
6.5;2.5 Discussion;51
7;References;53
8;3 Performance Evaluation Methods for Assistive RoboticTechnology;56
8.1;3.1 Introduction;56
8.2;3.2 Assistive Robotic Technologies;58
8.2.1;3.2.1 Autism Spectrum Disorders (ASD);58
8.2.1.1;3.2.1.1 End-User Evaluations;59
8.2.1.2;3.2.1.2 Discussion;60
8.2.2;3.2.2 Eldercare;60
8.2.2.1;3.2.2.1 End-User Evaluations;61
8.2.2.2;3.2.2.2 Discussion;61
8.2.3;3.2.3 Stroke Rehabilitation;62
8.2.3.1;3.2.3.1 End-User Evaluations;62
8.2.3.2;3.2.3.2 Discussion;63
8.2.4;3.2.4 Intelligent Wheelchairs;63
8.2.4.1;3.2.4.1 End-User Evaluations;64
8.2.4.2;3.2.4.2 Discussion;64
8.2.5;3.2.5 Assistive Robotic Arms;65
8.2.5.1;3.2.5.1 End-User Evaluations;65
8.2.5.2;3.2.5.2 Discussion;66
8.2.6;3.2.6 External Limb Prostheses;66
8.2.6.1;3.2.6.1 End-User Evaluations;67
8.2.6.2;3.2.6.2 Discussion;68
8.3;3.3 Case Studies;68
8.3.1;3.3.1 Designing Evaluations for an Assistive Robotic Arm;68
8.3.2;3.3.2 Designing Evaluations for Socially Assistive Robots;70
8.4;3.4 Incorporating Functional Performance Measures;71
8.5;3.5 Conclusions;74
9;References;76
10;4 Issues in Applying Bio-Inspiration, Cognitive Critical Mass and Developmental-Inspired Principles to Advanced Intelligent Systems;82
10.1;4.1 Introduction;82
10.2;4.2 The Increased Relevance of Bio-understanding;84
10.3;4.3 Cognitive Critical Mass and Cognitive Decathlon;87
10.3.1;4.3.1 Episodic Memory;90
10.3.2;4.3.2 Theory-of-Mind;91
10.3.3;4.3.3 Self-Awareness;92
10.4;4.4 Different Directions and Levels for Bio-inspiration;94
10.5;4.5 Developmental Robotics Methods;96
10.6;4.6 A Scientific Framework for Developmental Robotics;97
10.7;4.7 Developmental Principles Within an Embodied, Interactive Model;101
10.8;4.8 Summary;102
11;References;104
12;5 Evaluating Situation Awareness of Autonomous Systems;108
12.1;5.1 Introduction;108
12.2;5.2 Autonomous Agents and Situation Awareness;110
12.3;5.3 Criteria for Situation Awareness;111
12.3.1;5.3.1 Awareness of Ignorance;112
12.3.2;5.3.2 Model of Perception Abilities;113
12.3.3;5.3.3 Model of Information Relevance;114
12.3.4;5.3.4 Model of Information Dynamics;116
12.3.4.1;5.3.4.1 Spatial Information Dynamics;117
12.3.4.2;5.3.4.2 Temporal Information Dynamics;117
12.3.5;5.3.5 Spatio-Temporal Qualification;118
12.3.6;5.3.6 Information Sharing;119
12.4;5.4 Maintaining Situation Awareness;120
12.5;5.5 Application Scenario;122
12.5.1;5.5.1 Experimental Setting;122
12.5.2;5.5.2 Results;123
12.6;5.6 Discussion;123
12.7;5.7 Conclusion;124
13;References;125
14;6 From Simulation to Real Robots with Predictable Results: Methods and Examples;127
14.1;6.1 Introduction;127
14.1.1;6.1.1 Methodology for Algorithm Development;128
14.1.2;6.1.2 A Brief History of USARSim;129
14.2;6.2 Robot Platform Validation;130
14.3;6.3 Sensor Validation;135
14.3.1;6.3.1 Laser Range Finder;136
14.3.2;6.3.2 Global Positioning System;138
14.4;6.4 Algorithm Development;141
14.4.1;6.4.1 Criticisms and Advantages;143
14.5;6.5 Competitions;147
14.6;6.6 Conclusion;148
15;References;149
16;7 Cognitive Systems Platforms using Open Source;152
16.1;7.1 Introduction;152
16.1.1;7.1.1 The Rats Life Benchmark;153
16.1.2;7.1.2 The iCub Platform;153
16.1.3;7.1.3 The Swarm Platform of the Replicatorand SYMBRION Projects;153
16.2;7.2 The Rats Life Benchmark: Competing Cognitive Robots;154
16.2.1;7.2.1 Motivation;154
16.2.2;7.2.2 Existing Robot Competitions and Benchmarks;154
16.2.3;7.2.3 Rat's Life Benchmark: Standard Components;155
16.2.3.1;7.2.3.1 The e-puck mobile robot;155
16.2.3.2;7.2.3.2 LEGO bricks;156
16.2.3.3;7.2.3.3 The Webots robot simulation software;156
16.2.4;7.2.4 Rat's Life Benchmark Description;156
16.2.4.1;7.2.4.1 Software-only Benchmark;156
16.2.4.2;7.2.4.2 Configuration of the Maze;157
16.2.4.3;7.2.4.3 Virtual Ecosystem;157
16.2.4.4;7.2.4.4 Robotics and AI Challenges;158
16.2.5;7.2.5 Evolution of the Competition over Time;158
16.2.6;7.2.6 Discussion;160
16.3;7.3 The Open Source Humanoid Robot Platform iCub;161
16.3.1;7.3.1 The iCub;161
16.3.2;7.3.2 Mechanics;163
16.3.3;7.3.3 The Software: YARP;164
16.3.4;7.3.4 Research with the iCub;165
16.4;7.4 The iCub Simulator;167
16.4.1;7.4.1 Physics Engine;167
16.4.2;7.4.2 Rendering Engine;168
16.4.3;7.4.3 YARP Protocol for Simulated iCub;168
16.4.4;7.4.4 iCub Body Model;168
16.4.5;7.4.5 Simulator Testing and Further Developments;169
16.5;7.5 Symbiotic Robot Organisms: Replicator and SYMBRION Projects;170
16.5.1;7.5.1 Introduction;170
16.5.2;7.5.2 New Paradigm in Collective Robotic Systems;171
16.5.3;7.5.3 Example: Energy Foraging Scenario;173
16.5.4;7.5.4 Hardware and Software Challenges;174
16.5.5;7.5.5 Towards Evolve-Ability and Benchmarking of Robot Organisms;175
16.5.5.1;7.5.5.1 Bio-inspired/Bio-mimicking Approach;176
16.5.5.2;7.5.5.2 Engineering-Based Approach;176
16.5.6;7.5.6 Discussion;177
16.6;7.6 Other Projects and Future Work;177
17;References;179
18;8 Assessing Coordination Demand in Cooperating Robots;182
18.1;8.1 Introduction;182
18.1.1;8.1.1 Coordination Demand;183
18.2;8.2 Coordination Demand;184
18.2.1;8.2.1 Experimental Plan;185
18.3;8.3 Simulation Environment;186
18.3.1;8.3.1 MrCS---The Multirobot Control System;186
18.4;8.4 Experiment 1;187
18.4.1;8.4.1 Results;188
18.4.1.1;8.4.1.1 Human Interactions;189
18.5;8.5 Experiment 2;190
18.5.1;8.5.1 Procedure;191
18.5.2;8.5.2 Results;191
18.6;8.6 Experiment 3;192
18.6.1;8.6.1 Experimental Design;193
18.6.2;8.6.2 Results;193
18.6.2.1;8.6.2.1 Overall Performance;194
18.6.2.2;8.6.2.2 Coordination Effort;195
18.6.2.3;8.6.2.3 Analyzing Performance;196
18.7;8.7 Conclusions;196
19;References;198
20;9 Measurements to Support Performance Evaluation of Wireless Communications in Tunnels for Urban Search and Rescue Robots;200
20.1;9.1 Performance Requirements for Urban Searchand Rescue Robot Communications;200
20.2;9.2 Performance Evaluation Procedures;204
20.3;9.3 Measurement of Signal Impairments in a Tunnel Environment;207
20.3.1;9.3.1 The Test Environment;209
20.3.2;9.3.2 Measurements;210
20.3.2.1;9.3.2.1 Narrowband Received Power;210
20.3.2.2;9.3.2.2 Excess Path Loss and RMS Delay Spread;215
20.3.2.3;9.3.2.3 Tests of Robot Communications;219
20.4;9.4 Modeled Results;221
20.4.1;9.4.1 Single-Frequency Path Gain Models;221
20.4.2;9.4.2 Channel Capacity Model;224
20.5;9.5 Evaluating the Performance of a Robot in a Representative Tunnel Environment;227
20.6;9.6 Conclusion;230
21;References;231
22;10 Quantitative Assessment of Robot-Generated Maps;233
22.1;10.1 Introduction;233
22.2;10.2 Developing Test Scenarios for Robotic Mapping;236
22.2.1;10.2.1 Performance Singularity Identification and Testing;237
22.2.1.1;10.2.1.1 The Maze: Scenarios with Distinct Features;238
22.2.1.2;10.2.1.2 The Tube Maze: Scenarios with Occluded Features;239
22.2.1.3;10.2.1.3 The Tunnel: Scenarios with Minimal Features;240
22.3;10.3 Assessing Objective Performance Using Theoretical Analysis;240
22.3.1;10.3.1 The Case for Statistical Bounds;242
22.3.2;10.3.2 The CRB for Range-Finder Localization;243
22.3.3;10.3.3 The CRB for One-Shot Pose Tracking;243
22.3.4;10.3.4 The CRB for Pose Tracking Over a Trajectory;244
22.3.5;10.3.5 The CRB for Mapping and SLAM;245
22.4;10.4 Evaluating Local Metric Consistency of Robot-Generated Maps Using Force Field Simulation and Virtual Scans;246
22.4.1;10.4.1 Scan Alignment using Force Field Simulation;248
22.4.2;10.4.2 Augmenting Data Using Virtual Scans;248
22.4.3;10.4.3 Map Evaluation Using Virtual Scans;250
22.5;10.5 Evaluating Global Metric Consistency of Robot-Generated Maps;251
22.5.1;10.5.1 Harris-Based Algorithm;252
22.5.1.1;10.5.1.1 Closest Point Matching;252
22.5.1.2;10.5.1.2 Vectorial Space;253
22.5.2;10.5.2 Hough-Based Algorithm;253
22.5.3;10.5.3 Scale Invariant Feature Transform;254
22.5.4;10.5.4 Quality Measure;254
22.5.5;10.5.5 Limitations;257
22.6;10.6 Conclusion;257
23;Reference;258
24;11 Mobile Robotic Surveying Performance for Planetary Surface Site Characterization;261
24.1;11.1 Introduction;261
24.2;11.2 Local Sensor-Based Surveying;262
24.3;11.3 Remote Sensor-Based Surveying;264
24.3.1;11.3.1 Single-Site Remote Sensing Surveys;264
24.3.2;11.3.2 Single-Site Remote Survey Performance;267
24.3.3;11.3.3 Multiple-Site Remote Sensing Surveys;268
24.3.4;11.3.4 Multi-Site Remote Survey Performance;269
24.4;11.4 Characteristic Performance of Mobile Surveys;270
24.5;11.5 Enriching Metrics for Surveys on Planetary Surfaces;271
24.5.1;11.5.1 Consolidated Metric for Human-Supervised Robotic Prospecting ;273
24.5.2;11.5.2 Metrics for Real-Time Assessment of Robot Performance;276
24.5.2.1;11.5.2.1 Considerations for Real-Time Robot Performance Assessment;277
24.6;11.6 Summary and Conclusions;278
25;References;279
26;12 Performance Evaluation and Metrics for Perceptionin Intelligent Manufacturing;281
26.1;12.1 Introduction;281
26.2;12.2 Preliminary Analysis of Conveyor Dynamic Motion;283
26.2.1;12.2.1 Conveyor Motion Data Collection Method;284
26.2.2;12.2.2 Raw Motion Data Collected;285
26.2.3;12.2.3 Statistical Analysis of Linear Accelerations;287
26.2.4;12.2.4 Fast Fourier Transformation Analysis;287
26.2.5;12.2.5 Computed Speed and Position Data;289
26.2.6;12.2.6 Conclusion and Summary;291
26.3;12.3 Calibration of a System of a Gray Value Camera and MDSI Range Camera;291
26.3.1;12.3.1 Cross-Calibration Procedure;292
26.3.2;12.3.2 Experimental Results;294
26.3.3;12.3.3 Conclusion;296
26.4;12.4 Performance Evaluation of Laser Trackers;296
26.4.1;12.4.1 Introduction;296
26.4.2;12.4.2 The ASME B89.4.19 Standard;297
26.4.3;12.4.3 Large-Scale Metrology at NIST;297
26.4.4;12.4.4 Tracker Calibration Examples;298
26.4.5;12.4.5 Sensitivity Analysis;300
26.4.6;12.4.6 Summary;301
26.4.7;12.4.7 Conclusions;302
26.5;12.5 Performance of Super-Resolution Enhancement for LIDAR Camera Data ;302
26.5.1;12.5.1 Methodology;304
26.5.1.1;12.5.1.1 Preprocessing Stage;304
26.5.1.2;12.5.1.2 Triangle Orientation Discrimination (TOD) Methodology;306
26.5.2;12.5.2 LIDAR Camera;307
26.5.3;12.5.3 Data Collection;307
26.5.4;12.5.4 Data Processing;307
26.5.5;12.5.5 Perception Experiment;307
26.5.6;12.5.6 Results and Discussion;308
26.5.6.1;12.5.6.1 Assessment of Registration Accuracy;308
26.5.6.2;12.5.6.2 Triangle Orientation Discrimination Perception Experiment;309
26.5.7;12.5.7 Conclusion;311
26.6;12.6 Dynamic 6DOF Metrology for Evaluating a Visual Servoing Algorithm ;311
26.6.1;12.6.1 Purdue Line Tracking System;312
26.6.2;12.6.2 Experimental Set-up and Results;313
26.6.2.1;12.6.2.1 Stationary Tests;314
26.6.2.2;12.6.2.2 Linear Motion Tests;315
26.6.2.3;12.6.2.3 Shaking Motion Tests;317
26.6.3;12.6.3 Conclusions;318
26.7;12.7 Summary;319
27;References;320
28;13 Quantification of Line Tracking Solutions for Automotive Applications;323
28.1;13.1 Introduction;323
28.2;13.2 Quantifying Line Tracking Solutions;324
28.3;13.3 Experimental Setup and Performance Data Collection Method ;325
28.4;13.4 Quantification Test Cases;326
28.5;13.5 Quantification Results of the Encoder Based Line Tracking Solution;327
28.5.1;13.5.1 Rail Tracking Results;327
28.5.2;13.5.2 Arm Tracking Results;332
28.6;13.6 Quantification Results of the Encoder Plus Static Vision Line Tracking Solution;332
28.7;13.7 Quantification Results of the Analog Laser Based Line Tracking Solution;337
28.7.1;13.7.1 Analog Sensor Based Rail Tracking Results;339
28.7.2;13.7.2 Analog Sensor Based Arm Tracking Results;343
28.8;13.8 Conclusion and Automotive Assembly Applications;347
29;References;350