One of the breakout groups lead by John Barton from IBM focused on evaluation and challenges for ubicomp. The session focused on a conference scenario outlined by Alan Messer [ref] to focus he group on systems challenges and evaluation in this specific context. First the group brainstormed to better understand the scenario, and the open challenges. The system was understood to be an extension of an existing conference management system like EDAS with additional sensors and the use of mobile devices to automate a conference.

Challenges were roughly categorized into organizational or management challenges, and technical challenges. Organizational and management challenges included:

• regarding the portability of the system: can it be set up quickly and easily in a new conference location?
• Does it have a good cost-benefit?
• What is the overall quality of the user experience?
• Is the development not only a technical one, but also a social or political exercise?

Technical challenges included:

• indoor location sensing
• flash crowds where 300 people are using RFID and other wireless technologies in close range, lots of devices
• interoperability
• small and large screen user interface challenges
• issues of distraction – the system should be easy and fun, not difficult to use
• Testing, and set up, debugging
• Should the system be autonomic?
• Security – should easily allow conference participants, but not others to use it.
• Robustness in real time
• Power management
• Physical search problem, distributed electronic search across devices

Based on these challenges, a high level framework for evaluation was constructed. These consisted of evaluating users opinion of the system overall (reputation), how well it included all users at the conference (inclusiveness). To evaluate, a good feedback mechanism is required.

Within this evaluation framework, smaller scale features of the system were considered for a team to evaluate their success. Specifically the group focused on evaluating their indoor location system. This system would be involved in the following sub-scenarios involving the conference application “I need to meet with Roy after the workshop – but what workshop is he in?” and “Roy and three others want to meet after the meeting”, or answer the question for Roy “Where is my panel discussion?”

To evaluate the system, the group needed to compare it to a baseline, and specifically a less technical solution such as the use of a loudspeaker paging system. When comparing it to this type of system, the group realized that “noise” or “annoyance” level may be a way of evaluating the location system.

In a similar scenario such as “Find a projector for the workshop” one could compare a technical solution and compare it to a non technical solution involving searching by a number of people, and the reliability of actually finding the projector.

The group also discussed how a contractor may evaluate the system: how easy is it to make money installing it, time out of the box, set up, use of specialized equipment, mean time to failure, and operator cost.

This could be aggregated into a metric such as the “reputation” of the system. The group as a whole discussed the physical search problem. Can we use google or something similar to accomplish search in the real world? The hard problem is getting information about the physical world in an electronically search able format so that electronic search techniques can be applied.

Overall the discussion concluded that coming up with scenarios is critical for evaluation. From these, we can derive the possible benefits and frustrations of the system, and compare it to non/less technical solutions. It can be useful to come up with variations on a theme or category of scenarios that can be applied in a consistent way across systems for effective evaluation. The group agreed that if it is difficult to come up with scenarios, then evaluation will be a problem, and it may be useful to come up with a taxonomy of scenarios or themes for reuse in evaluation.