Motivation
The LifeClipper2 system is a prototype application for “Outdoor Augmented Reality”. A quintessential use of the system, tested during the course of the project, is to assess architectural and urban (landscape) planning.
The motivation for observing and questioning persons who used the system was as follows: to principally assess the acceptance of the system in an authentic urban planning context and identify other potential areas of deployment, but also shortcomings in the system.

Evaluation
The evaluation was undertaken with a total of 13 persons who used the system for approximately 45 minutes. The test subjects were asked to observe the planning model in augmented reality mode, choose a path through the model and then retrace the steps taken by indicating them on a plan. Moreover, the test subjects were requested to comment on their actions (think aloud); the statements were noted and additionally recorded on tape.
The test was concluded with a questionnaire (in German) that mainly considered three aspects:
a) Effectiveness: how well could the users gain an impression of the planning and how was the real-world experience?
b) Technology: how mature/usable is the technology deemed to be?
c) Usability: how well can the test subjects imagine assessing planning solely on the basis or AR, or using AR presentations for specific activities, as opposed to conventional planning views?

The evaluation was undertaken for the redesign of the Basle Rheinhafen, based on the project proposal “Undine“.

The test was conducted with the help of the current hardware settings. From the software point of view user interaction with a mouse or menu guidance was dispensed with in favour of a walk through in the AR model.

Findings
All users were able to complete the task and took full advantage of the allotted time to gain familiarisation with the system. The answers on the questionnaire were mostly in the range “Generally true“ (between 3.6 and 4.0 on a scale of 1 to 5), both for the quality of the effectiveness of the system and for the technical implementation. The potential uses for the system were, by and large, rated better (4.2) than the quality of the technical implementation (3.6).
In addition, it was clear that users with prior knowledge of the project competition (knowledge of the original site or planning documentation), rated the quality of the technical implementation more critically than users without prior knowledge of the location or planning (3.0 compared with 3.9), whilst the assessment of potential usability of the system is similar.
Statements emanating from the users during the tour (think aloud) were mostly to do with the technical shortcomings of the prototype model. In particular, problems with the system involving excessive tracking and reflection in the Head Mounted Display (HMD) when sunlight was streaming in from the side were mentioned. By comparison, qualitative statements concerning the scenery presented were less common.

Conclusions
It was established, in principle, that in spite of the prevailing technological shortcomings, the acceptance of the overall system and the assessment of the potential usability is extremely high. As a result, the possible uses for the system were rated highly by the users. This is even true in spite of difficult, external conditions such as the effects of the weather (strong sunlight, heat), the weight and size of the hardware prototype (7 kg) and the duration of the tour, approx. 45 minutes, and choice of test persons etc. A similar study on the general acceptance of a complex AR system in the past five years is not known to have taken place.
Although the prototype technology was generally rated positive and was not considered to be detrimental to the high acceptance of the overall system, it is not fully transparent for the user coming into contact with the equipment for the first time. This is the case even though the task presented (finding the way through the terrain) contained no predominantly technical aspects. It was almost exclusively spontaneous statements about the technological implementation of the LifeClipper2 system that were made by the users, whilst qualitative comments on the planning, in other words on the contents of the “ArchiViz“ scenario, were fewer or virtually non-existent.

Future prognosis
Use of the system at present is only possible when accompanied by a person who can intervene in dangerous traffic situations or if difficult terrain is involved. Modifications to the system so that a change can be made simply and rapidly to the real world thus allowing the user to travel alone are desirable.
The qualitative assessment of a park, as in this particular case, is generally challenging, regardless of the technology involved; to find a concrete task for an appropriate setting for evaluation purposes, on the other hand, is not easy. The challenge lies, among other things, in the size of the object to be assessed, in this case the park. For obvious reasons, therefore, an evaluation using smaller objects would not be out of place.
During the evaluation it became evident that in the retrospective assessment (a few days after the walk through) it was difficult for the test subjects to differentiate between real and virtual planning objects. It would seem that during the walk through the test subjects had little difficulty in being able to make this differentiation yet, in retrospect, were unable to distinguish between the real and virtual objects.