This paper deals with the design of a multi-modal system for pervasive context-aware service provision and humanenvironment interaction in augmented environments by the use of Personal Digital Assistants (PDA) or SmartPhones. The system enables mobile devices and remote displays to perform as interaction devices with pervasive applications which run on a dynamically composed server network. Visual interaction for service setup and provision are driven by appropriate graphical interfaces and XML-based protocols, which are dynamically composed according to the type of service and to the user current position by means of a mobile agent-based framework. The paper discusses both protocols, hardware and software system components. The first part of the document gives a general description of the system, which is managed by an entity-driven organization in augmented reality. The mobile and reference devices of the system framework are then discussed, along with the mobile agent software whic...