An ambient user interface is a set of hidden intelligent interfaces that recognize user's presence and provides services to immediate needs. There are several research activities on user interfaces and interactions which combining VR/AR, ubiquitous computing/ambient interfaces, and artificial intelligence. However, real-time and intelligent responses of user interfaces are still challenging problems. In this paper, we introduce the design of Context-aware Cognitive Agent Architecture (CCAA) for real-time and intelligent responses of ambient user interfaces in ubiquitous virtual reality, and discuss possible scenarios for realizing ambient interfaces. CCAA applies a vertically layered twopass agent architecture with three layers. The three layers are AR (augmented reality) layer, CA (context-aware) layer, and AI layer. The two passes interconnect the layers as an input or output. One of the passes of each layer is an input path from a lower layer or environmental sensors describing...