In this work, we propose a design for a user attention model featuring three core components. Our system components can work in real-time, offering indications of user attention from different sensory inputs (both visual and neurophysiological). Intention of the current work is to keep the equipment as unintrusive as possible, while keeping the confidence of the inputs as high as possible. We discuss potential applications of such a system, particularly with respect to evaluating user attentive behaviour during human-agent interactions and as a more natural interface for interacting with agents.