This work introduces an interactive robotic system for assistance, conceived to tackle some of the challenges that domestic environments impose. The system is organized into a network of heterogeneous components that share both physical and logical functions to perform complex tasks. It consists of several robots for object manipulation, an advanced vision system that supplies in-formation about objects in the scene and human activity, and a spatial augmented reality interface that constitutes a comfortable means for interacting with the system. A first analysis based on users’ experiences confirms the importance of having a friendly user interface. The inclusion of context awareness from visual perception enriches this interface allowing the robotic system to become a flexible and proactive assistant.