Semantic mapping of static environments has become a hot topic in robotics. The aim of the MERMAID project was to investigate the transfer of a sensor data interpretation approach for mapping to the problem of activity recognition in smart home applications such as elderly care. The basic structure of the semantic mapping approach, i.e., to assemble hypotheses of object aggregates in a closed-loop process of bottom-up raw data interpretation and top-down expectation generation from a domain ontology, can be extended to the temporal domain to include activity interpretation. This paper reports initial results, based on a study using point clouds from depth (RGB-D) sensor data.