This report documents the program and the outcomes of Dagstuhl Seminar 12491 “Interpreting Observed Action”. The aim of the seminar was to get a coherent picture, which transcends the borders of applications and disciplines, of existing approaches and problems in interpreting observed action in semantic terms – primarily action by humans, but action by artificial agents may play some role, too. The seminar brought together, on the one hand, researchers from the different camps of AI, robotics, and knowledge-based systems who are working on the various aspects and purposes of interpreting observed action by humans, or occasionally, other agents; on the other hand, it added some researchers from cognitive science (psychology, neurosciences) working on human perception of behaviour and action. The main outcome of the seminar were a set of guidelines for setting up a workbench, which can be used to explore and test methods and techniques related to interpreting observed action.