In this paper, we discuss challenges that are specific to testing of open IoT software systems. The analysis reveals gaps compared to wireless sensor networks as well as embedded software. We propose a testing framework which (a) supports continuous integration techniques, (b) allows for the integration of project contributors to volunteer hardware and software resources to the test system, and (c) can function as a permanent distributed plugtest for network interoperability testing. The focus of this paper lies in open-source IoT development but many aspects are also applicable to closed-source projects. Categories and Subject Descriptors D.2.5 [Testing and Debugging]: Testing tools, Diagnostics, Distributed debugging; B.8.1 [Performance and Reliability]: Reliability, Testing, and Fault-Tolerance; H.3.4 [Systems and Software]: Distributed systems Keywords open-source IoT, interoperability, test system architecture