There are many different implementation approaches to realize the vision of feature oriented software development, ranging from simple preprocessors, over feature-oriented programming, to sophisticated aspect-oriented mechanisms. Their impact on readability and maintainability (or program comprehension in general) has caused a debate among researchers, but sound empirical results are missing. We report experience from our endeavor to conduct experiments to measure the influence of different implementation mechanisms on program comprehension. We describe how to design such experiments and report from possibilities and pitfalls we encountered. Finally, we present some early results of our first experiment on comparing CPP with CIDE. Categories and Subject Descriptors. D.2.13 [Software Engineering]: Resuable Software; D.3.3 [Software Engineering]: Language Constructs and Features General Terms. Experimentation, Human Factors, Languages Keywords. Program Comprehension, Empirical Softwar...