We consider the problem of decomposing polygons (with holes) into various types of simpler polygons. We focus on the problem of partitioning a rectilinear polygon, with holes, into rectangles, and show an (n log n) lower bound on the timecomplexity. The result holds for any decomposition, optimal or approximative. The bound matches the complexity of a number of algorithms in the literature, proving their optimality and settling the complexity of approximate polygon decomposition in these cases. As a related result we show that any non-trivial approximation algorithm for the minimum gap-problem requires (n log n) time. Key words: Lower bounds, Minimum gap, Polygon decomposition, Algebraic decision trees, Computational Geometry