Two-dimensional and three-dimensional coordinate systems are the basic graphics symbols in many graphical documents. A robust coordinate system detection scheme is needed in order to further decompose other graphics elements accurately. In this paper, we propose a robust coordinate systems reconstruction approach that combines the geometric information of the coordinate systems with feature clustering and line segments verification techniques in the Hough space. Experiments on both 2D and 3D scanned diagram images show that the proposed approach is robust and applicable in constructing the coordinate systems when comparing with other projection-based coordinate systems detection approaches.