This paper presents a complete modeling system that extracts complex building structures with irregular shapes and surfaces. Our modeling approach is based on the use of airborne LiDAR which offers a fast and effective way to acquire models for a large urban environment. To verify and refine the reconstructed ragged model, we present a primitive-based model refinement approach that requires minor user assistance. Given the limited user input, the system automatically segments the building boundary, does the model refinement, and assembles the complete building model. By adapting a set of appropriate geometric primitives and fitting strategies, the system can model a range of complex buildings with irregular shapes. We demonstrate this system’s ability to model a variety of complex buildings rapidly and accurately from LiDAR data of the entire USC campus.