In this paper we present a sketch modelling system which we call Stilton. The program resembles a desktop VRML browser, allowing a user to navigate a 3D model in a perspective projection, or panoramic photographs, which the program maps onto the scene as a `floor' and `walls'. We place an imaginary 2D drawing plane in front of the user, and any geometric information that user sketches onto this plane may be reconstructed to form solid objects through an optimisation process. We show how the system can be used to reconstruct geometry from panoramic images, or to add new objects to an existing model. While panoramic imaging can greatly assist with some aspects of site familiarisation and qualitative assessment of a site, without the addition of some foreground geometry they offer only limited utility in a design context. Therefore, we suggest that the system may be of use in `just-intime' CAD recovery of complex environments, such as shop floors, or construction sites, by...