This paper presents a new photometric stereo method aiming to efficiently estimate BRDF and reconstruct glossy surfaces. Rough specular surfaces exhibit wide specular lobes under different lightings. They are ubiquitous and usually bring difficulties to both specular pixel removal and surface normal recovery. In our approach, we do not apply unreliable highlight separation and specularity estimation. Instead, an important visual cue, i.e. the cast shadow silhouette of the object, is employed to optimally recover global BRDF parameters. These parameter estimates are then taken into a reflectance model for robustly computing the surface normals and other local parameters using an iterative optimization. Within the unified framework, our method can also be used to reconstruct object surfaces assembled with multiple materials.