This paper describes the detection of rust defects on highway steel bridges, which are one of the most commonly observed defects on coating surfaces and thus have to be taken care of appropriately since they severely affect the structural integrity of bridges. A rust defect assessment method is presented that automatically detects the percentage of rust in a given digital image of bridge surface taken from a conventional digital camera. A training and detection algorithm is implemented to classify a given block of the image as rust or non-rust. The results of the algorithm are analyzed for its efficiency and possible optimization techniques are suggested.