Using statistical textons for texture classification has shown great success recently. The maximal response 8 (MR8) method, which extracts an 8-dimensional feature set from 38 filters, is one of state-of-the-art rotation invariant texture classification methods. However, this method has two limitations. First, it require a training stage to build a texton library, thus the accuracy depends on the training samples; second, during classification, each 8-dimensional feature is assigned to a texton by searching for the nearest texton in the library, which is time consuming especially when the library size is big. In this paper, we propose a novel texton feature, namely Binary Filter Response Pattern (BFRP). It can well address the above two issues by encoding the filter response directly into binary representation. The experimental results on the CUReT database show that the proposed BFRP method achieves better classification result than MR8, especially when the training dataset is limited...