This paper presents a cascaded scheme with block-based and pixel-based codebooks for background subtraction. The codebook is mainly used to compress information to achieve high efficient processing speed. In the block-based stage, 12 intensity values are employed to represent a block. The algorithm extends the concept of the Block Truncation Coding (BTC), and thus it can further improve the processing efficiency by enjoying its low complexity advantage. In detail, the blockbased stage can remove the most noise without reducing the True Positive (TP) rate, yet it has low precision. To overcome this problem, the pixel-based stage is adopted to enhance the precision, which also can reduce the False Positive (FP) rate. Moreover, this study also presents a color model and a match function which can classify an input pixel as shadow, highlight, background, or foreground. As documented in the experimental results, the proposed algorithm can provide superior performance to that of the former ...