This paper proposes a robust real-time, scalable and modular Field Programmable Gate Array (FPGA) based implementation of a spatiotemporal segmentation of video objects. The goal of this work is to translate an existing object segmentation algorithm into hardware to achieve real-time performance. The proposed implementation achieved an optimum processing speed of 133 MPixels/s while utilizing minimal hardware resources. The design was successfully simulated, synthesized and tested for real-time performance on an actual hardware platform which consists of a frame grabber with a user programmable FPGA - Xilinx Virtex-II Pro.