In this paper we analyze a nonlocal reaction-diffusion model which arises from the modeling of competition of phytoplankton species with incomplete mixing in a water column. The nonlocal nonlinearity in the model describes the light limitation for the growth of the phytoplankton species. We first consider the single species case and obtain a complete description of the long-time dynamical behavior of the model. Then we study the two species competition model and obtain sufficient conditions for the existence of positive steady states and uniform persistence of the dynamical system. Our approach is based on a new modified comparison principle, fixed point index theory, global bifurcation arguments, elliptic and parabolic estimates, and various analytical techniques.