A major difficulty for anomaly detection lies in discovering boundaries between normal and anomalous behavior, due to the deficiency of abnormal samples in the training phase. In this paper, a novel coevolutionary algorithm which attempts to simulate territory establishment in ecology is conceived to tackle anomaly detection problems. Two species in normal and abnormal behavior pattern space coevolve competitively and cooperatively. Competition prevents individuals in one species from invading the other’s territory; cooperation aims to achieve complete pattern coverage by adjusting the evolutionary environment according to the pressure coming from neighbors. In a sense, we extend the definition of cooperative coevolution from “coupled fitness” to “interaction of the evolutionary environment”. This coevolutionary algorithm, enhanced with features like niching inside of species, global and local fitness, and fuzzy sets, tries to balance overfitting and overgeneralization....