We initiate a systematic study of the Row Deletion(B) problem on matrices: Given an input matrix A and a fixed "forbidden submatrix" B, the task is to remove a minimum number of rows from A such that no row or column permutation of B occurs as a submatrix in the resulting matrix. An application of this problem can be found, for instance, in the construction of perfect phylogenies. Establishing a strong connection to variants of the NP-complete Hitting Set problem, we describe and analyze structural properties of B that make Row Deletion(B) NP-complete. On the positive side, the close relation with Hitting Set problems yields constant-factor polynomial-time approximation algorithms and fixed-parameter tractability results.