Traditionally, clustering problems are investigated under the assumption that all objects must be clustered. A shortcoming of this formulation is that a few distant objects, called outliers, may exert a disproportionately strong influence over the solution. In this work we investigate the k-min-sum clustering problem while addressing outliers in a meaningful way. Given a complete graph G = (V, E), a weight function w : E IN0 on its edges, and p IN0 a penalty function on its vertices, the penalized k-min-sum problem is the problem of finding a partition of V to k + 1 sets, S1, . . . , Sk+1, minimizing k i=1 w(Si) + p(Sk+1), where for S V w(S) = e={i,j}S we, and p(S) = iS pi. Our main result is a randomized approximation scheme for the metric version of the penalized 1-min-sum problem, when the ratio between the minimal and maximal penalty is bounded. For the metric penalized k-min-sum problem where k is a constant, we offer a 2-approximation.