The correspondence between a (96, 20, 4) symmetric design having regular automorphism group and a difference set with the same parameters has been used to obtain difference sets in groups of order 96. Starting from eight such symmetric designs constructed by the tactical decomposition method, 55 inequivalent (96, 20, 4) difference sets are distinguished. Thereby the existence of difference sets in 22 so far undecided nonabelian groups of order 96 is proved.