This paper presents a hybrid BIST architecture and methods for optimizing it to test systems-on-chip in a cost effective way. The proposed self-test architecture can be implemented either only in software or by using some test related hardware. In our approach we combine pseudorandom test patterns with stored deterministic test patterns to perform core test with minimum time and memory, without losing test quality. We propose two algorithms to calculate the cost of the test process. To speed up the optimization procedure, a Tabu search based method is employed for finding the global cost minimum. Experimental results have demonstrated the feasibility and efficiency of the approach and the significant decreases in overall test cost.