An extended algorithm of the relative reward strength algorithm is proposed. It is shown that the proposed algorithm ensures the convergence with probability 1 to the optimal path under the certain type of nonstationary environment. Several computer simulation results confirm the effectiveness of the proposed algorithm.