We propose a new method for recovering a 3-D object shape from an image sequence. In order to recover high-resolution relative depth without using the complex Markov random field (MRF) that includes a line process, we construct a recovery algorithm based on a belief propagation scheme using a multi-scale Bayesian network. With this algorithm, relative 3-D motion between a camera and an object can be determined together with relative depth, and the maximum a posteriori expectation-maximization (MAP-EM) algorithm is effectively used to determine a suitable approximation.