We present an approach for nonlinear optimization of the parameters of an endoscopic camera mounted on a surgery robot. The goal is to generate a depth map for each image in order to enhance the quality of medical light fields. The pose information provided by the robot is used as an initialization, where especially the orientation is inaccurate. Refinement of intrinsic and extrinsic camera parameters is performed by minimizing the back-projection error of 3-D points that are reconstructed by triangulation from image features tracked over an image sequence. Optimization of the camera parameters results in an enhancement of rendering quality in two ways: More accurate parameters lead to better interpolation as well as to better depth maps for approximating the scene geometry.