A Direct and Accurate Adaptive Semi-Lagrangian Scheme for the Vlasov-Poisson Equation