A proof of Alon's second eigenvalue conjecture and related problems