In this paper I will try to explain the nature of document understanding in all of its dimensions. Therefore I will first describe the characteristics of data, knowledge, and information in order to describe their synergetic interweaving. After that I will try to structure the inherent complexity of sub-problems of document understanding which may not be solved serially, but rather are attributes of individual documents. Thus, this paper focuses on system engineering challenges. However, I will show some recent work done on the different topics and give some insights in the individual techniques we chose at DFKI.