This paper investigates one of the existing methods for measuring usability – Logic Scoring Preference (LSP), and discusses the results of two formal experiments carried out to assess the extent to which LSP embodies the subjective perception of users in regards to Web usability. The two experiments used Computer Science students as experimental subjects. Our results suggest that scores obtained via LSP are significantly different from scores obtained via subjective opinion. In addition, we obtained contradictory results when investigating the consistency of LSP scores across subjects.