Web Quality of Experience (QoE) monitoring is a critical task for Internet Service Providers (ISPs), especially due to the key role played by customer experience in churn management. Previously, we have tackled the problem of Web QoE inference from the ISP perspective, relying on passive measurement of encrypted network traffic and machine learning models. In this paper, we exploit the broad heterogeneity of contents embedded in web pages to improve the state of the art performance in Web QoE inference, relying on web-content learning model tailoring. By analyzing the top-500 most popular web pages of the Internet through unsupervised learning, we discover different web page content classes which realize significantly different Web QoE inference performance. We train supervised learning inference models separately for each of these classes, using the well-known Speed Index (SI) metric as proxy to Web QoE. Empirical evaluations on a large corpus of Web QoE measurements for top popular websites demonstrate that our combined content-tailored approach improves the inference performance of the SI by almost 30% with respect to previous single-model approaches, reducing the QoE inference error in terms of mean opinion scores by more than 40%.