Ulcerative colitis is a chronic disease characterized by bleeding and ulcers in the colon. Currently, the gastroenterologist reviews the colonoscopy video to assess the disease severity using an endoscopic score. This task is time-consuming and does not consider the size and the number of lesions. Consequently, automatic detection methods were proposed enabling fine-grained assessment of lesion severity. However, they depend on the quality of the training set, and its specificity to the application context. To suit the local clinical setup, we opted for an internal training dataset containing only rough bounding box annotations around lesions. Color information is the primary indicator used by specialists to recognize the lesions. Thus, we propose to use linear models in suitable color spaces to detect lesions. We introduce an efficient sampling scheme for exploring the set of linear classifiers and removing trivial models i.e. those showing zero false negative or positive ratio. Using bounding boxes leads to ex- aggerated false negative/positive ratios due to mislabeled pixels, especially in the corners, resulting in decreased models’ accuracy. Therefore, we propose to evaluate the model sensitivity on the annotation level instead of the pixel level. Our sampling strategy can eliminate up to 25% of trivial models. Despite the limited annotations’ quality, the detectors achieved good performance (93% specificity/89% sensitivity for bleeding and 57% specificity/83% sensitivity for ulcers). The best models exhibit low variability when tested on a small subset of endoscopic images. However, the inter-patient model performance was variable suggesting that appearance normalization is critical in this context.