Objectives This study proposes and evaluates a deep learning method to detect pancreatic neoplasms and to identify main pancreatic duct (MPD) dilatation on portal venous computed tomography (CT) scans. Materials and Methods A total of 2890 portal venous CT scans from 9 institutions were acquired, among which 2185 had a pancreatic neoplasm and 705 were healthy controls. Each scan was reviewed by one in a group of 9 radiologists. Physicians contoured the pancreas, pancreatic lesions if present, and the MPD if visible. They also assessed tumor type and MPD dilatation. Data was split into a training and independent testing set of 2134 and 756 cases, respectively. A method to detect pancreatic lesions and MPD dilatation was built in three steps. First, a segmentation network was trained in a 5-fold cross-validation manner. Second, outputs of this network were postprocessed to extract imaging features: a normalized lesion risk, the predicted lesion diameter, and the MPD diameter in the head, body, and tail of the pancreas. Third, two logistic regression models were calibrated to predict lesion presence and MPD dilatation, respectively. Performance was assessed on the independent test cohort using receiver operating characteristic analysis. The method was also evaluated on subgroups defined based on lesion types and characteristics. Results The area under the curve (AUC) of the model detecting lesion presence in a patient was 0.98 (95% CI: 0.97, 0.99). A sensitivity of 0.94 (469 of 493, 95% CI: 0.92, 0.97) was reported. Similar values were obtained in patients with small (less than 2cm) and isodense lesions with a sensitivity of 0.94 (115 of 123, 95% CI: 0.87, 0.98) and 0.95 (53 of 56, 95% CI: 0.87, 1.0), respectively. The model sensitivity was also comparable across lesion types with values of 0.94 (95% CI: 0.91, 0.97), 1.0 (95% CI: 0.98, 1.0), 0.96 (0.97, 1.0) for pancreatic ductal adenocarcinoma, neuroendocrine tumor and intraductal papillary neoplasm, respectively. Regarding MPD dilatation detection, the model had an AUC of 0.97 (95% CI: 0.96, 0.98). Conclusions The proposed approach showed high quantitative performance to identify patients with pancreatic neoplasms and to detect MPD dilatation on an independent test cohort. Performance was robust across subgroups of patients with different lesion characteristics and types. Results confirmed the interest to combine a direct lesion detection approach with secondary features such as the MPD diameter, thus indicating a promising avenue for the detection of pancreatic cancer at early stages.