One-class support vector machines (OCSVM) have been recently applied in intrusion detection. Typically, OCSVM is kernelized by radial basis functions (RBF, or Gaussian kernel) whereas selecting Gaussian kernel hyperparameter is based upon availability of attacks, which is rarely applicable in practice. This paper investigates the application of nested OCSVM to detect intruders in network systems with data-driven hyperparameter optimization. The nested OCSVM is able to improve the efficiency over the proposed OCSVM applied in intrusion detection. In addition , the information of the farthest and the nearest neighbors of each sample is used to construct the objective cost instead of labeling based metrics such as geometric mean accuracy. The efficiency of this method is illustrated over the KDD99 dataset whereas the resulting estimated boundary, as well as intrusion detection performance, are comparable with existing methods. The experimental results show that the nested OCSVM method performs better than OCSVM for intrusion detection. The nested OCSVM with 12 density levels achieves 98.28% in accuracy and higher true alarming rate (TP) comparing to OCSVM.