Robust Statistical Anomaly Detection for Interpretable Intrusion Monitoring in Complex Network Systems
DOI:
https://doi.org/10.58429/pgjsrt.v5n4a240Keywords:
anomaly detection, robust statistics, network intrusion, outlier detection, threshold monitoringAbstract
Anomaly detection is essential for identifying abnormal observations that may signal operational errors, system failures, fraud, network misuse, or security threats in complex data systems. This study develops and evaluates a robust statistical framework for interpretable anomaly detection using the Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) network intrusion dataset as an empirical case. The framework combines descriptive profiling, data-quality auditing, benign-baseline calibration, robust location and scale estimation, interquartile range (IQR) boundaries, median absolute deviation (MAD) scoring, empirical probability-tail thresholds, and feature-count aggregation. The empirical dataset contains 2,830,743 observations distributed across eight flow-based files, including 2,273,097 benign records and 557,646 attack or anomaly records. The Monday benign-only traffic was used to calibrate the normal statistical baseline, while 2,300,825 observations from the remaining days were retained for holdout evaluation. Sixteen interpretable flow variables were selected for the statistical monitoring experiment after cleaning missing values, infinite values, and inconsistent labels. The most conservative probability-tail rule reached 84.80% accuracy, 87.30% precision, 43.63% recall, an F1-score of 0.5818, and a false-alarm rate (FAR) of 2.03%. The findings show that robust threshold monitoring is particularly useful when the main operational objective is high-confidence early warning with low false alarms rather than maximum attack recall. The study contributes a transparent, reproducible, and statistically grounded anomaly-detection procedure that can support cybersecurity monitoring and can be adapted to other risk-sensitive complex data systems.
Downloads
References
. Aggarwal, C. C. (2017). Outlier analysis (2nd ed.). Springer. https://doi.org/10.1007/978-3-319-47578-3
. Ahmed, M., Mahmood, A. N., & Hu, J. (2016). A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60, 19-31. https://doi.org/10.1016/j.jnca.2015.11.016
. Akoglu, L., Tong, H., & Koutra, D. (2015). Graph based anomaly detection and description: A survey. Data Mining and Knowledge Discovery, 29(3), 626-688. https://doi.org/10.1007/s10618-014-0365-y
. Bhuyan, M. H., Bhattacharyya, D. K., & Kalita, J. K. (2014). Network anomaly detection: Methods, systems and tools. IEEE Communications Surveys & Tutorials, 16(1), 303-336. https://doi.org/10.1109/SURV.2013.052213.00046
. Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000). LOF: Identifying density-based local outliers. Proceedings of the ACM SIGMOD International Conference on Management of Data, 93-104. https://doi.org/10.1145/335191.335388
. Buczak, A. L., & Guven, E. (2016). A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials, 18(2), 1153-1176. https://doi.org/10.1109/COMST.2015.2494502
. Campos, G. O., Zimek, A., Sander, J., Campello, R. J. G. B., Micenkova, B., Schubert, E., Assent, I., & Houle, M. E. (2016). On the evaluation of unsupervised outlier detection: Measures, datasets, and an empirical study. Data Mining and Knowledge Discovery, 30(4), 891-927. https://doi.org/10.1007/s10618-015-0444-8
. Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1-58. https://doi.org/10.1145/1541880.1541882
. Domingues, R., Filippone, M., Michiardi, P., & Zouaoui, J. (2018). A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern Recognition, 74, 406-421. https://doi.org/10.1016/j.patcog.2017.09.037
. Goldstein, M., & Uchida, S. (2016). A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PLOS ONE, 11(4), e0152173. https://doi.org/10.1371/journal.pone.0152173
. Gupta, M., Gao, J., Aggarwal, C. C., & Han, J. (2014). Outlier detection for temporal data: A survey. IEEE Transactions on Knowledge and Data Engineering, 26(9), 2250-2267. https://doi.org/10.1109/TKDE.2013.184
. He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263-1284. https://doi.org/10.1109/TKDE.2008.239
. Hubert, M., Rousseeuw, P. J., & Verdonck, T. (2012). A deterministic algorithm for robust location and scatter. Journal of Computational and Graphical Statistics, 21(3), 618-637. https://doi.org/10.1080/10618600.2012.672100
. Lazarevic, A., Ertoz, L., Ozgur, A., Kumar, V., & Srivastava, J. (2003). A comparative study of anomaly detection schemes in network intrusion detection. Proceedings of the Third SIAM International Conference on Data Mining, 25-36. https://doi.org/10.1137/1.9781611972733.3
. Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation forest. Proceedings of the IEEE International Conference on Data Mining, 413-422. https://doi.org/10.1109/ICDM.2008.17
. Maseer, Z. K., Yusof, R., Bahaman, N., Mostafa, S. A., & Foozy, C. F. M. (2021). Benchmarking of machine learning for anomaly-based intrusion detection systems in the CICIDS2017 dataset. IEEE Access, 9, 22351-22370. https://doi.org/10.1109/ACCESS.2021.3056614
. Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems. Proceedings of the Military Communications and Information Systems Conference, 1-6. https://doi.org/10.1109/MilCIS.2015.7348942
. Pang, G., Shen, C., Cao, L., & van den Hengel, A. (2021). Deep learning for anomaly detection: A review. ACM Computing Surveys, 54(2), 1-38. https://doi.org/10.1145/3439950
. Pimentel, M. A. F., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014). A review of novelty detection. Signal Processing, 99, 215-249. https://doi.org/10.1016/j.sigpro.2013.12.026
. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144. https://doi.org/10.1145/2939672.2939778
. Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019). A survey of network-based intrusion detection datasets. Computers & Security, 86, 147-167. https://doi.org/10.1016/j.cose.2019.06.005
. Rosay, A., Cheval, E., Carlier, F., & Leroux, P. (2022). Network intrusion detection: A comprehensive analysis of CIC-IDS2017. Proceedings of ICISSP 2022, 25-36. https://doi.org/10.5220/0010774000003120
. Rousseeuw, P. J., & Hubert, M. (2018). Anomaly detection by robust statistics. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(2), e1236. https://doi.org/10.1002/widm.1236
. Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., & Muller, K. R. (2021). A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5), 756-795. https://doi.org/10.1109/JPROC.2021.3052449
. Schmidl, S., Wenig, P., & Papenbrock, T. (2022). Anomaly detection in time series: A comprehensive evaluation. Proceedings of the VLDB Endowment, 15(9), 1779-1797. https://doi.org/10.14778/3538598.3538602
. Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. Proceedings of ICISSP 2018, 108-116. https://doi.org/10.5220/0006639801080116
. Sommer, R., & Paxson, V. (2010). Outside the closed world: On using machine learning for network intrusion detection. Proceedings of the IEEE Symposium on Security and Privacy, 305-316. https://doi.org/10.1109/SP.2010.25
. Tavallaee, M., Bagheri, E., Lu, W., & Ghorbani, A. A. (2009). A detailed analysis of the KDD CUP 99 data set. Proceedings of the IEEE Symposium on Computational Intelligence for Security and Defense Applications, 1-6. https://doi.org/10.1109/CISDA.2009.5356528
. Zimek, A., Schubert, E., & Kriegel, H. P. (2012). A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analysis and Data Mining, 5(5), 363-387. https://doi.org/10.1002/sam.11161
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Polaris Global Journal of Scholarly Research and Trends

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
