(Dept. of CSE at JNTUK, Kakinada, India)(2) Venkata Narayana
(Dept. of CSE at Lakireddy Bali Reddy College of Engineering, Mylavaram, India)*corresponding author
AbstractStreaming anomaly detection is a difficult task because the data distribution is changing, and concept drift degrades the performance of traditional detection methods. To solve the issue, this article suggests the HAD-CDA (Hybrid Anomaly Detection with Concept Drift Adaptation), which is a combined system that can detect local anomalies in the time series with LSTM Autoencoders (LSTM-AE) and global anomalies in the distribution with Quant Tree-EWMA (QT-EWMA). The proposed framework accomplishes three objectives: (i) the use of the dynamic weighting mechanism, which automatically changes the contribution of each component (lambda between 0.2 and 0.8) according to their effectiveness, (ii) two concept drift detectors are proposed, i.e., the Kolmogorov-Smirnov and Page-Hinkley tests, to allow detecting concept drift; and (iii) the use of Elastic Weight Consolidation (EWC) to reduce catastrophic forgetting during update of the model. Experiments on four real-world streaming datasets, HTTP, SMTP, ForestCover, and Shuttle, indicate that HAD-CDA has AUCs of 0.95-0.97, an 8-9% improvement over state-of-the-art methods, and F1-scores of 0.81-0.95. The recall measure obtained by the LSTM-AE element is 0.8094, compared to 0.9399, and the specificity of QT-EWMA is very high at 0.9399. The framework is highly adaptable to different types of drifts, regaining around 9092 performance levels before drift, and 1525 windows would be needed by a baseline method, with a low processing latency of 12.430.9 ms per window. Having a per-sequence complexity of O(1), memory cost of O(n), low DIS (approximately 0.08), and stability indices of 0.02-0.04, the suggested HAD-CDA framework is an accurate, efficient, and robust solution to real-world streaming anomaly detection in changing data
KeywordsAnomaly Detection, Streaming Data, Concept Drift, LSTM Autoencoder, QuantTree-EWMA, Dynamic Weighting
|
DOIhttps://doi.org/10.26555/ijain.v12i2.2283 |
Article metricsAbstract views : 185 | PDF views : 0 |
Cite |
Full Text Download
|
References
[1] N. Mokoena and I. C. Obagbuwa, “An analysis of artificial intelligence automation in digital music streaming platforms for improving consumer subscription responses: a review,” Front. Artif. Intell., vol. 7, p. 1515716, Jan. 2025, doi: 10.3389/frai.2024.1515716.
[2] R. Sultana and R. K. Moury, “Real-Time Streaming Analytics for Healthcare: An Architecture for Continuous Clinical Data Processing,” Am. J. Data Sci. Anal., vol. 07, no. 06, pp. 244–284, Jun. 2026, doi: 10.63125/gvqr2j17.
[3] K. Varalakshmi and J. Kumar, “Optimized predictive maintenance for streaming data in industrial IoT networks using deep reinforcement learning and ensemble techniques,” Sci. Rep., vol. 15, no. 1, p. 27201, Jul. 2025, doi: 10.1038/s41598-025-10268-8.
[4] Z. Li, V. Chang, H. Hu, M. Fu, J. Ge, and F. Piccialli, “Optimizing makespan and resource utilization for multi-DNN training in GPU cluster,” Futur. Gener. Comput. Syst., vol. 125, no. December, pp. 206–220, Dec. 2021, doi: 10.1016/j.future.2021.06.021.
[5] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” ACM Comput. Surv., vol. 46, no. 4, pp. 1–37, Apr. 2014, doi: 10.1145/2523813.
[6] T. Fernando, H. Gammulle, S. Denman, S. Sridharan, and C. Fookes, “Deep Learning for Medical Anomaly Detection – A Survey,” ACM Comput. Surv., vol. 54, no. 7, pp. 1–37, Sep. 2022, doi: 10.1145/3464423.
[7] C. Kumar and M. Kumar, “Session-based recommendations with sequential context using attention-driven LSTM,” Comput. Electr. Eng., vol. 115, no. April, p. 109138, Apr. 2024, doi: 10.1016/j.compeleceng.2024.109138.
[8] G.-L. Li, H.-R. Zhang, F. Min, and Y.-N. Lu, “Two-stage label distribution learning with label-independent prediction based on label-specific features,” Knowledge-Based Syst., vol. 267, no. May, p. 110426, May 2023, doi: 10.1016/j.knosys.2023.110426.
[9] N. Ramanan, R. Tahmasbi, M. Sayer, D. Jung, S. Hemachandran, and C. N. Coelho, “Real-time Drift Detection on Time-series Data,” pp. 1–5, Oct. 2021, Accessed: Jul. 19, 2026. [Online]. Available: https://arxiv.org/pdf/2110.06383.
[10] H. M. Gomes et al., “Adaptive random forests for evolving data stream classification,” Mach. Learn., vol. 106, no. 9–10, pp. 1469–1495, Oct. 2017, doi: 10.1007/s10994-017-5642-8.
[11] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff, “LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection,” pp. 1–5, Jul. 2016, Accessed: Jul. 19, 2026. [Online]. Available: http://arxiv.org/abs/1607.00148.
[12] N. Jeffrey, Q. Tan, and J. R. Villar, “A hybrid methodology for anomaly detection in Cyber–Physical Systems,” Neurocomputing, vol. 568, no. February, p. 127068, Feb. 2024, doi: 10.1016/j.neucom.2023.127068.
[13] N. Mohammadi Foumani, L. Miller, C. W. Tan, G. I. Webb, G. Forestier, and M. Salehi, “Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey,” ACM Comput. Surv., vol. 56, no. 9, pp. 1–45, Oct. 2024, doi: 10.1145/3649448.
[14] P. Malhotra, L. Vig, G. M. Shroff, and P. Agarwal, “Long Short Term Memory Networks for Anomaly Detection in Time Series,” in The European Symposium on Artificial Neural Networks, 2015, pp. 89–94. Accessed: Jul. 11, 2026. [Online]. Available: https://www.researchgate.net/publication/304782562.
[15] M. Tavana, H. Mousavi, A. Khalili Nasr, and H. Mina, “A fuzzy weighted influence non-linear gauge system with application to advanced technology assessment at NASA,” Expert Syst. Appl., vol. 182, no. November, p. 115274, Nov. 2021, doi: 10.1016/j.eswa.2021.115274.
[16] G. Pang, C. Shen, L. Cao, and A. Van Den Hengel, “Deep Learning for Anomaly Detection,” ACM Comput. Surv., vol. 54, no. 2, pp. 1–38, Mar. 2022, doi: 10.1145/3439950.
[17] M. Shen et al., “Artificial Intelligence for Web 3.0: A Comprehensive Survey,” ACM Comput. Surv., vol. 56, no. 10, pp. 1–39, Oct. 2024, doi: 10.1145/3657284.
[18] Girish Reddy Ginni and Srinivasa L. Chakravarthy, “A Hybrid Framework for Robust Anomaly Detection: Integrating Unsupervised and Supervised Learning with Advanced Feature Engineering,” Int. J. Comput. Exp. Sci. Eng., vol. 11, no. 2, pp. 1993–2017, Apr. 2025, doi: 10.22399/ijcesen.1383.
[19] A. Bifet and R. Gavaldà, “Learning from Time-Changing Data with Adaptive Windowing,” in Proceedings of the 2007 SIAM International Conference on Data Mining, Philadelphia, PA: Society for Industrial and Applied Mathematics, Apr. 2007, pp. 443–448. doi: 10.1137/1.9781611972771.42.
[20] K. Yamanishi, J. Takeuchi, G. Williams, and P. Milne, “On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms,” Data Min. Knowl. Discov., vol. 8, no. 3, pp. 275–300, May 2004, doi: 10.1023/B:DAMI.0000023676.72185.7c.
[21] Y. Qiao, K. Wu, and P. Jin, “Efficient Anomaly Detection for High-Dimensional Sensing Data With One-Class Support Vector Machine,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 1, pp. 404–417, Jan. 2023, doi: 10.1109/TKDE.2021.3077046.
[22] Y. Liu, H. Yuan, L. Cai, and S. Ji, “Deep Learning of High-Order Interactions for Protein Interface Prediction,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA: ACM, Aug. 2020, pp. 679–687. doi: 10.1145/3394486.3403110.
[23] X.-C. Yin, K. Huang, and H.-W. Hao, “DE2: Dynamic ensemble of ensembles for learning nonstationary data,” Neurocomputing, vol. 165, pp. 14–22, Oct. 2015, doi: 10.1016/j.neucom.2014.06.092.
[24] P. Zhou, “A survey of streaming data anomaly detection in network security,” PeerJ Comput. Sci., vol. 11, p. e3066, Aug. 2025, doi: 10.7717/peerj-cs.3066.
[25] T. S. Sethi and M. Kantardzic, “On the reliable detection of concept drift from streaming unlabeled data,” Expert Syst. Appl., vol. 82, pp. 77–99, Oct. 2017, doi: 10.1016/j.eswa.2017.04.008.
[26] D. Kwon, H. Kim, J. Kim, S. C. Suh, I. Kim, and K. J. Kim, “A survey of deep learning-based network anomaly detection,” Cluster Comput., vol. 22, no. S1, pp. 949–961, Jan. 2019, doi: 10.1007/s10586-017-1117-8.
[27] A. Mahmoud and A. Mohammed, “A Survey on Deep Learning for Time-Series Forecasting,” in Studies in Big Data, vol. 77, Springer Science and Business Media Deutschland GmbH, 2021, pp. 365–392. doi: 10.1007/978-3-030-59338-4_19.
[28] M. A. Talukder et al., “A dependable hybrid machine learning model for network intrusion detection,” J. Inf. Secur. Appl., vol. 72, no. February, p. 103405, Feb. 2023, doi: 10.1016/j.jisa.2022.103405.
[29] R. A. Ramadan and K. Yadav, “A Novel Hybrid Intrusion Detection System (IDS) for the Detection of Internet of Things (IoT) Network Attacks,” Ann. Emerg. Technol. Comput., vol. 4, no. 5, pp. 61–74, Dec. 2020, doi: 10.33166/AETiC.2020.05.004.
[30] A. K. Balyan et al., “A Hybrid Intrusion Detection Model Using EGA-PSO and Improved Random Forest Method,” Sensors, vol. 22, no. 16, p. 5986, Aug. 2022, doi: 10.3390/s22165986.
[31] E. Özer, M. İskefiyeli, and J. Azimjonov, “Toward lightweight intrusion detection systems using the optimal and efficient feature pairs of the Bot-IoT 2018 dataset,” Int. J. Distrib. Sens. Networks, vol. 17, no. 10, p. 155014772110522, Oct. 2021, doi: 10.1177/15501477211052202.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0