Traffic Anomaly Detection in Fog-IoT Networks Using a Combination of Improved Reinforcement Learning and Convolutional Neural Networks

Abstract

Despite significant advances in machine- and deep-learning-based intrusion detection systems, existing methods still face fundamental challenges. Many of these methods either rely on static and predefined patterns and have limited efficiency in detecting unknown and low-rate attacks, or are not suitable for practical deployment in fog and IoT environments due to their high computational complexity. In addition, the issues of data imbalance, traffic pattern dynamics, and limited computational resources in fog nodes mean that a simple combination of multiple learning models alone does not meet the practical needs of this area. In this research, a novel framework for anomaly detection in Fog-IoT networks is presented, in which a convolutional neural network extracts deep representations, and the Dueling-Q reinforcement learning algorithm learns the optimal decision-making policy. Also, to adapt the IDS monitoring data to the reinforcement learning framework, a data transformation module is designed to map traffic flows from a static feature space to structured state representations for the learning agent. To increase detection accuracy, the data imbalance is addressed using SMOTE, and the stability of the learning process is improved by designing a large Replay Buffer. Also, the model reward function is designed to be highly sensitive to false-negative errors so that low-rate, hidden, and distributed attacks are detected more accurately. The evaluation of the proposed model was carried out on two benchmark datasets, UNSW-NB15 and CICIDS2017, under a fog-layer deployment scenario. The results showed that the proposed model achieved 98.8% accuracy, 95.6% attack detection accuracy, 92.4% attack recall, 93.9% attack F1 score, and 96.78% false-negative rate on the UNSW-NB15 dataset. Also, on the CICIDS2017 dataset, the model performance improved further, achieving 98.9% accuracy, 95.8% attack detection accuracy, 92.9% attack recall, and 94.3% attack F1 score. These results indicate the model's strong ability to detect complex, sequence-based, low-rate attacks. The study of originality also showed that removing any of the key components of the model, including the CNN, the Dueling structure, reinforcement learning, or the SMOTE technique, leads to significant performance degradation. Comparisons with methods proposed for fog environments, such as Autoencoder-CNN-LSTM, FFL-IDS, Fog-Edge-SVM-FL, and Whale-ELM, also showed that the proposed method is significantly superior across all critical criteria, especially the attack recall criterion, the most important security indicator. Overall, the results of this research demonstrate that the combination of CNN and Dueling-Q can provide a lightweight, accurate, and practical solution for deployment on fog nodes and serve as a reliable IDS in real Fog-IoT environments.