Hybrid Data Preprocessing Framework for Enhanced Insider Threat Detection Using SMOTE and GANs

Abstract

This invention provides a solution to the problem of imbalanced datasets in insider threat detection, where malicious activities are significantly fewer compared to non-malicious activities. To tackle this issue, a hybrid data preprocessing framework is proposed, incorporating Synthetic Minority Over-sampling Technique (SMOTE) and Generative Adversarial Networks (GANs). By generating synthetic samples and improving class balance, this framework enhances the robustness of machine learning models, reducing the risk of overfitting while maintaining high detection accuracy in identifying insider threats.

Specification

Description:This invention addresses the critical issue of imbalanced datasets in insider threat detection, where malicious activities form a minority class compared to non-malicious activities. Traditional machine learning models tend to be biased towards the majority class, leading to poor detection of minority class instances, which are the actual insider threats.
To tackle this challenge, the invention introduces a hybrid data preprocessing framework that combines multiple techniques to improve class balance in training datasets. The process involves the following steps:
1. Input to SMOTE: The insider threat dataset is initially processed using the Synthetic Minority Over-sampling Technique (SMOTE).
2. Synthetic Sample Generation: SMOTE generates synthetic samples of the minority class, balancing it with the majority class.
3. Input for GAN's Generator: The synthetic minority class samples created by SMOTE are then fed as input to the generator component of a Generative Adversarial Network (GAN).
4. Refinement by GAN's Generator: The generator further refines these synthetic samples, creating regenerated minority class samples with enhanced quality and representation, based on the initial SMOTE data.
5. Discriminator's Evaluation: Both the real minority class data and the regenerated (synthetic) samples are taken as input by the GAN's discriminator.
6. Realistic Sample Generation: The discriminator evaluates the input and guides the generator in producing realistic, artificial samples that closely mimic the real minority class data, enhancing the overall dataset balance.
By enhancing the representation of malicious activities, this approach ensures a more balanced dataset, improving the learning capability of machine learning models. The combined use of SMOTE and GANs helps minimize the risk of overfitting and increases detection accuracy, adapting preprocessing strategies based on the specific characteristics of the dataset. This leads to better generalization and improved performance in identifying insider threats, even in highly imbalanced datasets.
The effectiveness of this approach is evaluated using key performance metrics such as precision, recall, and the Area Under the ROC Curve (AUC), ensuring a robust assessment of model capabilities in detecting rare but critical insider threats. Initial findings highlight that while oversampling techniques like SMOTE can enhance model performance, they may also lead to overfitting. The integration of GANs helps address this by generating more accurate and realistic synthetic samples, improving the overall robustness of the detection system.
, Claims:Claim 1 - Hybrid SMOTE and GAN-Based Data Augmentation for Insider Threat Detection:
A data preprocessing method for enhancing insider threat detection, specifically designed for imbalanced datasets, that integrates Synthetic Minority Over-sampling Technique (SMOTE) with Generative Adversarial Networks (GANs). The combined approach generates synthetic samples of the minority class (malicious insider activities) to improve class balance and enhance the accuracy of machine learning models in identifying insider threats.
Claim 2 - Adaptive SMOTE and GAN Framework for Minority Class Augmentation:
The method of Claim 1, wherein the preprocessing framework dynamically applies SMOTE to initially oversample the minority class, followed by the use of GANs to create realistic synthetic data. This combined approach reduces the bias of machine learning models towards the majority class and improves their capability to detect malicious activities in insider threat datasets.
Claim 3 - Performance Enhancement in Insider Threat Detection Using SMOTE-GAN Hybrid Model:
A system for insider threat detection that incorporates a hybrid data augmentation technique using SMOTE and GANs to preprocess training data. This system enhances model performance by mitigating the effects of data imbalance, achieving higher detection accuracy and reducing false-negative rates for malicious insider activity instances.

Documents