Art & Photos


Machine Learning based Research for Network Intrusion Detection: A State-of-the-Art

This paper reviews the machine learning based research carried out for network intrusion detection to lead a secure computer and network systems to the extent possible. Starting with an initial set of about 460 research articles, more than 105
of 20

Please download to get full document.

View again

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
  International Journal of Information & Network Security (IJINS) Vol.3, No.3, June 2014, pp. 01~20 ISSN: 2089-3299   31  Journal homepage : IJINS Machine Learning based Research for Network Intrusion Detection: A State-of-the-Art Kanubhai K. Patel*, Bharat V. Buddhadev** * CMPICA, Charotar University of Science & Technology ** Department of Computer Engineering, SS College of Engineering Article Info ABSTRACT    Article history: Received May 5 th , 2014 Revised May 20 th , 2014 Accepted May 26 th , 2014 This paper reviews the machine learning based research carried out for network intrusion detection to lead a secure computer and network systems to the extent possible. Starting with an initial set of about 460 research articles, more than 105 related studies in the period between 2000 and 2013 were selected focusing on using single or combine machine learning techniques for this review. Solutions using convergence of various techniques show a great promise and potential. Related studies are compared by their design, datasets used, and other experimental setups. Current achievements and limitations in developing intrusion detection systems using machine learning techniques are presented and discussed. We assume that reader has aware of basic concepts of machine learning techniques. A good number of future research directions are also provided.  Keyword: Intrusion detection Machine learning Survey Rule based learning Misuse detection Copyright @ 2014 Insitute of Advanced Engineeering and Science.  All rights reserved. Corresponding Author: Kanubhai K. Patel, CMPICA, Charotar University of Science & Technology Changa, India. Email:  1.   INTRODUCTION Intrusion is a set of actions aimed to compromise integrity, confidentiality or availability (CIA) of computing resources in a computing environment [1]. In general terms, intrusive behaviour can be considered as any behaviour that deviates from normal use of the system. There are general four categories of intrusion [2] [3]: — Denial of Service (DoS): The general task of DoS attacks is to interrupt some service on a host to prevent it from dealing with certain requests. For example, SYN flood, smurf and teardrop. — Probing: It is to gain information about the target host. For example, port-scan and ping-sweep. — User to Root (U2R): U2R attacks exploit vulnerabilities in operating systems and software to obtain root (administrator) access to the system. For example, buffer overflow attacks. — Remote to Local (R2L): The intruder does not have an account on the host and attempts to obtain local access across a network connection. For example, password guessing attacks. Intrusion Detection (ID) is the process of identifying and responding to intrusion activities [4]. ID is the process of monitoring and analyzing events that occur in a computer or network to detect behaviour of users that conflict with the intended use of the system [4]. An Intrusion Detection System (IDS) is software and/or hardware component that monitor the events in a computer or network, and analyze the activities for signs of possible violations of computer security policies. It employs techniques for modeling and recognizing intrusive behaviour in a computer system [4]. This is security system devoted to permanent inspection of computing environments, information technology (IT) infrastructure, and related assets such as      ISSN: 2089-3299 IJINS Vol. 3, No. 3, June 2014 : 01 – 20 32hosts, networks, application, servers, and databases. The objective of IDS is to identify ongoing attacks in real time and to establish an active or passive response in order to prevent successful attacks. IDS is the ‘burglar alarm’ of the computer security field. IDS have been around since the 1980s. James Anderson [5] introduced the concept of host-based IDS in 1980. In 1987, Dorothy Denning presented IDS design [6]. Then, Heberlein et al. [7] introduced network-based IDS in 1990. ID is an active research area in the field of computer and network security. Current ID technology often does not mature completely because of regular changes in computer, network and the general evolution of information and communication technology (ICT). Many improvements, from different perspectives, should be considered in ID technology, so that the challenging nature of the requirements for the current and future computing environments can be accommodated. Promising methodologies and technologies are required for the design and development of effective Intrusion Detection System (IDS). We have reviewed the research carried out for intrusion detection using machine learning techniques to lead a secure computer and network systems to the extent possible. Rule based learning is covered in greater detail and areas requiring further research and exploration are mentioned in the paper. Comparison among various machine learning techniques made by other researchers is also covered. The rest of this article is organized as follows. Section-2 describes main two methods of intrusion detection viz. i) misuse detection, and ii) anomaly detection. While Section-3 describes main two approaches of intrusion detection viz. i) Stateful detection approach, and ii) Stateless detection approach. Section-4 briefly presents research in intrusion detection using various machine learning techniques. This section covers both (supervised and unsupervised) machine learning techniques for intrusion detection. Section-5 briefly describes comparison of various techniques made by researchers along with summarization. Section-6 briefly describes open research areas, future challenges and opportunities in the field of intrusion detection through machine learning techniques. Section-7 concludes the article. 2.   INTRUSION DETECTION METHODS There are two main detection methods, i) misuse detection, and ii) anomaly detection [8] [9] [4]. These terms are also known as knowledge based and behaviour based intrusion [10]. The misuse detection method attempts to encode knowledge of known intrusions (misuse or abuse), typically as rules, and use this to screen events (also known as a signature based IDS) [11] [12]. As per Gollman [9], misuse detection is successful in commercial IDS. Misuse detection method is although being effective against known attacks, it fails to protect from novel threats. This brings anomaly detection into the focus of security research (e.g., [13] [14] [15] [16]. The anomaly detection method attempts to ‘learn’ the features of event patterns that constitute normal behaviour, and, by observing patterns that deviate from established norms, detect when an intrusion has occurred [6]. In recent years, researchers have incorporated techniques that allow misuse detection systems to be more flexible, being capable of detecting more variations of attacks. This has been made possible with machine learning techniques such as artificial neural networks (ANN) and fuzzy logic, which are built to be able to generalize their models of known attacks to classify unseen cases. This is also the case for rule based systems, which were deemed in the past to be unable to detect even slight variations of attacks due to rigid rules [17]. Rule based systems are now also capable of detecting variations of attacks, and may even be employed for anomaly detection, largely due to researchers incorporating fuzzy logic to define the rules. A broad review of anomaly detection can be found in the work of [18] [19]. A review of the main techniques applied in data preprocessing for anomaly based network intrusion can be found in the work of Davis and Clark [20]. 3.   INTRUSION DETECTION APPROACHES As per Engen [4], there are two main approaches to detect intrusions: i) Stateful, and ii) Stateless. Stateful approaches consider an attack as being composed of several events (stages), whilst stateless approaches attempt to classify single events as being an intrusion or not. 3.1. Stateful (Event correlation) detection approach Engen [4] has considered event correlation synonymous with stateful approaches for simplicity, which has been subject to extensive research and is commonly adopted in commercial IDSs, e.g., HP OpenView [21], EMERALD eXpert and eXpert-BSM [22] [23], and Snort [24]. Event correlation systems may analyze data both spatially and temporally, building deterministic and/or probabilistic models of intrusions [25]. Spatial systems analyze events from different sources simultaneously, whilst temporal systems consider not only the order of events to be significant, but also the time between them. Rule based systems are commonly used for event correlation [25] [22] [4]. The system will filter events according to a set of rules (signatures) that determine the pattern of intrusions. Kruegel et al. [26] has proposed stateful  IJINS ISSN: 2089-3299     Machine Learning based Research for Network Intrusion Detection: A State-of-the-Art (Kanubhai K Patel) 33intrusion detection for high-speed networks. Panichprecha et al. [27] presented an approach to multi-step scenario specification and matching. This aims to address some of the issues and problems inherent in to scenario specification and event correlation found in most previous research work. 3.2. Stateless intrusion detection approach Stateless IDS is classifying single events (e.g., network connections) as being intrusive or normal. Stateless intrusion detection is popularly adopted in the data mining and machine learning communities, treating the intrusion detection problem as a classification task. We need to transform the raw data, such as tcpdump for network based IDS, into suitable feature vectors such as those in MADAM/ID [28]. The feature vectors may also include some a priori knowledge, such as the count feature in the KDD Cup’99 data set [29], which contains information about the number of connections from a particular user within the last two seconds. Here, a challenge is to obtain a feature (input variable) vector that is comprehensive enough to separate normal data from intrusive data, but also keep the size of this vector as small as possible. Normally, the problem is more difficult to solve if we have more features. For many machine learning algorithms, increasing the number of features (the dimension of the problem) significantly increases the training time required to learn the intrusion task. It also slows down the run-time and increase memory requirements with more features. This is commonly referred to as ‘the curse of dimensionality’ [30]. Hence, much research has been devoted to developing efficient techniques to perform feature selection [4]. A feature construction algorithm consists of two steps: feature extraction and feature selection [4]. 4.   MACHINE LEARNING TECHNIQUES FOR INTRUSION DETECTION Machine Learning (ML) is a field of Artificial Intelligence (AI) that is concerned with constructing programs that can improve their behaviour with experience [31]. As per Mitchell [31], “ML refers to algorithmic mechanisms that allow computers to learn from experience, examples and analogy.” There are main two approaches of machine learning: i) Supervised learning, and ii) Unsupervised learning. In the case of intrusion detection, learning involves determining patterns of normal or intrusive behavior by examining the sample data. Within context of misuse detection, Sabhnani and Serpen [32] described application of machine learning techniques and algorithms to KDD Intrusion Detection Dataset, and also described why machine learning algorithms fail [33]. We have reviewed research in some of machine learning techniques viz. i) Rule based learning, ii) Decision tree, iii) Bayesian reasoning, iv) Neural Networks, v) Support Vector Machines (SVM), vi) Clustering, and vii) Nature-inspired. These techniques are presented in the next subsections. 4.1. Rule based learning Many machine learning techniques are applied to the problem of intrusion detection, however, there are few that emphasize on automatic rule learning and a fewer that learn rules online (i.e., in a single-pass). Automatic rule learning for intrusion detection is an active area of research. Some of classifier algorithms of rules are JRIP, Decision Tabel, PART, and OneR. The prominent techniques for learning rules specifically for intrusion detection are: i) Rule based expert system, and ii) Fuzzy rule based.   4.1.1. Rule based expert system Rule based expert system defines mechanisms to compare rules or signatures or scenarios against rule base or audit records. SRI International began research into an intrusion detection expert system in 1985 [34]. As a result of the research, the Intrusion Detection Expert System (IDES) has become a standard in intrusion detection systems. EMERALD [22], ASAX [35], and ORCHIDS [36] are the other examples of rule based expert system. In rule based expert system, the knowledge of human experts is encoded into a set of rules. EMERALD [22] is a forward chaining rule-based expert system. It generates a forward chain of rules which links audit records facts to signatures. While, ASAX [35] specifies signatures as pairs of conditions and actions. ORCHIDS [36] is based on the technique proposed in [37] whose idea is derived from ASAX. Here, the detection is performed by comparing events against application specific temporal logic expressions. RIPPER (Repeated Incremental Pruning to Produce Error Reduction) [38] is a popular rule mining algorithm that can be used to create a classifier, which is considered in several studies, for e.g. [39] [28] [40]. RIPPER is a sequential covering based rule learner, extended from IREP (Incremental Reduced Error Pruning) [41], which has been used by several researchers for learning rules for intrusion detection. Apriori [42] learns association rules by mining the frequent episodes and has also been used for intrusion detection by good number of researchers. Ramesh and Mahesh [43] proposed a framework to learn rules in two stages. First, the sequential covering algorithm is used to learn highly accurate rules indicating the presence of a target class. In the second stage, rules classifying the negation of the target class are learnt on the subset covered collectively by all positive rules.      ISSN: 2089-3299 IJINS Vol. 3, No. 3, June 2014 : 01 – 20 34Mahoney and Chan [44] [45] introduced a randomized rule generation algorithm which they called LERAD (Learning Rules for Anomaly Detection). LERAD generates simple if then conditional rules similar to association rules. This system was extended also to learn rules from system call sequences [46]. Maloof [47] extended the AQ11 algorithm, the incremental version of the sequential covering based AQ algorithm to AQ11-PM (i.e., AQ11 with partial memory). JAM [48] and ADAM [49] mined association rules from the training data and then use them to detect intrusions in the test data. The JAM worked in a misuse detection mode while the ADAM in the anomaly detection mode. Vollmer, Alves-Foss, and Manic [50] presented a combined approach that uses GA and anomaly-based IDS to create rules for a signature-based IDS. They produced set of optimal rules (rule-based) for a specific, anomalous instance previously detected by an anomaly IDS. Srinivasa et al. [51] presented a rule based IDS in which they use genetic algorithm (GA) to make IDS more efficient. The advantages of the rule based expert system approach are i) the simplicity, and ii) straightforwardness of the signature matching mechanism. But, this technique will not perform well in a case where number of rules is large. Rule based expert system is the most suitable for misuse detection method. But it suffers from low flexibility and robustness. 4.1.2. Fuzzy rule based technique Fuzzy logic [52] is an approach to obtain more flexible rules compared with crisp rule based expert systems. It is obvious that the nature of intrusion detection is fuzzy. Bridges and Vaughn [53] examined a combination of a rule based system (RBS) and fuzzy association rule mining to monitor network traffic and system audit trails. The RBS for misuse detection and the fuzzy association rule mining for anomaly detection. They found that fuzzy logic can help to extract more general patterns of intrusions. In their experiments, incorporating fuzzy logic into the rule mining reduced the number of false positives. Florez et al. [54] have extended the research of Bridges and Vaughn [53], making various improvements. Florez et al. [54] use prefix-trees for speeding the fuzzy association rule generation. In both studies above, a Genetic Algorithm (GA) has been used for feature selection and to optimize the fuzzy membership functions. Later, GAs have also been applied to rule learning by other researchers [55] [56] [57] and [58]. Dickerson et al. [59] developed the Fuzzy Intrusion Recognition Engine (FIRE) using fuzzy sets and fuzzy rules for detecting malicious activity in computer networks in a distributed fashion. FIRE is defined as a collection of autonomous agents. Each agent produces fuzzy information from input sensors. For each observed feature a fuzzy set is generated using a fuzzy c-means algorithm. Such information is combined by a fuzzy rule based system for determining the degree of normalcy. Experiments were conducted with synthetic data sets in three different scenarios: Port and Host scan, DoS attack and unauthorized services access. Tillapart et al. [60] proposed Fuzzy IDS (FIDS), a fuzzy rule based system. They provided numerous example rules. Cho [61] used a fuzzy rule reasoning mechanism in order to detect an anomaly. The input of the fuzzy reasoning mechanism are Hidden Markov Model (HMM) evaluation values (from different HMM models). The fuzzy rules are designed according to the set of HMM’s. A centroid defuzzyfication technique was applied for determining the final classification (abnormal or normal). Experiments were conducted for detecting user-to-root attacks on data collected from graduate students. Su et al. [62] proposed a novel method of incremental mining. They implemented fuzzy association rules in a real-time NIDS. Owens and Levary [17] utilized fuzzy set theory to develop an adaptive expert system for network based intrusion detection. Fuzzy rules can be created to perform both misuse and anomaly detection. Events are mapped to fuzzy sets, which are then classified by an expert system that determines an alert with a suspicion level of either ‘low’, ‘medium’ or ‘high’. Jahromi and Taheri [63] proposed a method for learning rule weights in fuzzy rule-based classification systems. While Toosi and Kahani [64] proposed a novel approach based on an evolutionary soft computing model for intrusion detection using neuro-fuzzy classifiers. Fuzzy association rule are employed by Tajbakhsh, Rahmati, and Mirzaei [65] for the building the classifier. Fries [66] proposed evolutionary optimization of a fuzzy rule based network IDS. It provides better performance in comparison to other evolutionary approaches. As per him, evolutionary based systems offer the ability to adapt to dynamic environments and thereby to identify unknown attack methods, while Fuzzy-based systems accommodate the fuzziness associated with altered and previously unidentified attack modes. Dhanalakshmi and Babu [67] proposed a system in which the fuzzy logic is integrated with the data mining methods using GA for intrusion detection. This system uses data mining to extract rules and Mamdami fuzzy inference system to determine the behaviour of the test data. Shanmugavadivu and Nagarajan [68] proposed anomaly based network IDS. They used fuzzy logic for identifying the intrusion activities in a network. This system generates fuzzy IF-THEN rules and with the help of fuzzy decision module the system identifies the appropriate classification of the test data. They used KDD Cup99 for the evaluation of IDS.  IJINS ISSN: 2089-3299     Machine Learning based Research for Network Intrusion Detection: A State-of-the-Art (Kanubhai K Patel) 35Om and Gupta [69] used fuzzy inference rules for host based IDS to monitor hardware profile changes and thereby to detect the unauthorized access in a computer system. A fuzzy logic technique has been used in correlation with ID because of its following characteristics [53] [59]: — Various quantitative parameters used for Intrusion Detection e.g., CPU usage time, activity frequency, connection interval, etc., are fuzzy in nature [70]. — The concept of security itself is fuzzy as stated by Bridges et al. [70]. — Fuzzy systems can readily combine inputs from varying sources [13]. — The degree of alert that can occur with intrusion is often fuzzy [13]. — Fuzzy rules allow us to easily construct if-then rules that help in describing security attacks. 4.1.3. Association rule discovery Association Rule mining is a very popular technique although it is very slow. It finds correlation between the attributes. It was initially applied to the so-called market basket analysis, which aims at finding regularities in shopping behavior of customers of supermarkets [71]. The concept of association rule mining for intrusion detection was introduced by Lee, et al. [72], and is extended by [73] [74] [75]. Disadvantages of Association rule discovery technique: — The execution time or association rule approach increases exponentially with respect to time as the number of attributes increases [65]. — There is vast number of rules, it is not possible to process all rules in turn. Audit Data Analysis and Mining (ADAM) [73] used classification algorithm and association rules to detect attacks in audit data. They try to improve the classification efficiency. Hanguang and Yu [76] applied the rule base deduced from Apriori algorithm to increase performance of the structure, which is the standard of the association rule mining. 4.1.4. Rule based languages There are two important rule based languages, i) rule based sequence evaluation language (RUSSEL) [77], and ii) production-based expert system tool set (P-BEST) [22]. RUSSEL was used in the advanced security audit trail analysis on UNIX (ASAX) project [77]. It is flexible and better to describe sequential event patterns and corresponding actions. The disadvantage is that to specify an attack pattern, user needs to write a program. P-BEST was developed for the Multiplexed information and computing service (Multics) Intrusion Detection and Alerting System (MIDAS). It was employed by IDES, NIDES, and EMERALD [22] later. Advantages of P-BEST are: — It has ability to invoke external C functions, — It is a language pre-processor, — It is quite small and intuitive, and — It does not depend on the structure of the input data. Disadvantages of P-BEST are: — It is a low-level language. — It is time consuming to specify the attack patterns. — Correctness of the rules is difficult to check due to the interaction of the related rules. 4.2. Decision tree Decision Tree (DT) is a simplest classifier. It uses a tree graph along with the probability to provide the best match for the input. DT is one of the most commonly used supervised learning techniques in IDS due to fast adaptation, its simplicity, and high detection accuracy. DT is widely used in misuse detection systems. It yields good performance and has benefits compare to other machine learning techniques. C4.5 algorithm [78] is the most popular DT classifier. NBTree [79] is a hybrid between decision tree and Naive Bayes. It creates trees whose leaves are NaiveBayes classifiers for the instances that reach the leaf. Other classifier algorithms based on DT are RandomForest and REPTree. As per Engen [4] drawbacks of DT are: — It cannot generalize to new attacks in the same manner as certain other machine learning approaches (similar to that of rule based systems). — It is not suitable for anomaly detection. Empirical findings also demonstrate that DT is very sensitive to the training data and does not learn well from imbalanced data [80]. Bouzida and Cuppens [81] applied DT for anomaly-based intrusion detection. They assign a default class to the test instance that is not covered by the tree. Then the default class is examined for unknown
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks