Date of Award

8-28-2013

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Science

First Advisor

Remzi Seker

Abstract

Worms and remote intrusions often utilize code injection to be successful. Execution of the injected code is crucial to exploit a vulnerability in an application. This proposal introduces a different approach than the previous approaches to system call analysis. System call layer reflects significant behavioral semantics of an application as it signifies change of application privileges to kernel mode. We introduce some of the well-established techniques and formalisms of Dynamical System Theory into analysis of application behavior via system calls analysis. We accept a program as a blackbox dynamical system whose internals are not known but whose output - system call - we can observe. The blackbox system observable in our model is the system calls the application makes. The collected system calls are treated as signals which are used to reconstruct the Dynamical system's phase space. Then, by using the well-established techniques from Dynamical System Theory, we quantify the amount of complexity in the system's (application) behavior. The change in the behavior of a compromised application is detected as anomalous behavior compared to the baseline established from a clean application. This approach is employed first on data collected through code injection simulation experiments on certain daemon applications. We test the proposed approach against DARPA-98 dataset for system call argument analysis. Finally, a real world exploit on ProFTP daemon is run to collect system call data and compare with normal execution system call data. Graphical analyses of deviations of Dynamical System measures of these datasets from their respective normal baseline values collectively emphasize applicability of our theory to detect anomalous application behavior.

Download

Share

COinS