Securing computing systems from the core: Kernel defense against insidious rootkit malware

Cyber security researchers at the Graduate School of Information Security (GSIS) at KAIST are setting a new trend, namely, a hardware-based security monitor, in order to create defense systems against insidious rootkit malwares that render all the existing in-host security measures useless by attacking the operating system kernel

Article  |  Fall 2014

The operating system kernel is at the core of all operations carried out in computing systems. A rootkit is a powerful class of malware that infects the OS kernels itself. Once the operating system kernel is compromised, the system is no longer trustworthy nor reliable.

The most challenging aspect in dealing with these vicious malwares is that they have the power to achieve stealth against or even impair all the in-host security measures deployed in the victimized system. Hence, researchers have explored various defense measures that operate outside of the compromised host system.

A joint research team consisting of Prof. Brent Byunghoon Kang from KAIST (Korea Advanced Institute of Science and Technology) and Prof. Yunheung Paek from SNU (Seoul National University) have introduced a hardware-based bus snooping system integrity monitor. The team’s proposed system, called Vigilare, drew attention from both academia and industry when it was presented at the ACM Conference on Computer and Communications Security (CCS) 2012, where Prof. Kang illustrated that the proposed system is capable of detecting an advanced rootkit attack technique called transient attacks that can be used to evade the existing snapshot-based hardware monitors [VIGILARE].

At USENIX Security 2013, the team presented KI-Mon, a breakthrough invention that creates a defense mechanism that utilizes a specialized hardware component designed to efficiently engage the kernel integrity monitor only when the monitored dynamic content is modified, necessarily requiring inspection. Monitoring dynamic data structures manipulated by modern advanced rootkits has been considered a difficult and complex problem as they are frequently and constantly modified during normal operations of the kernel [KIMON].

More recently, Prof. Kang’s team at KAIST has discovered a grave flaw, a fundamental limitation by design, that may allow an attacker to evade all the existing hardware-based monitors including their own system, KI-Mon. This new kind of attack, which was coined by the team as an “Address Translation Redirection Attack” (ATRA),” exploits the memory management subsystem that is required for translating the virtual address space that in which all the programs run into the physical address space that the real machine hardware operates on. This work was acknowledged for its seminal role as a guide and a motivation for designing robust hardware-based kernel integrity monitors. ATRA is to appear in ACM CCS 2014 [ATRA].
[ATRA] D. Jang, H. Lee, M. Kim, D. H. Kim, D. G. Kim and B. Kang. ATRA: Address Translation Redirection Attack against Hardware-based Kernel Integrity Monitors. ACM CCS 2014.

[KIMON] H. Lee, H. Moon, D. Jang, K. Kim, J. Lee, Y. Paek and B. Kang. KI-Mon: A Hardware-assisted Event-triggered Monitoring Platform for Mutable Kernel Object. USENIX Security 2013.

[VIGILARE] H. Moon, H. Lee, J. Lee, K. Kim, Y. Paek and B. Kang. Vigilare: Toward Snoop-based Kernel Integrity Monitor. ACM CCS 2012.