Geok Meng Ong, McAfee Avert Labs, USA
Ambiguity and variation in file format specifications and implementations has, in the past, led to numerous bugs and security issues. In varying interpretations of file parsers, developers may fail to handle anomalies or simply choose to ignore them and process them as normal files.
In the former, they often lead to vulnerabilities, causing unexpected behavior in the applications that may be exploited by those with malicious intent. Secure code auditors and researchers today often use effective techniques such as fuzzing to detect such cases, by making applications handle unexpected malformations in the file or protocol. In the latter, their impact to security has been less discussed. They often do not cause an exploitable vulnerability, but are more likely to be used to evade content filters and malicious file and packet inspectors. This is more true, in the dawn of increasing complexity in Internet file formats, where new software are being developed by the day to support high demand in multimedia applications for business, entertainment, perhaps even mobile – video, audio, documents, vector animations et cetera.
This paper discusses how the complexity and ambiguity of today's Internet file formats provides malware authors with an avenue to evade most traditional file and packet inspectors. We have seen how they were used in the wild to defeat a majority of security products, in major incidents such as world-wide exploitation of the Microsoft Animated Cursor 0-day vulnerability (Exploit-ANIFile.c); as well as in targeted attacks using Microsoft OLE document streams. Finally, we will discuss the challenges of detecting them and how ambiguities should almost always raise an alarm, and needs to be fixed in file implementations.