Resilient Self-Debugging Software Protection

EasyChair Preprint 3739

Abstract

Debuggers are a popular reverse engineering and tampering tool. Self-debugging is an effective technique for applications to defend themselves against hostile debuggers. In penetration tests on state-of-the-art self-debugging, we observed several opportunities through which it could be attacked, however. We therefore improved upon the existing technique, making it more resilient by introducing reciprocal debugging and making the transfers of control between protected application and self-debugger more stealthy. This paper presents the improved self-debugging design, and details our research efforts into realizing reciprocal debugging. In our evaluation we show that the improved design is significantly harder for attackers to defeat.

Keyphrases: Anti-tampering, anti-debugging, reverse engineering, self-debugging, software protection

@booklet{EasyChair:3739,
  author    = {Bert Abrath and Bart Coppens and Ilja Nevolin and Bjorn De Sutter},
  title     = {Resilient Self-Debugging Software Protection},
  howpublished = {EasyChair Preprint 3739},
  year      = {EasyChair, 2020}}