Unlocking Interaction via Pixel-Based Reverse Engineering

Morgan Dixon

User interface software has become a victim of its own success. Graphical event-driven systems are the dominant user interface for computational devices, realized via popular user interface toolkits. However, this success has induced a lock-in effect that demonstrably stifles innovation. For example, human-computer interaction researchers regularly propose new interaction techniques for desktop and mobile devices, enhancements for people with disabilities, better support for localization, and new collaboration facilities. However, these techniques rarely make it outside the laboratory because existing software tools cannot support them. Changing the ecosystem would require a mass adoption of new tools, and most of the applications we use today would need to be rewritten.

I will describe an approach that unlocks existing graphical application without their source code and without their cooperation, making it possible to explore and deploy many promising ideas in the context of existing real-world interfaces. Prefab is a system that reverse-engineers graphical interfaces from raw pixels. I will show how Prefab can be used to augment user interfaces with a variety of techniques, including accessibility enhancements, improved input for touch screens, UI language translation, and end-user customization. Some of these enhancements were proposed in the literature over the last two decades, but prior to Prefab they had been considered nearly impossible to deploy in practice.

Morgan Dixon is a PhD student in Computer Science & Engineering at the University of Washington working with James Fogarty. His research is centered around designing, building, and studying systems that make user interface software more flexible and extensible. He has also worked on novel interaction techniques for accessibility, dual-display mobile devices, health technologies, crossing-based interfaces, and data analysis tools. Morgan received a B.S. in Computer Science and Mathematics from the University of Maryland in 2008, and an M.S. from the University of Washington in 2010. His work on Prefab received a CHI Best Paper Award, and he was named a Microsoft Research PhD Fellow in 2011.