CITP Seminar – Auditing Large Language Model

The emergence of large language models (LLMs) represents a major advance in artificial intelligence (AI) research. However, the widespread use of LLMs is also coupled with significant ethical and social challenges. Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing procedures fail to address the governance challenges posed by LLMs, which are adaptable to a wide range of downstream tasks. To help bridge that gap, Mokander will present a novel blueprint for how to audit LLMs. By drawing on best practices from IT governance and system engineering, he and his collaborators at the University of Oxford propose a three-layered approach, whereby governance audits, model audits, and application audits complement and inform each other. Ultimately, this research seeks to expand the methodological toolkit available to technology providers and policymakers who wish to analyze and evaluate LLMs from technical, ethical, and legal perspectives. However, it is still work in progress, so all feedback is welcome.

Bio: Jakob Mökander is a Visiting Scholar at the Center for Information Technology Policy. His research – which sits at the intersection of ethics, law, management studies and systems engineering – focuses on developing and evaluating methods to audit automated decision-making systems. The aim thereby is to provide both tech companies and policymakers with the tools they need to ensure that automated decision-making systems are designed and deployed in ways that are legal, ethical, and safe.

Mökander’s home institution is the University of Oxford, where he is pursuing a doctorate in philosophy under the supervision of Luciano Floridi, Professor of Philosophy and Ethics of Information at the Oxford Internet Institute (OII). Mökander’s doctoral research is funded through an Oxford-AstraZeneca studentship.

Academically, Mökander holds one MSc in Social Science of the Internet from the University of Oxford, and one MSc in Industrial Engineering and Management from Linköping University in Sweden. Prior to joining the OII, Mökander was posted at the Swedish Trade & Invest Council in New Delhi, India, where he facilitated international industrial R&D projects.

For an overview of Mökander’s published work, visit his Google Scholar page.

---

To request accommodations for a disability please contact Jean Butcher, butcher@princeton.edu, at least one week prior to the event.

This talk will be recorded. The video will be posted to the CITP website, the CITP YouTube channel and the Princeton University Media Central channel.

Watch the webinar here.