今日云讲堂 | 苏炜杰：大语言模型水印统计框架

统计之都 · 公众号 · · 2024-05-25 12:05

正文

报告信息

主题：A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules

嘉宾：苏炜杰

地点：腾讯会议：782-711-380（或点击阅读原文）

时间：北京时间2024年05月25日（周六）21:00

报告摘要

Since ChatGPT was introduced in November 2022, embedding (nearly) unnoticeable statistical signals into text generated by large language models (LLMs), also known as watermarking, has been used as a principled approach to provable detection of LLM-generated text from its human-written counterpart. In this talk, we will introduce a general and flexible framework for reasoning about the statistical efficiency of watermarks and designing powerful detection rules. Inspired by the hypothesis testing formulation of watermark detection, our framework starts by selecting a pivotal statistic of the text and a secret key -- provided by the LLM to the verifier -- to enable controlling the false positive rate (the error of mistakenly detecting human-written text as LLM-generated). Next, this framework allows one to evaluate the power of watermark detection rules by obtaining a closed-form expression of the asymptotic false negative rate (the error of incorrectly classifying LLM-generated text as human-written). Our framework further reduces the problem of determining the optimal detection rule to solving a minimax optimization program. We apply this framework to two representative watermarks -- one of which has been internally implemented at OpenAI -- and obtain several findings that can be instrumental in guiding the practice of implementing watermarks. In particular, we derive optimal detection rules for these watermarks under our framework. These theoretically derived detection rules are demonstrated to be competitive and sometimes enjoy a higher power than existing detection approaches through numerical experiments.

嘉宾简介

Weijie Su is an Associate Professor at the University of Pennsylvania, with appointments in the Wharton Statistics and Data Science Department, Department of Mathematics (courtesy), and Department of Computer and Information Science (courtesy), where he is a co-director of the Penn Research in Machine Learning Center. Prior to joining Penn, he received his Ph.D. in Statistics from Stanford University in 2016 under the supervision of Emmanuel Candes and his bachelor’s degree in Mathematics from Peking University in 2011. His research interests span high-dimensional statistics, privacy-preserving data analysis, deep learning theory, and mathematical optimization. He serves as an associate editor of Journal of the American Statistical Association, Foundations and Trends in Machine Learning, Operations Research, and Journal of Machine Learning. He is a recipient of the Stanford Theodore Anderson Dissertation Award, an NSF CAREER Award, a Sloan Research Fellowship, the IMS Peter Gavin Hall Prize, the SIAM Early Career Prize in Data Science, the ASA Gottfried Noether Early Career Award, and the ICBS Frontiers of Science Award in Mathematics.

今日云讲堂 | 苏炜杰：大语言模型水印统计框架

正文

请到「今天看啥」查看全文