Privacy Risks of General-Purpose Language Models
Systematically measures privacy leakage risks in general-purpose language models.
Original source ↗
NUWA LAB · RESEARCH
NUWA Lab Research
Define risk. Build safety with evidence across frontier AI risk, agent safety, systems security, cybersecurity, and privacy.
86 research works 6 research themes 6 honours
FEATURED WORK
Featured research
Five research programs defining frontier risk and practical control.
Autonomy Comes with Costs: Detecting Denial-of-Service Vulnerabilities Caused by Resource Abusing in LLM-based Agents
Presents AgentDoS, a lifecycle-aware fuzzing framework for detecting resource-abuse DoS vulnerabilities in LLM-based agents.
Original source ↗
Large language model-powered AI systems achieve self-replication with no human intervention
Executable tests connect self-replication, adaptation, and shutdown-survival evidence.
Original source ↗MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction
Uses simulation-derived reasoning correction to reduce unsafe actions in computer-use agents while preserving task utility.
Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
Thought-Aligner corrects unsafe reasoning before action while preserving task utility.
Original source ↗RESEARCH & PUBLIC IMPACT
Research and public impact
International consensus, research agendas, and national security standards connected to our research themes.
Shanghai AI Safety Consensus
Signed by 32 global experts, including Geoffrey Hinton, Yoshua Bengio, and Andrew Yao; the signatories include Turing Award and Nobel Prize laureates.
Consensus Statement on Ensuring Alignment and Human Control of Advanced AI Systems to Safeguard Human Flourishing
An international scientific consensus on alignment, human control, safety assurance, and verifiable behavioural red lines for advanced AI.
SELECTED SIGNATORIES
- Geoffrey HintonTuring Award · Nobel Prize
- Yoshua BengioTuring Award
- 姚期智 Andrew YaoTuring Award
Other signatories include Stuart Russell · Sam Bowman · Max Tegmark
- Alignment
- Human control
- Verifiable red lines
Singapore Consensus on Global AI Safety Research Priorities
Formed by more than 100 global contributors across 13 countries and four research-priority areas.
The 2026 Singapore Consensus on Global AI Safety Research Priorities
A global agenda for urgent AI safety research shaped by contributors from frontier developers, government safety institutes, academia, and civil society.
- Risk assessment
- Safe development
- Control
- Societal resilience
GB/T 45654—2025
Current national standard; published 2025.04.25 and effective 2025.11.01.
Cybersecurity technology—Basic security requirements for generative artificial intelligence service
A national baseline for training-data security, model security, service safeguards, and security assessment of generative AI services.
- Training data
- Model security
- Safeguards
- Security assessment
RESEARCH INFRASTRUCTURE
Research infrastructure & benchmarks
Public evaluation environments and benchmarks that make agent cybersecurity capability observable.
AgentCyberRange
Evaluates frontier AI systems' autonomous cyber-attack capabilities in complex enterprise environments.
Official site ↗AutoControl Arena
Synthesizes executable test environments that combine deterministic code state with narrative dynamics for frontier AI risk evaluation.
View paper ↗RESEARCH THEMES
Research themes
Frontier risk & AI control
Measure autonomy, deception, proliferation, and control integrity.
REPRESENTATIVEAutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
View this theme →Agent & model safety
Protect reasoning, behavior, generative models, and human–AI trust.
REPRESENTATIVEAutonomy Comes with Costs: Detecting Denial-of-Service Vulnerabilities Caused by Resource Abusing in LLM-based Agents
View this theme →Software, systems & agent security
Find vulnerabilities across agents, software, firmware, and cloud systems.
REPRESENTATIVEBACAgent: LLM-Powered Detection of Broken-Access-Control Vulnerabilities in Web Applications
View this theme →Cybersecurity, privacy & real-world risk
Study cybercrime, privacy leakage, abuse ecosystems, and emerging threats.
View this theme →Trustworthy ML & model security
Study robustness, backdoors, watermarks, poisoning, and model trust.
REPRESENTATIVEBELT: Old-School Backdoor Attacks can Evade the State-of-the-Art Defense with Backdoor Exclusivity Lifting
View this theme →AI systems & learning methods
Complete the publication record across learning systems and methods.
View this theme →RECOGNITION
Honours and recognition
Best Paper Award · NDSS
Top-tier cybersecurity conference
Falling Walls Science Breakthroughs Shortlist
International interdisciplinary science breakthrough selection
Distinguished Paper Award · NDSS
Top-tier cybersecurity conference
Distinguished Paper Award · ACM SIGSOFT
Top-tier software-engineering conference
WAIC Yunfan Prize
Distinguished Paper Award · USENIX Security Symposium
Top-tier cybersecurity conference
PUBLICATIONS
Research index
86 results
202620 works＋
AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
Synthesizes executable risk-evaluation environments that combine deterministic code state with LLM-generated narrative dynamics.
Autonomy Comes with Costs: Detecting Denial-of-Service Vulnerabilities Caused by Resource Abusing in LLM-based Agents
Presents AgentDoS, a lifecycle-aware fuzzing framework for detecting resource-abuse DoS vulnerabilities in LLM-based agents.
BACAgent: LLM-Powered Detection of Broken-Access-Control Vulnerabilities in Web Applications
Studies security risks and defenses for agents and foundation models.
Better Safe than Sorry: Uncovering the Insecure Resource Management in App-in-App Cloud Services
Uncovers insecure resource-management practices in app-in-app cloud services.
FirmCred: Detecting Authentication Bypass Vulnerabilities in Firmware from Credential Initialization Perspective
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
FirmCross: Detecting Taint-style Vulnerabilities in Modern C-Lua Hybrid Web Services of Linux-based Firmware
Detects taint-style vulnerabilities across C and Lua boundaries in firmware web services.
FlowGuard: Towards Lightweight In-Generation Safety Detection for Diffusion Models via Linear Latent Decoding
Detects unsafe diffusion outputs during the generation process by approximating latent decoding, enabling earlier and cheaper NSFW intervention.
Khost: KVM-based Near Native MCU Firmware Rehosting
Rehosts MCU firmware on KVM with near-native execution for scalable security analysis.
MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction
Uses simulation-derived reasoning correction to reduce unsafe actions in computer-use agents while preserving task utility.
One Email, Many Faces: A Deep Dive into Identity Confusion in Email Aliases
Reveals identity-confusion risks in email alias ecosystems and measures their security impact.
One Step from Silicon Life: Autonomous AI Agents Capable of Uncontrolled Self-Proliferation
Demonstrates autonomous agents acquiring external computational resources and propagating across remote devices under controlled, simulated real-world conditions.
OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation
Builds a lightweight framework to evaluate deception risk and user trust dynamics in open-ended human-AI dialogue.
PHPBench: Automated Generation of Verifiable and Hierarchical Benchmarks for PHP Web Fuzzing
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Position: Preparing for AI Systems That Deceive Developers
Frames deception targeting developers as a distinct frontier-AI risk and proposes recommendations for monitorability, evaluation integrity, and non-evadable control.
PRISON: Unmasking the Criminal Potential of Large Language Models
Evaluates LLM criminal potential across traits such as false statements, framing, psychological manipulation, emotional disguise, and moral disengagement.
Reproducing Web Application Vulnerabilities with Patch-Guided Routing Inference and Sink Exploration
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
StruPhantom: Evolutionary Injection Attacks on Black-Box Tabular Agents Powered by Large Language Models
Proposes an evolutionary prompt-injection attack that targets black-box LLM-powered tabular agents under structural payload constraints.
Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
Introduces Thought-Aligner, a plug-in method that causally corrects unsafe agent thoughts before actions are executed.
Unveiling the Resilience of LLM-Enhanced Search Engines Against Black-Hat SEO Manipulation
Measures how LLM-enhanced search systems respond to adversarial SEO manipulation.
When Fun Turns Toxic: A First Look at Aggressive Advertising in Mini-games
Characterizes aggressive advertising behavior and user risk across mini-game ecosystems.
202519 works＋
ADSDx: Towards Automated Accident Diagnosis for High-level Autonomous Driving Systems
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
ApkDiffer: Accurate and Scalable Cross-Version Diffing Analysis for Android Applications
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Applying Fuzz Driver Generation to Native C/C++ Libraries of OEM Android Framework: Obstacles and Solutions
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
APSFUZZ: Simulation-Based Fuzzing Testing for Automated Parking Systems
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Beyond Exploit Scanning: A Functional Change-Driven Approach to Remote Software Version Identification
Identifies remote software versions through functional changes instead of exploit-only probes.
Effective Directed Fuzzing with Hierarchical Scheduling for Web Vulnerability Detection
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Email Cloaking: Deceiving Users and Spam Email Detectors with Invisible HTML Settings
Demonstrates invisible HTML techniques that mislead both recipients and spam detectors.
Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems
Studies whether models recognize evaluation contexts and alter behavior, identifying observer effects that threaten safety-evaluation integrity.
Frontier AI systems have surpassed the self-replicating red line
Evaluates whether frontier AI systems can autonomously self-replicate and reports successful self-replication in controlled trials.
HouseFuzz: Service-Aware Grey-Box Fuzzing for Vulnerability Detection in Linux-Based Firmware
Uses service-aware feedback to improve vulnerability discovery in Linux-based firmware.
Large language model-powered AI systems achieve self-replication with no human intervention
Extends self-replication evaluation across 32 AI systems and reports autonomous replication, self-exfiltration, adaptation, and shutdown-survival behaviors.
Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents
Introduces AgentFuzz, a directed greybox fuzzing framework for finding taint-style vulnerabilities in real-world LLM-based agents.
NOKEScam: Understanding and Rectifying Non-Sense Keywords Spear Scam in Search Engines
Studies nonsense-keyword spear scams in search engines and evaluates mitigation paths.
RAG-Thief: Scalable Extraction of Private Data from Retrieval-Augmented Generation Applications with Agent-based Attacks
Studies security risks and defenses for agents and foundation models.
Security Debt in LLM Agent Applications: A Measurement Study of Vulnerabilities and Mitigation Trade-offs
Measures security debt in LLM-agent applications by studying known vulnerabilities and the trade-offs introduced by mitigation strategies.
SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse
Studies security risks and defenses for agents and foundation models.
Unveiling the (Ab)usage of Serverless Cloud Function in the Wild
Measures malicious and abusive uses of serverless cloud functions in the wild.
XSSky: Detecting XSS Vulnerabilities through Local Path-Persistent Fuzzing
Finds cross-site scripting vulnerabilities with local path-persistent fuzzing.
You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
Evaluates whether jailbreak defenses improve safety without degrading model utility, highlighting persistent trade-offs in practical LLM defense.
202412 works＋
An Underground Industry Application Collection Method Based on Flow Analysis（一种基于突变流量的在野黑产应用采集方法）
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
BELT: Old-School Backdoor Attacks can Evade the State-of-the-Art Defense with Backdoor Exclusivity Lifting
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
HADES Attack: Understanding and Evaluating Manipulation Risks of Email Blocklists
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Interface Illusions: Uncovering the Rise of Visual Scams in Cryptocurrency Wallets
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Matryoshka: Exploiting the Over-Parametrization of Deep Learning Models for Covert Data Transmission
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
Misdirection of Trust: Demystifying the Abuse of Dedicated URL Shortening Service
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Neural Dehydration: Effective Erasure of Black-box Watermarks from DNNs with Limited Data
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Revealing the black box of device search engine: scanning assets, strategies, and ethical consideration
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
SCTrans: Constructing a Large Public Scenario Dataset for Simulation Testing of Autonomous Driving Systems
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Towards Practical Backdoor Attacks on Federated Learning Systems
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
VioHawk: Detecting Traffic Violations of Autonomous Driving Systems through Criticality-guided Simulation Testing
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
202310 works＋
Anti-FakeU: Defending Shilling Attacks on Graph Neural Network based Recommender Model
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Cracking White-box DNN Watermarks via Invariant Neuron Transforms
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Exorcising “Wraith”: Protecting LiDAR-based Object Detector in Automated Driving System from Appearing Attacks
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
MaSS: Model-agnostic, Semantic and Stealthy Data Poisoning Attack on Knowledge Graph Embedding
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
RØROS: Building a Responsive Online Recommender System via Meta-Gradients Updating
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
Simulation-Based Fuzzing for Autonomous Driving Systems: Landscapes, Challenges and Prospects
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Under the Dark: A Systematical Study of Stealthy Mining Pools (Ab)use in the Wild
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Understanding and Detecting Abused Image Hosting Modules as Malicious Services
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
20226 works＋
Analyzing Ground-Truth Data of Mobile Gambling Scams
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Exploring the Security Boundary of Data Reconstruction via Neuron Exclusivity Analysis
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
House of Cans: Covert Transmission of Internal Datasets via Capacity-Aware Neuron Steganography
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
MetaV: A Meta-Verifier Approach to Task-Agnostic Model Fingerprinting
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Towards Backdoor Attack on Deep Learning based Time Series Classification
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
20216 works＋
A Deep Learning Framework for Self-evolving Hierarchical Community Detection
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
Detection and Analysis Technology of Cybercrime（网络犯罪的检测分析技术）
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Enhancing Time Series Predictors with Generalized Extreme Value Loss
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
Facilitating Vulnerability Assessment through Poc Migration
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
TAFA: A Task-Agnostic Fingerprinting Algorithm for Neural Networks
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Understanding the Threats of Trojaned Quantized Neural Network in Model Supply Chains
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
20207 works＋
A Geometrical Perspective on Image Style Transfer with Adversarial Learning
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
BScout: Direct Whole Patch Presence Test for Java Executables
Directly tests whether complete security patches are present in Java executables.
How Android Developers Handle Evolution-induced API Compatibility Issues: A Large-scale Study
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Improving the Robustness of Wasserstein Embedding by Adversarial PAC-Bayesian Learning
Studies robustness, backdoors, watermarks, poisoning, and trust in machine learning models.
Justinian’s GAAvernor: Robust Distributed Learning with Gradient Aggregation Agent
Studies security risks and defenses for agents and foundation models.
Modeling Personalized Out-of-Town Distances in Location Recommendation
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
Privacy Risks of General-Purpose Language Models
Systematically measures privacy leakage risks in general-purpose language models.
20191 work＋
Modeling Extreme Events in Time Series Prediction
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
20185 works＋
Detecting Third-party Libraries in Android Applications with High Precision and Recall
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Geographical Feature Extraction for Entities in Location-based Social Networks
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
How You Get Shot in the Back: A Systematical Study about Cryptojacking in the Real World
Investigates privacy, abuse ecosystems, and real-world cybersecurity threats.
Invetter: Locating Insecure Input Validations in Android Services
Advances vulnerability discovery, testing, and protection for software and intelligent systems.
Theoretical Analysis of Image-to-Image Translation with Adversarial Learning
Develops learning methods and AI systems across recommendation, vision, and time-series modeling.
COLLABORATION
Research collaboration
Collaborate on frontier-risk evaluation, agent safety, systems security, cybersecurity, and privacy.
Contact research