Nicholas Carlini

Most-cited papers

Universal and Transferable Adversarial Attacks on Aligned Language Models
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr +2 more
2023arXiv.org3,942 citations
Semantic Scholar
Extracting Training Data from Large Language Models
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski +8 more
2020USENIX Security Symposium3,496 citations
Semantic Scholar
Membership Inference Attacks From First Principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song +2 more
2021IEEE Symposium on Security and Privacy1,326 citations Open access
Semantic Scholar
Extracting Training Data from Diffusion Models
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski +5 more
2023USENIX Security Symposium1,094 citations Open access
Semantic Scholar
Quantifying Memorization Across Neural Language Models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee +2 more
2022International Conference on Learning Representations1,035 citations
Semantic Scholar
Deduplicating Training Data Makes Language Models Better
Katherine Lee, Daphne Ippolito, A. Nystrom, Chiyuan Zhang +3 more
2021Annual Meeting of the Association for Computational Linguistics1,029 citations Open access
Semantic Scholar
Label-Only Membership Inference Attacks
Christopher A. Choquette-Choo, Florian Tramèr, Nicholas Carlini, Nicolas Papernot
2020International Conference on Machine Learning666 citations
Semantic Scholar
Scalable Extraction of Training Data from (Production) Language Models
Milad Nasr, Nicholas Carlini, J. Hayase, Matthew Jagielski +6 more
2023arXiv.org631 citations
Semantic Scholar
Are aligned neural networks adversarially aligned?
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski +7 more
2023Neural Information Processing Systems420 citations Open access
Semantic Scholar
Poisoning Web-Scale Training Datasets is Practical
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka +5 more
2023IEEE Symposium on Security and Privacy412 citations Open access
Semantic Scholar
Defeating Prompt Injections by Design
Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes +6 more
20252026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)382 citations Open access
Semantic Scholar
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot +1 more
2021IEEE Symposium on Security and Privacy323 citations Open access
Semantic Scholar
Poisoning and Backdooring Contrastive Learning
Nicholas Carlini, A. Terzis
2021International Conference on Learning Representations237 citations
Semantic Scholar
Stealing Part of a Production Language Model
Nicholas Carlini, Daniel Paleka, K. Dvijotham, Thomas Steinke +10 more
2024International Conference on Machine Learning220 citations
Semantic Scholar
Counterfactual Memorization in Neural Language Models
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski +2 more
2021Neural Information Processing Systems219 citations Open access
Semantic Scholar
(Certified!!) Adversarial Robustness for Free!
Nicholas Carlini, Florian Tramèr, K. Dvijotham, L. Rice +2 more
2022International Conference on Learning Representations186 citations Open access
Semantic Scholar
The Privacy Onion Effect: Memorization is Relative
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot +2 more
2022Neural Information Processing Systems176 citations Open access
Semantic Scholar
Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets
Florian Tramèr, R. Shokri, Ayrton San Joaquin, Hoang M. Le +3 more
2022Conference on Computer and Communications Security163 citations Open access
Semantic Scholar
Measuring Forgetting of Memorized Training Examples
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito +7 more
2022International Conference on Learning Representations160 citations Open access
Semantic Scholar
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander Schulhoff +10 more
2025arXiv.org159 citations
Semantic Scholar
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang +4 more
2022arXiv.org149 citations Open access
Semantic Scholar
Scalable Extraction of Training Data from Aligned, Production Language Models
Milad Nasr, Javier Rando, Nicholas Carlini, J. Hayase +6 more
2025International Conference on Learning Representations145 citations
Semantic Scholar
Backdoor Attacks for In-Context Learning with Language Models
Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, Nicholas Carlini
2023arXiv.org143 citations Open access
Semantic Scholar
Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang +4 more
2022International Conference on Natural Language Generation129 citations Open access
Semantic Scholar
Effective Prompt Extraction from Language Models
Yiming Zhang, Nicholas Carlini, Daphne Ippolito
2023120 citations
Semantic Scholar
Considerations for Differentially Private Learning with Large-Scale Public Pretraining
Florian Tramèr, Gautam Kamath, Nicholas Carlini
2022International Conference on Machine Learning116 citations Open access
Semantic Scholar
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot +1 more
2020International Conference on Machine Learning106 citations
Semantic Scholar
This paper is included in the Proceedings of the 32nd USENIX Security Symposium.
Milad Nasr, Jamie Hayes, T. Steinke, Borja Balle +6 more
94 citations
Semantic Scholar
SoK: Watermarking for AI-Generated Content
Xuandong Zhao, Sam Gunn, M. Christ, Jaiden Fairoze +10 more
2024IEEE Symposium on Security and Privacy86 citations Open access
Semantic Scholar
Query-Based Adversarial Prompt Generation
J. Hayase, Ema Borevković, Nicholas Carlini, Florian Tramèr +1 more
2024Neural Information Processing Systems73 citations
Semantic Scholar
Data Poisoning Won't Save You From Facial Recognition
Evani Radiya-Dixit, Sanghyun Hong, Nicholas Carlini, Florian Tramèr
2021International Conference on Learning Representations73 citations
Semantic Scholar
On Evaluating the Durability of Safeguards for Open-Weight LLMs
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang +6 more
2024International Conference on Learning Representations67 citations
Semantic Scholar
Debugging Differential Privacy: A Case Study for Privacy Auditing
Florian Tramèr, A. Terzis, T. Steinke, Shuang Song +2 more
2022arXiv.org67 citations
Semantic Scholar
Persistent Pre-Training Poisoning of LLMs
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi +4 more
2024International Conference on Learning Representations62 citations
Semantic Scholar
Privacy Side Channels in Machine Learning Systems
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette-Choo +4 more
2023USENIX Security Symposium62 citations Open access
Semantic Scholar
Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI
R. Honig, Javier Rando, Nicholas Carlini, Florian Tramèr
2024International Conference on Learning Representations56 citations
Semantic Scholar
Students Parrot Their Teachers: Membership Inference on Model Distillation
Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A. Choquette-Choo +2 more
2023Neural Information Processing Systems55 citations Open access
Semantic Scholar
Is Private Learning Possible with Instance Encoding?
Nicholas Carlini, Samuel Deng, Sanjam Garg, S. Jha +4 more
2021IEEE Symposium on Security and Privacy54 citations
Semantic Scholar
Forcing Diffuse Distributions out of Language Models
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter +1 more
2024arXiv.org53 citations
Semantic Scholar
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
J. Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr +1 more
2019arXiv.org48 citations
Semantic Scholar
An Attack on InstaHide: Is Private Learning Possible with Instance Encoding?
Nicholas Carlini, Samuel Deng, Sanjam Garg, S. Jha +5 more
2020arXiv.org41 citations
Semantic Scholar
No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"
Nicholas Carlini, V. Feldman, Milad Nasr
2022arXiv.org30 citations Open access
Semantic Scholar
Increasing Confidence in Adversarial Robustness Evaluations
R. Zimmermann, Wieland Brendel, Florian Tramèr, Nicholas Carlini
2022Neural Information Processing Systems25 citations Open access
Semantic Scholar
MLSys: The New Frontier of Machine Learning Systems
Alexander J. Ratner, Dan Alistarh, G. Alonso, D. Andersen +65 more
201925 citations
Semantic Scholar
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Yangsibo Huang, Milad Nasr, A. Angelopoulos, Nicholas Carlini +9 more
2025International Conference on Machine Learning24 citations
Semantic Scholar
Large-scale online deanonymization with LLMs
Simon Lermen, Daniel Paleka, Joshua P. Swanson, Michael Aerni +2 more
2026arXiv.org23 citations
Semantic Scholar
Position: Adversarial ML for LLMs Is Not Making Any Progress
Javier Rando, Jie Zhang, Nicholas Carlini, Florian Tramèr
202523 citations
Semantic Scholar
Report of the 1st Workshop on Generative AI and Law
A. F. Cooper, Katherine Lee, James Grimmelmann, Daphne Ippolito +31 more
2023Social Science Research Network23 citations
Semantic Scholar
SysML: The New Frontier of Machine Learning Systems
Alexander J. Ratner, Dan Alistarh, G. Alonso, D. Andersen +62 more
2019arXiv.org23 citations
Semantic Scholar
NeuraCrypt is not private
Nicholas Carlini, Sanjam Garg, S. Jha, Saeed Mahloujifar +2 more
2021arXiv.org21 citations
Semantic Scholar