Large Language Models (LLMs) present a complex array of opportunities and vulnerabilities. Prompt injection and jailbreaking techniques have emerged as indispensable methodologies for probing the resilience of these models, expanding the horizons of their capabilities while uncovering potential weaknesses.
Amidst the recent spotlight on Microsoft's PyRIT, a constellation of LLM pen testing tools has gained more attention:
1. garak by Leon Derczynski
Derczynski's garak stands as a testament to the potency of prompt injection techniques, providing a lens through which to scrutinize the fortitude of LLMs in the face of adversarial inputs.
2. HouYi by YI LIU and Gelei Deng
HouYi's contribution lies in its innovative approach to LLM security, illuminating potential vulnerabilities and avenues of exploitation within these models.
3. JailbreakingLLMs by Patrick Chao, Alex Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong
A collaborative endeavor, JailbreakingLLMs delves into the intricacies of jailbreaking tailored for LLMs, offering invaluable insights into fortifying these systems against malicious incursions.
4. llm-attacks by Andy Zou, Zifan Wang, and Zico Kolter
The llm-attacks framework presents a comprehensive toolkit for assessing the security posture of LLMs, encompassing a myriad of attack vectors and defensive strategies.
5. PromptInject by Fábio Perez and Ian Ribeiro
PromptInject emerges as a beacon of vigilance against prompt injection attacks, furnishing pragmatic solutions for bolstering the security resilience of LLMs.
6. LLM-Canary by Jamie Cohen and Jackson Gor
LLM-Canary introduces novel methodologies for detecting anomalous behaviors within LLMs, serving as a preemptive safeguard against potential breaches.
7. PyRIT by Microsoft
Microsoft's PyRIT commands attention as a recent addition to the list of LLM pen testing tools, underscoring the burgeoning significance of AI security in the contemporary cybersecurity discourse.
Credits to Idan Gelbourt and Simo Jaanus for researching this list.
For those inclined towards deeper exploration, please see our past research on prompt injection detection:
Data Leakage Detection and Response for Enterprise AI Search
Learn how to assess and remediate LLM data exposure via Copilot, Glean and other AI Chatbots with Knostic.
The Data Governance Gap in Enterprise AI
See why traditional controls fall short for LLMs, and learn how to build policies that keep AI compliant and secure.
Rethinking Cyber Defense for the Age of AI
Learn how Sounil Yu’s Cyber Defense Matrix helps teams map new AI risks, controls, and readiness strategies for modern enterprises.
Extend Microsoft Purview for AI Readiness
See how Knostic strengthens Purview by detecting overshared data, enforcing need-to-know access, and locking down AI-driven exposure.
Build Trust and Security into Enterprise AI
Explore how Knostic aligns with Gartner’s AI TRiSM framework to manage trust, risk, and security across AI deployments.
Real Prompts. Real Risks. Real Lessons.
A creative look at real-world prompt interactions that reveal how sensitive data can slip through AI conversations.
Stop AI Data Leaks Before They Spread
Learn how Knostic detects and remediates oversharing across copilots and search tools, protecting sensitive data in real time.
Accelerate Copilot Rollouts with Confidence
Equip your clients to adopt Copilot faster with Knostic's AI security layer, boosting trust, compliance, and ROI.
Reveal Oversharing Before It Becomes a Breach
See how Knostic detects sensitive data exposure across copilots and search, before compliance and privacy risks emerge.
Unlock AI Productivity Without Losing Control
Learn how Knostic helps teams harness AI assistants while keeping sensitive and regulated data protected.
Balancing Innovation and Risk in AI Adoption
A research-driven overview of LLM use cases and the security, privacy, and governance gaps enterprises must address.
Secure Your AI Coding Environment
Discover how Kirin prevents unsafe extensions, misconfigured IDE servers, and risky agent behavior from disrupting your business.
Tags:
GenAI research findings
