What security leaders need to know from our recent webinar with Knostic CTO Sounil Yu

The inflection point: Open source meets machine‑speed vuln discovery

The XKCD “tiny module in Nebraska” joke stopped being funny when one exhausted maintainer essentially confirmed it. AI‑assisted research is accelerating vulnerability discovery across widely integrated libraries. The result: an unpayable remediation debt landing on volunteer maintainers who can’t patch fast enough and are increasingly disclosing issues publicly by default. The community is entering a feedback loop where more findings yield more pressure and less capacity, precisely when downstream dependencies are ubiquitous.

For CISOs, the takeaway is not “abandon open source,” but to recognize that the risk surface is now paced by machine‑speed discovery while remediation remains human‑bounded. Governance and supplier expectations need to reflect that asymmetry.

Zero Trust in operations ≠ Zero Trust in construction

We’ve done decent work segmenting run‑time environments, enforcing identity controls, and tightening network pathways. But the construction side of the house, how software is assembled, still assumes third‑party modules are stable and safe. That’s where many exploitable conditions originate: inside dependencies and at the seams between them. The supply chain problem is shifting left into build systems, assistants, and the connective tissue of developer tools.

A provocative future: Custom code, no SBOM dependencies and a perverse incentive

Sounil outlined a scenario security teams are beginning to test: instruct coding assistants to generate all functionality without importing open‑source packages. You still benefit from the corpus as a template, but you ship bespoke code. The immediate side effect: your SBOM lists no third‑party components and your SCA shows zero known CVEs. On paper, that looks pristine.

This creates a compliance‑driven perverse incentive. If customers and regulators over‑weight “zero known CVEs,” suppliers may chase empty SBOMs, pushing more generated code whose provenance, variability, and security properties we don’t yet fully understand. The right governance response is not to penalize open‑source usage, but to evaluate outcomes (behavior under test) rather than ingredients alone.

Coding assistants in the danger zone

Developer machines are among the most privileged endpoints in the enterprise. Coding assistants amplify that privilege. A single risky action, say, an LLM‑generated rm -rf ~, can wipe a home directory or worse. Turn on “YOLO”/"dangerously skip permissions" modes and you’ve eliminated your last line of defense.

Add to that:

• Prompt/indirect injection via files, comments, and hidden characters

• Leaky environment variables that look like “credentials I found for you”

• Autonomous tool use with limited context on staging vs. production

It’s a perfect storm for accidental self‑harm and attacker abuse unless we add guardrails inside the IDE.

Operationalizing safety: The Agent’s Rule of Two

Meta recently reframed Chrome’s historical “Rule of Two” for agents: at any moment, allow an AI system only two of three properties:

1. Untrusted inputs

2. Access to sensitive data

3. Ability to communicate externally

If an agent must accept spammy email, don’t also let it exfiltrate or touch secrets without a human in the loop. If it can talk to the internet and read internal docs, lock down its inputs to trusted sources. In developer contexts, we routinely (and dangerously) enable all three.

What good looks like: detect when an assistant is about to enter the “all‑three” state, gate high‑risk actions, and require justification or review. Done well, this feels like a seatbelt that only clicks under hard braking.

Treat the IDE as the new endpoint (introducing Kirin)

Security has strong network controls and decent OS‑level controls. But developer risk now concentrates inside the IDE, where agents run tools, call MCP servers, and mutate code. Knostic’s Kirin takes an endpoint‑style approach to the IDE itself: instrumenting actions, applying the Agent’s Rule of Two, and catching dangerous patterns before they land. Think “browser security for developers,” tuned for MCP, extensions, tool invocations, and code‑mod operations.

Extensions and MCP servers: Trust, but verify (and verify again)

The emerging ecosystems around IDE extensions and MCP servers are vibrant and noisy. Malicious or sloppy packages can smuggle hidden instructions aimed squarely at coding assistants. Even when marketplaces respond quickly, repackaging and resurfacing are common. Choice paralysis doesn’t help either: when there are many unofficial connectors for the same SaaS, how do developers pick the right one? Until robust attestation, reputation, and verification catch up, security teams need compensating controls at point‑of‑use.

Practical steps:

• Maintain allow‑lists/deny‑lists of extensions and MCP endpoints, with a review path for exceptions.

• Scan for hidden control characters and embedded prompts across project files and diffs.

• Block or sandbox “dangerous by default” actions (credential access, DB schema changes, destructive file ops) without explicit human approval.

Making sense of “agents” vs. “agentic AI” using an OODA lens

Confused by the terminology? Use this working model:

• Tools: machines sense and act with human‑defined logic (e.g., auto‑patching a workstation).

• AI agents: add machine sense‑making (e.g., SOAR systems with LLM reasoning that choose a playbook).

• Agentic AI: go further. Machines also decide which goals or sequences to pursue.

The loop emerges when one system’s action becomes another’s input. Chain a few together and you shift from predictable workflows to non‑deterministic networks of behaviors. That’s powerful and it multiplies failure modes.

Determinism, non‑determinism, and why attackers love “weird machines”

Traditional IT felt deterministic: same inputs, same outputs. In reality, latent states and edge conditions always existed. Attackers just turned them into repeatable exploit paths. In the AI era, we start closer to non‑deterministic by design. Connect agentic systems and you get emergent behaviors, or what the exploitation literature calls weird machines: unintended, but functional behaviors living inside your intended system.

Not all weirdness is bad (SMS and GPS grew from unintended affordances). But in security, weird machines create surprising pathways for exfiltration, privilege escalation, or data destruction. The lesson for CISOs: stop assuming predictability; instrument for behavioral verification rather than just code provenance.

What CISOs and AI security leaders should do now

1. Instrument the IDE. Treat developer environments as first‑class security endpoints. Monitor assistant actions, MCP calls, and file/diff contexts rather than only network/OS events.

2. Enforce the Agent’s Rule of Two. Detect entry into the “all‑three” danger zone and gate it. Provide workflow‑friendly just‑in‑time approvals.

3. Shift from ingredients to outcomes. Don’t over‑rotate to empty SBOMs. Evaluate behavior via tests, adversarial prompts, and policy simulations on generated artifacts.

4. Harden extensions and MCP usage. Curate marketplaces via allow‑lists, provenance checks, and continuous scanning for hidden instructions and risky capabilities.

5. Guard privileged actions. Require human confirmation for destructive operations, credential access, schema migrations, and production data touchpoints.

6. Plan for non‑determinism. Assume agent chains will branch. Design observability and rollback around behaviors, not just releases. Favor reversible actions and fast recovery over fragile prevention.

7. Adopt moving‑target thinking. Frequent refactoring and regeneration can make exploitation harder, paired with robust tests and controls. (See Sounil’s D.I.E. lens: systems that are Distributed, Immutable, and Ephemeral are harder to pin down.)

Where this is headed

Agentic development workflows aren’t a thought experiment. They’re already here, unevenly distributed. The near future looks like:

• More generated code, fewer stable “upstream” references

• Richer IDE toolchains that behave like mini‑operating systems

• Marketplaces of MCP servers and extensions with variable quality and opaque trust signals

• A premium on runtime behavioral assurance over static attestations

Security’s north star in this world is simple to say and hard to do: constrain capabilities, verify outcomes, and recover fast when the weird emerges.

Watch the replay & continue the conversation

Want the full deep dive from Sounil Yu? Watch the webinar replay for the complete walkthrough of the Agent’s Rule of Two, the OODA model for agents, and how we’re operationalizing safeguards in real developer environments.

If you’re exploring how to secure coding assistants, MCP, extensions, and agentic workflows in your org, contact us. We’re happy to share what we’ve learned and show how Kirin helps teams instrument the IDE, enforce the Rule of Two, and keep agentic environments productive without letting them be weird.