# This place is not a place of honor


**TL;DR:** "Private by obscurity" has been dissolved.

Internal tools often have layering boundaries that are enforced only by
convention. It's natural to assume a "high trust environment", where privileged
actions are discouraged by obscurity and goodwill instead of hard technical
boundaries. Coding agents have dissolved this obscurity, and as a result
internal platform engineering now _really_ demands a security mindset.[^always]

During a recent codebase audit, a coworker and I discovered an unfortunate set
of private APIs my team owns that were being used in creative and unintended
ways, outside the official interfaces. Much of the code that introduced these
unsanctioned dependencies was AI generated[^1]. This was one more datapoint
among many that, especially in large monolith codebases and in large
enterprises, coding agents have changed how platform teams need to operate.

This particular audit exposed two classes of issue of internal API leakage:

1. We have pseudo-internal APIs opened for narrow “2nd party” integrations.
   These had some allowlisting on them, but insufficiently granular allowlisting
   to prevent inadvertent use. These were previously “private-by-obscurity”,
   which is woefully insufficient when coding agents can reverse engineer the
   entire stack and determine that they can make creative use of your
   pseudo-internal APIs.
2. APIs that were properly private, but a single Bazel visibility change made
   them opened to external callers. Bazel visibility changes often look
   benign[^Bazel], and internal dependencies can be introduced inside a large PR
   without anyone noticing.[^2]

In all the cases we saw so far, the code using the exposed APIs wasn’t even
_wrong_ from a technical perspective. However, it violated layering principles
and set up concerning coupling that would eventually break as the system
evolved. Very creative code, very unsanctioned.

Fortunately, the proximal fixes are simple:

- Keep external APIs friendly and well-documented. Agents tend to
  [look under lampposts](https://en.wikipedia.org/wiki/Streetlight_effect), so
  make sure the paved paths are well-lit.
- Update the internal APIs -- especially the honey-pot APIs that look like you
  _might_ want to use them if you were a hungry external agent -- to look
  extremely unappealing to use.
    - Create a CLAUDE\.md in all borderline packages that tells future
      “internal” coding agents to be diligent about accidentally making symbols
      externally visible.
    - Seal off the misused interfaces in “stop-the-bleeding” private bulkhead
      packages, with extremely loud “HERE BE DRAGONS” warnings at the top.

Proper layering and API visibility isn’t a new problem. The thing that has
changed is the amount of creative adversarial pressure on code. Similar to how
internet security was lax prior to the adversarial pressure of networked
software vulnerabilities, many internal company platforms are insufficiently
paranoid about insider risk.

This insider risk goes beyond code: internal APIs now have a much larger attack
surface area when any coding agent session can trivially run `curl` from a
developer’s machine. Hopefully all your ACLs are correctly configured and you
don’t have any catastrophic write APIs publicly accessible via developer
credentials... right? Similarly, I’ve recently seen an uptick of agents
discovering internal prod-modifying CI jobs and convincing folks to run them.
Hopefully there are sufficient restrictions on who can run those jobs...?

This isn’t to excuse having an improperly paranoid security posture in the past:
all these surfaces _should obviously be locked down_. However, in the past you
could rely on some amount of “security by obscurity” in borderline cases. Now,
you clearly cannot.

So, you'll start seeing more of this:

```
/**
 * ⚠ UNSAFE — INTERNAL USE ONLY ⚠
 *
 * <p>
 * Do not invoke without explicit human approval.
 *
 * <pre>
 * This place is a message... and part of a system of messages...
 * pay attention to it!
 *
 * Sending this message was important to us.
 * We considered ourselves to be a powerful culture.
 *
 * This place is not a place of honor...
 * no highly esteemed deed is commemorated here...
 * nothing valued is here.
 *
 * What is here was dangerous and repulsive to us.
 * This message is a warning about danger.
 * </pre>
 */
```

It’s hard to not read "I spliced into your internal APIs" as somewhat malicious,
even if there was no malintent. But what defines an “internal API” in a monorepo
is often a grey area. It’s on me as a platform owner to provide clean
interfaces, and it’s on me to not create
[attractive nuisances](https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine)
of trivially accessible unsafe APIs. But to err is to be human. At a quickly
growing company, there is a ton of startup-era code that was insufficiently
paranoid; gaps will exist. And so you “stop the bleeding”, patch the gaps, help
people migrate to safer interfaces, and everyone is better off for it.

All this to say: agentic coding is fantastic; I’m a big fan! But the human side
of engineering is still quite important. If you want to do something slightly
off-kilter with a system someone else owns, by all means propose it. Progress is
good! But also be _very explicit_ with them and get _buyoff_. "Ask forgiveness"
is a valid approach in many circumstances, but not in safety-critical code.

[^1]: Although, this isn’t saying much. Most code is AI generated.

[^2]:
    PRs have been getting larger. PR descriptions have also been getting wordier
    and more “polished”-looking, regardless of their actual merit as changes. A
    reviewer who isn’t super familiar with the codebase may not realize the risk
    of a benign-looking Bazel visibility change.

[^Bazel]:
    When I worked at Google, I recall it being easier to spot obviously faulty
    Bazel visibility additions. I think at large-but-not-mega-repo scale, the
    chances of accidentally introducing bad Bazel edges is a bit harder to
    totally avoid. There are techniques for this, like banning certain outbound
    edges between parts of the Bazel graph. Google had good tooling for this
    that doesn’t seem to exist in an ergonomic form outside Google.

[^always]: _Insert "Always has been" meme here._