Reachability is a property of the graph, not of the rule

Ask a team whether one service can reach a particular database and you usually get a rule as the answer. Someone opens the security group, points at an entry, and says no, only the application tier is allowed. That is not an answer to the question. It is a reading of one layer out of five.

Reachability is not written down anywhere in a cloud environment. It is an emergent property of several independent systems composing, and the composition is not visible from any single console. This is the same structural point as treating an IAM policy as a graph rather than a list, one layer further down: the permission graph decides who can act, the network graph decides what can talk, and both are computed rather than declared.

Five layers, and what each one can and cannot say

A packet gets from A to B only if every layer permits it. Each layer has a different model, and the mismatch between those models is where the surprises live.

Layer	What it expresses	What it cannot see
Route tables	Whether a path to the destination exists at all	Anything about ports or identity
Subnet-level ACLs	Stateless allow and deny, evaluated in order	Connection state, so return traffic needs its own rule
Instance-level groups	Stateful allow only, often referencing other groups rather than addresses	Deny. Absence of allow is the only deny.
Peering, transit, endpoints	Which networks are joined, and how transitively	What is joined two hops away through someone else’s link
Host firewall and service config	What the process actually binds and accepts	Everything above it

Two rows in there routinely break intuition. Stateless ACLs mean a correct inbound rule with no matching return-path rule produces a connection that establishes and then hangs, which gets debugged as an application timeout for a day. And group-to-group references mean the effective source set of a rule is not a list of addresses but the membership of another group, which changes every time someone launches an instance.

A rule that references another group is not a rule about hosts. It is a rule about a set that other people are editing.

The traps that produce real incidents

Egress is the path nobody reviews

Inbound rules get scrutinised because that is where an attack is imagined to arrive. Outbound rules are usually wide open, because tightening them breaks package installs and nobody wants that argument in a sprint planning meeting. The result is an environment where a compromised process cannot be reached from outside but can reach anything it likes on the way out, which is the direction that matters once something is already running.

The honest version of the tightening argument is not “deny all egress”, which nobody sustains past the first broken build. It is: private subnets reach the outside through one controlled path, that path is a proxy or gateway you can log, and provider APIs are reached over private endpoints so that traffic never enters the general internet route at all. Then a permissive egress rule is scoped to a network that has nowhere interesting to go.

A private subnet is not an offline subnet

“Private” in cloud vocabulary means no inbound route from the internet. It says nothing about outbound. Add a gateway for updates, which every environment does, and every host in that subnet now has a working path out. Teams that describe a subnet as isolated are usually describing the inbound half and forgetting that they built the outbound half themselves, deliberately, six months earlier.

Non-transitive peering, transitive appliances

Peering is deliberately non-transitive: if A peers with B and B peers with C, A cannot reach C. That property then gets treated as a security boundary, and someone puts a shared appliance in B. A proxy, an ingress controller, a service mesh gateway, a monitoring collector with a permissive egress rule. The appliance is reachable from A and can reach C. Transitivity is restored at layer seven, invisibly to every network diagram.

This is the most common way two environments documented as separated turn out not to be. The connection is not a route. It is a process that both sides are allowed to talk to.

Compute the answer, do not read it

Because reachability is a composition, the only reliable way to answer a question about it is to evaluate the composition. Every major provider ships a reachability analyser that does this, and they are underused, largely because they get filed under compliance tooling rather than under debugging.

What the evaluation looks like, stripped to its shape:

# Reachability is a conjunction across layers, plus a transitive closure
# over anything that forwards. Both halves matter.

def can_reach(src, dst, port, model):
    if not model.route_exists(src.subnet, dst.ip):
        return False, "no route"

    # stateless ACLs: both directions, evaluated in rule order
    if not model.acl_allows(src.subnet, dst.ip, port, "egress"):
        return False, "acl egress"
    if not model.acl_allows(dst.subnet, src.ip, port, "ingress"):
        return False, "acl ingress"
    if not model.acl_allows(dst.subnet, src.ip, EPHEMERAL, "egress"):
        return False, "acl return path"          # the one everyone forgets

    # group rules: resolve group references to current membership
    if not model.group_allows(src.groups, dst.groups, port):
        return False, "security group"

    if not model.host_listens(dst, port):
        return False, "nothing bound"

    return True, "reachable"


def effective_reachability(model):
    # direct edges, then close over forwarders: proxies, gateways, meshes,
    # anything that accepts from one side and originates to the other.
    edges = {(a, b) for a in model.hosts for b in model.hosts
                    if can_reach(a, b, ANY, model)[0]}
    for f in model.forwarders:
        inbound  = {a for (a, b) in edges if b == f}
        outbound = {b for (a, b) in edges if a == f}
        edges |= {(a, b) for a in inbound for b in outbound}
    return edges

The second function is the part worth internalising even if you never write it. Direct edges are what a rule review shows you. The closure over forwarders is what actually determines who can talk to whom, and it is invisible in every layer’s own console because no layer knows what a forwarder is.

Questions that get you a real answer

Reviews go badly when the question is “are the rules correct”, because rules are always locally defensible. These are harder to answer and worth more:

1. From this host, what is the full set of destinations reachable on any port? Not what is allowed. What is reachable. If nobody can produce the set, nobody knows the blast radius of that host being compromised.
2. Which paths exist only because of a forwarder? Those are the edges your diagram is missing.
3. What does the data store accept from, resolved to current group membership rather than to group names?
4. If every outbound connection were logged, what would we see that we cannot currently explain? Run it for a week in a non-production environment before arguing about the answer.
5. Which of these rules exists because of a decision, and which because of an incident at 2am? The second kind is where the permissive entries are, and they are never labelled.

None of this requires new tooling or a project. It requires accepting that the network topology you believe in is a summary somebody drew once, and the real one is a computed set that changes whenever an instance launches. The gap between those two is not a documentation problem. It is where the incident comes from.