LEStra - the vibe coded version of VestacP

mikho

It all started with a discussion about vibe coding, was it good or bad, ethical or unethical?

From that thread, a challenge was accepted in the cest-pit ... and a new thread were created about this project. https://lowendspirit.com/discussion/11268/vibe-coding-a-vestacp-alternative

That thread is the discussion thread about this project. This thread is only for the documentation and story abou the project.
As one of the main things said in the beginning was to keep manual entries at a minimum, most of the underlaying sources are created by Claude Code.

After each Plan phase and each Code phase, updates are timestamped and posted in a Decision.log.md file. That is the main source.

At first the plan was also to release this as seperate blog posts, but already, a couple of days into the project, we already have 10 parts, each part is aprox. a 10 minute read.

So this thread was created (and closed), to have everything without comments cluttering up the thread.
If you have a comment about it, use the thread mentioned above.

mikho

Not a patch. A demolition.

VestaCP has a function called search_objects. Ask it for a domain's IP address and it does not parse a config file. It runs eval on the config file. The value you asked for gets executed as shell code on its way back to you, on every list screen, every single time. This is not a bug report about one bad line. It's the actual read path for the entire panel, and it's the reason this series exists.

This is the first post in a new series: the build log for LESta, a from-scratch Laravel and React rewrite of VestaCP, written almost entirely through prompts to Claude Code rather than by hand. Each part covers real ground the project has already crossed, not a roadmap of what's planned. This part covers why the rewrite happened at all, which means it starts with what's wrong with the thing being replaced.

What Vesta actually is

If you've spent any time in the budget-VPS end of hosting, you've probably run into VestaCP already, or at least a fork of it (HestiaCP and others exist for a reason, and that reason is on this page). It's a free, open-source control panel: web server, DNS, mail, databases, cron, backups, all managed from one dashboard, no cPanel license fee attached. For a lot of small hosts and hobbyists, it's been the default answer to "how do I give someone a panel without paying per account for the privilege."

It's also, underneath the dashboard, a React single-page app talking to a PHP API that does nothing but build shell command strings, which get handed to 379 bash scripts, which read and write their state as flat text files under /usr/local/vesta/data. Four tiers, three languages, and the type system evaporates completely at every single boundary between them.

Part of why this project exists at all is that Vesta has been quietly load-bearing for the cheap end of hosting for well over a decade. It's the reason HestiaCP forked off it in 2017 rather than starting from nothing, and a fair few one-person hosting operations you've probably bought a VPS from at some point are running it, or a fork of it, right now, on a box you're paying five dollars a month for. That's not a knock. It's exactly why the state of the code underneath it is worth taking seriously instead of writing off as "old software, who cares."

A full engineering pass against the codebase, pinned at commit 5f4ee2ef on outroll/vesta so every number below is checked against one exact snapshot rather than "whatever's on GitHub today", came back with some numbers worth sitting with before touching any code:

• 102,000 lines across bash, PHP, and JavaScript.
• 379 bash scripts, 34,600 lines, doing literally all the business logic.
• 136 PHP endpoints, 42,900 lines, whose entire job is session handling and building shell commands.
• 4,666 lines duplicated across 12 near-identical list screens, the same seventeen functions copy-pasted with the field names swapped.
• Zero write locks, across every one of the 103 scripts that mutate shared state.

That last number is the one to actually worry about, but it's not even the strangest one on the list.

Reading a record means running it

Here's the read path, straight out of func/main.sh:341:

# /usr/local/vesta/data/users/admin/web.conf
DOMAIN='example.com' IP='10.0.0.1' U_DISK='420' SUSPENDED='no' ...

# func/main.sh:341, how search_objects reads a record back
search_objects() {
    for line in $(grep $2=\'$3\' $USER_DATA/$1.conf); do
        eval $line          # stored data becomes shell code
        eval echo \$$4
    done
}

Every record on disk is a line of shell variable assignments. Reading one back means feeding it to eval, twice, in the same function. This isn't a one-off shortcut somebody forgot to clean up. It's the load-bearing mechanism for the entire persistence layer, and a dedicated grep for it turns up 105 sites doing exactly this across bin/ and func/, out of 191 total eval calls in the tree once you count every use rather than just the record reads.

The format validators that are supposed to keep a stored value safe to eval are blocklists of shell metacharacters, not allowlists, and they're inconsistently applied between resource types. Which means the actual security boundary between "text a tenant typed into a form" and "code that runs as root the next time anyone lists that resource" is: whatever characters somebody remembered to blocklist for that particular field, checked once, at write time, years ago.

The read path for every record in the panel, in one function.

The second bug makes the first one invisible

Every single one of Vesta's v-list-* scripts builds its JSON output with echo, no escaping. A mail autoreply body, a cron command line, a user's last name, anything with a stray quote or backslash in it corrupts the JSON. That's 73 of the 80 v-list-* scripts, silently capable of shipping broken output the moment a tenant types an apostrophe somewhere reasonable, like their own name.

Normally that would at least throw a visible error. It doesn't, because line 2 of essentially every one of Vesta's API endpoint files reads error_reporting(NULL). PHP's json_decode gets the malformed JSON, returns null, the endpoint tries to iterate it anyway, and the whole thing fatals behind a wall that was specifically built to hide fatal errors from the person trying to debug them. The operator sees a blank page. The log stays empty. Nothing anywhere tells you what just happened, because that's the one thing this line of code was written to prevent.

This is also, not coincidentally, the reason a bug in the field that reports which file extensions get cached by the reverse proxy has apparently been shipping wrong output to every user of that feature for who knows how long, with zero errors logged, because nobody could see it happen.

And then there's the one that just writes files

No path constraint. None.

Buried in the API bootstrap, at web/api/index.php:105 and its byte-identical twin web/api/v1/index.php:105 (they're the same file, copy-pasted, which is its own small horror), there's a branch called v-make-tmp-file. It takes $_POST['arg2'] as a path and $_POST['arg1'] as content, and writes the second to the first. No path constraint. None. It sits behind authentication, which is the only thing standing between "logged in as a normal tenant" and "arbitrary file write anywhere the web server's PHP process can reach."

Put those three together, the eval-based reads, the error suppression that hides the JSON bug, and the unconstrained file write, and you get a fairly complete picture of why the answer here isn't "patch a few lines." And that's before getting to the missing locks. Every write in the panel is a full read-modify-write of a flat file, and there isn't a single flock anywhere in the 103 scripts that mutate one. Picture two admins on the same reseller account creating a domain each within the same second, which happens constantly on a busy panel. Both scripts read the account's current domain count, both add one, both write back the same "new" total. One domain gets created for real. The counter only ever heard about one of them. Nothing crashes, nothing logs an error, the quota just quietly drifts from reality until someone's disk usage stops adding up months later and support has no idea why.

None of this is a hypothetical read off a summary somewhere either. Every claim above was independently re-checked against that one pinned commit before it went into any planning document, because trusting a report about a codebase is exactly the kind of thing this project has opinions about. More on that discipline specifically in part 3, once there's an actual threat model to show for it.

So, instead of fixing it

The obvious move for a lot of these findings is: turn on error reporting, add locks, delete the duplicate file, ship a patch. That's genuinely what a fix-forward pass on Vesta would look like, and honestly not a bad plan if the goal were keeping Vesta alive.

That wasn't the goal here. The actual first move on this project wasn't opening an editor. It was a prompt, and a fairly blunt one: act like a senior technical lead who's going to own this thing for five years, don't just implement what's asked, challenge weak decisions, and work through the problem as four distinct roles in the same conversation, an Architect who designs, an Engineer who specifies concrete artifacts, a Reviewer who attacks the result for security and operational failure modes, and an Optimizer who checks it for long-term maintenance cost. Worth being precise about what that actually is: one model, Claude Code, reasoning through a design from four different angles in sequence, not four separate AIs arguing with each other. But the framing works, because it's the same discipline a real review process would apply, applied consistently instead of skipped under deadline pressure.

Four roles, one model, taken in sequence instead of skipped.

What came back from that first prompt was the governing plan LESta still measures itself against: Laravel 13 and React 19 on the control plane, holding every bit of relational desired state, quotas, audit history, and provisioning jobs. Nothing about the actual host, the web server config, the mail queue, the firewall, gets touched by Laravel directly. Every host change goes through a separately deployed Go node agent, over a versioned protocol, and only through named, schema-validated, idempotent operations. No shell string ever crosses that boundary.

The plan also wrote down, right at the start, a short list of things that are simply not allowed to exist in this codebase, ever, regardless of how convenient they'd be in a pinch: no generic RunCommand, no arbitrary command-plus-arguments API, no exec calls from a controller, no wildcard sudo, no raw user filesystem paths, no secrets in props or logs or job payloads, and no executable queue files. Each one maps to something specific found above. No raw filesystem paths is the direct answer to v-make-tmp-file taking a POST field straight to fopen(). No wildcard sudo answers the fact that every v-* script runs as root regardless of what it actually needs to touch, so a bug anywhere in 379 scripts is a bug with full host privilege. No executable queue files is the broadest one, closer to a principle than a citation. It generalizes the actual root cause underneath all of them, stored data getting treated as code, so the same failure mode can't quietly reappear somewhere else in LESta under a different name, in a queue payload or a config template instead of a .conf file.

What's coming

Part 2 covers what actually got built first, and it isn't a web hosting feature. It's the rules for how the underlying host gets set up in the first place, an entire installation architecture milestone, decided and documented before a single tenant-facing feature existed. Part 3 covers the moment a background research agent went and actually read the old Vesta source with intent, and came back with real product-policy questions nobody had thought to ask yet. Part 4 is where planning stops and code starts, including the four real bugs the first implementation actually shipped with, and how they got caught.

For now: a rewrite whose entire premise is "stop treating stored data as executable code" being substantially written by a large language model is either the funniest possible way to fix this, or the only genuinely sane one. The rest of this series is basically the argument for the second reading.

mikho

Part 1 covered why VestaCP got condemned rather than patched: a panel that reads its own config files back into the shell with eval, at 105 sites, with zero write locks anywhere, plus an authenticated arbitrary file write sitting in the public API. This part covers what LESta actually built first, and it wasn't a web hosting feature. It was an entire milestone on how a bare Ubuntu server gets turned into a working node in the first place, decided and written down before a single tenant-facing screen existed.

If you've only ever installed a panel with curl | bash, this is going to feel like overkill. That's the point.

The gap the plan left open

The governing rewrite plan from part 1 already committed to some big things: a Go node agent as the only thing allowed to touch a host, staged config rendering with syntax validation and atomic activation, and a "replace curl-to-root installation with a signed, pinned deployment process" line buried near the end. What it didn't say was how a blank Ubuntu box gets nginx, MariaDB, BIND, Exim, and the rest onto it in the first place, or who owns the resulting files on disk for the rest of the product's life.

That gap became the second real prompt in this project, and it's worth quoting the framing directly, because it's a good example of what "vibe coding with actual discipline" looks like in practice rather than just typing "build me an installer" and hoping:

Continue as the four-agent team defined in the first prompt... Act as a senior technical lead who will own this product for five or more years. Challenge weak decisions in the plan rather than implementing them faithfully... Do NOT write any code at this point, we are still in the planning phase of the project. As you work thru the prompt, ask me questions if there are any doubts on what option to chose, unless it is clearly decided.

That last line matters more than it looks like it should. A prompt that explicitly tells the model when to stop and ask, rather than when to guess, is doing a real chunk of the risk management for this whole project before a single design decision gets made.

Bootstrap and runtime, drawn as a hard line

The single most important decision to come out of this pass was drawing an explicit boundary between two things that Vesta's curl-based installer conflates completely: one-time host bootstrap (install packages, lay down a hardened baseline) and continuous runtime provisioning (render a tenant's config on every mutation, forever, for the life of the node).

The rule that fell out of that split: Laravel, the queue workers, the controllers, and the node agent itself never invoke installation logic at request time. Ever. Not "shouldn't", not "there's a permission check", the mechanism doesn't exist for the control plane to reach it at all. That single line is what actually closes off the entire category of bug this series keeps circling back to. It's one thing to say "no exec from controllers." It's another to make sure the thing controllers might have wanted to exec isn't even reachable from where they live.

A directory with an actual contract

The installation logic itself lives in a new .install directory at the repo root, and it comes with a written contract every service installer has to satisfy, checked, not just aspired to. The short version: no interactive prompts, no network fetch of unpinned code, no curl | bash under any circumstances, explicit version pinning and checksum verification on anything downloaded, deterministic exit codes, structured JSON output the control plane can actually record, a required dry-run mode, and a preflight check that verifies OS release, architecture, free disk, ports, and conflicting packages before a single byte gets written to disk.

Every service also declares its dependencies and provided capabilities as data, in a manifest, rather than as implicit script ordering that only the person who wrote it understands. Dependency order gets checked by walking that manifest graph, not by trusting that whoever adds a new service remembered to put it in the right place in a shell script.

There was even a small, genuinely funny moment of paranoia in this step: before anything else got built, the prompt specifically asked whether a dot-prefixed directory like .install might get silently dropped by .gitignore, .gitattributes, export-ignore rules, or the Vite/Composer build pipeline, and asked for that to be verified rather than assumed. It had, in fact, needed a .gitattributes fix to guarantee it survives a source archive build. Small thing. Also exactly the kind of thing that quietly bites you eighteen months later if nobody checks it on day one.

Four decisions that don't get to change their minds later

By the third prompt, the project was at the point where a few genuinely irreversible choices had to get made before any of this could actually run against a real service. These weren't left to drift, they got put in front of mikho as explicit options with a recommendation attached, and approved in one reply:

• Database: MariaDB 11.4 LTS over MySQL 8.4 LTS.
• Ubuntu support: both 24.04 and 26.04 LTS, not just one.
• ACME challenge mode: HTTP-01 and DNS-01, both, not one or the other. DNS-01 is the one that actually matters for wildcard certificates, and it's a lot more pleasant when LESta already owns the DNS zone.
• The web capability contract: one shared WebProvisioner contract, with web.nginx.v1 and web.apache.v1 as separate capabilities underneath it, plus a combined profile where nginx owns the public ports 80 and 443 and Apache sits behind it, loopback-only, on 127.0.0.1:8080.

That last one is worth sitting with for a second, because it's a direct answer to something Vesta does clumsily: supporting nginx-as-reverse-proxy-in-front-of-Apache as basically a special case bolted on top of two otherwise-separate code paths. Here it's one contract two implementations satisfy, decided up front, instead of a special case discovered halfway through building the second web server.

Deciding the order things get built in, on purpose

The fourth and final task in this pass wasn't a technical decision at all, it was a sequencing one: given three real strategies (build every installer first and the UI on top, build the UI first against fake adapters and wire up real services later, or take one service all the way to done, installer through UI through tests, before starting the next one), which order actually makes sense for one person building this alone, without a staffed team to parallelize across.

The chosen answer was the third one, a vertical slice per service, and the reasoning is worth spelling out because it's the kind of judgment call that's easy to get wrong under pressure to "just start building." Installing every service first risks designing the wrong contract before any real service has actually been touched by the UI that's supposed to use it, and risks a half-finished installation layer rotting before anything ever exercises it for real. Building the UI first against fakes risks the opposite: a beautiful frontend coupled to adapter contracts nobody's validated against an actual host yet. One service, start to finish, means every layer gets proven against something real before the next one starts, and it matches what the original governing plan already committed to for the web-hosting vertical slice in part 2's phase numbering. Web hosting first, fake WebProvisioner adapter before the real Go capability, then everything else follows the same pattern.

What actually shipped

For all that decision-making, the amount of executable code that shipped in this milestone was, deliberately, close to zero. No installer script, no shell gateway, no service implementation. What got committed as 11a1525 on 2026-08-26 was the ADR documenting all of the above, the .install directory skeleton with its README, manifest schema, and per-service metadata stubs for every planned service, from the base layer through mail, backups, and node health.

The verification list for that commit reads more like something you'd expect from a security audit than a first commit: every installation JSON file parses, web-profile metadata matches its declared listener rules, the installer contract explicitly defines dry-run, preflight, pinned artifacts, checksums, signatures, deterministic exit codes, and rollback expectations, and, checked directly rather than assumed, no executable installer, shell gateway, generic RunCommand, or service implementation exists anywhere in the diff. The working tree was clean after the push. Local and remote main point at the same commit.

A milestone almost entirely made of decisions and a verification list, no features yet.

A fresh session checks the homework

Everything above happened across three prompts run through GitHub Copilot Chat. The next session that touched this project was a fresh Claude Code session with zero memory of any of that prior work, starting cold from just the repo as it sat on disk. That's a genuinely useful setup for catching mistakes, since a model with no investment in its own prior reasoning has no reason to wave a questionable decision through.

It found three places where the existing ADR claimed to be "resolving a conflict" with the governing plan that, on an actual re-read, wasn't a conflict at all. The plan never mentioned PostgreSQL anywhere near the phase the ADR claimed it did. The plan already named both Ubuntu versions explicitly. The plan had already fully specified the nginx-or-Apache-or-both selection model the ADR was claiming credit for resolving. None of these were real disagreements, the ADR's own "resolutions" were just restating decisions the plan had already made, which is a small thing on its own, but exactly the kind of small thing that quietly rots into genuine confusion about what was actually decided versus what was merely repeated.

It also found a real defect, not a documentation nit: the firewall service manifest produced a capability named firewall.baseline.v1, but the base layer had separately self-declared its own same-purpose capability, base.firewall.v1, and nginx, Apache, and BIND had all been wired to depend on the wrong one. In plain terms, the dedicated firewall gate that the ADR's own ordering rationale describes as a hard prerequisite could be silently bypassed, because the dependency graph was pointing at a capability nothing actually produced. Two more services, mail and MariaDB, both of which open inbound ports, were missing that dependency entirely. All of it got fixed in the same pass, along with a stale nginx/README.md that still claimed nginx was the only supported web server, months after the three-profile decision had been made everywhere else.

That same session also settled a decision the earlier prompts had left open: whether the control plane's own MariaDB and the MariaDB LESta offers to tenants should be the same instance with logical separation, or genuinely separate. The answer landed on two separate server instances, the control plane on port 3306, tenant databases on port 3307, each with its own data directory, credentials, and resource limits, sharing the same physical node only because there's currently just the one node to share. Cleaner blast radius, worse resource density, a trade a hosting control panel should probably always take.

None of this is a dramatic story. It's a fresh pass finding three overclaimed "fixes" and one real bug in documentation that had already been reviewed once. Worth noting anyway, because it's the first concrete example in this series of a pattern that shows up again, and more sharply, in part 4: work that looked finished still had a bug in it, and the thing that caught it wasn't more careful writing, it was a second, independent pass actually checking the work against the source rather than trusting the summary of it.

What's coming

Part 3 covers the moment this project stopped designing in the abstract and actually went back to read the condemned Vesta codebase on purpose, resource by resource, to figure out exactly what a working panel needs to preserve and what it should quietly correct. That pass is where a background research agent came back with real product-policy questions nobody had thought to ask yet, the kind of thing that's a lot more interesting than "we found more bugs." Part 4 is where all of this planning finally turns into a database schema, and where the first real code shipped with four small bugs that a plan, however careful, was never going to catch on its own.

For a project that hadn't written a single feature yet at this point, it already had a stricter installer contract than most panels currently in production. Vesta included.

mikho

Part 2 ended with LESta having an entire installation architecture and not one tenant-facing feature. That changed here, but not by opening a code editor. It changed by going back to the codebase that started this whole series, the one with eval in its read path and zero write locks, and actually reading it closely, on purpose, resource by resource, to work out exactly what a working hosting panel needs to do and which parts of how Vesta does it should never make it into LESta at all.

Verify first, design second

Before any of this work started, the project cloned outroll/vesta locally as a read-only reference, pinned at the same commit as part 1's teardown, 5f4ee2ef, and explicitly not added to the LESta repo or pushed anywhere. That distinction matters more than it sounds like it should. The plan requires every vulnerability claim to be confirmed against a pinned commit before it's allowed to inform a real decision, not accepted because a prior document said so.

That discipline paid off almost immediately. The original teardown from part 1 reported error_reporting(NULL) across "all 136 endpoints." A direct, independent re-check against the pinned commit for this pass came back with a slightly different number: 133 files under web/api/v1/ specifically. Not a meaningfully different finding, still essentially the entire API surface with error visibility switched off, but a genuinely different number, confirmed by actually counting rather than by trusting the earlier count. A project that's this fussy about re-verifying its own prior claims about a legacy codebase is, not coincidentally, the same project that later catches its own bugs in part 4 rather than shipping them and finding out from a support ticket.

Reading Vesta with intent

The actual field-by-field extraction, resource fields, lifecycle verbs, quotas, suspension rules, role visibility, was handed to a background research agent rather than done inline, and it came back with more than a restatement of what the code does. It surfaced genuine lifecycle-contract gaps that the original teardown pass hadn't been scoped to look for, because it was reading for security bugs, not for product-policy bugs. A few of the sharper ones:

Editing a package's quotas, say lowering the web-domain limit from 100 to 10, runs a propagation script that force-cascades the new limit to every current subscriber, with a force flag that specifically skips the check that would otherwise reject the change if a subscriber is already over the new limit. So a subscriber sitting at 40 domains, under a package that just got edited down to a limit of 10, ends up 30 domains over quota, silently, with no rejection, no warning, no auto-suspension, nothing at all until they try to add domain number forty-one and hit a limit check for what feels like the first time.

v-unsuspend-user unconditionally reactivates every child resource belonging to an account, regardless of whether any of them were individually suspended before the account-level suspend happened. Suspend one domain by hand for a terms-of-service issue, then suspend the whole account for a billing problem, then resolve the billing problem and unsuspend the account, and that domain comes back online too, with its prior suspended state silently discarded.

And a genuinely unpleasant one: suspending a mail account doesn't flip a flag. It overwrites the account's real password hash in the Exim credentials file with the literal string SUSPENDED. State gets encoded directly into a credential field. Which is not merely inelegant, it's the kind of design where "restore the mailbox" and "restore the mailbox's actual password" become two different, easy-to-forget operations.

Four decisions that had no obviously correct answer

Most of what came out of this pass was an easy call: a live foreign-key reference for packages instead of the copy-on-assign snapshot Vesta uses, explicit suspended flags instead of overloading a credential field, transactional multi-step mutations instead of Vesta's unguarded fan-out. A real relational database with real transactions removes the technical reason Vesta did any of that in the first place, so there wasn't really a decision to make.

Four things genuinely were real product-policy questions though, the kind with no single correct answer, and they got put in front of mikho explicitly rather than resolved by whichever option Claude happened to reach for:

1. Package quota edits versus existing subscribers. Reject the edit outright, chosen over silently allowing it or allowing it with a flag. An admin cannot save a package edit that would put any current subscriber over the new limit, full stop, they have to either raise the limit or fix the affected accounts first.
2. Suspension-state provenance. Preserve prior individual state, the opposite of Vesta's always-reactivate-everything. Unsuspending an account only reactivates resources that were suspended solely because of that cascade, never one an admin suspended on its own beforehand.
3. Delete workflow. Automatic unsuspend-then-delete as one confirmed action, matching Vesta's actual UX rather than forcing a separate explicit unsuspend step first.
4. Admin cross-tenant visibility. A genuinely new read-only support view, distinct from and logged separately from full audited impersonation. Vesta has no concept of this at all, an admin who needs to look at a tenant's resources has to actually impersonate them via "login as" first, there's no lighter-weight way to just look.

What's actually notable here isn't any single answer, it's that the model doing the planning knew the difference between a bug to fix and a policy question to escalate, and treated them differently. An engineering correction like "packages should be a live reference, not a snapshot" got adopted without ceremony, because a real transactional database makes the old approach pointless. A genuine judgment call like "what should happen to a subscriber who's already over a newly lowered quota" got surfaced as a numbered decision with the reasoning laid out, and left for mikho to actually decide. That's a distinction a lot of AI-assisted work skips right past, quietly picking whichever answer sounds most reasonable and moving on. Here it didn't, and the read-only support view in particular, decision four, only exists in LESta today because that question got asked instead of assumed.

One fan-out, eight scripts, zero rollback

The single clearest illustration of why any of this matters shows up in how Vesta handles "add a web domain." It isn't one operation. It's up to eight separate script invocations chained together with no transaction and no rollback: the web domain itself, an optional DNS zone, DNS records for every alias, an optional mail domain, SSL or Let's Encrypt scheduling, a stats config, and an FTP account per entry, each one firing off its own credentials email as it goes.

If step four fails, the domain now exists with no DNS zone behind it, and nothing cleans that up automatically. Enabling Let's Encrypt on top of all that silently creates a system cron job for certificate renewal that doesn't count against the account's own cron quota and doesn't show up in the account's own cron list either, a resource that exists, consumes a cron slot on the box, and is invisible to the one person who'd need to know about it.

That's not an abstract argument for "we should use database transactions", it's the concrete, specific failure mode that makes the case for LESta's outbox-based provisioning model on its own. A partial multi-step mutation with no rollback and no visibility is exactly the shape of bug that a transactional record, one row progressing cleanly through a defined lifecycle, is built to make structurally impossible rather than merely less likely.

Five boundaries, each one earned

All of this fed into a formal threat model covering the five trust boundaries the original plan named back in part 1: browser to Laravel, Laravel to the node agent, agent to the operating system, tenant to tenant, and backup and secret storage. What makes it worth reading rather than boilerplate is that every boundary is grounded in a specific, cited legacy failure rather than a generic "here's what could go wrong in theory."

Tenant-to-tenant isolation in Vesta, for instance, isn't an enforced authorization model at all, it's a filesystem-path convention, data/users/<user>/*.conf, backed by nothing except the assumption that every script correctly resolves whose data it's touching. Combined with the error suppression from part 1, an authorization bug in that path would be unusually hard to even notice in production, since a failure just renders as an empty screen instead of a visible error. LESta's answer is that every query service scopes by ownership at the query level, using indexed fields, and nothing ever trusts a client-supplied account or tenant identifier as an authorization signal on its own.

The document closes with a section that's easy to skip past but is actually the whole point: a threat model isn't considered satisfied by being written and merged. Per the plan's own acceptance criteria, a capability doesn't get to claim it meets this threat model until the tests proving it actually exist and pass. Writing down the rule is the easy part. Part 4 is where that rule gets tested for real, against actual code, for the first time.

The protocol that closes the lost-update bug

The same pass also produced a first draft of the actual wire contract between Laravel and the Go node agent, which is worth pausing on, because it's the piece that turns "no exec from controllers" from a policy into something structurally impossible to violate. There is no generic operation type in this protocol at all. A message names one specific, versioned capability, web.nginx.v1, dns.bind9.v1, and so on, and the agent rejects anything naming a capability it never registered, before the payload even gets parsed. There's no field anywhere shaped like "command plus arguments" for a bug to hide inside.

Every operation carries a resource ID, a desired-state version, an idempotency key, a deadline, and a digest of the request itself, and every result the agent sends back echoes those same identifiers plus an explicit status: applied, already-applied, rejected, failed, or degraded. already_applied is the interesting one. If the same (resource_id, idempotency_key) pair shows up twice, because a response got lost on the wire and the control plane retried, the agent doesn't redo the mutation. It returns the stored result from the first attempt. If the retried request's content digest doesn't match what was stored the first time, the agent refuses outright rather than guessing which version is the real one.

And operations addressing the same resource are strictly serialized, one at a time, regardless of how the control plane happened to dispatch them. That's the direct, specific fix for the exact bug walked through in part 1: two concurrent writers to the same counter, both reading the old value, both writing the same "new" one, one update silently vanishing. A protocol that refuses to process two operations against one resource at the same time doesn't need every future engineer to remember to add a lock. The lock is the protocol.

What's coming

Part 4 is where four phases of planning finally turn into a running Laravel application, a real relational schema, real policies, a real dispatch mechanism for provisioning operations. It's also where the story stops being about a legacy codebase's mistakes and starts being about this one's, including a session that correctly refused to write any code at all because a safety rail was still active, and four genuine bugs the first real implementation shipped with. None of them were caught by planning. All four were caught by something that actually ran.

mikho

Three parts into this series and LESta still didn't have a database schema. It had a teardown of everything wrong with VestaCP, an installation architecture, a threat model, and a capability matrix full of decisions about how suspension state and package quotas ought to behave. All of it correct, none of it code. This part is where that changes, and where the series stops being a story about a legacy codebase's mistakes and starts being honest about this one's.

The session that refused to write anything

The plan approval for this phase came back about as clean as a green light gets: "Yes, and auto-accept," meaning proceed without pausing for confirmation on every individual step. Implementation should have started immediately.

It didn't. Claude Code has a plan mode, a state where the tool itself will not let the model touch a single file, regardless of what the surrounding conversation says, until someone explicitly exits it. That's a harness-level restriction, not a preference the model can reason its way around, the same category of guardrail as a filesystem permission rather than a suggestion. The session was still in that state when the approval above came in, and it stayed in that state. Told to implement, the delegated agent correctly refused to touch the repository at all, restriction still active, approval or no approval. What it produced instead was a fully resolved, execution-ready version of the plan checked against the real codebase, with exact code for every file it would eventually write, and ten explicitly flagged places where the original plan had been silent or shorthand. Small things: which models implement a marker interface the plan only described in prose, how an "impersonation stop" action finds the session it needs to restore, a named exception class for a decision the plan stated but never gave a concrete error type to. None of them changed what the plan intended. All of them were gaps a plan, however carefully written, leaves for the first person who actually has to type out every line.

All ten got reviewed and agreed with. Plan mode got exited a second time, properly. Only then did a second implementation pass actually write the code: 53 new files, 4 changed files, a real relational schema for accounts, memberships, roles, permissions, packages, nodes, node capabilities, audit events, idempotency receipts, and provisioning operations, plus the policies, the outbox and dispatch mechanism, and a full Pest test suite.

None of that "self-reported success" got taken at face value either. migrate:fresh got re-run independently: 12 migrations, clean. Pint: clean. PHPStan at Larastan level 7: zero errors. The full suite: 63 tests, 197 assertions, all passing, run again rather than just read about. Two files got opened and read directly rather than trusted on description alone, the authorization service provider and the account policy, specifically to confirm the one thing that mattered most: a provider admin with no membership on an account is provably denied by the policy, on its own, with no gate bypass leaking onto it. That's decision four from part 3, the read-only support view, actually holding.

Four bugs, none of them caught by planning

Here's where the series earns its subtitle. Four separate defects surfaced only once the code actually ran, not during any of the design or review passes covered in the last three parts. Each one got caught by an automated gate, a database constraint, an engine-specific limitation, a static analyzer, a flaky test failing, rather than by a person spotting it while reading. That's not a coincidence. It's the entire argument for running migrate, Pint, PHPStan, and the full test suite as a required gate instead of trusting a plan's correctness on paper, made concrete instead of abstract. And it's worth naming plainly what none of these four are: not one of them is an eval-on-stored-data bug, a missing lock, or an unconstrained file write. Three parts of threat modeling and non-negotiable prohibitions bought exactly what they were supposed to buy, the first real code shipped with ordinary, boring defects instead of the category of bug this whole series started with.

A missing NOT NULL value. idempotency_receipts.correlation_id is a required column, correctly, every row should be traceable back to whatever caused it. The plan's own code for creating a receipt never actually set it. That would have thrown a database exception on the very first real call. Fixed by generating a UUID at creation time, the same pattern already used everywhere else in the schema, once the gap was actually visible.

A query that works on MariaDB and silently doesn't on SQLite. The package-quota-violation check from part 3's decision one, reject an edit that would put a subscriber over the new limit, got written as a completely ordinary Eloquent pattern: withCount('memberships')->having('memberships_count', '>', $limitValue). SQLite rejects a HAVING clause on a query with no GROUP BY. MariaDB, the actual production target, tolerates it fine. Local development runs on SQLite, production runs MariaDB, and this is exactly the kind of latent database-portability bug the project had already been careful about elsewhere, the .install manifests from part 2 specifically avoid database-specific column features for this same reason. It surfaced here only because a test actually exercised the code path, not because anyone reviewed for it in advance. Fixed with a portable whereHas equivalent, verified to produce identical results.

A type mismatch from a configuration choice made in a completely different part of the app, long before this session. A result object was typed as Illuminate\Support\Carbon. This specific Laravel app configures Date::use(CarbonImmutable::class) globally, so every call to now() anywhere in the codebase actually returns a CarbonImmutable, not a Carbon. A type hint written from general Laravel knowledge, without checking this app's own configuration first, could never have accepted what the rest of the app was actually going to hand it. Fixed by retyping to the shared CarbonInterface both classes implement, so it stops mattering which one shows up.

A flaky test that wasn't actually a bug in the code under test. Two factory calls in a cascade-suspension test both drew a capability value from the same three-item random pool, and occasionally collided against the model's own unique constraint. It failed on the very first run, intermittent rather than a real defect, but exactly the kind of flakiness that quietly erodes trust in a test suite if it's left alone. Fixed by pinning explicit, distinct values in that one test.

None of these four changed what the plan intended. They're the ordinary gap between a design, however carefully reasoned through four roles and a threat model, and the first time it actually runs against a real database and a real static analyzer.

The commit that wasn't quite what it said it was

Everything above got committed and pushed as three separate commits rather than one, so the Phase 1 commit's message would stay accurate to only Phase 1. Right after, a fifth issue turned up that's more instructive than the first four, because it isn't a code defect at all.

bootstrap/providers.php had been correctly modified in the working tree, registering the two new service providers Phase 1 needed. It never made it into the Phase 1 commit. A plain git add mistake, nothing more exotic than that. Every verification command in the section above had genuinely passed, because the working tree, the actual files on disk, still had the fix. Nothing caught the gap until git status got checked again after the commit, against HEAD, and showed that one file still modified relative to what had supposedly just been committed.

Fixed with a follow-up commit. Then, deliberately, the entire verification suite got run a second time, against the now-clean working tree matching exactly what had actually been pushed, not against a working tree that happened to already contain the fix. That distinction is worth sitting with on its own: passing tests locally proves the working tree is correct. It does not, by itself, prove the commit is correct. Those are only the same claim once the working tree and HEAD actually match, and that's a thing worth checking explicitly rather than assuming.

The plot twist that already had an ending

A few days later, mikho pasted in a GitHub Actions log showing seven failed tests from CI, two distinct error signatures, an authorization exception on the node and package policy tests, and a container binding failure on the provisioning-dispatch tests.

The cause was exactly the bug from the section above: with the two service providers unregistered, the scoped admin bypass never runs, so a provider admin gets denied like anyone else, and nothing is bound to the Provisioner interface, so the container can't build one. This wasn't a new bug. It was the same bug, already found, already fixed, being reported for the second time by a system that was simply slower to notice than the person who'd already caught it.

Checked directly against gh run list and gh run view rather than assumed from the timestamps alone: the CI run for the original Phase 1 commit shows failure, with the exact same seven test names and the exact same two error signatures as the pasted log. The very next CI run, for the follow-up commit registering the providers, shows success, one job, zero failures. main had already been green for a while by the time the failure got reported. The pasted log wasn't a new problem surfacing. It was CI finally catching up to, and independently confirming, a bug that had already been found and fixed in-session, roughly two minutes after it was introduced.

CI, moving at its own pace, arriving to report a bug that had already been closed.

That's the cleanest example this project has produced so far of the thing this whole series keeps circling: a working tree that passed every local check while its commit quietly didn't match it, caught by discipline, checking the actual state of things rather than trusting a summary, rather than by luck.

What's actually next

Phase 1 is done: relational foundation, deny-by-default policies, the outbox and idempotency mechanism, all verified, all pushed, CI green. Per the plan from part 1, next up is Phase 2, the actual web hosting vertical slice, a fake WebProvisioner first, then the real Go nginx and Apache capabilities behind the shared contract decided back in part 2. That's the point where LESta stops being schema and policy and starts being something you could actually point a domain at.

Four parts in, the thesis of this whole series hasn't changed since the first paragraph of part 1. Vibe coding, done properly, doesn't mean skipping the boring parts. It means the boring parts, migrate:fresh, Pint, PHPStan, the full test suite, rereading git status after the commit instead of trusting the commit message, are exactly what make trusting the fast parts reasonable. This project has, so far, caught every one of its own mistakes before a tenant ever would have. Part 5 picks up whenever Phase 2 has enough of a story to tell.

mikho

Four parts in, LESta could suspend an account, deny an unauthorized admin, and dispatch a provisioning operation to nothing in particular, and it still couldn't do the one thing a hosting panel is actually for: let a customer point a domain at a server. ADR 0002 names the exact bar for this phase plainly: "a customer can manage a domain and see honest provisioning state." This part is where LESta clears it.

It's also the first time this project's own React frontend has had to do any real work. Everything through part 4 lived entirely in the database, the policies, and the outbox mechanism, backend all the way down, provable with php artisan test and nothing else. A customer doesn't experience a policy class or a migration. They experience a page that either lets them add a domain or doesn't, and tells them the truth about what state it's actually in while it does. That's a different kind of correctness to get right, and it's the reason this phase is where the series' first real UI bugs show up too.

Two decisions before any schema got written

Exploring the codebase for this phase turned up two real gaps, not bugs, just things nobody had needed yet. This app's Inertia frontend uses a <Form> render-prop component bound through Wayfinder everywhere, never the more common useForm hook, and there was no reusable confirmation-dialog component anywhere, every destructive action so far had rolled its own. More pressing, Pest's own browser-testing plugin, the one the plan actually requires for this phase (pestphp/pest-plugin-browser), and a table/pagination UI for the domains list, neither existed in the install yet.

Both are dependency decisions, and this project's own rule is no dependency change without explicit approval, so both went to mikho as real options rather than getting quietly assumed. Browser test coverage: add it now, since the plan explicitly requires real coverage for this phase, not deferred to some later cleanup pass. The list UI: hand-built in plain HTML and Tailwind, a simple prev/next paginator off Laravel's own paginator, a debounced search input from the existing plain Input component, no new frontend package of any kind. The shadcn CLI would have been the faster route, standard components, a small Radix package or two pulled in along with them. It lost anyway, on purpose, to keep this phase free of a dependency it didn't strictly need, one more small example of the project choosing the more deliberate option over the one that saves an afternoon.

The <Form>-over-useForm convention mattered for a quieter reason too: it meant the new domain create/edit/delete pages could match the app's existing shape exactly rather than introducing a second pattern for form state. A hand-written Textarea component (plain Tailwind, no shadcn CLI) had to be added along the way too, since the domain alias field needed one and none existed yet, small, but the same "match what's already here" instinct rather than reaching for a new dependency to solve a one-component gap.

The tension the design pass caught on its own

The schema design surfaced a real question it wasn't explicitly asked: should deleting an account cascade to delete its web domains at the database level, or should that relationship refuse to let the database do it silently? It picked the second, restrictOnDelete() on every foreign key from web_domains to accounts, nodes, and ip_allocations, without needing to be told why that mattered.

The reason is a direct callback to part 3. A silent database cascade deleting a tenant's domains the moment their account gets deleted would recreate, at the schema level, the exact "unguarded fan-out" failure mode documented in legacy Vesta's own eight-script, zero-rollback domain creation. With restrictOnDelete() instead, DeleteAccount has to explicitly delete each owned WebDomain through the real, audited DeleteWebDomain action before the account row can go. Forget that step in some future change, and the database throws an error instead of quietly orphaning provisioning state nobody meant to lose.

A third genuine product question came out of the same pass: when a package has no configured quota row at all for web_domains, not an explicit unlimited row, just nothing, should creating one be allowed or blocked? Blocked, was the answer, the stricter reading of deny-by-default. An explicit row with a null limit still means unlimited, unchanged from how Phase 1 already worked, but the complete absence of a row now means zero. Every package that's supposed to support web hosting needs to say so explicitly, even if what it says is "no limit," and every test or factory that creates a web domain has to remember to attach a limit row too, a deliberate constraint on the test suite itself, not an oversight if a test trips over it later.

None of the three questions above got resolved by picking whichever answer sounded more reasonable in the moment. Each one got framed as an actual choice, with the tradeoff stated, before an answer got picked. That's worth naming plainly, because it would have been easy, four phases into a project with an established track record, to start trusting the model's judgment on calls like these instead of still asking. It didn't happen here.

What actually got built

53 new files, 12 changed. Three migrations (ip_allocations, web_domains, web_domain_aliases), two enums, three models, the node/capability targeting rule (prefer nginx over Apache whenever a node has both registered, matching the public-listener rule from part 2), five domain actions, the exact cascade edits to the account-suspend and account-delete flows, a policy with no admin bypass, the full HTTP layer, three frontend pages, and the complete test list the plan called for. No new WebProvisioner PHP interface was needed either, Phase 1's provisioner mechanism was already capability-agnostic, exactly as it was designed to be back when there was nothing yet for it to provision.

None of that got taken on the implementation report's word. migrate:fresh re-run independently: 18 migrations, clean, in order. Pint: clean. PHPStan at its actual configured scope: zero errors. npm run build: clean, the first real frontend build this project has ever needed. The full test suite: 114 tests, 334 assertions, up from Phase 1's 63. Three files got read directly rather than trusted on description: the quota check genuinely distinguishes "no limit row" from "an explicit null-value row," the account-suspend cascade only touches domains that weren't already individually suspended and does it through the real audited action, and the policy has no admin bypass leaking onto it.

Five small deviations from the plan's literal text showed up along the way, all reviewed, none changing intent. The neatest one: the plan called for morphMany()->latestOfMany() to fetch a domain's latest provisioning operation, but this Laravel version's MorphMany genuinely doesn't support latestOfMany(), only HasOne, MorphOne, and HasOneThrough do. morphOne()->latestOfMany() is the framework's actual mechanism for exactly this, documented inline with the reasoning rather than silently swapped in. WebDomain also picked up a getRouteKeyName() override returning uuid, an addition the plan implied by giving the model a uuid column at all but never spelled out explicitly, route binding by internal auto-increment id would have been the actual inconsistency here. A handful of PHPStan-driven type and docblock corrections rounded out the rest, no behavioral change, just the static analyzer catching an overly loose array shape and a validation closure's signature before they could rot into something that mattered later. And CLAUDE.md's own PHP version line corrected itself from 8.5 to 8.4 as a side effect of Laravel Boost's post-update hook firing after the one authorized composer change, not a manual edit, left as-is because it was simply accurate.

The bugs that only showed up once someone actually clicked

Here's where this post earns its place in the series. The two browser test files existed, were wired into the test suite's own configuration, and looked complete. They still couldn't run, because pestphp/pest-plugin-browser needs Playwright, an npm package plus downloaded browser binaries, which hadn't been part of the one dependency change already approved for this task. Installing it was its own small decision, put to mikho rather than assumed, and approved.

Running the two browser tests for real, not just confirming they existed, found two genuine bugs in five minutes that reading the plan, the component code, and the test code in isolation had all missed completely. The domain edit page's suspend button has no confirmation dialog, delete is the only action that gets one, so one click is always enough. The test clicked it twice anyway. The first click suspended the domain and the button re-rendered as "Unsuspend." The second click, landing on that same now-relabeled button, silently undid the first action. The test failed expecting a suspended domain and got an active one.

Fixed that, ran again, hit a second bug immediately. The delete section has a heading that reads "Delete domain" directly above a button that also reads "Delete domain." A text-based click matched the heading, not the button, so the confirmation dialog never opened and nothing actually got deleted. Pest's browser plugin auto-saves a screenshot on failure, and the screenshot made the problem obvious at a glance, no dialog anywhere in frame. The fix was to target the component's existing data-test attributes instead of matching on visible text that wasn't unique, the same convention the codebase already used elsewhere for exactly this reason.

Both fixes together are a clean, small-scale replay of this whole series' argument. Neither bug was visible from reading anything in isolation. Both were visible in about ten seconds of watching the actual interaction fail. Re-ran the full browser suite twice after the fixes, two passes, two greens, no flakiness.

Shipped

Everything got committed as da02ae9, following the same working-tree-must-match-HEAD discipline part 4 learned the hard way: git status clean before the push, then the entire verification sequence, migrate:fresh, Pint, PHPStan, the Feature and Unit suite, and now the Browser suite too, re-run a second time against the exact committed state rather than a working tree that happened to already be right.

Then, rather than assume a push implies a passing build, the actual GitHub Actions run got watched to completion. Green, one job, under a minute. Phase 2 is done, verified, committed, pushed, and confirmed green in CI, the same bar Phase 1 had to clear, held for a second phase in a row rather than relaxed now that the pattern is familiar.

The point where LESta stopped being schema and policy and started being something you could point a domain at.

One honest gap, flagged rather than quietly left: CI's own workflow still only runs composer ci:check, which never touches the browser suite, deliberately excluded from phpunit.xml. The browser tests are real, reviewable, and passing, but only locally, until CI gets a browser-binary setup step of its own. Not a silent hole. A named one, for whenever it's actually worth the CI minutes to close.

What's coming

Part 6 steps back from the build log for one post to look at a pattern across the whole project so far: every single question this project has actually put to mikho, gathered in one place, and what it says about how "vibe coding" and "careful coding" turned out not to be opposites here. After that, the story picks back up wherever the actual Go agent work lands.

mikho

Five parts of this series have been a build log: teardown, plan, code, bugs, ship. This one is different. After Phase 2 shipped, the project did something it hadn't done before, went back over its own entire history and pulled out, in one place, every single question it had actually put to mikho, exactly as asked, with every option that was on the table and the answer that got picked. Sixteen questions, eight separate sessions, one real pattern sitting underneath all of it.

Why this exists

The prompt that produced it was simple: document the process and all questions and answers in the vault. What made it worth doing rather than busywork is what the audit turned up first. The project's own decision log, the same one this series has been drawing from since part 1, had been recording the earliest questions as narrative, the substance was all there, but not as the literal question and the literal options offered. Later entries, Phase 2's dependency and quota questions, already had that exact form. The earlier ones didn't.

Rather than rewrite history to make old entries match the new style, retroactively cleaner but less honest about how the record actually developed, the project wrote a second document instead: Questions and Answers.md, every question from the beginning, chronological, in one consistent shape, sitting alongside the narrative log rather than replacing it. The decision log stays the story. This is the reference underneath it.

The sixteen

They run from the very first ecosystem defaults (database engine, Ubuntu support, ACME mode, whether mail ships enabled by default) through the capability-matrix product calls covered in part 3, to Phase 2's dependency and quota questions covered in part 5. Every one keeps the same shape: the question as actually asked, every option that was genuinely on the table with a short reason for each, which one carried the "Recommended" label, and which one got picked. Reading the delete-workflow question from part 3 in that shape makes something visible that prose narrative tends to smooth over: "automatic unsuspend-then-delete" was recommended specifically because it matches the legacy UX, already effectively one action there, while "require explicit unsuspend first" was on the table as the stricter alternative and got named as one, not silently dropped for not being chosen. The road not taken is still on the page, which is exactly the point of building a document like this instead of just keeping the narrative.

A few of the sixteen are worth pulling out on their own.

One question didn't get answered by picking from the options offered at all. Asked how to source the legacy Vesta reference material for the capability matrix, the choices on the table were: clone the repo locally, use the earlier Claude-generated review artifact, or defer the whole thing. What actually happened was neither, clone locally and use the review artifact together, the checkout for direct verification, the artifact for an initial map of where to look. Worth noting for what it says about the shape of these questions generally: they were real options with tradeoffs, not a menu a model was quietly steering toward a foregone conclusion, and the record shows at least one case where the actual answer wasn't even on the menu.

Most, though, did follow the labeled recommendation. Twelve of sixteen. MariaDB over MySQL. Reject the quota edit outright. Preserve suspension state. Add the read-only support view. Add Playwright when the browser tests couldn't run without it. In each of those, the recommended option and the chosen option were the same thing, which is exactly what a recommendation is for when the reasoning behind it actually holds up.

One of the twelve deserves a second look on its own, because it's easy to misread as boring when it's actually the same instinct as the four exceptions below, just already wearing the recommended label. Asked whether mail should be enabled by default on a freshly bootstrapped node or require an explicit operator opt-in, the recommendation was opt-in, on the reasoning that a node with mail off has no mail attack surface at all until someone deliberately turns it on. That's not a case where the safer option happened to also be labeled "recommended" by coincidence. It's the same underlying preference this project keeps landing on, smaller default surface over fewer setup steps, showing up in a question where, this time, the recommendation had already done the conservative reasoning first.

Where documentation itself was allowed to live

A separate, smaller cluster of three questions is worth its own mention, not because any one of them was dramatic, but because all three landed on the same rule without that rule ever being stated as a rule first. Should the architecture decision record that was already committed and pushed to the repo also get mirrored into the Obsidian vault? Should the two JSON Schema files the node protocol document references move to the vault alongside the narrative that explains them? Should the .install directory's own README, which describes the installer contract itself, move too?

All three: no. Keep it in the repo. The reasoning given each time was the same reasoning, not three separate justifications that happened to agree: an ADR is conventionally versioned with the code it explains, a schema is a normative artifact future code will actually test against, and an installer README is the contract itself, not commentary about the contract. Prose that explains a decision belongs in the vault. Anything a machine will actually parse, validate, or enforce stays exactly where the code that depends on it lives. Nobody wrote that distinction down as a policy before these three questions came up. It only became visible as a pattern once all three answers sat next to each other in the same document.

That's part of the actual value of building Questions and Answers.md at all. A single question and its answer is just a decision. Sixteen of them, read together, are a value system, whether or not anyone set out to write one.

The four that didn't

Four times, the choice went the other way, and it's worth listing all four together because they don't look like four unrelated exceptions once you do:

1. Ubuntu support. The recommended option was 24.04 only, half the integration-test cost for a single developer. The answer was both Ubuntu 24.04 and 26.04.
2. ACME challenge mode. The recommended option was HTTP-01 only, a smaller failure surface with no DNS-capability dependency. The answer was HTTP-01 and DNS-01 both.
3. The domains list UI, from part 5. The recommended option was the shadcn CLI, standard components, faster to ship. The answer was hand-building it in plain HTML and Tailwind, no new dependency at all.
4. The web-domain quota default, also from part 5. The recommended option was unlimited until configured, consistent with how a null limit already means unlimited elsewhere. The answer was blocked until configured, the stricter, more literal reading of deny-by-default.

Four times, chosen deliberately: the route with more steps, not the shorter one.

Every one of these four trades a bit of near-term convenience for either a smaller blast radius or a more explicit, harder-to-misuse default. None of them make the system faster to build. All four make it slower to misconfigure later. That's not a coincidence repeated four times by chance, it's a legible preference showing up consistently enough across genuinely separate decisions, made in different sessions, about different parts of the system, to call it a real pattern rather than a story imposed on the data after the fact.

What this says about "recommended"

It would be easy to read "12 of 16 went with the recommendation" as evidence the recommendations were just being rubber-stamped, and the four exceptions as the only real thinking that happened. That reading doesn't survive contact with the actual list. The twelve that matched weren't twelve identical rubber stamps, they were twelve separate moments where a stated tradeoff got weighed and the labeled option held up under that weighing. Ubuntu, ACME, the list UI, and the quota default aren't twelve-minus-four either, they're the four cases where the same weighing process concluded the recommendation, reasonable as it was, wasn't actually the right call for what this specific project needed. A recommendation that gets overridden a quarter of the time by someone actually reading it isn't decoration. It's doing its job, giving a real default that a real decision can disagree with when it should.

The more accurate way to state the pattern isn't "the recommendation usually won" or "the stricter option usually won." It's that one specific outcome, the smaller default surface, the more explicit configuration requirement, the option with more steps up front and fewer ways to misconfigure later, kept winning regardless of which column it happened to sit in. Twelve times it was already the recommended column. Four times it wasn't, and the recommendation lost anyway. Mail's opt-in default and the web-domain quota's blocked-until-configured default are the same preference wearing two different labels, "Recommended" on one, "not the recommended option" on the other. The label never was the thing actually deciding it.

Why a build log needed a post like this

The first five parts of this series have all made some version of the same argument through different evidence: plan carefully, verify independently, catch your own mistakes before they ship. This one is different in kind, not degree. It's not a claim that the process worked. It's the receipts, in one place, checkable against the actual options offered at the time rather than reconstructed after the fact from memory of how the reasoning probably went. Sixteen questions, four overridden recommendations, one visible preference for the stricter, more explicit, more work-up-front option every single time a real choice got made. That's not a story this series is telling about itself. It's what the questions-and-answers list actually shows, whether or not anyone had written a build log to go with it.

It also closes a loop this series opened back in part 4: an implementation report isn't trusted on its own word, migrate:fresh and Pint and PHPStan and the actual test suite get re-run before anything counts as done. The same standard applies one level up. A build log's own claim that "the project consistently chose the stricter option" isn't trustworthy just because a build log said so either. Questions and Answers.md is what makes that specific claim checkable rather than just asserted, the exact same distinction, applied to a decision-making pattern instead of a database migration.

Questions and Answers

• Created: 2026-08-27T14:50Z (16:50 CEST)
  • Purpose: every question put to the user across this project's Claude Code sessions, with the exact options offered and the answer given, in one place. Decision Log.md is the narrative process record; this is the clean reference underneath it, for anyone later writing a full account of how this project's decisions actually got made.
  • Where the exact time isn't recorded: the practice of taking a real timestamp via the shell's date command started partway through the project (see Decision Log.md's note on this). Entries from before that point are dated but not timed; nothing here is a guessed time.
  ## 2026-08-26, during the service-installation and build-sequencing review Q1. Which database engine should the first implementation target as primary?
  • MariaDB 11.4 LTS only (Recommended). Matches Ubuntu's default repositories, one package/version to pin and test.
  • MySQL 8.4 LTS only. Matches broader panel-ecosystem familiarity, diverges from Ubuntu's default database.
  • Both, MariaDB primary + MySQL verified. Doubles the install/test matrix with no benefit yet.
  Answer: MariaDB 11.4 LTS only. Q2. Which Ubuntu LTS release(s) should the first implementation support?
  • 24.04 LTS only (Recommended). Halves integration-test cost for a single-developer project.
  • Both 24.04 and 26.04 simultaneously. Matches what the manifests already declared, doubles the test matrix from day one.
  Answer: Both 24.04 and 26.04 simultaneously. Q3. Which ACME challenge mode(s) should the web/ACME capability support first?
  • HTTP-01 only (Recommended). No dependency on the DNS capability, smaller failure surface.
  • HTTP-01 and DNS-01 both. Needed for wildcard certificates, but requires the DNS capability first.
  Answer: HTTP-01 and DNS-01 both. Q4. Should mail be enabled by default on a bootstrapped node, or opt-in?
  • Explicit operator opt-in (Recommended). A freshly bootstrapped node has no mail attack surface until deliberately turned on.
  • Enabled by default once all gates pass. Fewer steps for the operator, more default attack surface.
  Answer: Explicit operator opt-in (Recommended). ## 2026-08-26, before scoping the "move documentation to the vault" request Q5. The ADR is already committed and pushed to GitHub. Should it also move to the Obsidian vault, or stay in the repo?
  • Keep the ADR in the repo (Recommended). ADRs are conventionally versioned alongside the code they explain.
  • Move the ADR to the vault too. Full consistency: every explanatory document lives in Obsidian.
  Answer: Keep the ADR in the repo (Recommended). Q6. The node protocol doc references two JSON Schema files. Move those with the narrative, or keep them in the repo as testable contract files?
  • Keep schemas in the repo (Recommended). Normative, machine-validated artifacts future code will test against, not prose.
  • Move everything to the vault. Keeps all node-protocol material in one place.
  Answer: Keep schemas in the repo (Recommended). Q7. The .install directory's own README and per-service READMEs describe the installer contract itself. Move those too, or leave them in place?
  • Leave .install/ untouched (Recommended). Not standalone documentation, the installer contract itself.
  • Move these too.
  Answer: Leave .install/ untouched (Recommended). ## 2026-08-26, before extracting the Vesta capability matrix Q8. How should I source the legacy Vesta reference material for the capability matrix?
  • Clone outroll/vesta locally, read-only (Recommended). Pin to a commit, read purely for contract extraction.
  • Use your earlier Claude review artifact instead.
  • Defer the capability matrix, start elsewhere.
  Answer (a custom combination, not one of the listed options as-is): clone locally, read-only, never push to this repo, and use the Claude review artifact too, since it had since been added to the vault as "Engineering teardown · Vesta Control Panel". Both sources were used together: the checkout for direct verification, the teardown for the initial map of where to look. ## 2026-08-26, four capability-matrix product decisions Q9. When an admin edits a package and existing subscribers would exceed the new, lower quota, what should happen?
  • Reject the edit outright (Recommended). No account is ever left in a silently inconsistent quota state.
  • Allow it, flag affected accounts for admin review.
  • Allow it silently (matches the legacy gap this review had just found, not recommended).
  Answer: Reject the edit outright (Recommended). Q10. When an account that was suspended is unsuspended, should resources individually suspended beforehand stay suspended?
  • Preserve prior individual state (Recommended). Anything suspended before the account-level action stays suspended.
  • Always reactivate everything (matches legacy behavior).
  Answer: Preserve prior individual state (Recommended). Q11. Should deleting a suspended resource require an explicit separate unsuspend step first, or happen automatically as one action?
  • Automatic unsuspend-then-delete (Recommended). Matches the legacy UX, already effectively one action there.
  • Require explicit unsuspend first. Stricter, an extra safety checkpoint.
  Answer: Automatic unsuspend-then-delete (Recommended). Q12. Should a provider admin be able to view a tenant's resources without impersonating them?
  • Add a read-only support view (Recommended). Distinct from and logged separately from full audited impersonation.
  • Impersonation only, matching legacy exactly.
  Answer: Add a read-only support view (Recommended). ## 2026-08-27, before designing the Phase 2 schema Q13. Browser test coverage requires pestphp/pest-plugin-browser, not actually installed. Add it now for Phase 2?
  • Add it now (Recommended). Satisfies the plan's own browser-coverage requirement for this phase.
  • Defer browser tests, cover this phase with Feature-level tests only.
  Answer: Add it now (Recommended). Q14. The domains list page needs a table and pagination UI, neither exists in this shadcn install. Add via shadcn CLI, or hand-build without a table primitive?
  • Add via shadcn CLI (Recommended). Standard components, may pull in a small Radix package or two.
  • Build without a table primitive. No new npm packages at all.
  Answer: Build without a table primitive (the non-recommended option, chosen deliberately to keep this phase free of any new frontend dependency). ## 2026-08-27, the Phase 2 quota-default decision Q15. When a package has no configured quota row at all for web_domains, should creation be unlimited or blocked until a limit is set?
  • Unlimited until configured (Recommended). Absence of a row means no restriction, matching how an explicit null limit already means unlimited elsewhere in the schema.
  • Blocked until configured. Stricter, literal deny-by-default; every new package needs explicit setup before it's usable.
  Answer: Blocked until configured (the non-recommended, stricter option). ## 2026-08-27, mid-implementation of Phase 2 Q16. The two browser test files exist and are wired up, but Playwright isn't installed, so they can't actually run. Install it now?
  • Install it now (Recommended). Completes what the plan's own browser-coverage decision asked for.
  • Defer, commit as-is. phpunit.xml already excludes them from the default run and CI either way.
  Answer: Install it now (Recommended).

What's coming

Part 5 already named where the story goes next: wherever the real Go agent work lands, the point where LESta's provisioning stops targeting a fake adapter and starts talking to an actual node. That's a bigger jump than anything covered so far, real infrastructure instead of a database record describing intent, and it's the part of this project that hasn't happened yet. This series picks it up once it does.

mikho

LESta, part 7: the first Go code

Tags: laravel, react, hosting-panel, vibe-coding, security

Six parts of this series have all happened on one side of a boundary this whole project exists to enforce: Laravel constructs desired state, and something else, a separately deployed Go agent nobody had written yet, is the only thing allowed to touch a real host. This part is where that something else stops being a paragraph in a threat model and starts being actual code that renders a real nginx config and reloads a real nginx process.

It's also the point where this project stops being a single-language codebase. Every phase up to now has lived entirely inside one Laravel application, one language, one toolchain, one set of conventions to stay consistent with. This phase introduces a second language, a second module boundary, and a second set of tests that have to agree with the first set about what "correct" even means for the same operation. That's a genuinely different kind of risk than anything covered so far, not a bug in one function, but two independently written implementations of the same contract quietly drifting apart from each other over time if nothing forces them to keep agreeing.

Two decisions before a line of Go got written

Before design started, the machine itself got checked rather than assumed: no Go, no Multipass, no Vagrant, no Docker, no Lima installed anywhere on it. Two foundational questions got settled first, because guessing wrong here would mean redoing real infrastructure choices, not just refactoring code later.

Where should the agent's own code actually live? A subdirectory of the existing repo, agent/, not a separate one. One git history, the protocol schemas and the Go code that implements them versioned together, since Go modules already support independently tagged versions even from inside a subdirectory, so this doesn't fight the plan's own requirement that the agent ship as a separately deployable component.

How should "disposable Ubuntu" testing actually work, the kind ADR 0002 names as part of this phase's exit bar? Multipass locally, Canonical's own tool for exactly this, plus GitHub Actions' own Ubuntu runners in CI, no cloud account, no API credentials, nothing to provision ahead of time. Both Go and nginx got installed fresh on this machine specifically so the real-capability tests could actually run and be checked directly here, not just trusted on a future VM's word.

Deliberately narrow, on purpose

This phase scoped itself down hard, matching a pattern this whole project keeps returning to: prove one narrow, real slice completely rather than half-build several. In scope: the web.nginx.v1 capability only, all six protocol operations, tested by feeding envelope structs directly into Go tests against a shared contract suite. Explicitly out of scope, named rather than silently skipped: the actual network transport between Laravel and a running agent, no HTTP server, no mTLS, nothing over the wire yet, this phase is capability logic only. Apache and the combined web profile. And SSL rendering, ssl.mode gets parsed and stored, but every vhost this phase can produce is HTTP-only, since acting on a certificate that doesn't exist yet would just make the agent's own config validation fail against itself.

The design pass also caught two places where the ADR's general prose didn't quite match what the actual manifest files declared, and reconciled them explicitly rather than picking a side quietly. The ADR describes generations living under a shared path across all services; nginx's own manifest never declares that path at all, so generation history nests under nginx's own declared root instead, the manifest wins as the concrete, binding boundary. And generations turned out to need scoping per resource, one WebDomain at a time, not per node, since a single node-wide counter would tangle unrelated domains' rollback histories together.

Testing a config change without touching the live one

The actual mechanics of applying a change are the most satisfying part of this phase to read, because the whole design exists to answer one specific, concrete question: how do you validate a candidate nginx configuration without ever letting a broken one anywhere near the config that's currently serving real traffic?

The candidate gets rendered into a dotfile sitting in the live directory itself, invisible to nginx's own include glob, so it's on the same filesystem as the eventual real file without being anywhere near active. A synthetic full test config gets built by copying the real, read-only nginx.conf and swapping its include line for one pointing at a scratch directory populated with symlinks to every other currently-live fragment, plus this one candidate. Then nginx -t against that synthetic config, nginx's own validator, not a hand-rolled parser trying to guess what nginx would accept. On failure, the staging file gets deleted and the generation number isn't spent, a rejected attempt can retry cleanly. On success, an atomic rename swaps the candidate over the live file, nginx reloads as its own separate step, and a real HTTP request against the vhost checks for a 200 and a marker embedded in the rendered page, proof this specific vhost actually answered, not just that nginx didn't crash.

Failure semantics follow the plan's own stated table precisely. A create that fails has no prior generation to fall back to, so it's failed, never degraded. Everything else that fails re-stages the previous generation through that exact same pipeline, re-validating it rather than trusting stored state blindly, reloads, and re-checks health: degraded if that rollback comes up healthy again, failed only if even the rollback's own health check comes up bad.

A fake that speaks the same dialect as the real thing

Before any of the nginx-specific work, this phase needed a FakeCapability, the Go equivalent of the FakeProvisioner Phase 1 built back when there was nothing yet to provision for real. It would have been easy to let the two diverge, different languages, different authors in effect, no reason they'd naturally agree on anything. Instead the Go fake deliberately reuses the exact same formulas the PHP one already used, the identical digest scheme, the identical generation-id pattern, so the same fake behavior exists recognizably on both sides of the boundary this project keeps drawing between Laravel and the agent. It's a small thing, stateless, no disk I/O, runs identically on any OS, but it's the kind of small thing that keeps a two-language codebase from quietly growing two different ideas of what "fake" means.

The generation store underneath the real capability follows the same instinct toward consistency, a separate, service-agnostic package mirroring how Phase 1's RecordsProvisioningOperation stays provisionable-agnostic on the Laravel side. Activating a generation is an atomic rename plus a current/previous symlink swap, a persisted manifest, and pruning anything older than the last five generations per resource. Rolling back isn't a special code path bolted on afterward, it's the previous generation's stored content run back through that exact same stage-validate-activate pipeline. And the digest that proves a live directory matches what was actually applied hashes each file's path alongside its content, never raw concatenated bytes, because two byte-identical config fragments belonging to two different domains have to digest differently, or drift detection would miss exactly the kind of mistake it exists to catch.

The contract suite that exercises all of this runs the identical test code against both the fake and the real capability, duplicate idempotency keys return already_applied without re-rendering anything, a full create-suspend-unsuspend cycle ends with the original config's digest restored exactly, an invalid domain gets rejected cleanly rather than panicking. The real half of that suite gives each test its own fully disposable nginx process, not the machine's system-wide one, nginx's own flags relocate its entire working directory into a temp folder on an ephemeral loopback port, no sudo, no systemd, parallelizable, and identical whether it's running on this Mac, a Multipass VM, or a GitHub-hosted runner.

What actually got proven, not just claimed

The implementation report said the contract suite passed. That got checked directly rather than taken on faith, and this phase had a real advantage over the last few: since Go and nginx were now genuinely installed on this machine, the real-capability tests ran here too, not faked, not skipped. A real HTTP request against a real, disposable nginx process on an ephemeral port returned the exact body of a vhost that had just been created. Suspending it swapped in the real maintenance page. Unsuspending restored the original body exactly. Deleting removed the actual file from disk while a surviving vhost kept serving normally. The three failure-semantics paths got forced through a controlled reload-failure wrapper, not a mock, and each one reported exactly the status the plan's table said it should.

Independent verification caught two things the implementation report itself hadn't flagged. The new CI job's dependency-caching step pointed at a go.sum file that doesn't exist, since this module has zero external dependencies, left as written its behavior against a missing file was untested and unclear, fixed by an explicit cache: false instead of a caching key aimed at nothing. And the implementing agent had created its own local Claude Code settings file, something that had never existed in this repo before, confirmed by checking git history directly. Deleting it was itself blocked by Claude Code's own permission classifier, a sensible guard against an agent editing its own settings, worked around simply by never staging it into any commit rather than trying to force past the block.

Committed as c7edd5b, verified a second time against the exact committed state, pushed, and the actual GitHub Actions run watched to completion, both the existing job and the new agent job on its first run ever, including the step that installs nginx directly on the runner and runs the full real-capability suite there. Green on both, first try.

A short aside on how this project actually keeps going

Right after this phase shipped, a practical question came up: could a scheduled cloud routine pick this project back up automatically once a usage quota reset, rather than waiting for the next real session? Checking the actual constraint first turned up something worth recording on its own. A cloud-scheduled routine can only ever see whatever's in the git repository it clones fresh. This vault, the actual richest record of every decision and every piece of reasoning behind it, the decision log, the phase plans, Questions and Answers.md, lives on local storage by a deliberate choice made back in part 2, not in the repository. A cloud routine would see the code, the ADR, and the schemas, and nothing else this whole series has been drawing from.

The answer was to just resume the local session later instead, and the reason is worth naming plainly: this project's real continuity, across sessions and even across different tools on the same machine, comes entirely from the vault being local and readable, not from anything Claude Code or a cloud routine does automatically. It's a real boundary of how this whole setup works, not a workaround for a limitation nobody had thought about.

What's coming

Part 8 covers what came directly after: the first executable installer this project has ever shipped, and a security bug caught during review before it ever reached main, the sharpest single example yet of why every phase in this series gets independently verified rather than taken on the implementation's word.

mikho

LESta, part 8: the bug that almost shipped

Tags: laravel, react, hosting-panel, vibe-coding, security

Every part of this series so far has argued the same thing from different angles: verify independently, don't trust an implementation report on its word, catch mistakes before they ship. This part is where that argument stops being abstract. A real security issue got written, then caught, then fixed, entirely inside one session, before it ever reached main. This is that story, plus the installer it was found inside of.

The gate finally satisfied

The original task that scaffolded this whole project's installation architecture, back in part 2, drew a specific line: the first executable installer gets written only after the web capability passes both its fake and real-agent acceptance gates. Part 7 satisfied that. Three real next steps existed once it did: the real nginx installer itself, Apache's Go capability plus the combined web profile, or moving on to DNS entirely. The nginx installer won, the recommended option this time, since ADR 0002's own build sequence names installer and release hardening as the direct next step after a proven web capability, not a detour.

.install/ had existed since part 2 as a fully designed but entirely non-executable contract, manifests, a schema, a written contract document, zero scripts. This phase built the first one, install.sh, and everything downstream of it that a bare Ubuntu node actually needs to reach the state the Go agent's own code already assumes exists.

The shape of the script, and where it drew a hard line

One cohesive script, deliberately, not a generic multi-service runner. It handles base setup, the firewall baseline, and node-health as named internal phases before installing nginx itself, since nginx's own manifest hard-depends on all three and none of them had any install logic yet either. A manifest-driven dependency-graph runner would have exactly one real caller today, and this project's own architecture decision record already drew the same line elsewhere: keep the trusted component small, don't build infrastructure ahead of proven need.

Node-health gets reported honestly as "structurally installed, not yet control-plane-registered," a real distinction the installer contract had already drawn between a service passing its own health check and that service actually being known to the control plane, which needs network transport that doesn't exist yet.

Two real forks came out of the design pass, both put to mikho rather than picked silently. nginx.conf needs one specific include line for the agent's precondition to hold, but the manifest marks that exact file as both read-only and refused, a stronger pairing than anything else in this project. Should the installer insert that line itself, or require an operator to add it by hand first? Manual prerequisite won, the recommended option: the installer only ever reads the file to confirm the line exists, fails preflight with the exact remediation text if it doesn't, and never writes to a refused root under any circumstance. And how should the nginx package itself satisfy the contract's requirement for a pinned, verified, provenance-tracked artifact? Here the answer broke a pattern part 6 of this series identified across the project's first sixteen questions: live apt-get install, trusting APT's own GPG chain over the pinned Ubuntu archive, won over vendoring a pinned .deb file. Not the stricter option this time. The lighter one, chosen specifically to keep this phase scoped to the installer itself rather than growing it into a release-engineering pipeline, a different kind of discipline than caution, but discipline all the same.

What almost shipped

Here's the part that matters most. The first implementation draft made the shipped, production Go agent binary, the exact binary Phase 7 had already independently verified against a real nginx process, read environment variables to override its own hardcoded host paths. One of those variables, LESTA_NGINX_BINARY, named the executable path the agent's own code passes straight into exec.Command().

Sit with what that actually means for a second. Anything able to set that process's environment, a misconfigured service unit, a compromised adjacent process, a mistake in how the binary gets launched, could have redirected a privileged host-management component into executing an arbitrary binary instead of the real nginx. The stated reason for adding it wasn't malicious or careless, it was almost sympathetic: the installer's self-test needed some way to point the agent at a disposable, isolated nginx instance rather than the real host paths, and environment variables were the easy way to thread that through.

It got caught during review, not by the implementing agent, and not by any test that was already written, since the mechanism was new enough that nothing existing was positioned to catch it. The fix reverted main.go to exactly Phase 3's original state, zero environment sensitivity, the exact binary this project had already earned trust in. The self-test got redesigned instead of the binary: it now runs after install_nginx, not before it, against the real, just-installed system nginx, using the unmodified production binary directly. That required reordering the node-health bootstrap step to come after nginx installs, a real structural change to the script's own execution order, documented plainly as a deliberate departure from what the dependency graph alone would suggest, because that graph describes capability-usability gating for a hypothetical generic runner, not this specific script's own step sequence.

Two smaller things came out of that same review pass. The vendored agent binary got rebuilt with flags that strip debug symbols and local build-path strings, so the first binary blob this repository has ever committed doesn't leak the machine it was built on. And a CI job the approved plan explicitly called for had quietly never been added by the implementation, added now, before anything got committed.

It's worth being precise about why this specific shape of bug matters more than an ordinary logic error would. This series has been careful, since part 1's teardown of legacy Vesta, to treat "a privileged component that can be redirected by something an attacker controls" as its own distinct, named category of failure, not just a bug like any other. Legacy Vesta's entire read path ran stored tenant data through eval, an attacker-influenced value becoming attacker-influenced code. An environment variable read by a privileged Go binary and passed straight into exec.Command() is a structurally different mechanism, but the same shape of problem: something outside the trust boundary gets to decide what a privileged process actually executes. The whole reason this project drew a hard line at "no generic RunCommand, no exec from controllers" back in part 1 was to make that category of bug impossible by construction in Laravel. This was the first moment the Go side of the boundary came close to quietly reopening the same door from the other side, for a reason that had nothing to do with carelessness and everything to do with a test needing a quick way to point at a different target. That's exactly the kind of bug a "this class of mistake is not allowed to exist here" rule is supposed to catch regardless of how reasonable the reason for introducing it seemed in the moment.

A toolchain trap, and the discipline of not "fixing" working code

Running the script under plain macOS sh produced completely corrupted JSON output, every key and value empty. It would have been easy to start patching the script to work around whatever seemed broken. Instead, the actual cause got tracked down first: macOS's default sh plus BSD sed was mishandling one of the JSON-escaping passes, a real difference from Ubuntu's actual /bin/sh plus GNU sed combination this installer is written for and will actually run under. The fix was matching the real target toolchain locally, GNU sed via a PATH shim, dash as the actual shell, not touching a script that was already correct for the environment it was built to run in. Nothing in install.sh itself needed to change.

Four more bugs, found only by actually running it

Committed, then GitHub Actions ran the installer for real against a genuine Ubuntu host for the first time, and found four more problems across the next five pushes, each fixed and pushed as its own commit, each verified by watching the real run afterward rather than assuming a fix worked. ShellCheck's own --severity=style setting turned out to be its most verbose mode, not a filter, so notices about sourced library files failed the job until explicitly excluded. The runner image ships Apache pre-installed and has no nginx include line at all, both genuinely correct preflight rejections, not bugs, resolved by having CI perform exactly what an operator's own manual prerequisite already documents before the installer's preflight gets a fair run. A port-occupancy check compared a single process match against a literal string, which broke the moment the runner's dual-stack networking produced two matching lines instead of one, fixed by checking every matching line instead of just the first. And a rerun failed because deleting a resource doesn't erase its generation history, correct product behavior, a deleted resource's identity is never reused, but it meant the self-test's own fixed, hardcoded test domain collided with its own history on every second run, fixed by generating a fresh identifier per run instead.

After that fifth push, every job went green on the first attempt: the existing test suite, the agent's real-capability suite, and the new installer job, ShellCheck, a real dry run, a real apply against an actual Ubuntu host, a real idempotent rerun, and the agent's own contract suite running against the state the installer had just produced.

What's coming

Phase 5 has already started, the relational foundation for DNS, the next full service slice this project is taking on. It isn't finished yet as of this post, so per the pattern every part of this series has held to, it doesn't get written up until it is.

One more thing worth naming plainly before closing this one out. Nothing about this bug was caught because the plan was bad or the implementing pass was careless in some obvious way. The plan never asked for an environment-variable override; it was a reasonable-sounding shortcut invented mid-implementation to solve a real, narrow problem, how does a self-test reach an isolated nginx without needing the production binary to already know about test fixtures. That's exactly the kind of addition that's easy to wave through, because it's small, it's locally justified, and it doesn't look anything like the obvious kind of bug a reviewer goes looking for. It looked like test infrastructure. It was a privilege boundary. The gap between those two descriptions is the whole reason this project keeps insisting on reading the actual diff rather than trusting a summary of what changed, in this series and, going forward, in whatever gets shipped after it.

mikho

LESta, part 9: DNS gets a schema and a screen

Tags: laravel, react, hosting-panel, vibe-coding, security

With the nginx installer done and CI-green, this project faced the same fork it had already named as a real option back before Phase 4 even started: finish out the web profile with Apache, or move on to DNS entirely. DNS won, the recommended choice, matching ADR 0002's own build-sequencing text plainly: DNS comes right after installer hardening, before mail, cron, or anything else. This part covers the two phases that came out of that choice, the relational foundation and the screen built on top of it, one post because together they tell a single, complete story: DNS existing in this product for the first time.

A different shape than web hosting, on purpose

DNS zones and records aren't just web domains with a different name. The design pass for the relational foundation deliberately read two different precedents before writing any schema, WebDomain's own account-level cascade style, and NodeCapability's completely different one, because the two resources needed different answers.

A zone suspending cascades to its records as a raw model flip, a bare loop calling suspend() on each child, mirroring how a node suspends its capabilities, not how an account suspends its web domains through a full audited action call per child. The reasoning is concrete: every one of a zone's N records would otherwise generate an identical, zone-wide provisioning payload, one real operation dressed up as N redundant ones. A record suspended on its own, outside that cascade, still gets its own audit event, but the provisioning operation it triggers goes against the zone, the actual thing that gets provisioned, and carries the verb Update, not Suspend, since Suspend/Unsuspend are reserved for the zone itself changing state. Small, but the kind of distinction that's easy to blur if you're just copy-pasting the last resource's pattern instead of asking what's actually true of this one.

What a DnsZone deliberately doesn't store is its own small lesson in not building ahead of need. No IP field, an A record is a rendering-time default, not an intrinsic property of the zone itself, and this phase adds no auto-record-generation to justify one. No serial number, that's a BIND zone-file artifact with no relational meaning, and the provisioning operation's own existing version and digest fields already cover "what was last rendered" if a later phase ever needs it. No record count column, derived, never stored, the same choice WebDomain already made for its aliases. Each omission has a specific, stated reason rather than being an oversight, which matters later: it's the difference between a gap that was decided and a gap nobody noticed.

The quota shape branches too. Zones count per account, same as web domains always have. Records count per zone, not per account, confirmed against the legacy field list rather than assumed, which means an account with ten zones can have up to the package limit in each one, not the limit total across all ten combined. And per Q24, DnsRecord got its own UUID now rather than deferred, a small, deliberate exception to this project's usual "don't build ahead of need" instinct, justified because Phase 6's standalone record editing was already a decided, not hypothetical, near-term need.

A bug sqlite would never have shown you

Independent verification, reading every new model, action, and policy directly rather than trusting the implementation report, caught one real problem: the unique index on dns_records (dns_zone_id, name, type, value) sat on a text-typed value column, with a comment in the migration claiming this project's "only supported database driver is sqlite."

That's true of the .env sitting on this machine. It has never been true of the actual decided target. ADR 0002 settled MariaDB 11.4 LTS as the production database engine back before a single feature existed, the very first product decision this whole project made. A full-text column carrying a unique index works fine on SQLite, which doesn't enforce the same limit, and would fail the moment that table gets created against real MariaDB, whose InnoDB storage engine caps how much of an index key it will actually store. Local development running on sqlite is exactly why a passing local test suite would never have caught this, the constraint that breaks it doesn't exist in the database every test actually runs against.

Fixed by bounding value to a real length and correcting the comment to state the actual situation rather than the inverted one. This phase's CI run went green on the very first attempt, no follow-up commits needed, a real contrast with the nginx installer's five-push saga two parts ago. Worth naming plainly: that's not because this phase was more careful. It's because the mistakes here were the kind independent review of the actual code catches before a commit, where the installer's bugs were the kind that only a genuinely different execution environment, real Ubuntu instead of this Mac, could ever have surfaced. Different bug shapes need different kinds of verification, and this project has been running both kinds consistently, not just whichever one happens to be easier.

Building the screen without a fresh question

Moving from schema to UI didn't need a new "what's next" question this time. The DNS sub-phase sequence, foundation then UI then real capability then installer, had already been decided by precedent and wasn't a genuinely open fork anymore. That's worth noticing on its own: this project asks a real question when a real fork exists, and stops asking once a pattern has actually been established, rather than performing consultation on decisions that aren't decisions anymore.

The UI design pass caught two things worth a closer look before adopting them. First, a shared TypeScript types file: dns.ts couldn't just declare its own copies of SuspensionSource and ProvisioningStatus, since the app's actual types barrel re-exports everything from one module, and a naive mirror would create a duplicate-export collision for anything importing from @/types. Fixed by having dns.ts import those two types rather than redeclare them, caught by actually reading resources/js/types/index.ts directly rather than assuming the shape.

Second, and more interesting: a nested route, a record living under its own zone in the URL, needed an explicit guard confirming the record actually belongs to that zone, or a crafted URL could pair a real record with the wrong zone. The design's own first-draft explanation blamed a specific Laravel mechanism, a pluralization mismatch breaking route-scoping. That got checked, not accepted, by tracing Laravel's actual binding-resolution code directly in the framework source. The real reason turned out simpler and slightly different: Laravel only attempts that kind of scoped resolution when a route explicitly opts into it, which these plain routes never did, so there was no mechanism to break in the first place. The fix ends up identical either way, one explicit ownership check before anything else runs, not a security hole since the policy itself checks the record's real zone regardless of the URL, but a real correctness bug without it, the difference between an ugly error and a wrong one. Getting the actual reason right mattered less for the fix than for making sure the next person reading this code understands what's actually being guarded against.

Zones get pages, records get a table

One product decision shaped the whole frontend: zones get real top-level pages, a list, a create form, an edit screen, but records live entirely inline on their zone's edit page, no separate index anywhere. A record's name is zone-relative, its type and priority only make sense next to the zone that owns it, so a flat list of every record across every zone would show data with no useful shape to sort or filter by. It also just matches how every comparable panel already does this, cPanel and Plesk both manage records on the zone's own screen, not a separate global list, and there was no reason to invent a different convention here.

Record editing itself was a real, if small, decision rather than an assumption. It would have been easy to ship record creation and deletion only, forcing a typo fix through delete-and-recreate. That got rejected specifically because UpdateDnsRecord and its policy ability already existed from the relational-foundation phase, a concrete signal the record's lifecycle was meant to be fully reachable, not a speculative feature being built ahead of any evidence it was wanted.

Validating a DNS record turned out to need two new rules neither of the app's existing validators covered. A record's own name isn't a domain, @ and * are valid on their own, and real records like _dmarc or _sip._tcp need underscores the existing domain validator explicitly forbids. And a CNAME, NS, PTR, or MX target isn't quite a domain either, BIND's absolute-name notation allows a single bare label or an optional trailing dot that a normal hostname check would reject. Two purpose-built rules instead of stretching the existing one to cover cases it was never designed for. TXT, SRV, and CAA values stay bounded opaque strings, no attempt to parse their internal sub-fields, the same boundary the schema itself already drew a phase earlier, kept consistent rather than quietly expanded now that a form existed to fill it in.

A bug the implementation found on its own

This time, a real bug got caught by the implementing pass itself rather than by review. Inertia's <Form> component defaults one of its options, preserveState, to true on POST/PUT/DELETE submissions. On the zone edit page, several actions, adding a record, editing one, suspending or unsuspending one, all redirect back to that exact same page. With preserveState true, the page component never remounts, which meant a plain, uncontrolled dialog would just stay open after a successful save, showing a form that had already done its job.

Fixed by giving each of the three affected dialogs, add, edit, and suspend/unsuspend, explicit controlled open state, closed from the form's own success callback rather than left to component lifecycle. What's genuinely worth noting is the check on why the two delete dialogs, record and zone, didn't need the identical fix. A zone delete redirects to a completely different page, so the whole component tree it lived in is gone regardless. A record delete keeps you on the same page, but the deleted row itself vanishes from the refreshed data the moment the response lands, taking its dialog instance down with it. Two different reasons the same failure mode simply can't occur there, verified rather than assumed once the first bug made it worth double-checking every dialog on the page rather than just the one that broke.

Both done, both green

Phase 5 committed as 1000a2b, CI green on the first attempt. Phase 6 committed as 3f049b7, CI green on the first attempt too. Both independently re-verified beyond the implementation's own report: every new model, controller, policy, and Rule class read directly, the full test suite re-run rather than trusted, a real browser walkthrough of the golden path before either phase counted as done.

DNS now exists in LESta the way web hosting has since part 5: a tenant can create a zone, add and edit records inline, suspend and unsuspend at either level, and see it all reflected honestly, because underneath it the exact same provisioning discipline this whole project has held to since Phase 1 is doing the work. Two pieces remain for the DNS slice specifically, the real Go bind9 capability and its installer, and the first of those turned out to have the most interesting bug this series has covered yet.

What's coming

Part 10 covers that bug directly: a rollback design that would have worked perfectly in every test anyone thought to write, and still quietly broken production at some unpredictable point in the future, caught not by a test failing, but by someone tracing the actual math by hand before it ever shipped.

mikho

LESta, part 10: the bug that would have waited

Tags: laravel, react, hosting-panel, vibe-coding, security

Part 8 covered a bug that would have shipped immediately, a privilege boundary quietly reopened by an environment variable, caught before it ever reached main. This part covers a different shape of bug entirely, one that would have passed every test anyone thought to write, shipped clean, worked correctly for a while, and then broken production at some point nobody could have predicted in advance. It's the best evidence yet that "verify independently" means something more than "run the tests."

Checking BIND before designing anything

Phase 7 mirrors what Phase 3 already did for nginx: a real Go capability, dns.bind9.v1, tested against a real disposable BIND9 instance instead of a fake. The same discipline came first. BIND 9.20.27 got installed locally, and several real behaviors got confirmed directly rather than assumed, exactly the pattern this project has held to since the first time it touched an unfamiliar tool.

Does BIND's own config format support glob-pattern includes the way nginx's does, so the same owned-directory architecture could carry over unmodified? Confirmed, yes, verified via named-checkconf accepting a synthetic config built on one. Does an NS record need an in-zone glue record for every nameserver, or only when the nameserver's own hostname happens to live inside the zone being served? Confirmed by hand, with two deliberately different test zone files, one accepted, one rejected: an out-of-bailiwick nameserver, a fixed, external hostname the zone doesn't own, needs no glue record at all. And does named-checkconf -z perform one complete check, structure and every referenced zone file's actual content, in a single call, the direct BIND equivalent of nginx -t? Confirmed by pointing a config at a deliberately corrupted zone file and watching the plain check pass while the -z variant caught it.

That last finding did real design work on its own. A DnsZone, from part 9, deliberately has no IP address field and can exist with zero records. That's fine, and now provably fine, because a zone with nothing in it still validates and serves, as long as the capability synthesizes a fixed set of out-of-bailiwick nameservers at render time. No schema change needed to support the case that looked, on paper, like it might need one.

A bug caught before it was even presented

Here's the part worth sitting with. The design pass considered the obvious approach first: mirror nginx's rollback mechanism byte-for-byte, restage the previous generation's stored content verbatim into a new generation number. That's exactly what nginx does, and it's correct there, because an nginx vhost fragment is self-contained, nothing inside it references its own generation number.

A BIND zone stanza isn't self-contained the same way. Its file directive has to point at an actual path on disk, and that path necessarily includes a specific generation number, .../generations/<n>/zone.db. Byte-copying an old generation's content into a new generation slot doesn't fix that reference, it just carries the old, now-wrong generation number along for the ride, embedded in a file that's about to become the live, currently-served configuration.

Nothing about that breaks the very next reload. The rollback succeeds. DNS answers correctly. Every test anyone would think to write at that moment, does the zone still resolve, passes cleanly. The actual damage is patient: each additional rollback on the same resource, if it ever happens again, compounds the gap between the generation number a stanza references and the generation counter's own ever-climbing count. The generation store only keeps a fixed number of past generations, five by default, pruning older ones as new ones get created elsewhere in the system. At some point, on some resource, after some number of rollbacks nobody can predict in advance, purely because time passed and other operations kept incrementing the counter, the specific old generation a live stanza still points at falls outside that retention window and gets pruned, while still being the config a real nameserver is actively serving from. The failure shows up on whatever reload happens to come next, for whatever unrelated reason triggers it, with no obvious connection back to a rollback that might have happened days or weeks earlier.

This got caught by the design pass itself, before it was ever presented as a finished plan, and it didn't stop at spotting the problem. It got re-traced by hand afterward, independently, generation number by generation number, through an actual repeated-rollback scenario, to confirm the failure mode was real and not just a plausible-sounding worry. The fix that came out of it: rollback has to re-render fresh zone content into the new generation number, using the same stored payload every forward operation already renders from, never copy old bytes forward under a new label.

Two smaller findings, one dispatch decision

The same design pass caught a real zone-file syntax hazard. TXT and CAA record values stay deliberately opaque strings, a decision Phase 6 already made and this phase inherited rather than reopened, but an opaque string can still contain an unescaped quote or backslash, and a zone file is real syntax that a stray quote will happily corrupt. Nginx's own fields never needed this, because none of them accepted anything close to arbitrary tenant text. And a precise correction to the DNS semantics themselves: a node that no longer serves a zone at all has to answer REFUSED, not NXDOMAIN, since NXDOMAIN is specifically an assertion the zone's parent nameservers make about delegation, not something this node is in a position to claim on its own.

One real architectural fork got a name and a decision before implementation started: should the existing agent binary learn to dispatch between the nginx and bind9 capabilities based on which one an operation names, or should DNS get an entirely separate binary? One binary won, matching how the whole node protocol already frames "the agent" as a single long-lived process per node. Worth naming what this decision explicitly checked before being approved: whether adding a second capability would reopen the environment-variable exec-path vulnerability part 8 walked through in detail. It doesn't. Both capabilities' executable paths stay fully hardcoded in their own configuration function, with zero variability either way, the same discipline holding for a second capability that had already been earned for the first one.

The architecture underneath, in one sentence

Every zone this capability manages lives as two related files: a small stanza in BIND's own owned config directory, staged and activated through the identical dotfile-then-atomic-rename pattern nginx already established, and the actual zone data it points to, stored alongside the same generation manifest every resource in this system already keeps. The whole rollback bug lived in the seam between those two files, because nginx never had to think about that seam at all, one self-contained fragment was always the entire story. DNS made the two-file shape unavoidable, since a zone's data genuinely is separate from the config that references it, and unavoidable is exactly where a mechanical copy-paste of a working pattern stops being safe.

The disposable test harness mirrors nginx's closely on the surface, a hand-written, fully isolated named process with its own directory, its own port, its own control channel, started and stopped per test with no system-wide service touched. But it came with an honest caveat attached rather than a quiet assumption: the control-channel configuration syntax was confirmed correct by reading BIND's own documentation during design, not yet proven by actually starting a real disposable process, and that gap got closed before any test was written to depend on it, not after.

What verification actually looked like this time

The rollback fix didn't get trusted just because the design document said it was handled. The central dispatch file got read end to end, and the rollback path specifically got traced by hand again, this time against the real committed code, to confirm the fix that had been designed was the fix that had actually been written: a new generation's stanza and its zone data both freshly rendered, self-contained, never pointing at anything older or prunable.

The single test proving this mattered most got checked individually, not just run and trusted to pass. It doesn't just confirm DNS still answers after a rollback, that would pass even with the bug still present, for a while. It asserts the rolled-back result lands in a genuinely new generation ID, reads the live stanza file directly and confirms it references that new generation's own zone data path rather than the original, confirms that specific file actually exists on disk, and only after all of that re-queries DNS to confirm the original content is what actually gets served. Every link in that chain has to hold for the test to mean anything, and every link got checked individually rather than inferred from a green checkmark.

The implementing pass itself surfaced one more real finding along the way: BIND's own glob-include directive, unlike nginx's more forgiving equivalent, fails outright the moment it matches zero files. A freshly provisioned node with no zones yet would break on exactly that condition. That got independently re-verified against the real local BIND install too, a bare config pointed at a genuinely empty include directory does fail exactly as described, before the fix, a small self-healing placeholder file that keeps the glob non-empty, got accepted. Flagged plainly for later: whatever eventually builds the bind9 installer needs to seed that placeholder before BIND's very first start on a real node.

Shipped, and one genuine unknown resolved cleanly

Committed as e296cb4, and the actual CI run watched to completion rather than assumed from a green push notification, all three jobs passing on the first attempt, including the one real unknown going in: whether bind9-utils was actually the correct Ubuntu package name for the command-line tools this capability depends on, flagged honestly as unverified before the run rather than quietly hoped past. It resolved cleanly.

The DNS slice now has three of its four pieces: the relational foundation, the screen, and a real Go capability proven against real BIND9 the same way nginx was proven against real nginx four parts ago. Only the installer, the piece that takes a bare node to the state this capability already assumes exists, remains.

What's coming

That installer hasn't been built yet as of this post, so it doesn't get written up until it has been, the same rule this series has held to since part 4. What's worth carrying forward from this part specifically: the most dangerous bugs aren't always the ones that fail loudly and immediately. Sometimes the ones worth losing the most sleep over are the ones that pass every test you'd think to write today, and only ever fail later, for a reason that looks, from the outside, like it has nothing to do with the actual cause.

mikho

LESta, part 11: the firewall that closed its own doors

Quick recap for anyone new: this series follows LESta, a Laravel plus React rewrite of VestaCP, built almost entirely through prompts run against Claude Code rather than hand-typed. By the end of the last part, the DNS side of the product had a schema, a screen, and a real BIND9-speaking Go agent, three of four sub-phases done. The fourth was the installer, the script that actually gets bind9 running on a bare node. This part is that installer, and it turned out to be sitting on top of a bug that had nothing to do with DNS at all.

The second caller

Phase 4 built the real nginx installer and, buried in its own plan, left a note for whoever came next: the moment a second real installer exists, a question that had only ever been theoretical becomes answerable with evidence, whether shared infrastructure between installers should be extracted or left duplicated. Nobody guessed at the answer ahead of time. It just got flagged and left alone until there was a second data point to check it against.

bind9 is that second data point. Before writing a line of the new installer, the design pass went back and reread nginx's shipped one in full, and it found something that had been sitting there, live, since Phase 4: bootstrap_firewall_baseline renders the entire firewall table from scratch, every time it runs, from whichever installer happens to be calling it. nft -f against a full table definition replaces its contents. It does not merge. Run the bind9 installer on a node that's already serving nginx, and the ordinary case, one small box doing both web and DNS, would silently slam ports 80 and 443 shut the moment bind9's own ports got written in their place.

That's not a hypothetical edge case. It's the default outcome for the single most likely real-world deployment shape this whole product is built for.

Rejecting the tempting fix

The obvious patch is to make the firewall renderer aware of every service that could ever be installed, union their ports together from the manifests already sitting in the repo, and render the whole thing in one pass. It even sounds like the more thorough option.

It's wrong, and a Plan agent caught why before any of it got written. Deriving the port list from every manifest that exists in the checkout would open MariaDB's and mail's ports on a node that never installed either of them, contradicting the project's own deny-by-default posture from day one. Worse, it would silently open new ports on an already-running node the next time someone ran git pull, purely because a new service manifest happened to land in the repo, with no install action ever taken on that node at all. A firewall rule appearing because of an unrelated code change, not because anything was actually installed, is its own quiet category of bug.

The real fix is smaller and, once you see it, obvious: a per-node ports registry. Each installer writes its own fragment file, one file per service actually bootstrapped on that specific node, to /etc/lesta/firewall/ports.d/. Rendering the table means reading every fragment present and unioning them, order-independent, rerun-safe, and reflecting only what's actually been installed on that box, never what merely exists somewhere in a git checkout. nginx writes its fragment when it bootstraps. bind9 writes its own. Whichever runs second doesn't erase the first.

The bug that had been there since Phase 4

Building the regression proof for this fix turned up something nobody had gone looking for. The new verification step lists the live nft table's contents after both installers run in sequence, specifically to confirm nginx's rules survive bind9's install. What it found instead: nginx's own accept rules, tripled.

nft -f on a table that already exists doesn't replace a chain's content, it appends on top of whatever the kernel already holds. That behavior had been sitting in nginx's original bootstrap_firewall_baseline unchanged since Phase 4. It just never showed up, because nothing before this phase had ever actually inspected the rendered ruleset's contents across repeated applies, only whether the apply succeeded. Every rerun of the nginx installer had been quietly stacking a fresh, redundant copy of the same accept rules on top of the last one.

The fix is the standard nftables idiom, now living in the shared lib/firewall.sh: declare the table and chain empty first (a no-op if they already exist with the same type, hook, priority, and policy), flush the chain explicitly, then repopulate it, all inside one atomic nft -f transaction. It's the kind of bug that costs nothing functionally, duplicate accept rules still accept, until the day someone needs to actually read that ruleset to debug something else and finds three copies of everything staring back.

Three wrong turns before the right one

The other real chase in this phase started with a plain "permission denied" and took three attempts to actually close.

A test zone create kept failing its health check with a DNS-level SERVFAIL. The named journal blamed a permission error loading the zone file. /var/lib/lesta/bind is owned 0750 root:lesta, and the bind9 package's own named process runs as the bind system user, which had never been added to the lesta group. First fix: usermod -aG lesta bind, mirroring what the Go agent's own user already needed.

Same error, unchanged. The reason took another pass to find: bind9's package install itself starts named automatically during apt-get install, before the usermod call ever runs, and Linux only reads a process's supplementary groups at exec time via initgroups, never again afterward for an already-running process. systemctl enable --now is a no-op against a unit that's already active. rndc reload re-reads config and zones in place without re-executing the process. Second fix: force an unconditional systemctl restart named right after the group grant, so the new membership actually takes effect. Still the exact same denial.

At that point namei -l and id bind both confirmed every Unix permission and group membership were genuinely correct. The problem wasn't Unix permissions at all. Reading the actual AppArmor profile Ubuntu ships for named settled it: /etc/apparmor.d/usr.sbin.named allow-lists only /etc/bind/**, /var/lib/bind/**, and /var/cache/bind/**. Nothing about /var/lib/lesta/bind. AppArmor confines the process independent of standard file permissions entirely, and an earlier diagnostic step, running sudo -u bind test -x against the path, had given false reassurance the whole time, because AppArmor confinement attaches to the /usr/sbin/named binary itself, not to the identity running it, so a plain shell command under that user never actually runs inside the confined profile at all.

Third fix, the real one: append an allow rule to Ubuntu's own sanctioned extension point, /etc/apparmor.d/local/usr.sbin.named, the exact file the shipped profile's last line already includes for precisely this purpose, so nothing about the vendor-owned profile itself needs editing or risks getting silently overwritten on the next package upgrade. Reload with apparmor_parser -r, and the SERVFAIL was gone. The Unix group grant and the forced restart both stayed in the final script anyway, real independent hardening, not reverted just because they turned out not to be the actual fix.

Even that wasn't quite the end of it. Once the profile was genuinely enforcing, Phase 7's own already-shipped Go test suite broke too, because its disposable test harness points a real named process at a config file under a /tmp path the same production profile denies by design. The fix stayed narrow and scoped only to the disposable CI runner, never touching the enforce-mode profile a real node gets: switch AppArmor to complain mode for that one profile, right before the test step, using apparmor_parser directly rather than the friendlier aa-complain tool, which turned out to choke on two unrelated, pre-existing profiles the runner image ships for Microsoft Edge before it ever got anywhere near named.

The rest of the pile

Eleven distinct bugs surfaced across nine follow-up commits once this hit real CI, and the two above were the deepest, but not the only ones worth naming. The CI runner's ShellCheck build silently accepted a guard-clause shape the local one had been flagging, version drift in what counts style-clean, resolved by rewriting the ambiguous form outright rather than chasing which version was "right." A port-conflict check treated any existing listener on port 53 as a hard conflict, tripped by nothing more sinister than systemd-resolved's own stub resolver, which binds 127.0.0.53 by default on every Ubuntu box; a loopback-only bind and a later wildcard bind from a different process coexist fine, so the check learned to skip loopback listeners first. systemctl enable --now bind9 failed outright because bind9.service is only a systemd alias for the package's real unit, named.service, and enable refuses to operate on an alias the way start and stop happily do. And the health probe itself was borrowed from nginx's own curl-first design, which works great against an HTTP server and not at all against a DNS one, since curl's telnet:// scheme performs real protocol negotiation that named simply doesn't speak; a plain TCP connect check took its place.

One more is worth a line of its own for the sheer shrug value: the vendored agent binary shipped in this repo hadn't been rebuilt since Phase 4, and Phase 7 had added an entire new capability to it since then without anyone noticing, because Phase 7's own verification ran straight against source with go test, never touching the committed artifact. This installer was the first thing to actually exercise that binary against something other than nginx, and it promptly found it stale. Rebuilt, re-pinned, moved on. It would not be the last time a stale binary quietly outlived the code it was supposed to reflect, later parts of this series come back to that pattern more than once.

Where the slice lands

Final commit d680d7f, nine follow-up commits stacked on the original 4fd8ce1, CI run 33248074589 green across all three jobs. With this phase done, the DNS slice this project has been building since part 9 is complete end to end: a relational foundation, a real screen to manage it, a Go agent that actually speaks BIND9's config format, and now an installer that gets the whole thing running on a fresh node without stepping on whatever else that node happens to be running.

The through-line worth sitting with isn't really about DNS. It's that the most dangerous bug in this phase had nothing to do with the feature the phase was nominally about. A firewall table that silently replaces itself instead of merging is the kind of thing that looks completely fine in isolation, every single-service test passes, right up until a second real caller shows up and the assumption that was never actually written down anywhere gets tested for the first time. Phase 4's own plan knew enough to flag that this moment was coming. It just couldn't know in advance what shape the bug would take, only that leaving the door open for evidence to arrive later, instead of guessing at an abstraction with one data point, was the safer bet. It was.

mikho

LESta, part 12: the second web server, and the same bug twice

Part 11 closed the entire DNS slice, a nftables table that quietly closed its own doors and a bug that had been silently tripling firewall rules since part 4, found only because Phase 8 finally built the regression test that could see it. This part is a smaller story, mostly, except for the one moment where a bug from two parts ago shows up again wearing a different service's name, and gets recognized on sight.

Apache, but only the half that needs to exist

With DNS done, there was no established sequence saying what came next, so the choice went to an AskUserQuestion round rather than getting assumed: the Apache Go capability, hardening ACME certificate issuance, or wiring installer-contract CI enforcement into the pipeline. Apache won.

Three research passes ran in parallel before any design started, reading ADR 0002's exact wording on Apache and the both profile, the existing web.nginx.v1 capability's architecture file by file, and the Laravel side, WebDomain, Node, NodeCapability, the code that decides which capable node serves a given domain. The finding that mattered most: Laravel needed zero changes. WebDomain::toProvisioningPayload() was already fully server-agnostic, nothing nginx-specific baked in. NodeCapability was already a bare free-form string column, perfectly able to hold web.nginx.v1 and web.apache.v1 on the same node at once, with a test factory already faking exactly that scenario. And the routing logic that decides which server wins when a node runs both had already been written, its own doc comment stating the rule outright: "when a node has both active (the 'both' profile), nginx always wins." Nobody had to design that decision. It already existed, waiting for a second capability to make it matter.

That let the whole phase stay Go-agent-only, mirroring how the very first nginx capability was scoped back before the installer for it existed.

Two genuinely new findings came out of the design pass, one of them checked by hand against this machine's own built-in Apache binary rather than assumed from documentation. Apache's Include directive hard-fails the moment its glob pattern matches zero files, the same failure BIND9 hit in part 10, but where BIND9 needed a small self-healing placeholder file to keep its glob non-empty, Apache offers IncludeOptional, which tolerates an empty directory outright. No placeholder hack required, a cleaner solution than either of the two precedents this project had already built. And nginx's inline return 200 '...';, the trick that lets a resource serve static content directly from its own config fragment, has no Apache equivalent at all. The answer was mod_asis, designed by direct analogy to the two-file-per-resource pattern BIND9 already established: a thin config stanza pointing at a separate, generation-numbered content file.

What the Mac didn't know about Ubuntu

The implementation landed clean by every local measure, gofmt, go vet, a fresh go test ./..., all thirteen real-capability tests passing against a genuine disposable httpd process. Then CI's agent job failed on the very step meant to prove nothing needs a capability pre-installed to pass: every Apache test broke with module unixd_module is built-in and can't be loaded.

The cause was a platform gap invisible from a Mac. GitHub Actions' Ubuntu runners ship apache2 pre-installed already, and Ubuntu's real apache2 binary compiles mod_unixd statically into itself. This Mac's own /usr/sbin/httpd build doesn't, confirmed directly by asking it: httpd -l on the Mac reports only three modules built in, none of them unixd. The test harness's fixed LoadModule unixd_module ... line worked every time locally because the module genuinely needed loading there. On real Ubuntu, that same line tried to register a module Apache already owned statically, a hard startup error, not the harmless duplicate-load warning the harness's existing logic already knew how to shrug off.

The fix queried <binary> -l and skipped whichever modules came back already compiled in, applied to every module the harness loads, not just the one CI happened to name first. That mattered on the very next push. The rerun failed differently, a bare exit code with no stderr at all, a real regression from a failure that used to at least explain itself. Rather than guess twice, the harness got a diagnostic-only commit first: read and print Apache's own ErrorLog file on any startup failure, since a parse-time error prints to stderr but a later startup failure writes to the log file instead, once parsing has already moved past that line. That commit shipped alone, no guessed fix riding along with it.

The next CI run's real output showed two things at once. Ubuntu's apache2 also statically compiles mod_log_config, already caught by the general fix. And underneath it, a second, unrelated bug: mod_mime's default TypesConfig path is relative to Apache's own server root, and the harness's relocated, from-scratch root had no file there. mod_asis never needs MIME sniffing, every response it serves already carries an explicit Content-Type: header, so the fix was a throwaway mime.types file whose actual content doesn't matter, just its presence. Four commits total sat on top of the initial implementation before Apache's Go capability landed clean, run 33252996219, all three jobs green, Laravel genuinely untouched the whole time.

The wider option, chosen on purpose

Phase 10 asked the same "what's next" question again, Apache installer, ACME, or CI enforcement, and the Apache installer won again. Before writing a plan, direct research turned up more scope hiding inside that one bullet than expected: the backlog item bundled the installer itself with fixing nginx/install.sh's hard rejection of --web-server apache|both, and with building the actual reverse-proxy template that lets the both profile route traffic to Apache at all. A second round of questions surfaced a real, previously undocumented gap underneath that: nothing decided which domains, on a both-profile node, get served by nginx directly and which get proxied to Apache. WebDomain had no field for it. The routing logic's own doc comment said nginx always wins, full stop, with no path for Apache to ever actually receive traffic.

Three scope tiers went to the user for that reason, the narrowest recommended: just the standalone installer, leave the routing gap for a later phase, matching how this project has repeatedly deferred cross-cutting design until a concrete need forces it. The choice came back for the widest tier anyway, installer, both-profile wiring, and the real per-domain routing design, in one pass, even knowing the routing decision didn't exist yet and would have to get invented from scratch mid-phase.

INSTALLER-CONTRACT.md had already settled the shape of the harder architectural question before anyone had to argue for one side: nginx is explicitly named as "the web installer," with a --web-server nginx|apache|both flag scoped to it alone, while every other leaf-service installer, BIND9 included, carries no such flag. That resolved a genuine fork, does apache/install.sh stand entirely on its own, mirroring BIND9's independence, or does nginx's script absorb Apache's install logic into itself, with evidence rather than a coin flip. Apache's installer stayed fully standalone, its own main(), its own trap, a narrower --web-profile flag just for its own topology awareness, and nginx's script dispatches to it as a genuine child subprocess when asked for apache or both, never sourcing it, keeping the two scripts' error-trap isolation intact.

The routing design itself came out reasonably contained once the fork above was settled: a new web_server column on WebDomain, the node-resolution logic returning an ordered list of capable capabilities instead of a single winner, and toProvisioningPayload() overriding nginx's own template choice to a fixed apache-proxy sentinel whenever a domain is proxied rather than served directly. A new cross-capability test proved the whole path for real, a genuine disposable nginx instance actually forwarding a real HTTP request to a genuine disposable Apache instance's real content, not two capabilities individually mocked and trusted to compose correctly.

A bug this project had already met

Four real bugs surfaced through CI, three of them small and one of them familiar enough to be worth naming on its own.

The small ones first. apache2 -t's bare invocation fails outright without sourcing /etc/apache2/envvars first, a step every other real invocation of Apache already goes through, apt's own postinst script and the systemd unit included, but which this project's own installer had skipped. Fixed by switching to apache2ctl configtest, the tool built for exactly this. The vendored agent binary had gone stale again, last rebuilt for BIND9 and never refreshed once Apache's capability landed, so lesta-agent rejected web.apache.v1 as unsupported until a rebuild fixed it. And a pre-existing race inside BIND9's own test harness surfaced for the first time under this phase's added CI load, rndc stop returning before named had actually finished exiting, occasionally colliding with Go's own temp-directory cleanup. It had always been there; nothing had ever pushed the timing hard enough to expose it before.

Then the one worth sitting with. With the first three fixes in place, the both-profile installer's own self-test failed its health check with a real HTTP 403. Apache's worker processes run as www-data, and www-data needed to read the actual content file mod_asis serves at request time, a runtime filesystem read nginx never needs, since nginx bakes its content straight into an already-parsed config fragment. That is, name for name, exactly the gap part 11 already found and fixed for BIND9's named user, a Unix group grant alone isn't enough on a system whose default AppArmor profile denies by default, and the same enable --now-versus-restart subtlety underneath it, since apache2's own postinst script starts the service before the group grant ever runs, and Linux fixes a process's supplementary groups at the moment it starts, not whenever someone gets around to adding it to a group afterward.

The fix was a direct copy of BIND9's own solution: usermod -aG lesta www-data, an AppArmor local-override extension scoped to /var/lib/lesta/apache, and enable followed by an explicit restart rather than enable --now. What made this bug different from the first three wasn't its mechanism, it was the moment of recognition. Nobody re-derived the diagnosis from scratch. The shape was already on file from one part ago, and this time fixing it took minutes, not an installer's worth of trial and error.

Final CI run 33272765865, four jobs green, ci, installer, agent, and a new installer-both-profile job built specifically to prove the combined path. That closes the web capability set ADR 0002 originally scoped out, nginx, Apache, or both, real end to end, installer through Go capability through per-domain Laravel routing. The one honest gap left standing has nothing to do with Apache at all: no real network transport exists yet between Laravel and a deployed node's own running agent, a scope cut this project disclosed all the way back when the Go agent first got its node protocol, still unrelated to anything this pair of phases touched.

What carries forward

Two servers now share this project's config-management architecture without either one bending it out of shape, IncludeOptional instead of a placeholder file, mod_asis standing in for a directive Apache never had, the same generation-numbered content-file pattern doing the same job twice in two completely different config languages. And the bug worth remembering isn't really about Apache or www-data specifically. It's that a permission gap shaped exactly like "a worker process needs to read content a root-owned directory is guarding, and a Unix group alone won't cut it on a system with AppArmor turned on" is clearly a recurring category now, not a one-off. The next installer this project builds should probably expect to meet it a third time.

mikho

LESta, part 13: the protocol that couldn't wait

By part 10, LESta had DNS: a relational foundation, a screen, and a real Go capability talking to a real disposable BIND9. What it didn't have was a way to get a real certificate onto any of it. This part covers Phase 11, real ACME and TLS issuance, the largest, most novel phase in the project so far, and the one that made an architectural decision made all the way back at the start finally bite.

The constraint that had been waiting since part 2

Two things had to be checked before any design work could start, and both came from reading, not guessing: an exhaustive search of the codebase confirming WebDomain and DnsZone have no link to each other at all, and a direct read of the node protocol's own committed JSON Schemas confirming something this series has mentioned before but never had to reckon with directly. The protocol is strictly one-shot. One OperationEnvelope goes to the agent, one ResultEnvelope comes back, the process exits. No streaming, no callback, no way to say "check back with me in ninety seconds."

ACME, the protocol behind Let's Encrypt, is not a one-shot conversation. Ask for a challenge, wait for it to propagate, tell the certificate authority to go check, wait for that check to finish, download the finished certificate. Real wall-clock minutes can pass between any two of those steps, sometimes because a DNS record needs to propagate, sometimes because the certificate authority's own validation queue is busy. None of that fits inside a single agent invocation that has to exit the moment it answers.

That's not a bug to fix. It's a structural mismatch between what ACME needs and what the node protocol, by design, will ever provide. The real protocol client couldn't live in the Go agent at all. It had to move to Laravel, which already runs a persistent queue worker with a normal database connection and no deadline on how long a job is allowed to take. The Go side shrinks correspondingly, down to a capability that only ever writes files, tls.acme.v1, no protocol logic, no external process, no waiting on anything.

Given how much that reframes the shape of the whole phase, it went back to the user directly rather than getting assumed and built quietly. Confirmed as the right call. A second question settled the library choice, acmephp/core, an established PHP ACME v2 client, over hand-rolling the JWS signing and state machine from scratch, a real new Composer dependency flagged and approved per the project's own standing rule against silently adding dependencies. A third narrowed scope: both HTTP-01 and DNS-01 challenge types, since an earlier decision had already settled that firmly, but the new HTTPS server block only for nginx this pass, leaving Apache's own template as a separately boundable piece of work for later.

What actually got built

A new AcmeAccount model holds one row per ACME directory URL, its key pair encrypted at rest, registered lazily the first time it's needed, and never once touched by anything that flows through a queued ProvisioningOperation. That last part isn't incidental. The project's own governing document explicitly forbids ACME account keys from ever appearing in a queue payload or a log line, and the model's whole shape exists to make that true structurally rather than by convention.

The real work lives in a new job, IssueAcmeCertificate. It picks HTTP-01 by default and only reaches for DNS-01 when explicitly requested and an exact-match zone actually exists, no auto-detection guessing which zone might answer for a domain. When it does use DNS-01, the challenge TXT record never touches the real dns_records table at all. It gets injected into an ad-hoc payload, applied directly through the DNS capability, validated, and torn down again, verified by a test that asserts the real table is byte-identical before and after the whole dance. The zone stays exactly as clean as if the challenge had never happened, because as far as the database is concerned, it hadn't.

On the Go side, agent/internal/capability/acme/ ended up the simplest capability in the whole codebase. Two resource kinds, a challenge file and a certificate, no template rendering, no external binary to shell out to, no reload, no health check, just an atomic file write with path-traversal-safe validation on the token and domain, and a stricter file permission for anything that's a private key. Every nginx template picked up a shared .well-known/acme-challenge/ location, and a new HTTPS server block gets added once a certificate actually exists, with no forced redirect yet, a product decision left open rather than an engineering limitation.

Two orders, not one

Here's the bug worth sitting with, because it isn't a typo or an off-by-one. It's the AI getting the actual protocol wrong, not just the code around it.

The first draft called requestAuthorization() and then, once that authorization was validated, called requestCertificate() as a separate step. Read on its own, that looks entirely reasonable, request the right to prove you own the domain, prove it, then ask for the certificate. The problem is that in ACME v2's real object model, calling requestCertificate() separately creates a brand new order, and that new order doesn't inherit the validated authorization the first order already earned. Against Pebble, a real disposable ACME test server, this fails, because the certificate authority has no reason to believe the second order is entitled to anything the first one proved.

The fix was to stop treating authorization and certificate issuance as two separate requests at all, and instead call requestOrder() once and finalizeOrder() on that same order object all the way through. One order, one lifecycle, confirmed correct against real Pebble rather than a mock that would have happily accepted either shape.

Two smaller bugs came out of the same review pass. Account registration and the nginx-reload notification that follows a successful issuance were both reachable in a way that could either escape the job's own bounded retry design or silently overwrite a real success with a later failure, fixed by scoping each into its own try/catch so a notification failure can never masquerade as an issuance failure. And a version mismatch turned up between Pebble and the client library: newer Pebble builds advertise an extra challenge type whose response deliberately omits a token field, and acmephp/core reads that field unconditionally and throws. The Go test tolerates the missing field because it decodes into a plain struct; the PHP test doesn't, so the PHP-side Pebble tests got pinned to a specific known-good version instead.

That version pin produced its own small, honestly funny bug. The test file's own skip message told a future developer to install the latest Pebble to run these tests locally, which is exactly the version that makes them fail with a cryptic missing-key error instead of skipping cleanly. Caught during independent verification, not by whoever wrote the test, by actually trying to follow the instructions the test gave and watching them not work. Fixed the message and added a real explanation to the file, since Pebble's own version output is too generic to detect the mismatch automatically.

One more bug surfaced only once CI ran against a certificate that, for the first time in any local run, hadn't parsed successfully. openssl_x509_parse() returns either an array or false, and the code that reads a certificate's expiry date assumed the array unconditionally. Locally, every test run had a genuinely valid certificate to work with, so the false case never fired. PHPStan caught what every previous run had gotten lucky past, a small, clean example of why this project keeps a static-analysis gate in CI rather than trusting local runs alone.

Thirty-three hours, mostly spent asleep

Worth naming plainly, because it's a real part of how this project actually works day to day, not a footnote. This implementation pass took roughly thirty-three hours of wall-clock time, spanning two calendar days, almost none of it spent on the work itself. The local machine went to sleep mid-response five separate times, plus one session hit a rate limit. Each time, rather than restart the agent blind and risk redoing already-correct work, the actual state of the working tree got checked directly, git status, a fresh format and vet pass to find exactly which line had been left mid-edit, before resuming with the specific evidence of where things had actually stopped. One small, unrelated one-line edit that had crept into CLAUDE.md along the way got found and reverted between resumes.

None of that shows up in a changelog as its own line item. It's the same lesson this project keeps circling back to in different clothes: the vault being a plain, local, readable set of files is what makes "come back later and keep going" actually possible, not any single tool doing it automatically.

Shipped, with one gap CI still had to close

Committed as 7d3ecc7, and every file got read directly rather than trusted from the implementation's own report, including a specific correctness question worth tracing by hand: could reusing the same resource ID across a domain's challenge operation and its certificate operation ever cause challenge cleanup to accidentally delete the certificate that had just been installed? Reading the actual file-path logic settled it. Both operations derive the file path they touch purely from the payload's own token and domain fields, never from the resource ID, so the shared ID only ever does idempotency bookkeeping, never file lookup.

CI still found the openssl_x509_parse() gap described above at the static-analysis step, fixed and pushed as a small follow-up commit, 5ccf68d. Final run, 33558857102, all four jobs green, including a real Pebble install and a real Pebble-backed test run inside CI itself, not just locally.

Left deliberately open and disclosed rather than quietly skipped: Apache's own HTTPS template, the ACME installer script itself, wiring Pebble into the main PHP CI job rather than letting those tests self-skip there today, and a forced HTTP-to-HTTPS redirect, a product choice nobody has made yet, not a limitation.

What this phase actually proves

Every phase before this one built something the node protocol was already shaped to carry. This is the first one where the protocol's own shape said no, and the fix wasn't a workaround bolted onto the existing design, it was moving the work to the one place in the whole system that was actually built to hold it. The two-order bug that followed is the more interesting failure of the two, though: not a slip in translating a correct plan into code, but the plan itself being subtly wrong about how a real protocol's own object model works, caught only because the test target was a real ACME server and not a convenient stand-in for one.

mikho

LESta, part 14: the password that stayed the same

By this point in the rewrite, nginx, bind9, and Apache all had real installers, real Go capabilities, and real CI proof behind them. DNS had a schema and a screen. Certificates issued for real, against a real ACME server, from Laravel rather than the node agent, for reasons the last post went into at length. What LESta still didn't have was anywhere for a tenant's own application to put its data. Phase 12 built that: database.tenant.v1, a real MariaDB instance per node, provisioned, suspended, rotated, and deleted the same way everything else in this project gets provisioned, suspended, rotated, and deleted.

Offered the choice between tenant databases, Apache's own HTTPS parity, or finally wiring installer contracts into CI, mikho picked tenant databases, matching the order ADR 0002 had already committed to on paper months earlier. What followed was, on paper, the most straightforward phase in a while: no protocol reframing, no multi-minute challenge cycle forcing a rethink of where code has to live. Just a full vertical slice, model to installer, on the first attempt at full scope. It still found six real bugs, one of them the best slow-burn failure this series has covered yet.

a question none of the other capabilities had to answer

Every capability up to this point renders a file and reloads or validates it with an external binary. nginx writes a config and calls nginx -t. bind9 writes a zone file and calls named-checkconf -z. Apache writes a vhost and calls apache2ctl configtest. None of them ever hold a live connection to the thing they're managing.

A database capability can't get away with that. Talking to MariaDB means either a Go MySQL driver living inside the agent, or shelling out to the real mariadb CLI client and piping SQL over stdin. Put to mikho directly, since nothing already in the codebase set a precedent either way: exec the CLI. The agent module had held a zero-external-Go-dependency policy since its very first capability, and a database driver would have been the first real external Go package the agent ever needed. Piping DDL through a subprocess keeps that streak alive, at the cost of parsing whatever a CLI client decides to print instead of a typed driver response. A deliberate trade, made once, for a reason that will presumably get revisited the day a capability actually needs something a CLI can't give it.

The second real question was subtler and came straight out of the project's own paperwork contradicting itself. ADR 0002 section 3 says database credentials never appear in normal desired-state payloads or queue payloads, full stop, the same rule that already kept the last post's ACME account key out of every payload entirely. But an ACME account key never has to leave Laravel. A tenant's database password does, at least once, because the node is the thing that actually has to set it. The ADR's blanket rule and the tenant capability's basic requirement were flatly in tension, and rather than quietly picking a reading and moving on, that tension went back to mikho as its own decision. The answer: the password travels on create and on a dedicated rotate operation, and nowhere else. Every routine suspend, unsuspend, and delete carries no password at all.

That's not a comment in a docstring. Laravel's toProvisioningPayload() takes an explicit $includePassword parameter rather than deciding on its own whether to decrypt one, and the Go side has both a requirePassword() check and a forbidPassword() check, so a payload can be rejected for carrying a password it shouldn't even if Laravel would never have sent one. A dedicated test asserts a literal array_key_exists on the wire structure, not just "the value looks hidden." Two independent places enforcing the same rule, checked from both directions, because a rule that only lives on one side of a network boundary is a rule that's one refactor away from not existing.

One more small, concrete correctness detail worth naming on its own: rotating a tenant's password uses ALTER USER ... IDENTIFIED BY, never CREATE OR REPLACE USER. MariaDB documents the latter as a drop-then-recreate, which would silently wipe every grant that user had accumulated the moment a rotation ran. The obvious-looking statement was the wrong statement, and this project keeps finding that particular shape of trap in its own SQL.

a gap that got caught before it could cause anything

Independent verification on this phase didn't start by looking for bugs in the code that got written. It started by checking whether the code that should have gotten written was actually there.

Every prior installer phase, nginx, bind9, Apache, had earned a real disposable-Ubuntu CI proof as a matter of course. Reading through the implementation for this phase turned up nothing in .github/workflows/tests.yml at all, not a shellcheck entry, not a --apply step, nothing. The new installer touched a live multi-instance systemd mechanism, an AppArmor profile, and a real external apt repository, and none of it had ever run against real infrastructure. That's not a bug in the ordinary sense, since nothing was wrong yet. It's a bug in what didn't happen: a bar every earlier phase cleared by default, quietly not cleared here.

Fixed by adding the missing --dry-run/--apply/idempotent-reapply sequence to the existing installer job, plus a runner-prep step to purge whatever MySQL or MariaDB variant GitHub's own runner image ships preinstalled. Pushed specifically to generate real evidence rather than assume the gap, once closed, meant the installer actually worked. It didn't, not yet. That real evidence is what surfaced everything that follows.

three bugs this series has already met before

The version check failed first, and for a reason that had nothing to do with MariaDB itself. The pinned repository's real package reports its own version as 1:11.4.13+maria~ubu2404, and a bare shell case pattern matching against "11.4".* never matches a string that starts with 1:, an epoch prefix apt adds whenever a package's version numbering has ever needed disambiguating across a transition. The correct package had installed; the check just didn't recognize it. Fixed by stripping the epoch before comparing, and, while already in that code, replacing the weak version-string check with a real provenance check via apt-cache policy, confirming deb.mariadb.org actually provided the package rather than just eyeballing a version number that could in principle come from anywhere.

Then mysql, MariaDB's own worker identity, couldn't traverse its own datadir's parent directory. /var/lib/lesta is 0750 root:lesta, and mariadb-install-db, run automatically by the packaged systemd unit as user mysql, failed outright with a permission denied on a mkdir before it ever got near the already-correctly-owned subdirectory underneath. This is the third time this exact bug shape has shown up in this series: bind9's named needed the same grant in part 11, Apache's www-data needed it again in part 12, and now MariaDB's mysql needed it a third time. Same root cause every time, a worker account that isn't in the group that owns its own parent directory, fixed the identical way each time, usermod -aG lesta mysql. Worth naming plainly rather than treating as a coincidence: every installer in this project provisions a new system worker identity, and every one of them has hit the same gap in the same shared parent directory's group membership. Nobody has fixed the pattern itself yet, only its latest instance.

Then the vendored agent binary turned out to be stale, exactly the bug that hit part 12's own Apache phase, never rebuilt since. database.tenant.v1 was entirely absent from the shipped binary, confirmed directly with strings rather than assumed, the same way part 12 taught this project to check. Rebuilt via the project's own build command, pinned checksum updated to match. Third occurrence of that one too.

the bug that only showed up the second time

Here's the one worth slowing down for.

CREATE USER IF NOT EXISTS does exactly what it says: if the user already exists, MariaDB does nothing. It's a no-op. The password that user already has, server-side, doesn't change. But the installer script that runs this statement also generates a fresh random password and rewrites the credentials file with it, unconditionally, on every single apply, whether or not the user actually needed creating.

Run the installer once, and everything lines up: a new random password gets generated, the user gets created with it, and the file gets written with the same value. Run it again, for any reason, a config drift check, an idempotent-reapply test, a provider admin re-running a stuck operation, and the illusion breaks. CREATE USER IF NOT EXISTS sees the user already exists and does nothing to its password. The script generates a brand new random value anyway and overwrites the file with it. From that moment on, the file says one password and the server enforces a different one, and nothing about that state looks wrong from the outside. The installer reports success. The file exists. It's just lying.

This is exactly the kind of bug this series keeps drawing a line under: it wouldn't fail on the run that introduces it. It would only fail the next time anything actually tried to use the stored credential, which could be minutes or months later, and by then the installer script that caused it would be long gone from anyone's attention. The idempotent-rerun CI step this project runs for every installer phase caught it immediately, because that's exactly what it exists to catch: not "does this work," but "does this still work the second time." The self-test failed with Access denied for user 'lesta_agent'@'localhost' (using password: YES), MySQL's own label for a connection attempt that supplied a password and got the wrong one back.

The fix flips the earlier lesson on its head in a way that's worth sitting with. Password rotation, covered above, has to use ALTER USER, never CREATE OR REPLACE USER, specifically to avoid silently wiping a tenant's grants. This installer-owned bootstrap account needed the opposite statement, CREATE OR REPLACE USER, because the very next line in the same script unconditionally re-grants this account's own fixed, unchanging privilege set on every run regardless. There's no independently-managed grant state here to protect, so the drop-and-recreate that would be dangerous for a tenant's database is exactly correct for an account whose entire privilege set gets rewritten from scratch every single apply anyway. Same underlying MariaDB mechanism, opposite correct answer, depending entirely on what else is true about the account it's touching. Copying the fix from one bug to the other would have been wrong in both directions.

what actually shipped

Final CI run, all four jobs green: the regular test suite, the installer job (now including a full real mariadb/install.sh apply, an idempotent re-apply, and a real capability test against the installer-produced instance), the combined-profile job, and the agent's own Go test suite. The tenant database vertical slice, relational foundation, quota, UI, real Go capability, real installer, is fully real end to end, on the first attempt at building all of it at once.

Six real problems surfaced along the way, and not one of them came from a design that guessed wrong about MariaDB itself. One was a gap in what got tested at all, caught before it could produce a symptom. Three were installer-mechanics bugs this project has now hit, in the same shape, on three separate services. And one was a bug that could only ever be caught by asking the same question twice: not "did this work," but "does it still work if you run it again." Go, systemd, and GitHub Actions all did exactly what they were supposed to here. The gaps were entirely in what this project asked them to check, and, once asked properly, they didn't miss a thing.

Next up: cron gets its own capability, and with it, a design for keeping a tenant's raw shell commands out of a crontab line entirely.