Changelog

The 1.68.1 bump changed package.json and not the literal in src/lib/app-version.ts, so citaim.com rendered v1.68.0 after a deploy that Vercel reported as Ready and aliased (measured 2026-09-30, x-vercel-cache: HIT on a page prerendered by the NEW build). Nothing guarded the pair. Now scripts/check-version-sync.mjs runs first in prebuild and fails the build when the two disagree. No application change beyond the number.

Next 16.3.6 (2026-09-22) fixed a critical remote code execution in the Node ImageResponse from next/og (GHSA-vcvr-r3jv-pc5j): Satori wrote text into its SVG without escaping, and the rasteriser behind it turned crafted markup into code execution. Affected range 16.2.0–16.3.5. This repo sat on a pinned 16.3.3 for eight days after the fix, and it is the one project in the fleet with the exact vulnerable shape: /api/og-image/ runs in the Node runtime and puts title, subtitle and brand from the query string straight into the renderer (measured 2026-09-30: GET /api/og-image/?title=test on citaim.com answers 200 image/png). /r/[token]/opengraph-image renders the audited target the visitor typed in — the same input one hop removed. Whether anyone probed the endpoint in that window is not known; the Vercel logs for 22–30.09 are the only place that can answer it.

• next and eslint-config-next 16.3.3 → 16.3.8. 16.3.8 (2026-09-30) also closes SSRF in Image Optimization (needs images.remotePatterns, which this project has none of), SSG/ISR cache poisoning (needs a root catch-all, none here), a dynamicParams bypass on metadata images under webpack (the only dynamic OG route here is dynamic on purpose), use cache leaks (no cacheComponents) and an unauthenticated MCP endpoint in next dev.
• src/lib/og-text.ts — ogText() drops < > & " ' \ and control characters, then clamps. Both OG routes use it instead of the plain clamp(). The upgrade removes the known hole; this is the second lock on the same door for the next bug in the same library. Tested in og-text.test.ts`.
• npm audit --audit-level=high in the prebuild flagged 16.3.3 as critical since 22.09, so every build of this repo would have failed until this bump — the gate did its job.

A read-only audit of the whole codebase (five areas reviewed in parallel, every High finding re-checked in the code, the database ones probed with the public key inside a read-only transaction) found no Critical issues, four High and ten Medium ones. This release fixes all of them in code. Three steps still happen outside the code and are listed at the end.

• A password session no longer counts as signed in. Citaim signs people in with e-mailed links, one-time codes, OAuth and passkeys, never with a password. The auth provider still accepted password sign-ups, and with address confirmation switched off such a sign-up came back with an address marked as confirmed. getUser() now reads the session's amr claims and treats a password session as signed out, so the address on it is never trusted. Pending agency invites are claimed only when the stored account has a confirmed address that matches.
• Three database functions are executable by the server alone. Postgres gives EXECUTE to public by default, and every role inherits it, so a revoke … from anon, authenticated that leaves out public revokes nothing. The new increment revokes from all three for prune_page_views, record_domain_check and purge_binned_domains, and clamps retention to at least 30 days. A new test parses every SQL file and fails when a security definer function is left executable by public. Retention now actually runs: the daily domain scan prunes page views older than 400 days.
• MCP counts tool calls, not HTTP requests. A JSON-RPC batch is capped at 10 messages. Each tools/call costs one unit of an API key's daily quota; without a key, each costs from a shared per-IP allowance of 5 a minute and 50 a day. initialize and tools/list stay free.
• The report e-mail form is protected like the free tools. It needs a Turnstile check, has shared-store limits per IP, and sends at most three reports a day to one address. The subject and body name only the host that was checked, not the full address that was typed. Product e-mails go only to addresses that confirmed through a button on a page linked from the report e-mail; opening the link changes nothing, so mail scanners cannot confirm on anyone's behalf. Leads saved before this release have not confirmed and get no product e-mails until they do.
• Monitor alerts need a confirmed address. A signed-in user's own address is confirmed already; any other address gets the same confirmation e-mail an anonymous user gets. Anonymous visitors can create three monitors a day.
• API key limits follow the owner's current plan, looked up at request time and cached for five minutes, instead of the plan the key was created under.
• Sign-in and management tokens stay out of analytics. Google Analytics no longer loads under /auth, /monitor or /r. First-party page views skip /auth and store /monitor/:token and /monitor/confirm/:token as patterns; rows already stored are rewritten the same way by the increment.
• Admin sessions are revocable. Each session is registered in the key-value store and checked on every request, so signing out ends it on the server and a cookie minted anywhere else is refused. Every admin session open before this release ends at deploy. With Cloudflare Access on, the token sign-in route answers 404. Admin passkey sign-in requires user verification.
• Admin writes refuse cross-site requests. Every mutating admin route goes through one check that rejects a foreign Origin or a Sec-Fetch-Site other than same-origin.
• The client IP comes only from requests that proved they came through Cloudflare. The proxy drops cf-connecting-ip from requests admitted any other way, and drops client-sent copies of its own internal headers on every request, so rate limits and the trap log see the connection's address.
• Smaller fixes. — /account/?next= goes through the shared safeNext, which refuses backslashes and whitespace. — The speed check refuses ports other than 80, 443, 8080 and 8443. — safeFetch enforces its total deadline inside a hop, so a server that trickles bytes cannot hold a request open. — Gemini and PageSpeed keys travel in the x-goog-api-key header instead of the URL. The PageSpeed key can now be stored in Admin → Secrets. — The trap logs cross-site subresource requests without recording a block. — Lead unsubscribe links show a button: GET changes nothing, POST unsubscribes, and one-click unsubscribe keeps working. — Unsubscribe and confirmation links are signed with LEADS_UNSUB_SECRET when it is set; links signed with the old key keep working. Health reports security.unsubscribe_secret_shared until it is set. — The public error beacon files everything under client:, caps its fields, and keeps only same-site paths. The browser now posts it straight to /api/v1/errors/ instead of the bare path, which answers 308. — Unexpected errors in reminders, pilot and start-paid answer with a fixed sentence and are logged, instead of returning the underlying message. — One cronAuthorised() replaces seven copies of the cron check, and a test fails when a route compares the secret itself. The Cloudflare watchdog worker answers 503 without its secret and compares it in constant time. — A preview deployment refuses to write sealed data in plaintext when KMS is off, instead of writing it silently. — npm audit --omit=dev --audit-level=high runs in CI and before every build.

Still to do outside the code: turn on "Confirm email" in the Supabase dashboard, apply 20260927120000_lock_security_definer_functions.sql to the production database, and set LEADS_UNSUB_SECRET.

• Subscriptions: statement_descriptor: "CITAIM.COM" on every Stripe Product. Stripe picks the statement line for a subscription charge from the Invoice, then the Product, then the account. Until now the catalogue sync sent only the name and metadata, so every Citaim charge carried the descriptor of the company's shared Stripe account, which names a different product. A customer who does not recognise a line disputes it. The descriptor is sent on create and on every publication's update, so products created before this release pick it up the next time Admin → Billing publishes.
• One-off pilot: payment_intent_data.statement_descriptor_suffix: "CITAIM.COM". For card payments Stripe no longer accepts a full descriptor per payment, only a suffix appended to the account's shortened prefix (PREFIX* CITAIM.COM, 22 characters at most). Ten characters fit after the longest prefix Stripe allows (10 + * + 10 = 22). That matters because the prefix set on the shared account has not been measured: one sibling project assumes NSIWO, another recorded FAKTURY.CO.UK as the account descriptor.
• Build-time guard (lib/billing/statement-descriptor.test.ts, now in prebuild). Card networks validate the string only on the first live payment, so a bad character would surface at a customer, not in the build. The test checks both constants against the network rules (Latin printable, none of < > \ ' " *, a letter in each part, 5-22 for the full descriptor, at most 10 for the suffix) and reads the call sites from the code: a Product write or a payment_intent_data without the descriptor fails the build. The checkout and catalogue-publication tests now assert the value Stripe receives. Measured: with the two source files reverted, three tests fail; with them, all 1,464 pass.

Not verified against a live account: the exact statement line depends on the prefix in the Stripe Dashboard. To measure, read GET /v1/account (settings.card_payments.statement_descriptor_prefix) and make one test-mode payment; the payment view in the Dashboard shows the computed line.

• One constant pulled server code into 28 client bundles. The browser needs the name of the header that carries a solved Turnstile token. Six client components and the free tools' shared fetch helper imported it from lib/security/turnstile.ts, which is server code: since 1.62.0 it imports the error recorder, and through it node:crypto, the error store, the key-value layer and the KMS envelope code. In 1.63.0 the helper reached the WebMCP component in the root layout, which put the chain on every page.
• The two bundlers failed in different ways, and neither said so. The local dev server (next dev --webpack) cannot resolve node: imports for a browser and answered 500 on every page, the homepage included. The production build uses Turbopack, the Next 16 default, which substitutes a browser copy of Node's crypto library instead. Measured on citaim.com on 21 September: a 424 KB script (128 KB gzipped) loaded by the homepage carried that library together with the KMS envelope code — its error messages, configuration checks and the names of the AWS variables. No key, ARN or other secret value was in it: the code reads them from an environment the browser does not have.
• The header name now lives in a module with no imports (lib/security/turnstile-header.ts). Client code imports it from there, and the server module re-exports it for the routes.
• A test walks the imports of all 99 client components and fails when one reaches a Node built-in or a module marked server-only. On 1.66.2 it lists the 28 components and both chains (through the error recorder, and through the error store, the key-value layer and the KMS envelope). Run on older commits it dates the problem: 0 components on 1.61.0, 6 on 1.62.0, 28 on 1.63.0. It skips type-only imports and does not follow server actions, whose modules never reach the browser.
• The changelog page, checked in a browser now that the dev server works: section headings show inline code as code instead of printing the backticks (four headings), and long inline code may wrap. One unbroken expression in 0.84.0 made the whole page 444 px wide on a 375 px phone; it now fits.
• Verified in a browser at desktop and phone width: / and /changelog/ answer 200 under npm run dev, with no server errors and no new console errors. How much smaller the production scripts become is not measured yet; the next deploy will show it.
• Test suite: 1,461 passed, 0 failed.

• /changelog promised the whole file and showed only its list items. The parser behind the page kept - items and dropped every other line without a trace: paragraphs, the file's one numbered list, its one quote and its one code block. Measured on 2026-09-21: 951 lines in 52 of 202 releases never reached the page. 13 releases, among them 1.66.0 and 1.65.1, showed as a bare heading, and 39 more were cut short (1.47.0 showed one bullet and lost 54 lines).
• Every line of a release now reaches the page, in the file's order: paragraphs, bulleted and numbered lists, quotes and code blocks.
• [Unreleased] is an entry of its own. The oldest block in the file has that heading and no date. The parser did not recognise it and showed its items as part of 0.14.0. It now appears separately, without a date, and is not counted as a release ("202 releases since 9 July 2026" is unchanged).
• A test reads the real CHANGELOG.md line by line and fails when any line of a release is missing from the page or shown under a different release, so a block the parser does not know fails a test instead of disappearing. Run against the previous parser it reports 955 missing lines and the same 13 bare releases. A second test renders the page itself from the file and fails on the previous code as well.
• Hardening: a link whose target contains " is no longer turned into a link, because the quote would end the href attribute. The file has no such link, so nothing on the page changes; it matters now that every paragraph goes through the same code, not only list items.
• Until this change the newly shown text existed only in the private repository. It was checked for keys, personal data and descriptions of weaknesses that are still open, and none was found.
• Verified by tests that render the page from the real file, not in a browser: the local dev server (npm run dev) currently answers 500 on every page, the homepage included, because a client component imports node:crypto. That predates this change and is not fixed here.
• Test suite: 1,455 passed, 0 failed.

1.66.0 ("the database wins over the environment") changed getSecret and nothing else. Measured on 2026-09-21, starting from a clean 1.66.0:

• A test still asserted the old order. "lets the environment variable win" stored sk-table, set sk-env and expected sk-env, so it had failed since 1.66.0 (1 of 10 in secrets.test.ts). Seven tests of the new order replace it. The database value wins when both places hold one. A mismatch is logged by the key's NAME and never by either value (every console channel is checked), and nothing is logged when the two are equal. The environment answers when the table has no row, when the database cannot be read, when the stored key cannot be decrypted (local dev, previews), and when the row is refused as another key's envelope. A held database value outlives a database or KMS outage even with a variable set. Six of the seven fail on the resolver from before 1.66.0. The seventh (no warning when the values are equal) cannot tell the two orders apart by design: it guards against a false alarm.
• A rejected key from the environment was not re-checked early. reportSecretRejected returned at once whenever the variable was set. That was right while the environment always won, and wrong once the table did. After a provider's 401, a replacement pasted into the panel waited out the one-minute cache, and a table key that also had a variable was never re-checked early at all. The early return is gone, and the 5-second throttle still bounds the reads. Two new tests fail on 1.66.0: one database read where two are expected, and the rotated-out key served where the new one is. Affected: sendEmail (Resend), plus reportProviderStatus for the AI Visibility Checker and the five tracking engines.
• The Web Bot Auth signing key kept the old order. rawJwk() returned WEB_BOT_AUTH_PRIVATE_JWK from the environment before asking getSecret, so a key saved in the panel never replaced one set in Vercel. It was the only sealable key the new order had not reached. It now asks getSecret first. The environment stays the only source in the tsx leaderboard script, where secrets.ts cannot be imported; measured with tsx, the import throws on the server-only guard and the key still comes from the variable. The new test (the signature verifies with the panel's key, not the variable's) fails on 1.66.0.
• The panel said the opposite of what the code does. — Admin → Encryption labelled every key that had a Vercel variable "In Vercel env (wins)", even when a database copy existed and was the one in use. It now reads "Sealed in database · the Vercel variable is only a fallback". The label moved to whereKept() in secrets-meta.ts, where a test pins all four cases. The panel was not opened in a browser for this release (it needs the admin token), so these texts are verified by that test and by type-checking, not by a screenshot. — "Copy keys from Vercel" ended with "until then Vercel still wins". It now says the database copies take over within a minute, with no redeploy, and that the variables are only a fallback. — The email card and the email test route sent the owner to Vercel for EMAIL_API_KEY. While a saved copy exists, a changed variable is only the outage fallback, so both now point to Admin → Encryption (EMAIL_FROM stays in Vercel).
• Documentation and comments that still described the environment first: runbooks/AWS-KMS.md (the resolver summary, and rollout step 4, "Until then the Vercel variable still wins"), OPERATIONS.md (verify the gating keys in Vercel), and comments in secrets.ts, secrets-meta.ts, provider-key.ts, web-bot-auth.ts and vitest.setup.ts.

Found and deliberately not changed: supabase/0010_kms_encryption.sql (and the total generated from it) still says "the environment variable wins, the table is second". It is a comment in an applied migration, and applied migrations are never edited. The text exists only in the file; the database has no COMMENT ON for the table.

• APP_VERSION (src/lib/app-version.ts) is a hand-written literal, and it was not bumped in 1.64.0, 1.65.0, 1.65.1 or 1.66.0. The footer, /changelog, /about, the admin navigation, the MCP server and /api/health all reported 1.63.2 (the citaim.com homepage, measured 2026-09-21), so the post-deploy version check could not tell which of those releases was live. app-version.test.ts catches exactly this and had failed since 1.64.0, but nothing runs the test suite before a deploy: the prebuild checks only the SQL total. The literal and the lock file's version (also left at 1.63.2) now say 1.66.1.

Test suite: 1,450 passed, 0 failed. On a clean checkout of 1.66.0, measured the same day: 3 of 1,440 failed (the stale secrets test and both version checks).

The database now wins over the environment when both hold the same secret (fleet rule, owner decision of 2026-09-20).

The old order made the panel a liar: an operator could paste a rotated key, read "saved", and change nothing at all — a variable set in Vercel months earlier went on answering every call, and nothing reported the conflict. The screen and the running code disagreed in silence.

Now the place where a human can act is the place that decides, and rotating a key stops requiring a deploy. The environment keeps what it is genuinely good at: local development, a fresh deployment whose table is still empty, and the minutes when the database or KMS is unreachable (a held value still wins over both). When both sides hold a value and they DIFFER, the mismatch is logged by name — never by value — instead of being resolved quietly.

Check after deploying: if a key exists in both places, the database copy is now the one in use. Make sure it is the current one.

The stored mysentry key for this project had never been used: the token simply had not been pasted into the admin. "Nobody finished the setup" and "everything is fine" are the same empty monitor list, so nothing ever said so.

MYSENTRY_INGEST_URL is read as a FALLBACK under the admin-pasted link — the admin still wins, because that is where anyone rotating a token will look, but an env var can be set at deploy time, before someone remembers the screen.

Measured against the mysentry database on 20 September: vercel.json schedules seven crons, CRON_JOBS held six, and mysentry.ts hand-listed four. So domains-scan (daily), reminders (daily) and digest (weekly) had no monitor outside this deployment at all — and a missing monitor is indistinguishable from a healthy one: no tile, no alert, nothing to read.

• One list, derived. CRON_MONITORS is now computed from CRON_JOBS plus the watchdog, instead of being a second hand-kept copy. Two lists of the same thing always drift; the fix is to stop having two.
• One place that beats. recordCronRun() — already called by all seven handlers — now sends the outward heartbeat as well, and the four routes that called sendHeartbeat by hand no longer do. Two call sites for one fact is exactly how the other three got missed.
• Registration up front. registerMonitors() declares every scheduled job to mysentry from the watchdog, on every pass. A monitor is otherwise created by the FIRST beat, which a job that never starts never sends — the failure this module's own header describes (wrong slash form, renamed route, schedule never picked up).
• A refused report is logged. Callers ignore the result, correctly, so "403: this is a page token" used to look exactly like success. Fleet-wide that day: six mysentry projects with zero monitors against 23 scheduled jobs.
• Tests compare CRON_MONITORS with vercel.json directly, so the drift that caused this cannot come back silently.

Still required for any of this to be visible: this project's reporting token must be pasted in the admin (the stored key has never been used) and it must be a SERVER token — a page token cannot beat.

• Cron heartbeats now report their cadence (interval_minutes) instead of nothing at all. MySentry v1.73.00 derives the silence allowance from it, so the policy lives in one place and tuning it retunes every project at once.
• Until now this project sent no threshold, so every monitor kept MySentry's 26-hour default. For the two hourly crons that meant a full day of silence before anyone was told — and for the watchdog, whose whole job is noticing when other things stop, a day is most of the point of having it. Hourly monitors now get 60 minutes, daily ones 1440, which MySentry turns into allowances of 90 minutes and 26 hours respectively.
• A monitor whose name is not one of ours reports no cadence and keeps the default, deliberately: guessing a cadence nobody stated is worse than not having one.

Verified: tsc, eslint, 1,435 tests.

The security re-audit of 2026-09-18 reported that the consolidated SQL file was missing 0010_kms_encryption and 0011_admin_audit, so restoring from it would produce a database without KMS encryption and without the admin audit trail. Measured: the content had both, and npm run sql:verify executes the file twice on an embedded Postgres. What was wrong was the NAME. It carried the newest 14-digit migration id — 7 September — and could never move again, because 0012 sorts before 2026…. A name that cannot move is a name that will eventually be believed wrongly, and it was, by an auditor reading the repository exactly as it asks to be read.

The generator now names the file after the newest increment of every numbering series, so the name moves whenever either series grows, and scripts/check-total.mjs runs as prebuild: an increment added without re-running npm run sql:total fails npm run build before Next starts, locally and on Vercel. Verified by adding a throwaway 0012_*.sql and watching the build stop. The file's content is unchanged apart from its date line.

One more thing the audit did not ask about, written down rather than acted on: the file applies the 4-digit increments before the timestamped ones, which is not the order they were applied in (from git: 0001–0004 in July, the three timestamped files on 6–7 September, 0005–0011 between 9 and 17 September). Re-ordering a file that executes correctly today would be a change with no measured gain, so instead sql:verify now runs in CI — the day an increment starts depending on one the file applies later, CI says so.

safe-fetch.ts is the product's perimeter: every one of the free tools fetches an address a visitor typed. The re-audit's point was that the guard is right today and has to stay right after the next fifty tools. Its refusals are now regression-tested: a hostname that resolves to loopback, a DNS answer mixing a public record with a private one, an IPv4 address hidden inside IPv6 (both in the URL and from DNS), a redirect into the cloud metadata address, a redirect that changes scheme, a body over the cap, and a server that never answers.

Each case was proved to test something by switching the matching guard off and watching it go red — DNS validation, the IP-literal check, per-hop re-validation, the size cap, the request timeout — one at a time. The first attempt at the timeout case raced a real body against a 1 ms deadline and passed only when the machine was slow; it now stalls the server instead, which is deterministic.

.github/workflows/tests.yml runs the whole suite and sql:verify on every push and pull request. Tests that only run on a laptop stop running the day the laptop is busy.

poweredByHeader: false. Measured on production on 2026-09-18: every response carried x-powered-by: Next.js, which tells an attacker which advisories to read first and a visitor nothing at all. Measured again on a local production build after the change: the header is gone and the other security headers are untouched.

The scan added in 1.63.0 ran for the first time when that release reached GitHub (run 35252995411). It read all 175 commits in about four seconds and reported four findings, which is the gate working: the point of scanning the whole history is that something surfaces. Each finding was reviewed without printing its value — none of the four appears in the local environment file or anywhere else in the repository. Two are fixtures in tests whose database is mocked, one is an example inside a documentation comment (a counter name built from a hash prefix, not a credential), and one is the placeholder in the curl example on the API docs page. No secret is in this history.

.gitleaksignore now holds those four fingerprints, each with the reason written above it, and the procedure for the next failure: read the finding without printing the value, decide whether it is a real credential, and rotate a real one at its provider first — removing it from the history rewrites main and is the owner's decision, not a step taken in passing. Every entry names the commit that introduced the line, because the global form would also hide a real secret written on that line in any later commit. ci-secret-scan.test.ts enforces both rules and checks that a finding still fails the job, so the tempting fix for a red build — making the scan non-blocking — fails a test. Behaviour confirmed in the gitleaks 8.21.2 source: the git command reads .gitleaksignore from the directory it scans and skips comment lines.

package-lock.json was left at 1.62.0 by the 1.63.0 bump and now matches again.

The security review of 2026-09-17 read the code without testing production and listed five findings. Each was first confirmed in the current code, and each fix came with a test that failed on the code before the change and passes after it.

• A prompt could be deleted from another tenant's project. deletePromptAction checked that the caller could see the projectId they sent, then deleted the prompt by its id alone with the service-role key, which RLS does not restrain: any prompt whose UUID was known could be deleted through a project the caller could see. deletePrompt(sb, projectId, id) now filters on project_id and returns how many rows went. The other service-role writes behind the project pages were reviewed and were already scoped: prompts and competitors are added only after the project check, deleteCompetitor filters on the project (1.58.0), the plan actions on project_id, removeMember on the owner.
• The Stripe billing portal opened for anyone holding a checkout session_id — no sign-in, no tie to the account that paid, no expiry. The id sits in the address of /billing/success/, so in browser history and in what Google Analytics records; consent mode sends cookieless pings with the page address even before consent. /api/v1/billing/portal now behaves exactly like portal-account: the signed-in customer's own portal, 401 for anyone else, and the request body is not read. The unused ManageBillingButton that called the old route is gone, and Google Analytics no longer loads under /billing. Our own page-view beacon was checked as well: it stores the masked path only, and reads the address for campaign parameters and nothing else.
• A revoked API key could come back to life. authorizeAndMeter read the key record when a request started and wrote the whole record back at the end to count the call, so a revoke saved in between was overwritten by the stale copy. The count is now written with the existing KV compare-and-set against what was read; when something else wrote first, the record is read again and a revoked key is left alone. The daily quota was already an atomic counter.
• The free tools worked as an anonymous proxy carrying our verified-bot signature. 23 of the 26 tool routes counted requests only in the memory of the instance that served them, so the limit restarted on every instance, and none of them asked for Turnstile. Every tool route now goes through toolGate, counting in KV. An address gets a free allowance per hour (30, or 10 for the tools that fan out: compare, Core Web Vitals, subdomain search, security scan), then one Turnstile challenge that the page answers through the shared toolFetch (a solved challenge covers the free tools from that address for 30 minutes), then an hourly cap (120, or 40). The AI tools keep verifying every check. The security scanner also requested /.git/HEAD and /.env on any host anyone typed in. Those two requests now go out only for a domain the requester verified — the owner chose to build verification rather than drop the checks: an account-bound DNS TXT record citaim-verification=<token> on the registrable domain, trusted for 30 days, then looked up again. Without it the two checks show as not checked, with the reason, and the scanner offers verification. Through the public API they run only when the key's owner verified the domain; through MCP, never.
• Admin changes left no trail. A new admin_audit table (supabase/0011_admin_audit.sql) records every authenticated state-changing admin request — 20 handlers in 17 routes — with the actor (the Cloudflare Access e-mail; in token mode the admin session's method and id, which also ties a sign-in to what followed it), route, action, target, status and the ok the route reported, because some routes answer a failed save with 200. An entry never holds a submitted value (runbooks/ADMIN-AUDIT.md). In the same item: anonymous uptime monitors use double opt-in — an address given without an account gets one confirmation message (at most three a day per address) and no alerts until someone presses the confirm button, a button because mail scanners open links. Monitors created before this release keep their alerts, and an account's monitors are unchanged; monitor creation is now limited in KV rather than in one instance's memory. The security log keeps a keyed hash of the address and its /24 or /48 instead of the full IP, and rewrites older entries when the panel is opened; blocks keep the full address the edge enforces. A GitHub Actions workflow runs gitleaks over the full history on every push and pull request, redacting what it finds.

Two existing tests encoded the old behaviour and were extended rather than loosened: the gate test now asserts that exactly the three AI routes verify every check and that every other tool route is gated, and the admin session test proves by behaviour that only a signed, unexpired cookie yields a session.

The first attempt to deploy 1.63.0 failed: Vercel CLI tried to read the socket of the running CodeGraph daemon in .codegraph/ (Unknown system error -102). The CLI does not read .gitignore, and the file list of the production deployment then live (created 2026-09-17 05:05 UTC) showed what that meant: every CLI deploy from a working directory had uploaded what git ignores — .codegraph/codegraph.db (25 MB), .data/, the private /docs, and the .claude/worktrees copy of the repository. Deployment sources are visible to the Vercel team only, not to the public, but none of it belongs there. A new .vercelignore mirrors .gitignore and adds the local-only .claude/ and .unlazy/; src/lib/vercelignore.test.ts fails when a .gitignore rule is missing from it.

Deploy order: apply supabase/0011_admin_audit.sql first. It is safe either way — until the table exists every admin change reports an admin/audit error and still completes — but nothing is recorded in that window.

Verified: vitest 177 files / 1415 tests, TypeScript, lint and the production build pass; npm run sql:verify runs the regenerated total twice on PGlite. Not verified: nothing is deployed, so none of this has run in production; gitleaks is not installed locally, so its first CI run is also the first scan of the full history and may report old findings to rotate or allow-list.

A security review of 1.61.0 found four places where a missing secret, or a secret that was not what it looked like, quietly switched a protection off while the site kept working. Failing open hides the fault: nobody gets an error, so nobody looks. Each one now refuses on the production deployment and says why. "Production" means VERCEL_ENV=production (src/lib/security/deployment.ts), not NODE_ENV, because next build sets NODE_ENV=production on laptops and previews too. Development and preview behave as before.

What admins will notice: everyone signed in with the admin token or a passkey is signed out once and has to sign in again. Admins behind Cloudflare Access notice nothing.

• The admin session cookie no longer holds ADMIN_TOKEN. The token form copied the typed token into the cookie, and a passkey sign-in copied the environment value into it. So a cookie taken from a browser, a log or a backup was the permanent admin credential. The 8-hour Max-Age only told the browser when to drop it, the value also worked in the sign-in form, and the only way to revoke it was to rotate the token for everyone. The cookie is now a signed session, as1.<payload>.<HMAC-SHA256>, carrying iat, exp, a random session id and the sign-in method (src/lib/admin-auth.ts): — the key is derived from the token with a separate label, HMAC(ADMIN_TOKEN, "citaim:admin-session:v1"), so the key is never the token, and as1 is the key id for a later scheme; — the server checks the expiry (still 8 hours) and compares signatures in constant time; — rotating ADMIN_TOKEN ends every session; — the cookie is now __Host-admin_session (httpOnly, Secure, SameSite=Strict, Path=/), so a sibling subdomain cannot plant one. Path stays / because the page-view beacon reads the cookie to leave the operator out of the traffic numbers, and the __Host- prefix requires it. Sign-in (token and passkey) and sign-out also expire the old admin_session cookie, so the raw token stops sitting in browsers. The old cookie is never read again, which is why existing sessions end. Cloudflare Access is still checked first and does not use the cookie.
• Turnstile no longer waves requests through in production. Before, a missing TURNSTILE_SECRET_KEY turned verification off, and an unreachable or failing Cloudflare let every request through. In production (src/lib/security/turnstile.ts): — Turnstile is on whenever the public site key is set. The site key makes the widget render. A widget with no secret behind it used to be skipped without a word; it now refuses with unavailable:secret-missing. — A siteverify network error or non-2xx answer refuses too (unavailable:siteverify-…). — These refusals are our fault, not the visitor's. The free tools, inbox test, reminders and admin sign-in answer 503 "Verification is temporarily unavailable. Please try again in a few minutes." instead of blaming the visitor, and each reason is written to Admin → Errors at most once every 5 minutes. — A secret with no site key stays off. The browser cannot get a token without a widget, so enforcing would refuse every visitor over a setting only the operator can fix. The health report flags it instead. Measured before the change: production's CSP allows frame-src https://challenges.cloudflare.com, which is emitted only when the site key is set, and the 1.46.1 production log showed turnstile: "missing-token", which is only possible when both keys are set. So production already enforced, and nothing changes for visitors there.
• The origin gate refuses in production when ORIGIN_SHARED_SECRET is missing. An unset secret, or one under 32 characters (discarded as a typo), used to turn the gate off: every direct request to the Vercel URL got through, and cf-connecting-ip stopped being trusted without any warning. In production that state now answers 403 (src/lib/security/origin-auth.ts), logs once per instance from the proxy, and raises security.origin_secret_missing (down) in the health report. The exemptions are unchanged: the seven Vercel Cron paths, the e-mail worker's /api/v1/inbox/ingest, and the Cloudflare Access door on admin paths. Tests pin every exemption with the trailing slash the scheduler uses. Measured on 2026-09-16 without the header: https://crawlsonar.vercel.app/ → 403 Forbidden (the gate is active, so production's secret is set and at least 32 characters long) and /api/v1/cron/monitor/ → 401 (exempt, refused by its own CRON_SECRET). So the deployed configuration is not affected.
• Unsubscribe links can no longer be forged with the key in the repository. With neither LEADS_UNSUB_SECRET nor CRON_SECRET set, links were signed with the fixed string citaim-dev-unsubscribe, so anyone could unsubscribe any address. In production that state now issues and accepts no link (src/lib/leads/unsubscribe.ts). Lead nurture and the waitlist announcement check this before claiming a send, so no e-mail goes out without a working link and no one's step is used up. Admin → Waitlist shows "Not sent: unsubscribe secret not configured". The health report raises security.unsubscribe_secret_missing. The CRON_SECRET fallback stays (documented in OPERATIONS.md, and delivered links may use it), and the key derivation is byte-for-byte the same when a variable is set, which a test checks against the old formula, so delivered links still work. An empty variable now counts as unset.
• Health: new problem keys security.origin_secret_missing, security.turnstile_misconfigured and security.unsubscribe_secret_missing (production only, documented in runbooks/HEALTH-API.md). .env.example, runbooks/PASSKEYS.md and the Admin → Encryption notes describe the new behaviour.
• Reviewed and left alone on purpose: SCAN_LOG_SALT, which falls back to a fixed citaim-who-v1 in lib/who.ts and to CRON_SECRET or "" in the page-view and refusal hashes. These salts only make logs pseudonymous. Nothing is authorised by them, so changing them would only change the counting.

Decision 2026-09-16: everything sensitive moves to AWS KMS, the same way the rest of the fleet already works. Until now a copy of the database — a leaked backup, a misused service key — held every subscriber's tracked questions, brand and competitor names, the AI answers about them, their teammates' addresses, captured leads, subscriptions and the provider keys that spend money, all readable as they were written. This release makes that copy unreadable without the AWS role that only production holds.

• One envelope format across the fleet (v2.kms:, src/lib/kms-envelope.ts): a fresh AES-256 data key from KMS per write, AES-256-GCM, bound to {app: crawlsonar.com, environment, purpose, record, field} in both the KMS context and the AAD, with the key ARN in the header as the rotation hinge. An envelope copied to another row, key or column is refused before AWS is called. AES-256 is symmetric, so this layer needs no post-quantum change.
• Tracking tables (src/lib/client-crypto.ts, one map read by writers, readers and the migrator): brand and aliases, prompt text, competitor names, teammate e-mails, the competitors and citations of each answer, the answer excerpt, and the action plan's text. Fields written together share ONE envelope per row, because the citation gap reads up to 3 000 answers per project page and a Decrypt per field would multiply that. New rows get their id in the app, because the envelope is bound to it.
• What stays plain, and why — measured against every reader: domains (the subscriber's public site, filtered and indexed), ids, states, dates and numbers (filters, ORDER BY, joins, RLS), tp_actions.key (half of the upsert constraint), and run, snapshot and cost records (operational). Competitors are now sorted in the app, since the database cannot order a sealed name.
• Teammate addresses stay findable through a blind index: member_email_hmac, HMAC-SHA256 of the lower-cased address under BLIND_INDEX_KEY, carries the one-seat-per-address constraint and the invitation claim. The key lives in the environment, not the database, so a dump alone cannot test guessed addresses.
• Key-value store (src/lib/kv-sealing.ts): leads and sales notes, shared reports, monitors, reminders, inbox-test results, the waitlist, subscriptions, customers, pilot orders, the MySentry token and discovered or dismissed competitor names are sealed by key prefix, above both storage backends, so no caller changed. Compare-and-set still works: the layer compares what the caller read with the opened value and hands the database the raw envelope. Counters, markers, public caches, hashed API-key records, passkey public keys, and the capped security and error logs stay plain — the logs are written on attack-triggered paths, where a KMS call per event would turn a scan flood into a bill and an outage into lost telemetry.
• Provider keys can live sealed in the database (src/lib/secrets.ts): Anthropic, OpenAI, Perplexity, Gemini, SerpApi, Resend, the inbox-ingest secret and the Web Bot Auth signing key. The Vercel variable still wins. The row is re-read every 60 seconds and decrypted only when it changed; a provider's 401 or 403 forces a re-check; a held key keeps working through a database or KMS outage; a tampered envelope fails closed.
• The rule that decided which keys could move. A key moves only if every reader can be async and no synchronous gate decides "is this feature on?" from the environment — otherwise a key that exists only in the table makes the feature report "not configured" while the key is right there, and nothing fails loudly. So providerKey(), engineEnabled(), anyEngineEnabled(), aiVisibilityEnabled(), emailEnabled() and the Web Bot Auth checks became async, with every call site. Stripe keys stay in the environment: its synchronous gates run in server components in about twenty places, and billing is not switched on in production. The proxy's origin secret, Turnstile (fail-open), the admin token, cron and health tokens, the service key, the log salt, the unsubscribe secret and the blind index key stay too — each with its reason in Admin → Encryption.
• Modes. CLIENT_DATA_ENCRYPTION=kms seals every write; unset, writes stay plain. Reads accept both. An envelope that cannot be opened throws — it is never read as "no prompts" or "no subscription".
• Admin → Encryption: KMS self-test, copy keys from Vercel, store a key (write-only), and "Encrypt existing data", which seals tables and the key-value store in time-boxed batches, verifies each value by reading it back and opening it, and reports what failed or changed meanwhile. The count of what is still not encrypted comes from storage alone.
• Health and the provider readiness card count a key sealed in the database as configured.
• Rollout order (runbooks/AWS-KMS.md): apply supabase/0010_kms_encryption.sql first — additive, and safe under the old code — then set the AWS variables and BLIND_INDEX_KEY, deploy, test KMS, copy the keys and delete them from Vercel, then encrypt existing data. Backups taken before that keep the old values until their retention ends.
• The restore file's lower-casing of member addresses now skips sealed values: a re-run after sealing would otherwise have corrupted every envelope.

Reported the same day: "the system does not seem to track how many queries go to AI". It tracked one kind. The only code that wrote to ai_queries was the free AI-visibility tool, so Admin → Usage showed anonymous checks and nothing else. Tracking cycles (scheduled, run by hand, a new project's first cycle), prompt suggestions (1.56.2) and competitor discovery (1.58.0) all called providers and left no row there — the last two were written after the panel existed, and nothing noticed. The spend itself was always bounded: every call reserved and settled credit against the account's monthly cap and the platform's daily budget, and tracking runs kept their cost in tp_runs. It was invisible, not uncapped.

• Every provider call is recorded. The runner records each engine call with its trigger (tracking-cron, tracking-manual, tracking-bootstrap), the engine's provider and model, and whether it answered; prompt suggestions and competitor discovery record theirs the same way. Each call is one row: the runner records before settling credit, so a settle that throws cannot lose the row, and suggestions and discovery record a failure only when the provider did not answer, so a settle that throws afterwards cannot add a second one.
• Account queries are not domain checks. A new domainCheck flag keeps them out of the checked-domains counters, which exist to show what prospects look up. The free tool keeps its behaviour.
• A second card in Admin → Usage: "AI queries per day · from accounts", with sent and failed totals counted by the database, a breakdown by source and by provider, and the total across accounts and free tools. Cache hits are left out (they reach no provider); failures stay in, because a call that reached a provider and then failed can still be billed. The card says the record starts with this release, so the empty days before it do not read as a quiet month.
• A failed read no longer draws zeros. supabase-js reports a failed read in error instead of throwing, so the free-tool chart's catch never saw one and a broken read showed an empty month. A count reports even less: measured against production with the public key, a HEAD count on a missing table came back as status 204 with no error at all, only count: null. Both usage reads now treat an error, or a count with no number, as "could not be read".
• Paid plans see their monthly answers. Every plan has a monthly answer cap, enforced by the same runner, but only free accounts could see theirs. The account overview now shows "AI answers" used against the limit on paid plans; the "free cycle is complete" notice stays on the free plan.
• A guard against the next silent caller. ai-query-recording.test.ts scans src/ and fails when a file calls a provider entry point without recording the query, or sends a provider request outside the adapters. It has a positive control (the scan must find the four known callers) and a negative one (a synthetic caller that records nothing is flagged).

Not verified in production: the usage tab sits behind the admin sign-in. The change is covered by tests and a production build.

Asked the same day: why did the chart not show what the prompt checks and the citations showed? Two reasons, both in how the chart was built. It plots only full tracking cycles, and a check of a few prompts deliberately writes no point. And a cycle's snapshot stored share of voice and the share of answers naming the brand, but not citations, so an answer citing the brand's site could never appear on it. The single point on elsg.co.uk's chart came from the first cycle, run on prompts that had since been deleted.

• A citation line. Each cycle now records the share of its answers that cited the project's own site, per engine and overall, and the chart draws it as a third line ("Your site cited") with a card for the latest cycle. Snapshots from before this release have no value, and the line starts where the data does rather than drawing them as zero. Requires supabase/0009_snapshot_cited_rate.sql; until it is applied, snapshots are written and read without the column — as with 0007 — and the line does not appear.
• A check can put a point on the chart. When every active prompt fits in one check (three or fewer), "Check the prompt now and update the chart" checks them all and records a full cycle for today. Such a point rests on two samples per engine instead of the plan's usual three, and the schedule counts from it. Larger projects still check only their newest unanswered prompts, with no point.
• The chart says what it shows and when it will next move: points are full cycles, checks of a few prompts appear next to each prompt, and the next scheduled cycle is stated — computed from the project's cadence, its last cycle and the tracking cron's hour, with a test that fails if the cron schedule in vercel.json changes.

The runner ignored the result of writing a cycle's snapshot, so a failed write left no point and no log entry. Failures are now logged under tracking/snapshot.

Verified by unit tests — the migration and its place in the restore file (which also passed the idempotency check), recording and reading with and without the column, a series with gaps, what a check covers, and the next-cycle date against vercel.json — plus lint and build. Not yet verified in production.

Requested the same day: "Who wins the answers" was empty on both projects, because it counts only competitors a subscriber types in — while the answers were citing other firms, such as connaughtlaw.com and daniellecohenimmigration.com for ukimmigration.law's questions. With no tracked competitor, share of voice has nothing to be a share of.

• After each check or cycle — at most once every 20 hours per project — Citaim reads the answers from the last 30 days: the sites they cited (the brand's own left out) and excerpts of what they said. Claude is asked which of those are businesses competing for the same customers. Government bodies, regulators, forums, media, directories, review sites and charities are sources, not competitors, and are excluded.
• New competitors are added up to the plan's limit (Pro 3, Growth 5, Agency 10) and marked *auto* on the project page. Credit is reserved only once there is evidence to read.
• The subscriber stays in charge. Competitors can still be added by hand, and any that are not competitors can be removed: a removed name is remembered and never added back automatically.
• "Find competitors in AI answers" runs the same discovery at once (project owner, five times a day).

The remove action checked that the signed-in user could see the project named in the form, then deleted the competitor by id alone with the service key, which bypasses row-level security. A signed-in user could therefore remove another project's competitor by sending its id alongside a project of their own. Deletes are now scoped to the project.

Not yet verified in production: discovery on real answers. The names come from a model reading those answers, so an occasional wrong entry is to be expected — which is what removing, and remembering the removal, is for.

elsg.co.uk's project page said 0% while its PDF report listed elsg.co.uk among the sources AI cited. Both were right about different things. The page counted only answers that *name* the brand, and since 1.51.0 nothing on it showed citations of the brand's own site: the citation gap deliberately leaves the own site out, and it also counted an answer that cited the site without naming the firm as an answer "without you". For the prompt "deportacja z uk", one of the five latest answers cited elsg.co.uk and none of them wrote "ELSG", so the page showed "Mentioned in 0 of 5".

• Each prompt shows citations beside mentions: "Mentioned in 0 of 5 latest answers · your site cited in 1".
• Under the trend chart: how many analysed answers in the last 30 days cited the site, and a sentence saying the chart counts answers that name the brand.
• The citation gap no longer counts an answer that cites your site as one that left you out, so its pages come only from answers that neither named nor cited you.
• The PDF says what it measures: "is named in X% of answers" and "Named in answers", instead of "appears in".

Mentions still drive share of voice: a citation is not a recommendation by name, and the trend stays comparable with earlier cycles. Citation rates are not yet stored per cycle, so the chart has no citation line; that needs a schema change.

Verified by unit tests (gap counting with own-site citations by URL, subdomain and site; per-prompt summaries; plan and PDF tests unchanged), lint and build. Not yet verified in production.

Reported the same day: after a prompt was added by hand to a project that had already run a cycle, there was no way to see results for it. The only instant run, "Run the first cycle now", appeared only before any cycle existed, and the scheduled cycle runs once a day, only when the project's cadence is due — for a weekly project, up to a week later.

• "Check the new prompt now" appears under the prompt list once a cycle has run and some prompt has no answers yet (project owner only). It asks every engine on the plan about up to three such prompts, newest first, twice each — within about a minute — and returns to the page with the outcome.
• Each prompt shows its latest result: "Mentioned in 2 of 8 latest answers", or "Not checked yet". Until now the list showed only the question; the chart shows the whole project, so which questions the brand wins or loses could not be seen.

A check is deliberately not a cycle. It records the answers — they appear next to the prompt and feed the citation gap and the action plan — but it writes no trend snapshot, does not reset the cycle schedule, and sends no first-cycle e-mail or change alert. A snapshot of a few prompts would chart a subset as the project's share of voice, and a reset schedule would push the full cycle back a whole period. The "Run the first cycle now" sample was not reused for this: it takes the project's oldest prompts, so it would have skipped exactly the one just added.

Limits: ten checks per account per day, counted only when a check actually runs, within the plan's monthly answer allowance.

Not yet verified in production: a check on a real project. Signed-in project pages cannot be exercised locally without an account session.

For elsg.co.uk — a firm of Polish lawyers in the UK, with a homepage in Polish — "Suggest prompts with AI" proposed English questions about an unrelated industry. Measured: the model was given only the brand and the domain ("ELSG (website elsg.co.uk)"), so it guessed the business from a four-letter name. The homepage says plainly what ELSG does: lang="pl", the title "Polski prawnik w UK — prawo pracy i prawo imigracyjne", a Polish description, and LegalService structured data.

Suggestions now start from the website:

• The homepage is read first, the way the tools fetch client pages (SSRF-guarded, size-capped), and reduced to its language, title, description, main headings and structured-data types.
• The model works out what the business offers from that summary and writes every question in the site's language — Polish for elsg.co.uk.
• Nothing is guessed. When the website cannot be read (an error, a block, a page with no text before JavaScript runs), no credit is spent and the page says the site could not be read. When the site does not make clear what the business offers, the model answers UNKNOWN and the page says that instead.

Project creation uses the same suggestions, so a new project no longer starts with questions about the wrong business. Prompts already added from a wrong suggestion stay until they are removed.

Verified with unit tests — a fixture shaped like elsg.co.uk's homepage, the request the model receives, and each refusal — and by fetching elsg.co.uk through the app's own fetcher (200, lang pl). Not yet verified in production: the questions the model writes from that summary.

On a Growth project with a working Anthropic key, "Suggest prompts with AI" answered "No suggestions available (is the AI key configured?)". The key was not the problem, and nothing recorded what was. suggestPrompts turned every failure — no key, credit refused, a provider error, a reply its parser could not read — into an empty list, logged nothing, and the page guessed "the key" for all of them. Production logs held a single 200 response for the request and nothing else.

Measured before the change: the page and the suggestion route resolve the plan through the same activeSubscriptionForUser lookup, so both saw Growth; the model and headers are the ones the working AI Visibility check uses; and max_tokens (600) covers ten short questions. The causes that remain — credit refused, the provider failing, or an unreadable reply — cannot be told apart from outside, so this release makes each one say so:

• Every empty list carries a reason, and the page shows it: AI not switched on; paused because today's AI budget, this month's answers or a trial's credit is used up; the AI not answering; or a reply that could not be read as questions. The last two offer Try again. reserveCredit now reports which limit refused a reservation.
• Provider errors and unreadable replies are logged under tracking/suggest in the admin error log: the provider's own message, or the first 160 characters of the reply.
• The likeliest cause of an unreadable reply is gone. Suggestions were requested through the tracking call, whose system prompt asks the model to "respond naturally" — right for measuring answers, wrong for a list. The old parser took everything from the first [ to the last ], so a closing remark such as "(category: [law])" made the whole reply unreadable. Suggestions now have their own instructions (ten questions, one per line, nothing else), up to 800 tokens, and a parser that reads lines or a JSON array, ignores commentary and drops a last line cut off mid-question. Tracked answers are unchanged and still use the neutral prompt.

Not yet known: which of those causes produced the message on 2026-09-11. After this release the page names it.

Twenty-four places in the source pointed at /account without its trailing slash: /account?next=/projects in the header, homepage, pricing, alternatives page and mobile menu; /account?tab=… and /account in the account navigation; /account after checkout; the Stripe billing-portal return URLs; the sign-up link in nurture e-mails; and a percent-encoded next=%2Faccount%3Ftab%3Dapi on the API docs page. The fleet rule is that every internal link names its final URL, and now all of them do. The homepage's /tools/ai-visibility#objection got the same fix.

Measured on the dev server before the change: a request for /account?next=/projects answers 308. Links rendered by <Link> were already safe, because Next.js adds the slash when it renders them — none of the 42 links on the homepage and none of the 73 on /tools/ lacked one. The redirect was paid only by strings that never pass through <Link>: the nurture e-mail link, Stripe's return to the account page, and a next destination the account page follows with redirect(). router.push and server redirect() were not measured; both now receive the final form either way.

Why the guard missed all of them: src/lib/trailing-slash.test.ts checked href="…" attributes only when the path ran straight into the closing quote, so a link with a query string or a fragment was never checked. It now checks the path in front of ? and #. A new test fails on /account without its slash in every form found here — after a quote or backtick, after ${origin}, percent-encoded in a next value, or bare in a link, a router call or a redirect — and first proves its matcher catches known wrong samples and passes the final forms.

"Email me a sign-in link" used to ask Supabase to send the e-mail. Citaim now sends it: the server asks Supabase for a token with auth.admin.generateLink, which issues one and sends nothing, and e-mails a link to /auth/confirm/ through Resend, like every other Citaim e-mail.

For the person signing in:

• The link works in any browser. Supabase's link used the PKCE flow and could be exchanged only in the browser that requested it; opened on a phone or in a mail app's own browser, it failed. That is one of the two causes of "the sign-in link does not log the client in" (runbooks/MAGIC-LINK.md). The new link carries a token hash, which needs no cookie in any particular browser.
• A mail scanner cannot spend it. The confirm page (1.51.1) verifies only when Continue is pressed, and the e-mail now says so.
• The template is code (src/lib/auth/sign-in-link.ts), under test, rather than a dashboard setting that had never been changed.

What had to come with it:

• Limits of our own. Supabase rate-limits only the e-mails it sends. Without limits, this form would mail any inbox, as often as asked, from citaim.com. One e-mail per address per minute and five per hour, twenty per IP per hour, 200 in total per hour. Rate-limit keys hold hashes, never an address or an IP.
• Supabase's own e-mail as the fallback, whenever this path cannot be sure of doing better: e-mail or the service key not configured; the hourly total reached, so a flood cannot lock everyone out; an address the strict validator is unsure of; generateLink failing or issuing a type the confirm page cannot verify; or Resend failing.
• A breaker for a broken sender. A failed Resend send falls back at once and trips a ten-minute breaker held in KV. An unverified sender domain or a revoked key therefore cannot fail every sign-in the same way until someone notices.
• No code in the e-mail. generateLink also returns a 6-digit one-time code. Codes sent by e-mail must be at least 12 characters, and a link needs none.
• The type Supabase issued. An address Supabase has never seen gets a signup token, not a magiclink one; the link carries whichever was issued.

Verified by tests against mocked Supabase and Resend: sending, a new address, each fallback, flood limits, the breaker and its expiry, hashed keys, and an e-mail holding neither Supabase's code nor Supabase's own link. Not yet verified in production: a link requested in one browser signing in when opened in another.

Inviting a teammate wrote a row to tp_members and did nothing else. No e-mail was sent. The invited person found out only if they happened to sign in to Citaim with exactly that address, and nothing on the Team tab told the owner that nobody had been told.

• The invitation is e-mailed through Resend. It names who invited them, says what they will be able to do, and says which address to sign in with: an invitation is claimed by matching the e-mail address, so signing in with a different one shows nothing, silently. A reply reaches the person who invited (reply_to). The subject carries no address, because the logs record subjects.
• Resend on every pending invitation. It is also the only way invitations made before this release will ever reach anyone.
• A daily limit. Invitations go to people who never asked Citaim for anything. The seat cap bounds pending invitations but not remove-and-invite-again, so each owner may send 20 invitation e-mails a day. A double click sends one e-mail: the Resend idempotency key is per invitation, and per ten-minute window for a resend.
• One mailbox per invitation. The address must be a single deliverable mailbox — no comma, semicolon or angle bracket that could address a second person. An apostrophe is allowed: o'[email protected] is a real address.

Every refusal — no seats on the plan, seats full, an invalid address — redirected to /account?error=…, without tab=team. Those messages render only inside the Team tab, so the owner landed on the overview and read nothing. Redirects now go to /account/?tab=team at the trailing-slash address, and every outcome has a message there: sent, sent again, saved but not e-mailed, already joined, daily limit reached. The URL never carries the address.

The privacy policy now describes invited teammates' addresses and lists team invitations among the e-mails Resend delivers.

Not yet verified in production: an invitation e-mail arriving.

The Google AI Overviews engine (SerpApi, switched on by SERPAPI_KEY) was rewritten against SerpApi's documentation, checked on 2026-09-11, before the owner pays for a key. The previous adapter would have recorded misleading data in four ways:

• Deferred overviews were lost. Google sometimes loads the overview separately. SerpApi then returns only a page_token, which must be fetched straight away with engine=google_ai_overview. The adapter ignored it, so every such answer was a paid search recorded as a failed run.
• The cited pages' own text counted as Google's answer. The adapter collected every snippet in the overview, including references[].snippet — text from the source pages. A brand named on a source page counted as named by Google. The answer is now read only from text_blocks: paragraphs, headings, nested lists and expandable sections.
• It asked Google US. Searches now run with gl=uk, hl=en and google_domain=google.co.uk. AI Overviews appear only for English searches in a limited set of countries.
• "No overview" looked like an outage. A results page without an AI Overview is a real result. It is still kept out of share of voice, but it is recorded as NoAiOverview, distinct from NoAnswerReturned; an overview that SerpApi reports as failed is an engine error.

Cost is one or two SerpApi requests per answer. SerpApi does not document whether the follow-up uses a search credit, so it is charged as one — recorded cost can be too high, never too low — and the credit reservation rises to 0.06 USD, so a two-request answer never settles above what it reserved.

The key is now read through providerKey, like the AI providers' keys, so a pasted newline no longer breaks this engine alone; the guard test covers SERPAPI_KEY. runbooks/AI-PROVIDERS.md documents the engine.

Not verified with a real key: how often tracked UK questions get an AI Overview, and how often it is deferred.

The action plan (1.52.0) says what to do. This release shows what followed.

• Done actions on the trend chart. Every action marked done is a dated marker on the share-of-voice chart; hovering it shows what was done. The chart spaces cycles evenly — by cycle, not by calendar — so a marker sits between the two cycles whose dates bracket it, at the fraction of that gap its date represents. Placed by calendar across the whole axis, a marker could land on the wrong side of a cycle, and an action could appear to come after a rise that in fact preceded it. An action before the first cycle or after the latest one sits at that edge and says so; actions on the same spot share one marker.
• "After you acted". For each done action: share of voice across all engines, averaged over up to three cycles before it and three after — the cycles nearest the action. A cycle on the day of the action counts on neither side. With fewer than two cycles on either side there is no number, only how many cycles it is waiting for: a figure from a single run would be noise presented as a result. Beside the numbers, always: a change after an action is not proof that the action caused it.
• Log something you did. Work done outside the plan — a directory listing, a press mention, a rewritten page — can be logged with the date it was done, so it gets a marker and a comparison too. Past dates are allowed, because people log things afterwards. Future dates are not: a marker for something not yet done would draw a comparison around nothing. It is stored as a done action (kind = manual) and guarded like every plan write — the caller must see the project through their own RLS client before the service key writes.

Needs supabase/0008_action_plan.sql (1.52.0) applied in production.

The citation gap (1.51.0) shows where AI answers come from when they leave a brand out. This release turns it into a list a subscriber can work through, on the project page:

• Get mentioned on a page — a third-party page engines cite in answers without you, pages behind answers lost to a named competitor first.
• Publish your own page — when the cited page looks like a competitor's own site answering that question: the task is a page, not a listing.
• Answer a question on your site — a tracked question where, across at least two answers in 30 days, no engine mentioned the brand or cited its site. One answer proves nothing; two is the least that is not a coin flip.
• Check that crawlers can read the site — once per project, linking to Fix Pack pre-filled with the project's domain.

The plan is rebuilt after every tracking cycle and on demand. A rebuild never overrides a person's decisions. Each suggestion is a row with a stable key per project, and the rebuild upserts only its content — never state, done_at or binned_at — so a dismissed suggestion stays dismissed and a done one keeps the date it was done on, however often the same evidence comes back.

It is a full "our table": To do / Done / Dismissed / Pinned — the admin panel's Active / Archived / Bin / Safe in a subscriber's words — with 10/25/50 rows, multi-select, one search box over every column, and type-to-confirm on permanent delete. Delete exists only for dismissed items, which is what makes a pinned item impossible to delete. Dismissed items are purged 30 days after dismissal.

supabase/0008_action_plan.sql adds tp_actions. Signed-in users may only read it, and only when they can read its project. Every write goes through server actions that first confirm, through the caller's own RLS client, that the project is visible to them, and then scope every service-key query to that project, so an id belonging to another project matches nothing.

A trap in the restore file, avoided and measured. The read policy selects from tp_projects instead of calling the RLS helper private.tp_member_owners(). The consolidated restore file runs increments in name order, and this one sorts before the timestamped migration that moves that helper into private: a reference would have worked in production and broken disaster recovery on a fresh database. npm run sql:verify ran the regenerated total twice against a real engine.

• DataTable and StateTable moved from components/admin/ to components/table/. The customer panel uses them now, and a customer page importing from the admin folder is a dependency pointing the wrong way. StateTable takes labels, which rename tabs and verbs but cannot reorder or redefine the states.
• Fix Pack reads ?domain= and normalises input with the shared normalizeDomain instead of its own regex — one of the twenty hand-rolled normalisers found in 1.50.0.
• The checks that recognise a database behind the code — a missing column, now also a missing table — live together in schema-lag.ts.

Safe either way. Until 0008 is applied the project page says the plan appears once the database update is in, and a rebuild reports migration_pending rather than failing the tracking cycle.

Reported: "the sign-in link does not log the client in".

/auth/callback/ exchanged the link's code for a session inside a try and then redirected onward. auth-js does not throw when an exchange fails — it returns { error } — so the catch never ran. Every failed link was redirected to /projects without a session, bounced back to the sign-in page, and shown nothing, and nothing was logged anywhere. Measured on production: a made-up code answered 307 → /projects. The sign-in page had no way to show a link error at all.

The route now reads the returned error and Supabase's own error_code redirect parameter, sends the person back to /account/ with a reason, and records the reason — never the code — in the admin error log. The sign-in page says what happened and what to do:

• Opened in a different browser. @supabase/ssr uses the PKCE flow, which keeps the verifier in a cookie in the browser that requested the link, so the link cannot work anywhere else (pkce_code_verifier_not_found).
• Already used or expired — including when a mail security scanner opened the link first and spent the one-time token (otp_expired).
• Replaced by a newer link (bad_code_verifier).

Which of these hit the reporting client is not yet established; Supabase's auth logs will say. Both of the first two are structural rather than incidental.

/auth/confirm/ accepts a token hash and verifies it only when the person presses Continue. A token hash needs no browser-bound verifier, so any browser works, and a scanner that merely loads the page does not spend the token. It becomes the path real e-mails take once Supabase's Magic Link and Confirm signup templates point at it — a dashboard change documented, with the exact template text, in runbooks/MAGIC-LINK.md. Until then it is unused and harmless, and links already sitting in inboxes keep working.

The magic-link redirect now uses the final /auth/callback/ form, like the OAuth redirect: no 308 hop, and one exact address for Supabase's redirect allow-list.

Subscribers could see that a brand was missing from AI answers. Nothing told them where those answers came from instead — the one thing they cannot work out alone, and the thing that turns a chart into a list of places to be. The project page now lists the pages engines cited in the last 30 days in answers that did not mention the brand, pages from answers that named a tracked competitor first, with the questions, engines and rivals behind each one.

Pages, not sites. Until now every citation was reduced to its hostname before it reached the database. "Get onto legal500.com" is advice; "get onto the ranking the engine quoted for this question" is a task. tp_observations.source_urls (supabase/0007_citation_urls.sql) keeps the page, normalised in src/lib/tracking/citations.ts so that one page counts once. Fragments and tracking parameters go — OpenAI appends utm_source=openai to its citations, which would otherwise make one page a different page per engine — and so does a trailing slash on a non-root path. A bracket that belongs to an address (/wiki/Mercury_(planet)) survives; the one closing a markdown link does not. The site list is derived from the same URLs, so the two cannot disagree.

Each definition is a decision, and each is written into gap.ts:

• An answer left you out when neither the brand nor an alias is in it. Counting only answers that named a competitor would leave every free project — the free plan tracks no competitors — with an empty list forever.
• The brand's own site, subdomains included, is excluded: being cited without being named is a different problem with a different fix.
• Answers recorded before this release kept only the site. They are counted at site level and labelled, not dropped.
• A host carrying a competitor's name is flagged as likely theirs, because "get mentioned here" and "they have a page answering this" are different tasks.

The "Where AI cites you" card is gone. It listed every cited site rather than where the client was cited, and the new list is its actionable version.

The adapter collected sources by walking the whole overview object for URLs, which also picked up the overview thumbnail and, for a deferred overview, serpapi_link — a serpapi.com address stored as a page Google had quoted. Sources now come only from references[].link, the structure SerpApi documents at https://serpapi.com/ai-overview. No stored data was affected: production has no SERPAPI_KEY, so the engine has never run there.

Safe in either order, deliberately. Until 0007 is applied, the observation insert and the gap query both retry without the new column. Without that retry a deploy landing before its migration would have stopped tracking for every project, because a failed observation insert aborts the cycle. A warning is recorded once per process until the column exists; page-level detail starts with the first cycle after the migration.

AI-PROVIDERS.md, CRON-WATCHDOG.md, HEALTH-API.md, PASSKEYS.md and SOCIAL-LOGIN.md now live in runbooks/, and they are committed.

They had been written into the docs folder, which has been git-ignored in full since July (317cafa) so that generated documents — docx, pdf, business audits — never reach the repository. That rule is right for those files, and it silently swallowed the runbooks as well. Each file existed on disk, opened in the editor and passed the build; none of them was ever in git. Meanwhile earlier entries in this changelog (1.47.1 among them), several commit messages, and comments in the health endpoint, the cron watchdog, .env.example and the Cloudflare worker all sent readers to a path that did not exist in the repository. It was found only because git status failed to list a file that had just been edited.

The docs folder stays entirely private. Separating by location was chosen over separating by name pattern — tracking markdown there and excluding audits by name, as polbooks does — because a pattern would quietly commit the next business document saved as markdown without "audit" in its name, and a leaked business document is a worse failure than an unrecorded runbook.

The paths in the earlier entries below were corrected, because a changelog that points to a missing file is no use to the person reading it in six months. .gitignore now says where runbooks go, at the line someone would otherwise reach for.

runbooks.test.ts fails when code, configuration or this changelog references a runbook that does not exist, when anything points a reader to a markdown file in the private docs folder, or when that folder stops being ignored. The runbooks were scanned for secrets before being tracked for the first time: none found (two pattern matches were file paths).

Reported: the AI Visibility Checker returned 15 for ukimmigration.law and 85 for www.ukimmigration.law. Three independent things combined to make that possible:

1. Two spellings were two assessments. The checker normalised the scheme, the path and a trailing dot, but not www. — so the spellings were two cache keys and therefore two independent model calls.
2. Each call was a random draw. The request set no temperature, so it ran at the API default of 1.0, on what is in substance a yes/no judgement: "do you know this brand?"
3. The score has a cliff on that one boolean. A recognised brand keeps its confidence; an unrecognised one is capped at 15. For a borderline brand, a single flip of the flag is worth about 70 points — exactly the gap reported.

A hypothesis disproved by measurement: that the two spellings saw different evidence. curl -I with a shortened user agent got 403 on the www host, which would have meant one call had homepage context and the other had none. But that is not what the checker does. Running the checker's own fetch — GET, with the full user-agent string — on both spellings returned HTTP 200, the same title and the same 2 000 characters of text. The 403 was an artefact of the request method and of the site allow-listing the full UA string, which the comment on DEFAULT_USER_AGENT in safe-fetch.ts already warns about.

The fix, one line of reasoning per change:

• Both spellings resolve to one domain before the cache or the model is touched, through the shared normaliser below.
• The model call is pinned to temperature: 0. With a single sample, the honest answer is the model's most likely one, not a random one.
• The cache key moves to v2. v1 holds exactly the split this removes; keeping it would have served the apex spelling's random draw to both spellings for up to a day — consistent, and still a coin flip made at temperature 1.

Tracking is deliberately not pinned, and a test says so. It runs each prompt several times precisely to measure how much answers vary; a pinned temperature would make every run identical and turn share of voice into a frozen coin flip that looks like a trend.

What the score still is: the most likely answer of one model at one moment. Temperature 0 makes a repeat of the same input near-deterministic rather than guaranteed identical, and a model update can still move a borderline brand. That is a property of asking a model, not a bug, and the report already labels which model it asked.

MX, SPF and DMARC for ukimmigration.law exist only at the apex — measured on a public resolver; the www host has none of them. email-deliverability did not strip www., so checking www.ukimmigration.law would have told the owner their mail was unconfigured, about a domain publishing DMARC at p=quarantine. Nobody had reported it. After the fix, the tool's own code returns identical reports for both spellings — measured, not inferred. ct-logs searched %.www.<domain> and so, by construction, missed the subdomains it exists to find.

src/lib/domain-input.ts is now the one normaliser for tools that answer about a domain: AI Visibility, email deliverability, CT subdomains, DKIM, DNSSEC, HSTS, and usage recording. There were twenty written by hand, and two of them were exported under the same name, normalizeDomain, each doing the half of the job the other left out. Its rules:

• parsed with URL, so ports, userinfo and internationalised names come out right (bücher.de → xn--bcher-kva.de, the form DNS answers for);
• an email address yields its domain;
• a trailing dot is removed;
• www. is removed only when what remains is still a domain — www.com is a registered domain, and a blind strip would reject it;
• over 253 characters is rejected, not truncated, because a truncated domain is a different domain;
• other subdomains are kept, because they are genuinely different sites.

Deliberately unchanged, each for a reason: tools answering about one HTTP host (agent readiness, link previews, response headers), where www and the apex can genuinely differ; the blacklist, which classifies IP-or-domain first and whose DNSBL hostname semantics were not measured; and stored keys (monitors, watched domains, the API registry), where changing the spelling would orphan rows saved under the old one and needs a data migration first. admin/domains.ts still has its own function called normalizeDomain for that reason.

Checked and clean: tracking. Cited hosts and project domains are both www-stripped already, so share of voice was never split by spelling.

The daily spend chart shipped in 1.48.0 answers what the free tools cost. On its own that number is unreadable, and choosing it as the next thing to look at made the gap obvious: a low bar looks identical whether nobody came or everybody was refused at the door.

A request stopped by the gate never reaches a provider, so it appears nowhere in ai_queries. The free tools' limits — three AI Visibility checks per hour per address — were set by judgement before anyone had used them, and reviewing that number a month later would have meant guessing again with more confidence and no more information. Waiting for data would not have helped: the data was not being recorded.

supabase/0006_tool_refusals.sql records every refusal with the gate that caused it, kept apart because they mean opposite things and have different fixes: quota is our own limit being tight, verification is Turnstile failing real people, flood is abuse or a broken client.

The Usage chart now stacks turned-away on top of served, in the same bars, so the two are read together rather than in separate cards. Below it, the sentence the numbers exist for — because "twelve refusals" and "one person refused twelve times" are opposite findings:

The hourly cap turned away 12 checks from 11 different people. Spread across many visitors — the limit is refusing first attempts, not abuse.

Counting people without keeping them. No address and no user agent are stored. visitor is the same daily-salted hash the page-view analytics uses, so the same person counts once within a day and is unlinkable noise the next. That is what makes "how many different people hit the wall" answerable without holding anything worth stealing.

Refusals are awaited before the response is returned: the platform may freeze the function the moment it answers, and a floating write would be lost on exactly the busiest requests.

Two questions the Usage tab could not answer: how many AI queries the free tools make each day, and which domains people are checking.

Neither could be answered from what was being recorded. The only history was the analytics event list in KV, capped at 20 000 entries — correct for a funnel, which is a rolling window, and wrong for both of these. A daily chart would silently lose its left-hand side as traffic grew, and a domain the owner archived would fall out of the window and reappear as new, undoing the decision without telling anyone. A record with a state cannot live in a structure that forgets. Hence supabase/0005_usage_ai_queries.sql: ai_queries (one row per query) and checked_domains (one row per domain, carrying its state).

The chart counts what was spent, not what was asked. A repeat check inside the 24h cache window returns the same report without calling the provider, so it costs nothing; counting it would inflate the one number the owner reads to judge spend. Cache hits, failures and signed-in usage are shown beside the chart, where they answer different questions. Days are bucketed at local midnight through the same tested helper the Traffic tab uses — done in SQL it would draw the boundary in the database's timezone and put half the year's evening traffic on the wrong bar.

The domain list is a full "our table", because these rows accumulate forever and most are noise — a competitor checked once, a typo, somebody's test. Without a way to archive what has been seen and bin what is junk, a list stops being read after the first hundred rows, which is the same as not having it. src/components/admin/state-table.tsx adds the three things a plain table lacked, wrapping DataTable rather than copying it:

• Active / Archived / Bin / Safe as states of the row, never deletions; the Bin self-purges after 30 days and safe rows cannot be deleted at all.
• One search box over every value in the row — a person remembers a fragment of something, and which column it lived in is our problem.
• Type-to-confirm on the permanent delete only. Confirming everything teaches people to confirm without reading.

Counters are incremented in a single SQL statement. A read-modify-write from the application would lose counts whenever one domain is checked twice at once, which for a popular domain is ordinary rather than rare.

Honesty about the two ways a chart can be empty: when the store cannot be read the tab says so instead of drawing zero, because "nothing happened" and "we cannot see" look identical and mean opposite things.

• The AI Visibility route records each query, including whether the visitor was signed in. That flag is the presence of the session cookie, not a verified session — it is a statistic, not an authorisation, and the same predicate already stands in never-ban.ts. A forged cookie miscounts one row in a chart and grants nothing.

The logging added in 1.47.0 did its job on the first run and named the cause outright:

400 invalid_request_error
"This API key is not scoped to a workspace, so this request must include the
 anthropic-workspace-id header with the ID of the workspace to use."

Nothing about the key was wrong — valid, current, correctly pasted. It was created at the organisation level rather than inside a workspace, and Anthropic then requires the request to name one. The 1.47.0 guess that a trailing space was to blame was wrong; the trimming helper stays as worthwhile hardening, but it was not the cause.

• src/lib/anthropic.ts is now the only place that builds an Anthropic request. It sends anthropic-workspace-id when ANTHROPIC_WORKSPACE_ID is set and omits it otherwise, because a workspace-scoped key must not carry it. The headers had been assembled twice — the checker and the tracking engine — and both copies lacked the same header, so one missing line took out two features with two unrelated symptoms. A test forbids a third copy.
• The preferred fix needs no code at all: a key created inside a workspace. runbooks/AI-PROVIDERS.md records both routes, the whole variable table, and how to read a provider failure.

1.47.0 recorded the provider's status and body with console.error, which reaches Vercel's log — a live stream. Queried minutes later it was empty, and the fix that existed to make the next failure diagnosable was itself undiagnosable. The same fact now also goes to captureError, the durable error log the admin panel shows, which is where an owner actually looks.

tracking/claude.ts threw away the provider's body in exactly the way the checker did, keeping only the status. It now carries the message — which is what turned a day of guessing into one line of reading.

Third report on the free AI Visibility Checker in one afternoon, and the first two fixes were right about their own layer and blind to the one underneath. This entry records the whole path, because fixing it one layer at a time is what made it take an afternoon.

What the visitor saw: "Network error — please try again."

What actually happened: the request succeeded, the server answered, and the answer was HTTP 502 with a JSON body. Two independent defects turned that into the least informative sentence available.

Defect 1 — our own code returned 502. callClaude mapped "the AI provider answered with an error" to HTTP 502, which is also the status a serverless platform returns when the function itself dies. In Vercel's request log the two are identical: 502, message empty. Diagnosing this meant proving from the source that the 502 was ours — after first testing, and disproving by measurement, two hypotheses about a crashed process (an unhandled 'error' event on a truncated response stream, and a module-level throw). Provider failures now answer 503 with code: "provider", and core-web-vitals.ts loses the same false 502 in three places. 502 is now a status only the platform can produce, which is what makes it worth reading.

Defect 2 — the provider's answer was thrown away. The branch that handled a non-OK provider response discarded both the status and the body, so the only trace of a failed check anywhere in the system was that empty 502. It now logs provider_status and a truncated provider_body — safe, because the key travels in a header that is never echoed back, and asserted by a test.

Defect 3 — eighteen clients could not read an error. Every tool client did const data = await res.json() unguarded. res.json() throws on any body that is not JSON, and a non-JSON body is exactly what arrives when something fails upstream of our handler: a CDN error page, a platform 502, an empty response. So the one case where the visitor most needed a true sentence was the case that fell into catch and said "Network error". The tracked funnel event recorded status 0, so the analytics were wrong too. link-preview-client.tsx had already been fixed on its own, with a comment naming the CDN case, and the fix never spread.

Fixed across 48 parse sites in 40 files — public tools, account and admin panels alike — through one toolError() helper in src/lib/tools/read-response.ts, with a directory-scanning test so the nineteenth client cannot reintroduce it. A genuine network failure still says "Network error", correctly, because no response arrived.

Consistent with the evidence rather than proven by it: a trailing space is a legal HTTP header value, so the request is sent and the provider answers 401 — which is what we measured (a real response, not a crash). A trailing newline is worse: fetch throws Invalid header value before anything leaves, naming the header rather than the key. Neither is visible in a dashboard that renders the value as dots.

Nine call sites across four providers read process.env.*_API_KEY raw, so the same stray character failed four different ways. src/lib/provider-key.ts is now the only reader, it trims, and a whitespace-only key reads as *absent* — so the feature reports itself as not configured rather than broken. A test forbids reading those variables directly.

safeFetch listened for errors on the request but not on the response. An 'error' event with no listener is not a rejected promise: Node rethrows it outside every try, the process dies, and a serverless platform reports a bare 502 with no application log — the hardest failure there is, because nothing you wrote gets to speak. Reasoned, not reproduced: the truncation path was measured not to reach it, since destroy() with no argument emits 'close'.

• The AI Visibility route's catch-all is 500, not 422. 422 means "what you sent could not be processed", which blames the visitor for our fault and hides real breakage from anything watching 5xx. The other tool routes still use 422 for target failures, which is defensible there and deliberately untouched.

Reported straight after 1.46.1: the free checker answered "You've hit the hourly limit" on a first attempt. The verification fix had worked — the request now got past Turnstile and reached the rate limiter — but the limiter counted three requests that had already been made and refused.

The order was wrong, and it was wrong in three routes at once because the pattern had been copied. The limit ran before verification, so every request that failed Turnstile still consumed one of only three hourly checks. The auto-run bug fixed in 1.46.1 fired on every arrival from a link, silently spending the allowance; this session's own testing spent more of it, from the same address. The quota exists to cap what the AI providers can bill us, and a refused request bills nothing — charging it to the visitor punishes them for our failure, invisibly.

src/lib/security/tool-gate.ts now owns the order and the three routes call it:

• a flood guard in front, 60/hour, protecting the verification endpoint from something that cannot pass it;
• verification;
• the real quota behind it, counting only checks that were going to run.

The quota key carries a version segment. The counter now means "verified checks" rather than "requests", so a bucket filled under the old meaning must not be inherited — which also releases anyone still holding a bucket filled by the bug.

A test asserts the order inside the gate and that no tool route calls the rate limiter or Turnstile directly, because a route keeping its own copy of the order is a route that can get it wrong. Three did.

Reported from a real browser, and the production log named the cause exactly: turnstile: "missing-token". Not the AI key, not the Turnstile secret — the request carried no token at all.

A tool page can start itself from a ?url= link, and that is how a visitor arrives from the homepage: the homepage form navigates to /tools/ai-visibility?url=… and the page runs the check on arrival. The manual button is disabled until a Turnstile token exists — but the automatic run bypassed that guard entirely. It fired on mount, while the token was still null, so the request went out with no x-turnstile-token header and the server refused it. The one visitor who took the shortest path to the product met a failure message; anyone who typed the domain by hand did not.

The same pattern sat in three clients — AI Visibility, Blacklist and the Citability Checker added yesterday, which inherited the bug by being copied from the others. All three now return early until the token arrives and re-run when it does. The initial value is seeded from the link during render rather than pushed into state by the effect, and the call is deferred out of the effect's synchronous phase, so neither cascades a render.

src/components/tools/auto-run.test.ts scans the whole tools directory rather than asserting on three files, because the point of a copied bug is that the fourth copy has not been written yet.

Six findings from an external monetisation audit, each verified against the code or production before being acted on. One of the audit's claims did not survive that check and is corrected below.

Not every tracked engine searches the web. Verified in the request bodies: chatgpt sends a web_search tool, Perplexity uses Sonar's own retrieval and google_aio reads a live results page — while the Claude and Gemini calls send no search tool at all. A grounded engine tells you what a buyer is shown today; an ungrounded one tells you whether the model has heard of a brand. Five engines listed side by side implied five views of the same thing.

ENGINE_GROUNDED now records which is which, engineMeasurementLabel names them in the reader's words — *Web-grounded answer sample* against *Model knowledge check* — and the client report prints the distinction, but only when a cycle actually mixed the two kinds, because a note that always appears is a note nobody reads. A test fails if a search tool is ever added to Claude without the flag moving, and this is exactly the failure the site warns its own readers about under /learn/rag-grounding/.

canonical() appended a trailing slash to every entry, which is right except where Next declines to add one: a path whose last segment looks like a file. A leaderboard entry is a domain, so /leaderboard/renault.com/ answered 308 back to the slash-less form — every one of the 970 of them, invisibly, because a redirect is not an error. Measured on production; the rule is now "slash unless the last segment contains a dot".

Tool pages also come from the tool index instead of a hand-kept list, which held 49 of the 52 that exist — the Citability Checker among the missing three, a page shipped and never announced. Duplicates are dropped and /api/ endpoints are excluded: a sitemap is a list of pages to read, and /api/mcp answers POST.

/learn/structured-data/ sent readers to /tools/structured-data, which does not exist; the page is /tools/structured-data-checker. Nothing failed and nothing logged. A test now resolves every tool link in the knowledge base against the tool index.

It read as a prompt count beside a weekly cadence, hand-written, long after the free plan became a single baseline. There is a guard test for precisely this — and it missed, because it knew the digit 3 and not the spelled-out numeral. The copy now comes from freePlanLine, and the guard matches the shape of the promise (a count next to a cadence) rather than the digit, while still ignoring an essay about sample size that happens to mention three prompts.

www.citaim.com answered 200 with every page while the canonical tag named the apex: two hosts serving identical content, and a second identity for passkeys, cookies and the origin gate to reason about. It now 308s to the apex, before the origin gate, for the same reason the retired domain does.

A bracketed instruction asking the operator to insert the hosting and e-mail providers was live on the page an agency reads when evaluating a data processor. The processors are now named — Vercel (London region), Supabase, Cloudflare, Resend, Stripe, and the AI providers queried on the customer's behalf — rather than described as categories. A test fails on any remaining placeholder.

The audit reported the free-plan copy as inconsistent with the nine-answer limit. The generated line is correct: freePlanLine derives from the same limits the runner enforces and says "one complete baseline, no card". The inconsistency was one hand-written sentence on the homepage, fixed above. The distinction matters: the mechanism was working, one file had escaped it.

The Traffic panel counted views, visitors, pages, sources and countries — and never classified any of it. For a product whose entire proposition is being cited by AI, that meant ChatGPT and Perplexity referrals sat anonymously in a list of hosts while the one number the product exists to move went unreported.

src/lib/analytics/channels.ts classifies a stored source into AI · Search · Social · Campaigns · Other, and names the engine behind an AI referral. It is a single module read by the panel and by the tests, because two copies of the engine list give two different answers and nobody knows which is stale.

The limit is printed next to the number, not in a footnote: Google AI Overview referrals arrive as google.com and cannot be told apart from ordinary organic, and direct visits carry no referrer at all. Every AI figure here is a floor, never a total. Overstating that particular number would be overstating the product.

Two traps are covered by tests because getting either wrong is silent: gemini.google.com is an answer engine while www.google.com is search — collapsing them would report AI traffic that does not exist — and direct visits stay out of the referred denominator, since counting them inflates the total and shrinks every channel's share in the direction nobody checks.

Ordered to answer the questions in the order they are asked: what is happening now (three windows, each an exact database count rather than a slice of the sample — a breakdown that silently drops rows is a nuisance, a headline number that does is a lie), what shape it has (the daily chart, above the tables, because "noise or the start of something" comes first), where it came from (channels, with AI highlighted), then who and what.

The closing sentence now separates what is exact from what is sampled, instead of describing all of it as one thing.

The vault page Analityka pierwszostronna is rewritten as an obligatory format covering mechanics and presentation together, on the owner's decision. The reasoning is recorded there rather than implied: the same numbers arranged differently answer different questions, and an owner reading five panels should not have to switch between five ways of thinking. Three measurements that paid for the format are written down with it, including the PostgREST 1000-row cap that once made a twin's "Today" read 0 while the database held 169.

It was a flat list of ten links. Past roughly eight entries the eye stops reading a list and starts hunting, and there was nowhere for it to land.

• Five groups with headings — Growth, Money, Clients, Security & trust, System — ordered by the question an operator arrives with rather than the order the tabs happened to be built in. Groups collapse, and the collapse is remembered per browser. A group holding the current page never collapses: hiding where you are is disorienting in a way no saved preference justifies.
• Dashboard pinned above the groups, because it is the one entry nobody should have to look for.
• A search box inside the menu. Not ⌘K, which searches *data* — this filters the list already on screen. Enter opens the first hit, so three letters and a keystroke replace a hunt; Escape clears rather than closing, because closing loses the query just typed. An empty result says so instead of showing a blank column.
• The version number, next to the thing you are about to change.
• Icons are addressed by name, and only the name crosses the Server→Client boundary. Passing a render function through props from a server component throws at runtime and takes the whole panel down with a 500 that the compiler never sees; keeping the drawing in nav-icons.tsx makes that impossible to do by accident.

Two React mistakes were ported deliberately from the twin's comments, because both had already been paid for once: collapse state is read with useSyncExternalStore rather than copied into state by an effect (which paints the default layout and then repaints the stored one, so groups visibly pop open and shut on every page load), and getSnapshot caches its parse against the raw string (returning a freshly parsed object every render reads as "the store changed" and re-renders for ever).

One deliberate difference from the CMS it is modelled on: there, entries are routes and the active one is found by longest-prefix match. Here they are query-string tabs on one route, so the active entry is decided by id — simpler, and immune to the bug longest-match exists to solve. The mobile presentation also stays a horizontal scroller rather than becoming a drawer, per the fleet rule; groups use display: contents so their items flow into that single row, and only the desktop column honours a collapse.

Filter, Enter-to-first-hit, every link, group collapse and its persistence were exercised in a browser against a running panel. Two apparent failures during that check turned out to be the test's fault, not the code's: a second click into the search box appended to the existing value instead of replacing it, so the filter held revtraf, matched nothing, and Enter correctly did nothing.

The 1.37.0 notes said config redirects are evaluated *after* middleware on Vercel. Measuring the honeytrap on production today shows that is not true of the trailingSlash redirect: /wp-admin answers 308 to /wp-admin/ before the middleware runs, and only the slashed form reaches the trap. Paths carrying a file extension are exempt from that redirect, which is why /.env and /xmlrpc.php answer 410 on the first request.

What 1.37.0 actually established was narrower and still holds: the legacy-domain redirect works when it lives in the proxy. It was never demonstrated that redirects() runs later — that was inference presented as measurement, and the distinction is exactly the one this project keeps insisting on.

The consequence for the trap is small and stated rather than hidden: a scanner that follows redirects — nearly all do — still lands on 410 and is still recorded, at the cost of one hop. Escaping it entirely would mean skipTrailingSlashRedirect and re-implementing canonical URLs in middleware: a large amount of risk to buy back one hop. A test now pins that the matcher covers both forms, because matching only the slash-less one would mean the trap never fires at all.

Adding the DNS record behind Cloudflare's proxy before Vercel had issued a certificate left the origin with no TLS for that hostname, so Cloudflare's Full (strict) mode had nothing to hand-shake with. Vercel could not issue one either: proxied, it resolves to Cloudflare's addresses and Vercel reported the domain as "not configured".

Fixed by taking the record off the proxy long enough for the certificate to be issued, then putting it back. During the gap the host answered 403 from our own origin gate — a request that has not passed Cloudflare is refused, which is the gate working. admin.citaim.com now returns 302 to the Access login again.

Worth remembering as a sequence rather than a fact: certificate first, proxy second. The apex and www were configured that way round by accident and never showed the problem.

Customer accounts already had passkeys through Supabase. The admin panel had nothing: in token mode its entire authentication is one shared ADMIN_TOKEN typed into a password box — phishable, forwardable, identical for everyone who has ever held it, and impossible to revoke for one person without changing it for everybody. A passkey is worth more there than anywhere else in the product.

• No dependency and no CBOR parser. Registration normally means decoding the attestation object to extract the public key; the browser hands it over already in SPKI form via getPublicKey(), which WebCrypto imports directly. That removes the largest and most error-prone piece of a from-scratch implementation. Attestation is deliberately not verified — standard for passkeys, and verifying it would exclude password managers.
• Discoverable credentials (residentKey: "required"), so signing in sends no credential list up front and the site never tells an anonymous caller which passkeys exist.
• Verified on every assertion: ceremony type, challenge (constant-time), the origin exactly — never a suffix, because https://evil-citaim.com ends with citaim.com — the rpId hash binding the credential to this host, user presence, the signature over authenticatorData ‖ SHA-256(clientDataJSON), and a counter that advanced. ECDSA signatures are converted from DER to raw r‖s first; omitting that produces a verification that always fails, which reads as "the passkey is broken" and sends people looking in the wrong place.
• Four route rules, each asserted by a test because getting one wrong is silent — the feature still works, it just also works for the wrong person: registration (and even its challenge) requires an existing admin session and answers 404 otherwise; sign-in does not, and is rate-limited and behind Turnstile; challenges are single-use, browser-bound and consumed *before* verification so a failed attempt burns one too; and failure reasons go to the log, never to the client.
• Removing the last passkey does not disable the token sign-in. Locking yourself out of your own panel is a worse outcome than one extra way in.

The panel card says which mode is actually in force. Under Cloudflare Access, identity is Cloudflare's and a passkey here is a second factor and a fallback — not what is currently stopping anyone. Claiming otherwise would be reassurance rather than information.

SecureAccountNudge asks a new account to add a passkey, and it lived only on /account. Sign-up lands on /projects?welcome=1, and a customer reaches the account page rarely — often not for weeks. A nudge nobody sees is not a nudge. It now renders on /projects too, still disappearing on its own the moment a passkey exists so it cannot become the banner everyone learns to scroll past.

NEXT_PUBLIC_PASSKEYS_ENABLED is not set, so the customer-side buttons do not appear. That is the flag working as designed: it exists so the buttons cannot ship before Supabase Auth can honour them, because a button that errors on the first click does not get a second. Enable passkeys in Supabase first, then set the flag and redeploy. runbooks/PASSKEYS.md has both halves.

Single-use challenges need a key to stop existing the moment it is read. Expiring it in a second instead would leave a window exactly as wide as the replay it is meant to prevent.

Pointing the tool at citaim.com produced results that were wrong in ways a reader would have noticed before we did, which is the argument for doing it before shipping rather than after:

• An ISO date read as a number without a unit. The tool flagged its own "Last revised 2026-09-07" line. Matching from the day half only ever saw 26-09-07, which is not a date, so the window that recognises one had to be wide enough to contain the whole thing from either end.
• "At the moment they are deciding" counted as a vague time claim. It is a subordinate clause about the buyer, not an assertion of recency. A check that fires on correct writing teaches its reader to ignore it.
• A pronoun opening under a question heading reported as an orphan. The heading travels with the passage and supplies the antecedent: under "Is GEO just SEO?", "They overlap heavily" resolves perfectly well. FAQ-shaped content is the most citable shape there is, and a check that punished it would have been arguing against the tool's own advice.
• "The answer is not in the opening" on pages that answer in their first sentence. The opening window was measured from the raw start of the page, so a category label, an h1 and an "Also known as" line consumed the budget before any prose began. It now measures from the prose, and counts words rather than characters — a 58-character sentence that answered the question perfectly was failing a 60-character threshold.

Our own /learn/ pages went from 62 and 70 to 94 and 100 once the tool stopped being wrong about them.

Both are exactly what it exists to catch, so they are fixed rather than excused:

• The Citability Checker's own opening paragraph never used the word "citability", so nothing named the subject of the page a retriever would have quoted from.
• The homepage said "We ask from clean, neutral sessions." An engine cannot attribute a claim whose subject is a pronoun it cannot resolve; it now names Citaim.

The knowledge base goes from 29 entries to 39, and from 17 AI entries to 27. The new ten are all about measurement, because that is the part of this market where confident numbers are easiest to sell and hardest to check:

Designing the prompt set you track · Sampling and variance in AI answers · Choosing the competitor set · Which AI-visibility numbers are worth reporting · How the engines differ · Where AI gets its answers about your category · What does not work in AI visibility · Proving a change worked · Zero-click answers · Language and country in AI answers.

They say things that are inconvenient for a company selling tracking, and say them plainly: that four observations give a margin of error of roughly ±27 points, so a single-run figure is a coin flip; that no AI-visibility number can be attributed to revenue per deal; that AI referral traffic is a floor rather than a measure; that a precise AI Overview share cannot be measured from outside. A reader who checks one of those claims and finds it honest has a reason to believe the rest of the site.

Twenty-seven entries in a flat list is a wall, not a menu. They are now filed under five themes in the order somebody needs them: how answer engines work, being the source that gets quoted, measuring AI visibility, crawlers and policy, and what goes wrong. E-mail, DNS, security and performance keep their place below, with a sentence explaining why they are still here — none of them wins an AI answer, and any of them can lose you one.

An entry missing from every theme would vanish from the index while still existing as a page, so unthemedAiSlugs() exists and a test asserts it is empty.

The homepage carried a tool search box and six featured tools covering SSL, DNS and e-mail authentication, competing for attention with the thing the product is for. A page where everything is equally prominent has nothing prominent, and the visitor arrived asking whether AI mentions them.

Three featured tools now, all about being quoted — the Citability Checker first — under a heading about why AI names somebody else. The other 50 tools keep a plain sentence and a link. Nothing was deleted; it was demoted, which is the harder and more useful edit.

The product's own thesis is that engines retrieve *passages*, not pages, and quote a sentence they can attach to a claim. If that is true then citability is mostly a mechanical property of the prose — and a mechanical property can be measured deterministically, without asking any model.

That is the whole design, and it is a commercial argument as much as a technical one: the check costs no API call, returns instantly, and gives the same answer twice. The tracking product cannot do any of those things, because sampling a non-deterministic system is what it is for. The two halves fit: fix the structure here, then measure the effect there.

What it names, each with the offending sentence quoted verbatim:

• Passages that open with a pronoun. "This is why it matters" is meaningless once the paragraph above is removed — and removed is exactly how retrieval reads it. The single highest-value fix on most pages, and three words to make.
• Claims attributed to "we". An engine attributes a quote to a source; it cannot resolve "we" once the sentence leaves the page. The same sentence with the brand name in it is quotable.
• Numbers with no unit or date in the same sentence — a figure that cannot be stated safely is a figure taken from somebody else.
• Vague time words standing in for a date, when freshness is precisely what breaks ties between two otherwise adequate passages.
• Answers buried below the opening, headings that are puns rather than questions, sections too long to chunk cleanly, and missing dates or structured data.

The report ends with what it did not measure: structure is not authority. A well-shaped page making weak claims scores well here and still will not be quoted, and the tool says so rather than implying otherwise.

Running it against citaim.com immediately paid for itself:

• 7-day trial was reported as a number without a unit, because the check required whitespace between the figure and the noun. Hyphens now count.
• The pricing page — a grid of cards rather than prose — scored 0 on eighteen instances of three problems. Penalty is now charged per *kind* with a ceiling rather than per occurrence: a page with one bad habit repeated is not worse than a page with every bad habit once, and a zero tells the reader nothing about what to fix first.
• subject_absent fired on legitimate sub-sections of a long guide. A heading survives chunking and gives everything beneath it a topic, so it is now read as part of the passage. A noisy check discredits the ones that are right.

llms.txt invites AI agents to an MCP server and an agent-skills manifest; robots.txt forbade the whole of /api/, which told a compliant agent not to fetch the very endpoints being advertised. The site was contradicting itself in machine-readable form. Allow lines for /api/mcp, /api/markdown and /.well-known/ now sit beside the blanket rule, and being longer they win by the longest-match rule.

Also: the crawler policy moved into one module read by robots.txt, the trap matcher and the tests; training-only crawlers that return neither citation nor referral are now declined explicitly rather than by silence; and /llms-full.txt publishes the entire knowledge base as one markdown document, which this site was previously validating in other people's sites without offering itself.

The previous release shipped traps that deliberately blocked nothing, on the stated grounds that verifying a claimed crawler needs reverse DNS and the middleware runtime has no DNS. That reasoning was wrong. The edge runtime has no node:dns, but DNS-over-HTTPS is an ordinary fetch, and the fleet twin has been verifying crawlers this way in production for months. Conflating the two is exactly how a honeytrap ends up banning Bingbot — the failure the caution was meant to avoid.

The mechanism is now ported from that twin, with its reasoning:

• The verdict has three states, not two. yes, no, and unknown — and unknown never collapses into no. A dead resolver is not a disproved claim; blocking on it would let a DNS blip hand a lasting block to whichever search engine happened to pass through during the outage.
• Two vendor-documented methods. Reverse DNS with *forward confirmation* (the PTR must land in the vendor's domain and resolve back to the same address — one direction alone is forgeable), and published IP-range JSON for vendors that offer no rDNS. The range layer matches by address alone, so a crawler name nobody has heard of still passes; a User-Agent layer ages silently, and vendors do ship new names without announcement.
• Twelve crawler families, from Googlebot and Bingbot to OpenAI, Perplexity and Meta's link-preview fetchers. The list is one module, read by the trap and by the panel, because a panel with its own copy disagrees with the code that decides — silently.
• Exemptions in one entry point, applied when a block is imposed *and* when one is enforced. With the check only on the way in, "never banned" quietly means "from today", and an address blocked yesterday still cannot reach the panel where the block would be lifted. Cloudflare's whole 104.16.0.0/12 is exempt, including the WARP VPN exit that once put every WARP user behind a — Whitelist prefixes are capped at /24, because 104. contains that /12 while a representative-address check cannot see it.
• A targeted trap — a path only somebody who read our own documentation would try — gets its own event kind and a red block at the top of the panel, rather than one row among thirty WordPress scans.

Ordered to answer three questions in this order: is something happening now, is the defence alive, who exactly and when. A panel that opens with tables makes the reader scroll through data to reach the state, and the reader usually arrived because something pinged them.

• Three counters at the top, each an anchor into its section; a chart above the table, because "noise or the start of something" comes before "who and when".
• A failed read prints as a failed read, never as zero. A confident zero during a store outage is worse than an error message.
• An exemption does not silence the signal. A scan from a whitelisted address is still a scan, recorded with its own kind and colour — the whitelist is the first door anyone tries, and it must not be the one place nobody is watching.
• Protection is shown together with the state it depends on: a live test of the DoH resolver, the number of families verified by rDNS and the number of range sources, both counted from the code rather than transcribed, and the vendors that cannot be verified at all named out loud instead of left as a silent gap.
• The limit is stated where someone would otherwise assume otherwise: blocks are recorded and expire after 30 days, but they are not pushed to the Cloudflare edge, because that needs an API token this project does not hold. Evidence and a record of decisions, not an enforced ban.

admin.citaim.com now resolves through Cloudflare and is covered by the same Access application as the old host, so the audience tag in CF_ACCESS_AUD stays valid and no secret had to move. ADMIN_HOST points at it.

The investigation behind that: the crawlsonar.com zone has no Transform Rules at all, so nothing injects the origin-gate header there. The apex and www survive only because 1.37.0 redirects the retired domain before the gate; admin.crawlsonar.com passed Access, reached the app with no header, and was refused. Moving the panel to a zone whose rule works fixes it without reanimating a domain that is being retired.

admin.crawlsonar.com answered 403 after the shared secret was rotated: the zone's Transform Rule still injected the old value, so the origin gate refused a request that had already passed Cloudflare Access with a valid identity. The person able to repair that rule is the person the refusal locks out.

The gate now accepts a request on an admin path that carries Cloudflare Access's identity assertion, as a second door alongside the shared secret. The trade-off is stated in the code rather than implied: the middleware checks that the header is present, not that it is signed — verifying RS256 needs a JWKS fetch, and a network round trip does not belong in middleware. Anything reaching the origin this way meets src/lib/admin/guard.ts, which does verify the signature against Cloudflare's JWKS and answers 404 to anything else. The gain is that admin access survives a secret rotation; the cost is that admin routes are reachable-but-refused at the origin rather than refused one layer earlier. The signature was always the real control.

Tests pin both halves, including that the second door stays scoped to admin paths and that the guard still verifies the token.

/learn/ held fifteen entries, three of them about AI. It now holds twenty-nine, seventeen about AI, and the new material is the vocabulary a buyer uses while working out why their brand is missing from an AI answer:

Generative Engine Optimisation · Google AI Overviews & AI Mode · share of voice in AI answers · grounding and RAG · why one source gets cited · query fan-out · chunking and passage retrieval · embeddings and semantic search · entities and the knowledge graph · hallucinations about your brand · verifying AI crawlers · Content Signals (search / ai-input / ai-train) · AI referral traffic and attribution · prompt injection as a publishing problem.

Each is written twice, as the rest of the base is: once for someone who has never heard the term, once for someone who has to implement it. Several of them say plainly where measurement is weak — that AI Overview traffic is indistinguishable from ordinary organic clicks, that answers are non-deterministic so a single sample is an anecdote, and that referral counts are a floor rather than a measure. A product that sells measurement is the last one that should overstate it.

The argument these entries make is that structure decides citability, so the pages had to follow it.

• TechArticle with dateModified alongside the existing DefinedTerm and FAQPage, plus HowTo built from the fix steps, so a procedure is extractable as a procedure. One revision date for the whole base rather than one per entry: per-entry dates drift the moment someone edits without updating them, and a confidently wrong freshness signal is worse than a coarse honest one.
• speakable pointing at the answer-first block, which is now marked data-answer — one sentence that answers the question with nothing around it. A test enforces that no entry's one-liner opens with a pronoun, because that sentence has to survive being lifted out of the page.
• Markdown alternate for every entry. Accept: text/markdown on any /learn/<slug>/ URL returns the prose without markup or navigation chrome. The middleware passes the requested path in x-md-path, since the rewrite erases it from the URL.

The site sells an llms.txt checker and shipped a hand-written public/llms.txt that had already drifted: it described the knowledge base as thirteen terms while the base held twenty-nine. It is now generated at /llms.txt from the same three sources the pages are built from — the tool index, the sixteen guides, and the knowledge base — so that class of drift is gone rather than this instance of it.

The guide and the knowledge-base entry that both cover GEO now link to each other explicitly, with the entry defining the term and the guide walking through the work. Two pages on one topic either state their different jobs or compete for the same one.

GET /api/health/ used to be a public liveness stub: three fields, no authentication, and the application version printed for anyone who asked. It is now the fleet contract — status says whether something is wrong, summary says what in one line, and problems[] say why and how to fix it.

• The report is behind Authorization: Bearer $HEALTH_TOKEN, compared in constant time. A missing or wrong token answers 404, not 401: a 401 confirms the endpoint exists and invites a second attempt. With no token configured it answers 404 to everyone, so forgetting to set it cannot quietly publish the report. HEAD still answers 200 with no body for an uptime monitor that cannot send headers — that 200 means "the process answers", never "everything is well".
• The version moved behind the token. A version string is a map of which published vulnerabilities apply to this build.
• degraded stays on HTTP 200; only down returns 503. A monitor set to alarm on non-2xx should be alarming about outages. If a late weekly cron produced the same status code, the alarm would be muted inside a week — and the outage it exists for would be missed.
• Every check runs inside a 4-second guard. A guard that throws or times out produces a problem keyed <area>.failed rather than removing its section, because silence is never health.
• What is measured: a real round trip to the store with latency; all six scheduled jobs read from the cron registry (never_ran kept separate from dead — one is a configuration error that never self-heals, the other is a job that stopped); required secrets by presence only; the build the *edge* is serving, sampled through the canonical host with no cache-buster and read from a new data-app-version attribute on the footer; and HTTP probes of two cron paths, which must answer 401 and never 308.
• Analytics freshness is deliberately not a verdict. The fleet standard asks for a threshold derived from measured traffic; this site is too new for that, and a copied constant would be a lamp permanently lit. It returns when there is traffic to set it from.

runbooks/HEALTH-API.md carries the key table, the monitor configuration and the WAF skip rule for our own probes.

Twelve paths no legitimate visitor and no compliant crawler ever requests (abandoned WordPress surfaces, .env, .git/config, shell drops) now answer 410 Gone and record the hit.

• One list generates both the matcher and the Disallow: lines (src/lib/security/honeypots.ts). Two hand-maintained copies drift, and the drift is invisible until the day the thing caught is Googlebot.
• **Every user-agent group in robots.txt gets the block, not just *.** The AI-crawler group previously carried Allow: / and nothing else, which told exactly the crawlers this product most wants to keep that /admin, /api and the traps were fair game.
• Nothing is banned. Telling Googlebot from something claiming to be Googlebot needs reverse DNS with forward confirmation, and the middleware runtime has no DNS. Banning without that verification is the failure being guarded against — measured elsewhere in the fleet, 62% of one honeytrap's bans were Bingbot, Amazonbot and AhrefsBot. A hit is rewritten to a sink in the Node runtime, where the evidence is stored and where a reviewer can do the verification; blocking stays a human decision with that evidence in hand.
• 410 rather than 403 on purpose: if a legitimate crawler ever reaches a trap, 410 makes it drop the URL for good, while 403 invites it back tomorrow.

Against the fleet's security acceptance commands, on production: hybrid post-quantum key exchange negotiated (X25519MLKEM768), all six security headers present, all three cron paths answering 401 and never 308, and security.txt reachable. The honeytrap row was the only failure, and this release is that row.

crawlsonar.com answered 403 Forbidden to every visitor. The cause was the order of two mechanisms that are individually correct.

The origin gate refuses any request that does not carry the x-origin-auth header injected by Cloudflare, so nobody can reach the Vercel origin directly. Each zone injects that header with its own Transform Rule. When the shared secret was rotated during the rename, citaim.com got the new value and crawlsonar.com kept sending the old one — so the retired domain was refused by our own application, which is exactly what the gate is supposed to do to a stranger.

A domain that only forwards traffic does not need origin cloaking. The redirect therefore now runs in src/proxy.ts before the gate, and the old next.config.ts redirect is gone: config redirects are evaluated *after* middleware on Vercel, so it would never have executed. src/lib/legacy-redirect.test.ts pins the order, the 308 status (a POST to an old URL must not silently become a GET), the apex-and-www coverage, and the absence of a duplicate in the config.

Four things depend on the old domain continuing to answer, none of them recoverable once it stops: the URL inside the crawler's User-Agent that verified-bot programmes matched, the badge embed customers pasted onto their own sites, links in e-mails already delivered, and every indexed URL.

• NEXT_PUBLIC_SITE_URL is now https://citaim.com, so canonicals, the sitemap, llms.txt, e-mail links and the magic-link redirect all name the new domain.
• LEGACY_HOST=crawlsonar.com activates the redirect described above.

• The old domain is retired by redirecting, not by switching off: a permanent host-based redirect from crawlsonar.com and www.crawlsonar.com to the canonical host, path preserved. Four things depend on the old domain continuing to answer, and none of them survives deletion — the URL inside the crawler's User-Agent that verified-bot programmes matched, the badge embed customers pasted onto their own sites, links in e-mails already sitting in other people's inboxes, and every indexed URL.
• The redirect is gated on LEGACY_HOST so it cannot go live before the new domain answers. Enabling it early would make the old domain redirect into nothing, which is the same outage arriving by a different route. Set LEGACY_HOST=crawlsonar.com and deploy once citaim.com is verified.

• Renamed from Crawlsonar to Citaim across 153 files and 576 occurrences: brand name, citaim.com in every canonical URL, JSON-LD, sitemap, llms.txt, .well-known catalogue, e-mail copy and addresses, and the Cloudflare watchdog worker. Old CHANGELOG entries are left alone — they record what shipped under the old name, and rewriting them would falsify the history.
• The outbound crawler's User-Agent stays Crawlsonar/1.0, deliberately. Verified-bot programmes match the full string, so changing it drops that verification until every programme re-registers — weeks, run by other people — and sites that allow-listed the exact string would start refusing our scans with nothing in our logs to explain it. It changes in the same commit that re-registers the bot, not before. The URL inside it must keep resolving, which is one reason crawlsonar.com stays registered and redirecting rather than switched off. Both facts are now pinned by tests.
• ESLint no longer lints .claude/**. Parallel sessions get their own git worktree with its own .next output, and only the repo-root .next was ignored — so generated bundles nobody wrote were producing six errors and five warnings.

• NEXT_PUBLIC_SITE_URL is unset in production, so the code's fallback literal is what the site actually serves. This release is therefore not deployable until citaim.com answers: deploying it would repoint canonical URLs, the sitemap and every e-mail link at a domain that does not resolve yet.

A retry that never ran, and the product question hiding behind it.

• Entering a subdomain now returns the registration behind it. lookupWhois carried a retry commented "Retry once with the last two labels (helps when a subdomain was entered)". It sat inside a catch, so it ran only when the RDAP request *threw* — and the case it was written for does not throw. Measured on 1.33.0: Verisign answers HTTP 404 for blog.crawlsonar.com, fetchRdapAt maps 404 to null (correctly — that is how "not registered" is reported), and the lookup fell through to "No registration record found" in 387 ms without ever retrying, while the parent crawlsonar.com resolved fine. www.elsg.co.uk worked only because normalizeDomain strips a leading www.; no other prefix got that treatment. So the retry was dead for as long as it existed and its comment described behaviour that never happened — the same class of defect as the write-only consecutiveErrors counter fixed in 1.33.0. Not a regression from the registry-first change in that release: rdap.org answered the same subdomain with a 302 to the registry, which then 404'd, so the old path ended at the same null.
• The reminder widget and the reminder store now agree on the name. The preview lookup returned the registry's own spelling while the subscription was stored under the name that was typed. Those differ whenever a registry answers in a different form from the one queried — an IDN returns unicodeName where the lookup used punycode — and findByPair then misses, so the same person could collect two reminders for one domain. Both sides now use the name that was looked up.

• The answer is about a different name than the one typed, and now says so. Reporting example.com when somebody asked about blog.example.com is the useful thing to do — no registry holds a subdomain — but only if the substitution is visible; otherwise it is a confident answer to a question nobody asked. The report carries askedAbout alongside target, a plain sentence ("blog.example.com is a subdomain — this is the registration for example.com."), and a fact row so the substitution reaches every consumer of checks rather than only the one component that renders the notice. The result heading names the domain actually looked up. When nothing is registered, the error names both: "No registration record found for nothing-here.com (the registrable domain behind blog.nothing-here.com)".
• Resolving happens before the request, not after a failure, which also removes a wasted round trip: crawlsonar.com is asked directly rather than after a 374 ms 404 on the subdomain. A lookup for a name that is already registrable is unchanged — one request, no notice.
• A public suffix is refused rather than answered. Asking about co.uk returns 400 in 0 ms with an explanation, instead of a registration record. See below for why that matters.

• A bundled Public Suffix List decides where the registration boundary is. Counting labels cannot: blog.example.com sits under a two-label name and shop.example.co.uk under a three-label one, and nothing in the hostname says which. This is not a near miss — measured 2026-09-07, https://rdap.nominet.uk/uk/domain/co.uk answers HTTP 200 with a real registration record, as does org.uk. A "last two labels" rule would therefore have reported the registration of the entire .co.uk namespace as though the visitor owned it: a confident wrong answer, worse than the "not found" it replaced. IANA's RDAP bootstrap, already loaded in this file, cannot help — it knows TLDs, not that co.uk is one of them.
• The list is bundled (scripts/gen-public-suffix.mjs, ICANN section only, 7,396 rules) rather than fetched per request like the RDAP bootstrap, because the two have opposite failure modes: a missing bootstrap falls back to the rdap.org redirector and still answers correctly, whereas a missing suffix list would leave us unable to tell example.com from co.uk. The private section is deliberately excluded — github.io and uk.com describe who may create *sub*domains, but both are real registrations at their registries, which is what a WHOIS lookup is being asked about.
• Validated against the upstream vector file: 73 of 77 cases pass, and the four that differ are exactly the private-section uk.com cases the exclusion is meant to produce. Those vectors also caught a bug written blind: the list publishes internationalised rules in Unicode (公司.cn) while every hostname reaching us has been through new URL() and is punycode, so 446 of the 6,950 rules could never match. The generator now emits both forms.

A finding carried over from the fleet twin mysentry, where RDAP domain lookups had been failing for months unnoticed. Measured here before anything was changed: the bug does not reproduce in Crawlsonar, and the reason it does not is worth writing down. What did reproduce is the half of it that made the twin's outage invisible.

• RDAP lookups do not fail here for want of a User-Agent. Verified: rdap.org sits behind a WAF that answers HTTP 403 to any request carrying no User-Agent header — which is exactly what Node's built-in fetch sends by default, and how the twin broke. The same request with a User-Agent returns 200. Crawlsonar survives because its checks go through safeFetch, which supplies a default User-Agent one layer below every check, so whois.ts passing only an accept header is not the same code as the twin's. Nine domains across .com / .co.uk / .law / .pl / .app / .uk all resolved with correct expiry dates in 144–855 ms. This is a difference in architecture, not luck — but it is also undefended, which is the next item.

• A test pins the outbound User-Agent. Dropping that default would break every outbound check in the product while breaking neither the build nor the type checker — the exact failure mode that cost the twin months. outboundHeaders() is now a small exported function covered by safe-fetch.test.ts, which asserts the crawler identifies itself and, separately, that a caller passing its own headers (as the WHOIS check does) cannot silently displace it.
• A monitor whose check has stopped working now says so, and alerts once. consecutiveErrors had been incremented on every failed run since monitoring shipped and read by nothing — the comment in the runner claimed errors were "handled by the runner (consecutive-error tracking)", and that tracking did not exist. A check that errors keeps its last known state, so a monitor whose lookup fails forever went on showing a green "ok" badge, with "last checked" reading a minute ago. After three consecutive failures (roughly three hours at the hourly cron) a stalled alert goes out, once, on the crossing; the badge turns amber and reads "ok · not updating", keeping the last known state visible rather than hiding it.

• Domain lookups now ask the registry directly. IANA's bootstrap file (data.iana.org/rdap/dns.json, 1,200 TLDs, cached for a day) maps the TLD to its authoritative RDAP server; rdap.org stays as the fallback for TLDs the bootstrap omits, .io among them. This takes a third party — one that has demonstrably changed behaviour without notice, and fronts every registry we ask — off the critical path of every domain-expiry monitor, reminder refresh and WHOIS lookup. It is also faster, because rdap.org answers with a 302 that we then follow: measured 80–395 ms direct against 144–855 ms through the redirector, one round trip instead of two. Measured with the bootstrap URL deliberately broken: lookups still succeed through the fallback, and a stale copy of the map is preferred to no map at all.
• A failed RDAP lookup now names the layer and the status it failed at ("HTTP 403 (rdap.org)") instead of "The RDAP service returned an error". That string is what a monitor stores in its error field and shows as its latest result; a failure that does not say where it happened is a failure nobody diagnoses, which is precisely why the twin's went unexamined.

• The reminder cron no longer swallows a failed expiry refresh in silence. Before e-mailing someone, the cron re-checks the registry and falls back to the stored date if that fails — the right call for one run, since a slightly stale expiry beats skipping a reminder somebody is waiting on. But the failure was caught by a bare catch {} and left no trace anywhere, so an RDAP outage could run for months while every run reported success off dates nothing had rechecked. Failures are now counted into the run's response and captured, and the reminder still goes out.

Three findings from the conversion audit, all of them mine from the past two days, all verified by measurement before being fixed.

• Monthly answer caps were sized from the 4.33-week average instead of the fullest month. A month holding five weekly measurement dates needed 1,125 answers on Pro against a 1,000 cap and 7,500 on Growth against 6,600 — the fifth cycle was truncated silently, leaving a chart that just stops mid-month. Caps now come from MAX_CYCLES_PER_MONTH (five) with headroom for the suggestion calls that share the pool: Pro 1,200, Growth and Agency 7,800. Cost at full use, from the repo's own engine rates: $6.75/mo against Pro's $34, $33.75 against Growth's $119.
• The free plan was described by hand in five places, and four still promised "checked weekly" after 1.26.0 made it a single complete baseline. All five now render freePlanLine() from the entitlements, and a test fails if any file hand-writes that cadence again.
• The cookie banner claimed "analytics … only runs if you accept" while the first-party page-view counter added in 1.29.0 ran regardless. The counter is cookieless and stores no IP or user agent, so it is not what the banner gates — but the sentence on screen was still untrue. The copy now separates the two, matching what /privacy already said correctly.
• Absolute links in lifecycle and reminder e-mails used the slash-less form and earned a 308. An unsubscribe link that redirects is one more thing between a person and leaving.

• Running out of free AI-visibility checks is now treated as what it is — demand — rather than an error. The daily ceiling returns a distinct capacity code, the page answers with an offer instead of a red box ("today's pool is shared and it's gone; a free account has its own credit and measures weekly"), and the event is recorded so the owner sees the ceiling in the funnel rather than hearing about it from a stranger. The credit figure quoted comes from the admin catalogue, so it matches the pricing page.

• Every client fetch("/api/…") now uses the final path. Measured: a 308 does preserve the method and body, so these were never broken — each call simply paid an extra round trip since trailingSlash landed. A guard test covers fetch paths alongside links and cron paths.
• Corrected the reasoning recorded in 1.27.0 for the CSP report path. A redirected POST does not arrive empty; the real reason the slash matters there is that user agents do not follow redirects when delivering CSP reports at all — a redirected report is dropped. Right fix, wrong explanation, now measured.

• The post-deploy checklist still told you to curl /api/health without a trailing slash, which has answered 308 rather than JSON since trailingSlash: true landed in 1.27.0 — the documented proof that a release shipped no longer worked. The same applies to any external uptime check pointed at that URL: it passes only if it follows redirects.

• Sign in with Google and with Microsoft (Entra ID), listed in NEXT_PUBLIC_OAUTH_PROVIDERS and dormant until named — a button for a provider Supabase has not been configured with fails at the first click, so the list is the switch rather than a decoration. Both land on the existing /auth/callback/, which already exchanges the code, records the first sign-in and sends the welcome e-mail, so nothing provider-specific was added there. Entra is asked for the email scope explicitly; without it the profile arrives with no address and the account cannot be matched to a subscription.
• A "Secure your account" prompt on the dashboard, shown only while the account has no passkey and removed the moment one exists — a permanent banner is one people learn to look past. It enrols the passkey in place rather than pointing at a settings page, and says plainly why: with magic links alone, anyone who can read the user's inbox can sign in as them.
• runbooks/SOCIAL-LOGIN.md — the Google Cloud and Entra steps the owner must do, including the redirect URI being Supabase's rather than the site's, and the Entra client secret's expiry, which presents as "sign-in stopped working" with nothing in the app's own logs.

• The page_views migration failed with column "at" does not exist against a project that already had a page_views table. create table if not exists is silently a no-op when a table of that name exists in any shape, so the very next index ran against a foreign table and failed cryptically. The migration now adds each column explicitly, converging an existing table instead of assuming it owns the name, and applies the NOT NULL constraints only when the existing rows allow it — a bare set not null would abort the whole restore on legacy data.
• npm run sql:verify gained that scenario: it now runs the restore file against a clean database *and* against one already holding a differently shaped page_views with rows in it. The failure was reproduced in the harness before the fix was written, rather than reasoned about.

• The consolidated restore file was not idempotent, despite saying it was: running it against a database that already had the schema failed with function tp_member_owners() already exists in schema "private". alter function … set schema cannot be re-run, and the earlier increment had just re-created a copy of the function in public. Dropping that copy is not the answer either — the same increment re-creates the policies that depend on it. The stale private copy is now dropped first and the fresh public one moved in its place; policy dependencies follow the function's OID across a schema change, so nothing has to be rewritten.
• The fix sits in the increment itself rather than a later one, against the usual rule that applied migrations are never edited. The restore file is a single batch: a failure at that line aborts everything after it, so no subsequent increment can ever repair it. The end state is identical.
• npm run sql:verify runs the whole restore file twice against an isolated Postgres engine and asserts the RLS helper ends up only in private. The file has always claimed idempotency; nobody had executed it twice, so nothing contradicted the claim until it was needed.

• First-party page-view analytics: a page_views table, a beacon on path change, and Admin → Traffic. The owner sees their own traffic in their own database, on their own charts, permanently — not rented from a vendor who samples it and can withdraw it, and not dependent on an ad blocker letting a script through.
• The privacy contract is enforced by the table's shape: there is nowhere to put an IP address or a user agent, so neither can be stored by accident later. Uniqueness is a server-side hash with a daily salt — unique visitors today, irreversible noise tomorrow — and the browser stores nothing, so no consent banner gates it. Paths that can carry a secret (/i/:token, /unsubscribe/:token) are masked to their pattern before insert, because a saved token is a credential sitting in an analytics table and rows already written cannot be un-written.
• The operator does not appear in their own statistics: requests carrying the admin session cookie or coming from TRUSTED_IPS are dropped on arrival, bots are excluded by user agent, and the endpoint always answers 204 so a tracker can never break a page.
• Day buckets use local midnight rather than UTC, so an evening spike lands on the day it happened instead of shifting for half the year. Totals are counted in the database while breakdowns come from a capped sample, and the panel says which — a chart built from a slice that claims to be complete is worse than no chart.

• Self-reviving crons. Every job now records a completed run, and /api/v1/cron/watchdog/ walks an escalation ladder against each job's own interval: silence past 2.5× fires a quiet revival, past 3× revives and alarms, and a job with no completed run ever is alarmed immediately because that is a configuration error which never self-heals. One revival per job per interval, so a job crashing in a loop costs one attempt per cycle rather than one per pass.
• A second scheduler leg in cloudflare/cron-watchdog/: a free Cloudflare Worker that pokes the watchdog from outside Vercel, on an offset minute. The death of the whole platform scheduler is the one failure an internal watchdog cannot see, because it rides on what died.
• runbooks/CRON-WATCHDOG.md — what each job loses while it is dead (jobs reading a queue lose nothing permanent; jobs measuring the present lose that moment forever), the ladder and its reasoning, and the two-place CRON_SECRET rotation. Rotating only Vercel silently kills the backup leg, and a dead backup leg looks exactly like a healthy one until it is needed.
• A test asserting the schedule, the watched registry and the run-recording call sites against each other, so a cron added to vercel.json without a watcher fails the build instead of running unobserved.

• The existing heartbeats could not detect a cron that never started: a heartbeat-registered monitor does not exist until the first beat, so nothing was silent and nothing alarmed. That was the gap through which a sibling project lost seven crons from its first deploy.

• trailingSlash: true, the fleet standard, with every internal link, redirect and sitemap entry rewritten to the final form and the six cron paths in vercel.json given their slash. A path copied from another project is now always in the right form, which is the whole point of one convention: Vercel's scheduler calls the cron path exactly and does not follow redirects, so the wrong form earns a 308, the handler never runs, and the dashboard still reports the cron as executed.
• The origin gate now tolerates the trailing slash on its exact-match exemption list. Without this the change would have answered 403 to every cron from the edge — the same silent failure it was meant to prevent, arriving by a different door.
• The CSP report endpoint moved to /api/v1/csp-report/. A POST that earns a 308 arrives with an empty body, which would have quietly discarded the violation reports added in 1.23.0.

• A guard test covering the config, the schedule, the origin-gate exemption and internal links together, so the convention cannot drift back one file at a time.

• Annual billing is now the pre-selected option and states the saving as a number ("Save up to $598/yr", "Annual billing is 2 months free"), computed from the catalogue rather than typed into a sentence. The toggle's own comment had claimed annual was pre-selected since it was written; the state said monthly. Annual is cash up front and lower churn, and it only wins if it is the default a visitor opts out of.
• The free plan is one complete measurement cycle (3 prompts × 3 samples = 9 answers) instead of a trickle of single samples spread thinly over a month. A single sample is a number nobody should trust, and an allowance that runs out mid-month goes quiet without saying why. The account now shows how many answers are left and, when the cycle is done, says so with the upgrade decision at that exact moment.
• Plan copy will not promise "every week" to a plan whose allowance buys a fixed number of cycles; it says "one complete cycle" instead. The admin panel labels such a tier rather than rejecting it, and refuses only a cap too small to finish a single cycle — that one produces a half-measured, meaningless share of voice.

• Card-for-trial is a per-plan admin setting (default: required) rather than a constant, and the pricing card states it plainly — "7-day trial with $2.00 of AI credit · card required, cancel anytime". A card requirement discovered at the payment step costs more than the honest line. With the card waived, the Stripe subscription gets an explicit cancel-on-missing-payment end behaviour so a lapsed trial cannot hang.

• Plan limits — engines, samples per prompt, prompts, monthly answer cap — moved from a constant table into the admin catalogue. The seed values changed with them: paid tiers now take 3 samples per prompt per engine instead of 1, because a single non-deterministic draw reported as "share of voice" swings 20 → 40 % week to week on noise alone. Caps were raised to match the arithmetic (Pro 400 → 1,000, Growth/Agency 2,500/3,000 → 6,600); the admin panel refuses a cap below what a tier produces at full use, since that silently stops measurement mid-month.
• Engine mix by tier follows where buyers actually ask. Free samples ChatGPT rather than Claude — the free plan is the demo, and the demo must show a surface the buyer cares about. Pro gains Google AI Overviews, so no paid tier is missing the two surfaces that carry most real queries.
• Pricing bullets, the homepage cards, the sign-in copy and the welcome e-mail are now generated from the entitlements the code grants. They were string literals sitting next to the limits, so an admin edit would have changed what the product does while the page kept promising the old thing.

• Free-account AI credit as a single admin-owned number. Admin → Billing sets it; the pricing page and homepage advertise it, the customer sees their own balance in their account, and the engine spends it. Saving it refreshes the public pages by cache tag, so the promise on the page and the value in the panel cannot drift apart. It saves on its own rather than riding the price-publication flow: changing what a new free account gets must not require publishing prices to Stripe.
• The granted amount is written into the account's bucket on first contact and stays there. Raising the offer does not top up existing accounts and lowering it does not claw back credit somebody was already promised — a balance that moves without the holder doing anything is not a balance. Granting is atomic, so two concurrent first requests cannot grant twice, and an admin adjustment changes the grant without erasing spending history or producing a negative balance.

• A second CSP zone for the signed-in area (/account, /projects, /admin): per-request nonce with strict-dynamic, no 'unsafe-inline' and no analytics host. Those routes are already dynamically rendered, which is what nonces require, so the public pages keep the cacheable static policy and lose nothing. The pre-paint theme script is admitted by the sha256 of its own bytes rather than by reopening inline execution.
• CSP violation reporting at /api/v1/csp-report, feeding the existing Admin → Errors log as a csp kind. Both report shapes are accepted — the Reporting API array and the legacy csp-report object Safari and older Chromium still send — because supporting one loses about half the signal. Report bodies are unauthenticated and cross-origin, so every field is length-capped and only the document path is stored: a full URL can carry a one-time token or an e-mail address in its query string.

• Every JSON-LD block now goes through the escaping serializer instead of raw JSON.stringify. 38 of 63 injection sites bypassed it, and with 'unsafe-inline' in script-src a single </script> in any future value would have ended the element and started executing markup. A test fails if a raw one reappears.
• Google Analytics no longer loads in the signed-in area: the operator should not appear in their own statistics, and under the nonce policy it would have produced a blocked script and a violation report on every page view.

• The no-flash theme script moved out of next/script to a plain first-in-body <script>. Under a nonce policy next/script server-renders nonce="" while the client expects none, so every signed-in page hydrated with a mismatch.

• Sign-in failed with "Failed to fetch" the moment accounts went live: the Content-Security-Policy connect-src list never contained the Supabase origin, so the browser blocked every auth call before it left the page. The policy now names the configured project's exact origin — not https://*.supabase.co, and nothing at all while auth is dormant. The CSP moved out of next.config.ts into src/lib/csp.ts and is covered by tests, because a missing connect-src entry breaks no build, no test and no render; it fails one runtime call, opaquely.

• npm run sql:total generates the consolidated database restore file from the numbered increments, with the generation date as an argument so a diff shows schema changes rather than a new timestamp. A test fails when an increment is added without regenerating the total.
• An "Enabling accounts" runbook in OPERATIONS.md: the exact schema, Supabase Auth, environment and redeploy steps that turn the dormant sign-in on, and the verification order for confirming it.

• APP_VERSION had drifted behind package.json, so the footer, /changelog, /about and /api/health all reported 1.21.0 while 1.22.0 was live. Because the post-deploy check reads /api/health, a stale literal makes a successful release look like a failed one — it did, immediately after this release went out. A test now pins the literal to package.json and to the newest changelog heading.
• The consolidated restore file was written by hand and had drifted: it was two migrations behind and omitted 0001_kv_store.sql altogether, even though the newest migration creates an RPC on kv_store. Restoring the database from that one file — the whole reason it exists — would have failed on a fresh project. It is now generated from every increment in application order.
• .env.example pointed at individual increments for first-time setup, which invited exactly the cherry-picking that produced the gap above; it now points at the consolidated file and the runbook.

• Admin price drafts and atomic publication of immutable Stripe price IDs, stable product identities, stale quote checks and per-account checkout deduplication. Plan changes use the existing subscription's Stripe confirmation flow.
• A one-time agency pilot with admin-set price, paid-order storage, customer brief, operator fulfilment queue and refund status.
• A lead sales pipeline in Admin → Users: stages, notes, offer and next-contact date with protection against concurrent edits.
• Atomic AI credit reservations across tracking, onboarding and suggestions; provider token/cost evidence, configurable model rates, daily dollar limits and trial measurements on days 0, 3 and 6.
• Account display of contracted billing and remaining trial credit, plus a separate invoice review and confirmation to start paid billing early.

• Catalogue outages no longer resurrect default prices. Trial credit is stored with the original subscription and applies to instant runs. Revenue uses the agreed subscription amount and explicitly excludes trials.
• Empty provider answers are unavailable measurements, and ambiguous provider failures retain their cost reservation. Trial reminders use an invoice preview and retry a stable, idempotent email payload.
• Marketing copy now matches weekly cadence and account-wide prompt limits. Success/cancel pages report only verified payment status.

• Requires the 20260906092006_revenue_atomic_billing.sql migration, test-mode publication and owner-provided API keys before live activation. See OPERATIONS.md. No production changes are made by the local test suite.

• Commercial limits now protect margin: prompts and monthly answers are shared across an account, plans are enforced again when work runs, paid tracking defaults to weekly single samples, oldest projects run first, and provider spending stops when durable metering is unavailable.
• Five honest provider adapters: OpenAI uses Responses API web search, Perplexity and Gemini keep structured citations, Gemini defaults to a supported configurable model, Google AI Overview parsing handles nested blocks, and every provider model is configurable without code changes. Provider failure is recorded as failure and no longer becomes a zero-visibility snapshot.
• Identity-bound billing: checkout requires a verified account, reuses its Stripe customer, records the auth user id and grants one trial per account history. AI entitlements ignore unrelated API/monitoring subscriptions; portal and success pages are tied to the same identity; webhook storage failure returns 5xx and subscription creation dates remain stable. Trials are excluded from MRR.
• Database write boundary: a new migration removes browser mutation rights for tracking data and results, fixes the member-email conflict target, and adds provider/evidence fields. Authenticated server actions perform validated writes with the service client.
• Sellable launch path: pricing matches the enforced limits, offers a scoped $499 agency pilot, checkout errors stay visible, the full report-lead list has report/email actions, and the repository includes a first-revenue playbook and launch gate.

• The lifecycle cron bypasses the origin gate, every production cron fails closed without CRON_SECRET, Agency members see the owner's entitlements, and Open Graph theme constants no longer invalidate the production route build.

• Added provider, product-isolation, immutable-creation-date and margin-limit regressions. Full test, lint, typecheck and production webpack build are required by GATES.md.

• OPERATIONS.md corrected after the first real deploy: git push does not trigger a Vercel build for this project (the GitHub integration is inactive — 14 pushed commits produced no deployment); the deploy command is npx vercel deploy --prod from a clean tree. Also documented that vercel ls (CLI 59.x) prints every deployment URL twice, so the cleanup list must be de-duplicated before cutting it — otherwise the current production URL lands in the removal set.

• 1.19.1 went to production this day: /api/health reports 1.19.1, the lifecycle cron answers 401 (present, guarded), key pages and llms.txt serve 200. Old deployments pruned to production + one rollback spare (11 removed).

• OPERATIONS.md (repo root — /docs/ is deliberately uncommitted in this repo) — deploy procedure, the production env-var table with what each gates, the after-every-deploy checklist (health version, the mysentry "Send test heartbeat" click that registers all three cron monitors, the 401-not-404 cron spot-check, deployment cleanup) and the list of owner tasks code cannot do (legal read of /terms, real testimonials, competitor re-verification, Stripe promo code, Supabase migrations/passkeys). Written so the go-live steps live in the repo instead of a chat transcript.

• Smoke-tested the production build (Next 16.3.3) locally: /, /pricing, /alternatives, /terms, /tools, /leaderboard, /api/health, /llms.txt all 200; MCP tools/list 200; llms.txt serves the new Product section.

• Dependencies patched to npm audit 0 vulnerabilities (was 8: 2 moderate, 6 high): Next.js 16.2.10 → 16.3.3 (brings sharp 0.35.4, closing the libvips CVEs bundled with Next's image optimizer) and the postcss override 8.5.16 → 8.5.26 (the 8.5.16 pin was itself a security fix in 0.93.0; its version has since been flagged). Full test suite and production build pass on the new versions.
• ESLint and tsc are now completely clean (before: 18 errors + 9 warnings, 2 test-file type errors). Fixed across ~20 files: unescaped quotes/apostrophes in JSX; an <a href="/leaderboard"> swapped for <Link>; seven components deferring their initial state-setting effect a tick (the set-state-in-effect rule); five tool clients whose useCallback read the input state instead of its own argument (fixing the exhaustive-deps warnings without behaviour change — the argument is what actually ran); four unused imports/vars; the two long-standing test-file type errors (fixture cast, union narrowing).
• Output tracing fully scoped again under Next 16.3.3: the new tracer pulls the whole project into any route whose store calls fs.readdir on an env-overridable path — without the build warning 16.2.10 printed. Fixed in the monitor store, the reminders store and the admin dashboard's report counter (literal .data/... paths; the reminders test now stubs process.cwd() like the testimonials test). Verified: 0 routes above 100 traced src files, down from 9 × 500.
• llms.txt gains a Product section (pricing/tracking, agencies, alternatives, leaderboard, learn, API & MCP, terms) so AI readers see the product, not only the free tools.
• .env.example documents the last undocumented knobs (NEXT_PUBLIC_GA_ID, INBOX_INGEST_SECRET, SCAN_LOG_SALT, PDF_FONTS_DISABLED, the test-only store-dir overrides) and drops the now-dead MONITOR_STORE_DIR/REMINDER_STORE_DIR entries.

• Measured: npm audit 0 vulnerabilities; npx eslint src 0 problems; npx tsc --noEmit 0 errors; 593 tests pass; production build clean; nft-trace scan shows 0 routes >100 src files.

• Competitor price anchor verified at the source (fetched from otterly.ai/pricing on 23 August 2026): Otterly Standard is $189/month ($160/month billed annually) for 100 tracked prompts on four engines, with more engines as add-ons. The pricing page and /alternatives now quote that figure with the check date instead of the earlier unsourced "$189–199 for 100 prompts" range. Peec's page publishes no figures in plain text; /alternatives reports what third-party reviews say (Starter ≈ $95/50 prompts, Pro ≈ $245/150 prompts, three engines included) and labels it as such. Re-check both before any future copy change — prices move.
• Terms acceptance where it matters: the sign-in form states that continuing means agreeing to the Terms of Service and Privacy Policy (links), and the WebAPI JSON-LD termsOfService now points at /terms rather than /privacy.
• Every revenue-relevant cron is its own mysentry monitor: the daily tracking cron (<base>-tracking) and the daily lifecycle cron (<base>-lifecycle) now send heartbeats with their error state and processed counts, alongside the hourly monitor cron. The admin "Send test heartbeat" button pings all three, which registers them on mysentry from one click — closing the gap where a heartbeat-registered monitor cannot notice a cron that never started. The admin card says to press it after any deploy that adds a cron.
• Testimonial supply: the weekly digest for paid accounts ends with a one-line request for a sentence we may quote (name and company) — the honest way to fill the testimonials module added in 1.17.0. Free-tier digests keep the upgrade line instead.

• tsc clean apart from the two pre-existing test-file errors, eslint clean on every touched file, full test run green, production build clean with no whole-project tracing.
• Production observed from outside on 23 August 2026: /api/health reports version 1.6.0 — every release from 1.7.0 to this one is unshipped; /api/v1/cron/lifecycle returns 404 there (not deployed), the pricing page shows plans (billing enabled). The Vercel CLI is not installed on this machine, so production environment variable names could not be listed.

• Legal review of /terms; real customer quotes (the digest now asks); production env values (Resend, Supabase public vars, engine keys) — verify in the Vercel dashboard; press "Send test heartbeat" in Admin → Dashboard after the deploy; a Stripe promotion code for WINBACK_PROMO_CODE if win-back should carry a discount. Deployment only on explicit request.

• Terms of Service at /terms (footer link, sitemap, WebPage + breadcrumb JSON-LD): operator facts as on /privacy, what the free tools and paid plans are, acceptable use, trials and automatic renewal, the 14-day refund the pricing FAQ already promises, content ownership, a plain statement that AI-visibility numbers are sampled observations and not guarantees, liability caps, termination, England & Wales law. Written to match what the product does today; needs a legal read before it is relied on — the checkout and sign-in flows do not yet reference it.
• /alternatives — the checklist for choosing an AI visibility tracker and where Crawlsonar stands. Eight questions to ask any vendor, a fact table read from the same plan definitions the app enforces (engines, samples per prompt, metrics, competitors, alerts, white-label, price per tracked prompt, day-one experience, free plan), the existing price anchor ("Otterly and Peec list roughly $189–199/month for 100 prompts, at the time of writing"), and an honest "where we are not the right fit" box (no rank tracking, no SSO/SAML, no social listening, no guarantee of being recommended). No competitor feature claims beyond the price anchor — those would need verification. Linked from the pricing page and the homepage pricing section; FAQ JSON-LD; ISR hourly via the cached catalogue.
• Testimonials module — src/lib/testimonials.ts (KV document testimonials:v1, file locally; cached public read with tag testimonials), admin API /api/v1/admin/testimonials (GET/POST/PATCH/DELETE, admin only, validated), an "Social proof" panel under Admin → Users to add, publish/hide and remove quotes, and a <Testimonials /> section on the homepage and pricing page that renders nothing until the first quote is published. No placeholder quotes anywhere: the module is the form for the real ones.
• Cohorts, churn and LTV under Admin → Revenue: subscriptions grouped by the month they started, how many are still live, retention %, live MRR per cohort; 30-day cancellation rate over (live + churned); LTV as ARPU ÷ monthly churn, labelled as a rule-of-thumb until there are months of data. Pure computeCohorts with tests.
• MCP accepts the customer API key — /api/mcp with Authorization: Bearer <key> is metered on the key's daily quota (JSON-RPC -32001 with HTTP 401/429 when invalid or exhausted) instead of the anonymous 30-messages-per-minute-per-IP limit; authenticated traffic is attributed to the key in Admin → API & MCP and on Account → API. Documented on /api-docs, the MCP discovery GET and the admin limits card. Anonymous access is unchanged.
• Annual saving spelled out on the pricing cards: "billed $X/yr — 2 months free (save $Y)".

• Measured on the dev server: /terms, /alternatives, /pricing, /about render 200 with the expected titles; the pricing page contains no testimonial section while the list is empty; annual sub-copy shows "save $68" (Pro) and "save $238" (Growth); GET /api/mcp advertises optional auth; anonymous tools/list returns 16 tools; a bogus bearer key returns the JSON-RPC error with HTTP 401; the admin testimonials API returns 401 unauthenticated.
• tsc clean (two pre-existing test-file errors unchanged), eslint clean on all touched files, 593 tests pass, production build below.

• Legal review of /terms; real customer quotes; verification of any competitor claim before adding one; production env checks (Resend, billing flag, Supabase public vars, engine keys); MySentry registration of the lifecycle cron; Stripe promotion code for WINBACK_PROMO_CODE. Deployment is not performed without an explicit request.

• Win-back email about a month after a cancellation (30–44-day window, once per subscription, honours unsubscribes): the projects and history are still there on the free plan, here's the door — with WINBACK_PROMO_CODE quoted when set (create the code in Stripe; checkout accepts promotion codes). Runs from the daily lifecycle cron alongside trial reminders and lead nurture.
• Browser events are mirrored to GA4 (gtag('event', …) when the tag is loaded after consent), so the GA4 property sees the same scan / report / bridge / capture steps as the first-party store. Server-recorded conversions (signup, trial_started, upgrade, cancel) stay server-side by design; the GA4 admin tab now says exactly that instead of claiming events it never received.
• README replaces the create-next-app boilerplate: what the product is and for whom, the map of the codebase, how to run it, the conventions (version = changelog = commit; dormant-until-configured; privacy-minimal; server-recorded conversions) and a five-minute post-deploy check.

• 589 tests (+1), tsc and eslint clean. The win-back send itself is not exercised locally (no Resend); the window logic is unit-tested.

• /agencies — the Agency plan as a landing page: 25 client projects, 400 prompts, all five engines daily, 10 competitors, 10 seats, white-label PDF reports, the price from the live catalogue (revalidated hourly), "what a client sees each month", a FAQ with FAQPage JSON-LD, the trial CTA and a walkthrough e-mail link.
• /about — what the product does, how it scores, what data it touches, who runs it (company number, registered office) and how to reach a person. AboutPage JSON-LD.
• /changelog — every release rendered from CHANGELOG.md at build time (a small parser and a safe inline renderer, unit-tested): the public proof of pace. Linked in the footer together with About and For agencies.
• /leaderboard/{domain} — one static page per site in the snapshot (970): composite, AI-readiness and security scores with a plain-English band, rank overall and within the category, category neighbours as internal links, what the scores mean, "check your own site" and the tracking bridge; blocked sites get the bot-protection explanation instead. ItemPage + breadcrumb JSON-LD, listed in the sitemap (1,063 URLs now); the leaderboard table links to them.
• Learn — the empty Performance category gets core-web-vitals and caching-and-ttfb (beginner and professional explanations, fix steps, FAQ), with issue-id rules so performance findings link to them.
• The sitemap gains the two tools it was missing (/tools/blacklist-check, /tools/domain-expiry-reminder) plus the new pages.

• The homepage and /agencies rendered dynamically (Cache-Control: private, no-cache) despite revalidate = 3600: the KV client fetches with cache: "no-store", and any page that reads the catalogue during render inherits that. Public pages now read it through unstable_cache (hourly, tag billing-catalog, invalidated with revalidateTag(…, "max") when the admin saves prices). Measured on next start: both now answer x-nextjs-cache: HIT, s-maxage=3600.
• Five functions shipped the entire project — /r/[token], its OG image, the PDF route, the leads and reports APIs carried ~700 traced files each, ~490 of them project source. Found by diffing the routes' .nft.json; cause: fs.readdir(storeDir()) in the report store's sweep — a readdir on a runtime-computed path makes Turbopack's tracer give up and include everything. Pre-existing since the store was written. Fixed by reading the literal .data/reports directory (and never sweeping under test). The PDF route now traces 193 files, no route exceeds 100 source files, and the build warning "whole project was traced unintentionally" is gone. The PDF fonts moved to new URL(…, import.meta.url) asset references (emitted as server assets and listed in both PDF routes' traces; the outputFileTracingIncludes TTF globs are no longer needed) — verified on the production build: a freshly generated report embeds Geist, not the Helvetica fallback.

• 588 tests; tsc clean; production build: 1,141 static pages, zero warnings. On next start: /about, /agencies, /changelog, /leaderboard/github.com (scored), /leaderboard/aa.com (blocked), the two Learn entries and the sitemap render; an unknown domain 404s; a shared report's PDF embeds Geist. Nothing in this release needs a runtime service.

• Daily lifecycle cron GET /api/v1/cron/lifecycle (vercel.json, 10:00 UTC, CRON_SECRET): (1) trial-ending reminders 3 days and 1 day before the first charge — plan, date, amount and the portal link (src/lib/lifecycle/trials.ts; for a trialing Stripe subscription current_period_end is the trial end); (2) lead nurture — three emails to people who downloaded a PDF report, on day 1 (the tool-specific fix that moves their score most, linking their report), day 3 (who AI recommends instead → free account) and day 7 (track it weekly → plans). One email per address per run, steps in order, never twice (markers), never to leads older than 14 days — so the day this ships does not blast the historical list (src/lib/lifecycle/nurture.ts).
• Unsubscribe for lead email (/leads/unsubscribe): the link carries the address and an HMAC (LEADS_UNSUB_SECRET, falling back to CRON_SECRET), GET shows a confirmation page, POST implements RFC 8058 one-click; every nurture and waitlist email carries the link and the List-Unsubscribe headers. Opt-outs are markers keyed by a hash of the address.
• Waitlist announcement from Admin → Users → Waitlist: "{plan} is open", optional early-adopter promo code, once per address, honouring unsubscribes (src/lib/lifecycle/waitlist.ts, POST /api/v1/admin/waitlist). The email the waitlist was promised and never got.
• Weekly digest for the free tier too: the headline share of voice, the week's delta and cited sources, plus a plain statement of what Pro adds (competitors, three engines, alerts). Paid tiers unchanged.
• Leads now store the shared-report token so follow-ups can link the report; one-shot markers live in src/lib/lifecycle/markers.ts (KV with a 13-month TTL, files locally).

• 583 tests (+7: unsubscribe tokens and opt-out, nurture scheduling, trial timing); tsc and eslint clean. Live on the dev server: /leads/unsubscribe?token=bad → 400 (GET and POST); /api/v1/cron/lifecycle → both jobs report "skipped":"email not configured" — the expected dormant answer without Resend.
• Not exercised: actual sends (no Resend locally) and the Stripe currentPeriodEnd semantics during trial — taken from Stripe's documented behaviour. After deploy: register the new cron in MySentry (a heartbeat that never fired can't alert) or run it once by hand, and check Admin → Usage for lifecycle_email events.

• <NextStepBridge> under the result on all 49 tool pages, the Domain Audit and every shared report (/r/{token}): AI/SEO tools (readiness, llms.txt, robots, sitemap, schema, SEO, link preview, compare, fix pack, performance, generators for AI/SEO, utilities) bridge to AI-visibility tracking — "This is one check, today. Now see whether AI recommends you — every week" → /account?next=/projects, with /pricing as the secondary link; DNS, email, TLS and security tools (25 pages) bridge to monitoring first — "Fixed it? Get told before it breaks again" → /tools/monitoring, with tracking as the secondary link. The monitoring page itself gets none (it is the product).
• Clicks are tracked as bridge_clicked {tool, kind, target}, so Admin → Usage shows which tools actually send people into the product — the number the revenue plan was missing.
• Inserted by a codemod (<NextStepBridge tool="…" kind="…" /> immediately before each page's <ToolFaq>), so the placement is uniform and a page without a bridge is a lint-visible exception, not a forgotten one.

• tsc clean, 576 tests pass; the tool pages' 11 eslint findings (react/no-unescaped-entities, react-hooks/exhaustive-deps) are pre-existing — identical count with the change stashed. Rendered /tools/dns-lookup locally: bridge present below the content, above the FAQ.

• Server-recorded conversion events (src/lib/server-events.ts): signup (first sign-in ever, via a once-per-user marker in KV / a local file), project_created, first_cycle_ready, trial_started, upgrade, payment_failed, cancel — the last four derived from Stripe status *transitions* (previous record vs. incoming), so overlapping webhook events count each milestone once. The browser beacon endpoint no longer accepts signup/upgrade (a beacon can't fake a conversion any more); monitor_create_* (fired by the monitoring UI, rejected with 400 until now) and bridge_clicked / cta_clicked are accepted.
• Admin → Usage → Funnel, last 30 days: Scans → Leads → Accounts → Projects → First data → Trials → Paid with step-to-step conversion, plus payment failures and cancellations beside it. computeFunnel is pure and unit-tested.
• First cycle in a minute. Creating a project now suggests the first three prompts from the domain (Claude Haiku; skipped without ANTHROPIC_API_KEY) and runs an instant cycle — 3 prompts × 2 samples × the plan's engines, calls in parallel — before redirecting, so the project page opens with share of voice, winning competitors and cited sources instead of "No cycles yet". Projects that have prompts but no data get a Run the first cycle now button. Guards: owner only, 3 instant runs per account per day, the global TRACKING_DAILY_BUDGET, and plain messages while engine keys are dormant. The runner's per-project loop became runProjectCycle (shared by the cron — sequential as before — and the bootstrap — concurrent): src/lib/tracking/runner.ts, bootstrap.ts.
• Lifecycle emails (src/lib/lifecycle/emails.ts; dormant until Resend is configured, intent logged): welcome on first sign-in (one job: create the first project), "your first cycle is in" with the numbers (and a plan nudge on Free), payment failed (→ billing portal), cancelled (projects stay on Free, door open). Trial-ending reminders need a daily cron and ship in the next release.
• /projects?welcome=1 onboarding checklist; the create and run buttons narrate the 20–40 s wait (useFormStatus); both project pages declare maxDuration = 60 for the synchronous action.

• AI Visibility Free: Claude-sampled, 3 prompts weekly, first cycle in a minute (was "ChatGPT · Claude · Perplexity · Google AIO" and a "public leaderboard rank" that never existed). Growth: all five engines, 5 projects, 2 seats, 365-day history (was "API + MCP access" — a separate product — and "weekly digest for your team", which every paid tier gets). Agency now states 25 projects, 10 competitors, 10 seats. Monitoring: hourly checks (was 5-/1-minute), email + webhook alerts (was status page, SMS credits, white-label status page, team seats, priority alerting). API: daily quotas as enforced — 100 / 500 / 4,000 per day, 2 / 5 / 10 keys (was "10,000 / 100,000 per month" and "webhook results"). Tool counts come from the index.

• 576 tests pass (+4), tsc and eslint clean; /pricing rendered locally with the new bullets. Not exercised locally (no Supabase, engine or Resend keys in .env.local): the bootstrap run, the welcome / first-cycle emails and the webhook transitions — reasoned from code and written to fail soft (every branch degrades to the scheduled cron). First check after deploy: create a project on a free account → the page should open with numbers, and Admin → Usage → Funnel should show project_created and first_cycle_ready.

• Shared builders in src/lib/jsonld.ts — breadcrumbJsonLd (always starting at Home), webPageJsonLd (with isPartOf / publisher references) and graph — so every page links into one entity graph anchored on the /#organization and /#website nodes the layout declares. Same ids everywhere is what lets search and answer engines merge the pieces.
• Organization enriched with legalName, the Companies House identifier, the registered address, support email + contactPoint, foundingDate and knowsAbout; WebSite gains inLanguage and a description.
• New nodes: /tools — CollectionPage + an ItemList of all 51 tools as free WebApplications; /pricing — one Product per product line (AI Visibility, Monitoring, API) with an AggregateOffer and an Offer per active plan from the live catalogue (monthly and annual UnitPriceSpecification, free tier included); /leaderboard — Dataset (snapshot date, variables measured, technique) plus a CollectionPage; /api-docs — WebAPI (endpoint, documentation, provider, free-tier offer; MCP named in the description); /bot — WebPage about the crawler as a SoftwareApplication; /privacy — WebPage with dateModified; /guides and /learn — breadcrumbs. Tool pages' WebApplication now carries @id, publisher, isPartOf, isAccessibleForFree and a Home-first breadcrumb (one helper, 49 pages).
• /guides and /learn switched from raw JSON.stringify scripts to the escaping JsonLd component.

• Fetched 13 pages from the dev server and parsed every application/ld+json block: 0 invalid; each page carries the expected types (Organization + WebSite everywhere; SoftwareApplication + FAQPage on /; CollectionPage + ItemList on /tools; Product ×3 on /pricing; Dataset on /leaderboard; WebAPI on /api-docs; TechArticle + FAQPage on a guide; DefinedTerm + FAQPage on a Learn entry; WebApplication + FAQPage on a tool). 3 new unit tests (572 total); tsc and eslint clean.
• Rich-result eligibility (Product, FAQ, Breadcrumb, Dataset) is Google's call — run the Rich Results Test on /pricing, /leaderboard and a tool page after deploy.

• API keys are issued from the customer account only. The public generator on /api-docs is gone; the page now sends developers to Account → API (sign-in or a free account, then a key). POST /api/v1/keys requires a session (401 otherwise) and enforces the plan's key count (free 2, Dev 5, Scale 10); new GET /api/v1/keys lists the caller's keys with today's usage, and DELETE /api/v1/keys?id= revokes one. Every key now has an owner: records carry userId and a per-user index (apikeys:user:{id} in KV, a JSON file locally) makes list/revoke possible without scanning the store. Keys minted anonymously before this release keep working for requests; they simply belong to nobody and can't be listed.
• Account → API tab: plan, requests/day per key, active keys vs. the plan limit, a create form (label optional; the key is shown once), each key's usage today / total calls / last used, and Revoke. Free accounts see the API-plan upsell.
• Admin → "API & MCP" tab (/admin?tab=api): 7- and 30-day totals for the public API and the MCP server, rejected requests split by cause (burst rate-limit, daily quota, auth, malformed), a per-day table, and ranked breakdowns — API tools, API keys (by public id, matching Account → API), MCP tools and JSON-RPC methods. The limits in force are stated on the page: MCP 30 messages/min per IP, no key, no daily quota; API 60 requests/min per IP plus the per-key daily quota (free 100, Dev 500, Scale 4,000); the per-minute limiter is per serverless instance, the daily quota in KV is exact.
• Counters, not an event log: src/lib/api/metrics.ts keeps one hash per surface per day (metrics:api:YYYY-MM-DD → total, outcome, tool, key, method; 100-day TTL in KV, JSON files locally). Routes record through Next's after() so metering runs once the response is sent and can never fail a request. Unit-tested (+5 tests, 569 total).
• /api-docs "Limits & errors" now states the real quotas per plan and the per-IP burst cap; the 401 message for a missing key points to Account → API.

• tsc and eslint clean on every changed file; 569 tests pass (key ownership, revoke, path-traversal guard on user ids, metrics aggregation). Not exercised locally: the KV branch (hashes + user index) — it uses the same kvHincrby/kvRpushJson helpers already in production for the rate limiter and event stream.

• Sign in with a passkey on /account: a one-tap WebAuthn sign-in next to the magic link, via supabase.auth.signInWithPasskey() (auth-js 2.110, behind its auth.experimental.passkey flag). The same next continuation is honoured, so "Start free" → passkey → /projects. The email input advertises autocomplete="email webauthn" for conditional UI.
• Passkeys card on Account → Account: list (name, added, last used), add — auth.registerPasskey() then a friendly name via auth.passkey.update() (default "Chrome on macOS"-style from the user agent) — and remove. Clear recovery note: the email link always works, so a lost device is never a lock-out.
• Plain-English errors for every failure mode (NotAllowedError cancelled/timed out, InvalidStateError already registered, unsupported browser, insecure context, server not enabled, no passkey for this device) in src/lib/auth/passkeys.ts, unit-tested.
• Dormant until configured: NEXT_PUBLIC_PASSKEYS_ENABLED=1 shows the buttons and enables the client flag. Prerequisite on the Supabase project: passkeys enabled in Authentication settings (the /auth/v1/passkeys/* endpoints must respond) with the relying-party ID set to the site host. Without that the buttons would fail at the first click, which is why the flag defaults off. .env.example documents it.
• Why first-factor rather than MFA-only: auth-js exposes both; a second factor on top of a magic link adds friction without removing the inbox round-trip, while a passkey replaces it — the faster sign-in is the one that sells.

• 5 new unit tests (564 total); tsc and eslint clean. Not verifiable locally (no Supabase project with passkeys in .env.local): the live ceremony, which needs HTTPS on the real host. First thing to check after enabling: register a passkey on Account, sign out, sign in with it.

• Homepage rebuilt around the paid product. Hero: "Is your brand in the answer?" with one primary action (Start tracking free → /account?next=/projects) and a secondary one-shot check. A static product preview (trend chart, share of voice, "who wins the answers", labelled as sample data) replaces the abstract benefit tiles. Then: a proof strip with the real leaderboard numbers (970 sites, median 76/100, 17% blocked), "what you see every week", how it works, the one-shot AI Visibility check (/tools/ai-visibility, now tracked as scan_started {tool:"home-ai-visibility"}), plan cards from the editable billing catalogue (annual per-month price; feature bullets are the entitlements the code actually grants), the "Can't I just ask ChatGPT myself?" objection, and only then the free tools — the search box, six featured tools and "Browse all N tools" — visible to anyone who scrolls, absent from the first screen. FAQ with FAQPage JSON-LD, a closing CTA, and SoftwareApplication JSON-LD carrying Offers from the catalogue. The page is ISR (revalidate = 3600), so an admin price edit reaches the homepage within an hour without a deploy.
• The header sells an account: "Sign in" + Start free on desktop and mobile. "Domain audit" leaves the header button (it stays on /tools, the homepage and the mobile menu); the IP widget leaves the nav (it stays on /tools/what-is-my-ip).
• /tools gets the homepage's type-to-find search, and the tool count comes from the index (ALL_TOOLS.length, 51) — one number everywhere instead of "40+ / 50+ / 55".
• Funnel fixes from the revenue plan: the /pricing Free-plan CTA now creates an account (/account?next=/projects) instead of opening a one-shot tool; /billing/success sends the new customer to Create your first project (/projects?welcome=1) instead of back to the free tools; sign-in carries a same-site next through the magic-link callback, and a signed-in visitor arriving with ?next= is redirected straight there; the email-capture copy no longer promises that monitoring "launches" (it shipped in v0.60).
• Metadata: homepage title "Crawlsonar — is your brand in the AI answer?" with a tracking-first description; canonical unchanged.

• tsc clean for the change (two pre-existing test-file errors untouched), eslint clean on every changed file, 559 tests pass. Dev server: /, /tools and /account?next=/projects render; homepage copy checked section by section; plan prices $28 / $99 / $249 per month (annual) from the seed catalogue.
• Not verifiable locally: auth and billing are env-gated. With NEXT_PUBLIC_SUPABASE_URL / NEXT_PUBLIC_SUPABASE_ANON_KEY unset the new CTAs land on "Accounts aren't enabled yet" — confirm both are set in production before this goes live, and that NEXT_PUBLIC_BILLING_ENABLED=1 so the plan cards lead to checkout rather than the waitlist modal.

• Lead-magnet PDF (GET /api/v1/reports/{token}/pdf, AI Readiness / llms.txt / Domain Audit) is now a designed document instead of a text dump. Page one: navy cover band with the sonar mark, score ring with grade pill, severity pills, plain-English verdict (with the "capped by a high issue" explanation when it applies), four KPI tiles (issues by severity, checks passed, weakest area or module, AI-crawler rules / scan time / llms.txt status) and the top three priorities. Then: category score bars, issue cards (severity + category/module pill, Impact / Evidence / Fix in a spec-sheet layout, inline code, example code block, clickable "plain-English explainer" link into /learn/{slug}), the AI-crawler policy table with policy pills, "what an AI assistant actually reads" (front-matter and image syntax stripped from the extraction), the full checks list with drawn pass/fail/n-a glyphs and weights, a numbered "what to do next" (re-run link + share link with its expiry) and a closing bridge to AI Visibility monitoring with a UTM-tagged /pricing link. Running header/footer with "Page i of n", PDF outline bookmarks, document metadata.
• AI-Visibility tracking PDF (GET /api/v1/projects/{id}/report) rebuilt on the same system: headline share-of-voice ring with the change versus the previous cycle, KPI tiles (share of voice, presence, tone, change since the first cycle), a vector trend chart (share of voice + presence), per-engine table for the latest cycle, cycle history with ▲/▼ deltas, the latest cycle's winning competitors and cited sources as ranked bars (the route now passes latestInsights, so the PDF never says less than the project page), tracked prompts, competitor chips, a "how to read this report" note and a dashboard CTA. White-label (Agency) output carries no Crawlsonar name, mark, link or metadata — covered by a test that greps the uncompressed document.
• Typography: Geist + Geist Mono (SIL OFL 1.1, licence in src/lib/pdf/fonts/) embedded and subsetted per document — a five-page report is ~70–85 KB. If the TTFs cannot be read at runtime the renderer logs {"at":"pdf/fonts",…} and falls back to Helvetica/Courier under the same logical names: a missing asset degrades the look, never the download. next.config.ts force-includes the TTFs (and pdfkit's .afm metrics) into both PDF function bundles.
• New shared layer src/lib/pdf/{canvas,theme,format,fonts}.ts: measured text, ensure() page-break guards so no card, table, chart or heading straddles a page, tables that redraw their header after a break, ring / bar / line-chart / code-block primitives, pill / KPI / badge composites. Both renderers keep their public signatures (renderReportPdf, renderTrackingPdf), with an optional { compress: false } for tests.
• Tests: +9 (font embedding and fallback, llms.txt shape, multi-page pagination with embedded Geist and an outline, tracking branded / empty / white-label / dashboard link) — 559 total.

• Rendered the nine locally stored sample reports (ukimmigration.law, stripe.com, github.com, faktury.co.uk, example.com across all three report types) plus synthetic tracking data (branded, white-label, empty state) and inspected every page as a PNG. Unit tests pass; tsc is clean for the new files (the two pre-existing tsc errors in core-web-vitals.test.ts and reports/store.test.ts are untouched).
• Not yet verified on Vercel: that outputFileTracingIncludes ships the TTFs into the function bundles. After the first deploy, download any report and confirm the PDF embeds Geist (font panel, or strings report.pdf | grep Geist). A Helvetica-only PDF means the fallback kicked in and the function logs will show "at":"pdf/fonts".

• /pricing reads prices from the editable billing catalogue (KV, seed as default), so changing a plan’s price in the admin Billing tab now updates the public page too — no code edit. Curated feature copy stays in code; only the amount + which paid plans show (deactivated plans are hidden) come from the catalogue. The "Growth is $X/mo" anchor is computed from the live price. Verified: Pro $28 · Growth $99 · Agency $249 (annual/mo) from the seed.

• Usage tab: unique visitors, scans run, scans/visitor and total events, plus ranked bars for most-used tools, top countries, most-scanned targets and every event type — all from the first-party event stream (no external call).
• Revenue tab: MRR (annual normalised to monthly), ARR, paying accounts (active/trialing), ARPU, new/churned this month, past-due to recover, and an MRR-by-plan table with share — from the Stripe-synced subscription records.
• GA4 tab: connection status (Measurement ID), deep links to Realtime / Reports / Search Console, and exactly which events we send — with an offer to wire the GA4 Data API (service account) for embedded live numbers.
• Admin sidebar reorganised: Dashboard · Usage · Revenue · Users · Billing · GA4 · Errors. New src/lib/admin/insights.ts computes usage + revenue.

• Unified customer dashboard with the same left vertical sidebar as the admin panel (Overview / Projects / Subscription / Team / Account), a plan badge + sign-out in the footer, and a responsive top-strip on mobile. /account and /projects now share one shell (CustomerShell).
• Overview: welcome header + at-a-glance stat cards (projects used/limit, prompts, engines, cadence/history), primary quick actions, a "shared with you" card, and an empty-state nudge to create the first project.
• Subscription: current plan + what it unlocks, an Upgrade/Change-plan CTA, and a "Manage subscription" button that opens Stripe's secure customer portal for the signed-in user (new authenticated POST /api/v1/billing/portal-account — resolves the Stripe customer from the *verified session* e-mail, so a user can only ever manage their own billing).
• Team (seats invite/manage) and Account (sign-in details) as dedicated sections. Cognitive-bias-aware copy throughout (anchoring on limits, clear single primary action per view).

• The admin can now edit the plan list and prices directly in the panel (like the polbooks flow): add/remove plans, rename, change monthly price, toggle active — saved to the KV store (billing:catalog), with the code seed as the default until the first save.
• Price-aware Stripe sync: because Stripe prices are immutable, changing a price now creates a new Price carrying the same lookup_key (transfer_lookup_key) and archives the old one — new checkouts get the new amount, existing subscribers keep theirs (grandfathering). The status table flags each row as Synced / Price changed / Missing / To archive with catalogue-vs-Stripe amounts.
• Checkout validates the plan against the editable catalogue (not the static list); the stripe:setup CLI and the admin button share one engine (stripe-catalog.ts). Admin API validates every field with zod and never leaks secret values. +2 tests.

• Admin redesign: the panel now uses a left vertical sidebar (CMS-style) with icons — Dashboard / Errors / Users / Billing — a brand header, identity + sign-out in the footer, and a full-width content area. Replaces the old top tab-strip (admin-tabs.tsx removed). Responsive: the sidebar collapses to a scrollable top bar on mobile.
• Billing tab (Stripe product management, like the polbooks flow): shows a live readiness banner, a configuration checklist (secret key + test/live mode, webhook secret, pricing-UI flag, catalogue completeness), and a table of every plan × interval with its lookup key, price and Created/Missing status. A "Create products in Stripe" button runs the same idempotent sync as the CLI — right from the admin UI.
• Shared catalogue engine src/lib/billing/stripe-catalog.ts (getCatalogStatus / syncStripeCatalog) now backs both the admin button and npm run stripe:setup (DRY). New admin API POST/GET /api/v1/admin/billing (behind the admin gate; returns 4xx not 5xx per the CF-swallows-5xx rule; never leaks secret values — booleans only).

• Root cause: every tool API returned HTTP 5xx (502/504) for *expected, target-caused* failures (the scanned site blocks bots, times out, DNS-fails, or returns 4xx). Cloudflare (in front of crawlsonar.com) replaces any origin 5xx with its own HTML error page, so the browser never received our JSON {error} — res.json() threw and the client fell back to the generic "Network error — please try again."
• Fix: target/upstream-failure responses now return HTTP 422 (which Cloudflare passes through untouched, like our existing 400/429) across all 25 tool routes + the WHOIS/RDAP, Certificate-Transparency and link-preview libs. The client now sees the real, specific message (e.g. "Could not reach that page — it may block bots or be too slow").
• Link-preview also: client now parses the response body defensively (a non-JSON edge page can never surface as a bogus "Network error"), the fetch timeout drops 10s → 8s, and it now runs bot-wall detection — a Cloudflare/Akamai/etc. challenge served as HTTP 200 (e.g. "Just a moment…") is reported as "X challenge is blocking our crawler" instead of returning a junk preview scraped from the interstitial.

• Each AI answer that mentions the brand now gets a tone score −1..1 from a cheap, deterministic lexicon (positive/negative terms + simple negation flipping) — no per-answer LLM cost, protecting the paid product's margin. analyzeSentiment scores only the sentence(s) naming the brand; analyzeAnswer attaches it; aggregatePrompt exposes avgSentiment; latestInsights averages it per cycle.
• Migration 0004 adds a nullable tp_observations.sentiment numeric(4,3) (additive; existing RLS/grants cover it). Runner persists it.
• Project page gains a "How AI talks about you" card (Positive / Neutral / Negative + score). +7 tests.

• Leaderboard generator now runs two gentle retry passes (concurrency 3, 45s timeout) over *transient* errors — recovering the ~96 marquee brands (bankofamerica, cloudflare, hsbc…) that timed out under load — while leaving deterministic robots.txt/anti-bot skips alone. data.json regenerated.
• Email deliverability: the Resend mailer now supports one-click List-Unsubscribe (RFC 8058) headers (Gmail/Yahoo bulk requirement); wired into the domain-expiry reminders. (Note: transactional email was already fully implemented — this hardens inbox placement.)

• GA4 is already wired (consent-gated, CSP-nonce) — set NEXT_PUBLIC_GA_ID to activate.

• Run supabase/0004_observation_sentiment.sql (or supabase/Total_incorporating_0004.sql).

• Team seats. An Agency (or Growth) owner can invite teammates by e-mail from /account; an invited teammate shares read + day-to-day collaboration (prompts & competitors) on all the owner's tracking projects. Project create/delete and billing stay owner-only.
• New table tp_members (migration 0003) with its own RLS (owner manages seats; a member reads only rows naming them). The six tracking tables are widened from "owner-only" to "owner OR member" via one security definer set-function tp_member_owners() (self + inviters) — chosen to avoid RLS policy recursion and keep every policy a one-liner. SELECT = owner/member; INSERT/UPDATE/DELETE = per-table (prompts/competitors collaborative; projects/runs/snapshots owner-only writes). Cross-tenant negative-test SQL is inline in the migration.
• Invites are claimed server-side with the service key (claimInvites binds member_user_id when the invited e-mail signs in) — identity never trusted from the client.
• Correct ownership limits: a project's prompt/competitor/cadence limits now follow the project owner's plan (entitlementsForOwnerId), so a member on a personal free plan can still work a 400-prompt Agency project; the create-project quota counts only owned projects (countOwnProjects). Seats: Growth 2 · Agency 10 (maxSeats).
• /account gains a Team panel (seat usage, invite, remove, Active/Invited status) + a "Shared with you" card; /projects labels shared projects and hides Delete on them. +3 tests. Dormant until Supabase Auth is configured — single-user behaviour unchanged.

• Run supabase/0003_agency_workspaces.sql (or the consolidated supabase/Total_incorporating_0003.sql) in the Supabase SQL editor.

• API keys are now plan-tiered. A signed-in user minting a key gets it on their subscription tier (apiLimits: free 100/day, api_dev 500/day, api_scale 4000/day), baked into the key record (plan, dailyLimit, userId); authorizeAndMeter enforces the key's own daily quota. Anonymous keys keep the free limit unchanged.
• Monitors are now plan-capped. A signed-in user's monitors belong to them (Monitor.userId) and are limited by plan (monitoringLimits: free 2, mon_pro 20, mon_business 100); the create route rejects over-limit with a 403. Anonymous flow (IP rate-limit) untouched.
• Multi-product aware: activePlansForEmail returns all active plans, so one user can hold AI-Visibility + Monitoring + API subscriptions at once. Entirely dormant until Supabase Auth is configured — anonymous behaviour is byte-for-byte unchanged. +2 tests.
• Remaining: I (Agency workspaces + seats — RLS migration + negative tests, next) and J-sentiment.

• ✨ Suggest prompts with AI. On a project, one click asks Claude (Haiku, cheap) for 10 realistic buyer questions in the brand's category; tick the ones you want and add them in a batch (src/lib/tracking/suggest.ts, GET /api/v1/projects/[id]/suggest — auth + RLS + rate-limited, addPromptsAction respects the plan's prompt cap). Solves the cold-start — an empty tracking project is now one click from useful. Dormant until ANTHROPIC_API_KEY is set.
• Remaining: J-sentiment (needs a schema column + is a heuristic — deferred), K (gate Monitoring + API to Stripe entitlements) and I (Agency workspaces + seats) — both are ownership-model + RLS changes, done carefully next.

• Crawler User-Agent is now version-stable — Crawlsonar/1.0 (+https://crawlsonar.com/bot; diagnostic scan) (no longer embeds the app version), so verified-bot registrations (Cloudflare Signed Agent) that match the full UA don't drift each release. The performance-check UA is aligned and also points at /bot.

• G · Latest-cycle drill-down. Project pages now surface the data we already store per run: "Where AI cites you" (ranked domains the models leaned on — the get-featured list) and "Who wins the answers" (competitors named most this cycle). latestInsights() aggregates the newest cycle's observations.
• H · Weekly digest email (Pro+). New /api/v1/cron/digest (Mondays 09:00, src/lib/tracking/digest.ts) emails each paid project owner their current share of voice, the week's delta, and top sources/competitors — a retention touch-point. Dormant until Supabase service + Resend are set.
• Web Bot Auth: the key directory now signs its own response (Signature/Signature-Input, tag http-message-signatures-directory, covering @authority) — Cloudflare requires a signed directory. Request signing was already compliant (Signature-Agent, tag web-bot-auth, JWK-thumbprint keyid, ed25519, @authority+signature-agent).

• Public /bot page. Documents the Crawlsonar crawler: its User-Agent, that it reads server-rendered HTML only, respects robots.txt, signs requests (Web Bot Auth), its rate limits, and exactly how to allow or block it (robots.txt snippet + WAF guidance) plus a contact. Linked in the footer + sitemap. This is what Cloudflare/DataDome verified-bot programs and site owners look for.
• robots.txt enforcement. New src/lib/security/robots-gate.ts (robotsAllows / assertRobotsAllowed, per-host cached, fails open) wired into the automated scanner (scanReadiness — the leaderboard + admin watchlist): we now **skip any site that disallows the Crawlsonar token or * in robots.txt**. +4 tests.
• User-Agent now points at the bot page — +https://crawlsonar.com/bot — so a site owner who sees it in their logs lands on how to identify/allow/block us.

• Leaderboard scaled to ~1000 sites. src/lib/leaderboard/sites.ts expanded from 138 to 970 unique well-known domains across ~25 categories; the batch runner now de-dupes and tolerates bad domains (graceful "error"). The "Ranked by composite readiness" table is now paginated (50/page), sortable by every column, and searchable across every column (src/components/leaderboard-table.tsx).
• Per-engine trend view. The project trend chart now stores + shows a snapshot per engine plus the "all" aggregate, with an engine selector (groupSnapshotsByEngine, updated TrendChart).
• AI-visibility change alerts (Pro+). When a cron cycle moves a project's share of voice ≥10 pts — or the brand drops out of AI answers — the owner gets an email (src/lib/tracking/alerts.ts); threshold-gated, dormant until Resend is configured, free tier excluded.
• White-label PDF report. GET /api/v1/projects/[id]/report renders a per-project AI-visibility PDF (latest metrics, history table, competitors, prompts) via pdfkit; Agency tier gets it white-labelled (Crawlsonar branding dropped). "Download PDF" on the project page.
• Tests: +3 (alert change logic).

• Four more engines. src/lib/tracking/engines.ts: ChatGPT (OpenAI gpt-4o-mini), Perplexity (sonar, web-grounded), Gemini (gemini-2.0-flash) and Google AI Overviews (via SerpAPI — a SERP, not an LLM API), alongside Claude. Each is a separate adapter, dormant until its own key is set (OPENAI_API_KEY / PERPLEXITY_API_KEY / GEMINI_API_KEY / SERPAPI_KEY), all using neutral stateless sessions + cheap models.
• Cron now samples every allowed engine. runner.ts resolves each project owner's plan → allowed engines, intersects with the keys actually configured, and samples each prompt N× per engine; a single flaky engine call no longer sinks the cycle. Writes per-engine trend snapshots + an "all" aggregate. Plan gating updated (Pro = Claude/ChatGPT/Perplexity; Growth+ adds Gemini + Google AIO).
• Margin: one global daily budget across all engines (TRACKING_DAILY_BUDGET), per-engine cost estimates. Tests: +3 (engine gating); entitlement engine test updated for Phase 2. Env documented.

• Copy button next to the header IP — a small clipboard icon beside the desktop nav IP copies it to the clipboard (with a brief ✓).

• Share-of-voice trend chart on /projects/[id] — pure-SVG, theme-aware two-series chart (share of voice + mention rate, %) over time from tp_trend_snapshots, plus latest-cycle stat tiles and a "12% → 34% since <date>" delta. Empty state until the first cron cycle. src/components/tracking/trend-chart.tsx + listSnapshots.
• This completes the tracking product loop end to end: create project → add prompts/competitors → scheduled cron samples Claude → watch the trend climb. All dormant until Supabase Auth + ANTHROPIC_API_KEY are configured.

• Tracking engine. src/lib/tracking/runner.ts + /api/v1/cron/tracking (daily, CRON_SECRET-auth, origin-gate exempt): for each due project (isDue from cadence + last_run_at) it runs each active prompt N=3× through Claude (stateless = neutral sessions), records every tp_run + tp_observation, writes a daily tp_trend_snapshot (share of voice), and stamps last_run_at. Service-role Supabase client (src/lib/supabase/service.ts) bypasses RLS for the cross-user job.
• Margin protection. Global daily Claude-call budget in KV (TRACKING_DAILY_BUDGET, default 2000) + per-run caps (25 projects, 100 prompts). Errors captured to the admin Errors tab. Dormant unless ANTHROPIC_API_KEY + Supabase service are set.
• Tests: +4 (isDue cadence logic).

• Crawler User-Agent now carries the real version — Crawlsonar/0.1 → Crawlsonar/${APP_VERSION} (e.g. Crawlsonar/0.92.0 (+https://crawlsonar.com; diagnostic scan)).

• End-user accounts (Supabase Auth, magic-link). /account sign-in (passwordless), /auth/callback code exchange, sign-out. Session is scoped to the tracking/account area only (not global middleware) to avoid the getUser-on-every-request perf trap. Dormant until NEXT_PUBLIC_SUPABASE_URL + NEXT_PUBLIC_SUPABASE_ANON_KEY. New deps @supabase/ssr + @supabase/supabase-js.
• Entitlement gating. currentEntitlements(user) resolves the user's active subscription (planForEmail) → plan → tracking limits; enforced server-side in every Server Action (defense-in-depth on top of the disabled-in-UI controls).
• Project management UI. /projects (list + create, gated by maxProjects) and /projects/[id] (add/remove prompts and competitors, gated by plan). All mutations are Server Actions using the user-scoped Supabase client, so RLS enforces per-user isolation — the app never trusts a client-supplied user id.
• Nav: "Account" in the footer. Env documented in .env.example.
• Next: the sampling cron (runs prompts N× via Claude → stores runs/observations/snapshots) and the trend-chart UI.

Foundation for the paid tracking product (scoped & approved). Pure, tested, dormant — no auth/UI/cron yet.

• Schema + per-user RLS. supabase/0002_ai_visibility_tracking.sql (+ regenerated Total_incorporating_0002.sql): tp_projects / tp_prompts / tp_competitors / tp_runs / tp_observations / tp_trend_snapshots, each with owner-only RLS (auth.uid() = user_id, transitively for children) so user A can never see user B's data. Indexed for the cron's due-selection.
• Analysis engine (the "hard part"). src/lib/tracking/analysis.ts — token-boundary brand detection, competitive ranking (brand's position among named brands), cited-source extraction, and share-of-voice aggregation across the N sampling runs. Fully unit-tested.
• Claude adapter. src/lib/tracking/claude.ts — stateless Haiku calls (= clean, neutral sessions by construction; no account bias), small max_tokens, dormant until ANTHROPIC_API_KEY.
• Plan entitlements. src/lib/billing/entitlements.ts — plan → tracking limits (projects, prompts, cadence, N runs, competitors, history, engines); engines intersected with LIVE_ENGINES so we never promise an engine the runner can't execute (Phase 1 = Claude only).
• Tests: +12 (analysis + entitlements). Next increments: Supabase Auth (magic-link) + entitlement gating, the sampling cron, and the project UI + trend chart.

• Admin tables now paginate (10/25/50). The Errors and Users tabs use the reusable DataTable (page-size selector + pagination) instead of raw dumps — honouring the "all data tables paginate" rule. New client tables: errors-table.tsx, subscribers-table.tsx, waitlist-table.tsx.
• Early-access waitlist is now visible in admin. The Users tab shows a second paginated table of the pricing-page early-access signups (email · wanted plan · when) alongside live subscribers — so the pre-billing list is actionable at launch.
• /pricing “Why upgrade if the tools are free?” section — condensed objection-handler (track over time · representative data · alerts & reports) with a link to the full AI-Visibility case. Reinforces the value story on the money page.

• Full subscription checkout. stripe SDK + a billing catalogue (src/lib/billing/plans.ts, single source of truth: 7 plans × monthly/annual, annual = ×10). POST /api/v1/billing/checkout creates a Stripe Checkout Session (subscription mode, 14-day trial, promo codes) resolving the price by lookup_key; the pricing page swaps its waitlist CTA for a real "Start 14-day trial" when NEXT_PUBLIC_BILLING_ENABLED=1, and falls back to the waitlist otherwise.
• Webhook sync. POST /api/v1/billing/webhook verifies the Stripe signature and upserts a subscriptions store (src/lib/billing/subscriptions.ts, durable KV / in-memory) from checkout.session.completed and customer.subscription.*. Errors captured to the admin Errors tab.
• Customer portal + result pages. POST /api/v1/billing/portal (opened via the post-checkout session_id, so no email-based billing exposure) + /billing/success (shows plan/trial/receipt + Manage-billing) and /billing/cancel.
• Admin Users tab is now real — lists live subscribers (email, plan, billing, status, renews) from the store.
• Setup script: npm run stripe:setup (STRIPE_SECRET_KEY=sk_test_… ) creates all products + prices with lookup_keys, idempotently. Env documented in .env.example.
• Everything is dormant until STRIPE_SECRET_KEY (+ webhook secret) are set — the app behaves exactly as before without them. Tests: +6 (catalogue math + subscriptions upsert/list).

• New /pricing with a monthly/annual toggle (annual billed for 10 months = 2 months free, pre-selected for higher LTV). Three product blocks — AI Visibility (Free/Pro $34/Growth $119/Agency $299), Monitoring (Free/Pro $12/Business $39), API (Free/Dev $24/Scale $89) — prices trimmed from the research card.
• Cognitive-bias-driven, honest copy: reciprocity banner ("all 55 tools stay free"), competitor anchor (struck-through $189–199 vs Growth), center-stage "Most popular" featured tier, default-annual toggle framed as a gain, loss-framed objection link. No fake scarcity/countdowns.
• Early-access capture (pre-billing): paid CTAs open a modal that records the email + wanted plan to a new waitlist store (src/lib/waitlist/store.ts, durable KV / in-memory) via POST /api/v1/waitlist (rate-limited + honeypot). Converts to real Stripe checkout later; builds the launch list now.
• Nav: "Pricing" added to the desktop header, mobile menu and footer.

• AI Visibility: “Can’t I just ask ChatGPT myself?” objection-handler. Customer-facing section on /tools/ai-visibility that answers the #1 GEO objection head-on — non-determinism (one chat = one sample), account contamination (clean neutral sessions), breadth (many prompts × engines), trend-over-time as the real value, competitor + cited-source extraction, and drop-out alerts. Honest (the free check is framed as a baseline snapshot) and written to bridge toward the paid tracking tiers (loss-framed close). Same copy will seed /pricing.

• Admin is now tabbed: Dashboard · Errors · Users. /admin keeps everything that was there under Dashboard (extracted to src/components/admin/dashboard-tab.tsx); tabs switch via ?tab= (src/components/admin/admin-tabs.tsx).
• Errors tab — real error monitoring. New capped error log (src/lib/errors/store.ts, durable KV in prod / in-memory locally, last 500) with a fail-safe captureError() helper (src/lib/errors/capture.ts). Server failures are now recorded from the reminders cron, monitor cron and Resend email sends; uncaught client-side errors are reported via a global ErrorReporter (window.onerror / unhandledrejection) → POST /api/v1/errors (rate-limited, size-capped). The tab lists newest-first with time / kind / source / message / where, plus a Clear log action (DELETE /api/v1/admin/errors, admin-only).
• Users tab — placeholder for the accounts/subscribers view we'll build with billing.
• Tests: +5 (error store record/list/clear/cap, captureError never-throws & non-Error handling).

• "Remind me before it expires" button was dead. The Turnstile widget mounts on component load via a useEffect([]), but its container only rendered *inside* the save form (shown after a lookup), so it never initialised and no token arrived — leaving the submit button permanently disabled. The Turnstile container now mounts at the component root (interaction-only, so still invisible), so the token is ready when the user submits.
• Privacy-policy link in the consent notice now opens in a new tab (target="_blank" rel="noopener noreferrer") so it doesn't drop someone out of the form mid-signup.

• Expiry reminder offered on every domain check. The /tools/whois-lookup page now includes the reminder opt-in ("Never lose this domain") — for any expiry date, even years out — not just the dedicated tool.
• Visitor IP moved into the desktop header nav (left of "Tools"), a compact IP · x.x.x.x inline via a new variant="nav" on YourIp. On mobile the full strip stays at the bottom of the homepage (now sm:hidden so it isn't duplicated on desktop).

• New tool /tools/domain-expiry-reminder. Check any domain's registry expiry (live via RDAP), then opt in to be emailed at 90 / 60 / 30 days before it lapses. Two-step UI (src/components/domain-expiry-reminder.tsx): look up the date first (show value), then capture email with an explicit consent checkbox + Turnstile.
• Full reminder loop. New src/lib/reminders/{types,store,email}.ts: durable KV store (Supabase/Upstash) with filesystem fallback and a hashed domain+email pair-index for idempotent de-dup; branded Resend email with a working one-click unsubscribe; decideReminder threshold engine (sends only the single most-urgent note when someone joins late, never double-sends).
• API + cron. POST/GET /api/v1/reminders (authoritative RDAP lookup server-side, rate-limited, Turnstile, consent-gated), POST /api/v1/reminders/unsubscribe + /reminders/unsubscribe confirm page, and a daily cron /api/v1/cron/reminders (added to vercel.json + origin-gate exempt) that re-checks RDAP at each threshold to detect a renewal and skip nagging.
• Monetisation (honest, light-touch). The reminder email cross-sells our own uptime/SSL monitoring (100% margin) and carries an optional REGISTRAR_AFFILIATE_URL "renew or transfer & save" CTA — dormant until set (renewals are poorly commissioned; transfers/new-regs are what pay). The real asset is the high-intent, recurring email list.
• Compliance: explicit opt-in consent + timestamp, one-click unsubscribe, sender identity (Northstar Infinity Works Ltd), rate-limit + Turnstile so nobody can sign another address up. Dormant-until-configured for email (EMAIL_*).
• Tests: +16 (threshold engine incl. late-join & renewal, store create/get/find-by-pair/token-safety, dormant email).

• Dedicated admin host. New ADMIN_HOST (e.g. admin.crawlsonar.com): src/proxy.ts serves the dashboard at that host's root (redirects / → /admin) and **refuses /admin* + /api/v1/admin/* with 404 on every other host**, so the marketing domain no longer exposes an admin login. Dormant until ADMIN_HOST is set.
• Cloudflare Access JWT verification (second layer). New src/lib/security/cf-access.ts verifies the Cf-Access-Jwt-Assertion header — RS256 signature against the team JWKS (cached, refetch-on-kid-miss), plus aud / iss / exp/nbf checks — so a request reaching the origin without passing Access (e.g. a spoofed Host on the raw Vercel URL) is rejected. Dormant until CF_ACCESS_TEAM_DOMAIN + CF_ACCESS_AUD are set.
• Unified admin gate. src/lib/admin/guard.ts now resolves either mode: with Access configured the signed identity is the sole app-layer auth (the legacy token form is skipped and its cookie ignored); otherwise it falls back to the ADMIN_TOKEN cookie for local/dev. Who may sign in is decided entirely by the Cloudflare Access policy — the app keeps no email/domain allow-list. The admin page shows the signed-in email and, in Access mode, Sign out ends the edge session (/cdn-cgi/access/logout).
• Tests: +10 (Access JWT verify: valid / forged / bad aud / wrong iss / expired / non-RS256 / disabled; proxy admin-host routing: dormant / refused-off-host / dashboard-on-host / apex-untouched).

• Safe JSON-LD serializer. New src/components/json-ld.tsx (<JsonLd> / serializeJsonLd) escapes <, >, & and U+2028/U+2029 that JSON.stringify leaves raw — closing the theoretical </script> breakout in structured-data blocks. Applied to the site-wide + reused/content-driven spots (layout org data, ToolFaq on every tool page, leaderboard, guide articles). +2 tests. (Inputs are static today; this makes the pattern safe by construction.)
• Lead email is now format-validated — z.string().trim().toLowerCase().max(254).pipe(z.email()) instead of a bare length cap (rejects junk before storage/send).
• Admin sign-in rate limit is now durable — switched from the per-instance in-memory limiter to rateLimitKv (holds across serverless instances), on top of the existing Turnstile + constant-time compare.
• Extra response headers: HSTS gains preload; added Cross-Origin-Opener-Policy: same-origin and X-Permitted-Cross-Domain-Policies: none. (Deliberately NOT Cross-Origin-Resource-Policy — the embeddable AI-Ready badge is loaded cross-origin by design.)
• Dependencies: pinned postcss to 8.5.16 via overrides (Next bundled a flagged 8.4.31) — npm audit is now 0 vulnerabilities.

• Resend email delivery wired. The email stub is now a real send: src/lib/leads/email.ts exposes sendEmail() that POSTs to Resend's API with Authorization: Bearer (accepts EMAIL_API_KEY or RESEND_API_KEY, plus EMAIL_FROM), and report deliveries (sendReportEmail) and monitoring alerts (src/lib/monitor/alerts.ts) now use it with branded HTML. Fails safe (never throws into a route/cron); dormant until EMAIL_API_KEY + EMAIL_FROM are set, logging intent otherwise. New admin Email (Resend) card + /api/v1/admin/email route to check status and send a test email. +5 tests (466 total).

• Mobile hamburger menu. Below the sm breakpoint the inline header nav is replaced by a hamburger button (src/components/mobile-nav.tsx) that opens a full-width dropdown with the nav links + Domain audit; the theme toggle stays in the bar. Closes on link tap, on Escape, or by toggling. Desktop nav unchanged.

• Light / Dark / Auto theme toggle (geo-aware). A 3-way switch (☀ / ◐ / ☾) in the header. Tailwind's dark: variant now keys off a data-theme attribute (via @custom-variant) instead of the OS media query, so the choice is controllable and persisted in localStorage. A beforeInteractive init script sets the theme before first paint (no flash). Auto resolves via Vercel geo: src/proxy.ts reads x-vercel-ip-timezone on HTML navigations and sets a short-lived geo_theme cookie (day → light, night → dark for the visitor's location); the client falls back to its own clock, then the OS preference. A prefers-color-scheme CSS fallback covers the no-JS case.
• WebMCP (progressive enhancement). src/components/web-mcp.tsx registers Crawlsonar tools with the emerging navigator.modelContext.provideContext() API so an in-browser AI agent can use the site's actions — list_crawlsonar_tools, check_ai_readiness, check_agent_readiness (calling our public endpoints). Feature-detected, so it's a no-op in every browser that lacks the API today — an early-mover / dogfood flex for an agent-readiness product, with zero downside.

• Homepage rebuilt for laypeople + conversion. Reworked the front page around one clear job, plain language, and ethically-applied cognitive biases (real data only — no dark patterns): — Hero: a friendly question headline ("Can AI find and recommend your website?") + plain-English subhead (60 seconds, 0–100 score, free, no signup), and a single prominent CTA — removed the jargon (llms.txt) and the competing elements (IP strip, tool search) from above the fold. — Trust line anchors on real leaderboard data: "We've scored 138 of the world's best-known sites — the median is 78/100" with a "See where you'd land" curiosity hook. — "What you'll get" (score / issues / fixes) kills ambiguity before the click; "How it works" is three tiny steps; a benchmark section reuses the leaderboard (median 78, 15% blocked) for social proof + relativity; "What we check for you" reframes tools as outcomes a non-expert cares about. — Secondary paths (full audit, all 50+ tools, tool search, the IP strip) moved below the fold so they never compete with the primary action. Verified in light/dark, desktop/mobile.

• Content Signals (Cloudflare Content Signals Policy) — end to end. — Published on our own robots.txt. Converted the robots.txt from Next's structured MetadataRoute to a text route handler (src/app/robots.txt/route.ts) so it can emit custom directives, and added Content-Signal: search=yes, ai-input=yes, ai-train=yes (we're a GEO product — all uses allowed) to both the * group and the AI-crawler group. — Detected in the Agent Readiness Checker. New src/lib/checks/content-signals.ts parses a Content-Signal: directive; the checker adds a Content Signals declared in robots.txt check (weight 8) with an info finding + example when absent. Weights rebalanced (AI-crawler-rules 12→8, OpenAPI 12→8) so the module still sums to 100. +3 tests. — Generated by the AI robots.txt Generator. The generated policy now includes a Content-Signal directive derived from your allow/block choices (search / ai-input / ai-train), e.g. allow-all → all yes, block-all → all no, block-training → ai-train=no. +4 tests (461 total).

• mysentry uptime heartbeat. Crawlsonar now sends a heartbeat to mysentry (the operator's monitoring app) so its panel shows Crawlsonar alive — a dead-man's switch: if the pings stop, mysentry alerts. New src/lib/mysentry.ts POSTs { monitor, ok, processed } with Authorization: Bearer <token> to mysentry's /api/ingest/heartbeat (fire-and-forget, never throws into the cron; +5 tests). Configured from a new admin card — paste your mysentry project ingest token (stored server-side in KV, never returned to the client), pick a monitor name, and Send test heartbeat to confirm it lands. The hourly monitor cron carries the heartbeat (ok reflects whether the run had errors). Admin APIs /api/v1/admin/mysentry (+ /test) are session-guarded. Dormant until a token is set.
• /api/health — a lightweight public liveness endpoint (GET/HEAD → 200 with { status, version, store, time }), handy for any external HTTP uptime check.

• DNS for AI Discovery (DNS-AID) check in the Agent Readiness Checker. New src/lib/checks/dns-aid.ts probes _index._agents.<domain> and _a2a._agents.<domain> over DoH for ServiceMode SVCB/HTTPS records (RFC 9460) and reports whether the discovery answer is DNSSEC-authenticated (AD flag). Surfaces a concrete finding + fix (publish ServiceMode SVCB/HTTPS with alpn+endpoint, DNSSEC-sign the zone) with an example record, plus a dedicated explainer on /tools/agent-readiness linking the IETF DNS-AID draft and RFC 9460. Wired into the report (weight 10; llms.txt & MCP trimmed 15→10 to keep the module total at 100). +3 tests.
• Leaderboard average & median. The leaderboard now shows median + average of composite, AI-readiness and security across the scored sites (computeStats in src/lib/leaderboard/score.ts, +2 tests). When a visitor runs the AI Readiness Checker, a new "How you compare" panel shows their score against the leaderboard's AI-readiness median & average (of the well-known sites), with a distribution bar — stats passed server-side so the client bundle stays lean.
• Turnstile on admin sign-in. The admin login form now carries an (invisible) Turnstile check and the login route verifies it — defense-in-depth on top of the existing 5/5-min brute-force rate limit. Dormant unless Turnstile is configured.

• robots.txt: AI crawlers now get unrestricted access. The AI-crawler group (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …) is now Allow: / with no disallows — maximally readable and citable by AI, which is the point of a GEO product. The general * group keeps /api/, /r/, /admin, /monitor/ out.

• Admin domain watchlist + reusable data table. New reusable DataTable (src/components/admin/data-table.tsx): page-size selector (10/25/50), pagination, and optional mass-selection with bulk actions — applied across the admin. New Watched domains section in /admin: add domains, scan now (live AI-readiness + security composite via the shared scanReadiness engine), and a full lifecycle — Active (Archive / Bin), Archived (Unarchive / Bin), Bin (Restore / Delete now). Binned domains auto-delete after 30 days (lazy on read + in the daily cron). crawlsonar.com and mysentry.co.uk are seeded so the operator's own properties are tracked out of the box. Durable store in KV with an in-memory fallback for local dev; state-machine, purge, normalisation and dedupe are pure and unit-tested (+7 tests, 443 total). Admin APIs (/api/v1/admin/domains + /action + /settings) are session-guarded.
• Daily domain auto-scan cron. New /api/v1/cron/domains-scan (Vercel Cron, 0 6 * * *) purges expired binned domains then scans the day's least-recently-checked auto-scan domains, up to the operator-set auto-scan/day count (a box in the admin). Added to the origin-gate exemption in src/proxy.ts (Vercel Cron bypasses Cloudflare, so it must not be 403'd by origin cloaking) — covered by a proxy test.
• The existing Recent tests and Recent leads admin tables are now paginated via the same DataTable.
• scanReadiness extracted to src/lib/leaderboard/scan.ts and shared by the leaderboard generator and the watchlist (DRY).

• The AI & Web Readiness Leaderboard (/leaderboard). We scan a curated list of well-known public global sites through our own engine and rank them by a composite of AI readiness (55%) + HTTP security & headers (45%) — a data-journalism / backlink-magnet page (only big, public entities; no shaming of small sites). Sites that answer with an anti-bot challenge are detected via the bot-wall engine, marked blocked and surfaced separately with the vendor — a telling stat in its own right ("N% blocked our fetch — and, likely, AI crawlers too"). Each row links into the AI Readiness Checker (funnel). Static snapshot in src/lib/leaderboard/data.json, generated by npm run gen:leaderboard (scripts/gen-leaderboard.ts); the page prerenders from it, so zero per-request scanning. Pure composite/ranking logic in src/lib/leaderboard/score.ts (+5 tests, 436 total). First scan: 138 sites, 113 scored, 21 (15%) blocked. Linked from the header + footer nav.

• Blacklist Check tool (/tools/blacklist-check) — "is your IP or domain on a spam blocklist?". New src/lib/checks/blacklist.ts engine queries DNSBLs over DoH: an IP is checked against IP lists; a domain is checked against domain lists (DBL) and its MX servers' IPs (up to 3) against the IP lists. Results feed the standard explainable score (a listing is a high-severity issue that caps the score; unreachable lists are excluded, never counted as clean). Per-listing results cached in KV for ~1h. New API route /api/v1/tools/blacklist (zod + durable per-IP rate limit 15/h + Turnstile-gated) and the tool is registered in the search index.
• Dormant Spamhaus DQS integration. With SPAMHAUS_DQS_KEY set, the tool queries Spamhaus ZEN (IP) + DBL (domain) via the Data Query Service — the sanctioned, serverless-friendly path (Spamhaus blocks public-resolver queries and forbids commercial use of the free mirror). Without a key it runs the best-effort public list (SpamCop) and shows a "limited coverage — connect a free Spamhaus DQS key" notice. Blocked/misconfig sentinels (127.255.255.x) are treated as *unavailable*, never a false listing. +8 tests (431 total).

• Origin gate no longer breaks the service webhooks when enabled. The origin-cloaking gate (src/proxy.ts, dormant until ORIGIN_SHARED_SECRET is set) now exempts two self-authenticating service endpoints that legitimately arrive without the Cloudflare-injected header: /api/v1/inbox/ingest (Cloudflare Email Worker subrequest — a Worker's fetch into our zone isn't guaranteed to pick up the Transform Rule) and /api/v1/cron/monitor (Vercel Cron invokes the deployment directly, never traversing Cloudflare). Both stay protected by their own strong, constant-time secrets (INBOX_INGEST_SECRET, CRON_SECRET), so exempting them doesn't weaken cloaking. Exact-match allow-list (never prefix). Prevents inbound test emails and monitoring from silently dying the moment cloaking is switched on at launch. +6 tests (423 total) covering dormant pass-through, gated 403, correct-header pass, and both exemptions.

• Turnstile is now invisible (appearance: "interaction-only"). The widget no longer renders a visible box in the common case — it stays zero-height and still returns a token, only becoming visible if Cloudflare genuinely needs an interactive challenge. Fixes the bulky/ugly widget on the inbox and AI Visibility tools.

• Cloudflare Turnstile privacy addendum. New "Bot protection (Cloudflare Turnstile)" section on /privacy explaining what Cloudflare processes (IP + browser/device/interaction signals), that it's strictly-necessary and not used for cross-site tracking/ads, lawful basis (legitimate interests), and a link to Cloudflare's Turnstile Privacy Policy. Turnstile added to the processor list.
• Corrected the GA4 contradiction. The privacy policy still claimed "we set no cookies… no cookie-consent banner is required", which became false when GA4 + the consent banner shipped (0.71–0.72). The Cookies section is rewritten into strictly-necessary vs consent-based analytics; the Summary and Usage-analytics sections now describe Google Analytics 4 under Consent Mode v2 (disabled until you accept), with Google added to the processor list. Last-updated bumped to 10 July 2026.

• Compact "Your IP" strip. The IP box (homepage + What Is My IP tool) was a tall card with a large text-lg address; it's now a single-line pill (px-3 py-2, text-sm) showing IP · version · browser/OS · Copy on one row, taking far less vertical space.

• Cloudflare Turnstile bot mitigation (no-CAPTCHA) on the two costly/abusable endpoints: inbox address minting (/api/v1/inbox/new, creates DB rows) and AI Visibility (/api/v1/tools/ai-visibility, spends LLM tokens). Defense-in-depth — the per-IP rate limits stay as the cost floor; Turnstile raises the bar for automation on top. New src/lib/security/turnstile.ts verifies the token server-side with Cloudflare's siteverify (fails closed on a missing/denied token, open if Cloudflare itself is unreachable — same philosophy as the rate limiter, and the per-IP limit still caps abuse in that window). New src/components/turnstile.tsx useTurnstile() hook loads the widget script once and hands the client a single-use token sent in the x-turnstile-token header. Dormant until configured: with no NEXT_PUBLIC_TURNSTILE_SITE_KEY + TURNSTILE_SECRET_KEY nothing renders and no token is required, so behaviour is unchanged — requiring both keys avoids locking users out with a half-config. CSP widens to https://challenges.cloudflare.com (script-src + frame-src) only when a site key is set; dormant deployments keep the tighter frame-src 'none'. +8 tests (417 total).

• GDPR cookie consent + Google Consent Mode v2. GA4 now loads in a consent-aware state with all storage denied by default (analytics_storage, ad_storage, ad_user_data, ad_personalization set to denied, with wait_for_update: 500), set in the GA init before gtag('config') — so no analytics cookie is written until the visitor opts in. A new bottom cookie banner (CookieBanner, production-only) offers Accept / Reject with a link to the privacy policy; on Accept it calls gtag('consent','update',{ analytics_storage:'granted' }) and persists the choice to localStorage (cs_consent). Returning visitors who accepted get consent re-applied immediately on load; those who rejected (or never chose) stay denied. A "Cookie settings" link in the footer reopens the banner so the choice can be changed at any time (GDPR withdrawal of consent). Ad-related signals stay denied regardless — this is analytics-only.

• Inbox test: friendlier rate-limit message. When the durable limiter on /api/v1/inbox/new returns 429 (a visitor minted many test addresses quickly), the client now shows a dedicated "You've created a lot of test addresses just now. Please wait a minute, then refresh this page." notice instead of the misleading generic "Couldn't start the test" error.

• Google Analytics 4 (property G-FG8G8SKBVB) loaded via next/script (afterInteractive) in the root layout — production only, so local dev doesn't pollute the property. Configurable via NEXT_PUBLIC_GA_ID. The Content-Security-Policy was widened to allow it: script-src adds www.googletagmanager.com; connect-src adds the Google Analytics collection domains. (Note: GA4 sets cookies — a GDPR cookie-consent gate is the recommended next step.)

• Hardening: durable, shared rate limiting on the abusable/costly endpoints. Added rateLimitKv — a fixed-window limiter backed by the KV store (Supabase), so the limit holds across all serverless instances, unlike the in-memory limiter whose counter is per-instance (weak on Vercel). Applied to key-minting (/api/v1/keys), AI Visibility (/api/v1/tools/ai-visibility, each call spends LLM tokens), shareable-report creation (/api/v1/reports, re-runs a scan), and inbox-test address minting (/api/v1/inbox/new, creates DB rows). Falls back to the in-memory limiter when KV is unconfigured and fails open on a KV error (never blocks a legitimate user). +2 tests (409 total).

• Charts in the admin dashboard. The plain tables are now visual: an Activity — last 14 days bar chart (total events per day, with the tests portion highlighted), and horizontal bar charts for the Funnel and Tests by tool. Rendered server-side as pure CSS/SVG bars — no chart-library dependency, theme-aware, with hover tooltips. Backed by a new daily-activity aggregation in admin-data.ts.

• Inbox test: a clear 3-step "how to use" on the tool (copy the address → email it → read your score), so a first-time visitor knows exactly what to do.
• Inbox tests now show in the admin "Recent tests" log authoritatively — logged server-side in the ingest endpoint the moment an email is analysed (tool inbox-test, score, sender as the target), so they appear even if the visitor closed the tab. Removed the duplicate client-side event.

• Inbox Deliverability Test (/tools/inbox-test) — the "send us an email, get a spam/inbox score" tool (mail-tester style). The visitor gets a unique one-time address (test-<token>@crawlsonar.com); Cloudflare Email Routing forwards the raw message to POST /api/v1/inbox/ingest (authenticated by a shared INBOX_INGEST_SECRET, timing-safe), which parses the receiving MTA's Authentication-Results (SPF/DKIM/DMARC verdicts), extracts the sending IP, resolves reverse DNS, checks a couple of bulk-sender signals, and stores a scored report in Supabase keyed by the token. The page mints the address and auto-polls until the email lands, then shows the inbox-readiness score with fixes and Learn links. — New endpoints: POST /api/v1/inbox/new, POST /api/v1/inbox/ingest, GET /api/v1/inbox/result/[token]. Pure, unit-tested MIME-header parsing + scoring (inbox.ts, 7 tests, 407 total). Hex token survives MTA lowercasing. Dormant-until-configured (needs Supabase + INBOX_INGEST_SECRET). First revenue-shaped tool (free now; credit-gating to follow with the paid OG tier).

• Full target coverage in the admin "Recent tests" log. The target domain/URL is now recorded for every domain/URL tool (18 clients: DNS, SSL, HTTP headers, robots, sitemap, performance, link-preview, security scanner, structured data, CT subdomains, redirect, DNS focus, domain audit, plus the shared ReportScanClient covering SEO/WHOIS/deliverability/etc., and the earlier HSTS/DNSSEC/DKIM/Agent Readiness). Tools without a single domain target by nature (secret/hex generator, user-agent parser, log analyzer, email-header paste) correctly show none. So the admin audit log now answers "who tested what & when" with the actual target.

• Web Bot Auth — signed outbound crawler requests (RFC 9421 HTTP Message Signatures). When a signing key is configured, safeFetch now signs every hop of our outbound requests with an Ed25519 signature (Signature / Signature-Input / Signature-Agent headers, tag="web-bot-auth"), so the sites we scan can cryptographically verify the request genuinely came from Crawlsonar rather than an impersonator. The matching public key is published at /.well-known/http-message-signatures-directory (JWKS). — Dormant-until-configured: set WEB_BOT_AUTH_PRIVATE_JWK (an Ed25519 private JWK, server-only secret) to enable; with no key we don't sign and the directory 404s — zero overhead and safe to ship. — Pure, unit-tested core (web-bot-auth.ts): RFC 9421 signature-base construction, RFC 7638 JWK thumbprint key id, and a full sign→verify round-trip test (generate an Ed25519 key, sign a request, verify the signature against the published public key). 5 tests (400 total). Completes the honestly-achievable agent-readiness set.

• Admin: "Recent tests — who, what & when" audit log. The admin dashboard now shows a live table of individual test runs across every tool: when (timestamp), what (tool + target domain), who and the score. Captured server-side on the events pipeline, so it covers all 24 tools automatically. — Privacy-respecting "who" (new who.ts, unit-tested): we store the two-letter country (from the CDN geo header) and a short salted one-way hash of the IP (cid) — never a raw IP or user agent, keeping the cookieless/no-PII discipline while still letting the owner tell repeat visitors and abuse apart. — Target domain is recorded for the scans that report it (HSTS, DNSSEC, DKIM, Agent Readiness now; more to follow); who/when/tool is captured for every tool. Durable on Supabase.

• Markdown content-negotiation — an AI agent that requests the homepage with Accept: text/markdown now gets a clean, link-rich markdown briefing (generated from the tool registry at /api/markdown) instead of the full HTML. Wired into the existing origin proxy, scoped to / and gated on the Accept header, so ordinary browsers (which send text/html) are completely unaffected. Completes another agent-readiness signal — the last of the honestly-achievable ones before Web Bot Auth.

• Agent Skills discovery index at /.well-known/agent-skills/index.json (Agent Skills Discovery RFC v0.2.0). Generated live from the public API tool registry: every check is a "skill" an agent can invoke via POST /api/v1/run, each with a genuine sha256 digest of its descriptor (not a placeholder) and clear invocation instructions (HTTP + MCP). Part of making crawlsonar itself agent-ready.

• Supabase (Postgres) as the durable-store backend — src/lib/kv.ts now speaks to two interchangeable backends behind one interface, chosen by env, with Supabase preferred: set SUPABASE_URL + SUPABASE_SECRET_KEY (the new sb_secret_… key; legacy SUPABASE_SERVICE_ROLE_KEY also accepted) and every dormant feature (shareable reports, benchmark histogram, monitoring, API keys, AI cache) lights up. Upstash/Vercel KV (Redis REST) remains as the alternative; with neither set, the app falls back to the filesystem stores (local dev unchanged). — Redis-style ops map onto three tables + atomic RPC functions: kv_store (SET/GET/INCR + TTL, with expiry-aware counter reset), kv_hash (HINCRBY/HGETALL), kv_list (RPUSH/LTRIM/LRANGE, negative-index aware). SQL in supabase/0001_kv_store.sql (+ consolidated Total_incorporating_0001.sql): RLS enabled on every table with no policies (only the secret key / service_role bypasses), and RPC EXECUTE revoked from anon/authenticated — least privilege by default. — The Upstash wire format is byte-for-byte unchanged; +8 unit tests cover the new Supabase path and backend precedence (390 total). Dormant-safe: shipping this changes nothing until the Supabase env vars are set.

• MCP Server Card at the SEP path — crawlsonar now also serves its MCP discovery manifest at /.well-known/mcp/server-card.json (SEP-1649 shape: serverInfo, transport, capabilities), in addition to /.well-known/mcp.json, so agents and audits that look for either convention find it. The Agent Readiness Checker now probes both paths too.

• Random Hex & Secret Generator (/tools/secret-generator) — generate a cryptographically secure random secret with a user-set length (1–512 bytes) in hex, base64url or base64, with an optional uppercase toggle. Uses the Web Crypto API (crypto.getRandomValues) entirely client-side — nothing is sent to a server. Shows entropy (bits) and character count; one-click copy and regenerate. Live-verified: 32 bytes → 64 hex chars, length control works. Wired into homepage search, sitemap, /tools and llms.txt.

• Dogfooding: made Crawlsonar itself agent-ready — added the real, standards-based discovery surface our own Agent Readiness Checker probes (no faking: OAuth is intentionally omitted because we use API-key auth, not OAuth): — /.well-known/security.txt (RFC 9116) — machine-readable security contact. — /.well-known/mcp.json — a manifest describing our live MCP endpoint (/api/mcp, streamable-http, protocol 2025-06-18). — /openapi.json — a compact OpenAPI 3.1 description of the public API (/api/v1/run, /api/v1/keys, /api/mcp). — /.well-known/api-catalog (RFC 9727 linkset) pointing at the OpenAPI + docs. — Discovery Link headers on the homepage response (rel="describedby" → llms.txt, rel="service-desc" → OpenAPI, rel="api-catalog"), emitted from next.config headers. — Together with the existing llms.txt and explicit AI-crawler rules, this takes crawlsonar.com's own Agent Readiness score from failing to a strong pass — the tool now practises what it preaches.

• OG Image Generator (/tools/og-image-generator) — a free tool that renders a crisp 1200×630 social-share PNG from a title, subtitle, brand and one of six themes, server-side via next/og (/api/og-image). Instant, no sign-up, no watermark, zero marginal cost. Live preview + one-click PNG download. Live-verified: the endpoint returns a valid 1200×630 image/png. Wired into homepage search, sitemap, /tools and llms.txt. — Sets up the paid tier: the page carries a dormant "AI artwork — coming soon" upsell. The paid path (AI-generated backgrounds via OpenAI gpt-image-1, sold as one-off credit packs through Stripe with a KV credit ledger) is deferred until KV + Stripe + OPENAI_API_KEY are configured — money code shouldn't ship untested against real keys.

• "Learn" — a plain-English knowledge base (/learn). Every concept behind our checks, explained twice on its own SEO page: once for a total beginner (with an everyday analogy), once for a professional, plus how-to-fix steps, an FAQ and links to the relevant tools. 13 launch entries covering HSTS, SPF, DKIM, DMARC, DNSSEC, HTTP security headers, structured data, canonical URLs, meta descriptions, AI crawlers & robots.txt, llms.txt, crawler/WAF bot-blocking and SSL/TLS. Each page emits DefinedTerm + FAQPage + BreadcrumbList JSON-LD; the index emits DefinedTermSet. Wired into the header nav, sitemap and llms.txt.
• "Learn what it means, in plain English →" links on every issue. The shared IssueList now maps each finding to its knowledge-base entry (learnSlugForIssue), so any issue we can explain automatically gets a beginner-friendly Learn link — across every tool at once. For laypeople building their first site, not just pros.
• New data-driven kb.ts (entries + issue→slug resolver, 6 unit tests, 382 total). Easily extensible — adding a topic is pure content.

• Shareable report links no longer mint dead URLs when the report database isn't connected. On Vercel's per-invocation serverless filesystem (no KV), a created share link would 404 from another instance. The share API now detects that persistence isn't durable and returns a clear "shareable links are being switched on" message instead of a broken link; the Share button shows it gracefully. Activates fully the moment Upstash KV is configured. (Local dev still works via the filesystem store.)

• Agent Readiness Checker (/tools/agent-readiness) — a new standalone tool for the emerging "agentic web": where the AI Readiness Checker scores how an LLM reads a *page*, this probes the origin-level discovery surface autonomous AI agents use. Nine real, observable signals: llms.txt, explicit AI-crawler rules in robots.txt, Markdown content-negotiation (Accept: text/markdown), an MCP manifest (/.well-known/mcp.json or /mcp), OpenAPI / API-catalog description, OAuth discovery metadata (/.well-known/oauth-authorization-server / -protected-resource), discovery Link headers, and security.txt (RFC 9116). We only score signals we can actually observe over HTTP/DNS — speculative agentic-commerce / bot-auth proposals are deliberately left out so the score stays honest. — Pure buildAgentReadinessReport + SSRF-safe checkAgentReadiness (parallel probes, never throws). 5 unit tests (376 total). Wired into the homepage search, public API + MCP (agent-readiness), sitemap, llms.txt and /tools. Live-verified: github.com correctly detects its security.txt; crawlsonar.com detects llms.txt + 9 AI-crawler rules.

• False negatives when a site is behind a bot wall / WAF, and the "dates/privacy" checks. Triggered by a real report on www.ukimmigration.law (privacy/terms and publication dates flagged missing when they exist): — Bot-wall detection (new bot-wall.ts, unit-tested) — when a site sits behind Cloudflare Bot Fight Mode / a managed challenge (often served as HTTP 200 with an interstitial) or another WAF (Akamai, Imperva, PerimeterX, DataDome, Sucuri), our crawler was parsing the challenge page and reporting everything as missing. The AI Readiness Checker now detects the block, marks the unreadable content checks not-applicable instead of failed, and raises one headline finding: *"Your site is blocking automated crawlers"* — with the key insight that the same block hits GPTBot / ClaudeBot / PerplexityBot / Google-Extended / Googlebot, so the content can be invisible to AI and search. Fix guidance points at Cloudflare "Verified bots" / WAF allow rules. — Privacy/terms detection broadened — now matches the link text as well as the href, across privacy / terms / cookie(s) / GDPR / legal / disclaimer / data-protection / impressum. (Fixed the ukimmigration.law false negative.) — Dates detection broadened — the check is about dates being *visible*, so it now also accepts a clearly-visible date in the text ("8 July 2026", "July 8, 2026", "2026-07-08"), plus article:published_time / article:modified_time / itemprop date meta, not only <time> and JSON-LD. — Live-verified on www.ukimmigration.law: privacy + dates now pass (100/A); +7 tests (371 total).

• Plain-language layer + "Fix-it" everywhere — for millions, not geeks. Two cross-cutting UX changes to the shared report components, so they land on *every* tool at once (the SEO checker, security scanner, HSTS, DNSSEC, audit — anything using ScoreCard/IssueList): — Plain-English verdict under every score: one honest, non-alarmist line that tells a non-technical visitor what the number means (e.g. *"In plain English: this needs attention — several important things are failing below."* vs *"…this looks healthy — no urgent problems found."*). — One-click fixes — issue Fix text now renders backtick snippets as real inline code, and any fix that contains a copy-pasteable snippet gets a "Copy fix" button (plus a Copy button on example blocks). One consistent affordance across the whole site via a new shared <CopyButton>. — New pure, unit-tested report-format.ts (plainVerdict, splitInlineCode, firstCodeSnippet; 7 tests). Live-verified: HSTS on example.com → Grade F, plain verdict shown, "Copy fix" button on the missing-HSTS fix.

• Score benchmarking — the first KV growth loop (moat + linkbait). Every completed scan now contributes its 0–100 score to a per-tool histogram in KV, and the report shows the visitor where they land: *"Better than X% of sites we've checked with this tool · based on N scans."* The distribution improves with every scan (a data moat that can't be copied) and a top-decile result is eminently shareable. — Privacy by design: we store only a tool id and an integer score bucket (bench:v1:<tool>, fields "0".."100") — never the domain, IP or any identifier. Nothing to leak. — Dormant-until-configured: with no Upstash/KV set, the endpoint returns { enabled: false } and the badge renders nothing — so it's invisible and harmless until KV is wired (activates the moment KV_REST_API_URL/KV_REST_API_TOKEN are set). Percentile is also hidden below a 30-scan minimum so it's never misleading. — New pure computePercentile() (unit-tested, 7 tests), recordAndRank()/getRank() (never throw), POST /api/v1/benchmark, and a reusable <BenchmarkBadge> wired into the Domain Audit, HSTS Checker and DNSSEC Checker. Added kvHincrby/kvHgetall to the KV client.

• Structured-data (schema.org) presence check in the On-Page SEO Checker. The SEO checker now flags whether a page has any structured data — JSON-LD, microdata (itemscope/itemtype) or RDFa (typeof) — so a missing-schema page is caught in the ordinary SEO audit, with the detected @types shown as evidence and a fix that links to the JSON-LD Generator and Structured Data Checker. Also flags invalid JSON-LD blocks. Live-verified: crawlsonar.com → passed (Organization, WebSite); example.com → failed (no structured data). Reuses the already-tested parseJsonLd; feeds the /tools/seo-checker tool and the domain audit.

• HSTS Checker (/tools/hsts-checker) — a dedicated Strict-Transport-Security tool. Fetches the site over HTTPS, parses max-age / includeSubDomains / preload, and queries the official hstspreload.org status API to report whether the domain is actually on the browser preload list. Catches the real footguns: max-age=0 (HSTS disabled), a short lifetime, missing includeSubDomains, a preload directive that isn't eligible, "eligible but not submitted", and — importantly — "on the preload list but the live header no longer qualifies". Pure parseHsts + buildHstsReport, unit-tested; live-verified github.com = 100/A, on the preload list.
• DNSSEC Checker (/tools/dnssec-checker) — validates the full chain of trust (root → TLD → zone) over DNS-over-HTTPS. Shows the three links (zone signed via DNSKEY, anchored via DS in the parent, validates to root via the resolver AD flag) and flags the dangerous states that quietly take domains offline: DS with no DNSKEY (a live resolution outage — critical), bogus (DS/DNSKEY mismatch after a key rollover — high), island of security (signed but not anchored — medium), and deprecated crypto (RSASHA1 / SHA-1 DS). Pure parsers + buildDnssecReport, unit-tested; live-verified cloudflare.com = 100/A.
• Both tools are wired into the homepage search, the public API + MCP (hsts, dnssec tool ids), sitemap.xml, llms.txt and the /tools index.
• 27 new unit tests (HSTS 13, DNSSEC 14). 349 total.

• DNSSEC algorithm parsing — Cloudflare's DoH renders the signing algorithm as a mnemonic name (e.g. 256 3 ECDSAP256SHA256 …), not a number. The first cut parsed it as a number, dropped every DNSKEY, and falsely reported healthy domains (cloudflare.com!) as a "critical broken chain". algNumber() now accepts both the mnemonic and numeric forms; caught in live verification before shipping, with a regression test added.

• Rich, option-by-option educational content on every generator — expanded the below-the-fold SEO/GEO narrative on all 9 generator tool pages so each one explains *what the thing is*, documents *every option*, and warns of the common mistakes. This deepens topical authority (each page now targets the long-tail "what does X option do" queries) and helps users configure correctly the first time: — Security Headers Generator — HSTS (max-age / includeSubDomains / preload), CSP presets (strict / basic / none) + report-only rollout + nonce path, X-Frame-Options (DENY / SAMEORIGIN / none), nosniff, Referrer-Policy, Permissions-Policy. — SPF Generator — a, mx, ip4/ip6, include, and -all/~all/?all/+all, plus the one-record and 10-lookup rules. — DMARC Generator — p, sp, pct, rua/ruf, adkim/aspf, and the safe three-stage rollout. — JSON-LD (Schema) Generator — when to use Organization / WebSite / Article / LocalBusiness / Product, and the visible-content rule. — Open Graph Generator — every og:/twitter: field, image sizing (1200×630), and cache/absolute-URL pitfalls. — CAA Generator — CA issue list, issuewild, and iodef reporting. — MTA-STS & TLS-RPT Generator — mode (testing / enforce / none), MX hosts, max_age, TLS-RPT rua, and how to publish the policy. — AI robots.txt Generator — the three presets (protect-training / allow-all / block-all) and the per-crawler allow/block toggles + opt-out tokens. — llms.txt Generator — what llms.txt is and how the draft is built.
• Content-only change (no logic touched): build compiles, all 323 tests pass, generator pages remain statically pre-rendered.

• Footer version showed "· v" with no number. Next 16 does not inline env: { NEXT_PUBLIC_APP_VERSION } into the statically prerendered server-component layout, so the value was empty in production. Replaced the env/process.env indirection with a deterministic generated literal (src/lib/app-version.ts), imported by the layout — works identically across static prerender, dynamic render and Vercel serverless. Removed the now-dead fs read + env block from next.config.ts. Verified · v0.49.0 live.

• Homepage tool search — a type-to-find search box above "Popular tools" that narrows suggestions as you type across all 40+ tools, with keyword matching (type dkim, hsts, spf, core web vitals…), keyboard navigation and Enter-to-open. Backed by a new canonical src/lib/tools-index.ts (single searchable index with keywords). Makes the growing toolset discoverable.
• DKIM Checker (/tools/dkim-checker) — the missing third of the SPF/DKIM/DMARC trio as a standalone tool. Looks up a domain's DKIM public key by selector (or probes the common provider selectors), validates it, and reports key type, estimated strength (flags weak ~1024-bit RSA), and testing/revoked flags — with fixes. Pure record parser + report builder, unit-tested. Live-verified against github.com (selector google, RSA, strong). — 11 new unit tests (DKIM parsing/scoring + tool-search ranking). 323 total. Build compiles. HSTS was already covered by the HTTP Header Checker + Security Headers Generator (now instantly findable via search).

• Fix Pack (/tools/fix-pack) — the "don't just find problems, fix them" tool. Enter a domain and get copy-paste-ready, pre-filled snippets for the most common SEO/AI/security gaps: HTTP security headers, an AI-crawler robots.txt policy, SPF + DMARC anti-spoofing records, Organization JSON-LD and an llms.txt skeleton — each with where-to-apply guidance and a copy button. Fully client-side (reuses the pure generator libs), so it's instant and private. Live-verified: 6 blocks generate pre-filled for the domain. Registered in /tools, sitemap, llms.txt.

• MCP server — Crawlsonar is now agent-native. A zero-dependency Model Context Protocol server at POST /api/mcp (Streamable HTTP, JSON-RPC 2.0) exposes all 13 registry tools (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview) as agent-callable tools — so an AI assistant can audit a domain itself. Reuses the shared tool registry, so MCP, the public API and the UI run one implementation. initialize / tools/list / tools/call / ping / notifications handled; SSRF-safe; rate-limited. Documented on /api-docs and in llms.txt. — 7 new unit tests over the pure JSON-RPC router. 312 total. Live-verified: full handshake + a real tools/call (whois google.com → registrar + score via MCP).

• Explicit AI-crawler policy in robots.txt. Added a dedicated rule allowing every known AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, CCBot, Google-Extended and more) — we *want* to be readable and citable by AI (GEO), so we opt in explicitly rather than relying on the wildcard. Also added the host directive and kept /monitor/ out of crawling. Resolves the "no explicit AI crawler policy" finding from the self-audit and lifts the AI Readiness + robots.txt module scores.

• Persuasion-optimized default OG image. Redesigned the social-share card for click-through using honest cognitive-bias techniques: a curiosity gap (a big "?" as "your AI score" — the number you don't yet know), loss framing ("invisible to AI?" in alarm-amber beats "readable by AI"), concrete numbers ("40+ free checks"), recognisable authority (ChatGPT, Claude, Perplexity) and a "Free" anchor. No fake social proof or scarcity — on-brand for an honest-audit product. Renders via next/og (verified 200 PNG).

• App version in the footer — the running version (from package.json, exposed via NEXT_PUBLIC_APP_VERSION in next.config.ts) now shows as · v<x.y.z> in the global footer, so it's obvious which build is live.

• Completed the FAQ rollout — added ToolFaq (visible accordion + FAQPage structured data) to the remaining 24 tool pages, so every tool page now has an FAQ (only the /tools index doesn't, as it's a listing). Better AEO/rich-result coverage and thicker content on the lighter utility/generator pages. 305 tests green; build compiles; live-verified schema + visible FAQ render (incl. escaped-quote content).

• Restored static rendering. Reverted the per-request nonce CSP (0.43.0) and the force-dynamic root layout back to a static CSP baseline, so tool pages and guides are prerendered/CDN-cacheable again — build went from 44 dynamic / 2 static to ~29 static / 20 dynamic (the remainder are API routes and genuinely dynamic pages). Recovers TTFB, LCP and crawl efficiency; the static CSP keeps frame-ancestors 'none', object-src 'none' etc. (the app has no script-injection sink, so 'unsafe-inline' is an acceptable trade for static generation).
• Site-level structured data. Added global Organization + WebSite JSON-LD (with SearchAction pointing at the domain audit) in the root layout — the brand entity search and AI systems need. Previously the site had none.
• Default social card. Added a branded 1200×630 opengraph-image (auto-applied to every page) plus a default twitter:card = summary_large_image and og:site_name. The site no longer ships imageless share links.
• Icons & manifest. Added dynamic icon, apple-icon and manifest.webmanifest; removed the leftover create-next-app files (next.svg, vercel.svg, file/globe/window.svg) from public/.
• Titles. Trimmed 15 page titles that exceeded ~60 characters so the keyword-rich part isn't truncated in search results.
• Homepage metadata. Added a canonical, a keyword-rich description and Open Graph for / (it previously only inherited the layout defaults).
• FAQ structured data. New self-contained ToolFaq component (visible accordion + FAQPage JSON-LD); added FAQs to 16 key tool pages (audit, AI Visibility/Readiness, deliverability, CWV, SSL, DNS, HTTP headers, security, SPF, DMARC, WHOIS, subdomains, link preview, robots, sitemap). toolJsonLd already emitted BreadcrumbList for every tool. — 305 tests green; production build compiles; verified live: homepage emits Organization+WebSite+SearchAction, OG image serves a 200 PNG, tool pages carry FAQPage schema + visible FAQ.
• Not code-fixable (operational): the only deployment is behind Vercel SSO and crawlsonar.com isn't connected yet, so nothing is indexable until a public production deploy; wire Google Search Console + analytics after launch.

• Guides rebuilt into a comprehensive knowledge base — the go-to reference behind the tools, engineered to be maximally search- and AI-friendly. Replaced the 4 hand-written guide pages with a data-driven engine: each guide is a structured object (src/content/guides/*.ts) rendered by one template (src/components/guide-article.tsx) that auto-generates, for every guide: — Structured data: TechArticle + BreadcrumbList + FAQPage + optional HowTo JSON-LD (verified: the GEO pillar emits all four). — An anchored table of contents, reading time, category, published/updated dates, semantic headings with deep-link anchors, callouts, tables, code and step lists, an FAQ accordion, and dense internal linking to related tools and guides.
• 15 in-depth guides across 6 topics (up from 4): AI readiness & GEO (Generative Engine Optimization, llms.txt, SSR-for-AI, controlling AI crawlers), Email (deliverability pillar, SPF, DMARC, DKIM), SEO (structured data, Open Graph), Security (security headers, CSP, SSL errors), DNS (records explained, DNSSEC), Performance (Core Web Vitals). Each ends with the free tool to check your own site.
• A categorized /guides hub with CollectionPage JSON-LD, and the full set surfaced in sitemap.xml (with real updated dates) and llms.txt (categorized) for AI discovery. — Build compiles (52 routes); 305 tests green; /guides/<slug> renders with TOC + FAQ + 4 schemas, /guides lists all 15, unknown slugs 404. Old static guide routes removed (migrated into data).

• Strict, nonce-based CSP — completed the hardening flagged in 0.42.0. src/proxy.ts now issues a fresh per-request nonce and sets script-src 'self' 'nonce-…' 'strict-dynamic', so 'unsafe-inline' (and 'unsafe-eval') are gone from script-src in production. Next.js reads the nonce from the request CSP header and stamps it on its own inline scripts. application/ld+json is data (not gated by script-src), so structured data stays intact with no nonce. The five static headers (HSTS, nosniff, X-Frame-Options, Referrer-Policy, Permissions-Policy) remain in next.config.ts. — Verified in a production build (next start): CSP nonce and the nonce on Next's scripts match within a single request, no 'unsafe-inline'/'unsafe-eval', JSON-LD present, client hydration + fetches work. 305 tests green. — Trade-off (deliberate): a per-request nonce can't live in a prerendered page, so the root layout is now export const dynamic = "force-dynamic" — the whole app renders at request time (no static/CDN HTML caching). For a free-tools SEO site that's a real speed/cost cost for a defense-in-depth gain (the app has no script-injection sink — React escapes all input; the only dangerouslySetInnerHTML is our own JSON-LD). To restore static generation, revert this commit (drops back to the 0.42.0 'unsafe-inline' baseline).

• The app now sends its own security headers on every response (previously it had the header *generator/checker* for users but shipped none itself — it would have failed the audit it runs on others). Added via next.config.ts headers(): Content-Security-Policy, Strict-Transport-Security (2y, includeSubDomains), X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: strict-origin-when-cross-origin, a locked-down Permissions-Policy, and X-DNS-Prefetch-Control. — CSP is tuned to the app: frame-ancestors 'none', object-src 'none', base-uri 'self', form-action 'self', img-src 'self' data: https: (the Link Preview tool renders remote og:images), connect-src 'self', upgrade-insecure-requests. script-src/style-src keep 'unsafe-inline' because every tool page emits inline JSON-LD (structured data is core to the product) and Next hydrates inline; dev also allows 'unsafe-eval' + ws: for HMR. — Verified live: all six headers present, no CSP console violations, client fetches (connect-src 'self'), hydration and inline JSON-LD all intact. — Follow-up (tracked): tighten script-src to a per-request nonce (middleware-injected) to drop 'unsafe-inline' — the last hardening step.

• AI Crawler Log Analyzer (/tools/ai-crawler-log) — the observational counterpart to the robots.txt generator: upload or paste a web-server access log to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended and ~15 more) actually visited, how often, and whether they were blocked (403/401) — which quietly makes you invisible to those AI systems. [Tier 2] — Fully client-side (same privacy stance as the Email Header Analyzer): the log is parsed in the browser and never uploaded. Pure analyzeCrawlerLog / parseLogLine (combined-log UA extraction, bot matching, blocked-hit detection, sorting) are unit-tested. 305 total tests. Build 86/86. Live-verified: 4 AI hits across GPTBot (2, incl. one 403), ClaudeBot and PerplexityBot, human traffic ignored.

• Compare Two Websites (/tools/compare) — head-to-head audit: runs the full Domain Audit on two domains in parallel and scores them module by module, highlighting the winner of each check and overall. Turns an abstract score into a competitive to-do list; naturally shareable. Reuses runDomainAudit; pure compareAudits(a, b) diff is unit-tested (per-module winner, ties, ran-only-one-side, overall). [Tier 3] — Route maxDuration = 60, tight rate limit (two audits). 300 total tests. Build 85/85. Live-verified: github.com 74 vs gitlab.com 73 in 3.7s, with a sensible per-module breakdown (ties handled).

• Public developer API — every check from one authenticated endpoint (POST /api/v1/run with { tool, url }), plus a docs page at /api-docs with a one-click free-key generator, curl example, the tool table and limits. Opens integrations, CI and agent workflows (and the paid-tier upsell). [Tier 3] — Accountless API keys: POST /api/v1/keys mints a cs_live_… key, shown once; only its SHA-256 hash is stored (KV in prod, filesystem in dev). Auth via Authorization: Bearer or x-api-key. Per-key daily quota metered in KV (free tier 100/day, API_FREE_DAILY_LIMIT) with x-ratelimit-* headers, plus a per-IP burst cap and a mint cap. — A single tool registry (src/lib/api/registry.ts) maps 13 tool ids to the existing check modules (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview), normalising input per tool — so the API and the UI share one implementation. Typed check errors are mapped to sensible HTTP statuses; every call stays SSRF-safe. — 8 new unit tests (key mint/hash, header extraction, auth/metering round-trip, registry). 297 total. Build 83/83. Live-verified: 401 unauthenticated → mint key → authenticated dns run 200 with rate-limit headers → unknown-tool 400 listing the 13 tools.

• Website & Domain Monitoring (/tools/monitoring) — the Tier-3 recurring-value engine that turns one-off tools into a reason to come back. Create a monitor for a domain and get alerted on uptime, SSL-certificate expiry (14-day warning), domain-registration expiry (30-day warning) or DNS-record change, via email or webhook. Accountless: creating a monitor returns a private manage link (/monitor/{token}), the same capability-URL pattern as shared reports — no auth to build yet. — Cron (/api/v1/cron/monitor, hourly via vercel.json, CRON_SECRET-gated with a timing-safe check) iterates all monitors, re-runs each check, and dispatches alerts. Alerting is a pure decideAlert(type, prevState, prevAlarm, outcome) function (first-run seeding, threshold enter/leave, change detection) — fully unit-tested — so it only fires on a genuine change and again on recovery (no spam). — Checks reuse the existing TLS / WHOIS / DNS / HTTP modules. Store is the same KV-in-prod / filesystem-in-dev dual pattern as reports. Webhooks are SSRF-safe: safeFetch gained POST-body support so user-supplied webhook URLs are delivered through the same public-IP-pinned fetcher (can't be pointed at internal services). Email reuses the dormant delivery pattern (logs intent until a provider is wired). — 9 new unit tests (alert-decision matrix). 289 total. Build 80/80. Live-verified full lifecycle: create → seeded check (example.com SSL "51 days") → view → cron re-check (no false alert) → manage page (noindex) → delete → 404.

• The hourly cron needs a Vercel Pro plan (Hobby caps cron at daily); expiry checks are fine daily, uptime benefits from the hourly cadence. Set CRON_SECRET in production.

• AI Visibility Checker (GEO) (/tools/ai-visibility) — the Tier-2 hero and the product's differentiator. Asks an LLM what it knows about a brand from training, compares that to the brand's own site (fetched for the accuracy check), and returns a 0–100 visibility score + grade, whether it's recognised / accurate / positive, and concrete GEO recommendations to get cited in AI answers. — Margin-protected (global "unit economics for AI features" rule): dormant until ANTHROPIC_API_KEY is set (503 otherwise); cheap Haiku model with small max_tokens; 24h per-domain result cache in KV so repeat scans don't re-bill; a global daily budget counter in KV (AI_VISIBILITY_DAILY_BUDGET, default 500) on top of a hard 3-per-hour per-IP limit. — Honest framing throughout: reflects one model's training knowledge (labelled), not a live crawl of every engine; being unknown is common for smaller brands and is what GEO fixes. LLM output is defensively normalised (enum whitelisting, confidence clamp, recommendation cap) so malformed JSON can't break the report. — Added kvIncr / kvExpire to the KV client for the budget counter. Pure transform + normaliser unit-tested (recognised/unrecognised scoring, finding generation, coercion). 280 total tests. Live-verified dormant path (503 without a key) and page render; the LLM happy-path runs once the key is set in prod.

• Link Preview & Open Graph Checker (/tools/link-preview) — shows exactly how a URL renders as a card on Google, X/Twitter, Facebook/LinkedIn and Slack/Discord, with the Open Graph / Twitter Card fixes to improve it (missing og:image, over-length title/description, missing twitter:card). Reuses the existing HTML parser; extracts OG + Twitter tags, resolves relative image/canonical/favicon URLs to absolute, and renders four faithful preview cards. Viral, shareable, SEO-adjacent. [Tier 1] — SSRF-safe (page fetched via safeFetch). Pure extractPreview(html, url) transform, unit-tested (OG-over-base fallback, relative-URL resolution, warning generation, favicon default). 274 total tests. — Live-verified against github.com (all four cards render, og:image loads, sensible length warnings). Registered in /tools, sitemap, llms.txt.

• DNS propagation tool deferred: a meaningful multi-geo propagation checker needs geographically-distributed resolvers, but DoH-JSON over the SSRF port allowlist (80/443/8080/8443) is effectively only Cloudflare + Google (both global anycast → always agree). Not worth shipping a weak version; revisit with a proper resolver source.

• Subdomain Finder (/tools/subdomain-finder) — passively enumerate a domain's subdomains and certificate issuers from public Certificate Transparency logs (crt.sh), and flag exposed dev/staging/admin/internal hosts. A bespoke discovery UI (stat tiles, issuer chips, highlighted sensitive hostnames) rather than the scored report shape, since this is enumeration not pass/fail. [Tier 1] — SSRF-safe (only outbound call is to crt.sh via safeFetch); passive — never scans or brute-forces the target. Route maxDuration = 30 (crt.sh is slow). Pure entries→report transform, unit-tested (SAN parsing, wildcard stripping, domain scoping, issuer aggregation, sensitive-host flagging). 270 total tests. — Live-verified against anthropic.com: 126 subdomains from 3,798 certs, correctly surfacing staging, test, internal.api and portal hosts. Registered in /tools, sitemap, llms.txt.

• Email Deliverability Test (/tools/email-deliverability) — the sending-authentication trio (SPF, DKIM, DMARC) plus MTA-STS / TLS-RPT / BIMI in one focused, scored report, with the exact fix per failing check. Adds DKIM selector probing (queries ~20 common provider selectors) on top of the SPF/DMARC parsers reused from the DNS module, and weights everything for inbox deliverability (Gmail/Yahoo now require SPF+DKIM+DMARC for bulk senders). [Tier 1] — DNS-only (no SSRF surface). DKIM is selector-based and can't be fully enumerated — the tool probes common selectors and says so, so a custom-selector setup reads as "no DKIM found" with a clear caveat (verified: google.com's rotating date selectors aren't guessable; github.com's google selector is found). — Pure resolved-facts→report transform, unit-tested (healthy vs broken vs no-SPF). 266 total tests. Registered in /tools, sitemap, llms.txt. Live-verified against google.com (82/B) and github.com (91/A).

• WHOIS / RDAP Lookup (/tools/whois-lookup) — registrar, registration/expiry dates, registry status, nameservers and DNSSEC for any domain, via the modern RDAP protocol (structured JSON successor to WHOIS). The actionable win is expiry: flags a domain expiring within 30 days (critical if already lapsed), plus any blocking status (hold / pending delete). Feeds the upcoming expiry monitor. [Tier 1] — SSRF-safe: resolved through the rdap.org redirector using safeFetch, which re-validates every redirect hop's IP, so the TLD-dependent final host can't be abused. Falls back to the registrable domain when a subdomain is entered. — 7 new unit tests over a pure RDAP→report transform + input normaliser — 262 total. Registered in /tools, sitemap, llms.txt. Live-verified against google.com and bbc.co.uk (multi-part .co.uk TLD handled).

• Core Web Vitals Checker (/tools/core-web-vitals) — the biggest missing SEO/perf signal. Powered by Google PageSpeed Insights: real-user field data (CrUX) for LCP / INP / CLS when available, falling back to the Lighthouse lab result for low-traffic sites, plus the overall Lighthouse performance score. Explainable scoring against Google's official "good" thresholds (LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.1), one issue+fix per failing metric. This is the first item of the Tier-1 one-stop-shop expansion. — No SSRF surface (Google fetches the target; our only outbound call is to googleapis.com). Works keyless at low volume; GOOGLE_PSI_API_KEY optional for production reliability. Kept out of the synchronous Domain Audit on purpose — Lighthouse is slow (route maxDuration = 60, tight 6-per-5-min rate limit). — 5 new unit tests over a pure PSI→report transform (field-preferred, lab fallback, INP-not-applicable, severity capping) — 255 total. Registered in /tools, sitemap and llms.txt. Live-verified: graceful 429 handling keyless; page renders.

• Durable serverless store — Vercel-ready. The three filesystem stores (shareable reports, leads, analytics events) now transparently use Upstash Redis in production, since serverless filesystems are ephemeral/read-only. New zero-dependency Upstash REST client (src/lib/kv.ts, explicit JSON (de)serialisation, 5s timeout) behind the existing store interfaces. Report TTL is now native (SET … EX), so shares self-expire with no sweep/cron. The event list is capped (LTRIM) to bound growth. — Dormant-until-configured (same pattern as origin cloaking / email / admin): with no KV_REST_API_URL+KV_REST_API_TOKEN (or UPSTASH_REDIS_REST_URL/TOKEN), the app keeps using the filesystem stores — so local dev needs no Redis and nothing else changed for it. — 11 new unit tests (mocked-fetch REST wiring, env gating, JSON round-trip) — 250 total. Report share round-trip re-verified live over HTTP on the filesystem path.

• next.config.ts: outputFileTracingIncludes force-bundles pdfkit's .afm font-metric files into the PDF route, avoiding the classic ENOENT … Helvetica.afm failure on Vercel.
• .env.example documents the KV vars and that they're required on Vercel.

• The in-memory rate limiter is per-instance on serverless (state isn't shared across Vercel instances) — acceptable at launch; move to Redis if abuse warrants.
• The Domain Audit route already sets maxDuration = 60 (needs a Vercel Pro plan; Hobby caps functions at 60s but with fewer resources).

• Privacy policy: filled the operator's statutory details from Companies House. Northstar Infinity Works Ltd — company number 17326480 (incorporated 7 July 2026), registered office C/O Elsg Ltd, Regus, Building 2, Marlins Meadow, Watford, England, WD18 8YA. Removed the [company number] / [registered office address] placeholders and the publish-blocker TODO, so /privacy is now legally complete for launch.

• Rebrand to Crawlsonar (crawlsonar.com). The product is now Crawlsonar everywhere it matters: metadata title/template, llms.txt, PDF report brand, shared-report Open Graph images, User-Agent strings (Crawlsonar/0.1 (+https://crawlsonar.com; …)), JSON-LD publisher/author, the AI robots.txt generator's output comment, and the privacy policy. Default site URL is now https://crawlsonar.com (still overridable via NEXT_PUBLIC_SITE_URL).
• Global site chrome — added a sticky header (sonar wordmark + Tools / Guides / Domain-audit nav) and a shared footer across every page, so all ~40 tool pages now carry consistent branding and a way back home. Removed the homepage's bespoke footer (the global one replaces it).
• Privacy policy contact set to [email protected]; .env.example example domains updated to crawlsonar.com (site URL, abuse, from-address).

• Share + PDF + email-capture for the Domain Audit — the most shareable artefact is now shareable. The /audit/domain report gets the "🔗 Share this report" button (tokenised /r/{token} link, authentic-by-construction re-run), the PDF lead magnet and email capture, exactly like the other tools. — Extended the report store to a three-way discriminated union (ai-readiness | llms-txt | domain-audit) via a new ReportPayload type; POST /api/v1/reports and POST /api/v1/leads re-run runDomainAudit server-side for audit shares. — PDF renderer now handles the audit shape (overall score/grade header, per-module score breakdown, merged issue list from every module). — Extracted an AuditBody presentational component shared by the live audit page and the read-only /r/{token} page, so the two can't drift.
• 1 new unit test (audit PDF renders a valid %PDF) — 189 total. Verified live end-to-end: shared a github.com audit → /r/{token} renders (200, noindex, audit body) → 5.4 KB PDF with full module breakdown + all 9 issues.

• Full Domain Audit (/audit/domain + POST /api/v1/audit/domain) — the product's headline promise: one input, one coherent report (Requirements v1.0 §1/§7/§8). Classifies the input (domain/URL/email → origin+url) and runs all seven modules in parallel via Promise.allSettled with per-module error isolation (a module that can't run — e.g. TLS on an HTTP-only host — is shown but excluded from the score, so one failure never sinks the report). Aggregates an impact-weighted overall score (AI 25, HTTP 18, TLS 15, DNS 15, robots/sitemap/llms 9 each), severity counts and a merged, severity-sorted issue list; each module card links to its dedicated tool prefilled with the target. Stricter rate limit (5 / 5 min) and maxDuration: 60.
• Homepage secondary CTA to the audit, sitemap + own llms.txt entries. The audit reuses every existing module — no new checks, pure composition.
• 8 new unit tests (input classification, grade mapping, weighted aggregation excluding errored modules, severity tallies) — 188 total. Verified live: github.com 77/B (~2.6s), stripe.com 83/B (~3.4s), all 7 modules running; invalid input → 400.

• robots.txt Checker (/tools/robots-txt-checker + POST /api/v1/tools/robots): reuses the shared robots parser + AI-crawler matrix to detect full-site blocks (critical), missing Sitemap directives, and whether an explicit AI crawler policy exists; shows the parsed AI matrix and raw file. Clarifies robots.txt controls crawling, not indexing.
• Sitemap Checker (/tools/sitemap-checker + POST /api/v1/tools/sitemap): reuses parseSitemapXml to confirm sitemap.xml exists and is valid, counts URLs (following the first child of a sitemap index) and samples up to 6 URLs (SSRF-safe fetch) for reachability.
• Both wired: homepage (all "coming soon" cards now live — 11 tools total), sitemap, own llms.txt, cross-links.
• 4 new unit tests (robots analysis: absent, healthy, full-site block, missing sitemap/AI policy) — 180 total. Verified live: github.com robots 66/C (no sitemap directive, no AI policy), vercel.com sitemap 100/A (500 URLs, 0 broken), sites without a sitemap → no_sitemap. Confirmed SSRF still blocks localhost self-scan.

• SSL / TLS Checker (/tools/ssl-checker + POST /api/v1/tools/ssl). Inspects the certificate: validity window, days-to-expiry (with expiring-soon warnings), hostname match (SAN/CN incl. single-label wildcards), chain trust vs default roots, negotiated protocol (flags < TLS 1.2), key strength, plus the full chain and connected IP. SSRF-safe: resolves the host, validates every IP with isPublicIp, and connects to the pinned IP with servername set for SNI/identity. Pure cert-evaluation helpers (SAN parse, wildcard match, validity) are unit-tested.
• 11 new unit tests — 176 total. Verified live across badssl.com fixtures: cloudflare.com 100/A, expired.badssl.com → cert_expired, wrong.host → cert_hostname_mismatch, self-signed → cert_untrusted.

• Key-strength false positive: ECDSA certificates (e.g. Cloudflare's P-256) were wrongly flagged as weak keys because the check assumed RSA. Now ECDSA keys are recognised as strong; only RSA < 2048 is flagged. Caught during live verification (cloudflare.com went from a false 90 to a correct 100/A).

• HTTP Header Checker (/tools/http-headers-checker + POST /api/v1/tools/http-headers). Through the SSRF-safe fetcher: redirect chain with per-hop status, HTTPS-final + HTTP→HTTPS-upgrade probe, security-header inspection (HSTS, CSP, X-Frame-Options/frame-ancestors, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP/COEP/CORP), cookie-flag analysis (Secure/HttpOnly/SameSite) and server/framework version-leak detection. Header analysis is a pure, unit-tested function. Redirect-chain view + security-header table + issues. Homepage card, sitemap, own llms.txt, cross-links.
• 7 new unit tests (Set-Cookie parsing, secure vs insecure response analysis, cookie flags, long redirect chain) — 165 total. Verified live: github.com 89/B (HSTS+CSP+X-Frame+nosniff+Referrer, flags one insecure cookie); insecure HTTP target correctly scored, and slow targets return a clean 502 on timeout (no crash).

• DNS Lookup (/tools/dns-lookup + POST /api/v1/tools/dns) — first sprint-3 commodity tool (internal-link hub). Resolves A/AAAA/CNAME/MX/TXT/NS/SOA/CAA via node:dns with bounded timeouts, parses SPF (with a 10-DNS-lookup-limit count) and DMARC (policy strength). Issue detection: multiple SPF records (critical), SPF over 10 lookups (high), missing SPF/DMARC (medium), p=none (low), missing CAA (low), no address records (high). Records tables, email-authentication panel, explainable score. Input accepts a bare domain, URL or email address (domain extracted). Homepage card (DNS no longer "coming soon"), sitemap, own llms.txt, cross-links.
• 9 new unit tests (SPF lookup counting, SPF/DMARC parsing incl. case + multiple records) — 158 total. Verified live: github.com 94/A, email-address extraction ([email protected] → stripe.com), invalid input → 400.

• AI robots.txt Generator (/tools/ai-robots-txt-generator) — the actionable other half of the AI Readiness checker's crawler matrix (Requirements v1.0 Appendix B). Per-crawler Allow/Block toggles grouped by purpose (training / AI search / user-triggered), three presets ("Block training, allow AI search" default, "Allow all", "Block all"), live copyable/downloadable robots.txt output. Honest framing baked in: robots.txt is advisory, and Google-Extended/Applebot-Extended are opt-out tokens (noted only when blocked). Pure client-side — no network, no SSRF surface.
• Cross-linked: the AI Readiness no_ai_crawler_policy issue fix now points at this generator; homepage card, sitemap entry, own llms.txt entry, related-tools links.
• 7 new unit tests including a round-trip through the checker's own robots.txt parser proving the generator and checker agree — 149 total. Verified live: presets, per-bot toggles, custom state and opt-out-token note.

• Email-capture + PDF lead magnet (Roadmap v1.1 sprint 2 — the free→paid bridge). "Get this report as a PDF" appears on both live checkers and on shared /r/{token} pages. — Real PDF, no headless browser: src/lib/pdf/report-pdf.ts renders either report type with pdfkit (header, score, category breakdown, severity-coloured issues, full checks list, footer disclaimer). serverExternalPackages: ["pdfkit"] keeps its font metrics loadable at runtime. Downloadable at GET /api/v1/reports/{token}/pdf (token-validated, rate-limited). Verified live: valid 4.3 KB %PDF with correct content-type/disposition. — GDPR-first capture: explicit consent checkbox is mandatory (consent: z.literal(true); submit disabled until email valid + consent checked). Leads stored minimally (email, tool, target, consent timestamp, no IP) in a swappable filesystem LeadStore (.data/leads/leads.jsonl). POST /api/v1/leads re-runs the scan server-side to persist an authentic report, stores the lead, and returns a token + PDF URL so the download works immediately. — Dormant email delivery (src/lib/leads/email.ts): active only when EMAIL_API_KEY + EMAIL_FROM are set (provider send is a launch TODO); until then it logs intent (domain only, never the full address) and the UI honestly offers the download without claiming an inbox delivery.
• 7 new unit tests (PDF buffer validity, email validation, lead store no-IP, dormant-email gating) — 142 total. Verified live end-to-end: consent enforced (400 without), invalid email rejected (400), lead stored without IP, dormant email logged, PDF rendered.

• Shareable reports (Roadmap v1.1 sprint 2 + third-party report policy). "🔗 Share this report" on both checkers creates a tokenised link at /r/{token}. — Authentic by construction: POST /api/v1/reports RE-RUNS the scan server-side rather than accepting client JSON, so a shared report about someone else's site cannot be fabricated. — Security: 128-bit crypto-random base64url tokens; every filesystem path gated by a strict token regex (path-traversal-proof, verified live with ..%2F..%2Fetc%2Fpasswd → 404); 30-day expiry with lazy-delete-on-read; 512 KB size cap; report-creation rate limit (12 / 5 min per IP). — Policy: /r/* pages are noindex, nofollow and disallowed in robots.txt; visible point-in-time disclaimer ("not a security certification"); owner opt-out contact via optional NEXT_PUBLIC_ABUSE_EMAIL; reports store only public target data (no visitor IP/headers).
• Report store (src/lib/reports/) as a swappable ReportStore interface with a filesystem implementation (.data/reports, gitignored). NOTE documented: swap for object store / DB before any serverless deploy (ephemeral FS).
• Shared presentational components (src/components/report/): ScoreCard, IssueList, ChecksList, CategoryBars, CrawlerTable, AiReadinessBody, LlmsTxtBody. Both live checkers refactored to use them, so the live and shared-report views can never drift.
• 7 new unit tests (token validation incl. traversal vectors, store round-trip, expiry deletion) — 135 total. Verified live end-to-end: create link in UI → open /r/{token} → renders, noindex, disclaimer, report_viewed{shared:true} event.

• AI Readiness Checker (/tools/ai-readiness-checker + POST /api/v1/tools/ai-readiness) — the flagship differentiator (Roadmap v1.1 sprint 2). Scores a page across seven weighted categories summing to 100 (Requirements v1.0 §17): crawlability (20), content extraction (20), entity clarity (15), structured data (15), trust signals (10), AI crawler governance (10), agent readiness (10). Per-category breakdown bars, evidence+fix issue cards, an AI crawler policy matrix (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended → allowed/blocked/partial/unspecified from robots.txt), and a markdown extraction preview ("what an LLM sees"). Analysis is on server-rendered HTML only (no JS) — stated honestly in the UI, since that's what crawlers/LLM fetchers see.
• Supporting modules: robots-rules.ts (robots.txt parser with Google-style longest-match path matching, per-agent group selection, wildcard/$ patterns, AI crawler matrix) and html-analysis.ts (main-content extraction + noise ratio + markdown, heading outline, JSON-LD parsing, form-label a11y, link-text quality, alt coverage, page signals).
• Homepage hero now routes to the AI Readiness Checker; new tool card, sitemap entry, own llms.txt entry, and cross-links between all four tools.
• 24 new unit tests (robots matching/matrix, HTML extraction/JSON-LD/a11y). Caught two real bugs before ship: JSON-LD blocks were unreadable because parseHtml dropped <script> text, and a noise-ratio fixture was miscalibrated. 128 tests total. Verified live against MDN (no JSON-LD → schema 0) and stripe.com (WebSite+Organization → schema 15/15).

• llms.txt Generator (/tools/llms-txt-generator + POST /api/v1/tools/llms-txt-generator): builds a draft llms.txt from sitemap.xml (sitemap-index support, 500-URL cap) or homepage nav links as fallback, with H1/summary from homepage title/og/meta description. URL categorisation into Documentation / API / Products / Pricing / Blog / Company / Legal sections; per-section (12) and total (40) link caps keep output curated. Editable textarea, copy + download, "this is a draft — curate it" guidance. Full SEO page template; cross-linked with the llms.txt Checker both ways (checker's missing-llms.txt fix now points at the generator). Verified live: vercel.com (501 sitemap URLs → 40 links, 7 sections) and llmstxt.org via UI.
• 20 new unit tests (sitemap XML parsing, URL categorisation, slug humanisation, section capping, output structure) — 104 total.

• Origin cloaking behind Cloudflare (dormant until launch): src/proxy.ts gate + src/lib/security/origin-auth.ts. When ORIGIN_SHARED_SECRET is set, every request must carry the Cloudflare-injected x-origin-auth header or the app answers a bare 403 — direct hits on the origin get nothing. Constant-time comparison, minimum 32-char secrets, comma-separated dual secrets for zero-downtime rotation. Unset in dev = no-op. Verified live: dormant 200; active → no header 403, wrong header 403, secret #1 200, rotated secret #2 200, API without header 403.
• Rate limiter now keys on cf-connecting-ip when (and only when) origin cloaking is active — behind Cloudflare the leftmost x-forwarded-for value is client-controlled and would allow rate-limit evasion.
• .env.example: ORIGIN_SHARED_SECRET documentation with generation command and launch-time reminder.
• 11 new unit tests (84 total).

Sprint 1 increment 1 (per Roadmap v1.1): secure, instrumented platform skeleton with the first differentiated public tool.

• Next.js 16 + TypeScript app (App Router, Tailwind 4, React 19) — modular monolith per Roadmap v1.1 principle 5.
• SSRF-pinned safe fetcher (src/lib/security/): IP validation blocklist (RFC1918, link-local/cloud-metadata 169.254.0.0/16, CGNAT, multicast/reserved, IPv6 loopback/link-local/ULA/site-local, IPv4-mapped/NAT64/6to4/Teredo embedded forms), validation inside the DNS lookup callback so the validated address IS the connected address (atomic pin — no TOCTOU/rebinding window), whole-fetch refusal when any resolved record is non-public, manual redirect handling with per-hop revalidation, scheme+port allowlists, body-size and time caps. Verified live: metadata IP, decimal-encoded 127.0.0.1 and 127.0.0.1.nip.io rebinding all blocked.
• Common issue schema + explainable scoring (src/lib/issues/): severity/category/impact/evidence/fix per Requirements v1.0 §22; critical/high issues cap module scores; not-applicable checks excluded from scoring.
• llms.txt checker (src/lib/checks/llms-txt.ts + /tools/llms-txt-checker): availability, content-type/HTML-fallback detection, llmstxt.org structure validation (H1, summary blockquote, sections, links), link-liveness sampling, thin-file detection, llms-full.txt check. Full SEO page template: tool above fold, explanation with honest "Google ignores llms.txt" note, example, common problems, FAQ, unique metadata, canonical, OG, JSON-LD (WebApplication + BreadcrumbList).
• API POST /api/v1/tools/llms-txt: zod validation, per-IP rate limiting (10/min anonymous, verified live), no internal error leakage.
• First-party cookieless funnel analytics: scan_started/scan_completed/report_viewed (+ reserved names for the full funnel) via sendBeacon to /api/v1/events; no cookies, no IP, no PII stored.
• Homepage with one-input-first hero routing into the checker; product's own llms.txt, robots.txt, sitemap.xml (dogfooding, launch DoD).
• Tests: 73 unit tests (SSRF IP validation incl. negative vectors, llms.txt parser fixtures, scoring) — all passing; production build clean.

• Privacy & Data Protection Policy (/privacy) — a UK GDPR / Data Protection Act 2018 policy with Northstar Infinity Works Ltd as the named data controller, tailored to what the Service actually processes: cookieless analytics with no IP, client-side-only email-header parsing, consent-based lead capture, transient IP for rate-limiting, 30-day shared reports, processors (hosting / Cloudflare / email provider), international transfers, retention, data-subject rights and the ICO complaint route. Linked from the email-capture consent notice, a new homepage footer, and the sitemap. (Draft — placeholders for company number, registered office and contact email to be filled; legal review recommended before launch.)

• Content-negotiation bug: HTML analysers were served markdown. The safe fetcher's default Accept header preferred text/plain, text/markdown over text/html. On sites that content-negotiate (e.g. AI-optimised builds that serve a clean markdown alternate to markdown-preferring clients — ukimmigration.law does exactly this), our HTML analysers received markdown with no <head>, and wrongly reported missing meta description, missing title and no structured data — penalising sites that are, ironically, *more* AI-ready. Changed the default Accept to prefer HTML (text/html,application/xhtml+xml,…). Affected the AI Readiness, On-Page SEO, Structured Data, Security, Privacy checkers and the llms.txt Generator's homepage-metadata fetch. Found via a real-world scan reported by the user.
• Verified: ukimmigration.law went from a false low score to 89/B with JSON-LD (LegalService + WebSite) correctly detected; the llms.txt checker (which fetches static .txt) is unaffected. 239 tests still pass.

• Admin dashboard (/admin, Requirements v1.0 §28 — "admin scan monitoring"). Token-gated, dormant-until-configured: with no ADMIN_TOKEN the admin area does not exist (routes 404); with a token set it requires sign-in. Security-by-design: httpOnly + secure + sameSite=strict session cookie, constant-time token comparison (reusing the origin-cloaking primitive), rate-limited login (5 / 5 min), noindex + robots-disallowed, and it sits behind origin cloaking when that's enabled. Read-only dashboard showing the funnel (event counts), scans by tool, lead count + recent leads, and shared-report count.
• Analytics events are now persisted to a filesystem store (.data/events/events.jsonl, no IP/PII) in addition to stdout, feeding the dashboard; readLeads added. NOTE documented: swap the filesystem stores for a DB before serverless.
• 4 new unit tests (admin enabled-state + constant-time verification) — 239 total. Verified live end-to-end: 404 without token; 401 on wrong token; login sets the cookie; dashboard renders the persisted funnel data.

• DNSSEC validation + DNSKEY/DS + HTTPS/SVCB records over DNS-over-HTTPS (Requirements v1.0 §10 — completes the record coverage node's resolver can't reach). New src/lib/checks/doh.ts queries Cloudflare DoH (application/dns-json) through the SSRF-safe fetcher; a pure parseDohJson + deriveDnssec are unit-tested. The DNS Lookup now reports DNSSEC status (validated / signed / unsigned, from the Authenticated-Data flag plus DNSKEY/DS presence) and whether an HTTPS/SVCB record exists, both surfaced in the tool and scored (unsigned → a low DNSSEC issue). Verified live: cloudflare.com DNSSEC validated (AD + DNSKEY + DS), HTTPS record present.
• 7 new unit tests — 235 total.

• DNS depth (Requirements v1.0 §10): the DNS Lookup now also detects BIMI, MTA-STS and TLS-RPT records (advanced email hardening), surfaced in the tool's email panel and scored for mail-enabled domains. Verified live: google.com → MTA-STS + TLS-RPT present.
• SSL depth (§12): the SSL Checker now explicitly probes TLS 1.2 and TLS 1.3 support (two extra version-pinned handshakes) and flags a missing TLS 1.3. Shown in the certificate panel. Verified live: cloudflare.com supports both.

• Deeper DNS records (DNSKEY/DS/TLSA/SVCB/HTTPS) and DNSSEC validation need DNS-over-HTTPS (node's resolver can't fetch those types) — parked. IP ASN/geolocation + VPN/Tor detection need a paid data provider (MaxMind/IPinfo). Field Core Web Vitals need a headless browser. OpenAPI checker + llms-full/OpenAPI-starter/MCP generators are lower-value/phase-3.

• Three more generators (Requirements v1.0 Appendix B), pure client-side builders + tests: — Open Graph Generator (/tools/og-tag-generator): og: + twitter: meta tags for rich link previews, XML-escaped. — CAA Generator (/tools/caa-generator): DNS CAA records (issue/issuewild + iodef), including the "disallow all" case. — MTA-STS & TLS-RPT Generator (/tools/mta-sts-generator): the policy file plus both TXT records, with a testing→enforce rollout and a minimum max_age.
• 6 new unit tests — 228 total. All three pages verified 200; wired into the /tools Generators group, sitemap and llms.txt.

• On-Page SEO Checker (/tools/seo-checker, Requirements v1.0 §15): title length, meta description, canonical, indexability, single H1 + heading order, lang, mobile viewport, Open Graph, image alt coverage and anchor-text quality — reusing the shared HTML-analysis helpers, framed for SEO. Verified live: stripe.com 85/B.
• Privacy & Cookie Scanner (/tools/privacy-scanner, Requirements v1.0 §21): third-party tracker detection, cookie Secure/HttpOnly/SameSite flags, privacy-policy link and forms-over-HTTPS — with risk wording, not legal conclusions, and an honest note that consent-timing needs a real browser. Verified live: theguardian.com correctly reports policy link + insecure cookies.
• ReportScanClient — a reusable, RSC-safe generic client (serialisable string props only) for standard {score, issues, checks} tools, used by both new pages. Two thin API routes generated from the shared template. 222 tests (both reuse already-tested analysis helpers; orchestration verified live).

• Website Speed Test (/tools/performance + POST /api/v1/tools/performance, Requirements v1.0 §20 MVP). SSRF-safe timed request (resolve → validate → connect to the pinned IP) captures DNS / TCP / TLS / TTFB / total via socket timing hooks, plus HTML size, compression (gzip/br/zstd), cache-control and CDN detection (Cloudflare/CloudFront/Fastly/Akamai/Vercel/Netlify). Timing waterfall bars + issues (slow TTFB, no compression, large HTML). Field Core Web Vitals (LCP/CLS/INP) are explicitly out of scope — they need a real browser (Phase 2, noted in the UI). Verified live: cloudflare.com TTFB 141ms (CDN detected), github.com 164ms (gzip).

• Structured Data Checker (/tools/structured-data-checker + POST /api/v1/tools/structured-data, Requirements v1.0 §16): fetches a page, extracts and validates all JSON-LD (reusing the AI-readiness parser), lists the schema.org types, flags invalid JSON, and notes Microdata/RDFa presence. Verified live: stripe.com 100/A (WebSite + Organization).
• JSON-LD Generator (/tools/schema-generator, Appendix B): pure client-side builder for Organization, WebSite (with SearchAction), Article, LocalBusiness (PostalAddress) and Product (Offer) from a form; omits empty fields, outputs a ready <script type="application/ld+json"> block. Cross-linked with the checker and the structured-data guide.
• 5 new unit tests (JSON-LD building per type incl. conditional Offer/address, valid-JSON render) — 222 total.

• Website Security Scanner (/tools/security-scanner + POST /api/v1/tools/security-scan, Requirements v1.0 §19). Passive checks plus a few light probes of well-known paths, all through the SSRF-safe fetcher with a tighter rate limit (8/min): exposed .git (critical) and .env (critical), mixed content on HTTPS, CORS wildcard-with-credentials, server/framework version leakage, security.txt presence, WordPress REST exposure, and third-party tracker detection (GA/GTM/Meta/TikTok/Hotjar/LinkedIn). UI and AUP disclaim the light active probing ("only scan sites you own or are authorised to test"). Pure findMixedContent + detectTrackers helpers unit-tested.
• 4 new unit tests — 217 total. Verified live: github.com 100/A (clean, has security.txt); a tracker-heavy site correctly flagged Google Analytics + missing security.txt.

• Guides / fix-content (Tier 3 — topic clusters + AdSense prep). New /guides index plus four written, cross-linked articles for the most common issues our tools flag: fixing multiple SPF records, adding HTTP security headers (HSTS/CSP), adding structured data (JSON-LD), and server-side rendering for AI. Each guide has Article JSON-LD, unique metadata, and links to the relevant tool/generator; a shared GuideLayout keeps them consistent. Homepage, sitemap (index + articles) and own llms.txt updated. All 5 pages statically generated and verified 200.

• Embeddable AI Readiness badge (Tier 3 — backlink growth loop). /tools/badge runs a full audit, then hands you HTML and Markdown snippets for a shields-style score badge that links back to a live audit of your domain — the classic SSL-Labs/GTmetrix backlink + brand loop. The badge is a lightweight SVG served from GET /api/v1/badge?label&score&grade, coloured by score and hard-cached (content fully determined by query params). Inputs are sanitised and clamped (score 0–100, label stripped, XML-escaped) to prevent SVG/markup injection — verified live: <script> label → "script", score 999 → 100.
• Added to the /tools Utilities group and sitemap. Verified live: valid SVG renders as a green "AI Readiness 83/100 (B)" badge; /tools/badge returns 200.

• Generators pack (Requirements v1.0 Appendix B) — three client-side, deterministic generators (pure builder libs, no network): — Security Headers Generator (/tools/security-headers-generator): HSTS (max-age/includeSubDomains/preload), CSP preset (strict/basic), X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy, output as raw headers or nginx/Apache config. — SPF Generator (/tools/spf-generator): builds a single v=spf1 record from mx/a/ip4/ip6/includes with a live 10-DNS-lookup-limit count. — DMARC Generator (/tools/dmarc-generator): policy, subdomain policy, pct, rua/ruf, adkim/aspf alignment.
• Each cross-links to its matching checker; new Generators group on /tools; sitemap + own llms.txt updated.
• 7 new unit tests (header building incl. nginx/apache render, SPF assembly + lookup count, DMARC assembly) — 213 total. Verified live: Security Headers Generator produces correct HSTS/CSP/X-Frame output and switches to nginx add_header … always; format.

• Email Header Analyzer (/tools/email-header-analyzer, Requirements v1.0 §11) — paste raw email headers to see SPF/DKIM/DMARC pass/fail (from Authentication-Results), the delivery path with per-hop delays, sending IP and core message fields, with issues and fixes ("why did my email go to spam"). Fully client-side — the pure src/lib/email-headers.ts parser (header unfolding, auth-result extraction, Received-hop ordering + delay calc) runs in the browser, so pasted headers never reach our server (§11's "never expose pasted headers", taken to its logical end). Cross-links to the SPF/DMARC checkers.
• 7 new unit tests (header unfolding + body cut-off, auth-result parsing incl. softfail, hop ordering/delay/sending-IP, full analysis pass/fail cases) — 206 total. Verified live: failing SPF/DMARC headers score 45/D with correct issues and sending IP.

• 8 programmatic SEO landing pages (Requirements v1.0 Appendix A), each targeting a specific high-volume keyword and reusing an existing engine — no new checks, just focused views + tailored content: — SPF Checker, DMARC Checker, MX Lookup, CAA Checker — a shared DnsFocusClient renders only the relevant slice of the DNS report (record + focused issues) and links to the full DNS report. — Redirect Checker — reuses the HTTP-headers engine to show just the redirect chain. — User-Agent Parser — fully client-side (parseUserAgent extracted to a Node-free src/lib/user-agent.ts), decodes browser/OS/device with a bot flag. — What Is My IP — dedicated page around the IP widget.
• /tools index page grouping every tool (Full audit / AI & SEO / DNS & email / Security & HTTP / Utilities). Homepage now links to it; sitemap rewritten data-driven with all 19 routes; own llms.txt lists the new tools; shared toolJsonLd helper for landing-page structured data.
• Verified live: all 8 new pages return 200; SPF Checker shows github.com's record with the 8/10-lookup count. 199 tests still green (new pages are presentational over tested engines).

• Dynamic Open Graph images for shared reports — every /r/{token} link now unfurls as a branded score card (tool, target, big score, grade, issue count, accent-coloured by score) on X / Slack / LinkedIn / WhatsApp. Multiplies the reach of the existing share mechanic — more unfurl clicks → more scans → more email capture. Rendered via next/og (opengraph-image.tsx, Node runtime so it can read the report store); og:image meta auto-wired plus twitter: summary_large_image. Works despite the page being noindex (unfurls ignore indexing). Verified live: 1200×630 PNG, HTTP 200, correct score/grade/issues.

• "Your IP address" widget on the homepage (Requirements v1.0 §9). Shows the visitor's public IP (with IPv4/IPv6), parsed browser + OS, and a copy button. Reads the IP from proxy headers (cf-connecting-ip first — trustworthy behind our Cloudflare proxy — then x-real-ip/x-forwarded-for), so it shows the real public IP in production; on a direct/localhost connection it honestly notes the loopback/private address instead of pretending. GET /api/v1/ip endpoint. Pure pickClientIp + parseUserAgent helpers.
• 10 new unit tests (IP header priority incl. IPv6 and XFF hop selection; user-agent parsing for Chrome/Edge/Firefox/Safari across macOS/Windows/Linux/iOS) — 199 total. Verified live: browser shows its real Chrome/macOS UA, loopback note on localhost, cf-connecting-ip honoured.

• Dev server: switched npm run dev to next dev --webpack. Next 16 makes Turbopack the default, which on this machine fails to generate the app-router route manifest — every app route returned 404 while static public/ files still served. Webpack dev compiles the routes correctly. (Production next build was always fine.)

• Revised roadmap: docs/AI_SEO_Network_Toolbox_Roadmap_v1.1.docx (v1.1, 9 July 2026). Supersedes Section 29 of Requirements v1.0. Key changes: inverted build order (AI-readiness wedge ships in sprints 1–2, commodity tools in 3–4, public launch in sprint 2), paid monitoring pulled forward to sprint 5 with AdSense demoted to a revenue floor after it, plus new sections absent from v1.0 — pricing strategy, measurement plan, SSRF fetcher security gates (IP pinning + expanded blocklist), sensitive-data handling, third-party report policy, and a risk register. Architecture directive changed to modular monolith + Postgres queue (pg-boss) instead of 13 worker microservices + Redis.
• Initialised git repository.
• Product requirements specification: docs/AI_SEO_Network_Toolbox_Requirements.docx (v1.0, 8 July 2026).
• CHANGELOG.md (this file) and .gitignore.