157 releases since 9 July 2026, newest first. Each entry says what changed and why — and what was verified, and what wasn’t. Running version: v1.37.0.
NEXT_PUBLIC_SITE_URL is now https://citaim.com, so canonicals, the sitemap, llms.txt, e-mail links and the magic-link redirect all name the new domain.LEGACY_HOST=crawlsonar.com activates the redirect described above.crawlsonar.com and www.crawlsonar.com to the canonical host, path preserved. Four things depend on the old domain continuing to answer, and none of them survives deletion — the URL inside the crawler's User-Agent that verified-bot programmes matched, the badge embed customers pasted onto their own sites, links in e-mails already sitting in other people's inboxes, and every indexed URL.LEGACY_HOST so it cannot go live before the new domain answers. Enabling it early would make the old domain redirect into nothing, which is the same outage arriving by a different route. Set LEGACY_HOST=crawlsonar.com and deploy once citaim.com is verified.citaim.com in every canonical URL, JSON-LD, sitemap, llms.txt, .well-known catalogue, e-mail copy and addresses, and the Cloudflare watchdog worker. Old CHANGELOG entries are left alone — they record what shipped under the old name, and rewriting them would falsify the history.Crawlsonar/1.0, deliberately. Verified-bot programmes match the full string, so changing it drops that verification until every programme re-registers — weeks, run by other people — and sites that allow-listed the exact string would start refusing our scans with nothing in our logs to explain it. It changes in the same commit that re-registers the bot, not before. The URL inside it must keep resolving, which is one reason crawlsonar.com stays registered and redirecting rather than switched off. Both facts are now pinned by tests..claude/**. Parallel sessions get their own git worktree with its own .next output, and only the repo-root .next was ignored — so generated bundles nobody wrote were producing six errors and five warnings.NEXT_PUBLIC_SITE_URL is unset in production, so the code's fallback literal is what the site actually serves. This release is therefore not deployable until citaim.com answers: deploying it would repoint canonical URLs, the sitemap and every e-mail link at a domain that does not resolve yet.lookupWhois carried a retry commented "Retry once with the last two labels (helps when a subdomain was entered)". It sat inside a catch, so it ran only when the RDAP request *threw* — and the case it was written for does not throw. Measured on 1.33.0: Verisign answers HTTP 404 for blog.crawlsonar.com, fetchRdapAt maps 404 to null (correctly — that is how "not registered" is reported), and the lookup fell through to "No registration record found" in 387 ms without ever retrying, while the parent crawlsonar.com resolved fine. www.elsg.co.uk worked only because normalizeDomain strips a leading www.; no other prefix got that treatment. So the retry was dead for as long as it existed and its comment described behaviour that never happened — the same class of defect as the write-only consecutiveErrors counter fixed in 1.33.0. Not a regression from the registry-first change in that release: rdap.org answered the same subdomain with a 302 to the registry, which then 404'd, so the old path ended at the same null.unicodeName where the lookup used punycode — and findByPair then misses, so the same person could collect two reminders for one domain. Both sides now use the name that was looked up.example.com when somebody asked about blog.example.com is the useful thing to do — no registry holds a subdomain — but only if the substitution is visible; otherwise it is a confident answer to a question nobody asked. The report carries askedAbout alongside target, a plain sentence ("blog.example.com is a subdomain — this is the registration for example.com."), and a fact row so the substitution reaches every consumer of checks rather than only the one component that renders the notice. The result heading names the domain actually looked up. When nothing is registered, the error names both: "No registration record found for nothing-here.com (the registrable domain behind blog.nothing-here.com)".crawlsonar.com is asked directly rather than after a 374 ms 404 on the subdomain. A lookup for a name that is already registrable is unchanged — one request, no notice.co.uk returns 400 in 0 ms with an explanation, instead of a registration record. See below for why that matters.blog.example.com sits under a two-label name and shop.example.co.uk under a three-label one, and nothing in the hostname says which. This is not a near miss — measured 2026-09-07, https://rdap.nominet.uk/uk/domain/co.uk answers HTTP 200 with a real registration record, as does org.uk. A "last two labels" rule would therefore have reported the registration of the entire .co.uk namespace as though the visitor owned it: a confident wrong answer, worse than the "not found" it replaced. IANA's RDAP bootstrap, already loaded in this file, cannot help — it knows TLDs, not that co.uk is one of them.scripts/gen-public-suffix.mjs, ICANN section only, 7,396 rules) rather than fetched per request like the RDAP bootstrap, because the two have opposite failure modes: a missing bootstrap falls back to the rdap.org redirector and still answers correctly, whereas a missing suffix list would leave us unable to tell example.com from co.uk. The private section is deliberately excluded — github.io and uk.com describe who may create *sub*domains, but both are real registrations at their registries, which is what a WHOIS lookup is being asked about.uk.com cases the exclusion is meant to produce. Those vectors also caught a bug written blind: the list publishes internationalised rules in Unicode (公司.cn) while every hostname reaching us has been through new URL() and is punycode, so 446 of the 6,950 rules could never match. The generator now emits both forms.fetch sends by default, and how the twin broke. The same request with a User-Agent returns 200. Crawlsonar survives because its checks go through safeFetch, which supplies a default User-Agent one layer below every check, so whois.ts passing only an accept header is not the same code as the twin's. Nine domains across .com / .co.uk / .law / .pl / .app / .uk all resolved with correct expiry dates in 144–855 ms. This is a difference in architecture, not luck — but it is also undefended, which is the next item.outboundHeaders() is now a small exported function covered by safe-fetch.test.ts, which asserts the crawler identifies itself and, separately, that a caller passing its own headers (as the WHOIS check does) cannot silently displace it.consecutiveErrors had been incremented on every failed run since monitoring shipped and read by nothing — the comment in the runner claimed errors were "handled by the runner (consecutive-error tracking)", and that tracking did not exist. A check that errors keeps its last known state, so a monitor whose lookup fails forever went on showing a green "ok" badge, with "last checked" reading a minute ago. After three consecutive failures (roughly three hours at the hourly cron) a stalled alert goes out, once, on the crossing; the badge turns amber and reads "ok · not updating", keeping the last known state visible rather than hiding it.data.iana.org/rdap/dns.json, 1,200 TLDs, cached for a day) maps the TLD to its authoritative RDAP server; rdap.org stays as the fallback for TLDs the bootstrap omits, .io among them. This takes a third party — one that has demonstrably changed behaviour without notice, and fronts every registry we ask — off the critical path of every domain-expiry monitor, reminder refresh and WHOIS lookup. It is also faster, because rdap.org answers with a 302 that we then follow: measured 80–395 ms direct against 144–855 ms through the redirector, one round trip instead of two. Measured with the bootstrap URL deliberately broken: lookups still succeed through the fallback, and a stale copy of the map is preferred to no map at all.error field and shows as its latest result; a failure that does not say where it happened is a failure nobody diagnoses, which is precisely why the twin's went unexamined.catch {} and left no trace anywhere, so an RDAP outage could run for months while every run reported success off dates nothing had rechecked. Failures are now counted into the run's response and captured, and the reminder still goes out.MAX_CYCLES_PER_MONTH (five) with headroom for the suggestion calls that share the pool: Pro 1,200, Growth and Agency 7,800. Cost at full use, from the repo's own engine rates: $6.75/mo against Pro's $34, $33.75 against Growth's $119.freePlanLine() from the entitlements, and a test fails if any file hand-writes that cadence again./privacy already said correctly.capacity code, the page answers with an offer instead of a red box ("today's pool is shared and it's gone; a free account has its own credit and measures weekly"), and the event is recorded so the owner sees the ceiling in the funnel rather than hearing about it from a stranger. The credit figure quoted comes from the admin catalogue, so it matches the pricing page.fetch("/api/…") now uses the final path. Measured: a 308 does preserve the method and body, so these were never broken — each call simply paid an extra round trip since trailingSlash landed. A guard test covers fetch paths alongside links and cron paths./api/health without a trailing slash, which has answered 308 rather than JSON since trailingSlash: true landed in 1.27.0 — the documented proof that a release shipped no longer worked. The same applies to any external uptime check pointed at that URL: it passes only if it follows redirects.NEXT_PUBLIC_OAUTH_PROVIDERS and dormant until named — a button for a provider Supabase has not been configured with fails at the first click, so the list is the switch rather than a decoration. Both land on the existing /auth/callback/, which already exchanges the code, records the first sign-in and sends the welcome e-mail, so nothing provider-specific was added there. Entra is asked for the email scope explicitly; without it the profile arrives with no address and the account cannot be matched to a subscription.docs/SOCIAL-LOGIN.md — the Google Cloud and Entra steps the owner must do, including the redirect URI being Supabase's rather than the site's, and the Entra client secret's expiry, which presents as "sign-in stopped working" with nothing in the app's own logs.column "at" does not exist against a project that already had a page_views table. create table if not exists is silently a no-op when a table of that name exists in any shape, so the very next index ran against a foreign table and failed cryptically. The migration now adds each column explicitly, converging an existing table instead of assuming it owns the name, and applies the NOT NULL constraints only when the existing rows allow it — a bare set not null would abort the whole restore on legacy data.npm run sql:verify gained that scenario: it now runs the restore file against a clean database *and* against one already holding a differently shaped page_views with rows in it. The failure was reproduced in the harness before the fix was written, rather than reasoned about.function tp_member_owners() already exists in schema "private". alter function … set schema cannot be re-run, and the earlier increment had just re-created a copy of the function in public. Dropping that copy is not the answer either — the same increment re-creates the policies that depend on it. The stale private copy is now dropped first and the fresh public one moved in its place; policy dependencies follow the function's OID across a schema change, so nothing has to be rewritten.npm run sql:verify runs the whole restore file twice against an isolated Postgres engine and asserts the RLS helper ends up only in private. The file has always claimed idempotency; nobody had executed it twice, so nothing contradicted the claim until it was needed.page_views table, a beacon on path change, and Admin → Traffic. The owner sees their own traffic in their own database, on their own charts, permanently — not rented from a vendor who samples it and can withdraw it, and not dependent on an ad blocker letting a script through./i/:token, /unsubscribe/:token) are masked to their pattern before insert, because a saved token is a credential sitting in an analytics table and rows already written cannot be un-written.TRUSTED_IPS are dropped on arrival, bots are excluded by user agent, and the endpoint always answers 204 so a tracker can never break a page./api/v1/cron/watchdog/ walks an escalation ladder against each job's own interval: silence past 2.5× fires a quiet revival, past 3× revives and alarms, and a job with no completed run ever is alarmed immediately because that is a configuration error which never self-heals. One revival per job per interval, so a job crashing in a loop costs one attempt per cycle rather than one per pass.cloudflare/cron-watchdog/: a free Cloudflare Worker that pokes the watchdog from outside Vercel, on an offset minute. The death of the whole platform scheduler is the one failure an internal watchdog cannot see, because it rides on what died.docs/CRON-WATCHDOG.md — what each job loses while it is dead (jobs reading a queue lose nothing permanent; jobs measuring the present lose that moment forever), the ladder and its reasoning, and the two-place CRON_SECRET rotation. Rotating only Vercel silently kills the backup leg, and a dead backup leg looks exactly like a healthy one until it is needed.vercel.json without a watcher fails the build instead of running unobserved.trailingSlash: true, the fleet standard, with every internal link, redirect and sitemap entry rewritten to the final form and the six cron paths in vercel.json given their slash. A path copied from another project is now always in the right form, which is the whole point of one convention: Vercel's scheduler calls the cron path exactly and does not follow redirects, so the wrong form earns a 308, the handler never runs, and the dashboard still reports the cron as executed./api/v1/csp-report/. A POST that earns a 308 arrives with an empty body, which would have quietly discarded the violation reports added in 1.23.0.strict-dynamic, no 'unsafe-inline' and no analytics host. Those routes are already dynamically rendered, which is what nonces require, so the public pages keep the cacheable static policy and lose nothing. The pre-paint theme script is admitted by the sha256 of its own bytes rather than by reopening inline execution./api/v1/csp-report, feeding the existing Admin → Errors log as a csp kind. Both report shapes are accepted — the Reporting API array and the legacy csp-report object Safari and older Chromium still send — because supporting one loses about half the signal. Report bodies are unauthenticated and cross-origin, so every field is length-capped and only the document path is stored: a full URL can carry a one-time token or an e-mail address in its query string.JSON.stringify. 38 of 63 injection sites bypassed it, and with 'unsafe-inline' in script-src a single </script> in any future value would have ended the element and started executing markup. A test fails if a raw one reappears.next/script to a plain first-in-body <script>. Under a nonce policy next/script server-renders nonce="" while the client expects none, so every signed-in page hydrated with a mismatch.connect-src list never contained the Supabase origin, so the browser blocked every auth call before it left the page. The policy now names the configured project's exact origin — not https://*.supabase.co, and nothing at all while auth is dormant. The CSP moved out of next.config.ts into src/lib/csp.ts and is covered by tests, because a missing connect-src entry breaks no build, no test and no render; it fails one runtime call, opaquely.npm run sql:total generates the consolidated database restore file from the numbered increments, with the generation date as an argument so a diff shows schema changes rather than a new timestamp. A test fails when an increment is added without regenerating the total.APP_VERSION had drifted behind package.json, so the footer, /changelog, /about and /api/health all reported 1.21.0 while 1.22.0 was live. Because the post-deploy check reads /api/health, a stale literal makes a successful release look like a failed one — it did, immediately after this release went out. A test now pins the literal to package.json and to the newest changelog heading.0001_kv_store.sql altogether, even though the newest migration creates an RPC on kv_store. Restoring the database from that one file — the whole reason it exists — would have failed on a fresh project. It is now generated from every increment in application order..env.example pointed at individual increments for first-time setup, which invited exactly the cherry-picking that produced the gap above; it now points at the consolidated file and the runbook.20260906092006_revenue_atomic_billing.sql migration, test-mode publication and owner-provided API keys before live activation. See OPERATIONS.md. No production changes are made by the local test suite.CRON_SECRET, Agency members see the owner's entitlements, and Open Graph theme constants no longer invalidate the production route build.GATES.md.git push does not trigger a Vercel build for this project (the GitHub integration is inactive — 14 pushed commits produced no deployment); the deploy command is npx vercel deploy --prod from a clean tree. Also documented that vercel ls (CLI 59.x) prints every deployment URL twice, so the cleanup list must be de-duplicated before cutting it — otherwise the current production URL lands in the removal set./api/health reports 1.19.1, the lifecycle cron answers 401 (present, guarded), key pages and llms.txt serve 200. Old deployments pruned to production + one rollback spare (11 removed).OPERATIONS.md (repo root — /docs/ is deliberately uncommitted in this repo) — deploy procedure, the production env-var table with what each gates, the after-every-deploy checklist (health version, the mysentry "Send test heartbeat" click that registers all three cron monitors, the 401-not-404 cron spot-check, deployment cleanup) and the list of owner tasks code cannot do (legal read of /terms, real testimonials, competitor re-verification, Stripe promo code, Supabase migrations/passkeys). Written so the go-live steps live in the repo instead of a chat transcript./, /pricing, /alternatives, /terms, /tools, /leaderboard, /api/health, /llms.txt all 200; MCP tools/list 200; llms.txt serves the new Product section.npm audit 0 vulnerabilities (was 8: 2 moderate, 6 high): Next.js 16.2.10 → 16.3.3 (brings sharp 0.35.4, closing the libvips CVEs bundled with Next's image optimizer) and the postcss override 8.5.16 → 8.5.26 (the 8.5.16 pin was itself a security fix in 0.93.0; its version has since been flagged). Full test suite and production build pass on the new versions.<a href="/leaderboard"> swapped for <Link>; seven components deferring their initial state-setting effect a tick (the set-state-in-effect rule); five tool clients whose useCallback read the input state instead of its own argument (fixing the exhaustive-deps warnings without behaviour change — the argument is what actually ran); four unused imports/vars; the two long-standing test-file type errors (fixture cast, union narrowing).fs.readdir on an env-overridable path — without the build warning 16.2.10 printed. Fixed in the monitor store, the reminders store and the admin dashboard's report counter (literal .data/... paths; the reminders test now stubs process.cwd() like the testimonials test). Verified: 0 routes above 100 traced src files, down from 9 × 500.NEXT_PUBLIC_GA_ID, INBOX_INGEST_SECRET, SCAN_LOG_SALT, PDF_FONTS_DISABLED, the test-only store-dir overrides) and drops the now-dead MONITOR_STORE_DIR/REMINDER_STORE_DIR entries.npm audit 0 vulnerabilities; npx eslint src 0 problems; npx tsc --noEmit 0 errors; 593 tests pass; production build clean; nft-trace scan shows 0 routes >100 src files./alternatives now quote that figure with the check date instead of the earlier unsourced "$189–199 for 100 prompts" range. Peec's page publishes no figures in plain text; /alternatives reports what third-party reviews say (Starter ≈ $95/50 prompts, Pro ≈ $245/150 prompts, three engines included) and labels it as such. Re-check both before any future copy change — prices move.termsOfService now points at /terms rather than /privacy.<base>-tracking) and the daily lifecycle cron (<base>-lifecycle) now send heartbeats with their error state and processed counts, alongside the hourly monitor cron. The admin "Send test heartbeat" button pings all three, which registers them on mysentry from one click — closing the gap where a heartbeat-registered monitor cannot notice a cron that never started. The admin card says to press it after any deploy that adds a cron.tsc clean apart from the two pre-existing test-file errors, eslint clean on every touched file, full test run green, production build clean with no whole-project tracing./api/health reports version 1.6.0 — every release from 1.7.0 to this one is unshipped; /api/v1/cron/lifecycle returns 404 there (not deployed), the pricing page shows plans (billing enabled). The Vercel CLI is not installed on this machine, so production environment variable names could not be listed./terms; real customer quotes (the digest now asks); production env values (Resend, Supabase public vars, engine keys) — verify in the Vercel dashboard; press "Send test heartbeat" in Admin → Dashboard after the deploy; a Stripe promotion code for WINBACK_PROMO_CODE if win-back should carry a discount. Deployment only on explicit request./terms (footer link, sitemap, WebPage + breadcrumb JSON-LD): operator facts as on /privacy, what the free tools and paid plans are, acceptable use, trials and automatic renewal, the 14-day refund the pricing FAQ already promises, content ownership, a plain statement that AI-visibility numbers are sampled observations and not guarantees, liability caps, termination, England & Wales law. Written to match what the product does today; needs a legal read before it is relied on — the checkout and sign-in flows do not yet reference it./alternatives — the checklist for choosing an AI visibility tracker and where Crawlsonar stands. Eight questions to ask any vendor, a fact table read from the same plan definitions the app enforces (engines, samples per prompt, metrics, competitors, alerts, white-label, price per tracked prompt, day-one experience, free plan), the existing price anchor ("Otterly and Peec list roughly $189–199/month for 100 prompts, at the time of writing"), and an honest "where we are not the right fit" box (no rank tracking, no SSO/SAML, no social listening, no guarantee of being recommended). No competitor feature claims beyond the price anchor — those would need verification. Linked from the pricing page and the homepage pricing section; FAQ JSON-LD; ISR hourly via the cached catalogue.src/lib/testimonials.ts (KV document testimonials:v1, file locally; cached public read with tag testimonials), admin API /api/v1/admin/testimonials (GET/POST/PATCH/DELETE, admin only, validated), an "Social proof" panel under Admin → Users to add, publish/hide and remove quotes, and a <Testimonials /> section on the homepage and pricing page that renders nothing until the first quote is published. No placeholder quotes anywhere: the module is the form for the real ones.computeCohorts with tests./api/mcp with Authorization: Bearer <key> is metered on the key's daily quota (JSON-RPC -32001 with HTTP 401/429 when invalid or exhausted) instead of the anonymous 30-messages-per-minute-per-IP limit; authenticated traffic is attributed to the key in Admin → API & MCP and on Account → API. Documented on /api-docs, the MCP discovery GET and the admin limits card. Anonymous access is unchanged./terms, /alternatives, /pricing, /about render 200 with the expected titles; the pricing page contains no testimonial section while the list is empty; annual sub-copy shows "save $68" (Pro) and "save $238" (Growth); GET /api/mcp advertises optional auth; anonymous tools/list returns 16 tools; a bogus bearer key returns the JSON-RPC error with HTTP 401; the admin testimonials API returns 401 unauthenticated.tsc clean (two pre-existing test-file errors unchanged), eslint clean on all touched files, 593 tests pass, production build below./terms; real customer quotes; verification of any competitor claim before adding one; production env checks (Resend, billing flag, Supabase public vars, engine keys); MySentry registration of the lifecycle cron; Stripe promotion code for WINBACK_PROMO_CODE. Deployment is not performed without an explicit request.WINBACK_PROMO_CODE quoted when set (create the code in Stripe; checkout accepts promotion codes). Runs from the daily lifecycle cron alongside trial reminders and lead nurture.gtag('event', …) when the tag is loaded after consent), so the GA4 property sees the same scan / report / bridge / capture steps as the first-party store. Server-recorded conversions (signup, trial_started, upgrade, cancel) stay server-side by design; the GA4 admin tab now says exactly that instead of claiming events it never received.create-next-app boilerplate: what the product is and for whom, the map of the codebase, how to run it, the conventions (version = changelog = commit; dormant-until-configured; privacy-minimal; server-recorded conversions) and a five-minute post-deploy check.tsc and eslint clean. The win-back send itself is not exercised locally (no Resend); the window logic is unit-tested./agencies — the Agency plan as a landing page: 25 client projects, 400 prompts, all five engines daily, 10 competitors, 10 seats, white-label PDF reports, the price from the live catalogue (revalidated hourly), "what a client sees each month", a FAQ with FAQPage JSON-LD, the trial CTA and a walkthrough e-mail link./about — what the product does, how it scores, what data it touches, who runs it (company number, registered office) and how to reach a person. AboutPage JSON-LD./changelog — every release rendered from CHANGELOG.md at build time (a small parser and a safe inline renderer, unit-tested): the public proof of pace. Linked in the footer together with About and For agencies./leaderboard/{domain} — one static page per site in the snapshot (970): composite, AI-readiness and security scores with a plain-English band, rank overall and within the category, category neighbours as internal links, what the scores mean, "check your own site" and the tracking bridge; blocked sites get the bot-protection explanation instead. ItemPage + breadcrumb JSON-LD, listed in the sitemap (1,063 URLs now); the leaderboard table links to them.core-web-vitals and caching-and-ttfb (beginner and professional explanations, fix steps, FAQ), with issue-id rules so performance findings link to them./tools/blacklist-check, /tools/domain-expiry-reminder) plus the new pages./agencies rendered dynamically (Cache-Control: private, no-cache) despite revalidate = 3600: the KV client fetches with cache: "no-store", and any page that reads the catalogue during render inherits that. Public pages now read it through unstable_cache (hourly, tag billing-catalog, invalidated with revalidateTag(…, "max") when the admin saves prices). Measured on next start: both now answer x-nextjs-cache: HIT, s-maxage=3600./r/[token], its OG image, the PDF route, the leads and reports APIs carried ~700 traced files each, ~490 of them project source. Found by diffing the routes' .nft.json; cause: fs.readdir(storeDir()) in the report store's sweep — a readdir on a runtime-computed path makes Turbopack's tracer give up and include everything. Pre-existing since the store was written. Fixed by reading the literal .data/reports directory (and never sweeping under test). The PDF route now traces 193 files, no route exceeds 100 source files, and the build warning "whole project was traced unintentionally" is gone. The PDF fonts moved to new URL(…, import.meta.url) asset references (emitted as server assets and listed in both PDF routes' traces; the outputFileTracingIncludes TTF globs are no longer needed) — verified on the production build: a freshly generated report embeds Geist, not the Helvetica fallback.tsc clean; production build: 1,141 static pages, zero warnings. On next start: /about, /agencies, /changelog, /leaderboard/github.com (scored), /leaderboard/aa.com (blocked), the two Learn entries and the sitemap render; an unknown domain 404s; a shared report's PDF embeds Geist. Nothing in this release needs a runtime service.GET /api/v1/cron/lifecycle (vercel.json, 10:00 UTC, CRON_SECRET): (1) trial-ending reminders 3 days and 1 day before the first charge — plan, date, amount and the portal link (src/lib/lifecycle/trials.ts; for a trialing Stripe subscription current_period_end is the trial end); (2) lead nurture — three emails to people who downloaded a PDF report, on day 1 (the tool-specific fix that moves their score most, linking their report), day 3 (who AI recommends instead → free account) and day 7 (track it weekly → plans). One email per address per run, steps in order, never twice (markers), never to leads older than 14 days — so the day this ships does not blast the historical list (src/lib/lifecycle/nurture.ts)./leads/unsubscribe): the link carries the address and an HMAC (LEADS_UNSUB_SECRET, falling back to CRON_SECRET), GET shows a confirmation page, POST implements RFC 8058 one-click; every nurture and waitlist email carries the link and the List-Unsubscribe headers. Opt-outs are markers keyed by a hash of the address.src/lib/lifecycle/waitlist.ts, POST /api/v1/admin/waitlist). The email the waitlist was promised and never got.src/lib/lifecycle/markers.ts (KV with a 13-month TTL, files locally).tsc and eslint clean. Live on the dev server: /leads/unsubscribe?token=bad → 400 (GET and POST); /api/v1/cron/lifecycle → both jobs report "skipped":"email not configured" — the expected dormant answer without Resend.currentPeriodEnd semantics during trial — taken from Stripe's documented behaviour. After deploy: register the new cron in MySentry (a heartbeat that never fired can't alert) or run it once by hand, and check Admin → Usage for lifecycle_email events.<NextStepBridge> under the result on all 49 tool pages, the Domain Audit and every shared report (/r/{token}): AI/SEO tools (readiness, llms.txt, robots, sitemap, schema, SEO, link preview, compare, fix pack, performance, generators for AI/SEO, utilities) bridge to AI-visibility tracking — "This is one check, today. Now see whether AI recommends you — every week" → /account?next=/projects, with /pricing as the secondary link; DNS, email, TLS and security tools (25 pages) bridge to monitoring first — "Fixed it? Get told before it breaks again" → /tools/monitoring, with tracking as the secondary link. The monitoring page itself gets none (it is the product).bridge_clicked {tool, kind, target}, so Admin → Usage shows which tools actually send people into the product — the number the revenue plan was missing.<NextStepBridge tool="…" kind="…" /> immediately before each page's <ToolFaq>), so the placement is uniform and a page without a bridge is a lint-visible exception, not a forgotten one.tsc clean, 576 tests pass; the tool pages' 11 eslint findings (react/no-unescaped-entities, react-hooks/exhaustive-deps) are pre-existing — identical count with the change stashed. Rendered /tools/dns-lookup locally: bridge present below the content, above the FAQ.src/lib/server-events.ts): signup (first sign-in ever, via a once-per-user marker in KV / a local file), project_created, first_cycle_ready, trial_started, upgrade, payment_failed, cancel — the last four derived from Stripe status *transitions* (previous record vs. incoming), so overlapping webhook events count each milestone once. The browser beacon endpoint no longer accepts signup/upgrade (a beacon can't fake a conversion any more); monitor_create_* (fired by the monitoring UI, rejected with 400 until now) and bridge_clicked / cta_clicked are accepted.computeFunnel is pure and unit-tested.ANTHROPIC_API_KEY) and runs an instant cycle — 3 prompts × 2 samples × the plan's engines, calls in parallel — before redirecting, so the project page opens with share of voice, winning competitors and cited sources instead of "No cycles yet". Projects that have prompts but no data get a Run the first cycle now button. Guards: owner only, 3 instant runs per account per day, the global TRACKING_DAILY_BUDGET, and plain messages while engine keys are dormant. The runner's per-project loop became runProjectCycle (shared by the cron — sequential as before — and the bootstrap — concurrent): src/lib/tracking/runner.ts, bootstrap.ts.src/lib/lifecycle/emails.ts; dormant until Resend is configured, intent logged): welcome on first sign-in (one job: create the first project), "your first cycle is in" with the numbers (and a plan nudge on Free), payment failed (→ billing portal), cancelled (projects stay on Free, door open). Trial-ending reminders need a daily cron and ship in the next release./projects?welcome=1 onboarding checklist; the create and run buttons narrate the 20–40 s wait (useFormStatus); both project pages declare maxDuration = 60 for the synchronous action.tsc and eslint clean; /pricing rendered locally with the new bullets. Not exercised locally (no Supabase, engine or Resend keys in .env.local): the bootstrap run, the welcome / first-cycle emails and the webhook transitions — reasoned from code and written to fail soft (every branch degrades to the scheduled cron). First check after deploy: create a project on a free account → the page should open with numbers, and Admin → Usage → Funnel should show project_created and first_cycle_ready.src/lib/jsonld.ts — breadcrumbJsonLd (always starting at Home), webPageJsonLd (with isPartOf / publisher references) and graph — so every page links into one entity graph anchored on the /#organization and /#website nodes the layout declares. Same ids everywhere is what lets search and answer engines merge the pieces.legalName, the Companies House identifier, the registered address, support email + contactPoint, foundingDate and knowsAbout; WebSite gains inLanguage and a description./tools — CollectionPage + an ItemList of all 51 tools as free WebApplications; /pricing — one Product per product line (AI Visibility, Monitoring, API) with an AggregateOffer and an Offer per active plan from the live catalogue (monthly and annual UnitPriceSpecification, free tier included); /leaderboard — Dataset (snapshot date, variables measured, technique) plus a CollectionPage; /api-docs — WebAPI (endpoint, documentation, provider, free-tier offer; MCP named in the description); /bot — WebPage about the crawler as a SoftwareApplication; /privacy — WebPage with dateModified; /guides and /learn — breadcrumbs. Tool pages' WebApplication now carries @id, publisher, isPartOf, isAccessibleForFree and a Home-first breadcrumb (one helper, 49 pages)./guides and /learn switched from raw JSON.stringify scripts to the escaping JsonLd component.application/ld+json block: 0 invalid; each page carries the expected types (Organization + WebSite everywhere; SoftwareApplication + FAQPage on /; CollectionPage + ItemList on /tools; Product ×3 on /pricing; Dataset on /leaderboard; WebAPI on /api-docs; TechArticle + FAQPage on a guide; DefinedTerm + FAQPage on a Learn entry; WebApplication + FAQPage on a tool). 3 new unit tests (572 total); tsc and eslint clean./pricing, /leaderboard and a tool page after deploy./api-docs is gone; the page now sends developers to Account → API (sign-in or a free account, then a key). POST /api/v1/keys requires a session (401 otherwise) and enforces the plan's key count (free 2, Dev 5, Scale 10); new GET /api/v1/keys lists the caller's keys with today's usage, and DELETE /api/v1/keys?id= revokes one. Every key now has an owner: records carry userId and a per-user index (apikeys:user:{id} in KV, a JSON file locally) makes list/revoke possible without scanning the store. Keys minted anonymously before this release keep working for requests; they simply belong to nobody and can't be listed./admin?tab=api): 7- and 30-day totals for the public API and the MCP server, rejected requests split by cause (burst rate-limit, daily quota, auth, malformed), a per-day table, and ranked breakdowns — API tools, API keys (by public id, matching Account → API), MCP tools and JSON-RPC methods. The limits in force are stated on the page: MCP 30 messages/min per IP, no key, no daily quota; API 60 requests/min per IP plus the per-key daily quota (free 100, Dev 500, Scale 4,000); the per-minute limiter is per serverless instance, the daily quota in KV is exact.src/lib/api/metrics.ts keeps one hash per surface per day (metrics:api:YYYY-MM-DD → total, outcome, tool, key, method; 100-day TTL in KV, JSON files locally). Routes record through Next's after() so metering runs once the response is sent and can never fail a request. Unit-tested (+5 tests, 569 total)./api-docs "Limits & errors" now states the real quotas per plan and the per-IP burst cap; the 401 message for a missing key points to Account → API.tsc and eslint clean on every changed file; 569 tests pass (key ownership, revoke, path-traversal guard on user ids, metrics aggregation). Not exercised locally: the KV branch (hashes + user index) — it uses the same kvHincrby/kvRpushJson helpers already in production for the rate limiter and event stream./account: a one-tap WebAuthn sign-in next to the magic link, via supabase.auth.signInWithPasskey() (auth-js 2.110, behind its auth.experimental.passkey flag). The same next continuation is honoured, so "Start free" → passkey → /projects. The email input advertises autocomplete="email webauthn" for conditional UI.auth.registerPasskey() then a friendly name via auth.passkey.update() (default "Chrome on macOS"-style from the user agent) — and remove. Clear recovery note: the email link always works, so a lost device is never a lock-out.NotAllowedError cancelled/timed out, InvalidStateError already registered, unsupported browser, insecure context, server not enabled, no passkey for this device) in src/lib/auth/passkeys.ts, unit-tested.NEXT_PUBLIC_PASSKEYS_ENABLED=1 shows the buttons and enables the client flag. Prerequisite on the Supabase project: passkeys enabled in Authentication settings (the /auth/v1/passkeys/* endpoints must respond) with the relying-party ID set to the site host. Without that the buttons would fail at the first click, which is why the flag defaults off. .env.example documents it.tsc and eslint clean. Not verifiable locally (no Supabase project with passkeys in .env.local): the live ceremony, which needs HTTPS on the real host. First thing to check after enabling: register a passkey on Account, sign out, sign in with it./account?next=/projects) and a secondary one-shot check. A static product preview (trend chart, share of voice, "who wins the answers", labelled as sample data) replaces the abstract benefit tiles. Then: a proof strip with the real leaderboard numbers (970 sites, median 76/100, 17% blocked), "what you see every week", how it works, the one-shot AI Visibility check (/tools/ai-visibility, now tracked as scan_started {tool:"home-ai-visibility"}), plan cards from the editable billing catalogue (annual per-month price; feature bullets are the entitlements the code actually grants), the "Can't I just ask ChatGPT myself?" objection, and only then the free tools — the search box, six featured tools and "Browse all N tools" — visible to anyone who scrolls, absent from the first screen. FAQ with FAQPage JSON-LD, a closing CTA, and SoftwareApplication JSON-LD carrying Offers from the catalogue. The page is ISR (revalidate = 3600), so an admin price edit reaches the homepage within an hour without a deploy./tools, the homepage and the mobile menu); the IP widget leaves the nav (it stays on /tools/what-is-my-ip)./tools gets the homepage's type-to-find search, and the tool count comes from the index (ALL_TOOLS.length, 51) — one number everywhere instead of "40+ / 50+ / 55"./pricing Free-plan CTA now creates an account (/account?next=/projects) instead of opening a one-shot tool; /billing/success sends the new customer to Create your first project (/projects?welcome=1) instead of back to the free tools; sign-in carries a same-site next through the magic-link callback, and a signed-in visitor arriving with ?next= is redirected straight there; the email-capture copy no longer promises that monitoring "launches" (it shipped in v0.60).tsc clean for the change (two pre-existing test-file errors untouched), eslint clean on every changed file, 559 tests pass. Dev server: /, /tools and /account?next=/projects render; homepage copy checked section by section; plan prices $28 / $99 / $249 per month (annual) from the seed catalogue.NEXT_PUBLIC_SUPABASE_URL / NEXT_PUBLIC_SUPABASE_ANON_KEY unset the new CTAs land on "Accounts aren't enabled yet" — confirm both are set in production before this goes live, and that NEXT_PUBLIC_BILLING_ENABLED=1 so the plan cards lead to checkout rather than the waitlist modal.GET /api/v1/reports/{token}/pdf, AI Readiness / llms.txt / Domain Audit) is now a designed document instead of a text dump. Page one: navy cover band with the sonar mark, score ring with grade pill, severity pills, plain-English verdict (with the "capped by a high issue" explanation when it applies), four KPI tiles (issues by severity, checks passed, weakest area or module, AI-crawler rules / scan time / llms.txt status) and the top three priorities. Then: category score bars, issue cards (severity + category/module pill, Impact / Evidence / Fix in a spec-sheet layout, inline code, example code block, clickable "plain-English explainer" link into /learn/{slug}), the AI-crawler policy table with policy pills, "what an AI assistant actually reads" (front-matter and image syntax stripped from the extraction), the full checks list with drawn pass/fail/n-a glyphs and weights, a numbered "what to do next" (re-run link + share link with its expiry) and a closing bridge to AI Visibility monitoring with a UTM-tagged /pricing link. Running header/footer with "Page i of n", PDF outline bookmarks, document metadata.GET /api/v1/projects/{id}/report) rebuilt on the same system: headline share-of-voice ring with the change versus the previous cycle, KPI tiles (share of voice, presence, tone, change since the first cycle), a vector trend chart (share of voice + presence), per-engine table for the latest cycle, cycle history with ▲/▼ deltas, the latest cycle's winning competitors and cited sources as ranked bars (the route now passes latestInsights, so the PDF never says less than the project page), tracked prompts, competitor chips, a "how to read this report" note and a dashboard CTA. White-label (Agency) output carries no Crawlsonar name, mark, link or metadata — covered by a test that greps the uncompressed document.src/lib/pdf/fonts/) embedded and subsetted per document — a five-page report is ~70–85 KB. If the TTFs cannot be read at runtime the renderer logs {"at":"pdf/fonts",…} and falls back to Helvetica/Courier under the same logical names: a missing asset degrades the look, never the download. next.config.ts force-includes the TTFs (and pdfkit's .afm metrics) into both PDF function bundles.src/lib/pdf/{canvas,theme,format,fonts}.ts: measured text, ensure() page-break guards so no card, table, chart or heading straddles a page, tables that redraw their header after a break, ring / bar / line-chart / code-block primitives, pill / KPI / badge composites. Both renderers keep their public signatures (renderReportPdf, renderTrackingPdf), with an optional { compress: false } for tests.tsc is clean for the new files (the two pre-existing tsc errors in core-web-vitals.test.ts and reports/store.test.ts are untouched).outputFileTracingIncludes ships the TTFs into the function bundles. After the first deploy, download any report and confirm the PDF embeds Geist (font panel, or strings report.pdf | grep Geist). A Helvetica-only PDF means the fallback kicked in and the function logs will show "at":"pdf/fonts"./pricing reads prices from the editable billing catalogue (KV, seed as default), so changing a plan’s price in the admin Billing tab now updates the public page too — no code edit. Curated feature copy stays in code; only the amount + which paid plans show (deactivated plans are hidden) come from the catalogue. The "Growth is $X/mo" anchor is computed from the live price. Verified: Pro $28 · Growth $99 · Agency $249 (annual/mo) from the seed.src/lib/admin/insights.ts computes usage + revenue./account and /projects now share one shell (CustomerShell).POST /api/v1/billing/portal-account — resolves the Stripe customer from the *verified session* e-mail, so a user can only ever manage their own billing).billing:catalog), with the code seed as the default until the first save.lookup_key (transfer_lookup_key) and archives the old one — new checkouts get the new amount, existing subscribers keep theirs (grandfathering). The status table flags each row as Synced / Price changed / Missing / To archive with catalogue-vs-Stripe amounts.stripe:setup CLI and the admin button share one engine (stripe-catalog.ts). Admin API validates every field with zod and never leaks secret values. +2 tests.admin-tabs.tsx removed). Responsive: the sidebar collapses to a scrollable top bar on mobile.src/lib/billing/stripe-catalog.ts (getCatalogStatus / syncStripeCatalog) now backs both the admin button and npm run stripe:setup (DRY). New admin API POST/GET /api/v1/admin/billing (behind the admin gate; returns 4xx not 5xx per the CF-swallows-5xx rule; never leaks secret values — booleans only).{error} — res.json() threw and the client fell back to the generic "Network error — please try again."analyzeSentiment scores only the sentence(s) naming the brand; analyzeAnswer attaches it; aggregatePrompt exposes avgSentiment; latestInsights averages it per cycle.0004 adds a nullable tp_observations.sentiment numeric(4,3) (additive; existing RLS/grants cover it). Runner persists it.data.json regenerated.List-Unsubscribe (RFC 8058) headers (Gmail/Yahoo bulk requirement); wired into the domain-expiry reminders. (Note: transactional email was already fully implemented — this hardens inbox placement.)NEXT_PUBLIC_GA_ID to activate.supabase/0004_observation_sentiment.sql (or supabase/Total_incorporating_0004.sql)./account; an invited teammate shares read + day-to-day collaboration (prompts & competitors) on all the owner's tracking projects. Project create/delete and billing stay owner-only.tp_members (migration 0003) with its own RLS (owner manages seats; a member reads only rows naming them). The six tracking tables are widened from "owner-only" to "owner OR member" via one security definer set-function tp_member_owners() (self + inviters) — chosen to avoid RLS policy recursion and keep every policy a one-liner. SELECT = owner/member; INSERT/UPDATE/DELETE = per-table (prompts/competitors collaborative; projects/runs/snapshots owner-only writes). Cross-tenant negative-test SQL is inline in the migration.claimInvites binds member_user_id when the invited e-mail signs in) — identity never trusted from the client.entitlementsForOwnerId), so a member on a personal free plan can still work a 400-prompt Agency project; the create-project quota counts only owned projects (countOwnProjects). Seats: Growth 2 · Agency 10 (maxSeats)./account gains a Team panel (seat usage, invite, remove, Active/Invited status) + a "Shared with you" card; /projects labels shared projects and hides Delete on them. +3 tests. Dormant until Supabase Auth is configured — single-user behaviour unchanged.supabase/0003_agency_workspaces.sql (or the consolidated supabase/Total_incorporating_0003.sql) in the Supabase SQL editor.apiLimits: free 100/day, api_dev 500/day, api_scale 4000/day), baked into the key record (plan, dailyLimit, userId); authorizeAndMeter enforces the key's own daily quota. Anonymous keys keep the free limit unchanged.Monitor.userId) and are limited by plan (monitoringLimits: free 2, mon_pro 20, mon_business 100); the create route rejects over-limit with a 403. Anonymous flow (IP rate-limit) untouched.activePlansForEmail returns all active plans, so one user can hold AI-Visibility + Monitoring + API subscriptions at once. Entirely dormant until Supabase Auth is configured — anonymous behaviour is byte-for-byte unchanged. +2 tests.src/lib/tracking/suggest.ts, GET /api/v1/projects/[id]/suggest — auth + RLS + rate-limited, addPromptsAction respects the plan's prompt cap). Solves the cold-start — an empty tracking project is now one click from useful. Dormant until ANTHROPIC_API_KEY is set.Crawlsonar/1.0 (+https://crawlsonar.com/bot; diagnostic scan) (no longer embeds the app version), so verified-bot registrations (Cloudflare Signed Agent) that match the full UA don't drift each release. The performance-check UA is aligned and also points at /bot.latestInsights() aggregates the newest cycle's observations./api/v1/cron/digest (Mondays 09:00, src/lib/tracking/digest.ts) emails each paid project owner their current share of voice, the week's delta, and top sources/competitors — a retention touch-point. Dormant until Supabase service + Resend are set.Signature/Signature-Input, tag http-message-signatures-directory, covering @authority) — Cloudflare requires a signed directory. Request signing was already compliant (Signature-Agent, tag web-bot-auth, JWK-thumbprint keyid, ed25519, @authority+signature-agent)./bot page. Documents the Crawlsonar crawler: its User-Agent, that it reads server-rendered HTML only, respects robots.txt, signs requests (Web Bot Auth), its rate limits, and exactly how to allow or block it (robots.txt snippet + WAF guidance) plus a contact. Linked in the footer + sitemap. This is what Cloudflare/DataDome verified-bot programs and site owners look for.src/lib/security/robots-gate.ts (robotsAllows / assertRobotsAllowed, per-host cached, fails open) wired into the automated scanner (scanReadiness — the leaderboard + admin watchlist): we now **skip any site that disallows the Crawlsonar token or * in robots.txt**. +4 tests.+https://crawlsonar.com/bot — so a site owner who sees it in their logs lands on how to identify/allow/block us.src/lib/leaderboard/sites.ts expanded from 138 to 970 unique well-known domains across ~25 categories; the batch runner now de-dupes and tolerates bad domains (graceful "error"). The "Ranked by composite readiness" table is now paginated (50/page), sortable by every column, and searchable across every column (src/components/leaderboard-table.tsx).groupSnapshotsByEngine, updated TrendChart).src/lib/tracking/alerts.ts); threshold-gated, dormant until Resend is configured, free tier excluded.GET /api/v1/projects/[id]/report renders a per-project AI-visibility PDF (latest metrics, history table, competitors, prompts) via pdfkit; Agency tier gets it white-labelled (Crawlsonar branding dropped). "Download PDF" on the project page.src/lib/tracking/engines.ts: ChatGPT (OpenAI gpt-4o-mini), Perplexity (sonar, web-grounded), Gemini (gemini-2.0-flash) and Google AI Overviews (via SerpAPI — a SERP, not an LLM API), alongside Claude. Each is a separate adapter, dormant until its own key is set (OPENAI_API_KEY / PERPLEXITY_API_KEY / GEMINI_API_KEY / SERPAPI_KEY), all using neutral stateless sessions + cheap models.runner.ts resolves each project owner's plan → allowed engines, intersects with the keys actually configured, and samples each prompt N× per engine; a single flaky engine call no longer sinks the cycle. Writes per-engine trend snapshots + an "all" aggregate. Plan gating updated (Pro = Claude/ChatGPT/Perplexity; Growth+ adds Gemini + Google AIO).TRACKING_DAILY_BUDGET), per-engine cost estimates. Tests: +3 (engine gating); entitlement engine test updated for Phase 2. Env documented./projects/[id] — pure-SVG, theme-aware two-series chart (share of voice + mention rate, %) over time from tp_trend_snapshots, plus latest-cycle stat tiles and a "12% → 34% since <date>" delta. Empty state until the first cron cycle. src/components/tracking/trend-chart.tsx + listSnapshots.ANTHROPIC_API_KEY are configured.src/lib/tracking/runner.ts + /api/v1/cron/tracking (daily, CRON_SECRET-auth, origin-gate exempt): for each due project (isDue from cadence + last_run_at) it runs each active prompt N=3× through Claude (stateless = neutral sessions), records every tp_run + tp_observation, writes a daily tp_trend_snapshot (share of voice), and stamps last_run_at. Service-role Supabase client (src/lib/supabase/service.ts) bypasses RLS for the cross-user job.TRACKING_DAILY_BUDGET, default 2000) + per-run caps (25 projects, 100 prompts). Errors captured to the admin Errors tab. Dormant unless ANTHROPIC_API_KEY + Supabase service are set.isDue cadence logic).Crawlsonar/0.1 → Crawlsonar/${APP_VERSION} (e.g. Crawlsonar/0.92.0 (+https://crawlsonar.com; diagnostic scan))./account sign-in (passwordless), /auth/callback code exchange, sign-out. Session is scoped to the tracking/account area only (not global middleware) to avoid the getUser-on-every-request perf trap. Dormant until NEXT_PUBLIC_SUPABASE_URL + NEXT_PUBLIC_SUPABASE_ANON_KEY. New deps @supabase/ssr + @supabase/supabase-js.currentEntitlements(user) resolves the user's active subscription (planForEmail) → plan → tracking limits; enforced server-side in every Server Action (defense-in-depth on top of the disabled-in-UI controls)./projects (list + create, gated by maxProjects) and /projects/[id] (add/remove prompts and competitors, gated by plan). All mutations are Server Actions using the user-scoped Supabase client, so RLS enforces per-user isolation — the app never trusts a client-supplied user id..env.example.supabase/0002_ai_visibility_tracking.sql (+ regenerated Total_incorporating_0002.sql): tp_projects / tp_prompts / tp_competitors / tp_runs / tp_observations / tp_trend_snapshots, each with owner-only RLS (auth.uid() = user_id, transitively for children) so user A can never see user B's data. Indexed for the cron's due-selection.src/lib/tracking/analysis.ts — token-boundary brand detection, competitive ranking (brand's position among named brands), cited-source extraction, and share-of-voice aggregation across the N sampling runs. Fully unit-tested.src/lib/tracking/claude.ts — stateless Haiku calls (= clean, neutral sessions by construction; no account bias), small max_tokens, dormant until ANTHROPIC_API_KEY.src/lib/billing/entitlements.ts — plan → tracking limits (projects, prompts, cadence, N runs, competitors, history, engines); engines intersected with LIVE_ENGINES so we never promise an engine the runner can't execute (Phase 1 = Claude only).DataTable (page-size selector + pagination) instead of raw dumps — honouring the "all data tables paginate" rule. New client tables: errors-table.tsx, subscribers-table.tsx, waitlist-table.tsx.stripe SDK + a billing catalogue (src/lib/billing/plans.ts, single source of truth: 7 plans × monthly/annual, annual = ×10). POST /api/v1/billing/checkout creates a Stripe Checkout Session (subscription mode, 14-day trial, promo codes) resolving the price by lookup_key; the pricing page swaps its waitlist CTA for a real "Start 14-day trial" when NEXT_PUBLIC_BILLING_ENABLED=1, and falls back to the waitlist otherwise.POST /api/v1/billing/webhook verifies the Stripe signature and upserts a subscriptions store (src/lib/billing/subscriptions.ts, durable KV / in-memory) from checkout.session.completed and customer.subscription.*. Errors captured to the admin Errors tab.POST /api/v1/billing/portal (opened via the post-checkout session_id, so no email-based billing exposure) + /billing/success (shows plan/trial/receipt + Manage-billing) and /billing/cancel.npm run stripe:setup (STRIPE_SECRET_KEY=sk_test_… ) creates all products + prices with lookup_keys, idempotently. Env documented in .env.example.STRIPE_SECRET_KEY (+ webhook secret) are set — the app behaves exactly as before without them. Tests: +6 (catalogue math + subscriptions upsert/list)./pricing with a monthly/annual toggle (annual billed for 10 months = 2 months free, pre-selected for higher LTV). Three product blocks — AI Visibility (Free/Pro $34/Growth $119/Agency $299), Monitoring (Free/Pro $12/Business $39), API (Free/Dev $24/Scale $89) — prices trimmed from the research card.src/lib/waitlist/store.ts, durable KV / in-memory) via POST /api/v1/waitlist (rate-limited + honeypot). Converts to real Stripe checkout later; builds the launch list now./tools/ai-visibility that answers the #1 GEO objection head-on — non-determinism (one chat = one sample), account contamination (clean neutral sessions), breadth (many prompts × engines), trend-over-time as the real value, competitor + cited-source extraction, and drop-out alerts. Honest (the free check is framed as a baseline snapshot) and written to bridge toward the paid tracking tiers (loss-framed close). Same copy will seed /pricing./admin keeps everything that was there under Dashboard (extracted to src/components/admin/dashboard-tab.tsx); tabs switch via ?tab= (src/components/admin/admin-tabs.tsx).src/lib/errors/store.ts, durable KV in prod / in-memory locally, last 500) with a fail-safe captureError() helper (src/lib/errors/capture.ts). Server failures are now recorded from the reminders cron, monitor cron and Resend email sends; uncaught client-side errors are reported via a global ErrorReporter (window.onerror / unhandledrejection) → POST /api/v1/errors (rate-limited, size-capped). The tab lists newest-first with time / kind / source / message / where, plus a Clear log action (DELETE /api/v1/admin/errors, admin-only).captureError never-throws & non-Error handling).useEffect([]), but its container only rendered *inside* the save form (shown after a lookup), so it never initialised and no token arrived — leaving the submit button permanently disabled. The Turnstile container now mounts at the component root (interaction-only, so still invisible), so the token is ready when the user submits.target="_blank" rel="noopener noreferrer") so it doesn't drop someone out of the form mid-signup./tools/whois-lookup page now includes the reminder opt-in ("Never lose this domain") — for any expiry date, even years out — not just the dedicated tool.IP · x.x.x.x inline via a new variant="nav" on YourIp. On mobile the full strip stays at the bottom of the homepage (now sm:hidden so it isn't duplicated on desktop)./tools/domain-expiry-reminder. Check any domain's registry expiry (live via RDAP), then opt in to be emailed at 90 / 60 / 30 days before it lapses. Two-step UI (src/components/domain-expiry-reminder.tsx): look up the date first (show value), then capture email with an explicit consent checkbox + Turnstile.src/lib/reminders/{types,store,email}.ts: durable KV store (Supabase/Upstash) with filesystem fallback and a hashed domain+email pair-index for idempotent de-dup; branded Resend email with a working one-click unsubscribe; decideReminder threshold engine (sends only the single most-urgent note when someone joins late, never double-sends).POST/GET /api/v1/reminders (authoritative RDAP lookup server-side, rate-limited, Turnstile, consent-gated), POST /api/v1/reminders/unsubscribe + /reminders/unsubscribe confirm page, and a daily cron /api/v1/cron/reminders (added to vercel.json + origin-gate exempt) that re-checks RDAP at each threshold to detect a renewal and skip nagging.REGISTRAR_AFFILIATE_URL "renew or transfer & save" CTA — dormant until set (renewals are poorly commissioned; transfers/new-regs are what pay). The real asset is the high-intent, recurring email list.EMAIL_*).ADMIN_HOST (e.g. admin.crawlsonar.com): src/proxy.ts serves the dashboard at that host's root (redirects / → /admin) and **refuses /admin* + /api/v1/admin/* with 404 on every other host**, so the marketing domain no longer exposes an admin login. Dormant until ADMIN_HOST is set.src/lib/security/cf-access.ts verifies the Cf-Access-Jwt-Assertion header — RS256 signature against the team JWKS (cached, refetch-on-kid-miss), plus aud / iss / exp/nbf checks — so a request reaching the origin without passing Access (e.g. a spoofed Host on the raw Vercel URL) is rejected. Dormant until CF_ACCESS_TEAM_DOMAIN + CF_ACCESS_AUD are set.src/lib/admin/guard.ts now resolves either mode: with Access configured the signed identity is the sole app-layer auth (the legacy token form is skipped and its cookie ignored); otherwise it falls back to the ADMIN_TOKEN cookie for local/dev. Who may sign in is decided entirely by the Cloudflare Access policy — the app keeps no email/domain allow-list. The admin page shows the signed-in email and, in Access mode, Sign out ends the edge session (/cdn-cgi/access/logout).src/components/json-ld.tsx (<JsonLd> / serializeJsonLd) escapes <, >, & and U+2028/U+2029 that JSON.stringify leaves raw — closing the theoretical </script> breakout in structured-data blocks. Applied to the site-wide + reused/content-driven spots (layout org data, ToolFaq on every tool page, leaderboard, guide articles). +2 tests. (Inputs are static today; this makes the pattern safe by construction.)z.string().trim().toLowerCase().max(254).pipe(z.email()) instead of a bare length cap (rejects junk before storage/send).rateLimitKv (holds across serverless instances), on top of the existing Turnstile + constant-time compare.preload; added Cross-Origin-Opener-Policy: same-origin and X-Permitted-Cross-Domain-Policies: none. (Deliberately NOT Cross-Origin-Resource-Policy — the embeddable AI-Ready badge is loaded cross-origin by design.)postcss to 8.5.16 via overrides (Next bundled a flagged 8.4.31) — npm audit is now 0 vulnerabilities.src/lib/leads/email.ts exposes sendEmail() that POSTs to Resend's API with Authorization: Bearer (accepts EMAIL_API_KEY or RESEND_API_KEY, plus EMAIL_FROM), and report deliveries (sendReportEmail) and monitoring alerts (src/lib/monitor/alerts.ts) now use it with branded HTML. Fails safe (never throws into a route/cron); dormant until EMAIL_API_KEY + EMAIL_FROM are set, logging intent otherwise. New admin Email (Resend) card + /api/v1/admin/email route to check status and send a test email. +5 tests (466 total).sm breakpoint the inline header nav is replaced by a hamburger button (src/components/mobile-nav.tsx) that opens a full-width dropdown with the nav links + Domain audit; the theme toggle stays in the bar. Closes on link tap, on Escape, or by toggling. Desktop nav unchanged.dark: variant now keys off a data-theme attribute (via @custom-variant) instead of the OS media query, so the choice is controllable and persisted in localStorage. A beforeInteractive init script sets the theme before first paint (no flash). Auto resolves via Vercel geo: src/proxy.ts reads x-vercel-ip-timezone on HTML navigations and sets a short-lived geo_theme cookie (day → light, night → dark for the visitor's location); the client falls back to its own clock, then the OS preference. A prefers-color-scheme CSS fallback covers the no-JS case.src/components/web-mcp.tsx registers Crawlsonar tools with the emerging navigator.modelContext.provideContext() API so an in-browser AI agent can use the site's actions — list_crawlsonar_tools, check_ai_readiness, check_agent_readiness (calling our public endpoints). Feature-detected, so it's a no-op in every browser that lacks the API today — an early-mover / dogfood flex for an agent-readiness product, with zero downside.llms.txt) and the competing elements (IP strip, tool search) from above the fold. — Trust line anchors on real leaderboard data: "We've scored 138 of the world's best-known sites — the median is 78/100" with a "See where you'd land" curiosity hook. — "What you'll get" (score / issues / fixes) kills ambiguity before the click; "How it works" is three tiny steps; a benchmark section reuses the leaderboard (median 78, 15% blocked) for social proof + relativity; "What we check for you" reframes tools as outcomes a non-expert cares about. — Secondary paths (full audit, all 50+ tools, tool search, the IP strip) moved below the fold so they never compete with the primary action. Verified in light/dark, desktop/mobile.MetadataRoute to a text route handler (src/app/robots.txt/route.ts) so it can emit custom directives, and added Content-Signal: search=yes, ai-input=yes, ai-train=yes (we're a GEO product — all uses allowed) to both the * group and the AI-crawler group. — Detected in the Agent Readiness Checker. New src/lib/checks/content-signals.ts parses a Content-Signal: directive; the checker adds a Content Signals declared in robots.txt check (weight 8) with an info finding + example when absent. Weights rebalanced (AI-crawler-rules 12→8, OpenAPI 12→8) so the module still sums to 100. +3 tests. — Generated by the AI robots.txt Generator. The generated policy now includes a Content-Signal directive derived from your allow/block choices (search / ai-input / ai-train), e.g. allow-all → all yes, block-all → all no, block-training → ai-train=no. +4 tests (461 total).src/lib/mysentry.ts POSTs { monitor, ok, processed } with Authorization: Bearer <token> to mysentry's /api/ingest/heartbeat (fire-and-forget, never throws into the cron; +5 tests). Configured from a new admin card — paste your mysentry project ingest token (stored server-side in KV, never returned to the client), pick a monitor name, and Send test heartbeat to confirm it lands. The hourly monitor cron carries the heartbeat (ok reflects whether the run had errors). Admin APIs /api/v1/admin/mysentry (+ /test) are session-guarded. Dormant until a token is set./api/health — a lightweight public liveness endpoint (GET/HEAD → 200 with { status, version, store, time }), handy for any external HTTP uptime check.src/lib/checks/dns-aid.ts probes _index._agents.<domain> and _a2a._agents.<domain> over DoH for ServiceMode SVCB/HTTPS records (RFC 9460) and reports whether the discovery answer is DNSSEC-authenticated (AD flag). Surfaces a concrete finding + fix (publish ServiceMode SVCB/HTTPS with alpn+endpoint, DNSSEC-sign the zone) with an example record, plus a dedicated explainer on /tools/agent-readiness linking the IETF DNS-AID draft and RFC 9460. Wired into the report (weight 10; llms.txt & MCP trimmed 15→10 to keep the module total at 100). +3 tests.computeStats in src/lib/leaderboard/score.ts, +2 tests). When a visitor runs the AI Readiness Checker, a new "How you compare" panel shows their score against the leaderboard's AI-readiness median & average (of the well-known sites), with a distribution bar — stats passed server-side so the client bundle stays lean.Allow: / with no disallows — maximally readable and citable by AI, which is the point of a GEO product. The general * group keeps /api/, /r/, /admin, /monitor/ out.DataTable (src/components/admin/data-table.tsx): page-size selector (10/25/50), pagination, and optional mass-selection with bulk actions — applied across the admin. New Watched domains section in /admin: add domains, scan now (live AI-readiness + security composite via the shared scanReadiness engine), and a full lifecycle — Active (Archive / Bin), Archived (Unarchive / Bin), Bin (Restore / Delete now). Binned domains auto-delete after 30 days (lazy on read + in the daily cron). crawlsonar.com and mysentry.co.uk are seeded so the operator's own properties are tracked out of the box. Durable store in KV with an in-memory fallback for local dev; state-machine, purge, normalisation and dedupe are pure and unit-tested (+7 tests, 443 total). Admin APIs (/api/v1/admin/domains + /action + /settings) are session-guarded./api/v1/cron/domains-scan (Vercel Cron, 0 6 * * *) purges expired binned domains then scans the day's least-recently-checked auto-scan domains, up to the operator-set auto-scan/day count (a box in the admin). Added to the origin-gate exemption in src/proxy.ts (Vercel Cron bypasses Cloudflare, so it must not be 403'd by origin cloaking) — covered by a proxy test.DataTable.scanReadiness extracted to src/lib/leaderboard/scan.ts and shared by the leaderboard generator and the watchlist (DRY)./leaderboard). We scan a curated list of well-known public global sites through our own engine and rank them by a composite of AI readiness (55%) + HTTP security & headers (45%) — a data-journalism / backlink-magnet page (only big, public entities; no shaming of small sites). Sites that answer with an anti-bot challenge are detected via the bot-wall engine, marked blocked and surfaced separately with the vendor — a telling stat in its own right ("N% blocked our fetch — and, likely, AI crawlers too"). Each row links into the AI Readiness Checker (funnel). Static snapshot in src/lib/leaderboard/data.json, generated by npm run gen:leaderboard (scripts/gen-leaderboard.ts); the page prerenders from it, so zero per-request scanning. Pure composite/ranking logic in src/lib/leaderboard/score.ts (+5 tests, 436 total). First scan: 138 sites, 113 scored, 21 (15%) blocked. Linked from the header + footer nav./tools/blacklist-check) — "is your IP or domain on a spam blocklist?". New src/lib/checks/blacklist.ts engine queries DNSBLs over DoH: an IP is checked against IP lists; a domain is checked against domain lists (DBL) and its MX servers' IPs (up to 3) against the IP lists. Results feed the standard explainable score (a listing is a high-severity issue that caps the score; unreachable lists are excluded, never counted as clean). Per-listing results cached in KV for ~1h. New API route /api/v1/tools/blacklist (zod + durable per-IP rate limit 15/h + Turnstile-gated) and the tool is registered in the search index.SPAMHAUS_DQS_KEY set, the tool queries Spamhaus ZEN (IP) + DBL (domain) via the Data Query Service — the sanctioned, serverless-friendly path (Spamhaus blocks public-resolver queries and forbids commercial use of the free mirror). Without a key it runs the best-effort public list (SpamCop) and shows a "limited coverage — connect a free Spamhaus DQS key" notice. Blocked/misconfig sentinels (127.255.255.x) are treated as *unavailable*, never a false listing. +8 tests (431 total).src/proxy.ts, dormant until ORIGIN_SHARED_SECRET is set) now exempts two self-authenticating service endpoints that legitimately arrive without the Cloudflare-injected header: /api/v1/inbox/ingest (Cloudflare Email Worker subrequest — a Worker's fetch into our zone isn't guaranteed to pick up the Transform Rule) and /api/v1/cron/monitor (Vercel Cron invokes the deployment directly, never traversing Cloudflare). Both stay protected by their own strong, constant-time secrets (INBOX_INGEST_SECRET, CRON_SECRET), so exempting them doesn't weaken cloaking. Exact-match allow-list (never prefix). Prevents inbound test emails and monitoring from silently dying the moment cloaking is switched on at launch. +6 tests (423 total) covering dormant pass-through, gated 403, correct-header pass, and both exemptions.appearance: "interaction-only"). The widget no longer renders a visible box in the common case — it stays zero-height and still returns a token, only becoming visible if Cloudflare genuinely needs an interactive challenge. Fixes the bulky/ugly widget on the inbox and AI Visibility tools./privacy explaining what Cloudflare processes (IP + browser/device/interaction signals), that it's strictly-necessary and not used for cross-site tracking/ads, lawful basis (legitimate interests), and a link to Cloudflare's Turnstile Privacy Policy. Turnstile added to the processor list.text-lg address; it's now a single-line pill (px-3 py-2, text-sm) showing IP · version · browser/OS · Copy on one row, taking far less vertical space./api/v1/inbox/new, creates DB rows) and AI Visibility (/api/v1/tools/ai-visibility, spends LLM tokens). Defense-in-depth — the per-IP rate limits stay as the cost floor; Turnstile raises the bar for automation on top. New src/lib/security/turnstile.ts verifies the token server-side with Cloudflare's siteverify (fails closed on a missing/denied token, open if Cloudflare itself is unreachable — same philosophy as the rate limiter, and the per-IP limit still caps abuse in that window). New src/components/turnstile.tsx useTurnstile() hook loads the widget script once and hands the client a single-use token sent in the x-turnstile-token header. Dormant until configured: with no NEXT_PUBLIC_TURNSTILE_SITE_KEY + TURNSTILE_SECRET_KEY nothing renders and no token is required, so behaviour is unchanged — requiring both keys avoids locking users out with a half-config. CSP widens to https://challenges.cloudflare.com (script-src + frame-src) only when a site key is set; dormant deployments keep the tighter frame-src 'none'. +8 tests (417 total).analytics_storage, ad_storage, ad_user_data, ad_personalization set to denied, with wait_for_update: 500), set in the GA init before gtag('config') — so no analytics cookie is written until the visitor opts in. A new bottom cookie banner (CookieBanner, production-only) offers Accept / Reject with a link to the privacy policy; on Accept it calls gtag('consent','update',{ analytics_storage:'granted' }) and persists the choice to localStorage (cs_consent). Returning visitors who accepted get consent re-applied immediately on load; those who rejected (or never chose) stay denied. A "Cookie settings" link in the footer reopens the banner so the choice can be changed at any time (GDPR withdrawal of consent). Ad-related signals stay denied regardless — this is analytics-only./api/v1/inbox/new returns 429 (a visitor minted many test addresses quickly), the client now shows a dedicated "You've created a lot of test addresses just now. Please wait a minute, then refresh this page." notice instead of the misleading generic "Couldn't start the test" error.G-FG8G8SKBVB) loaded via next/script (afterInteractive) in the root layout — production only, so local dev doesn't pollute the property. Configurable via NEXT_PUBLIC_GA_ID. The Content-Security-Policy was widened to allow it: script-src adds www.googletagmanager.com; connect-src adds the Google Analytics collection domains. (Note: GA4 sets cookies — a GDPR cookie-consent gate is the recommended next step.)rateLimitKv — a fixed-window limiter backed by the KV store (Supabase), so the limit holds across all serverless instances, unlike the in-memory limiter whose counter is per-instance (weak on Vercel). Applied to key-minting (/api/v1/keys), AI Visibility (/api/v1/tools/ai-visibility, each call spends LLM tokens), shareable-report creation (/api/v1/reports, re-runs a scan), and inbox-test address minting (/api/v1/inbox/new, creates DB rows). Falls back to the in-memory limiter when KV is unconfigured and fails open on a KV error (never blocks a legitimate user). +2 tests (409 total).admin-data.ts.inbox-test, score, sender as the target), so they appear even if the visitor closed the tab. Removed the duplicate client-side event./tools/inbox-test) — the "send us an email, get a spam/inbox score" tool (mail-tester style). The visitor gets a unique one-time address (test-<token>@crawlsonar.com); Cloudflare Email Routing forwards the raw message to POST /api/v1/inbox/ingest (authenticated by a shared INBOX_INGEST_SECRET, timing-safe), which parses the receiving MTA's Authentication-Results (SPF/DKIM/DMARC verdicts), extracts the sending IP, resolves reverse DNS, checks a couple of bulk-sender signals, and stores a scored report in Supabase keyed by the token. The page mints the address and auto-polls until the email lands, then shows the inbox-readiness score with fixes and Learn links. — New endpoints: POST /api/v1/inbox/new, POST /api/v1/inbox/ingest, GET /api/v1/inbox/result/[token]. Pure, unit-tested MIME-header parsing + scoring (inbox.ts, 7 tests, 407 total). Hex token survives MTA lowercasing. Dormant-until-configured (needs Supabase + INBOX_INGEST_SECRET). First revenue-shaped tool (free now; credit-gating to follow with the paid OG tier).safeFetch now signs every hop of our outbound requests with an Ed25519 signature (Signature / Signature-Input / Signature-Agent headers, tag="web-bot-auth"), so the sites we scan can cryptographically verify the request genuinely came from Crawlsonar rather than an impersonator. The matching public key is published at /.well-known/http-message-signatures-directory (JWKS). — Dormant-until-configured: set WEB_BOT_AUTH_PRIVATE_JWK (an Ed25519 private JWK, server-only secret) to enable; with no key we don't sign and the directory 404s — zero overhead and safe to ship. — Pure, unit-tested core (web-bot-auth.ts): RFC 9421 signature-base construction, RFC 7638 JWK thumbprint key id, and a full sign→verify round-trip test (generate an Ed25519 key, sign a request, verify the signature against the published public key). 5 tests (400 total). Completes the honestly-achievable agent-readiness set.who.ts, unit-tested): we store the two-letter country (from the CDN geo header) and a short salted one-way hash of the IP (cid) — never a raw IP or user agent, keeping the cookieless/no-PII discipline while still letting the owner tell repeat visitors and abuse apart. — Target domain is recorded for the scans that report it (HSTS, DNSSEC, DKIM, Agent Readiness now; more to follow); who/when/tool is captured for every tool. Durable on Supabase.Accept: text/markdown now gets a clean, link-rich markdown briefing (generated from the tool registry at /api/markdown) instead of the full HTML. Wired into the existing origin proxy, scoped to / and gated on the Accept header, so ordinary browsers (which send text/html) are completely unaffected. Completes another agent-readiness signal — the last of the honestly-achievable ones before Web Bot Auth./.well-known/agent-skills/index.json (Agent Skills Discovery RFC v0.2.0). Generated live from the public API tool registry: every check is a "skill" an agent can invoke via POST /api/v1/run, each with a genuine sha256 digest of its descriptor (not a placeholder) and clear invocation instructions (HTTP + MCP). Part of making crawlsonar itself agent-ready.src/lib/kv.ts now speaks to two interchangeable backends behind one interface, chosen by env, with Supabase preferred: set SUPABASE_URL + SUPABASE_SECRET_KEY (the new sb_secret_… key; legacy SUPABASE_SERVICE_ROLE_KEY also accepted) and every dormant feature (shareable reports, benchmark histogram, monitoring, API keys, AI cache) lights up. Upstash/Vercel KV (Redis REST) remains as the alternative; with neither set, the app falls back to the filesystem stores (local dev unchanged). — Redis-style ops map onto three tables + atomic RPC functions: kv_store (SET/GET/INCR + TTL, with expiry-aware counter reset), kv_hash (HINCRBY/HGETALL), kv_list (RPUSH/LTRIM/LRANGE, negative-index aware). SQL in supabase/0001_kv_store.sql (+ consolidated Total_incorporating_0001.sql): RLS enabled on every table with no policies (only the secret key / service_role bypasses), and RPC EXECUTE revoked from anon/authenticated — least privilege by default. — The Upstash wire format is byte-for-byte unchanged; +8 unit tests cover the new Supabase path and backend precedence (390 total). Dormant-safe: shipping this changes nothing until the Supabase env vars are set./.well-known/mcp/server-card.json (SEP-1649 shape: serverInfo, transport, capabilities), in addition to /.well-known/mcp.json, so agents and audits that look for either convention find it. The Agent Readiness Checker now probes both paths too./tools/secret-generator) — generate a cryptographically secure random secret with a user-set length (1–512 bytes) in hex, base64url or base64, with an optional uppercase toggle. Uses the Web Crypto API (crypto.getRandomValues) entirely client-side — nothing is sent to a server. Shows entropy (bits) and character count; one-click copy and regenerate. Live-verified: 32 bytes → 64 hex chars, length control works. Wired into homepage search, sitemap, /tools and llms.txt./.well-known/security.txt (RFC 9116) — machine-readable security contact. — /.well-known/mcp.json — a manifest describing our live MCP endpoint (/api/mcp, streamable-http, protocol 2025-06-18). — /openapi.json — a compact OpenAPI 3.1 description of the public API (/api/v1/run, /api/v1/keys, /api/mcp). — /.well-known/api-catalog (RFC 9727 linkset) pointing at the OpenAPI + docs. — Discovery Link headers on the homepage response (rel="describedby" → llms.txt, rel="service-desc" → OpenAPI, rel="api-catalog"), emitted from next.config headers. — Together with the existing llms.txt and explicit AI-crawler rules, this takes crawlsonar.com's own Agent Readiness score from failing to a strong pass — the tool now practises what it preaches./tools/og-image-generator) — a free tool that renders a crisp 1200×630 social-share PNG from a title, subtitle, brand and one of six themes, server-side via next/og (/api/og-image). Instant, no sign-up, no watermark, zero marginal cost. Live preview + one-click PNG download. Live-verified: the endpoint returns a valid 1200×630 image/png. Wired into homepage search, sitemap, /tools and llms.txt. — Sets up the paid tier: the page carries a dormant "AI artwork — coming soon" upsell. The paid path (AI-generated backgrounds via OpenAI gpt-image-1, sold as one-off credit packs through Stripe with a KV credit ledger) is deferred until KV + Stripe + OPENAI_API_KEY are configured — money code shouldn't ship untested against real keys./learn). Every concept behind our checks, explained twice on its own SEO page: once for a total beginner (with an everyday analogy), once for a professional, plus how-to-fix steps, an FAQ and links to the relevant tools. 13 launch entries covering HSTS, SPF, DKIM, DMARC, DNSSEC, HTTP security headers, structured data, canonical URLs, meta descriptions, AI crawlers & robots.txt, llms.txt, crawler/WAF bot-blocking and SSL/TLS. Each page emits DefinedTerm + FAQPage + BreadcrumbList JSON-LD; the index emits DefinedTermSet. Wired into the header nav, sitemap and llms.txt.IssueList now maps each finding to its knowledge-base entry (learnSlugForIssue), so any issue we can explain automatically gets a beginner-friendly Learn link — across every tool at once. For laypeople building their first site, not just pros.kb.ts (entries + issue→slug resolver, 6 unit tests, 382 total). Easily extensible — adding a topic is pure content./tools/agent-readiness) — a new standalone tool for the emerging "agentic web": where the AI Readiness Checker scores how an LLM reads a *page*, this probes the origin-level discovery surface autonomous AI agents use. Nine real, observable signals: llms.txt, explicit AI-crawler rules in robots.txt, Markdown content-negotiation (Accept: text/markdown), an MCP manifest (/.well-known/mcp.json or /mcp), OpenAPI / API-catalog description, OAuth discovery metadata (/.well-known/oauth-authorization-server / -protected-resource), discovery Link headers, and security.txt (RFC 9116). We only score signals we can actually observe over HTTP/DNS — speculative agentic-commerce / bot-auth proposals are deliberately left out so the score stays honest. — Pure buildAgentReadinessReport + SSRF-safe checkAgentReadiness (parallel probes, never throws). 5 unit tests (376 total). Wired into the homepage search, public API + MCP (agent-readiness), sitemap, llms.txt and /tools. Live-verified: github.com correctly detects its security.txt; crawlsonar.com detects llms.txt + 9 AI-crawler rules.www.ukimmigration.law (privacy/terms and publication dates flagged missing when they exist): — Bot-wall detection (new bot-wall.ts, unit-tested) — when a site sits behind Cloudflare Bot Fight Mode / a managed challenge (often served as HTTP 200 with an interstitial) or another WAF (Akamai, Imperva, PerimeterX, DataDome, Sucuri), our crawler was parsing the challenge page and reporting everything as missing. The AI Readiness Checker now detects the block, marks the unreadable content checks not-applicable instead of failed, and raises one headline finding: *"Your site is blocking automated crawlers"* — with the key insight that the same block hits GPTBot / ClaudeBot / PerplexityBot / Google-Extended / Googlebot, so the content can be invisible to AI and search. Fix guidance points at Cloudflare "Verified bots" / WAF allow rules. — Privacy/terms detection broadened — now matches the link text as well as the href, across privacy / terms / cookie(s) / GDPR / legal / disclaimer / data-protection / impressum. (Fixed the ukimmigration.law false negative.) — Dates detection broadened — the check is about dates being *visible*, so it now also accepts a clearly-visible date in the text ("8 July 2026", "July 8, 2026", "2026-07-08"), plus article:published_time / article:modified_time / itemprop date meta, not only <time> and JSON-LD. — Live-verified on www.ukimmigration.law: privacy + dates now pass (100/A); +7 tests (371 total).ScoreCard/IssueList): — Plain-English verdict under every score: one honest, non-alarmist line that tells a non-technical visitor what the number means (e.g. *"In plain English: this needs attention — several important things are failing below."* vs *"…this looks healthy — no urgent problems found."*). — One-click fixes — issue Fix text now renders backtick snippets as real inline code, and any fix that contains a copy-pasteable snippet gets a "Copy fix" button (plus a Copy button on example blocks). One consistent affordance across the whole site via a new shared <CopyButton>. — New pure, unit-tested report-format.ts (plainVerdict, splitInlineCode, firstCodeSnippet; 7 tests). Live-verified: HSTS on example.com → Grade F, plain verdict shown, "Copy fix" button on the missing-HSTS fix.bench:v1:<tool>, fields "0".."100") — never the domain, IP or any identifier. Nothing to leak. — Dormant-until-configured: with no Upstash/KV set, the endpoint returns { enabled: false } and the badge renders nothing — so it's invisible and harmless until KV is wired (activates the moment KV_REST_API_URL/KV_REST_API_TOKEN are set). Percentile is also hidden below a 30-scan minimum so it's never misleading. — New pure computePercentile() (unit-tested, 7 tests), recordAndRank()/getRank() (never throw), POST /api/v1/benchmark, and a reusable <BenchmarkBadge> wired into the Domain Audit, HSTS Checker and DNSSEC Checker. Added kvHincrby/kvHgetall to the KV client.itemscope/itemtype) or RDFa (typeof) — so a missing-schema page is caught in the ordinary SEO audit, with the detected @types shown as evidence and a fix that links to the JSON-LD Generator and Structured Data Checker. Also flags invalid JSON-LD blocks. Live-verified: crawlsonar.com → passed (Organization, WebSite); example.com → failed (no structured data). Reuses the already-tested parseJsonLd; feeds the /tools/seo-checker tool and the domain audit./tools/hsts-checker) — a dedicated Strict-Transport-Security tool. Fetches the site over HTTPS, parses max-age / includeSubDomains / preload, and queries the official hstspreload.org status API to report whether the domain is actually on the browser preload list. Catches the real footguns: max-age=0 (HSTS disabled), a short lifetime, missing includeSubDomains, a preload directive that isn't eligible, "eligible but not submitted", and — importantly — "on the preload list but the live header no longer qualifies". Pure parseHsts + buildHstsReport, unit-tested; live-verified github.com = 100/A, on the preload list./tools/dnssec-checker) — validates the full chain of trust (root → TLD → zone) over DNS-over-HTTPS. Shows the three links (zone signed via DNSKEY, anchored via DS in the parent, validates to root via the resolver AD flag) and flags the dangerous states that quietly take domains offline: DS with no DNSKEY (a live resolution outage — critical), bogus (DS/DNSKEY mismatch after a key rollover — high), island of security (signed but not anchored — medium), and deprecated crypto (RSASHA1 / SHA-1 DS). Pure parsers + buildDnssecReport, unit-tested; live-verified cloudflare.com = 100/A.hsts, dnssec tool ids), sitemap.xml, llms.txt and the /tools index.256 3 ECDSAP256SHA256 …), not a number. The first cut parsed it as a number, dropped every DNSKEY, and falsely reported healthy domains (cloudflare.com!) as a "critical broken chain". algNumber() now accepts both the mnemonic and numeric forms; caught in live verification before shipping, with a regression test added.max-age / includeSubDomains / preload), CSP presets (strict / basic / none) + report-only rollout + nonce path, X-Frame-Options (DENY / SAMEORIGIN / none), nosniff, Referrer-Policy, Permissions-Policy. — SPF Generator — a, mx, ip4/ip6, include, and -all/~all/?all/+all, plus the one-record and 10-lookup rules. — DMARC Generator — p, sp, pct, rua/ruf, adkim/aspf, and the safe three-stage rollout. — JSON-LD (Schema) Generator — when to use Organization / WebSite / Article / LocalBusiness / Product, and the visible-content rule. — Open Graph Generator — every og:/twitter: field, image sizing (1200×630), and cache/absolute-URL pitfalls. — CAA Generator — CA issue list, issuewild, and iodef reporting. — MTA-STS & TLS-RPT Generator — mode (testing / enforce / none), MX hosts, max_age, TLS-RPT rua, and how to publish the policy. — AI robots.txt Generator — the three presets (protect-training / allow-all / block-all) and the per-crawler allow/block toggles + opt-out tokens. — llms.txt Generator — what llms.txt is and how the draft is built.env: { NEXT_PUBLIC_APP_VERSION } into the statically prerendered server-component layout, so the value was empty in production. Replaced the env/process.env indirection with a deterministic generated literal (src/lib/app-version.ts), imported by the layout — works identically across static prerender, dynamic render and Vercel serverless. Removed the now-dead fs read + env block from next.config.ts. Verified · v0.49.0 live.dkim, hsts, spf, core web vitals…), keyboard navigation and Enter-to-open. Backed by a new canonical src/lib/tools-index.ts (single searchable index with keywords). Makes the growing toolset discoverable./tools/dkim-checker) — the missing third of the SPF/DKIM/DMARC trio as a standalone tool. Looks up a domain's DKIM public key by selector (or probes the common provider selectors), validates it, and reports key type, estimated strength (flags weak ~1024-bit RSA), and testing/revoked flags — with fixes. Pure record parser + report builder, unit-tested. Live-verified against github.com (selector google, RSA, strong). — 11 new unit tests (DKIM parsing/scoring + tool-search ranking). 323 total. Build compiles. HSTS was already covered by the HTTP Header Checker + Security Headers Generator (now instantly findable via search)./tools/fix-pack) — the "don't just find problems, fix them" tool. Enter a domain and get copy-paste-ready, pre-filled snippets for the most common SEO/AI/security gaps: HTTP security headers, an AI-crawler robots.txt policy, SPF + DMARC anti-spoofing records, Organization JSON-LD and an llms.txt skeleton — each with where-to-apply guidance and a copy button. Fully client-side (reuses the pure generator libs), so it's instant and private. Live-verified: 6 blocks generate pre-filled for the domain. Registered in /tools, sitemap, llms.txt.POST /api/mcp (Streamable HTTP, JSON-RPC 2.0) exposes all 13 registry tools (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview) as agent-callable tools — so an AI assistant can audit a domain itself. Reuses the shared tool registry, so MCP, the public API and the UI run one implementation. initialize / tools/list / tools/call / ping / notifications handled; SSRF-safe; rate-limited. Documented on /api-docs and in llms.txt. — 7 new unit tests over the pure JSON-RPC router. 312 total. Live-verified: full handshake + a real tools/call (whois google.com → registrar + score via MCP).host directive and kept /monitor/ out of crawling. Resolves the "no explicit AI crawler policy" finding from the self-audit and lifts the AI Readiness + robots.txt module scores.next/og (verified 200 PNG).package.json, exposed via NEXT_PUBLIC_APP_VERSION in next.config.ts) now shows as · v<x.y.z> in the global footer, so it's obvious which build is live.ToolFaq (visible accordion + FAQPage structured data) to the remaining 24 tool pages, so every tool page now has an FAQ (only the /tools index doesn't, as it's a listing). Better AEO/rich-result coverage and thicker content on the lighter utility/generator pages. 305 tests green; build compiles; live-verified schema + visible FAQ render (incl. escaped-quote content).force-dynamic root layout back to a static CSP baseline, so tool pages and guides are prerendered/CDN-cacheable again — build went from 44 dynamic / 2 static to ~29 static / 20 dynamic (the remainder are API routes and genuinely dynamic pages). Recovers TTFB, LCP and crawl efficiency; the static CSP keeps frame-ancestors 'none', object-src 'none' etc. (the app has no script-injection sink, so 'unsafe-inline' is an acceptable trade for static generation).SearchAction pointing at the domain audit) in the root layout — the brand entity search and AI systems need. Previously the site had none.opengraph-image (auto-applied to every page) plus a default twitter:card = summary_large_image and og:site_name. The site no longer ships imageless share links.icon, apple-icon and manifest.webmanifest; removed the leftover create-next-app files (next.svg, vercel.svg, file/globe/window.svg) from public/./ (it previously only inherited the layout defaults).ToolFaq component (visible accordion + FAQPage JSON-LD); added FAQs to 16 key tool pages (audit, AI Visibility/Readiness, deliverability, CWV, SSL, DNS, HTTP headers, security, SPF, DMARC, WHOIS, subdomains, link preview, robots, sitemap). toolJsonLd already emitted BreadcrumbList for every tool. — 305 tests green; production build compiles; verified live: homepage emits Organization+WebSite+SearchAction, OG image serves a 200 PNG, tool pages carry FAQPage schema + visible FAQ.crawlsonar.com isn't connected yet, so nothing is indexable until a public production deploy; wire Google Search Console + analytics after launch.src/content/guides/*.ts) rendered by one template (src/components/guide-article.tsx) that auto-generates, for every guide: — Structured data: TechArticle + BreadcrumbList + FAQPage + optional HowTo JSON-LD (verified: the GEO pillar emits all four). — An anchored table of contents, reading time, category, published/updated dates, semantic headings with deep-link anchors, callouts, tables, code and step lists, an FAQ accordion, and dense internal linking to related tools and guides./guides hub with CollectionPage JSON-LD, and the full set surfaced in sitemap.xml (with real updated dates) and llms.txt (categorized) for AI discovery. — Build compiles (52 routes); 305 tests green; /guides/<slug> renders with TOC + FAQ + 4 schemas, /guides lists all 15, unknown slugs 404. Old static guide routes removed (migrated into data).src/proxy.ts now issues a fresh per-request nonce and sets script-src 'self' 'nonce-…' 'strict-dynamic', so 'unsafe-inline' (and 'unsafe-eval') are gone from script-src in production. Next.js reads the nonce from the request CSP header and stamps it on its own inline scripts. application/ld+json is data (not gated by script-src), so structured data stays intact with no nonce. The five static headers (HSTS, nosniff, X-Frame-Options, Referrer-Policy, Permissions-Policy) remain in next.config.ts. — Verified in a production build (next start): CSP nonce and the nonce on Next's scripts match within a single request, no 'unsafe-inline'/'unsafe-eval', JSON-LD present, client hydration + fetches work. 305 tests green. — Trade-off (deliberate): a per-request nonce can't live in a prerendered page, so the root layout is now export const dynamic = "force-dynamic" — the whole app renders at request time (no static/CDN HTML caching). For a free-tools SEO site that's a real speed/cost cost for a defense-in-depth gain (the app has no script-injection sink — React escapes all input; the only dangerouslySetInnerHTML is our own JSON-LD). To restore static generation, revert this commit (drops back to the 0.42.0 'unsafe-inline' baseline).next.config.ts headers(): Content-Security-Policy, Strict-Transport-Security (2y, includeSubDomains), X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: strict-origin-when-cross-origin, a locked-down Permissions-Policy, and X-DNS-Prefetch-Control. — CSP is tuned to the app: frame-ancestors 'none', object-src 'none', base-uri 'self', form-action 'self', img-src 'self' data: https: (the Link Preview tool renders remote og:images), connect-src 'self', upgrade-insecure-requests. script-src/style-src keep 'unsafe-inline' because every tool page emits inline JSON-LD (structured data is core to the product) and Next hydrates inline; dev also allows 'unsafe-eval' + ws: for HMR. — Verified live: all six headers present, no CSP console violations, client fetches (connect-src 'self'), hydration and inline JSON-LD all intact. — Follow-up (tracked): tighten script-src to a per-request nonce (middleware-injected) to drop 'unsafe-inline' — the last hardening step./tools/ai-crawler-log) — the observational counterpart to the robots.txt generator: upload or paste a web-server access log to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended and ~15 more) actually visited, how often, and whether they were blocked (403/401) — which quietly makes you invisible to those AI systems. [Tier 2] — Fully client-side (same privacy stance as the Email Header Analyzer): the log is parsed in the browser and never uploaded. Pure analyzeCrawlerLog / parseLogLine (combined-log UA extraction, bot matching, blocked-hit detection, sorting) are unit-tested. 305 total tests. Build 86/86. Live-verified: 4 AI hits across GPTBot (2, incl. one 403), ClaudeBot and PerplexityBot, human traffic ignored./tools/compare) — head-to-head audit: runs the full Domain Audit on two domains in parallel and scores them module by module, highlighting the winner of each check and overall. Turns an abstract score into a competitive to-do list; naturally shareable. Reuses runDomainAudit; pure compareAudits(a, b) diff is unit-tested (per-module winner, ties, ran-only-one-side, overall). [Tier 3] — Route maxDuration = 60, tight rate limit (two audits). 300 total tests. Build 85/85. Live-verified: github.com 74 vs gitlab.com 73 in 3.7s, with a sensible per-module breakdown (ties handled).POST /api/v1/run with { tool, url }), plus a docs page at /api-docs with a one-click free-key generator, curl example, the tool table and limits. Opens integrations, CI and agent workflows (and the paid-tier upsell). [Tier 3] — Accountless API keys: POST /api/v1/keys mints a cs_live_… key, shown once; only its SHA-256 hash is stored (KV in prod, filesystem in dev). Auth via Authorization: Bearer or x-api-key. Per-key daily quota metered in KV (free tier 100/day, API_FREE_DAILY_LIMIT) with x-ratelimit-* headers, plus a per-IP burst cap and a mint cap. — A single tool registry (src/lib/api/registry.ts) maps 13 tool ids to the existing check modules (domain-audit, dns, ssl, http-headers, whois, email-deliverability, ai-readiness, llms-txt, performance, core-web-vitals, security, subdomains, link-preview), normalising input per tool — so the API and the UI share one implementation. Typed check errors are mapped to sensible HTTP statuses; every call stays SSRF-safe. — 8 new unit tests (key mint/hash, header extraction, auth/metering round-trip, registry). 297 total. Build 83/83. Live-verified: 401 unauthenticated → mint key → authenticated dns run 200 with rate-limit headers → unknown-tool 400 listing the 13 tools./tools/monitoring) — the Tier-3 recurring-value engine that turns one-off tools into a reason to come back. Create a monitor for a domain and get alerted on uptime, SSL-certificate expiry (14-day warning), domain-registration expiry (30-day warning) or DNS-record change, via email or webhook. Accountless: creating a monitor returns a private manage link (/monitor/{token}), the same capability-URL pattern as shared reports — no auth to build yet. — Cron (/api/v1/cron/monitor, hourly via vercel.json, CRON_SECRET-gated with a timing-safe check) iterates all monitors, re-runs each check, and dispatches alerts. Alerting is a pure decideAlert(type, prevState, prevAlarm, outcome) function (first-run seeding, threshold enter/leave, change detection) — fully unit-tested — so it only fires on a genuine change and again on recovery (no spam). — Checks reuse the existing TLS / WHOIS / DNS / HTTP modules. Store is the same KV-in-prod / filesystem-in-dev dual pattern as reports. Webhooks are SSRF-safe: safeFetch gained POST-body support so user-supplied webhook URLs are delivered through the same public-IP-pinned fetcher (can't be pointed at internal services). Email reuses the dormant delivery pattern (logs intent until a provider is wired). — 9 new unit tests (alert-decision matrix). 289 total. Build 80/80. Live-verified full lifecycle: create → seeded check (example.com SSL "51 days") → view → cron re-check (no false alert) → manage page (noindex) → delete → 404.CRON_SECRET in production./tools/ai-visibility) — the Tier-2 hero and the product's differentiator. Asks an LLM what it knows about a brand from training, compares that to the brand's own site (fetched for the accuracy check), and returns a 0–100 visibility score + grade, whether it's recognised / accurate / positive, and concrete GEO recommendations to get cited in AI answers. — Margin-protected (global "unit economics for AI features" rule): dormant until ANTHROPIC_API_KEY is set (503 otherwise); cheap Haiku model with small max_tokens; 24h per-domain result cache in KV so repeat scans don't re-bill; a global daily budget counter in KV (AI_VISIBILITY_DAILY_BUDGET, default 500) on top of a hard 3-per-hour per-IP limit. — Honest framing throughout: reflects one model's training knowledge (labelled), not a live crawl of every engine; being unknown is common for smaller brands and is what GEO fixes. LLM output is defensively normalised (enum whitelisting, confidence clamp, recommendation cap) so malformed JSON can't break the report. — Added kvIncr / kvExpire to the KV client for the budget counter. Pure transform + normaliser unit-tested (recognised/unrecognised scoring, finding generation, coercion). 280 total tests. Live-verified dormant path (503 without a key) and page render; the LLM happy-path runs once the key is set in prod./tools/link-preview) — shows exactly how a URL renders as a card on Google, X/Twitter, Facebook/LinkedIn and Slack/Discord, with the Open Graph / Twitter Card fixes to improve it (missing og:image, over-length title/description, missing twitter:card). Reuses the existing HTML parser; extracts OG + Twitter tags, resolves relative image/canonical/favicon URLs to absolute, and renders four faithful preview cards. Viral, shareable, SEO-adjacent. [Tier 1] — SSRF-safe (page fetched via safeFetch). Pure extractPreview(html, url) transform, unit-tested (OG-over-base fallback, relative-URL resolution, warning generation, favicon default). 274 total tests. — Live-verified against github.com (all four cards render, og:image loads, sensible length warnings). Registered in /tools, sitemap, llms.txt./tools/subdomain-finder) — passively enumerate a domain's subdomains and certificate issuers from public Certificate Transparency logs (crt.sh), and flag exposed dev/staging/admin/internal hosts. A bespoke discovery UI (stat tiles, issuer chips, highlighted sensitive hostnames) rather than the scored report shape, since this is enumeration not pass/fail. [Tier 1] — SSRF-safe (only outbound call is to crt.sh via safeFetch); passive — never scans or brute-forces the target. Route maxDuration = 30 (crt.sh is slow). Pure entries→report transform, unit-tested (SAN parsing, wildcard stripping, domain scoping, issuer aggregation, sensitive-host flagging). 270 total tests. — Live-verified against anthropic.com: 126 subdomains from 3,798 certs, correctly surfacing staging, test, internal.api and portal hosts. Registered in /tools, sitemap, llms.txt./tools/email-deliverability) — the sending-authentication trio (SPF, DKIM, DMARC) plus MTA-STS / TLS-RPT / BIMI in one focused, scored report, with the exact fix per failing check. Adds DKIM selector probing (queries ~20 common provider selectors) on top of the SPF/DMARC parsers reused from the DNS module, and weights everything for inbox deliverability (Gmail/Yahoo now require SPF+DKIM+DMARC for bulk senders). [Tier 1] — DNS-only (no SSRF surface). DKIM is selector-based and can't be fully enumerated — the tool probes common selectors and says so, so a custom-selector setup reads as "no DKIM found" with a clear caveat (verified: google.com's rotating date selectors aren't guessable; github.com's google selector is found). — Pure resolved-facts→report transform, unit-tested (healthy vs broken vs no-SPF). 266 total tests. Registered in /tools, sitemap, llms.txt. Live-verified against google.com (82/B) and github.com (91/A)./tools/whois-lookup) — registrar, registration/expiry dates, registry status, nameservers and DNSSEC for any domain, via the modern RDAP protocol (structured JSON successor to WHOIS). The actionable win is expiry: flags a domain expiring within 30 days (critical if already lapsed), plus any blocking status (hold / pending delete). Feeds the upcoming expiry monitor. [Tier 1] — SSRF-safe: resolved through the rdap.org redirector using safeFetch, which re-validates every redirect hop's IP, so the TLD-dependent final host can't be abused. Falls back to the registrable domain when a subdomain is entered. — 7 new unit tests over a pure RDAP→report transform + input normaliser — 262 total. Registered in /tools, sitemap, llms.txt. Live-verified against google.com and bbc.co.uk (multi-part .co.uk TLD handled)./tools/core-web-vitals) — the biggest missing SEO/perf signal. Powered by Google PageSpeed Insights: real-user field data (CrUX) for LCP / INP / CLS when available, falling back to the Lighthouse lab result for low-traffic sites, plus the overall Lighthouse performance score. Explainable scoring against Google's official "good" thresholds (LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.1), one issue+fix per failing metric. This is the first item of the Tier-1 one-stop-shop expansion. — No SSRF surface (Google fetches the target; our only outbound call is to googleapis.com). Works keyless at low volume; GOOGLE_PSI_API_KEY optional for production reliability. Kept out of the synchronous Domain Audit on purpose — Lighthouse is slow (route maxDuration = 60, tight 6-per-5-min rate limit). — 5 new unit tests over a pure PSI→report transform (field-preferred, lab fallback, INP-not-applicable, severity capping) — 255 total. Registered in /tools, sitemap and llms.txt. Live-verified: graceful 429 handling keyless; page renders.src/lib/kv.ts, explicit JSON (de)serialisation, 5s timeout) behind the existing store interfaces. Report TTL is now native (SET … EX), so shares self-expire with no sweep/cron. The event list is capped (LTRIM) to bound growth. — Dormant-until-configured (same pattern as origin cloaking / email / admin): with no KV_REST_API_URL+KV_REST_API_TOKEN (or UPSTASH_REDIS_REST_URL/TOKEN), the app keeps using the filesystem stores — so local dev needs no Redis and nothing else changed for it. — 11 new unit tests (mocked-fetch REST wiring, env gating, JSON round-trip) — 250 total. Report share round-trip re-verified live over HTTP on the filesystem path.next.config.ts: outputFileTracingIncludes force-bundles pdfkit's .afm font-metric files into the PDF route, avoiding the classic ENOENT … Helvetica.afm failure on Vercel..env.example documents the KV vars and that they're required on Vercel.maxDuration = 60 (needs a Vercel Pro plan; Hobby caps functions at 60s but with fewer resources).[company number] / [registered office address] placeholders and the publish-blocker TODO, so /privacy is now legally complete for launch.crawlsonar.com). The product is now Crawlsonar everywhere it matters: metadata title/template, llms.txt, PDF report brand, shared-report Open Graph images, User-Agent strings (Crawlsonar/0.1 (+https://crawlsonar.com; …)), JSON-LD publisher/author, the AI robots.txt generator's output comment, and the privacy policy. Default site URL is now https://crawlsonar.com (still overridable via NEXT_PUBLIC_SITE_URL).[email protected]; .env.example example domains updated to crawlsonar.com (site URL, abuse, from-address)./audit/domain report gets the "🔗 Share this report" button (tokenised /r/{token} link, authentic-by-construction re-run), the PDF lead magnet and email capture, exactly like the other tools. — Extended the report store to a three-way discriminated union (ai-readiness | llms-txt | domain-audit) via a new ReportPayload type; POST /api/v1/reports and POST /api/v1/leads re-run runDomainAudit server-side for audit shares. — PDF renderer now handles the audit shape (overall score/grade header, per-module score breakdown, merged issue list from every module). — Extracted an AuditBody presentational component shared by the live audit page and the read-only /r/{token} page, so the two can't drift.%PDF) — 189 total. Verified live end-to-end: shared a github.com audit → /r/{token} renders (200, noindex, audit body) → 5.4 KB PDF with full module breakdown + all 9 issues./audit/domain + POST /api/v1/audit/domain) — the product's headline promise: one input, one coherent report (Requirements v1.0 §1/§7/§8). Classifies the input (domain/URL/email → origin+url) and runs all seven modules in parallel via Promise.allSettled with per-module error isolation (a module that can't run — e.g. TLS on an HTTP-only host — is shown but excluded from the score, so one failure never sinks the report). Aggregates an impact-weighted overall score (AI 25, HTTP 18, TLS 15, DNS 15, robots/sitemap/llms 9 each), severity counts and a merged, severity-sorted issue list; each module card links to its dedicated tool prefilled with the target. Stricter rate limit (5 / 5 min) and maxDuration: 60./tools/robots-txt-checker + POST /api/v1/tools/robots): reuses the shared robots parser + AI-crawler matrix to detect full-site blocks (critical), missing Sitemap directives, and whether an explicit AI crawler policy exists; shows the parsed AI matrix and raw file. Clarifies robots.txt controls crawling, not indexing./tools/sitemap-checker + POST /api/v1/tools/sitemap): reuses parseSitemapXml to confirm sitemap.xml exists and is valid, counts URLs (following the first child of a sitemap index) and samples up to 6 URLs (SSRF-safe fetch) for reachability./tools/ssl-checker + POST /api/v1/tools/ssl). Inspects the certificate: validity window, days-to-expiry (with expiring-soon warnings), hostname match (SAN/CN incl. single-label wildcards), chain trust vs default roots, negotiated protocol (flags < TLS 1.2), key strength, plus the full chain and connected IP. SSRF-safe: resolves the host, validates every IP with isPublicIp, and connects to the pinned IP with servername set for SNI/identity. Pure cert-evaluation helpers (SAN parse, wildcard match, validity) are unit-tested./tools/http-headers-checker + POST /api/v1/tools/http-headers). Through the SSRF-safe fetcher: redirect chain with per-hop status, HTTPS-final + HTTP→HTTPS-upgrade probe, security-header inspection (HSTS, CSP, X-Frame-Options/frame-ancestors, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP/COEP/CORP), cookie-flag analysis (Secure/HttpOnly/SameSite) and server/framework version-leak detection. Header analysis is a pure, unit-tested function. Redirect-chain view + security-header table + issues. Homepage card, sitemap, own llms.txt, cross-links./tools/dns-lookup + POST /api/v1/tools/dns) — first sprint-3 commodity tool (internal-link hub). Resolves A/AAAA/CNAME/MX/TXT/NS/SOA/CAA via node:dns with bounded timeouts, parses SPF (with a 10-DNS-lookup-limit count) and DMARC (policy strength). Issue detection: multiple SPF records (critical), SPF over 10 lookups (high), missing SPF/DMARC (medium), p=none (low), missing CAA (low), no address records (high). Records tables, email-authentication panel, explainable score. Input accepts a bare domain, URL or email address (domain extracted). Homepage card (DNS no longer "coming soon"), sitemap, own llms.txt, cross-links./tools/ai-robots-txt-generator) — the actionable other half of the AI Readiness checker's crawler matrix (Requirements v1.0 Appendix B). Per-crawler Allow/Block toggles grouped by purpose (training / AI search / user-triggered), three presets ("Block training, allow AI search" default, "Allow all", "Block all"), live copyable/downloadable robots.txt output. Honest framing baked in: robots.txt is advisory, and Google-Extended/Applebot-Extended are opt-out tokens (noted only when blocked). Pure client-side — no network, no SSRF surface.no_ai_crawler_policy issue fix now points at this generator; homepage card, sitemap entry, own llms.txt entry, related-tools links./r/{token} pages. — Real PDF, no headless browser: src/lib/pdf/report-pdf.ts renders either report type with pdfkit (header, score, category breakdown, severity-coloured issues, full checks list, footer disclaimer). serverExternalPackages: ["pdfkit"] keeps its font metrics loadable at runtime. Downloadable at GET /api/v1/reports/{token}/pdf (token-validated, rate-limited). Verified live: valid 4.3 KB %PDF with correct content-type/disposition. — GDPR-first capture: explicit consent checkbox is mandatory (consent: z.literal(true); submit disabled until email valid + consent checked). Leads stored minimally (email, tool, target, consent timestamp, no IP) in a swappable filesystem LeadStore (.data/leads/leads.jsonl). POST /api/v1/leads re-runs the scan server-side to persist an authentic report, stores the lead, and returns a token + PDF URL so the download works immediately. — Dormant email delivery (src/lib/leads/email.ts): active only when EMAIL_API_KEY + EMAIL_FROM are set (provider send is a launch TODO); until then it logs intent (domain only, never the full address) and the UI honestly offers the download without claiming an inbox delivery./r/{token}. — Authentic by construction: POST /api/v1/reports RE-RUNS the scan server-side rather than accepting client JSON, so a shared report about someone else's site cannot be fabricated. — Security: 128-bit crypto-random base64url tokens; every filesystem path gated by a strict token regex (path-traversal-proof, verified live with ..%2F..%2Fetc%2Fpasswd → 404); 30-day expiry with lazy-delete-on-read; 512 KB size cap; report-creation rate limit (12 / 5 min per IP). — Policy: /r/* pages are noindex, nofollow and disallowed in robots.txt; visible point-in-time disclaimer ("not a security certification"); owner opt-out contact via optional NEXT_PUBLIC_ABUSE_EMAIL; reports store only public target data (no visitor IP/headers).src/lib/reports/) as a swappable ReportStore interface with a filesystem implementation (.data/reports, gitignored). NOTE documented: swap for object store / DB before any serverless deploy (ephemeral FS).src/components/report/): ScoreCard, IssueList, ChecksList, CategoryBars, CrawlerTable, AiReadinessBody, LlmsTxtBody. Both live checkers refactored to use them, so the live and shared-report views can never drift./r/{token} → renders, noindex, disclaimer, report_viewed{shared:true} event./tools/ai-readiness-checker + POST /api/v1/tools/ai-readiness) — the flagship differentiator (Roadmap v1.1 sprint 2). Scores a page across seven weighted categories summing to 100 (Requirements v1.0 §17): crawlability (20), content extraction (20), entity clarity (15), structured data (15), trust signals (10), AI crawler governance (10), agent readiness (10). Per-category breakdown bars, evidence+fix issue cards, an AI crawler policy matrix (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended → allowed/blocked/partial/unspecified from robots.txt), and a markdown extraction preview ("what an LLM sees"). Analysis is on server-rendered HTML only (no JS) — stated honestly in the UI, since that's what crawlers/LLM fetchers see.robots-rules.ts (robots.txt parser with Google-style longest-match path matching, per-agent group selection, wildcard/$ patterns, AI crawler matrix) and html-analysis.ts (main-content extraction + noise ratio + markdown, heading outline, JSON-LD parsing, form-label a11y, link-text quality, alt coverage, page signals).parseHtml dropped <script> text, and a noise-ratio fixture was miscalibrated. 128 tests total. Verified live against MDN (no JSON-LD → schema 0) and stripe.com (WebSite+Organization → schema 15/15)./tools/llms-txt-generator + POST /api/v1/tools/llms-txt-generator): builds a draft llms.txt from sitemap.xml (sitemap-index support, 500-URL cap) or homepage nav links as fallback, with H1/summary from homepage title/og/meta description. URL categorisation into Documentation / API / Products / Pricing / Blog / Company / Legal sections; per-section (12) and total (40) link caps keep output curated. Editable textarea, copy + download, "this is a draft — curate it" guidance. Full SEO page template; cross-linked with the llms.txt Checker both ways (checker's missing-llms.txt fix now points at the generator). Verified live: vercel.com (501 sitemap URLs → 40 links, 7 sections) and llmstxt.org via UI.src/proxy.ts gate + src/lib/security/origin-auth.ts. When ORIGIN_SHARED_SECRET is set, every request must carry the Cloudflare-injected x-origin-auth header or the app answers a bare 403 — direct hits on the origin get nothing. Constant-time comparison, minimum 32-char secrets, comma-separated dual secrets for zero-downtime rotation. Unset in dev = no-op. Verified live: dormant 200; active → no header 403, wrong header 403, secret #1 200, rotated secret #2 200, API without header 403.cf-connecting-ip when (and only when) origin cloaking is active — behind Cloudflare the leftmost x-forwarded-for value is client-controlled and would allow rate-limit evasion..env.example: ORIGIN_SHARED_SECRET documentation with generation command and launch-time reminder.src/lib/security/): IP validation blocklist (RFC1918, link-local/cloud-metadata 169.254.0.0/16, CGNAT, multicast/reserved, IPv6 loopback/link-local/ULA/site-local, IPv4-mapped/NAT64/6to4/Teredo embedded forms), validation inside the DNS lookup callback so the validated address IS the connected address (atomic pin — no TOCTOU/rebinding window), whole-fetch refusal when any resolved record is non-public, manual redirect handling with per-hop revalidation, scheme+port allowlists, body-size and time caps. Verified live: metadata IP, decimal-encoded 127.0.0.1 and 127.0.0.1.nip.io rebinding all blocked.src/lib/issues/): severity/category/impact/evidence/fix per Requirements v1.0 §22; critical/high issues cap module scores; not-applicable checks excluded from scoring.src/lib/checks/llms-txt.ts + /tools/llms-txt-checker): availability, content-type/HTML-fallback detection, llmstxt.org structure validation (H1, summary blockquote, sections, links), link-liveness sampling, thin-file detection, llms-full.txt check. Full SEO page template: tool above fold, explanation with honest "Google ignores llms.txt" note, example, common problems, FAQ, unique metadata, canonical, OG, JSON-LD (WebApplication + BreadcrumbList).POST /api/v1/tools/llms-txt: zod validation, per-IP rate limiting (10/min anonymous, verified live), no internal error leakage.scan_started/scan_completed/report_viewed (+ reserved names for the full funnel) via sendBeacon to /api/v1/events; no cookies, no IP, no PII stored.llms.txt, robots.txt, sitemap.xml (dogfooding, launch DoD)./privacy) — a UK GDPR / Data Protection Act 2018 policy with Northstar Infinity Works Ltd as the named data controller, tailored to what the Service actually processes: cookieless analytics with no IP, client-side-only email-header parsing, consent-based lead capture, transient IP for rate-limiting, 30-day shared reports, processors (hosting / Cloudflare / email provider), international transfers, retention, data-subject rights and the ICO complaint route. Linked from the email-capture consent notice, a new homepage footer, and the sitemap. (Draft — placeholders for company number, registered office and contact email to be filled; legal review recommended before launch.)Accept header preferred text/plain, text/markdown over text/html. On sites that content-negotiate (e.g. AI-optimised builds that serve a clean markdown alternate to markdown-preferring clients — ukimmigration.law does exactly this), our HTML analysers received markdown with no <head>, and wrongly reported missing meta description, missing title and no structured data — penalising sites that are, ironically, *more* AI-ready. Changed the default Accept to prefer HTML (text/html,application/xhtml+xml,…). Affected the AI Readiness, On-Page SEO, Structured Data, Security, Privacy checkers and the llms.txt Generator's homepage-metadata fetch. Found via a real-world scan reported by the user..txt) is unaffected. 239 tests still pass./admin, Requirements v1.0 §28 — "admin scan monitoring"). Token-gated, dormant-until-configured: with no ADMIN_TOKEN the admin area does not exist (routes 404); with a token set it requires sign-in. Security-by-design: httpOnly + secure + sameSite=strict session cookie, constant-time token comparison (reusing the origin-cloaking primitive), rate-limited login (5 / 5 min), noindex + robots-disallowed, and it sits behind origin cloaking when that's enabled. Read-only dashboard showing the funnel (event counts), scans by tool, lead count + recent leads, and shared-report count..data/events/events.jsonl, no IP/PII) in addition to stdout, feeding the dashboard; readLeads added. NOTE documented: swap the filesystem stores for a DB before serverless.src/lib/checks/doh.ts queries Cloudflare DoH (application/dns-json) through the SSRF-safe fetcher; a pure parseDohJson + deriveDnssec are unit-tested. The DNS Lookup now reports DNSSEC status (validated / signed / unsigned, from the Authenticated-Data flag plus DNSKEY/DS presence) and whether an HTTPS/SVCB record exists, both surfaced in the tool and scored (unsigned → a low DNSSEC issue). Verified live: cloudflare.com DNSSEC validated (AD + DNSKEY + DS), HTTPS record present./tools/og-tag-generator): og: + twitter: meta tags for rich link previews, XML-escaped. — CAA Generator (/tools/caa-generator): DNS CAA records (issue/issuewild + iodef), including the "disallow all" case. — MTA-STS & TLS-RPT Generator (/tools/mta-sts-generator): the policy file plus both TXT records, with a testing→enforce rollout and a minimum max_age./tools Generators group, sitemap and llms.txt./tools/seo-checker, Requirements v1.0 §15): title length, meta description, canonical, indexability, single H1 + heading order, lang, mobile viewport, Open Graph, image alt coverage and anchor-text quality — reusing the shared HTML-analysis helpers, framed for SEO. Verified live: stripe.com 85/B./tools/privacy-scanner, Requirements v1.0 §21): third-party tracker detection, cookie Secure/HttpOnly/SameSite flags, privacy-policy link and forms-over-HTTPS — with risk wording, not legal conclusions, and an honest note that consent-timing needs a real browser. Verified live: theguardian.com correctly reports policy link + insecure cookies.ReportScanClient — a reusable, RSC-safe generic client (serialisable string props only) for standard {score, issues, checks} tools, used by both new pages. Two thin API routes generated from the shared template. 222 tests (both reuse already-tested analysis helpers; orchestration verified live)./tools/performance + POST /api/v1/tools/performance, Requirements v1.0 §20 MVP). SSRF-safe timed request (resolve → validate → connect to the pinned IP) captures DNS / TCP / TLS / TTFB / total via socket timing hooks, plus HTML size, compression (gzip/br/zstd), cache-control and CDN detection (Cloudflare/CloudFront/Fastly/Akamai/Vercel/Netlify). Timing waterfall bars + issues (slow TTFB, no compression, large HTML). Field Core Web Vitals (LCP/CLS/INP) are explicitly out of scope — they need a real browser (Phase 2, noted in the UI). Verified live: cloudflare.com TTFB 141ms (CDN detected), github.com 164ms (gzip)./tools/structured-data-checker + POST /api/v1/tools/structured-data, Requirements v1.0 §16): fetches a page, extracts and validates all JSON-LD (reusing the AI-readiness parser), lists the schema.org types, flags invalid JSON, and notes Microdata/RDFa presence. Verified live: stripe.com 100/A (WebSite + Organization)./tools/schema-generator, Appendix B): pure client-side builder for Organization, WebSite (with SearchAction), Article, LocalBusiness (PostalAddress) and Product (Offer) from a form; omits empty fields, outputs a ready <script type="application/ld+json"> block. Cross-linked with the checker and the structured-data guide./tools/security-scanner + POST /api/v1/tools/security-scan, Requirements v1.0 §19). Passive checks plus a few light probes of well-known paths, all through the SSRF-safe fetcher with a tighter rate limit (8/min): exposed .git (critical) and .env (critical), mixed content on HTTPS, CORS wildcard-with-credentials, server/framework version leakage, security.txt presence, WordPress REST exposure, and third-party tracker detection (GA/GTM/Meta/TikTok/Hotjar/LinkedIn). UI and AUP disclaim the light active probing ("only scan sites you own or are authorised to test"). Pure findMixedContent + detectTrackers helpers unit-tested./guides index plus four written, cross-linked articles for the most common issues our tools flag: fixing multiple SPF records, adding HTTP security headers (HSTS/CSP), adding structured data (JSON-LD), and server-side rendering for AI. Each guide has Article JSON-LD, unique metadata, and links to the relevant tool/generator; a shared GuideLayout keeps them consistent. Homepage, sitemap (index + articles) and own llms.txt updated. All 5 pages statically generated and verified 200./tools/badge runs a full audit, then hands you HTML and Markdown snippets for a shields-style score badge that links back to a live audit of your domain — the classic SSL-Labs/GTmetrix backlink + brand loop. The badge is a lightweight SVG served from GET /api/v1/badge?label&score&grade, coloured by score and hard-cached (content fully determined by query params). Inputs are sanitised and clamped (score 0–100, label stripped, XML-escaped) to prevent SVG/markup injection — verified live: <script> label → "script", score 999 → 100./tools Utilities group and sitemap. Verified live: valid SVG renders as a green "AI Readiness 83/100 (B)" badge; /tools/badge returns 200./tools/security-headers-generator): HSTS (max-age/includeSubDomains/preload), CSP preset (strict/basic), X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy, output as raw headers or nginx/Apache config. — SPF Generator (/tools/spf-generator): builds a single v=spf1 record from mx/a/ip4/ip6/includes with a live 10-DNS-lookup-limit count. — DMARC Generator (/tools/dmarc-generator): policy, subdomain policy, pct, rua/ruf, adkim/aspf alignment./tools; sitemap + own llms.txt updated.add_header … always; format./tools/email-header-analyzer, Requirements v1.0 §11) — paste raw email headers to see SPF/DKIM/DMARC pass/fail (from Authentication-Results), the delivery path with per-hop delays, sending IP and core message fields, with issues and fixes ("why did my email go to spam"). Fully client-side — the pure src/lib/email-headers.ts parser (header unfolding, auth-result extraction, Received-hop ordering + delay calc) runs in the browser, so pasted headers never reach our server (§11's "never expose pasted headers", taken to its logical end). Cross-links to the SPF/DMARC checkers.DnsFocusClient renders only the relevant slice of the DNS report (record + focused issues) and links to the full DNS report. — Redirect Checker — reuses the HTTP-headers engine to show just the redirect chain. — User-Agent Parser — fully client-side (parseUserAgent extracted to a Node-free src/lib/user-agent.ts), decodes browser/OS/device with a bot flag. — What Is My IP — dedicated page around the IP widget./tools index page grouping every tool (Full audit / AI & SEO / DNS & email / Security & HTTP / Utilities). Homepage now links to it; sitemap rewritten data-driven with all 19 routes; own llms.txt lists the new tools; shared toolJsonLd helper for landing-page structured data./r/{token} link now unfurls as a branded score card (tool, target, big score, grade, issue count, accent-coloured by score) on X / Slack / LinkedIn / WhatsApp. Multiplies the reach of the existing share mechanic — more unfurl clicks → more scans → more email capture. Rendered via next/og (opengraph-image.tsx, Node runtime so it can read the report store); og:image meta auto-wired plus twitter: summary_large_image. Works despite the page being noindex (unfurls ignore indexing). Verified live: 1200×630 PNG, HTTP 200, correct score/grade/issues.cf-connecting-ip first — trustworthy behind our Cloudflare proxy — then x-real-ip/x-forwarded-for), so it shows the real public IP in production; on a direct/localhost connection it honestly notes the loopback/private address instead of pretending. GET /api/v1/ip endpoint. Pure pickClientIp + parseUserAgent helpers.cf-connecting-ip honoured.npm run dev to next dev --webpack. Next 16 makes Turbopack the default, which on this machine fails to generate the app-router route manifest — every app route returned 404 while static public/ files still served. Webpack dev compiles the routes correctly. (Production next build was always fine.)docs/AI_SEO_Network_Toolbox_Roadmap_v1.1.docx (v1.1, 9 July 2026). Supersedes Section 29 of Requirements v1.0. Key changes: inverted build order (AI-readiness wedge ships in sprints 1–2, commodity tools in 3–4, public launch in sprint 2), paid monitoring pulled forward to sprint 5 with AdSense demoted to a revenue floor after it, plus new sections absent from v1.0 — pricing strategy, measurement plan, SSRF fetcher security gates (IP pinning + expanded blocklist), sensitive-data handling, third-party report policy, and a risk register. Architecture directive changed to modular monolith + Postgres queue (pg-boss) instead of 13 worker microservices + Redis.docs/AI_SEO_Network_Toolbox_Requirements.docx (v1.0, 8 July 2026).CHANGELOG.md (this file) and .gitignore.