Skip to content
← All posts
seo infrastructure

Making an SPA tell crawlers the truth

How I built crawler-only HTML snapshots for Royal Subz without replacing the SPA, lying with soft 404s, or reloading Nginx on every content update.

Royal Subz already had pages for every published blog post and active product. A customer could open one, move through the app, and see exactly what they expected.

A crawler had a less certain experience.

The site is a single-page application, so the first HTML response is mostly the shared application shell. Route-specific titles, descriptions, canonical URLs, structured data, and content arrive through the application. That is fine for a browser. For a crawler deciding what a URL means from the response it receives, it leaves too much room for ambiguity.

The obvious requirement was “generate HTML for crawlers.” The real requirement turned out to be much wider: generate the right HTML, for the right routes, from the current deployed application, without changing what normal visitors receive—and fail in a way that tells search engines the truth.

That last part shaped most of the system.

I did not need a second website

The goal was never to recreate Royal Subz as a parallel server-rendered application. The SPA was already the customer experience. I only needed a reliable response layer for two route families where discovery matters:

  • /blog/:slug
  • /product/:slug

I planned the work around that boundary. Human requests should continue to fall through to the normal index.html. Recognized search crawlers should receive route-specific snapshot HTML. Everything else about the application should remain untouched.

Keeping the surface small mattered. Every extra route supported by the snapshot system becomes another route whose publication rules, metadata, deletion behavior, and failure modes have to remain synchronized. Blog posts and products already had clear lifecycle states in Supabase: posts are published or unpublished, and products are active or inactive. That gave the generator an authoritative definition of what should exist.

The snapshot was not a new source of truth. It was a compiled output of the sources I already trusted.

Start from what is actually deployed

My sync script begins by reading the deployed SPA template from SPA_TEMPLATE_PATH. It does not keep its own permanent copy of index.html, and it does not guess which JavaScript or CSS files are current.

That decision looks small until the frontend is deployed.

Production builds generate hashed asset filenames. A snapshot created from last week’s template can contain perfectly current metadata while still pointing at JavaScript and CSS bundles that no longer exist. The crawler gets an HTML document that looks correct at the top and is stale underneath.

I added explicit SEO_SNAPSHOT_START and SEO_SNAPSHOT_END markers to the application template. The script uses that controlled region to inject the route-specific SEO tags and JSON-LD. On every eligible run, it rereads the deployed template, fetches the current published posts and active products from Supabase, and rebuilds each snapshot against the assets that are actually live.

The output is deliberately predictable:

snapshots/
  blog/[slug].html
  product/[slug].html

Nginx can resolve those files with try_files; the sync job can compare and prune them without searching through the application build; and an operator can inspect the directory without needing to understand the script first.

A missing snapshot has two different meanings

The most important planning decision was to stop using the filesystem as proof that a route exists.

Imagine a crawler asks for /product/current-plan. The product is active in Supabase, but its snapshot is briefly absent because a sync is in progress or a deploy has just replaced the files. Returning 404 would tell the crawler that the product does not exist. That is false.

Now imagine the request is for /product/retired-plan. That product is no longer active. Falling back to the generic SPA shell with a 200 would also be false. It creates a soft 404: the server claims there is a page even though the route is not part of the current catalog.

The system needed to distinguish route truth from snapshot availability.

For every sync, I generate an Nginx include containing the valid blog and product routes. That route map answers one question: does the application currently recognize this slug? The snapshot directory answers another: is the crawler document available right now?

That produces three honest outcomes:

known route + snapshot present   -> 200 with snapshot HTML
known route + snapshot missing   -> 503 with Retry-After
unknown route                    -> 404

A temporary generation problem is not permanent absence, so it receives 503. An unknown or retired slug really is absent, so it receives 404. Normal browsers still receive the SPA template and let the application’s own routing take over.

The snapshot file is an artifact. The route map is the truth. Treating them as the same thing would make every temporary filesystem problem look like deleted content.

Nginx only needs to know when routes change

The next distinction was between a content change and a routing change.

Editing a product description changes its snapshot HTML, but /product/example is still a valid route. Publishing a new post or deactivating a product changes the set of valid routes. Only the second kind of change needs Nginx to reload its generated include.

The sync script therefore compares the newly generated route-truth file with the current one. If the route set is identical, it leaves the include alone and does not reload Nginx. Snapshot HTML can update as often as needed without turning every copy edit into a configuration event.

When the route map does change, the script writes the new include and runs the configured Nginx test command before reloading. The commands are environment-controlled—nginx -t and nginx -s reload by default—so the deployment can match the server’s actual installation.

This reduced the operational blast radius of the feature. The web server is not asked to reconsider its configuration because a meta description gained three words.

File writes had to be boring

Cron jobs make simple scripts concurrent. A run can take longer than expected, the next minute arrives, and suddenly two processes are both pruning, comparing, and replacing the same set of files.

I handled that at two levels.

First, the script owns a lock file. If a previous run still holds it, the new invocation exits cleanly. It does not wait, overlap, or assume the older process is broken. The default lock lives at /var/lock/royalsubz-meta-sync.lock; when the cron user is not root, it can be moved to a writable lock directory beneath the snapshot root.

Second, snapshots use atomic temp-file swaps. A process writes the complete new document separately, then replaces the destination. Nginx should see the old complete file or the new complete file, never half an HTML document caught during a write.

The script also compares content before replacing a file. Unchanged snapshots stay untouched. Removed or unpublished records are pruned so the directory cannot slowly become a second, stale catalog.

These are not SEO features. They are what make the SEO feature safe to schedule unattended.

I separated cron frequency from sync frequency

I wanted the job to recover naturally after a missed run without filling the crontab with complicated timing logic. The final setup invokes the script every minute, while the script itself uses META_SYNC_INTERVAL_MINUTES—30 minutes by default—to decide whether real work is due.

* * * * * . /etc/royalsubz-meta-sync.env && cd /path/to/project && node scripts/sync-crawler-snapshots.mjs >> /var/log/royalsubz-meta-sync.log 2>&1

That split gives me a simple scheduler and a testable interval gate. FORCE_META_SYNC=true bypasses the gate when I need an immediate run. DRY_RUN=true lets me inspect planned changes without applying them.

The Supabase service-role key never goes into the crontab. It lives with the other settings in /etc/royalsubz-meta-sync.env, owned by root and set to mode 600. The crontab sources the file, changes into the project, runs Node, and appends output to a dedicated log.

It is tempting to treat secret placement as deployment housekeeping and leave it until the end. Here it was part of the implementation. A working snapshot generator with a service-role key exposed in process instructions or a world-readable file would not be a finished system.

Deploying the frontend is part of the sync protocol

Rereading the deployed template solves asset drift on the next eligible sync. “Next eligible” is not good enough immediately after a production build.

If I deploy new hashed assets at 10:01 and the last sync ran at 10:00, crawlers can receive snapshots referencing the previous build until the interval opens again. The content may be correct, but the document is attached to files the deployment has just removed.

So the deploy plan includes an explicit final step:

FORCE_META_SYNC=true node scripts/sync-crawler-snapshots.mjs

The same rule applies after events that change route truth: publishing or unpublishing a blog post, changing a blog slug, activating or deactivating a product, or changing a product slug. Waiting thirty minutes is technically self-healing. For search crawlers, it is still thirty minutes of avoidable inconsistency.

This is the part I would have missed if I had treated the script as a standalone utility. The generator, frontend deployment, content lifecycle, and Nginx route map are one protocol. The order of their updates matters.

Health needed a simpler signal than “cron exists”

A configured cron entry proves only that a command is being attempted. It does not prove Supabase was reachable, the template markers were found, the files were writable, or the Nginx configuration passed its test.

At the end of every successful run, the script writes SNAPSHOT_ROOT/.last_sync. The interval gate maintains .meta-sync-state.json, the lock path shows whether a run is active, and the generated include exposes the route set currently known to Nginx.

The first operational check is intentionally plain: if .last_sync becomes stale, the job is no longer completing and needs investigation. From there, the log, state file, lock, snapshot directory, and generated route map narrow down where it stopped.

I did not need a monitoring platform to make the system observable. I needed one reliable success marker and artifacts that explain the last decision.

What the implementation changed for me

I started this work thinking about metadata. I finished it thinking about truthful failure states.

Generating a title tag and JSON-LD was the easy part. The harder questions were what to return while that generated file is missing, how to distinguish stale content from an invalid route, when Nginx truly needs a reload, how two cron invocations avoid corrupting each other, and how a new frontend build reaches every old snapshot.

The final system keeps those responsibilities separate:

  • Supabase decides which posts and products are current.
  • The deployed SPA template supplies the current application shell and asset hashes.
  • The sync script compiles route-specific snapshots and removes stale ones.
  • The generated include tells Nginx which crawler routes are real.
  • Nginx serves snapshots only to crawlers and preserves the SPA for everyone else.
  • Cron provides repeated opportunities to converge, while the lock and interval gate keep those attempts controlled.

That separation is what makes the setup understandable. Each piece answers one question, and no temporary failure is allowed to pretend it means something else.

The crawler does not need a second Royal Subz. It needs the first response from Royal Subz to be specific, current, and honest. Building that response took more than injecting metadata. It took making route truth, file state, deployment state, and server behavior agree.

← All posts