Website Security Header, CORS, robots.txt Checker

How it works

Website Header & Crawl-Policy Preflight is a pre-launch check for a site you own or are allowed to test. It tells you which curl commands to run, then reads the raw headers, robots.txt and sitemap you paste back. It is a header and crawl-policy preflight, not a security audit.

Phase 1 — Commands. Exact curl commands (GET, OPTIONS preflight with an Origin) and what to paste back.

Phase 2 — Score. A transparent 100-point rubric over HSTS, CSP, X-Content-Type-Options, framing, Referrer-Policy, Permissions-Policy and cookie flags, with the evidence line for each point.

Phase 3 — CORS and crawl policy. Flags risky combinations such as a reflected origin with credentials, and reviews robots.txt and sitemap.xml.

Phase 4 — Fixes and gaps. Header snippets for nginx, Apache and Cloudflare, plus an explicit list of what was not checked.

Sample output

Real, unedited output from running this Skill on a sample problem:

## Scope and permission

- Site: example-shop.test (owner confirmed in the request; launching Friday).
- Blocks received: Block 2 (`https://example-shop.test/`, final response, `HTTP/2 200`), Block 4 (CORS preflight `OPTIONS https://example-shop.test/api/`, `HTTP/2 204`), `robots.txt`, `sitemap.xml`. Blocks 1, 3, 5 and `curl -I` on sampled sitemap URLs were not provided.
- This is a header and crawl-policy preflight, not a security audit or penetration test. It reads only what you pasted, and a good score does not mean the site is secure.

## Score

Scored response: Block 2, `https://example-shop.test/`, `HTTP/2 200`. Cookie lines were pasted, so the score is out of 100. The cookie values are already `REDACTED`, which is correct.

| Item | Earned/Max | Evidence line |
|---|---|---|
| Strict-Transport-Security | 10/20 | `strict-transport-security: max-age=86400` (about 1 day, below 15552000, so partial credit; no `includeSubDomains` or `preload`, which are not required) |
| Content-Security-Policy | 0/25 | `header absent in block 2` (no `Content-Security-Policy` or `-Report-Only`) |
| X-Content-Type-Options | 10/10 | `x-content-type-options: nosniff` |
| Clickjacking protection | 15/15 | `x-frame-options: SAMEORIGIN` (valid on its own; no CSP `frame-ancestors` present, so nothing to conflict with it) |
| Referrer-Policy | 10/10 | `referrer-policy: strict-origin-when-cross-origin` |
| Permissions-Policy | 0/5 | `header absent in block 2` |
| Cookie flags | 5/15 | `set-cookie: session=REDACTED; Path=/; HttpOnly` (missing `Secure` and `SameSite`, -10). See the note below on `cart`. |
| **Total** | **50/100** | One response scored (Block 2). |

Why the lost points matter:
- HSTS: a one-day lifetime gives browsers little protection against downgrade to plain HTTP once the entry expires.
- CSP: with no policy, the browser has no second line of defence if any page ever renders untrusted content.
- Permissions-Policy: powerful browser features (camera, microphone, geolocation) are not explicitly switched off for a site that does not use them.
- Cookies: the `session` cookie can be sent over plain HTTP (no `Secure`) and has no explicit cross-site rule (no `SameSite`).

Cookie note: `set-cookie: cart=REDACTED; Path=/; Secure; SameSite=Lax` has no `HttpOnly`. I scored only `session`, since the rubric covers cookies holding session or login state. If `cart` also carries login or session state, it loses 5 more points (cookie row 0/15, total 45/100). Tell me if so.

## CORS findings

Source: Block 4 only (`HTTP/2 204`). I sent the untrusted test origin `https://not-my-site.test`. Headers seen: `access-control-allow-origin`, `access-control-allow-credentials`, `access-control-allow-methods`, `access-control-allow-headers`, `access-control-max-age`. `Vary` was not present.

| Severity | Evidence line | Finding and fix |
|---|---|---|
| FINDING (high) | `access-control-allow-origin: https://not-my-site.test` and `access-control-allow-credentials: true` | The server reflected an untrusted origin and allows credentials, so any site could read authenticated API responses. Fix: compare `Origin` to an exact allow-list and echo only on a match. |
| FINDING (medium) | `Vary` header absent in block 4 | A reflected origin without `Vary: Origin` lets a shared cache serve one origin's CORS answer to another. Fix: add `Vary: Origin`. |
| NOTE | `access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS` | This is a fixed list, not an echo of the requested method, but check that the API really needs `PUT` and `DELETE` cross-origin. `access-control-allow-headers: content-type, authorization` is plausible for a credentialed API. |
| OK | `HTTP/2 204`, `access-control-max-age: 600` | Preflight succeeds with a short cache lifetime. |
| NOT CHECKED | Block 5 not provided | I cannot compare the preflight with a real GET, so a preflight/GET mismatch is untested. |

Limit: one test origin shows reflection, but a prefix or suffix match bug may need more origins. I can give you two more `Origin` values to test if you want deeper coverage.

## Crawl policy findings

**robots.txt**

| Severity | Evidence line | Finding |
|---|---|---|
| FINDING | `Sitemap: /sitemap.xml` | The sitemap URL is not absolute. Use `Sitemap: https://example-shop.test/sitemap.xml`. (Placement above `User-agent` is not itself the issue, since `Sitemap` is not tied to a group; check current search engine docs.) |
| FINDING | `Disallow: /products/` with sitemap entry `https://example-shop.test/products/blue-widget` | robots.txt blocks an important section that your sitemap asks crawlers to visit. If the shop's product pages should be found, remove this line. |
| FINDING | `Noindex: /thank-you/` | `Noindex` in robots.txt is not a supported directive for major search engines (check current search engine docs). Use `<meta name="robots" content="noindex">` or an `X-Robots-Tag: noindex` header on `/thank-you/`, and do not `Disallow` that path or the crawler will never see the tag. |
| NOTE | `Disallow: /admin/` and `Disallow: /private-reports/` | robots.txt is a request to well-behaved crawlers, not access control. It publicly advertises these paths, so protect them with authentication. |
| NOTE | `Disallow: /products/` | `Disallow` stops crawling, not indexing: a blocked URL can still appear in results if other pages link to it. |

No BOM or HTML was visible in the pasted text, and `User-agent: *` does not use a site-wide `Disallow: /`. Rule precedence and wildcard behaviour follow RFC 9309.

**sitemap.xml**

| Severity | Evidence line | Finding |
|---|---|---|
| FINDING | `<loc>http://example-shop.test/</loc>` | `http://` URL in a sitemap for an HTTPS site, and the scheme differs from the other entry. Use `https://example-shop.test/`. |
| FINDING | `<lastmod>10/10/2026</lastmod>` | Not W3C date format. Use `2026-10-10`. (The first entry's `2026-10-10` is correct.) |
| FINDING | `https://example-shop.test/products/blue-widget` | This URL is disallowed by `Disallow: /products/` in robots.txt (see above). |
| OK | `<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">` | Valid root element and namespace; well-formed in the pasted portion; no duplicates; two URLs, far below the 50,000 limit (check current limits in search engine docs). |
| NOT CHECKED | n/a | URL reachability and redirects: run `curl -sS -I https://example-shop.test/` and the same for the product URL if you want that covered. |

## Fixes

Test in staging first. A wrong CSP, framing or HSTS value can lock out your own pages or subdomains.

**HSTS** (raise the lifetime in steps; add `includeSubDomains` only once every subdomain serves HTTPS; `preload` is a commitment, check current hstspreload.org guidance):

```nginx
# nginx (server block; "always" adds headers on error responses too)
add_header Strict-Transport-Security "max-age=15552000" always;
# later, once stable: max-age=31536000; includeSubDomains

CSP (start report-only, then tighten and switch to enforcing):

add_header Content-Security-Policy-Report-Only "default-src 'self'; frame-ancestors 'self'" always;

Permissions-Policy:

add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;

Apache (mod_headers) equivalents:

Header always set Strict-Transport-Security "max-age=15552000"
Header always set Content-Security-Policy-Report-Only "default-src 'self'; frame-ancestors 'self'"
Header always set Permissions-Policy "camera=(), microphone=(), geolocation=()"

Cloudflare: Rules > Transform Rules > Modify Response Header > "Set static" for each header above (menu names change; check current Cloudflare docs). Cloudflare also has an HSTS toggle under its SSL/TLS settings; do not set the header in two places with different values.

Cookies: set the session cookie with the attribute string Secure; HttpOnly; SameSite=Lax, for example session=<value>; Path=/; Secure; HttpOnly; SameSite=Lax. (If cart holds session state, add HttpOnly there too.)

CORS: compare Origin to an exact list, echo only on a match, add Vary: Origin, and never combine * with credentials. nginx pattern:

# http block
map $http_origin $cors_origin {
    default "";
    "https://example-shop.test" $http_origin;
    "https://admin.example-shop.test" $http_origin;
}
# api location
add_header Access-Control-Allow-Origin $cors_origin always;
add_header Access-Control-Allow-Credentials "true" always;
add_header Vary "Origin" always;

An unlisted origin gets an empty Access-Control-Allow-Origin and so no credentialed access. Re-run command 4 with https://not-my-site.test afterwards and expect no reflection.

robots.txt and sitemap: corrected version.

Sitemap: https://example-shop.test/sitemap.xml
User-agent: *
Disallow: /admin/

Protect /admin/ and /private-reports/ with authentication rather than relying on robots.txt. Add the noindex meta tag or X-Robots-Tag on /thank-you/. In the sitemap, change the first <loc> to https:// and the second <lastmod> to 2026-10-10.

Not checked

• Block 1 (home page without following redirects) and Block 3 (plain-HTTP entry point): redirect to HTTPS and any headers on the redirect response itself.
• Block 5 (normal API GET with Origin): real API response CORS headers, and comparison with the preflight.
• Headers on pages other than the one pasted (product, cart, checkout, API, error pages).
• TLS protocol and certificate quality.
• Redirect chains beyond what was pasted.
• Application vulnerabilities, authentication and session logic, dependency issues.
• CSP effectiveness against real injection (no CSP present to evaluate).
• Subresource integrity.
• Sitemap URL reachability (no curl -I on sampled URLs provided).
• Whether cart carries session or login state (affects the cookie score).

This checks what the server told us, not whether the site is safe.

## Use cases

- Pre-launch check of a new site's headers
- Reviewing an API's CORS behaviour
- Fixing robots.txt and sitemap.xml mistakes
- Getting copy-paste header snippets for nginx, Apache or Cloudflare

## FAQ

**Is this a security audit?** No. It only reads headers you paste and cannot find application vulnerabilities.

**Does it scan my site?** No. You run the curl commands yourself, on sites you own or may test.

**Which servers?** Fix snippets for nginx, Apache and Cloudflare.