Open Sesame - Web Scraper

Get clean, structured data out of the sites that are hardest to scrape — JavaScript-heavy single-page apps, dynamically-rendered pages, and sites whose sitemaps are empty because content hides behind a search box.

Point it at a site or a topic. Get back clean, deduplicated, source-tagged text as RAG-ready chunks — or markdown, plain text, or JSON. It self-heals: it retries stubborn pages and skips junk instead of handing you garbage.

The problem it solves

A normal crawler asks a site for its sitemap and reads the raw HTML. On modern sites that gets you almost nothing: the content is rendered by JavaScript after load, or the real pages hide behind a search form, or what you do get is 80% cookie banners, menus, and boilerplate. So your dataset ends up empty, stale, or poisoned with noise.

How it works

1. Scout — maps the site and estimates cost. If the sitemap is empty (the classic single-page-app tell), it says so and switches strategy.
2. Discover — for sites that hide content behind search, it finds the real pages by topic search, then dedups them by the stable ID in each URL — one clean entry per document.
3. Harvest — fully renders each page (retrying with heavier rendering if the first pass comes back empty or as boilerplate), runs a quality gate + an LLM usefulness judge, chunks it, dedups, and tags it with its source.

The "no bad data" guarantee

Every page passes a deterministic quality gate (real content — not boilerplate, JSON, or a stub) and an optional LLM judge that catches junk which looks like content (error pages, empty shells, walls). Anything that fails is retried, then skipped with a reason logged in the manifest. You never pay for a pile of cookie banners.

Real example

A site that returned just 2 URLs from its sitemap (a JS-rendered search app) — Open Sesame's topic-search discovery turned that into 30 unique, deduplicated document permalinks, then harvested them into clean, source-tagged chunks. A dead/error URL in the same run was retried 3×, judged junk, and skipped automatically.

Outputs (you pick the format)

• jsonl — RAG-ready chunks with content, source label, and content hash (optional embeddings)
• markdown — one clean file per page
• text — plain text
• json — one structured record per page
• raw — the raw render, for your own pipeline

Every run also writes a manifest (kept vs. skipped, and why) and a cost report.

What you give it

Just a site URL, or a topic on that site. That's it.

Good for

RAG pipelines • research & datasets • competitive/market monitoring • legal, regulatory, and government content • any site that returns nothing useful to a normal scraper.

Use it on sites you're permitted to access, and respect each site's terms. It extracts the pages you point it at — verify anything you rely on against the source.