Hey! Another quick question about a use case we're setting up, want to check what already

exists before we build around it.

We want to track what the market is saying about us: crawl a set of public

pages daily, land them in a knowledge base, and have an agent produce a weekly

trend summary. The knowledge base + agent + schedule trigger side looks like a

great fit already.

Three things we're unsure about:

1. Does the Web Crawler render JavaScript? A lot of the pages we care about are

SPAs where the content only appears after client-side routing, so a static

fetch gets an empty shell. Is there a headless-render mode we're missing?

2. The crawler seems to stay on the same host as the seed URL. Is that a

deliberate design decision? We'd love one connector with multiple seed

origins, or an allowlist of extra hosts, instead of one connector per site.

3. If we do the fetching ourselves, is POST /api/knowledge-files the right way

to push content into a knowledge base from an external pipeline? Or is that

more of an internal endpoint we shouldn't build on?

Happy to be told we're holding it wrong on any of these.

hey 👋

so you’re reading it correctly - the web crawler knowledge connector currently supports static HTML on a single origin (no support yet for JavaScript rendering / multi-site support).

we will add support though, this makes a lot of sense as a feature imo.

hi there 👋 if you want to upgrade to v1.4.0-beta.3 and try this out, it is now configurable to allow this