Website connector
Crawl and index public websites, SSRF-safe.
Scoping
Start URL with depth and domain limits.
SSRF protection
Internal addresses are effectively blocked.
Fair crawling
robots-friendly and rate-limited.
Permission-true from the start
The source stays authoritative.
The connector transfers content and the relevant permission information into the knowledge index. Queries are then checked against those rights.
SRCWebsite
Knowledgeworker ConnectSync · Parse · ACL
AIAI applications