Websites
In this article, you'll learn how to add website content to your AI Agent's knowledge base by crawling pages, selected URLs, or a sitemap.
Before you start
You need Owner, Admin, or AI Agent Admin role in Enjo.
At least one AI Agent must be configured in your workspace. See Create AI Agent.
The website you want to crawl must be publicly accessible. Private pages are not crawled unless appropriate authentication is configured.
How website indexing works
Enjo crawls and indexes the selected website content as a one-time operation. The AI Agent uses this indexed content to answer questions and includes citations linking back to the source pages.
💡 Website knowledge sources are indexed once and do not update automatically. To keep the content current, remove and re-add the knowledge source to trigger a fresh crawl.
Step 1 — Open Knowledge
Go to AI Agents and select your agent.
Click the Knowledge tab.
Click Add Knowledge.
Select Website from the options.
Step 2 — Choose or create a knowledge source
Use an existing knowledge source Select it from the list and click Add.
Create a new knowledge source Click Create a new knowledge source and continue to Step 3.
Step 3 — Select an extraction method
Choose how you want Enjo to extract content from the website:
Method | What it does |
|---|---|
Extract data from all website pages | Crawls your entire website including internal cross-links. Limit: 10,000 pages. |
Extract data from selected pages | Extracts content from specific pages you choose. Click Next to select the pages to index. |
Extract data from sitemap URL | Extracts content exclusively from the URLs listed in a provided sitemap. One sitemap URL at a time. |
💡 Use Extract data from sitemap URL if your website has a sitemap — it gives you more precise control over which pages are indexed compared to a full crawl.
Step 4 — Name your knowledge source and add metadata
Enter a name for the knowledge source.
Optionally add metadata as key-value pairs to describe the content — for example,
topic: IT supportorlanguage: English. Metadata improves search accuracy.Click Start Indexing.
💡 If you are unsure what metadata to add, leave it blank. You can add or update metadata later.
After indexing
Once indexing is complete, the knowledge source appears in the Knowledge tab with a status of Done. The AI Agent can now use the indexed content to answer questions.
Frequently asked questions
What is the page limit for a full website crawl? The crawl limit is 10,000 pages when using the Extract data from all website pages option.
Can I extract data from multiple sitemaps? Extraction is currently limited to one sitemap URL at a time. To index multiple sitemaps, add each one as a separate knowledge source.
Will the AI Agent update automatically when the website changes? No — website knowledge sources are indexed once. To update the content, remove the existing knowledge source and re-add it to trigger a new crawl.
Are private pages crawled? Only publicly accessible pages are crawled by default. Private pages require appropriate authentication to be configured before they can be indexed.
What's next
AI Agent Knowledge — Manage all knowledge sources connected to your agent.
Supported File Formats — Upload documents directly if you prefer file-based knowledge sources.
Bulk Testing — Validate AI responses across your knowledge sources at scale.
Guidelines for Knowledge Base Articles — Best practices for structuring knowledge content for AI retrieval.