` HTML tags. xpath\_filter is always included inside this parent xpath : `/html/body/descendant::text()[not(ancestor::style) and not(ancestor::script) and not(ancestor::header) and not(ancestor::footer) and {xpath_filter}]` Useful xpath examples : ```yaml theme={null} ancestor::div[contains(@id, 'cal_page-article-fiche')]/header/div not(ancestor::*[contains(@id, 'ref-bottom') or contains(@id, 'ref-header')]) not(ancestor::*[contains(@class, 'navbar')]) and ancestor::p ``` Configure HTTP headers per domain : ```yaml theme={null} slug: Crawler config: websiteURL: - https://www.mywebsite.fr/ websites_config: mywebsite.fr: headers: someCustom: header ``` Provide a `websiteURL` ending with **sitemap.xml** to only crawl the URLs listed in that sitemap : ```yaml theme={null} slug: Crawler config: websiteURL: - https://www.example.com/sitemap.xml crawlerId: example.com ``` Set up recurring crawl schedules: ```yaml theme={null} slug: Crawler config: periodicity: 259200 ``` This configuration schedules a crawl to be run every 3 days. You can choose different content extraction strategies, separately for HTML or Documents. ```yaml theme={null} slug: Crawler config: websites_config: docs.prisme.ai: parsers: documents: tika html: xpath ``` This configuration will use docling to parse the documents and html from the domain `docs.prisme.ai`. Possible values are: * Documents: `unstructured` (default) or `tika` * html: `xpath` (default) You can send additional arguments specific to Tika/Unstructured by creating a dedicated Alias for them. Here is an example showing the available options: ```yaml theme={null} slug: Crawler config: parsers: tikaOCR: type: tika skip_ocr: false # Default value is true (no OCR) tikaOCRFull: type: tika skip_ocr: false pdf_ocr_strategy: 'ocr_and_text_extraction' # Default value is auto (no OCR if any raw text is detected) unstructuredOCR: type: unstructured strategy: auto # Default strategy is fast (= no OCR) websites_config: docs.prisme.ai: parsers: documents: tikaOCR html: xpath ``` ## Common Use Cases The Crawler app enables a wide range of use cases: Build comprehensive AI knowledge bases from website content: * Documentation portals * Product information sites * Support knowledge bases * Internal wikis Keep track of changes and updates on important websites: * Competitor websites * Industry news sources * Regulatory publications * Product documentation Gather structured data from web sources: * Product catalogs * Price information * Company directories * Research publications Transform websites into conversational AI agents: * Company websites * Documentation portals * Educational resources * Knowledge repositories ## Integration with Other Prisme.ai Products The Crawler app works seamlessly with other Prisme.ai products: The Crawler is a primary data source for Knowledges: * Extract web content for knowledge bases * Keep information current through scheduled crawls * Process and structure content for optimal retrieval * Preserve source attribution for transparency This integration enables the creation of AI agents that can answer questions based on website content. Use the Crawler in your automation workflows: * Trigger automations based on website changes * Process and transform web content * Extract specific data for decision-making * Integrate web content with other data sources This enables sophisticated automations that leverage web data. Combine the Crawler with Custom Code for advanced processing: * Apply custom transformations to extracted content * Implement specialized parsing logic * Analyze content with custom algorithms * Generate derived insights from crawled data This combination provides maximum flexibility for handling web content. Store crawled content in Collection for persistence and querying: * Save structured content for later use * Build queryable repositories of web information * Track changes over time * Combine with other data sources This integration provides persistent storage and retrieval capabilities for crawled content. ## Example: Creating a Website RAG Agent One of the most popular use cases for the Crawler is creating a Retrieval-Augmented Generation (RAG) agent based on website content: Set up the Crawler to extract content from the target website: ```yaml theme={null} slug: Crawler config: websiteURL: - https://www.example.com/ websites_config: mywebsite.fr: xpath_filter: '(ancestor:article or ancestor::*[contains(@class, 'main-content')])' ``` Set up a new project in Knowledges to house the crawled content: * Create a new project * Configure embedding and chunk settings * Set up appropriate processing options Configure the connection between the Crawler and Knowledges: * Select the crawler as a data source * Map crawled content to knowledge base structure * Configure metadata and attribute mapping Execute the first crawl to populate your knowledge base: * Monitor the crawl progress * Verify content extraction quality * Address any issues or adjustments needed Set up an AI agent that uses the crawled content: * Configure prompt templates * Set retrieval parameters * Define response formatting * Test with sample questions Make the agent available and keep it current: * Publish the agent to Agent Creator * Set up scheduled crawls for updates * Monitor usage and performance * Refine based on feedback The result is an AI agent that can answer questions based on the content of the website, providing accurate and up-to-date information. ## Best Practices Follow these recommendations to get the most from the Crawler app: Be a good web citizen: * Respect robots.txt directives * Implement appropriate rate limiting * Identify your crawler with a meaningful user agent * Crawl during off-peak hours when possible * Only extract content you have permission to use These practices help maintain good relationships with the websites you crawl. Be selective about what you crawl: * Focus on the most valuable content * Use specific CSS selectors for precision * Exclude boilerplate and repetitive content * Be mindful of content that changes frequently * Consider the information architecture of the site Targeted crawling improves efficiency and content quality. Optimize ongoing crawling: * Use incremental updates when possible * Schedule crawls based on content update frequency * Focus on changed content rather than recrawling everything * Implement change detection mechanisms * Archive historical versions when appropriate These approaches minimize resource usage while keeping content current. Prepare for crawling challenges: * Monitor for crawl failures * Implement retries for transient errors. * Have fallback mechanisms for critical content * Set up notifications for persistent issues * Regularly review crawl logs Robust error handling ensures reliable content extraction. ## Limitations and Considerations When using the Crawler app, be aware of these considerations: * **JavaScript-Heavy Sites**: Some websites rely heavily on JavaScript for content rendering. The Crawler includes JavaScript processing capabilities, but complex single-page applications might present challenges. * **Authentication Requirements**: Websites requiring login can be crawled, but require additional configuration for authentication handling. * **Legal and Terms of Service**: Always ensure you have the right to crawl and use the content from websites, respecting their terms of service. * **Dynamic Content**: Content that changes based on user interaction or personalization may not be fully captured. * **Resource Intensity**: Comprehensive crawling can be resource-intensive. Consider scope and frequency based on your needs and available resources. ## Next Steps Learn about connecting to external APIs Execute custom logic within your workflows Manage data with simplified database access Follow a detailed tutorial on creating a website RAG agent