Crawler & SearchEngine Microservices

Deploy and configure the Prisme.ai web crawling and search engine capabilities for knowledge base creation and content discovery

The Crawler and SearchEngine microservices work together to provide powerful web content indexing and search capabilities for your Prisme.ai platform. These services enable the creation of knowledge bases from web content, support external data integration, and power the search functionality across your applications.

Overview

The Crawler and SearchEngine microservices function as a pair to deliver comprehensive web content processing:

prismeai-crawler

Discovers, fetches, and processes web content from specified sources, managing crawl schedules and content extraction

prismeai-searchengine

Indexes processed content and provides search capabilities with relevance ranking and content highlighting

These services must be deployed together - you cannot use one without the other as they form a complete indexing and search solution.

Installation Prerequisites

Before deploying these microservices, ensure you have access to the following dependencies:

ElasticSearch

Required for document storage and search functionality

• Can use the same ElasticSearch instance as the core deployment
• Stores indexed content and search metadata
• Provides the search functionality backend

Redis

Required for inter-service communication

• May share an instance only if it uses noeviction and has enough capacity
• Manages crawl queues and job scheduling
• Facilitates communication between services
• Must run with maxmemory-policy=noeviction

Redis stores engine definitions and crawl queues; it is not a cache. prismeai-searchengine checks the eviction policy at startup in every mode. prismeai-crawler also checks it in durable-webhooks and full modes. If Redis denies CONFIG GET, the services start with a warning and you must enforce the policy in your deployment. See Redis configuration.

Configuration

Environment Variables

Configure the Crawler and SearchEngine microservices with the following environment variables:

Common Environment Variables

Variable Name	Description	Default Value	Affected Services
REDIS_URL	Redis connection URL for communication between services	redis://localhost:6379	Both
REDIS_PASSWORD	Redis connection password		Both
REDIS_TLS_CA_FILE	Redis CA certificate filepath		Both
REDIS_MAX_CLIENTS	Maximum number of Redis client connections in the pool	20	Both
DOCUMENTS_STORAGE_DRIVER	Documents storage driver : elasticsearch or opensearch	elasticsearch	Both
ELASTIC_URL	Elasticsearch connection URL for document storage	http://localhost:9200	Both
ELASTIC_USER	Elasticsearch user	“	Both
ELASTIC_PASSWORD	Elasticsearch password	“	Both
ELASTIC_INDICES_PREFIX	Elasticsearch indices name prefix	“	Both
ELASTIC_VERIFY_CERTS	Whether to validate Elasticsearch https certificate	true	Both

Crawler-Specific Environment Variables

Variable Name	Description	Default Value	Affected Services
MAX_CONTENT_LEN	Maximum length (in characters) of documents crawled	1500000	prismeai-crawler
CONCURRENT_REQUESTS	The maximum number of concurrent (i.e. simultaneous) requests that will be performed by the Scrapy downloader	16	prismeai-crawler
CONCURRENT_REQUESTS_PER_DOMAIN	The maximum number of concurrent (i.e. simultaneous) requests that will be performed to any single domain.	16	prismeai-crawler
DOWNLOAD_DELAY	Minimum seconds to wait between 2 consecutive requests to the same domain.	0	prismeai-crawler
REQUEST_QUEUES_POLLING_INTERVAL	Interval in seconds between each poll of the Redis queue for new requests to process. Lower values increase responsiveness but also Redis load.	2	prismeai-crawler
REQUEST_QUEUES_POLLING_SIZE	Number of requests to start from the queue in a single poll	1	prismeai-crawler
USER_AGENT	Crawler HTTP user agent	Prisme.ai (https://prisme.ai)	prismeai-crawler
ROBOTSTXT_OBEY	Whether the crawler should respect the site’s robots.txt. - Recommended: True for all public websites. - For internal portals (e.g. SharePoint, intranet), you may set False if robots.txt prevents access to content you are authorized to crawl. ⚠️ Disabling this option should only be done in controlled/internal environments.	True	prismeai-crawler
DEFAULT_ENGINE_WEBPAGES_LIMIT	Maximum pages that can be indexed per search engine before crawling stops	30000	prismeai-crawler
MAX_SEARCHENGINES_NB	Maximum number of search engines to run concurrently. -1 means unlimited. Must stay -1 when CRAWLER_RELIABILITY_MODE=full.	-1	prismeai-crawler
TARGET_SEARCHENGINES	Restrict a pod to a comma-separated list of search engine ids (e.g. myworkspace/mycrawler). Must be unset when CRAWLER_RELIABILITY_MODE=full.		prismeai-crawler
SEARCHENGINES_SYNC_METHOD	How engine configuration updates are propagated: streams (durably ordered) or pubsub. Must stay streams when CRAWLER_RELIABILITY_MODE=full.	streams	Both

ACK & Reliability Environment Variables

These settings control how the crawler recovers work after a crash.

Variable Name	Description	Default Value	Affected Services
ACK_ENABLED	Enable/disable the request acknowledgment mechanism. When enabled, requests are tracked atomically in Redis and automatically reclaimed if the crawler crashes during processing. Disabling this reverts to the original behavior where crashed requests are lost. Must stay True when CRAWLER_RELIABILITY_MODE=full.	True	prismeai-crawler
ACK_DUPEFILTER_DROPS	When True, requests dropped by the deduplication filter are also ACKed (cleaned up from pending structures).	True	prismeai-crawler
SKIP_NON_RETRIABLE_ERRORS	When True, HTTP 400/401/403/404/410 errors and certain exceptions (UnsupportedFormat, CorruptedFile, AuthenticationFailed) are moved to the DLQ immediately rather than retried.	True	prismeai-crawler
PENDING_VISIBILITY_TIMEOUT	Time in seconds before a pending request is considered stale and eligible for reclaim. Set this to 2-3x your longest document processing time (e.g., large PDFs). If a request takes longer than this timeout, it will be retried.	900 (15 minutes)	prismeai-crawler
PENDING_RECLAIM_INTERVAL	Interval in seconds between checks for stale pending requests. The reclaim loop runs at this frequency to find and re-queue abandoned tasks.	60	prismeai-crawler
PENDING_RECLAIM_BATCH_SIZE	Maximum number of stale tasks to reclaim per reclaim loop iteration.	10	prismeai-crawler
MAX_RETRY_ATTEMPTS	Maximum number of times a task can be reclaimed/retried before it is moved to the DLQ permanently. Set to 0 to disable retry limits.	3	prismeai-crawler
DLQ_MAX_SIZE	Monitoring alert threshold: when the Dead Letter Queue grows beyond this size, warnings are emitted. Does not hard-cap the DLQ. Once limit is reached a log is shown periodically but it doesn’t affect crawling.	1000	prismeai-crawler
DLQ_RETENTION_DAYS	Number of days DLQ entries are retained before cleanup.	7	prismeai-crawler

Resource Throttling Environment Variables

These settings control adaptive resource monitoring to prevent OOM (Out of Memory) crashes through CPU and memory monitoring. The crawler will pause polling when resources are constrained and resume when they recover.

Thresholds are percentages of container limits, not system resources. Limits are auto-detected from cgroup files (cgroup v1/v2). For restricted Kubernetes clusters where cgroup detection fails, you can set CONTAINER_CPU_LIMIT_MILLICORES and CONTAINER_MEMORY_LIMIT_BYTES as fallbacks.

Variable Name	Description	Default Value	Affected Services
RESOURCE_THROTTLING_ENABLED	Enable/disable resource-based throttling. When enabled, the crawler monitors memory and CPU usage and pauses polling when thresholds are exceeded to prevent OOM crashes.	True	prismeai-crawler
RESOURCE_THROTTLING_MODE	What resources to monitor. Options: memory_only (recommended - Kubernetes handles CPU throttling effectively), cpu_only, both, none. Memory monitoring is critical as memory exhaustion leads to OOMKilled.	memory_only	prismeai-crawler
MEMORY_WARNING_THRESHOLD	Memory usage percentage (of container limit) above which the crawler pauses polling for new requests. Existing requests continue processing.	80	prismeai-crawler
MEMORY_CRITICAL_THRESHOLD	Memory usage percentage (of container limit) above which the crawler stops all processing. This is a last-resort protection against OOM.	95	prismeai-crawler
CPU_WARNING_THRESHOLD	CPU usage percentage (of container limit) above which the crawler pauses polling. Only applies when RESOURCE_THROTTLING_MODE includes CPU monitoring.	85	prismeai-crawler
CPU_CRITICAL_THRESHOLD	CPU usage percentage (of container limit) above which the crawler stops all processing. Only applies when RESOURCE_THROTTLING_MODE includes CPU monitoring.	95	prismeai-crawler
RESOURCE_CHECK_INTERVAL	Interval in seconds between resource usage checks. Lower values provide faster response to resource pressure but increase overhead.	1.0	prismeai-crawler
RESOURCE_CIRCUIT_BREAKER_THRESHOLD	Number of consecutive resource check failures before escalating to a CRITICAL alert. This helps identify persistent resource issues vs. temporary spikes.	10	prismeai-crawler
RESOURCE_CIRCUIT_BREAKER_SHUTDOWN_TIMEOUT	Seconds the crawler can remain stuck in critical state before initiating automatic pod shutdown. The pod exits gracefully, allowing Kubernetes to restart it with fresh state. Set to 0 to disable auto-shutdown.	900 (15 minutes)	prismeai-crawler
RESOURCE_THROTTLE_FAST_RECLAIM	When True and a critical resource threshold is hit, tasks are immediately released back to the Redis queue for other pods, rather than waiting for the visibility timeout to expire.	True	prismeai-crawler
CONTAINER_CPU_LIMIT_MILLICORES	Fallback CPU limit in millicores (e.g. 2000 = 2 cores) for restricted Kubernetes clusters where cgroup files are inaccessible. Used only if cgroup v1/v2 detection fails.		prismeai-crawler
CONTAINER_MEMORY_LIMIT_BYTES	Fallback container memory limit in bytes for restricted Kubernetes clusters where cgroup files are inaccessible. If all detection methods fail, falls back to system total memory.		prismeai-crawler
RESOURCE_CIRCUIT_BREAKER_FORCE_EXIT_TIMEOUT	Hard timeout in seconds for graceful reactor shutdown during circuit breaker auto-shutdown. If the reactor doesn’t stop within this time, os._exit(1) is called to force termination.	60	prismeai-crawler

Document Processing Environment Variables

These settings control how documents (PDFs, DOCX, etc.) are parsed and processed by the crawler.

Variable Name	Description	Default Value	Affected Services
DOCUMENTS_DEFAULT_PARSER	Default parser for document processing. Options: unstructured (recommended, uses the Unstructured library), tika (Apache Tika - requires TIKA_PATH to be configured). Can be overridden per-searchengine via parser configuration.	unstructured	prismeai-crawler
UNSTRUCTURED_DEFAULT_STRATEGY	Default parsing strategy for the Unstructured library. Options: fast (no OCR, fastest), auto (automatic detection), hi_res (full OCR processing, slowest but most accurate for scanned documents), ocr_only (OCR only). Can be overridden per-searchengine.	fast	prismeai-crawler
TIKA_OCR_SKIP	Skip OCR (Tesseract) processing when using Apache Tika parser. Set to True for faster parsing at the cost of not extracting text from images within documents. Similar to Unstructured’s fast strategy. Can be overridden per-searchengine using parser config: {"type": "tika", "skip_ocr": true}.	True	prismeai-crawler
TIKA_PATH	Path to the directory containing tika-server.jar. If not set, Tika is disabled and the default parser falls back to Unstructured.	/app/tika/ (in Docker image)	prismeai-crawler
TIKA_PARSING_TIMEOUT	Timeout in seconds for a single Tika parsing request.	600	prismeai-crawler
TIKA_MAX_RETRIES	Maximum retries for Tika parse requests before giving up.	3	prismeai-crawler

Resource Considerations

When planning your deployment, consider these resource recommendations:

Memory Requirements

• Crawler: Min 1GB, recommended 2GB+
• SearchEngine: Min 1GB, recommended 2GB+
• Scale based on crawl volume and index size

CPU Allocation

• Crawler: Min 0.5 vCPU, recommended 1+ vCPU
• SearchEngine: Min 0.5 vCPU, recommended 1+ vCPU
• Consider additional resources for high request volumes

Storage Needs

• ElasticSearch: Plan for index growth based on content volume
• Redis: Plan capacity for engine state, crawl queues, and the webhook outbox
• Consider storage class with good I/O performance

Network Configuration

• Internet access for the Crawler service
• Internal network access between services
• Consider bandwidth requirements for crawl activities

Reliability mode

Set CRAWLER_RELIABILITY_MODE on both services to choose the reliability level. The default is legacy, so upgrading does not enable the new delivery path automatically.

Mode	What it enables
legacy (default)	Best-effort webhooks; crawler processes remain active while waiting for work.
durable-webhooks	Adds a Redis-backed webhook outbox with retries and dead-lettering. Delivery is at least once.
full	Adds atomic crawl admission and lets idle crawl workers stop and restart when work arrives.

Use the same mode on prismeai-crawler and prismeai-searchengine. During a rolling upgrade, update the crawler first and the searchengine second. Downgrade them in the reverse order.

Prerequisites

• Use a dedicated or correctly sized Redis instance with maxmemory-policy=noeviction in every mode. Do not use a cache-tuned LRU instance.
• For durable-webhooks and full, enable Redis AOF persistence with appendfsync everysec or stronger so a Redis restart does not undermine the delivery guarantees.
• Webhook consumers must be idempotent in durable-webhooks and full. Deduplicate events by the top-level id or dedup_key.
• In full, any crawler pod can process any engine. Keep these required settings at their defaults:

Setting	Required value at full
ACK_ENABLED	True
MAX_SEARCHENGINES_NB	-1
TARGET_SEARCHENGINES	unset
SEARCHENGINES_SYNC_METHOD	streams on both services

Enable full mode

Set the variable on both services in your values.yaml, then deploy them together:

prismeai-crawler:
  env:
    - name: CRAWLER_RELIABILITY_MODE
      value: "full"

prismeai-searchengine:
  env:
    - name: CRAWLER_RELIABILITY_MODE
      value: "full"

Unknown values fall back to legacy and produce a warning.

Monitor durable webhooks

In durable-webhooks and full, monitor the outbox.stream.depth and outbox.dispatcher.pel.length gauges. Alert well before the outbox reaches WEBHOOK_OUTBOX_MAXLEN (100,000 entries by default).

At WEBHOOK_OUTBOX_MAXLEN, Redis may trim the oldest entries before delivery. Treat the limit as an emergency memory guard, not a normal operating level.

API behaviour at full

In full, /index admits each crawl batch atomically. Clients must handle the following responses:

Status	Meaning	Client action
200	All batches were accepted	—
400	Invalid batch	Fix the request
404	Unknown engine	—
503 + Retry-After: 30	The engine queue remained above its limit	Retry the same request after the delay
500	Internal admission failure	Do not retry automatically; investigate the service

Below full, /index keeps its historical behaviour: a direct queue write with no backpressure refusal.

Large /index requests are split into batches. A 503 can therefore arrive after earlier batches were accepted; retrying the full request may create duplicates. Engine creation, update, and reset retry seed admission in the background instead of returning 503.

Downgrading

• Downgrade prismeai-searchengine first, then prismeai-crawler.
• Before returning to legacy, stop webhook producers or wait for the outbox to drain. legacy does not consume entries left in the outbox.

Deployment Process

Follow these steps to deploy the Crawler and SearchEngine microservices:

1. Configure Dependencies

   Ensure ElasticSearch and Redis are accessible:

   1. Verify ElasticSearch connection with:

      curl -X GET "[ELASTIC_SEARCH_URL]:9200"
      

      You should receive a response with version and cluster information

   2. Verify Redis connection with:

      redis-cli -u [REDIS_URL] ping
      

      The response should be “PONG”

2. Deploy Microservices

   Deploy both services using Helm.

   Ensure both services are included in your values.yaml configuration:

   prismeai-crawler:
     enabled: true
     config:
       redis:
         url: "redis://redis-service:6379"
       elasticsearch:
         url: "elasticsearch-service:9200"
       
   prismeai-searchengine:
     enabled: true
     config:
       redis:
         url: "redis://redis-service:6379"
       elasticsearch:
         url: "elasticsearch-service:9200"
   
3. Verify Deployment

   Check that both services are running correctly:

   kubectl get pods -n apps | grep 'crawler\|searchengine'
   

   Both services should show Running status and be ready (e.g., 1/1).

4. Configure Network Access

   Ensure the services can access:

   1. ElasticSearch and Redis internally
   2. Internet access for the Crawler service
   3. Access from other Prisme.ai services that will use search functionality

Microservice Testing

After deploying the Crawler and SearchEngine microservices, verify their operation with these steps:

1. Create a Test SearchEngine

   Create a searchengine instance to crawl a test website:

   curl --location 'http://localhost:8000/monitor/searchengine/test/test' \
   --header 'Content-Type: application/json' \
   --data '{
       "websites": [
           "https://docs.eda.prisme.ai/en/workspaces/"
       ]
   }'
   

   If successful, you should receive a complete searchengine object that includes an id field.

2. Check Crawl Progress

   After a few seconds, check the crawl history and statistics:

   curl --location --request GET 'http://localhost:8000/monitor/searchengine/test/test/stats' \
   --header 'Content-Type: application/json' \
   --data '{
       "urls": ["https://docs.eda.prisme.ai/en/workspaces/"]
   }'
   

   Verify that:

   • The metrics.indexed_pages field is greater than 0
   • The metrics.pending_requests field indicates active crawling
   • The crawl_history section shows pages that have been processed
3. Test Search Functionality

   Perform a test search query to verify indexing and search:

   curl --location 'http://localhost:8000/search/test/test' \
   --header 'Content-Type: application/json' \
   --data '{
       "query": "workspace"
   }'
   

   The response should include a results array containing pages from the crawled website that match your search term. Each result should include relevance information and content snippets.

If all tests pass, congratulations! Your Crawler and SearchEngine microservices are up and running correctly.

Features and Capabilities

Web Crawling

The Crawler service provides advanced web content discovery and extraction:

• Configurable crawl depth: Control how many links deep the crawler will explore
• URL filtering: Include or exclude specific URL patterns
• Rate limiting: Respect website terms of service with configurable crawl rates
• Content extraction: Parse and clean HTML to extract meaningful content
• Metadata extraction: Capture titles, descriptions, and other metadata
• Scheduled crawls: Set up periodic recrawling to keep content fresh
• Robots.txt compliance: Respect website crawling policies

Search Capabilities

The SearchEngine service delivers powerful search functionality:

• Full-text search: Find content across all indexed documents
• Relevance ranking: Surface the most relevant content first
• Content highlighting: Highlight matching terms in search results
• Faceted search: Filter results by metadata fields
• Synonym handling: Find content using related terms
• Language support: Index and search content in multiple languages
• Query suggestions: Support for “did you mean” functionality
• Result snippets: Show context around matching terms

Integration with Prisme.ai

The Crawler and SearchEngine microservices integrate with other Prisme.ai components:

Knowledges

• Create knowledge bases from crawled web content
• Enrich existing knowledge bases with web information
• Use search capabilities for better information retrieval

Builder

• Build custom search interfaces using search API
• Integrate search results into workflows
• Trigger crawls programmatically in automations

Agent Creator

• Power research agents with web crawling capabilities
• Create domain-specific search tools
• Develop content discovery applications

Custom Code

• Extend crawling behavior with custom functions
• Process search results with specialized logic
• Create advanced search and discovery experiences

Advanced Configuration

Crawl Configuration Options

When creating a searchengine, you can specify advanced crawl options:

{
  "websites": ["https://example.com"],
  "options": {
    "maxDepth": 3,
    "includePatterns": ["*/blog/*", "*/products/*"],
    "excludePatterns": ["*/admin/*", "*/login/*"],
    "respectRobotsTxt": true,
    "crawlDelay": 1000,
    "userAgent": "Prisme.ai Crawler",
    "maxPagesPerSite": 1000
  }
}

These options allow you to fine-tune crawling behavior for different use cases.

ElasticSearch Index Management

The services automatically create and manage ElasticSearch indices. For advanced use cases, you can:

• Configure index settings like sharding and replication
• Set up index lifecycle policies for managing index growth
• Implement custom analyzers for specialized search needs
• Configure cross-cluster search for large-scale deployments

Consult the ElasticSearch documentation for more information on these advanced configurations.

Performance Tuning

To optimize performance for your specific needs:

• Adjust MAX_CONTENT_LEN to balance comprehensiveness with resource usage
• Configure crawler concurrency settings for faster crawling
• Implement ElasticSearch performance optimizations
• Consider Redis caching strategies for frequent searches
• Use horizontal scaling for high-volume crawling and search scenarios

Troubleshooting

Crawling Issues

Symptom: Web pages are not being crawled or indexed

Possible causes:

• Network connectivity issues
• Website robots.txt restrictions
• Rate limiting by target websites
• URL pattern configuration excluding relevant pages

Resolution steps:

1. Check crawler logs for specific error messages
2. Verify network connectivity to target websites
3. Review website robots.txt for restrictions
4. Adjust crawl rate settings to avoid being blocked
5. Check URL pattern configurations

Search Problems

Symptom: Search results are missing or irrelevant

Possible causes:

• Content not properly indexed
• ElasticSearch configuration issues
• Query formatting problems
• Content exceeding maximum length limits

Resolution steps:

1. Verify content was successfully crawled and indexed
2. Check ElasticSearch connectivity and health
3. Review search query format and parameters
4. Check if content exceeds MAX_CONTENT_LEN setting
5. Test simple queries to validate basic functionality

Performance Issues

Symptom: Slow crawling or search response times

Possible causes:

• Insufficient resources allocated
• ElasticSearch performance problems
• Redis bottlenecks
• Large crawl queues or index sizes

Resolution steps:

1. Monitor resource usage during operations
2. Check ElasticSearch performance metrics
3. Verify Redis isn’t running out of memory
4. Consider scaling resources horizontally or vertically
5. Implement more targeted crawling strategies

Tika troubleshooting

Since Tika crawling takes place in a separate process than the main prisme.ai-crawler process, troubleshooting Tika issues may require auditing Tika logs directly in prisme.ai-crawler container :

• /app/tika/tika.log
• /app/tika/tika-server.log

Security Considerations

When deploying and using the Crawler and SearchEngine microservices, keep these security considerations in mind:

Network Security

• Implement appropriate network policies
• Consider using a dedicated proxy for outbound crawling
• Monitor for unusual traffic patterns

Content Security

• Be mindful of crawling and indexing sensitive content
• Implement URL patterns to exclude sensitive areas
• Consider content filtering before indexing

Authentication

• Secure ElasticSearch and Redis with strong authentication
• Implement API access controls for search endpoints
• Use TLS for all service communications

Compliance

• Respect website terms of service when crawling
• Consider data retention policies for crawled content
• Be aware of copyright implications of content indexing

For any issues or questions during the deployment process, contact [e-mail] for assistance.