> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prisme.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Crawler & SearchEngine Microservices
> Deploy and configure the Prisme.ai web crawling and search engine capabilities for knowledge base creation and content discovery
The Crawler and SearchEngine microservices work together to provide powerful web content indexing and search capabilities for your Prisme.ai platform. These services enable the creation of knowledge bases from web content, support external data integration, and power the search functionality across your applications.
## Overview
The Crawler and SearchEngine microservices function as a pair to deliver comprehensive web content processing:
Discovers, fetches, and processes web content from specified sources, managing crawl schedules and content extraction
Indexes processed content and provides search capabilities with relevance ranking and content highlighting
These services must be deployed together - you cannot use one without the other as they form a complete indexing and search solution.
## Installation Prerequisites
Before deploying these microservices, ensure you have access to the following dependencies:
Required for document storage and search functionality
* Can use the same ElasticSearch instance as the core deployment
* Stores indexed content and search metadata
* Provides the search functionality backend
Required for inter-service communication
* Can use the same Redis instance as the core deployment
* Manages crawl queues and job scheduling
* Facilitates communication between services
* Stores temporary processing data
## Configuration
### Environment Variables
Configure the Crawler and SearchEngine microservices with the following environment variables:
| Variable Name | Description | Default Value | Affected Services |
| -------------------------- | ------------------------------------------------------- | ------------------------ | ----------------- |
| `REDIS_URL` | Redis connection URL for communication between services | `redis://localhost:6379` | Both |
| `REDIS_PASSWORD` | Redis connection password | | Both |
| `REDIS_TLS_CA_FILE` | Redis CA certificate filepath | | Both |
| `REDIS_MAX_CLIENTS` | Maximum number of Redis client connections in the pool | `20` | Both |
| `DOCUMENTS_STORAGE_DRIVER` | Documents storage driver : elasticsearch or opensearch | `elasticsearch` | Both |
| `ELASTIC_URL` | Elasticsearch connection URL for document storage | `http://localhost:9200` | Both |
| `ELASTIC_USER` | Elasticsearch user | \`\` | Both |
| `ELASTIC_PASSWORD` | Elasticsearch password | \`\` | Both |
| `ELASTIC_INDICES_PREFIX` | Elasticsearch indices name prefix | \`\` | Both |
| `ELASTIC_VERIFY_CERTS` | Whether to validate Elasticsearch https certificate | `true` | Both |
| Variable Name | Description | Default Value | Affected Services |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | ----------------- |
| `MAX_CONTENT_LEN` | Maximum length (in characters) of documents crawled | `150000` | prismeai-crawler |
| `CONCURRENT_REQUESTS` | [The maximum number of concurrent (i.e. simultaneous) requests that will be performed by the Scrapy downloader](https://docs.scrapy.org/en/latest/topics/settings.html#concurrent-requests) | `16` | prismeai-crawler |
| `CONCURRENT_REQUESTS_PER_DOMAIN` | The maximum number of concurrent (i.e. simultaneous) requests that will be performed to any single domain. | `16` | prismeai-crawler |
| `DOWNLOAD_DELAY` | [Minimum seconds to wait between 2 consecutive requests to the same domain.](https://docs.scrapy.org/en/latest/topics/settings.html#download-delay) | `0` | prismeai-crawler |
| `REQUEST_QUEUES_POLLING_INTERVAL` | Interval in seconds between each poll of the Redis queue for new requests to process. Lower values increase responsiveness but also Redis load. | `2` | prismeai-crawler |
| `REQUEST_QUEUES_POLLING_SIZE` | Number of requests to start from the queue in a single poll | `1` | prismeai-crawler |
| `USER_AGENT` | Crawler HTTP user agent | `Prisme.ai (https://prisme.ai)` | prismeai-crawler |
| `ROBOTSTXT_OBEY` | Whether the crawler should respect the site's `robots.txt`.
- Recommended: **`True`** for all public websites.
- For **internal portals (e.g. SharePoint, intranet)**, you may set **`False`** if robots.txt prevents access to content you are authorized to crawl.
⚠️ Disabling this option should only be done in controlled/internal environments. | `True` | prismeai-crawler |
| `DEFAULT_ENGINE_WEBPAGES_LIMIT` | Maximum pages that can be indexed per search engine before crawling stops | `30000` | prismeai-crawler |
| `MAX_SEARCHENGINES_NB` | Maximum number of search engines to run concurrently. `-1` means unlimited. | `-1` | prismeai-crawler |
These settings control the acknowledgment mechanism that ensures zero data loss when the crawler crashes during request processing.
| Variable Name | Description | Default Value | Affected Services |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------ | ----------------- |
| `ACK_ENABLED` | Enable/disable the request acknowledgment mechanism. When enabled, requests are tracked atomically in Redis and automatically reclaimed if the crawler crashes during processing. Disabling this reverts to the original behavior where crashed requests are lost. | `True` | prismeai-crawler |
| `ACK_DUPEFILTER_DROPS` | When `True`, requests dropped by the deduplication filter are also ACKed (cleaned up from pending structures). | `True` | prismeai-crawler |
| `SKIP_NON_RETRIABLE_ERRORS` | When `True`, HTTP 400/401/403/404/410 errors and certain exceptions (`UnsupportedFormat`, `CorruptedFile`, `AuthenticationFailed`) are moved to the DLQ immediately rather than retried. | `True` | prismeai-crawler |
| `PENDING_VISIBILITY_TIMEOUT` | Time in seconds before a pending request is considered stale and eligible for reclaim. Set this to 2-3x your longest document processing time (e.g., large PDFs). If a request takes longer than this timeout, it will be retried. | `900` (15 minutes) | prismeai-crawler |
| `PENDING_RECLAIM_INTERVAL` | Interval in seconds between checks for stale pending requests. The reclaim loop runs at this frequency to find and re-queue abandoned tasks. | `60` | prismeai-crawler |
| `PENDING_RECLAIM_BATCH_SIZE` | Maximum number of stale tasks to reclaim per reclaim loop iteration. | `10` | prismeai-crawler |
| `MAX_RETRY_ATTEMPTS` | Maximum number of times a task can be reclaimed/retried before it is moved to the DLQ permanently. Set to `0` to disable retry limits. | `3` | prismeai-crawler |
| `DLQ_MAX_SIZE` | Monitoring alert threshold: when the Dead Letter Queue grows beyond this size, warnings are emitted. Does not hard-cap the DLQ. Once limit is reached a log is shown periodically but it doesn't affect crawling. | `1000` | prismeai-crawler |
| `DLQ_RETENTION_DAYS` | Number of days DLQ entries are retained before cleanup. | `7` | prismeai-crawler |
These settings control adaptive resource monitoring to prevent OOM (Out of Memory) crashes through CPU and memory monitoring. The crawler will pause polling when resources are constrained and resume when they recover.
**Thresholds are percentages of container limits**, not system resources. Limits are auto-detected from cgroup files (cgroup v1/v2). For restricted Kubernetes clusters where cgroup detection fails, you can set `CONTAINER_CPU_LIMIT_MILLICORES` and `CONTAINER_MEMORY_LIMIT_BYTES` as fallbacks.
| Variable Name | Description | Default Value | Affected Services |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------- |
| `RESOURCE_THROTTLING_ENABLED` | Enable/disable resource-based throttling. When enabled, the crawler monitors memory and CPU usage and pauses polling when thresholds are exceeded to prevent OOM crashes. | `True` | prismeai-crawler |
| `RESOURCE_THROTTLING_MODE` | What resources to monitor. Options: `memory_only` (recommended - Kubernetes handles CPU throttling effectively), `cpu_only`, `both`, `none`. Memory monitoring is critical as memory exhaustion leads to OOMKilled. | `memory_only` | prismeai-crawler |
| `MEMORY_WARNING_THRESHOLD` | Memory usage percentage (of container limit) above which the crawler pauses polling for new requests. Existing requests continue processing. | `80` | prismeai-crawler |
| `MEMORY_CRITICAL_THRESHOLD` | Memory usage percentage (of container limit) above which the crawler stops all processing. This is a last-resort protection against OOM. | `95` | prismeai-crawler |
| `CPU_WARNING_THRESHOLD` | CPU usage percentage (of container limit) above which the crawler pauses polling. Only applies when `RESOURCE_THROTTLING_MODE` includes CPU monitoring. | `85` | prismeai-crawler |
| `CPU_CRITICAL_THRESHOLD` | CPU usage percentage (of container limit) above which the crawler stops all processing. Only applies when `RESOURCE_THROTTLING_MODE` includes CPU monitoring. | `95` | prismeai-crawler |
| `RESOURCE_CHECK_INTERVAL` | Interval in seconds between resource usage checks. Lower values provide faster response to resource pressure but increase overhead. | `1.0` | prismeai-crawler |
| `RESOURCE_CIRCUIT_BREAKER_THRESHOLD` | Number of consecutive resource check failures before escalating to a CRITICAL alert. This helps identify persistent resource issues vs. temporary spikes. | `10` | prismeai-crawler |
| `RESOURCE_CIRCUIT_BREAKER_SHUTDOWN_TIMEOUT` | Seconds the crawler can remain stuck in critical state before initiating automatic pod shutdown. The pod exits gracefully, allowing Kubernetes to restart it with fresh state. Set to `0` to disable auto-shutdown. | `900` (15 minutes) | prismeai-crawler |
| `RESOURCE_THROTTLE_FAST_RECLAIM` | When `True` and a critical resource threshold is hit, tasks are immediately released back to the Redis queue for other pods, rather than waiting for the visibility timeout to expire. | `True` | prismeai-crawler |
| `CONTAINER_CPU_LIMIT_MILLICORES` | Fallback CPU limit in millicores (e.g. `2000` = 2 cores) for restricted Kubernetes clusters where cgroup files are inaccessible. Used only if cgroup v1/v2 detection fails. | | prismeai-crawler |
| `CONTAINER_MEMORY_LIMIT_BYTES` | Fallback container memory limit in bytes for restricted Kubernetes clusters where cgroup files are inaccessible. If all detection methods fail, falls back to system total memory. | | prismeai-crawler |
| `RESOURCE_CIRCUIT_BREAKER_FORCE_EXIT_TIMEOUT` | Hard timeout in seconds for graceful reactor shutdown during circuit breaker auto-shutdown. If the reactor doesn't stop within this time, `os._exit(1)` is called to force termination. | `60` | prismeai-crawler |
These settings control how documents (PDFs, DOCX, etc.) are parsed and processed by the crawler.
| Variable Name | Description | Default Value | Affected Services |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | ----------------- |
| `DOCUMENTS_DEFAULT_PARSER` | Default parser for document processing. Options: `unstructured` (recommended, uses the Unstructured library), `tika` (Apache Tika - requires TIKA\_PATH to be configured). Can be overridden per-searchengine via parser configuration. | `unstructured` | prismeai-crawler |
| `UNSTRUCTURED_DEFAULT_STRATEGY` | Default parsing strategy for the Unstructured library. Options: `fast` (no OCR, fastest), `auto` (automatic detection), `hi_res` (full OCR processing, slowest but most accurate for scanned documents), `ocr_only` (OCR only). Can be overridden per-searchengine. | `fast` | prismeai-crawler |
| `TIKA_OCR_SKIP` | Skip OCR (Tesseract) processing when using Apache Tika parser. Set to `True` for faster parsing at the cost of not extracting text from images within documents. Similar to Unstructured's `fast` strategy. Can be overridden per-searchengine using parser config: `{"type": "tika", "skip_ocr": true}`. | `True` | prismeai-crawler |
| `TIKA_PATH` | Path to the directory containing `tika-server.jar`. If not set, Tika is disabled and the default parser falls back to Unstructured. | `/app/tika/` (in Docker image) | prismeai-crawler |
| `TIKA_PARSING_TIMEOUT` | Timeout in seconds for a single Tika parsing request. | `600` | prismeai-crawler |
| `TIKA_MAX_RETRIES` | Maximum retries for Tika parse requests before giving up. | `3` | prismeai-crawler |
### Resource Considerations
When planning your deployment, consider these resource recommendations:
* **Crawler**: Min 1GB, recommended 2GB+
* **SearchEngine**: Min 1GB, recommended 2GB+
* Scale based on crawl volume and index size
* **Crawler**: Min 0.5 vCPU, recommended 1+ vCPU
* **SearchEngine**: Min 0.5 vCPU, recommended 1+ vCPU
* Consider additional resources for high request volumes
* **ElasticSearch**: Plan for index growth based on content volume
* **Redis**: Minimal requirements for queue management
* Consider storage class with good I/O performance
* Internet access for the Crawler service
* Internal network access between services
* Consider bandwidth requirements for crawl activities
## Deployment Process
Follow these steps to deploy the Crawler and SearchEngine microservices:
Ensure ElasticSearch and Redis are accessible:
1. Verify ElasticSearch connection with:
```bash theme={null}
curl -X GET "[ELASTIC_SEARCH_URL]:9200"
```
You should receive a response with version and cluster information
2. Verify Redis connection with:
```bash theme={null}
redis-cli -u [REDIS_URL] ping
```
The response should be "PONG"
Deploy both services using Helm.
Ensure both services are included in your `values.yaml` configuration:
```yaml theme={null}
prismeai-crawler:
enabled: true
config:
redis:
url: "redis://redis-service:6379"
elasticsearch:
url: "elasticsearch-service:9200"
prismeai-searchengine:
enabled: true
config:
redis:
url: "redis://redis-service:6379"
elasticsearch:
url: "elasticsearch-service:9200"
```
Check that both services are running correctly:
```bash theme={null}
kubectl get pods -n apps | grep 'crawler\|searchengine'
```
Both services should show `Running` status and be ready (e.g., `1/1`).
Ensure the services can access:
1. ElasticSearch and Redis internally
2. Internet access for the Crawler service
3. Access from other Prisme.ai services that will use search functionality
## Microservice Testing
After deploying the Crawler and SearchEngine microservices, verify their operation with these steps:
Create a searchengine instance to crawl a test website:
```bash theme={null}
curl --location 'http://localhost:8000/monitor/searchengine/test/test' \
--header 'Content-Type: application/json' \
--data '{
"websites": [
"https://docs.eda.prisme.ai/en/workspaces/"
]
}'
```
If successful, you should receive a complete searchengine object that includes an `id` field.
After a few seconds, check the crawl history and statistics:
```bash theme={null}
curl --location --request GET 'http://localhost:8000/monitor/searchengine/test/test/stats' \
--header 'Content-Type: application/json' \
--data '{
"urls": ["https://docs.eda.prisme.ai/en/workspaces/"]
}'
```
Verify that:
* The `metrics.indexed_pages` field is greater than 0
* The `metrics.pending_requests` field indicates active crawling
* The `crawl_history` section shows pages that have been processed
Perform a test search query to verify indexing and search:
```bash theme={null}
curl --location 'http://localhost:8000/search/test/test' \
--header 'Content-Type: application/json' \
--data '{
"query": "workspace"
}'
```
The response should include a `results` array containing pages from the crawled website that match your search term. Each result should include relevance information and content snippets.
If all tests pass, congratulations! Your Crawler and SearchEngine microservices are up and running correctly.
## Features and Capabilities
The Crawler service provides advanced web content discovery and extraction:
* **Configurable crawl depth**: Control how many links deep the crawler will explore
* **URL filtering**: Include or exclude specific URL patterns
* **Rate limiting**: Respect website terms of service with configurable crawl rates
* **Content extraction**: Parse and clean HTML to extract meaningful content
* **Metadata extraction**: Capture titles, descriptions, and other metadata
* **Scheduled crawls**: Set up periodic recrawling to keep content fresh
* **Robots.txt compliance**: Respect website crawling policies
The SearchEngine service delivers powerful search functionality:
* **Full-text search**: Find content across all indexed documents
* **Relevance ranking**: Surface the most relevant content first
* **Content highlighting**: Highlight matching terms in search results
* **Faceted search**: Filter results by metadata fields
* **Synonym handling**: Find content using related terms
* **Language support**: Index and search content in multiple languages
* **Query suggestions**: Support for "did you mean" functionality
* **Result snippets**: Show context around matching terms
## Integration with Prisme.ai
The Crawler and SearchEngine microservices integrate with other Prisme.ai components:
* Create knowledge bases from crawled web content
* Enrich existing knowledge bases with web information
* Use search capabilities for better information retrieval
* Build custom search interfaces using search API
* Integrate search results into workflows
* Trigger crawls programmatically in automations
* Power research agents with web crawling capabilities
* Create domain-specific search tools
* Develop content discovery applications
* Extend crawling behavior with custom functions
* Process search results with specialized logic
* Create advanced search and discovery experiences
## Advanced Configuration
When creating a searchengine, you can specify advanced crawl options:
```json theme={null}
{
"websites": ["https://example.com"],
"options": {
"maxDepth": 3,
"includePatterns": ["*/blog/*", "*/products/*"],
"excludePatterns": ["*/admin/*", "*/login/*"],
"respectRobotsTxt": true,
"crawlDelay": 1000,
"userAgent": "Prisme.ai Crawler",
"maxPagesPerSite": 1000
}
}
```
These options allow you to fine-tune crawling behavior for different use cases.
The services automatically create and manage ElasticSearch indices. For advanced use cases, you can:
* Configure index settings like sharding and replication
* Set up index lifecycle policies for managing index growth
* Implement custom analyzers for specialized search needs
* Configure cross-cluster search for large-scale deployments
Consult the ElasticSearch documentation for more information on these advanced configurations.
To optimize performance for your specific needs:
* Adjust `MAX_CONTENT_LEN` to balance comprehensiveness with resource usage
* Configure crawler concurrency settings for faster crawling
* Implement ElasticSearch performance optimizations
* Consider Redis caching strategies for frequent searches
* Use horizontal scaling for high-volume crawling and search scenarios
## Troubleshooting
**Symptom**: Web pages are not being crawled or indexed
**Possible causes**:
* Network connectivity issues
* Website robots.txt restrictions
* Rate limiting by target websites
* URL pattern configuration excluding relevant pages
**Resolution steps**:
1. Check crawler logs for specific error messages
2. Verify network connectivity to target websites
3. Review website robots.txt for restrictions
4. Adjust crawl rate settings to avoid being blocked
5. Check URL pattern configurations
**Symptom**: Search results are missing or irrelevant
**Possible causes**:
* Content not properly indexed
* ElasticSearch configuration issues
* Query formatting problems
* Content exceeding maximum length limits
**Resolution steps**:
1. Verify content was successfully crawled and indexed
2. Check ElasticSearch connectivity and health
3. Review search query format and parameters
4. Check if content exceeds `MAX_CONTENT_LEN` setting
5. Test simple queries to validate basic functionality
**Symptom**: Slow crawling or search response times
**Possible causes**:
* Insufficient resources allocated
* ElasticSearch performance problems
* Redis bottlenecks
* Large crawl queues or index sizes
**Resolution steps**:
1. Monitor resource usage during operations
2. Check ElasticSearch performance metrics
3. Verify Redis isn't running out of memory
4. Consider scaling resources horizontally or vertically
5. Implement more targeted crawling strategies
Since Tika crawling takes place in a separate process than the main prisme.ai-crawler process, troubleshooting Tika issues may require auditing Tika logs directly in prisme.ai-crawler container :
* /app/tika/tika.log
* /app/tika/tika-server.log
## Security Considerations
When deploying and using the Crawler and SearchEngine microservices, keep these security considerations in mind:
* Implement appropriate network policies
* Consider using a dedicated proxy for outbound crawling
* Monitor for unusual traffic patterns
* Be mindful of crawling and indexing sensitive content
* Implement URL patterns to exclude sensitive areas
* Consider content filtering before indexing
* Secure ElasticSearch and Redis with strong authentication
* Implement API access controls for search endpoints
* Use TLS for all service communications
* Respect website terms of service when crawling
* Consider data retention policies for crawled content
* Be aware of copyright implications of content indexing
For any issues or questions during the deployment process, contact [support@prisme.ai](mailto:support@prisme.ai) for assistance.