Skip to main content
The Knowledge Bases API allows you to create and manage knowledge bases that agents can query for information. Knowledge bases organize collections and data sources for AI-powered retrieval with customizable chunking strategies and embedding models.

​
Key Features

  • Customizable Chunking: Recursive, hierarchical, or fixed strategies
  • Structured Data Support: Process CSV and Excel files
  • Multiple Embedding Providers: Azure OpenAI, Mistral, Telekom OTC
  • AI-Powered Retrieval: Vector-based semantic search
  • Milvus Processing: Production-ready vector database engine
  • Flexible Data Selection: Combine collections and individual data sources
  • Status Monitoring: Track processing progress with detailed metadata
  • Agent Integration: Direct use in agent configurations

​
Authentication

All endpoints require authentication using your API key:
  • API Key: x-api-key: <key>

​
Available Endpoints

List Knowledge Bases

Retrieve all knowledge bases

Create Knowledge Base

Create a new knowledge base

Get Knowledge Base

Retrieve knowledge base details

Update Knowledge Base

Modify knowledge base settings

Delete Knowledge Base

Remove a knowledge base

Check Status

Monitor processing status

Available Collections

List collections ready for use

​
Processing Engine

Milvus Vector Database
  • High-performance vector similarity search
  • Scalable for large document collections
  • Advanced HNSW indexing algorithms
  • Real-time updates and queries
  • Production-ready for enterprise deployments
Milvus is currently the only supported processing engine. Additional engines may be added in future releases.

​
Knowledge Base Lifecycle

​
1. Creation

Define knowledge base with data sources, chunking strategy, and embedding configuration: 
{
  "name": "Product Knowledge Base",
  "description": "Product documentation and guides",
  "processing_engine": "milvus",
  "selected_collections": ["coll_123"],
  "selected_datasources": ["ds_456"],
  "chunking_strategy": {
    "strategy": "recursive",
    "chunk_size": 500,
    "chunk_overlap": 50,
    "split_by": "word",
    "recursive_separators": ["\n\n", "\n", ". ", " "]
  },
  "embedding_config": {
    "provider": "mistral",
    "model": "mistral-embed",
    "dimensions": 1024
  }
}

​
2. Processing

Knowledge base processes data source content in two phases: Phase 1: Chunking (0-20% progress)
  • Extract text from documents (PDF, DOCX, TXT, MD, LOG)
  • Process structured data (CSV, Excel) with custom row batching
  • Apply chunking strategy to split into searchable chunks
  • Track per-document progress
Phase 2: Embedding & Indexing (20-100% progress)
  • Generate embeddings via batch API (Mistral, Azure OpenAI, or Telekom OTC)
  • Stream embeddings to Milvus vector database
  • Build HNSW indices for fast similarity search
  • Validate data quality and store metadata

​
3. Completion

Ready knowledge bases can be used by agents for retrieval. Check status, progress, and success_rate to monitor health.

​
Knowledge Base Status

Knowledge base created but processing not started
Collections are being processed and indexed
All collections processed and ready for queries
Processing failed - check error details and logs

​
Response Schema

Knowledge base responses include comprehensive processing details:
​
datasources
array
Array of data source objects included in this knowledge base
​
datasource_count
integer
Total number of data sources
​
ingested_datasources
array
Successfully processed data sources with full metadata
​
failed_datasources
array
Data sources that failed processing with error details
​
datasource_errors
object
Map of datasource_id to error messages for failed sources
​
datasource_snapshots
object
Snapshot of datasource details at ingestion time with ingestion status
​
success_rate
number
Percentage of successfully processed data sources (0-100)
​
chunking_strategy
object
Applied chunking configuration (recursive, hierarchical, or fixed)
​
embedding_config
object
Embedding model configuration (provider, model, dimensions)
​
processing_metadata
object
Processing progress details including current phase and document being processed

​
Supported File Types

Documents:
  • PDF, DOC, DOCX
  • TXT, MD (Markdown), LOG
Structured Data:
  • CSV (row-level or file-level processing)
  • XLSX, XLS (Excel with batch row processing)

​
Chunking Strategies

Three strategies available for document processing:

​
Recursive

Splits text using multiple separators in order. Best for general documents.
  • Configurable separators (paragraphs, sentences, words)
  • Flexible chunk sizes and overlap
  • Supports word or character splitting only

​
Hierarchical

Creates multi-level chunks preserving document structure. Ideal for research papers.
  • Multiple block sizes in descending order
  • Maintains document hierarchy
  • Word splitting only

​
Fixed

Simple fixed-size chunks. Fastest processing.
  • Consistent chunk sizes
  • Configurable overlap
  • Word splitting only
See Create Knowledge Base for detailed configuration.

​
Embedding Providers

Azure OpenAI
  • text-embedding-ada-002 (1536 dimensions)
Mistral AI
  • mistral-embed (1024 dimensions)
Telekom OTC
  • text-embedding-bge-m3 (1024 dimensions)
  • jina-embeddings-v2-base-de (768 dimensions) - German-optimized
  • jina-embeddings-v2-base-code (768 dimensions) - Code-optimized
  • tsi-embedding-colqwen2-2b-v1 (1024 dimensions)

​
Agent Integration

Use knowledge bases in agent configurations: 
name: "Support Agent"
agent_type: "llm_agent"
system_prompt: "You are a helpful support agent..."
knowledge_bases:
  - name: "product_kb"
    knowledge_base_type: "milvus"
    knowledge_base_id: "kb_123"
    retrieval_config:
      strategy: "dense"
      top_k: 10
      search_ef: 64
      metric_type: "COSINE"

​
Retrieval Configuration

  • Search parameters: Control relevance and results
  • Context integration: How retrieved content is used
  • Fallback behavior: When no relevant content is found
Test knowledge base queries before deploying agents to ensure relevant results.

​
Performance Considerations

​
Indexing Time

  • Collection size: Larger collections take longer to process
  • Content complexity: Rich documents require more processing
  • Engine choice: Different engines have varying performance characteristics

​
Query Performance

  • Index optimization: Proper indexing improves search speed
  • Result filtering: Limit results for faster responses
  • Caching: Frequently accessed content is cached

​
Scalability

  • Concurrent queries: Multiple agents can query simultaneously
  • Update frequency: How often content changes
  • Storage requirements: Vector storage grows with content
Large knowledge bases may require significant processing time. Plan for appropriate indexing windows.

​
Monitoring and Troubleshooting

​
Health Checks

  • Processing progress: Track completion percentage
  • Error rates: Monitor failed data sources
  • Query performance: Response times and accuracy

​
Common Issues

  • Collection dependencies: Ensure collections are completed
  • Processing failures: Check data source formats and content
  • Memory limits: Large knowledge bases may hit resource limits
Use the status endpoint to monitor knowledge base health and processing progress.