# Knowledge & Data Sources ## 🧠 **Powering Your Agent with Domain Knowledge** The Knowledge tab is where you transform your AI agent from a general assistant into a domain expert. By connecting relevant data sources, documents, and knowledge bases, you give your agent access to the specific information it needs to provide accurate, contextual responses. *** ## 🎯 **Knowledge Source Types** ### **πŸ“„ Document Upload** Upload files directly to your agent's knowledge base for semantic search and retrieval. #### **Supported File Types** * **PDF Documents**: Research papers, manuals, reports * **Word Documents**: (.docx) Policies, procedures, guides * **Text Files**: (.txt, .md) Documentation, notes, plain text * **HTML Files**: (.html) Web pages, formatted documentation * **Spreadsheets**: (.xlsx, .xls, .csv) Data tables, catalogs, structured data #### **Document Processing Features** * **Intelligent Chunking**: Configurable chunking strategies for optimal knowledge retrieval * **Text Extraction**: Automatic text extraction from supported formats * **Space Normalization**: Remove extra whitespace for cleaner text #### **Best Practices for Document Upload** ``` βœ… DO: - Use clear, descriptive dataset names - Keep documents focused on specific topics - Ensure text is selectable (not just images) - Configure chunking based on document structure ❌ AVOID: - Uploading duplicate or conflicting information - Using documents with poor formatting - Including sensitive personal information without proper access controls - Overwhelming with too many similar documents ``` ### **πŸ“Š Table Upload** Upload structured data in tabular format for precise lookups and semantic search. #### **Supported Table Formats** * **CSV Files**: Comma-separated values * **Excel Files**: (.xlsx, .xls) Spreadsheets with single or multiple sheets * **Manual Entry**: Create tables directly in the interface #### **Table Configuration** * **Column Types**: TEXT, NUMBER, INTEGER, BOOLEAN, DATE * **Semantic Columns**: Mark columns for semantic search indexing * **Column Sequencing**: Define display order for columns * **Schema Analysis**: Automatic type detection from uploaded files #### **Table Use Cases** * Product catalogs with specifications and pricing * Customer records and interaction history * FAQ databases with questions and answers * Knowledge articles with categorization * Configuration and settings databases ### **πŸ—„οΈ Database Schema (Manual)** Create database-like schemas for structured knowledge organization. #### **Database Format Features** * **Custom Schema Design**: Define your own table structures * **Column Type Support**: TEXT, NUMBER, INTEGER, BOOLEAN, DATE * **Manual Data Entry**: Populate data through the interface * **Structured Queries**: Enable precise data retrieval *** ## βš™οΈ **Knowledge Processing Settings** ### **Chunking Strategy** Control how documents are broken down for processing and retrieval. #### **Chunking Configuration Options** * **Chunk Type**: Strategy for dividing content * **Max Tokens**: Maximum size per chunk (configurable) * **Separator**: Custom separator for chunk boundaries * **Remove Extra Spaces**: Clean up whitespace * **Remove URLs/Emails**: Filter out contact information #### **Best Practices** ``` Document Length Guidelines: - Short documents (< 5 pages): Larger chunks (1000+ tokens) - Medium documents (5-50 pages): Medium chunks (500-1000 tokens) - Long documents (50+ pages): Smaller chunks (200-500 tokens) - Technical docs with code: Preserve code blocks intact ``` *** ## πŸ” **Agent Knowledge Configuration** Configure how your agent retrieves and uses knowledge during conversations. ### **Retrieval Mode** #### **Auto Retrieval** (Default: Off) ``` Automatic Knowledge Access: - Agent automatically retrieves relevant knowledge for every query - No explicit tool call needed - Best for: Agents that always need domain knowledge - Trade-off: May retrieve unnecessary information ``` #### **Manual Tool Call** (Default: On) ``` On-Demand Knowledge Access: - Agent decides when to retrieve knowledge using available tools - More control over when knowledge is accessed - Best for: General-purpose agents that sometimes need knowledge - Trade-off: Requires agent to recognize when knowledge is needed ``` ### **Search Strategy** #### **Hybrid Search** (Default - Recommended) ``` Combined Approach: - Semantic search: Understanding context and meaning - Full-text search: Exact keyword matching - Best for: Most use cases, balances accuracy and coverage - Returns: Semantically relevant + keyword-matched results - Cost extra credits for re-ranking documents. ``` #### **Semantic Search Only** ``` Vector-Based Retrieval: - Understanding context and intent - Finding conceptually related information - Handling synonyms and variations - Best for: Natural language queries and conceptual searches ``` #### **Full-Text Search Only** ``` Keyword-Based Retrieval: - Exact term matching - Faster for specific lookups - Good for technical terminology and precise queries - Best for: Known keywords and exact phrase matching ``` ### **Retrieval Parameters** #### **Top K** (Default: 5, Range: 1-10) ``` Number of Knowledge Chunks to Retrieve: - Higher values: More comprehensive context, higher cost - Lower values: Focused context, lower cost - Recommended: 3-5 for most use cases - Adjust based on: Query complexity and knowledge base size ``` #### **Threshold** (Default: 0.5, Range: 0.0-1.0) ``` Relevance Score Threshold: - Higher threshold (0.7-1.0): Only highly relevant results - Medium threshold (0.4-0.7): Balanced relevance - Lower threshold (0.0-0.4): Include more potential matches - Recommended: Start at 0.5, adjust based on retrieval quality ``` #### **Query Rewrite** (Default: On) ``` Query Optimization: - Rewrites user query before knowledge retrieval - Improves search relevance by clarifying intent - Expands abbreviations and adds context - Recommended: Enable for most use cases ``` #### **Rerank** (Default: Off) ``` Result Reranking: - Post-processes retrieved results for better ordering - Uses cross-encoder models for more accurate relevance - Trade-off: Better results but additional latency - Recommended: Enable for critical accuracy use cases ``` ### **Connected Datasets** #### **Multiple Dataset Support** * Connect up to **100 datasets** per agent * Each dataset appears as a searchable knowledge source * Datasets maintain their own: * Name and ID * Source type (UPLOAD, MANUAL) * Format type (TEXT, TABLE, DATABASE) * Processing status #### **Dataset Information Display** For each connected dataset, the agent has access to: * Dataset name (user-friendly identifier) * Dataset ID (unique identifier) * Source type (how data was added) * Status (PENDING, SUCCESS, FAILURE) * Format type (TEXT, TABLE, DATABASE) *** ## πŸ“Š **Knowledge Analytics & Management** ### **Dataset Status Monitoring** #### **Processing States** * **PENDING**: Dataset creation or update in progress * **SUCCESS**: Dataset ready for use * **FAILURE**: Processing encountered errors #### **Progress Tracking** * Monitor document import progress * Track embedding generation status * View chunk processing metrics ### **Embedding Updates** #### **Manual Embedding Refresh** ``` When to Update Embeddings: - After modifying dataset content - After bulk row updates in tables - To incorporate new document versions ``` *** ## πŸ”§ **Knowledge Configuration Best Practices** ### **Initial Setup Process** 1. **Audit Existing Information**: Catalog what knowledge you have 2. **Choose Dataset Format**: TEXT for documents, TABLE for structured data 3. **Configure Processing**: Set chunking and parsing options 4. **Select Embedding Model**: Choose based on language and domain 5. **Test Retrieval**: Verify agent responses with sample queries ### **Dataset Organization Strategies** #### **By Topic** ``` Product Knowledge: β”œβ”€β”€ Product Features (TEXT dataset) β”œβ”€β”€ Technical Specifications (TABLE dataset) β”œβ”€β”€ Pricing & Plans (TABLE dataset) └── Troubleshooting Guides (TEXT dataset) Customer Support: β”œβ”€β”€ Common Issues (TEXT dataset) β”œβ”€β”€ Resolution Procedures (TEXT dataset) └── Product Updates (TEXT dataset) ``` #### **By Source Type** ``` Documentation: β”œβ”€β”€ User Manual.pdf β†’ TEXT dataset β”œβ”€β”€ API Reference.pdf β†’ TEXT dataset └── FAQ.csv β†’ TABLE dataset Internal Knowledge: β”œβ”€β”€ Training Materials β†’ TEXT dataset β”œβ”€β”€ Process Documents β†’ TEXT dataset └── Policy Database β†’ DATABASE dataset ``` ### **Optimization Guidelines** #### **Document Preparation** ``` Before Upload: - Remove duplicate content across documents - Ensure consistent formatting - Split very large documents into logical sections - Use descriptive filenames - Remove or redact sensitive information ``` #### **Table Design** ``` Column Configuration: - Mark relevant columns as "semantic" for search - Use appropriate data types (TEXT, NUMBER, DATE, etc.) - Include descriptive column names - Maintain data consistency across rows - Consider creating separate tables for different entity types ``` #### **Retrieval Tuning** ``` Adjustment Process: 1. Start with defaults (Hybrid, Top K=5, Threshold=0.5) 2. Test with representative queries 3. If too few results: Lower threshold, increase Top K 4. If too many irrelevant results: Raise threshold, enable rerank 5. If missing semantic matches: Switch to Semantic search 6. If missing exact matches: Switch to Full-text search ``` *** ## πŸš€ **Advanced Features** ### **Multi-Dataset Retrieval** When connecting multiple datasets to an agent: * Agent can search across all connected datasets * Results merged and ranked by relevance * Each result includes source dataset information * Useful for comprehensive knowledge coverage ### **Semantic Column Configuration** For TABLE and DATABASE formats: * Mark specific columns for semantic search indexing * Non-semantic columns remain queryable but not embedded * Reduces embedding costs for large tables * Improves search focus on relevant fields *** ## 🎯 **Knowledge Integration Checklist** Before activating your agent's knowledge base: * [ ] **Dataset Created**: Upload documents or create tables * [ ] **Processing Complete**: Verify dataset status is SUCCESS * [ ] **Embeddings Generated**: Check embedding progress is 100% * [ ] **Search Strategy Configured**: Choose Hybrid, Semantic, or Full-text * [ ] **Retrieval Parameters Set**: Configure Top K, threshold, rerank * [ ] **Retrieval Mode Selected**: Auto or manual tool call * [ ] **Test Queries Run**: Verify knowledge retrieval with sample questions * [ ] **Access Controls Set**: Ensure appropriate workspace permissions *** Your agent's knowledge is its competitive advantageβ€”invest in building a comprehensive, well-organized knowledge base that enables intelligent, accurate responses.