RAG System (Auto-Build) ======================= The RAG System provides tools for automatically building vector databases from various sources including documentation websites, GitHub repositories, and PDF files. It features web crawling, document processing, and integration with Hugging Face for database distribution. Overview -------- The RAG auto-build system (``pantheon.toolsets.utils.rag``) provides: - **Automated Web Crawling**: Deep crawl documentation sites and extract content - **Multi-Source Support**: Build from websites, GitHub READMEs, PDFs, and local files - **Vector Database Creation**: Automatic chunking and embedding with LanceDB - **Hugging Face Integration**: Upload and download pre-built databases - **Caching System**: Intelligent caching for embeddings and build progress - **YAML Configuration**: Define sources and parameters in YAML files YAML Configuration Format ------------------------- The YAML configuration file defines one or more vector databases with their sources: Basic Structure ~~~~~~~~~~~~~~~ :: database_name: description: Description of the database type: vector_db parameters: embedding_model: text-embedding-3-large chunk_size: 4000 chunk_overlap: 200 items: item_name: type: source_type url: source_url description: Item description Supported Source Types ~~~~~~~~~~~~~~~~~~~~~~ - **package documentation**: Deep crawls documentation websites - **tutorial**: Processes tutorial websites with multi-page content - **github readme**: Fetches README files from GitHub repositories - **pdf**: Downloads and processes PDF documents Example Configuration ~~~~~~~~~~~~~~~~~~~~~ :: single-cell-python-packages: description: Vector database of single-cell python packages documentation type: vector_db parameters: embedding_model: text-embedding-3-large chunk_size: 4000 chunk_overlap: 200 items: scanpy: type: package documentation url: https://scanpy.readthedocs.io/en/stable/ description: Scanpy toolkit for single-cell analysis sc-best-practices: type: tutorial url: https://www.sc-best-practices.org/ description: Best practices guide for single-cell analysis star: type: pdf url: https://raw.githubusercontent.com/alexdobin/STAR/master/doc/STARmanual.pdf description: STAR RNA-seq aligner manual Command Line Usage ------------------ Build from YAML Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Build the database:: python -m pantheon.toolsets.utils.rag build config.yaml ./output_dir Upload to Hugging Face ~~~~~~~~~~~~~~~~~~~~~~ Share your built database:: # Set your Hugging Face token export HUGGINGFACE_TOKEN=your_token_here # Upload to default repo (NaNg/pantheon_rag_db) python -m pantheon.toolsets.utils.rag upload ./output_dir # Or specify custom repo python -m pantheon.toolsets.utils.rag upload ./output_dir --repo-id your-username/your-repo Download Pre-built Database ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download existing databases:: # Download from default repo python -m pantheon.toolsets.utils.rag download ./local_dir # Download from custom repo python -m pantheon.toolsets.utils.rag download ./local_dir --repo-id username/repo --filename custom.zip Build Process Details --------------------- Directory Structure ~~~~~~~~~~~~~~~~~~~ After building, the output directory contains:: output_dir/ └── database_name/ ├── metadata.yaml # Database configuration ├── info_cache.json # Build progress tracking ├── raw/ # Downloaded raw content │ └── item_name/ │ ├── *.md # Markdown files │ └── *.pdf # PDF files └── lancedb/ # Vector database files └── database_name.lance/ Processing Pipeline ~~~~~~~~~~~~~~~~~~~ 1. **Download Phase**: - For documentation/tutorials: Deep crawl with configurable depth - For GitHub READMEs: Direct file download - For PDFs: Binary file download 2. **Content Extraction**: - HTML → Markdown conversion for web content - PDF text extraction using PyMuPDF - Duplicate removal via content hashing 3. **Text Processing**: - Smart chunking with configurable size and overlap - Metadata preservation (source, URL) - Context maintenance across chunks 4. **Vector Storage**: - Embedding generation via OpenAI API - LanceDB storage with PyArrow schema - Metadata indexing for filtering Progress Tracking ~~~~~~~~~~~~~~~~~ The system maintains build state in ``info_cache.json``:: { "item_name": { "success": true, "created_at": "2024-01-01T12:00:00", "download_success": true } } Key behaviors: - Successfully cached items (``success: true``) are skipped on subsequent builds - Failed items can be retried without re-processing successful ones - To force re-download: Set ``download_success`` to ``false`` in ``info_cache.json`` - To force complete rebuild: Set ``success`` to ``false`` in ``info_cache.json`` - Then re-run the build command to process the modified items Programmatic Usage ------------------ Using VectorDB Class ~~~~~~~~~~~~~~~~~~~~ Direct database operations:: from pantheon.toolsets.utils.rag.vectordb import VectorDB # Load existing database db = VectorDB("./output_dir/database_name") # Query the database results = await db.query( query="How to perform clustering analysis?", top_k=5, source="scanpy" # Optional: filter by source ) # Insert new content await db.insert( text="Your new content here", metadata={"source": "custom", "date": "2024-01-01"} ) # Insert from file with automatic chunking await db.insert_from_file( file_path="./new_doc.md", metadata={"source": "local_docs"} ) Building Programmatically ~~~~~~~~~~~~~~~~~~~~~~~~~ Build databases from Python code:: import asyncio from pantheon.toolsets.utils.rag.build import build_vector_db db_config = { "type": "vector_db", "parameters": { "embedding_model": "text-embedding-3-large", "chunk_size": 4000, "chunk_overlap": 200 }, "items": { "my_docs": { "type": "package documentation", "url": "https://mydocs.example.com", "description": "My documentation" }, "manual": { "type": "pdf", "url": "https://example.com/manual.pdf", "description": "User manual" } } } asyncio.run(build_vector_db("my_knowledge_base", db_config, "./output")) Full Build Example ~~~~~~~~~~~~~~~~~~ Complete workflow from YAML to usage:: import asyncio from pantheon.toolsets.utils.rag.build import build_all # Build all databases defined in YAML asyncio.run(build_all("./config.yaml", "./output")) Special Features ---------------- PDF Support ~~~~~~~~~~~ The system can process PDF documents:: items: research_paper: type: pdf url: https://arxiv.org/pdf/2301.00001.pdf description: Research paper on transformers PDFs are: - Downloaded as binary files - Text extracted using PyMuPDF - Processed like other text documents - Stored with original URL metadata Web Crawling Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~ For documentation and tutorial types: - **max_depth**: Controls crawling depth (default: 1) - **include_external**: Include external links (default: false) - Uses BFS (Breadth-First Search) strategy - Automatic markdown extraction from HTML - Removes duplicate content via SHA-256 hashing Embedding Models ~~~~~~~~~~~~~~~~ Supported OpenAI embedding models: - ``text-embedding-3-large``: Best quality, 3072 dimensions - ``text-embedding-3-small``: Faster, 1536 dimensions - ``text-embedding-ada-002``: Legacy model, 1536 dimensions Caching System ~~~~~~~~~~~~~~ Two-level caching for efficiency: 1. **Embedding Cache**: Disk-based cache prevents redundant API calls 2. **Build Cache**: ``info_cache.json`` tracks processing status Hugging Face Integration ------------------------ Upload and Distribution ~~~~~~~~~~~~~~~~~~~~~~~~ The system integrates with Hugging Face for sharing databases: 1. **Packaging**: Creates ZIP archive of entire database 2. **Upload**: Pushes to Hugging Face dataset repository 3. **Versioning**: Uploaded as ``latest.zip`` by default :: # Upload with authentication export HUGGINGFACE_TOKEN=hf_xxxxx python -m pantheon.toolsets.utils.rag upload ./my_db --repo-id myorg/my-rag-db Download and Use ~~~~~~~~~~~~~~~~ Download pre-built databases:: # Download and extract python -m pantheon.toolsets.utils.rag download ./local_db --repo-id myorg/my-rag-db # Use immediately from pantheon.toolsets.utils.rag.vectordb import VectorDB db = VectorDB("./local_db/database_name") Real-World Examples ------------------- Single-Cell Analysis Knowledge Base ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Build a comprehensive single-cell analysis database:: # single_cell.yaml single-cell-tools: description: Comprehensive single-cell analysis tools documentation type: vector_db parameters: embedding_model: text-embedding-3-large chunk_size: 4000 chunk_overlap: 200 items: scanpy: type: package documentation url: https://scanpy.readthedocs.io/en/stable/ description: Core single-cell analysis toolkit scvi-tools: type: package documentation url: https://docs.scvi-tools.org/en/stable/ description: Deep learning for single-cell cellranger: type: pdf url: https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/using/tutorial_ct description: 10x Genomics Cell Ranger manual Machine Learning Documentation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create ML framework knowledge base:: # ml_docs.yaml ml-frameworks: description: Machine learning frameworks documentation type: vector_db parameters: embedding_model: text-embedding-3-large chunk_size: 3000 chunk_overlap: 300 items: pytorch: type: package documentation url: https://pytorch.org/docs/stable/ description: PyTorch deep learning framework tensorflow: type: package documentation url: https://www.tensorflow.org/api_docs description: TensorFlow ML platform papers: type: pdf url: https://arxiv.org/pdf/1706.03762.pdf description: Attention Is All You Need paper Best Practices -------------- 1. **Choose Appropriate Chunk Sizes**: - Larger chunks (4000-5000) for narrative content - Smaller chunks (1000-2000) for API references - Balance between context and precision 2. **Optimize Embedding Models**: - Use ``text-embedding-3-large`` for best quality - Consider ``text-embedding-3-small`` for large datasets - Monitor API costs 3. **Organize Sources Logically**: - Group related documentation in same database - Use descriptive item names - Add clear descriptions for each source 4. **Handle Build Failures**: - Check ``info_cache.json`` for error details - Failed items can be retried individually - Network issues don't affect completed items 5. **Use Hugging Face for Distribution**: - Share pre-built databases to save compute - Version control via repository tags - Collaborate on knowledge bases 6. **Regular Updates**: - Rebuild periodically for fresh content - Use caching to minimize redundant work - Track changes via git Troubleshooting --------------- Common Issues ~~~~~~~~~~~~~ **Build Failures**: - Check network connectivity for web crawling - Verify URLs are accessible (not behind authentication) - Review error messages in ``info_cache.json`` - Ensure sufficient disk space **PDF Processing Errors**: - Verify PDF URL is directly accessible - Check if PDF requires authentication - Some PDFs may have text extraction restrictions - Install PyMuPDF: ``pip install pymupdf`` **Embedding Errors**: - Ensure ``OPENAI_API_KEY`` is set correctly - Check API rate limits and quotas - Verify model name is correct - Monitor token usage **Storage Issues**: - Ensure sufficient disk space (databases can be large) - Check write permissions on output directory - Clean old cache files if needed - Verify LanceDB installation Performance Tips ~~~~~~~~~~~~~~~~ - Use embedding cache to avoid redundant API calls - Enable progress tracking to resume interrupted builds - Process sources in parallel when system allows - Consider smaller embedding models for very large datasets - Use local caching for frequently accessed databases Integration with VectorRAGToolSet ---------------------------------- The VectorRAGToolSet class provides a toolset interface for agents to interact with the built databases. Basic Usage ~~~~~~~~~~~ Create a VectorRAGToolSet with your built database:: from pantheon.toolsets.vector_rag import VectorRAGToolSet # Initialize with database path rag_toolset = VectorRAGToolSet( name="knowledge_assistant", db_path="./output/single-cell-tools" ) # The toolset provides these tools: # - query_vector_db: Query the database with optional source filtering # - get_vector_db_info: Get metadata about the database Configuration Parameters ~~~~~~~~~~~~~~~~~~~~~~~~ Full configuration options:: rag_toolset = VectorRAGToolSet( name="custom_rag", db_path="./output/ml-frameworks", worker_params=None, # Optional worker configuration allow_insert=False, # Enable insert_vector_db tool allow_delete=False, # Enable delete_vector_db tool db_params={} # Additional database parameters ) Available Tools ~~~~~~~~~~~~~~~ **query_vector_db**: Main query interface - ``query``: Query string - ``top_k``: Number of results (default: 3) - ``source``: Optional source filter **get_vector_db_info**: Returns database metadata including description **insert_vector_db** (optional): Add new content - Enabled with ``allow_insert=True`` - Parameters: ``text``, ``metadata`` **delete_vector_db** (optional): Remove content - Enabled with ``allow_delete=True`` - Parameter: ``id`` (string or list) Using with Agents ~~~~~~~~~~~~~~~~~ Connect the toolset to an agent as a remote service:: from pantheon.agent import Agent from pantheon.toolsets.vector_rag import VectorRAGToolSet import asyncio async def main(): # Create RAG toolset rag_toolset = VectorRAGToolSet( name="bio_knowledge", db_path="./output/single-cell-tools", allow_insert=True # Allow agent to add new knowledge ) # Start toolset service await rag_toolset.start_service() # Create agent agent = Agent( name="bioinformatics_expert", instructions="You are a single-cell analysis expert. Use get_vector_db_info first to understand the database, then query_vector_db to find relevant information." ) # Connect agent to toolset await agent.remote_toolset(rag_toolset.service_id) # Query through agent response = await agent.run("How do I perform trajectory inference?") asyncio.run(main()) Programmatic Direct Usage ~~~~~~~~~~~~~~~~~~~~~~~~~ Use the toolset directly without agents:: async def search_knowledge(): rag = VectorRAGToolSet( name="ml_rag", db_path="./ml_db/ml-frameworks" ) # Get database info info = await rag.get_vector_db_info() print(f"Database: {info['description']}") # Query the database results = await rag.query_vector_db( query="transformer architecture", top_k=5, source="pytorch" # Filter by source ) for result in results: print(f"Score: {result['score']}") print(f"Text: {result['text'][:200]}...") print(f"Source: {result['metadata']['source']}") print("---")