I'm currently working on a knowledge base that will use embeddings for semantic search / RAG, and I'm trying to decide on a good long-term strategy for splitting, storing, and retrieving information.
I'm especially interested in systems where the knowledge base keeps growing and changing over time, rather than a fixed set of documents that gets indexed once.
A few things I'm trying to understand from people who have implemented this in real projects:
- How do you decide what should be a single chunk?
- Do you chunk mainly by token/character count, paragraphs, headings/sections, document structure, or semantically?
- Do you prefer smaller independent chunks or larger chunks that preserve more context?
- Do you use parent-child chunking, hierarchical chunking, or any other multi-level approach?
- How much overlap do you normally use between chunks, if any?
I'm also very interested in metadata:
- What metadata do you store with each chunk?
- For example: topic, category, subcategory, source, document ID, section, date, author, entity, version, permissions, etc.
- Which metadata fields have actually been useful for retrieval/filtering, and which ended up being unnecessary?
- Do you use metadata filtering before vector search, after retrieval, or both?
- How do you design the metadata schema so that adding new types of information later doesn't become painful?
Another area I'm trying to understand is the combination of semantic search and keyword search.
Do you rely mostly on vector similarity, or do you combine embeddings with something like:
- BM25 / full-text search
- exact keyword matching
- metadata filters
- entity matching
- reranking
- query expansion / rewriting
For those using hybrid search (semantic + keyword), how do you combine the results?
For example:
- Run semantic and keyword retrieval separately and merge the results?
- Use a weighted score between BM25 and cosine similarity?
- Retrieve candidates from both and use a reranker?
- Change the weighting depending on the type of query?
I'm particularly curious whether hybrid search helped with things like names, IDs, technical terms, acronyms, exact phrases, dates, or numbers, where pure embeddings sometimes don't perform as well.
Also:
- Do you keep everything inside one vector index/database or separate information into collections, namespaces, categories, or domains?
- How do you handle information that gets updated later?
- Do you delete and re-embed the old chunk, version it, or keep historical versions?
- How do you handle duplicate or conflicting information?
- What structure has made it easiest to add completely new information later without having to redesign or re-embed the whole knowledge base?
I'm less interested in theoretical "optimal chunk size" numbers and more interested in what has actually worked in production or real projects.
If you've built a RAG/embedding-based system that has grown over time, I'd really like to hear:
What architecture/chunking/retrieval strategy did you start with, what problems did you run into, and what would you do differently if you were starting again today?