arXiv RAG — Retrieval that keeps itself current
A RAG system that ingests new papers on its own, runs hybrid retrieval with local models, and costs nothing to operate.
The problem
Most RAG demos index a corpus once and freeze. The retrieval works, the answers cite sources, and six months later everything it knows is stale. The hard part of a production RAG isn’t the retrieval — it’s staying current without anyone maintaining it.
I wanted a system where new arXiv papers show up in the index on their own, and where the answers are grounded enough that “I don’t have that information” is a valid response.
Architecture decisions
Incremental ingestion, not periodic reindexing. A state table tracks every paper by id and content hash. Each run asks arXiv what’s new, compares hashes, and skips anything unchanged. Hashing costs microseconds; embedding costs seconds. Putting the cheap check before the expensive work is what makes “incremental” mean something.
Chunk deletion and state update share one transaction. When a paper is revised, its old chunks are deleted and the new ones inserted in the same transaction as the state update. There is no window where both versions exist in the index. This is the bug that silently poisons retrieval quality, and it’s the reason the vector store is Postgres and not a dedicated vector database — I wanted real transactions across chunks and ingestion state.
Categories live in a table, not in code. Adding cs.IR to the tracked set is an INSERT. No redeploy, no code change. Anything that changes more often than the logic doesn’t belong in the logic.
Hybrid retrieval, then rerank. Dense vector search and Postgres full-text run in parallel and fuse with Reciprocal Rank Fusion — ranks, not scores, because cosine similarity and ts_rank live on incomparable scales. The fused top 20 goes through a cross-encoder that keeps 5. Recall first, precision second.
Three processes, one database. Ingestion, the web API, and the Telegram bot never talk to each other — only to Postgres. Any one can fail without taking the others down. The query pipeline is a single shared module; web and Telegram are entry points, not implementations.
The interesting trade-off
The VPS had 3.2 GB of headroom. bge-m3 and its matching reranker wanted 2.7 GB of that, which meant living one traffic spike away from an OOM kill. I swapped both for 384-dimension small models — the entire ML footprint dropped to roughly 400 MB, idle memory to 466 MB.
Benchmarks say I gave up a few points of retrieval quality. On English abstracts the difference is invisible, and I traded it for a demo that doesn’t fall over when someone opens it. A portfolio piece that’s down is worth less than one that’s slightly less precise.
The generation layer runs on OpenRouter’s free tier through its auto-router rather than a pinned model id. Free models rotate and disappear; hardcoding one guarantees an outage on a schedule I don’t control.
Result
Live at rag.leanalvarez.com. Ingestion runs every six hours across cs.CL, cs.IR and cs.AI, catches up in about three seconds when there’s nothing new, and has never needed manual intervention. Embeddings and reranking run locally; generation runs on a free tier. Operating cost is zero.
Ask it something outside the corpus and it says so.