Build a Chat-With-Your-Docs App That Survives a Doc Update

Key takeaway: Every chat-with-your-docs tutorial stops at the first upload. Interviews start at the second one. Your index is a copy of the documents, and it goes stale on four kinds of change: an edit, a delete, a rename, and a cached answer. Build the app with a stable document ID, a content hash, and a sync loop that deletes as well as inserts, then prove each case with a test. The decision this post helps you make: which RAG project to build this weekend, and what it must survive before you put it on your CV.

Most RAG project ideas lists end with "upload PDFs, ask questions". That demo is a commodity now. You can find it listed as a fixed-price job on freelance marketplaces.

The gig version can't answer the question interviewers ask next: "what happens when the document changes?" This post is one build that can: the scope, the schema, the sync loop, the four changes that break a naive app, and a five-minute demo that shows all of it.

Previous post: AI Engineer Interview Guide 2026: Build These 5 Apps.

What this post covers

• What a chat-with-your-docs app really is, and why it goes stale
• The build: scope, stack, and the two tables that matter
• Four changes that break a naive app, and the fix for each
• The sync loop, in about 25 lines
• The questions the interviewer will ask, with answer skeletons
• A worked example and a five-minute demo script

A chat-with-your-docs app is a copy of your documents

What it literally is. Retrieval-augmented generation (RAG) fetches passages from a store and puts them in the model's prompt before it answers. Lewis et al. (2020) named the pattern. One of their selling points was that you could update what the model knows by swapping the index instead of retraining.

Where it sits. That index is a derived copy. You parse the source, split it into chunks, embed them, and store them. The source keeps living after that. A wiki page gets edited. A PDF gets replaced. A folder gets cleaned up. The copy only changes when your code changes it.

Technical explanation: the model answers from whatever chunks retrieval returns. If old chunks are still in the index, they compete with new ones, and nothing in the prompt can tell which is current.

Business explanation: picture an internal IT helpdesk bot. The team retires the old VPN client and deletes its setup guide. A week later an employee asks how to connect from home. The bot walks them through installing software that no longer works, and cites the deleted guide.

The build: a RAG project for one weekend, two tables

Scope. About 100 documents from one real source you can edit: a folder of PDFs and Markdown, or a wiki export. Chat UI optional. A CLI is fine.

Stack. Any parser, any embedding model, and one store that holds vectors and metadata together (Postgres with pgvector works). Add BM25 if you already built hybrid search.

The two tables that matter:

• documents: doc_id, source_path, content_hash, version, ingested_at
• chunks: chunk_id, doc_id, version, text, embedding

The doc_id must survive a rename. Use the source system's ID if it has one (wiki page ID, Drive file ID). For a plain folder, assign an ID on first sight. When a new path shows up with the same hash as a stored document whose old path is missing from this sync's listing, treat it as a move. If the old path is still there, it's a copy and gets its own ID. The file path is metadata, not identity.

Four changes that break a naive app

1. Edit. The naive app appends new chunks and keeps the old ones. Both rank. Fix: on a hash change, delete every chunk for that doc_id, then insert the new version.
2. Delete. The naive app never notices, because it only looks at files that exist. Fix: list the whole source on each sync and remove any doc_id that is missing.
3. Rename or move. If identity is the path, a rename looks like a delete plus a new file. You re-embed for nothing, and if the delete path is missing, you now have duplicates. Fix: identity is doc_id; the path is an updatable field.
4. Cached answer. You added a response cache to cut cost. The index is fresh, but the cache still serves last week's answer. Fix: key cache entries by the doc_id and version of the chunks that produced them, and drop entries when a version changes.

The sync loop

def sync(source, store, cache):
    listing = {i.path: (i, source.read(i))      # full listing first
               for i in source.list_all()}
    seen = set()
    for path, (item, text) in listing.items():
        h = sha256(text)
        doc_id = store.resolve_id(item, h, listing)  # source ID, or a move
        seen.add(doc_id)
        row = store.get_document(doc_id)
        if row and row.content_hash == h:
            store.update_path(doc_id, path)        # unchanged or moved
            continue
        version = (row.version + 1) if row else 1
        chunks = chunk(text)
        vectors = embed([c.text for c in chunks])
        with store.transaction():
            store.delete_chunks(doc_id)            # old version out first
            store.insert_chunks(doc_id, version, chunks, vectors)
            store.upsert_document(doc_id, path, h, version)
        cache.invalidate(doc_id)

    for doc_id in store.all_doc_ids() - seen:      # deletes
        with store.transaction():
            store.delete_chunks(doc_id)
            store.delete_document(doc_id)
        cache.invalidate(doc_id)

Three details interviewers check:

• The delete and insert share a transaction. Otherwise a query between them sees no chunks, or both versions.
• The listing is complete. A partial listing turns into mass deletes. LangChain's indexing API warns about the same thing for its full cleanup mode, and adds scoped_full for loaders that can't return everything at once (API reference).
• Only changed documents are re-embedded. The hash check is what keeps re-sync cheap.

Frameworks cover part of this. LangChain's record manager and LlamaIndex's UPSERTS_AND_DELETE docstore strategy, which matches by document ID and then compares hashes (API reference), handle the edit and delete cases. Both identify documents by a source ID or doc ID that is random or path-derived by default, so rename detection is still yours, and so is cache invalidation. Build the loop by hand once so you can explain where the framework stops.

What the interviewer will ask

1. "How do you detect a change?" Skeleton: content hash, not modified time. Copies and restores change the timestamp without changing the text.
2. "What about deletes?" Skeleton: full listing per sync, set difference, delete by doc_id. Name the risk of a partial listing.
3. "How stale can an answer be?" Skeleton: your sync interval plus the sync's run time. State the number from your own app, then say what would make you move to change events from the source. Mention the edge case: in a plain folder with no source IDs, a doc edited and renamed in the same window gets a new doc_id and a full re-embed, which is correct but not free.
4. "How do you know it still works?" Skeleton: freshness cases in the golden set. Edit a fact, delete a doc, rename a doc, and assert on the answer and the cited doc_id after each sync.

Worked example: Meera and the retired VPN guide

Fictional scenario. Meera is a QA engineer at a large IT services company, interviewing for an AI engineer role. She demos her IT helpdesk bot. The interviewer deletes vpn-setup-old.pdf from the source folder, runs Meera's sync, and asks, "How do I connect to VPN from home?"

The bot answers from the new guide and cites it.

"Okay," the interviewer says. "Now rename the new guide."

Meera renames it. The sync log shows one path update, zero embeddings. The answer is unchanged.

"Why didn't it re-embed?"

"Same doc_id, same hash. Only the path changed," Meera says. "If I'd keyed on the path, that rename would have been a delete plus an insert, and before I fixed my delete path it left two copies of the guide in the index."

Things to note: she showed the delete case without being asked, she knew the cost of the rename (zero embeddings), and she named the bug she had already hit and fixed.

How to demo it in five minutes

1. Ask a question and show the cited doc_id (one minute).
2. Edit a fact in the source, sync, ask again, and show the new answer and version number (one minute).
3. Delete a document, sync, and show the bot no longer cites it (one minute).
4. Rename a document, sync, and show zero re-embeds in the log (one minute).
5. Run the golden set, including the freshness cases (one minute).

TLDR

• Your index is a copy, and it goes stale on edits, deletes, renames, and cached answers.
• Identity is a stable doc_id. The path is metadata. Change detection is a content hash.
• The sync loop deletes old chunks inside the same transaction that inserts new ones, and removes documents that left the source.
• Freshness cases belong in your golden set, next to accuracy cases.
• Build this RAG project this weekend and demo the update path, because that is what the interviewer will test.

Where to start

Build the two tables first, then the sync loop, then the four freshness tests. Do the delete case before anything else, because a naive app gets it wrong without any error.

This is Day 3 of the 30-day series on interview prep and build guides for AI app builders. Follow Flow on X for the rest. Running this sync loop for a dozen sources at work? Inherent's context engine handles managed re-ingestion and invalidation, with a receipt for every answer. DM for a design-partner slot.

Next read: Batch Ingestion Is Why Your RAG Answers Go Stale