Company policies and project docs end up scattered across a pile of Markdown files, and finding a single sentence means digging for a while. This post covers how to set up local document Q&A: have Claude Code write a script, ask.py, that uses only the Python standard library. It reads your documents, retrieves the most relevant original sentences, and notes which file each one came from. If the documents don't have the answer, it simply says "not found in the documents".
We tested it on 3 policy documents: the script was written in 67 seconds, all 4 questions were answered correctly, and for "gym subsidy," which isn't in the documents, it honestly answered "not found in the documents". The script itself calls no API. For the script-writing step, I connected AICodeWith to Claude Code.

本地文档问答:ask.py 实跑结果
The finished result: ask.py answers 4 questions, and each answer includes the original sentence and its source file. For question 4, which the documents don't cover, the script outputs "not found in the documents."
Set up the environment first: two steps to connect AICodeWith
You only configure this once, and it works for every scenario that follows:
- Install Claude Code: follow the official Claude Code installation docs to install it. In your terminal, type
claude --versionand check that a version number shows up. - Create a Key and edit the global config: after logging in, go to "Key Management" in the console and click "Create Key." Then open the global config file
~/.claude/settings.json, and inenv, fill in the Base URL and Key:
{
"env": {
"ANTHROPIC_BASE_URL": "https://api.aicodewith.ai",
"ANTHROPIC_AUTH_TOKEN": "sk-你的Key"
}
}
Don't add/v1to the end of the Base URL. After editing, restart your terminal and runclaude. If you can chat normally, you're connected. For what each field means and more error handling, see “How to Connect Claude Code to a Third-Party API”.
Step 1: Prepare your docs and have it write the script in one sentence
Create a new folder and put your docs in a docs/ subdirectory (Markdown format), then run this inside the folder: claude. The prompt we used in testing:
docs/ 文件夹里是我们公司的几份制度文档。帮我做一个本地问答小工具:写一个不依赖任何第三方 API Key 的 Python 脚本 ask.py,读取 docs/ 下的 Markdown,按段落切块,用关键词/BM25 检索出最相关的段落,输出答案和出处(文件名+原句)。然后用它回答这 4 个问题并把结果写进 qa-results.md:1. 去上海出差住宿每晚最多报多少?2. 入职 12 年有几天年假?3. 借测试手机找谁、能借多久?4. 公司有没有健身补贴?(文档里没有的要明确说"文档中未找到",不能编)
Be sure to include that last line, "if something isn't in the docs, say clearly that it wasn't found in the docs, and don't make anything up." It decides whether the script forces an answer out of thin air.
Step 2: How the script it writes works
The script that Claude Code wrote, ask.py It uses only the Python standard library, so there's nothing to install:
- It reads
docs/and splits every Markdown file under it into paragraphs, using blank lines and list items as boundaries. - Chinese text is tokenized into two-character groups, and BM25 scores each paragraph.
- It then calculates the share of question keywords that appear in the best-matching paragraph. If that share is below 30%, it outputs "Not found in the docs" outright instead of padding the answer with loosely related entries.
- Output the original sentence and the source file name.
Usage is simple:
python3 ask.py "入职 12 年有几天年假?"
python3 ask.py -k 3 "借测试手机找谁?" # 看前 3 个候选段落
Step 3: Test results
| Question | Sentence returned by the script | Source | Final answer |
|---|---|---|---|
| What is the maximum nightly lodging reimbursement for a business trip to Shanghai? | Travel lodging: up to 600 yuan per night in tier-one cities, 400 yuan in other cities. | 报销制度.md | 600 yuan |
| How many days of annual leave do I get after 12 years with the company? | Annual leave: 5 days after 1 year of service, 10 days after 10 years, 15 days after 20 years. | 请假制度.md | 10 days |
| Who do I ask to borrow a test phone, and for how long? | Register with Wang Lei in IT to borrow a test phone. The maximum loan period is 14 days. | 设备借用.md | Ask Wang Lei in IT, 14 days max |
| Does the company offer a gym subsidy? | Not found in the documents | — | Not found in the documents |
What the script does and what Claude infers
It's worth being clear on this point:ask.py only does retrieval. It returns the original sentences from your documents and does no reasoning.
- For question 1, the source text says "first-tier city" and never mentions Shanghai. The step "Shanghai counts as a first-tier city, so it's 600 yuan" is Claude's inference. To make "Shanghai" match that sentence, Claude also added a small synonym table to the script that maps Beijing, Shanghai, Guangzhou, and Shenzhen to "first-tier city".
- For question 2, the script returns the full annual leave sentence. The step "12 years of service is at least 10 but under 20, so it's 10 days" is also Claude's inference, which it wrote into
qa-results.md.
When you run ask.py on its own, you get the original sentences and their sources, and you read them to reach a conclusion yourself. If you want an answer that states the conclusion directly, ask in Claude Code. Have it call ask.py first, then answer based on the original sentences, and tell it to note which parts are inference.
Common errors and self-checks
| Symptom | What to do |
|---|---|
| The answer is in the documents, but it returns "Not found in the documents" | The wording in your question doesn't match the documents (for example, "Shanghai" vs. "first-tier city"). Ask Claude to add the corresponding terms to the synonym table |
| It returns the wrong passage | Add -k 3 to see the top candidates, or ask Claude to change how the documents are split into passages |
| 401 or request failure | Check that the Key is complete and that the Base URL is https://api.aicodewith.ai (without /v1). Reopen the terminal after making changes |
FAQ
Does local document Q&A upload my documents to an external service?
ask.py reads files and runs retrieval locally and never calls any API. When you use Claude Code to write the script and have it read the results, the relevant content is sent to the model, so check your company's policy first if the documents are sensitive.
Do I need to install a vector database or other dependencies?
No. In our test, the script used only the Python standard library, so you can copy it to any computer with Python 3 and run it.
How long does it take?
In our test, going from writing the script to answering all 4 questions took 67 seconds. After that, running ask.py on its own is purely local computation and doesn't call the model again.
Does it support PDF or Word?
We tested with Markdown. For other formats, convert them to Markdown or plain text first, put them in docs/, and then use the same script.
Does it still work with a lot of documents?
You can have Claude adjust the passage splitting and retrieval for your document volume. Our test used 3 small documents. For large collections, try a few questions first to check that the answers are correct.
Related reading
- To make "every answer must cite its source" a standing rule, see "How to Write CLAUDE.md for Claude Code".
- To turn Q&A results into a report, see "Making Slides with Claude".
- If you're not sure which model to use, see "Which Model to Use with Claude Code".
Sign up for AICodeWith → View pricing →
References
- Claude Code installation: code.claude.com/docs/en/setup
- Introduction to the BM25 algorithm: en.wikipedia.org/wiki/Okapi_BM25
Similar scenarios
For more ways to build a local knowledge base with AI, see the AI local knowledge base scenario page.



