Designing an AI Course Assistant That Cites Its Sources
How to build a course or policy assistant students can trust: retrieval over approved documents, answers with citations, refusing when unsure, privacy-aware hosting and an evaluation set that keeps it accurate.
Students ask the same questions every semester: deadlines, late penalties, whether they may use AI tools, how to apply for an extension. An AI assistant can answer at 2 a.m. — but only if it answers from your rules, shows where the answer came from, and admits when it does not know.
Try the retrieval step
The demo below runs entirely in your browser. It searches a sample course handbook, picks the best passage and cites the section. Ask “Can I use ChatGPT for my essay?” — then ask about car parking and see it decline to guess.
This demo shows the retrieval-and-citation step. In production we add a language model hosted in your approved environment to phrase answers — always grounded in, and linked to, the cited passages.
The pattern: retrieve, then answer
This approach is often called retrieval-augmented generation (RAG):
- Collect approved sources — handbook, assessment briefs, policies, FAQs. Nothing else.
- Split them into passages that keep section numbers and titles.
- Retrieve the most relevant passages for each question (keyword search, vector search, or both).
- Answer from those passages only, using a language model instructed to quote and cite.
- Refuse when the retrieved passages don’t support an answer, and point to a human contact.
A citation lets a student verify the answer and lets staff spot errors quickly. Make every claim link back to a section, and show the passage on click.
Design decisions that matter
Source control. Keep the document set small and owned. When the handbook changes, re-index it — and keep a version history so you know what the assistant knew on a given date.
Hybrid retrieval. Keyword search handles exact terms (“APA 7th”, “Room 9.21”); vector search handles paraphrases (“I’m sick before the deadline”). Combining both is more robust than either alone.
Refusal threshold. Tune a minimum relevance score below which the assistant says it cannot find the answer. Wrong-but-confident is the failure that destroys trust.
Model and hosting. Choose a model and hosting arrangement that matches your institution’s data policy — an approved cloud service with appropriate data terms, a university-hosted gateway, or an open-weight model on your own infrastructure. Avoid sending student personal data to the assistant at all where possible.
Keep it accurate with an evaluation set
Before launch, write 50–100 real questions with the expected answer and section. Run them automatically whenever documents, prompts or models change.
| Check | What it catches |
|---|---|
| Correct section retrieved | Search or chunking problems |
| Answer matches the source | Hallucination or paraphrase errors |
| Refuses out-of-scope questions | Over-confident guessing |
| Same answer in English and Chinese | Language gaps |
Track the pass rate over time. If it drops after a model update, you’ll know before students do.
Rolling out responsibly
- Tell users what it is: an assistant that answers from the course handbook, not a tutor or a decision-maker.
- Keep humans in the loop for anything consequential, such as extensions or misconduct.
- Review logs (without personal identifiers) in the first weeks to find gaps in the documentation itself — often the biggest win.
- Agree on data retention for questions asked.
Interested in an assistant for your course, programme or student services? See our AI services or ask us for a short scoping call.
Planning a project like this?
Gigantic Work builds web platforms, research portals, AI tools, Unity/VR experiences and cloud infrastructure for Hong Kong universities, grant-funded projects, NGOs and SMEs — and hands over everything at the end.