“AI will replace researchers” says one headline. “AI is useless and makes things up” says another.
Both are wrong, and the reason most people hold one of these extreme views is that they don’t have a working mental model of what AI tools actually are.
You don’t need a computer science degree here. You need 5 concepts. Once you have them, the capabilities and the limitations of AI tools will make intuitive sense — and you’ll stop being surprised by both.
Concept 1: LLMs Are Autocomplete, Scaled Up Vastly
A Large Language Model (LLM) — the technology behind ChatGPT, Claude, Gemini, and similar tools — was trained by reading an enormous amount of text and learning to predict: given this text, what word (token) comes next?
That’s not a dismissal. That simple learning objective, applied to hundreds of billions of examples, produces something that can:
- Explain a concept clearly
- Identify logical inconsistencies in an argument
- Rewrite a paragraph in a different tone
- Translate between languages
What it cannot do is look something up in real-time (unless it has a search tool), access unpublished information, or remember your last conversation (unless memory is enabled).
The mental model to hold
Concept 2: Training Data Has a Cutoff — And Biases
LLMs are trained on a snapshot of the internet and digitised text up to a certain date. After that date: nothing.
ChatGPT-4’s training data cuts off in early 2024. Ask it about a paper published in late 2024 and it genuinely cannot know it exists.
More importantly: the training data reflects the distribution of text on the internet. Topics that are well-documented in English, discussed in mainstream publications, and present in digitised text are well-represented. Topics that are:
- Primarily documented in non-English languages
- From disciplines with small online footprints
- Recent, niche, or contested
- Primarily in unpublished grey literature
…are poorly represented. AI tools will be less reliable on these topics and may confidently produce plausible-sounding wrong information.
Concept 3: Hallucination Is a Feature, Not a Bug
This term frightens people but it’s entirely predictable once you understand Concept 1.
If an LLM is producing text that sounds like a research paper citation, its training has taught it how a citation is structured: author, year, title, journal. It will produce something in that format. If the specific citation doesn’t exist in training data, it will produce a citation that fits the pattern — i.e., a plausible-looking fabrication.
This is not lying. It’s pattern completion without grounding.
The practical implication: never use AI-generated citations without verifying them independently. Use a database like Semantic Scholar, PubMed, or Google Scholar to confirm a paper exists and that the AI’s description of it is accurate.
Where hallucination risk is low
AI is much less likely to hallucinate when:
- You provide the source and ask it to work with that content
- The task is structural (reformat, rewrite, summarise text you provide)
- You’re asking about well-established, broadly documented concepts
The risk is high when you ask it to retrieve specific facts, cite specific sources, or describe recent events.
Concept 4: Your Prompt Is the Only Thing the Model Has
Unlike a search engine that understands what you probably want, an LLM responds to exactly what you wrote (plus however its system prompt configures it).
This has two implications:
Vague prompts produce vague output. “Tell me about epistemology” → generic summary. “Explain the difference between foundationalism and coherentism as it applies to the philosophy of social science, suitable for a first-year PhD seminar” → targeted, useful response.
Context in the prompt dramatically improves quality. If you paste in your research question, your methodology, and three example papers you think are high-quality, the model’s response will be calibrated to your actual context rather than its average training distribution.
A practical prompt structure for researchers:
Context: [Your field, your level, the specific paper/project you're working on]
Task: [What you want the AI to do]
Constraints: [Tone, length, what to avoid]
Output format: [Bullet points, prose, table, etc.]
Concept 5: AI Tools Are Not Peers — They Are Sophisticated Assistants
The biggest misuse of AI in research I see is treating the model’s output as a conclusion rather than a starting point.
An LLM can help you:
- Draft a structure for an argument
- Identify potential counterarguments you hadn’t considered
- Translate complex text into plain language
- Surface themes across text you provide
- Rewrite a passage more clearly
It cannot:
- Evaluate whether a methodology is appropriate for your research question
- Judge whether the interpretation of your results is valid
- Ensure your literature review is comprehensive
- Verify that a citation is accurate
- Understand the tacit knowledge of your discipline
Your job as a researcher is to set the task, evaluate the output, provide domain expertise, and be accountable for what you publish. The AI is a tool in that process.
Putting It Together: A Practical Classification of AI Tasks
| Task | AI Reliability | What to Do |
|---|---|---|
| Drafting/rewriting my own text | High | Use freely, edit output |
| Explaining a concept I already know | High | Good check for clarity |
| Summarising text I provide | High | Verify key claims |
| Generating citations | Low | Always independently verify |
| Describing recent publications | Low | Cross-reference databases |
| Evaluating my methodology | Medium | Treat as brainstorm, not verdict |
| Translating language | Medium-High | Review, especially technical terms |
Where to Learn More
If this sparked curiosity and you want to go deeper without a CS degree, these resources are genuinely accessible:
- 3Blue1Brown’s “Neural Networks” series — visual, no maths required
- Andrej Karpathy’s “Intro to LLMs” (YouTube) — 1 hour, remarkably clear
- “Calling Bullshit” by Carl Bergstrom and Jevin West — critical thinking framework applicable to AI claims
Next in this series: How to Write Better AI Prompts for Academic Research →