EPPS Math and Coding Camp
AI as Assistant
A typical research workflow
- Research question
- Literature review
- Theoretical framework
- Data collection
- Methodology
- Analysis
- Interpretation
- Conclusion and discussion …
What AI can do for you?
- Safe zone
- Danger zone, requires further human validation
- No go zone, might violate academic integrity
Safe zone
- Brain storming
- Writing and fixing errors in non-production codes
- Generating simple plots and tables
- Simple data collection
Example: let AI download tables for you
Danger zone, requires further human validation
Writing and fixing errors in production codes
Why this can be dangerous?
- Pattern matching
- Scope creep
- Task substitution
- Ignoring conventions
- Over-engineering: creating unnecessary abstraction, long method chaining, etc.
- Under-engineering: symptom patching, routing, short-cutting
- Obscure naming
What it does well
- Pattern completion
- Translation between natural language and code
- Local code refactoring
- Generating alternatives and first drafts
- Feedback-driven iteration
What it is bad at
- Establishing what is true
- Inferring unstated requirements
- Reasoning across an entire system
- Making trade-offs between competing objectives
- Verifying its own work reliably
- Maintaining intent across long, multi-step tasks
Collet and manage Literatures
Root Issue: Frontier AI models are trained on vast majority of the human knowledge, not specific to your domain.
Literature collection vs literature review
- Literature collection is a retrieval task for AI
- Literature review is a contextual understanding and synthesis task for AI
- Contextual understanding: AI are trained on general knowledge, not prioritized for your domain
- Synthesis: AI may not know what to leave out and what to include
You are the domain expert, you need to know these.
Why this can be dangerous?
- Citation hallucination
- Coverage expansion or shrinkage
- Position bias, bias towards popular results and ignoring niche results
- Recency bias
What it does well
- Query expansion
- Candidate discovery
- Metadata extraction and normalization
- Deduplication, tagging, and file organization
- Search-update monitoring
What it is bad at
- Guaranteeing complete coverage
- Deciding relevance, quality, or eligibility
- Distinguishing terminology and concepts that might be niche in your domain
- Extracting what are important for your research
Assisting some research methods and measurement
Root Issue: AI has growing capabilities for solving mathematical, logical, and statistical problems, but might not be there yet.
Why this can be dangerous?
- Method mis-specification
- Assumption blindness
- Bias amplification
What it does well
- Generate survey questions and options
- Generate vignettes and scenarios
- Help creating codebooks
- Explain formula and equations
- Method and analysis option generation
- Descriptive statistics
- Simple data analysis and visualization
What it might be still bad at (may improve over time)
- Choosing the final study design, estimand, and model
- Diagnosing and deciding the model assumptions with underlying domain knowledge
- Independently implement a complex causal inference analyses
Interpreting results
What AI read in your results?
- Tables
- Text
- Figures
Root Issue: Not all models are good at reading plots.
Why this can be dangerous?
- Picking a model that cannot handle plots reliably
- Vibe check
- Statistical dichotomization
- Overgeneralization
- Hedging
What it does well
- Explain a clean table of means, percentages, or regression estimates
- Describe an obvious pattern in a simple bar, line, or scatter plot
What it is bad at
- Explain what a reported effect size, confidence interval, or test statistic represents
What it is bad at:
- Distinguishing groups or cohorts by color or pattern
- Resolving conflicts between a table, figure, and written result
- Deciding whether a result generalizes beyond the study sample
- Apply the interpretation in the domain context, rather than just restating the numbers
Example of Usage
Prompt A: “This function is broken, can you clean it up and make it better?”
Prompt B: “Fix only the error in the for loop on line XX. Keep the existing variable names and style. Do not change anything else.”
Which prompt is better?
Prompt A: “Search my library folder for papers on social media and political polarization, and list them in APSA format, sort by publication years.”
Prompt B: “Find me ten papers that prove social media causes political polarization.”
Which prompt is better?
Given input prompt: “Fix the bug where the function crashes when the this field is empty.”
Response A: “I found the crash and fixed it. While I was in there, I also refactored the authentication module to use a cleaner session handling pattern and added input validation across all the other forms.”
Response B: “The crash happened because the code called .strip() on the password before checking if it was empty. I added a check for empty or missing passwords before that line. I did not change anything else.”
Which response is better given the prompt?
No go zone
- Write papers
- Creating data (not the same as data collection or data generation)
- Replacing human participants (without declaring and justifying the “silicone sample”)
- Upload protected data or other materials to publicly available AI tools
- Concealing AI assisted works
- Letting AI certify its own claims
