EPPS Math and Coding Camp

AI as a research method

Instructor: Xingyuan Zhao

How this differ from AI as an assistant?

Assistant:

  • AI helps your research pipeline, but not in them

Method:

  • AI IS a part of your research pipeline. It can be methodology, or the target being studied, or something else.

How to do it?

  • Using chatbot Graphical User Interface (GUI)
  • Using API
  • Deploy local model

Applications

Data standardization

  • Standardize inconsistent formats, cases, codes, units, and delimiters
  • Detect and repair typos, invalid values, outliers, and inconsistent encodings
  • Impute missing values using related variables or documented external context

Data integration

  • Entity Matching: Link records that refer to the same real-world entity across datasets
  • Schema Matching: Identify semantic correspondences between elements across heterogeneous schemas from different sources

Data enrichment

  • Data Annotation: Assign semantic or structural labels to tables, columns, and cells
  • Data Profiling: Derive structured metadata from semantic datasets

Example:

Given a series of unstandardized dates -> AI models -> Standardized dates

from openai import OpenAI

# Client points at the local vLLM server; api_key is unused but required by the SDK
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

# Raw data: noisy scraped values that need one consistent format
raw_data = [
    "March 3rd, 2021",
    "2021.03.03",
    "03/03/21",
]

# Instruction applied to every input value
prompt_template = (
    "Rewrite the date below as YYYY-MM-DD. "
    "Return only the date, nothing else.\nDate: {value}"
)

# The LLM call: one request per input value
standardized_data = []
for value in raw_data:
    response = client.chat.completions.create(
        model="google/gemma-4-12B-it",
        messages=[{"role": "user", "content": prompt_template.format(value=value)}],
        temperature=0.0,
        max_tokens=64,
    )
    # Returns: pull the reply text out of the response object
    standardized_data.append(response.choices[0].message.content.strip())

Example output

["2021-03-03", "2021-03-03", "2021-03-03"]

Natural language processing (NLP) tasks

  • Sentiment analysis
  • Topic modeling
  • Named entity recognition
  • Relation extraction
  • Argument extraction
  • Semantic similarity

Example: Given a list of event descriptions from the Global Terrorism Database -> AI models -> Victims and aggressors, extracted and labeled

from openai import OpenAI
import json

# Client points at the local vLLM server; api_key is unused but required by the SDK
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

# Raw data: short event descriptions to pull named entities from
events = [
    "Gunmen linked to the Riverlands Front stormed a market in Dalcroft, killing two vendors before being detained by local police.",
    "A bomb attributed to the Kestrel Brigade damaged a bus station in Merrow, injuring three commuters.",
]

# Options: the prompt states the two roles the model must choose between
prompt_template = (
    "Read the event below and extract every named person or group mentioned. "
    "Classify each one as one of two options: victim or aggressor.\n"
    "Event: {event}"
)

# Output format: coerce the reply into this structure; each list only
# accepts strings, so every entity must be sorted into one of the two options
response_format = {"type": "json_schema", "json_schema": {
    "name": "entity_extraction",
    "schema": {
        "type": "object",
        "properties": {
            "victims": {"type": "array", "items": {"type": "string"}},
            "aggressors": {"type": "array", "items": {"type": "string"}},
        },
    },
}}

# The LLM call: one request per event
extracted_entities = []
for event in events:
    response = client.chat.completions.create(
        model="google/gemma-4-12B-it",
        messages=[{"role": "user", "content": prompt_template.format(event=event)}],
        temperature=0.0,
        max_tokens=200,
        response_format=response_format,
    )
    # Returns: parse the JSON reply into its victims and aggressors lists
    parsed = json.loads(response.choices[0].message.content)
    extracted_entities.append(parsed)

Example output

[
    {"victims": [], "aggressors": ["Riverlands Front"]},
    {"victims": [], "aggressors": ["Kestrel Brigade"]}
]

Create a pipeline

You can stack multiple tasks together to create a pipeline.

Example: Given a series of scraped candidate wiki pages -> AI models (clean, then annotate) -> Partisanship and election metadata per candidate

from openai import OpenAI
import json

# Client points at the local vLLM server; api_key is unused but required by the SDK
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

# Raw data: scraped wiki infobox HTML for a series of election candidates
raw_html_pages = [
    "<div class='infobox'><h2>Barack Obama</h2><p>Obama was nominated by the <b>Democratic Party</b> for <b>President of the United States</b> in the <i>2008</i> election and defeated Republican nominee John McCain.</p></div>",
    "<div class='infobox'><h2>John McCain</h2><p>McCain was the <b>Republican Party</b> nominee for <b>President of the United States</b> in the <i>2008</i> election and lost to Democrat Barack Obama.</p></div>",
]

# Stage 1 instruction: clean each scraped page into plain biography text
clean_prompt_template = "Strip the HTML below to plain text. Return only the text.\nHTML: {html}"

# Stage 1: one cleaning call per scraped page
clean_bios = []
for html in raw_html_pages:
    response = client.chat.completions.create(
        model="google/gemma-4-12B-it",
        messages=[{"role": "user", "content": clean_prompt_template.format(html=html)}],
        temperature=0.0,
        max_tokens=200,
    )
    clean_bios.append(response.choices[0].message.content.strip())

# Stage 2 instruction: annotate partisanship and election details from the clean text
annotate_prompt_template = (
    "Read the biography below and report the candidate's party, the office "
    "they sought, the election year, and whether they won.\nBiography: {bio}"
)

# Stage 2 output format: coerce the reply into these four fields
response_format = {"type": "json_schema", "json_schema": {
    "name": "candidate_annotation",
    "schema": {
        "type": "object",
        "properties": {
            "party": {"type": "string"},
            "office": {"type": "string"},
            "election_year": {"type": "string"},
            "outcome": {"type": "string", "enum": ["won", "lost"]},
        },
    },
}}

# Stage 2: one annotation call per cleaned biography, chained from stage 1's output
annotations = []
for bio in clean_bios:
    response = client.chat.completions.create(
        model="google/gemma-4-12B-it",
        messages=[{"role": "user", "content": annotate_prompt_template.format(bio=bio)}],
        temperature=0.0,
        max_tokens=100,
        response_format=response_format,
    )
    # Returns: parse the JSON reply into its four annotated fields
    annotations.append(json.loads(response.choices[0].message.content))

Example output

[
    {"party": "Democratic Party", "office": "President of the United States", "election_year": "2008", "outcome": "won"},
    {"party": "Republican Party", "office": "President of the United States", "election_year": "2008", "outcome": "lost"}
]

Danger zone

  • LLM-as-judge
  • Silicone sample

LLM-as-judge

Why they can be dangerous?

  • Position and order bias
  • Verbosity bias
  • Self-enhancement bias
  • Authority bias
  • Bandwagon-effect bias
  • Sycophancy

Caveats if you have to use it:

  • Define a task-specific rubric and external ground truth before judging; do not let the LLM define quality by itself.
  • Separate answer generation from evaluation; do not use the same model as both generator and judge.
  • Randomize and counterbalance answer order, repeat judgments across permutations, and report instability rather than accepting one verdict.
  • Run task-bias specific robustness audits before deployment: perturb length, identity, citation, and framing while holding substantive content fixed.
  • Validate a sample against independent human ground truth and escalate consequential or uncertain cases to human review.

Silicone sample

Why they can be dangerous?

  • Biased training data: Outputs can underrepresent within-group diversity, flatten variation, and obscure cultural and demographic differences.
  • Biased training data: They reflect what goes into the training data, and the selected pool of humans creating “public writing,” rather than the whole population or what humans do.
  • Guardrail/safety: Anti-harm refusals can be detrimental to simulations.
  • Sycophancy can shift simulations toward user-pleasing, unrealistically agreeable, or prosocial outputs.
  • Proprietary training data: Researchers cannot diagnose which component of pretraining data drives replication success or failure; results may not be reproducible.

Caveats if you have to use it:

  • Robustness permutations audits, such as temperature variation, prompt translation round-trip, context rewording.
  • Use fine-tuned domain specific models, rather than general purpose models.
  • Use prompt engineering, context-rich prompting or researcher defined pipelines to represent latent features or isolate pollutions.

Validation

triangle Human Human AI AI Human->AI Ground Truth Ground Truth Human->Ground Truth

AI outputs are not considered as the ground truth. It is human’s responsibility to validate them.