Using an LLM API as a Data Labeling Tool

What We're Building

A pipeline that labels a large batch of unstructured customer-feedback text with a custom business category ("Billing Issue," "Bug Report," "Feature Request," "Praise," "Other") — categories with no existing labeled training data anywhere, making dsproj2-1's own from-scratch approach genuinely impractical here. The LLM applies the labeling scheme directly, at scale, the moment it can be described in a prompt — then the results get analyzed statistically with this track's own established tools.

Step 1: The Unlabeled Batch

import pandas as pd

feedback = pd.read_csv("customer_feedback.csv")   # one column: raw text, no labels at all
print(feedback.shape)

Step 2: Designing the Labeling Prompt

CATEGORIES = ["Billing Issue", "Bug Report", "Feature Request", "Praise", "Other"]

PROMPT_TEMPLATE = """Classify the following customer feedback into exactly one of these categories: {categories}.

Feedback: "{text}"

Respond with only valid JSON: {{"category": "...", "confidence": "high"|"medium"|"low"}}"""

Step 3: Calling the API — claude-adv1's Own Territory, Applied Minimally

import anthropic, json, time

client = anthropic.Anthropic()   # API key read from environment, per claude-adv1

def label_feedback(text):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=100,
        messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(categories=CATEGORIES, text=text)}],
    )
    time.sleep(0.5)
    return json.loads(response.content[0].text)

The full API surface — streaming, tool use, system prompts — is claude-adv1's own territory in depth. This chapter uses exactly one call shape, deliberately, since the point here is what the API call accomplishes inside a larger pipeline, not the API itself.

Step 4: Running the Batch and Parsing Results

results = []
for text in feedback["text"]:
    try:
        label = label_feedback(text)
        results.append({"text": text, **label})
    except (json.JSONDecodeError, KeyError):
        results.append({"text": text, "category": "parse_error", "confidence": None})

labeled = pd.DataFrame(results)

Same JSON-parsing reflex ds1-3 established, now applied to a live model's own output instead of a file or a conventional API response — and the same defensive try/except instinct this track has used since dsproj1-1's own scraper, since a model occasionally returning malformed JSON is a real, expected possibility, not a bug to be surprised by.

Step 5: Analyzing the Labeled Data — This Track's Own Familiar Tools

import matplotlib.pyplot as plt

category_counts = labeled["category"].value_counts()
category_counts.plot(kind="barh", color="#15803D")
plt.title("Feedback Volume by Category")
plt.savefig("feedback_categories.png")

Nothing new here at all — .value_counts() and a bar chart, the exact ds1-level analysis skill this whole track opened with, applied here to data an LLM helped create rather than data that arrived pre-labeled.

The Honest Trade-Off, Stated Directly

What's Next

Chapter 8: Capstone — An End-to-End ML-Powered Mini App — collection, cleaning, feature engineering, model training/evaluation, and a minimal interface, combined into one real application, with a chapter-attribution table spanning both dsproj1 and dsproj2.