Automated Keyword Research and AI Clustering, End to End

By Irina Saprykina, working in SEO and digital marketing since 2006.

Manual keyword research and clustering could swallow a whole day. In this article I'll show how I automate keyword research almost end to end – using Claude Code to write the scripts, a keyword-research API for the data, Python to tie it together, and my own rules for how the grouping should actually work. One run costs from $0.09 in API credits and about 15 minutes.

Below you'll find the whole keyword research workflow in detail – essentially how to automate keyword research yourself – plus a step-by-step setup plan you can follow even if you've never written a line of Python.

Why Automate It at All

Keyword research isn't hard so much as it is repetitive and rule-bound. I have a fairly fixed way of deciding what belongs in a group, what's a separate intent, and what to throw out. That's exactly the kind of judgement you can write down once and reuse, instead of re-explaining it to a person (or to yourself) every time. The goal was never to remove my judgement from the loop, only to stop spending it on the mechanical 80%.

How the Core Pipeline Works

The main flow is five steps, and only the first touches a paid API:

1. I give it seeds and a region. Seed keywords and a location, and optionally negative keywords and other filters.
2. The Python script pulls related keywords. Up to 1,000 of them from DataForSEO, each with search volume, competition and a 12-month trend.
3. The script drafts the clusters. A first-pass grouping (no API cost), saved as a JSON file plus a prompt for Claude to refine the groups according to my rules.
4. Claude refines the groups. It asks clarifying questions if anything is ambiguous, then returns a cleaned-up JSON, which goes back into the Python script.
5. I get grouped keywords in Google Sheets. Organised by topic, with volumes, competition and the past 12 months of trend for each keyword.

The split matters: Python does the fetching, deduping and the boring first draft; Claude does the semantic judgement; Python does the writing-up. Each tool does the part it's actually good at, and the expensive API call happens exactly once per run.

A tip if you're still tuning your grouping rules: have the script cache the pulled keywords to a local file. Testing a different grouping prompt then means re-running only the grouping step, so you're not charged again for the same keywords every time you restart the script.

Keyword Clustering: The Prompt Does the Real Work

Of the five steps, one deserves singling out, because it's where good clusters are won or lost: the rules I hand Claude in step 4. The Python first pass only buckets keywords crudely. The real call – does running shoes for flat feet belong with best running shoes, or is it a separate search intent? – comes down to how precisely I've written the keyword grouping instructions.

That prompt is the heart of the whole workflow. It's where my domain judgement lives, written down once: what counts as a separate intent, when a modifier splits a group and when it doesn't, what to discard outright, how large a group is allowed to get. Vague rules give you a tidy-looking sheet that's subtly wrong; specific rules give you groups you can actually build pages around. The API pulls the data, but the prompt decides whether the grouping is any good – so that's where I spend my time.

Two Scripts for Going Deeper

Grouped keywords are a map, not the whole job. Two more scripts help me decide how to treat specific keywords and what the winning pages actually look like.

SERP Overlap Check

I usually take the most popular keywords from the main groups and run this script to compare how much their search results overlap. If two keywords return largely the same pages, Google treats them as one intent and they belong in one group; if the results diverge, they should be split. The overlap check is the tie-breaker for the cases where semantics alone don't settle whether two keywords are really the same search. With the overlap data in hand, you can go back to Claude and ask it to re-sort the keywords, taking the new overlap information into account.

Competitor Keyword Analysis

For a few chosen keywords, this script reads the SERP and then, for the top-ranking pages, digs into what the competitors are doing. It returns two things:

• Their other keywords. The rest of the keywords those top pages also rank for.
• Their on-page language. Which relevant phrases they use, and how those phrases are distributed across the page.

The result is a picture of both the demand (what else these pages capture) and the execution (how they actually phrase it on the page).

By the end I have keyword groups organised by topic, enriched with competitors' keywords and the language they use on their winning pages.

Keywords for AI Search: Fan-Out Queries

Everything above optimises for what people type into Google. But AI search doesn't run your query once: when you put a question to an assistant like ChatGPT, Gemini or Perplexity, the model quietly expands it into a set of fan-out queries – related sub-questions it retrieves pages for before writing its answer.

Those fan-out queries are a keyword layer of their own – not just long-tail keywords – and most don't exist in any classic tool: AirOps found roughly 95% have no search volume at all. Covering this layer is the keyword-research side of generative engine optimization (GEO) – optimising to be surfaced and cited by AI engines, not just ranked by Google. (Why fan-out coverage matters for getting cited is a separate topic, covered in my AI citation ranking factors guide.)

DataForSEO's AI Optimization API surfaces this layer two ways, and I feed both into the same pipeline:

• LLM Responses. Send a prompt to ChatGPT, Claude, Gemini or Perplexity through one endpoint, and the response comes back with a fan_out_queries field – the actual sub-questions each engine broke your topic into.
• AI Keyword Search Volume. Estimates how often each fan-out gets used in AI search, up to 1,000 keywords per request with a 12-month trend – so you can rank a long, mostly-zero-volume list. It's a proprietary index that leans on Google's "People Also Ask" questions as a proxy, on the logic that PAA captures the same conversational, question-shaped phrasing people put to an assistant.

The fan-out layer slots into the main flow: seed prompts → LLM Responses → dedupe → AI Keyword Search Volume to rank → the same clustering step. Classic keyword research tells you what to rank for; this tells you what the AI goes looking for once it's answering.

What It Costs (July 2026 Prices)

Here's the current per-run cost in Live mode, reflecting DataForSEO's July 2026 price update. All of these are pay-as-you-go – you're billed per request, not by subscription.

So, in practice:

• Core run (grouped keywords in Sheets): about $0.09 – just the related-keywords pull, since Claude does the clustering for free.
• Fuller run (plus competitor analysis and a few SERP-overlap checks): around $0.20–0.25.
• Fan-out layer: a few cents per seed prompt on top.
• Claude: from $20/month (the Pro plan) – but this is only one of many tasks I run it for, so it hardly counts as a cost of this workflow alone.
• Setup: several hours of testing, but a one-time cost.

Not bad for what used to take the better part of a day.

From Zero: Build This Yourself with Claude Code

Here's the part that trips people up: you don't need to know how to code. I lean on Claude Code – Anthropic's terminal agent – to actually write the Python. My job is to describe the workflow, hand over my rules, and test what comes back. If you've never opened a terminal, this is roughly an afternoon of setup; after that, changes are just conversation.

The order below is the one I'd follow from a clean machine.

1. Install Python

The scripts run on Python, so install it first from python.org.

• Windows: run the installer and tick "Add Python to PATH" on the first screen – this one checkbox saves a lot of grief later.
• macOS: the python.org installer is the simplest route; Homebrew (brew install python) works too if you already have it.

Check it worked by opening a terminal (PowerShell on Windows, Terminal on macOS) and running python --version – it should print a 3.x version. If it doesn't, Claude Code can diagnose the PATH problem for you in the next step.

2. Install Claude Code

Claude Code needs a paid Claude plan – Pro or Max (the free Claude.ai tier doesn't include it). Then install it one of two ways:

• Terminal (recommended).
• Windows PowerShell: irm https://claude.ai/install.ps1 | iex
• macOS / Linux: curl -fsSL https://claude.ai/install.sh | bash
• Prefer not to touch the terminal? Install the Claude Code desktop app instead – same agent, graphical interface.

Then make a new empty folder for the project, open a terminal inside it, and run claude. On first launch it opens your browser to log in with your Claude account – no separate API key to manage, since Claude Code bills against your subscription. From here on, you mostly talk to it: "install the libraries you need," "create a .env file for my secrets," and so on. If Python from step 1 isn't found, tell Claude Code and let it walk you through fixing PATH.

3. Get your DataForSEO credentials

The keyword and AI data come from DataForSEO, which is pay-as-you-go with a $50 minimum top-up (that credit lasts a long time at these prices). Sign up, add funds, and copy your API login and password from the dashboard.

Don't paste those into the script. Ask Claude Code to keep them in a .env file (a plain text file that stays on your machine and out of your code), like:

DATAFORSEO_LOGIN=your_login_here
DATAFORSEO_PASSWORD=your_password_here

Tell Claude Code to read the keys from .env and to add that file to .gitignore so you can never accidentally share it. You don't need a separate Anthropic API key for this – Claude Code already covers the Claude side.

4. Give the scripts access to Google Sheets

This is the fiddliest part, because Google makes a script prove it's allowed to touch your sheets. The mechanism is a service account – a robot Google user with its own email address that you explicitly invite to your spreadsheet. Walk through it once:

1. Go to the Google Cloud Console and create a new project.
2. Enable two APIs for it: Google Sheets API and Google Drive API.
3. Create a service account, then create a key for it and download the JSON file. Save it in your project folder (it's usually named something like credentials.json).
4. Open that JSON file and copy the client_email – it looks like something@your-project.iam.gserviceaccount.com.
5. Open the Google Sheet you want the script to write to, hit Share, paste that email, and give it Editor access. This is the step everyone forgets – without it the script gets a "permission denied" even though the code is fine.

Then tell Claude Code you have credentials.json in the folder and want to write results to a specific Google Sheet; it'll wire up the gspread library and the rest. Treat credentials.json like a password – add it to .gitignore too.

5. Let Claude Code build the pipeline

Now describe what you want, in plain language. The prompts that worked for me were specific about both the data and the rules:

• "Fetch related keywords from the DataForSEO Google Ads Keywords For Keywords live endpoint for these seeds and this location, up to 1,000, with search volume, competition and the 12-month trend."
• "Do a first-pass clustering in Python, then export a JSON plus a prompt so I can refine the groups. Here are my grouping rules: …"
• "Write the final groups to my Google Sheet, one tab, with volumes and trends."
• Housekeeping that pays off: "print the actual API cost after each run," "add error handling and retries," "cache the raw API response so I can re-run the clustering for free."

Expect to go back and forth. You'll test on a small seed, spot something wrong, and describe the fix – that conversation is the development. Start cheap: a single seed keyword is a few cents, so you can iterate a lot before it adds up.

One optional shortcut worth knowing: DataForSEO publishes an MCP server, which you can connect to Claude Code so it can call the DataForSEO API directly while it builds and debugs, instead of you copy-pasting sample responses. It's not required, but it makes the build loop noticeably faster.

None of this needs prior coding skill – it needs a clear description of what you want and the patience to test. The scripts are the easy part now; the judgement about what belongs in a group is still yours, which is exactly as it should be.

Other Keyword APIs in DataForSEO Worth Knowing

My workflow only touches a handful of endpoints, but DataForSEO has more keyword-related APIs you could bolt on. A few I keep coming back to (all pay-as-you-go, most of the DataForSEO Labs ones at roughly $0.01 per task plus $0.0001 per keyword returned):

• Bulk Keyword Difficulty API – a keyword-difficulty score from 0 to 100 (on a logarithmic scale) for your odds of reaching the top-10 organic results, up to 1,000 keywords at a time. Handy for triaging which keywords in a group are actually realistic to target.
• Search Intent API – auto-classifies each keyword as informational, navigational, commercial or transactional, with a probability score, up to 1,000 per request. It's a machine version of the intent call my clustering prompt makes by hand.
• Keyword Suggestions and Keyword Ideas APIs – two more ways to expand a seed. Suggestions returns long-tail phrases that contain your seed keyword; Ideas returns keywords from the same product or service category that may not contain the seed word at all.
• Historical Keyword Data API – past search-volume history for a keyword, useful for spotting seasonality or a term that's quietly declining.
• Rank-checking APIs – the SERP API (Google Organic) returns the live results, and any domain's position, for a keyword; in DataForSEO Labs, Ranked Keywords lists everything a domain already ranks for and Historical Rank Overview tracks how those positions move over time.
• SERP Competitors API – the domains that show up most often across a set of your keywords, so you can see who you're really competing with before you write a line.

None of these extra APIs are required for the pipeline above; they're just the pieces I reach for when a project needs more than grouped keywords.

Am I Fully Satisfied?

During the build I went through the full range of opinions – from "this is impossible" to "this is almost perfect." My current version works well, though it isn't flawless. I tested it on different kinds of keywords:

• Small and medium groups: it gets almost everything right.
• Very wide groups: the results get messier. But even then it hands me the full map, and I can zoom into the narrow groups and work them closely – which is far faster than starting from a blank sheet.

The honest framing is the one I keep landing on with these experiments: automation doesn't replace the judgement, it removes the repetition around it. I still make the calls on the wide, ambiguous groups. I just don't spend my day getting there anymore.

About the Author

I'm Irina Saprykina, and I've worked in SEO and digital marketing since 2006, mostly in-house, promoting international SaaS companies. I'm a mathematician by training, which is probably why I'd rather test and automate a process than take a claim on faith. datairi is where I publish those experiments and the tools that come out of them.