# How to find competitors' keywords in their content and code

> curl their robots.txt, sitemap and headings to get the target list, then add positions with one lookup and join the two.

URL: https://claudegpt.si/news/how-to-find-competitors-keywords-in-content  
Published: 2026-10-26T05:09:24.423Z  
Author: ClaudeGPT Si Newsroom  
Section: Guides  
Topics: Developer tools, Competitor keywords  
Publisher: ClaudeGPT Si

## Key takeaways

- robots.txt usually names the sitemap, and sitemap slugs plus title and H1 tags reveal the phrases a rival targets.
- Content alone shows plans, not positions or traffic.
- seogeoaeo.ai's Competitor Keywords adds positions, volumes and ranking URLs for 35 credits, in the browser or over REST.

Here is how to find competitors' keywords in content they publish, using nothing but a terminal: their sitemap, their robots.txt, and the title and heading tags of their pages. It takes a few commands, and it tells you what a rival is targeting. It cannot tell you what they rank for, so the last section covers the one lookup that adds positions and traffic.

## Step 1: find the sitemap

Start with robots.txt, because it usually names the sitemap:

```bash
curl -s https://rival.example/robots.txt | grep -i '^sitemap:'
```

If nothing comes back, try `/sitemap.xml` and `/sitemap_index.xml` directly. Large sites use an index that points to child sitemaps split by type, often with names like `post-sitemap.xml` or `docs-sitemap.xml`. The blog and docs children are where keyword-led pages live.

## Step 2: turn URLs into phrases

Most teams put the target phrase in the slug. Extract the locations and convert hyphens to spaces:

```bash
curl -s https://rival.example/post-sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
  | awk -F/ '{print $NF ? $NF : $(NF-1)}' \
  | tr '-' ' ' | sort -u > slugs.txt
```

You now have a rough keyword list. Drop dates, IDs and pagination before you read it. Watch for patterns: a run of "x vs y" slugs, a cluster of "how to" guides, or a set of integration pages often shows where the rival is spending its writing time.

## Step 3: read titles and headings

The slug is a hint. The title tag and H1 are the clearer signal, since most SEO teams put the primary phrase in both, and the H2s often hold secondary phrases. A short loop over the pages you care about:

```bash
while read url; do
  html=$(curl -sL "$url")
  echo "== $url"
  echo "$html" | grep -o '<title>[^<]*' | sed 's/<title>//'
  echo "$html" | grep -oE '<h[12][^>]*>[^<]*' | sed -E 's/<h[12][^>]*>/  /'
done < urls.txt
```

This misses headings that contain nested tags or that a page renders with JavaScript, so treat the output as a quick read, not a parser.

## Step 4: spot-check with site:

For a topic that matters to you, search Google with `site:rival.example` plus the topic. It shows which of their pages Google ties to it. Google's documentation says the results are not always exhaustive and large sites should not expect every URL to show up, so this is a sample.

## What this gets you, and what it misses

You have their intentions: the phrases they wrote pages for and the words they chose. That is useful for understanding positioning and naming.

What is missing is the outcome. A rival can publish pages for hundreds of phrases and rank for a small fraction. Your slug list has no positions, no search volumes, no visits. You cannot tell the page that brings them a quarter of their traffic from the one that sits on page six.

## Adding rankings with one lookup

The [seogeoaeo.ai](https://seogeoaeo.ai) Competitor Keywords tool fills that gap. Give it the rival's domain, or any URL on it, and a country. It returns up to 100 keywords the domain ranks for in Google, ordered by estimated traffic, each with position, volume, difficulty, intent, cost per click, estimated visits and the ranking URL. It also lists the pages that bring the domain the most visits, which you can join against your slug list to see which planned pages paid off.

It costs 35 credits per run, and the price shows before the run. It also runs over REST or MCP with a workspace API key, so the same job fits in a script:

```bash
curl -X POST https://seogeoaeo.ai/api/v1/tools/competitor-keywords/runs \
  -H "Authorization: Bearer $SEOGEOAEO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":{"url":"https://rival.example","market":"us"}}'
```

Limits worth knowing: it covers the whole domain, not one folder; one country per run; visits and volumes are estimates from a data provider; and it does not show what a competitor bids on in Google Ads. It also does not track rankings over time, so if you need a history of your own positions, a rank tracker such as Semrush Position Tracking is the right tool for that part.

## Putting the two together

Join your slug list against the ranking URLs. Pages in both are plans that worked. Pages only in the sitemap did not make the top 100 by traffic, which often marks a topic where a better page from you can still compete. Plan pricing starts at $29 a month for 2,000 credits; see [seogeoaeo.ai/pricing](https://seogeoaeo.ai/pricing).

## Frequently asked questions

### Where do I find a competitor's sitemap?

Check the Sitemap line in their robots.txt, or try /sitemap.xml and /sitemap_index.xml.

### Do title tags show what a page ranks for?

They show what it targets. Rankings need keyword data, such as a Competitor Keywords lookup.

## Sources

- [Competitor Keywords: what a site ranks for](https://seogeoaeo.ai/tools/competitor-keywords), seogeoaeo.ai
- [seogeoaeo.ai tool list for agents](https://seogeoaeo.ai/llms-full.txt), seogeoaeo.ai
- [site: search operator](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site), Google Search Central
- [seogeoaeo.ai pricing](https://seogeoaeo.ai/pricing), seogeoaeo.ai
- [How to find a competitor's keywords](https://seogeoaeo.ai/blog/competitor-keywords), seogeoaeo.ai