How to find competitors' keywords in their content and code

Here is how to find competitors' keywords in content they publish, using nothing but a terminal: their sitemap, their robots.txt, and the title and heading tags of their pages. It takes a few commands, and it tells you what a rival is targeting. It cannot tell you what they rank for, so the last section covers the one lookup that adds positions and traffic.

Step 1: find the sitemap

Start with robots.txt, because it usually names the sitemap:

curl -s https://rival.example/robots.txt | grep -i '^sitemap:'

If nothing comes back, try /sitemap.xml and /sitemap_index.xml directly. Large sites use an index that points to child sitemaps split by type, often with names like post-sitemap.xml or docs-sitemap.xml. The blog and docs children are where keyword-led pages live.

Step 2: turn URLs into phrases

Most teams put the target phrase in the slug. Extract the locations and convert hyphens to spaces:

curl -s https://rival.example/post-sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
  | awk -F/ '{print $NF ? $NF : $(NF-1)}' \
  | tr '-' ' ' | sort -u > slugs.txt

You now have a rough keyword list. Drop dates, IDs and pagination before you read it. Watch for patterns: a run of "x vs y" slugs, a cluster of "how to" guides, or a set of integration pages often shows where the rival is spending its writing time.

Step 3: read titles and headings

The slug is a hint. The title tag and H1 are the clearer signal, since most SEO teams put the primary phrase in both, and the H2s often hold secondary phrases. A short loop over the pages you care about:

while read url; do
  html=$(curl -sL "$url")
  echo "== $url"
  echo "$html" | grep -o '<title>[^<]*' | sed 's/<title>//'
  echo "$html" | grep -oE '<h[12][^>]*>[^<]*' | sed -E 's/<h[12][^>]*>/  /'
done < urls.txt

This misses headings that contain nested tags or that a page renders with JavaScript, so treat the output as a quick read, not a parser.

Step 4: spot-check with site:

For a topic that matters to you, search Google with site:rival.example plus the topic. It shows which of their pages Google ties to it. Google's documentation says the results are not always exhaustive and large sites should not expect every URL to show up, so this is a sample.

What this gets you, and what it misses

You have their intentions: the phrases they wrote pages for and the words they chose. That is useful for understanding positioning and naming.

What is missing is the outcome. A rival can publish pages for hundreds of phrases and rank for a small fraction. Your slug list has no positions, no search volumes, no visits. You cannot tell the page that brings them a quarter of their traffic from the one that sits on page six.

Adding rankings with one lookup

The seogeoaeo.ai Competitor Keywords tool fills that gap. Give it the rival's domain, or any URL on it, and a country. It returns up to 100 keywords the domain ranks for in Google, ordered by estimated traffic, each with position, volume, difficulty, intent, cost per click, estimated visits and the ranking URL. It also lists the pages that bring the domain the most visits, which you can join against your slug list to see which planned pages paid off.

It costs 35 credits per run, and the price shows before the run. It also runs over REST or MCP with a workspace API key, so the same job fits in a script:

curl -X POST https://seogeoaeo.ai/api/v1/tools/competitor-keywords/runs \
  -H "Authorization: Bearer $SEOGEOAEO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":{"url":"https://rival.example","market":"us"}}'

Limits worth knowing: it covers the whole domain, not one folder; one country per run; visits and volumes are estimates from a data provider; and it does not show what a competitor bids on in Google Ads. It also does not track rankings over time, so if you need a history of your own positions, a rank tracker such as Semrush Position Tracking is the right tool for that part.

Putting the two together

Join your slug list against the ranking URLs. Pages in both are plans that worked. Pages only in the sitemap did not make the top 100 by traffic, which often marks a topic where a better page from you can still compete. Plan pricing starts at $29 a month for 2,000 credits; see seogeoaeo.ai/pricing.