Skip to content

fix(title): pick best <title> when a page has multiple (Medium fix) - #11

Merged
rebelchris merged 1 commit into
masterfrom
fix/medium-multiple-title-tags
Jul 23, 2026
Merged

fix(title): pick best <title> when a page has multiple (Medium fix)#11
rebelchris merged 1 commit into
masterfrom
fix/medium-multiple-title-tags

Conversation

@rebelchris

Copy link
Copy Markdown

Problem

Recently-scraped Medium articles are being stored with the title Medium.

Medium serves two <title> tags on article pages — the real one (… | by Author | … | Medium) and a bare <title>Medium</title>. In the non-JS HTML our scraper (post-scraper-one-ai / yggdrasil via this newspaper fork) receives, the bare tag can appear first, and get_title() blindly used title_element[0]Medium.

Fix

Instead of always taking the first <title>, choose the best candidate among all <title> tags:

  1. prefer the one whose filtered text contains og:title (the descriptive article title), else
  2. the longest candidate.

The existing |/- splitter then trims the | by … | Medium suffix as before. Single-<title> pages are unaffected.

Verification

Ran against the real Medium article HTML (both tag orderings) + a normal single-title page:

as-fetched (real first)    -> 'High performance has a human requirement'
medium-first (worst case)  -> 'High performance has a human requirement'
single-title sanity        -> 'My Great Post'

Notes / follow-up

This handles the reachable-article case. Genuinely walled/404 Medium pages still only contain <title>Medium</title> (no real title tag), so a defensive guard in post-scraper-one-ai to reject a final title of Medium/Just a moment... is still worth adding separately.

Medium serves two <title> tags on article pages, one of which is a bare
site name (<title>Medium</title>). In the non-JS HTML our scraper receives
that bare tag can come first, so get_title() picked "Medium" as the post
title.

Instead of blindly using title_element[0], choose the best candidate among
all <title> tags: prefer the one whose filtered text contains og:title,
otherwise the longest. Single-title pages are unaffected.
@rebelchris
rebelchris requested review from idoshamun and vpol July 23, 2026 13:59
@rebelchris
rebelchris merged commit 36bedbb into master Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants