Skip to content

fix: avoid selecting feedback form text as article content - #201

Open
ZaynQuinn wants to merge 1 commit into
buriy:masterfrom
ZaynQuinn:fix/feedback-form-extraction
Open

ZaynQuinn wants to merge 1 commit into
buriy:masterfrom
ZaynQuinn:fix/feedback-form-extraction

Conversation

@ZaynQuinn

Copy link
Copy Markdown

On a saved HTML snapshot of the OAIC Privacy Act overview, Document.summary() selects the feedback form's feedback__section-2-content container. The result contains prompts such as “Rate your experience” and omits the introduction to the Act and the overview of the 13 Australian Privacy Principles.

To reproduce on 3e7a832, save the page's original HTML as oaic.html and run:

from pathlib import Path

from readability import Document

url = "https://www.oaic.gov.au/privacy/privacy-legislation/the-privacy-act"
print(Document(Path("oaic.html").read_bytes(), url=url).summary())

Article assembly moves the selected node into the output without its enclosing form, so removing <form> elements during cleanup misses the feedback text. This change keeps the existing <article> and <main> recovery logic. If the resulting candidate is a form or is inside a form, and scored candidates exist outside forms, it reruns selection using those candidates.

Validation

  • Added a synthetic HTML regression test that checks that both article paragraphs are kept and feedback text is excluded. It fails before the fix and passes after it.
  • On the same OAIC snapshot, the normalized extracted text grew from 130 to 645 characters, recovering the introduction and the overview of the privacy principles and excluding the feedback prompts.
  • make pre-release passed on Python 3.13 with 63 passing tests. The 166-page benchmark scored F1 0.973253 before and after, with identical matched, extracted, and reference token counts on every page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant