Skip to content

Add record-metadata exporter (WARC/WET/WAT → CSV/JSON) with text and link modes - #26

Open
zodetron wants to merge 4 commits into
commoncrawl:mainfrom
zodetron:main
Open

zodetron wants to merge 4 commits into
commoncrawl:mainfrom
zodetron:main

Conversation

@zodetron

@zodetron zodetron commented Nov 6, 2026

Copy link
Copy Markdown

Adds an exporter under exporter/export_records.py that converts WARC/WET/WAT record metadata to CSV or JSON. Includes a text mode (WET page text), a links mode (page links from WARC HTML), a unit-test suite (tests/test_exporter.py), a Makefile target, and README docs. Happy to adjust to match project conventions.

@wumpus

wumpus commented Nov 6, 2026

Copy link
Copy Markdown
Member

Amusingly enough, we just shipped (a few hours before this PR) a tool named warc2zip that overlaps with this functionality. We also have a header-only dataset that's in progress.

And, fundamentally, this repo is intended to showcase tools in other repos.

What should we do?

@lfoppiano
lfoppiano self-requested a review November 19, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants