[VisLab]

No description provided.

[Copilot]

🟡 Changes recommended

Reference expansion generates exponentially many redundant combinations and materializes the entire Cartesian product, making small sidecars expensive to validate and risking memory exhaustion.

4 open issues

• Deduplicate repeated references before building the substitution product · New
• Stream substitution combinations instead of sorting the full product · New
• Preserve lone CR characters when normalizing line endings · New
• Normalize CRLF to keep line checks aligned with pandas rows · New

What changed in this PR

This PR strengthens HED tabular validation by reporting blank and incomplete TSV lines before validating annotations. It also expands sidecar repeated-expression checks to include removed references.

Changes:

• Record empty and wrong-width TSV lines, report their file locations, and block assembly.
• Check sidecar substitutions with references removed and deduplicate repeated-expression reports.
• Add regression tests and document the validation changes.

File	Description
tests/​validator/​test_spreadsheet_validator.py	Tests line errors, stage ordering, and file-line reporting.
tests/​validator/​test_sidecar_validator.py	Tests removed-reference substitutions and repeat deduplication.
tests/​tools/​bids/​test_bids_file_group.py	Tests malformed lines with invalid sidecars and HED eligibility.
tests/​models/​test_base_input.py	Covers line scanning, loading, and assembly refusal.
tests/​data/​validator_tests/​bids_events_invalid_columns.tsv	Adds a missing trailing cell to preserve fixture intent.
hed/​validator/​spreadsheet_validator.py	Reports line problems during initial validation.
hed/​validator/​sidecar_validator.py	Checks removal substitutions and deduplicates repeat issues.
hed/​tools/​bids/​bids_file_group.py	Allows line errors to be reported despite sidecar errors.
hed/​models/​base_input.py	Records malformed lines and prevents their assembly.
hed/​errors/​error_types.py	Adds internal codes for line errors.
hed/​errors/​error_messages.py	Adds line-specific validation messages.
CHANGELOG.md	Documents TSV and sidecar validation changes.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Newline-only named tables lose their blank header during fallback, leaving blank_columns empty and incorrectly bypassing TABULAR_INVALID (c) validation even with an annotating sidecar.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Preserve blank headers for newline-only named tables

hed/​models/​base_input.py:912

For a named table containing only LF or CRLF, this early return discards the recorded blank header, leaving blank_columns empty and skipping TABULAR_INVALID (c), even with an annotating sidecar. Preserve header_cells through the fallback for named input; genuinely empty text is already handled above. Update the newline-only expectation in tests/models/test_base_input.py:568-569 to expect a blank column.

4 resolved since last review

• Stream substitution combinations instead of sorting the full product
• Deduplicate repeated references before building the substitution product
• Normalize CRLF to keep line checks aligned with pandas rows
• Preserve lone CR characters when normalizing line endings

_{Review effort: Balanced}