Skip to content

Add read_scan_raw to read every field of a scan as stored - #52

Open
mohsaad wants to merge 2 commits into
dancergraham:mainfrom
mohsaad:feat/read-scan-raw
Open

mohsaad wants to merge 2 commits into
dancergraham:mainfrom
mohsaad:feat/read-scan-raw

Conversation

@mohsaad

@mohsaad mohsaad commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Addresses part of #25 and #19.

Motivation

read_points returns xyz, color and intensity with the pose applied, invalid points dropped and color normalized to 0–1. Several things can't be read at all today:

  • row and column indices (gridded / structured scans)
  • return index and return count (multi-return sensors)
  • timestamps
  • invalid-state flags, for filtering points yourself
  • extension fields such as las:pointSourceId
  • the original integer color values, for example 16-bit color

API

raw = e57.read_scan_raw(path, index)                          # dict[str, np.ndarray]
raw = e57.read_scan_raw(path, index, fields=["rowIndex", "columnIndex"])
  • Keys are the E57 field names (cartesianX, colorRed, rowIndex, returnIndex, timeStamp, …). Extension fields are keyed as namespace:name.
  • Values are 1-D arrays with one entry per record in the scan.
  • No processing: no pose, no spherical→cartesian conversion, no normalization, and invalid points are kept.
  • Dtypes follow the field's declared type:
    • Single → float32, Double → float64, ScaledInteger → its scaled float64 value
    • Integer → the smallest of uint8 / uint16 / int32 / int64 that holds the declared min/max, so 8-bit color is uint8 and 16-bit color is uint16
  • fields= returns only the listed fields and skips allocating memory for the others. An unknown field raises ValueError and lists the fields that are available.
  • Index errors: negative indices count from the end, and an out-of-range index raises IndexError.

The keys and value semantics match pye57's read_scan_raw, so code can move between the two easily. I checked values against pye57 on every test file it can read, and they matched within float32 rounding. pye57 can't read returnIndex.e57 at all: it forces color to uint8, and the file has 16-bit values (ErrorValueNotRepresentable).

Tests

  • New tests are in tests/test_read_scan_raw.py:
    • field names and order
    • row/column indices
    • invalid points kept, with their state values
    • pose not applied
    • color returned as raw uint8
    • 16-bit color, return index/count and an extension field
    • scaled integers returned as float64
    • fields= subsets and unknown fields
    • index errors and a missing file
  • New test files, described in testdata/README.md:
    • gridFields.e57: generated with pye57 by testdata/generate_grid_fields.py
    • returnIndex.e57 and scaledIntegerIntensity.e57: 9 KB and 2 KB, copied from the cry-inc/e57 test data (MIT, attributed)
  • pytest: 29 passed. cargo fmt --check and cargo clippy -- -D warnings are clean.

Notes

🤖 Generated with Claude Code

mohsaad and others added 2 commits November 15, 2026 13:06
read_points returns only xyz, color and intensity, with the pose applied,
invalid points dropped and color normalised. There was no way to get row
and column indices, return index/count, timestamps, invalid-state flags,
extension fields, or the original integer color values.

read_scan_raw(filepath, index, *, fields=None) returns a dict of 1-D numpy
arrays keyed by E57 field name, with one entry per record and no
transformation. Floats keep their stored precision, scaled integers are
returned as float64, and integers use the smallest dtype that holds the
field's declared range, so 16-bit color comes back as uint16. `fields`
limits the result (and memory) to the named fields.

Adds a generated grid test file plus two small files from the cry-inc/e57
test data (MIT) covering multiple returns, 16-bit color, an extension
field and scaled integers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohsaad
mohsaad marked this pull request as ready for review November 15, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant