Skip to content

Test fixtures that accept from a non-blocking std listener fail on macOS #503

Open

Description

@aram356

Problem

Eight test fixtures bind a std::net::TcpListener and call set_nonblocking(true) on it, so the accept loop can poll a deadline. They then read the accepted stream as if it were blocking, relying on set_read_timeout.

The two platforms treat the accepted socket differently:

  • On Linux, accept(2) does not copy O_NONBLOCK from the listener, so the read waits up to its timeout.
  • On macOS it does, so the read returns WouldBlock at once whenever the client's bytes have not arrived yet.

Under parallel test load the fixtures therefore fail on macOS with WouldBlock, or with "SDK did not contact fixture". Which tests fail changes from run to run. CI runs only on ubuntu-latest, so CI never shows this.

The probe below, run on macOS (Darwin 27.2.0), shows the difference. It accepts from a non-blocking listener, sets a 2 s read timeout and reads with no data sent. The read returns WouldBlock after 1.5 µs.

let listener = TcpListener::bind("127.0.0.1:0")?;
listener.set_nonblocking(true)?;
let _client = TcpStream::connect(listener.local_addr()?)?;
// accept() polled until a stream arrives
accepted.set_read_timeout(Some(Duration::from_secs(2)))?;
accepted.read(&mut [0_u8; 1]); // macOS: Err(WouldBlock) at once

Affected fixtures

File Added by
crates/edgezero-store-aws/src/secrets_manager.rs #414
crates/edgezero-store-aws/src/dynamodb_tests.rs #414
crates/edgezero-adapter-axum/tests/native_aws_startup.rs #414
crates/edgezero-cli/tests/native_store_generated_proof.rs #414
crates/edgezero-cli/tests/dynamodb_generated_proof.rs #414
crates/edgezero-store-celld/src/client.rs #422
crates/edgezero-adapter-axum/tests/native_celld_startup.rs #422
crates/edgezero-cli/tests/celld_generated_proof.rs #422

Expected behavior

Each fixture puts every accepted std stream back into blocking mode, or reads through an API that waits, before it relies on read and write timeouts. The tests then behave the same on macOS as on Linux.

Also failing on macOS, cause not yet traced

Four production_hosting tests fail on macOS through tests/fixtures/native_proxy.rs (#409), and they also fail on #409's own head:

  • native_selected_families_conflicts_malformed_sets_gaps_and_exhaustion
  • native_source_bound_single_proxy_and_mapped_cidr_match_only_owned_peers
  • native_two_proxy_chain_stops_at_untrusted_hop_and_gateway_replaces_prefix
  • native_verified_tls_ingress_overwrites_public_assertions_and_rejects_wrong_name

They get an empty response, or a TLS UnexpectedEof. native_proxy.rs accepts through tokio, so the cause above does not explain them.

Acceptance

  • On macOS (Apple Silicon), with the default test thread count, these all pass: edgezero-store-aws and edgezero-store-celld with all features, native_aws_startup, native_celld_startup, and the generated celld, DynamoDB and native-store proofs.
  • The four production_hosting tests either pass on macOS, or are documented as Linux-only with the reason.

Found while building #495.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions