Skip to content

Instrument only host code in the coverage build - #3193

Open
ndgrigorian wants to merge 3 commits into
masterfrom
fix-coverage-device-counters
Open

ndgrigorian wants to merge 3 commits into
masterfrom
fix-coverage-device-counters

Conversation

@ndgrigorian

Copy link
Copy Markdown
Collaborator

This PR fixes a random segfault of the OpenCL CPU runtime in the coverage build, which fails the coverage job of PRs, such as #3157 in test_sumprod.py, and makes it rely on the retry elsewhere. The crash is always at libintelocl.so+0x2d5f026: a TBB worker in NDRange::Init calls clDevCmdStatusChanged on a freed command.

In a coverage build, -fprofile-instr-generate -fcoverage-mapping also instrument device code, and the device compiler puts every __profc_ counter of a module into each of its kernel images, where SYCL treats them as device globals. This happens for the dpnp sources compiled into the libtensor modules (choose.cpp and putmask.cpp give each _tensor_impl image about 800 counters), and for every target that keeps the global flags: the pybind11 extensions, dpnp_backend_c and the Cython modules. With them, a few dpnp.tensor.concat calls on opencl:cpu segfault in about half the runs; concat copies with _copy_usm_ndarray_into_usm_ndarray of _tensor_impl.

Device counts never reach the profile, so this PR instruments the host side only (-Xarch_host), as the static libtensor sources already are: in the global PROFILE_FLAGS, for the dpnp sources of the libtensor modules and under DPNP_TENSOR_GENERATE_COVERAGE_FOR_PYBIND11_EXTENSIONS. tensor_build_flags.cmake now strips PROFILE_FLAGS as a whole. Release builds are unchanged.

Checked on a local coverage build of master, on opencl:cpu (Core Ultra 7 265U):

  • The reproducer (10 runs of the concat calls) crashes 0 times with the change and 5 times without it; third_party/cupy/math_tests/test_sumprod.py passes 4 runs out of 4, against 3 crashes out of 3 without it.
  • No kernel image of any module has a __profc_ counter left (13261 of 23417 images had them before), and host coverage is still recorded, e.g. 78% of the regions of choose.cpp after one dpnp.choose call.
  • With DPNP_TENSOR_GENERATE_COVERAGE_FOR_PYBIND11_EXTENSIONS=ON, the libtensor sources get only the -Xarch_host forms; that configuration was not built.

The full coverage job wasn't run locally; the CI run of this PR is the check for that.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

Device-side -fprofile-instr-generate puts every __profc_ counter of a
module into each of its kernel images as a device global. The OpenCL CPU
runtime then segfaults at random in NDRange::Init, calling
clDevCmdStatusChanged on a freed command; dpt.concat in _tensor_impl,
whose images carry the counters of choose.cpp and putmask.cpp, crashes
in about half the runs. Device counts never reach the profile, so
instrument the host side only, as the static libtensor sources already
are.
@github-actions

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3193/index.html

@github-actions

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_39 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

No base build to compare — fix-coverage-device-counters into master

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants