Repository navigation
Instrument only host code in the coverage build - #3193
Open
ndgrigorian wants to merge 3 commits into
Open
ndgrigorian wants to merge 3 commits into
ndgrigorian wants to merge 3 commits into
Conversation
Device-side -fprofile-instr-generate puts every __profc_ counter of a module into each of its kernel images as a device global. The OpenCL CPU runtime then segfaults at random in NDRange::Init, calling clDevCmdStatusChanged on a freed command; dpt.concat in _tensor_impl, whose images carry the counters of choose.cpp and putmask.cpp, crashes in about half the runs. Device counts never reach the profile, so instrument the host side only, as the static libtensor sources already are.
ndgrigorian
requested review from
antonwolfy and
vlad-perevezentsev
as code owners
November 20, 2026 22:14
Contributor
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3193/index.html |
Contributor
|
Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_39 ran successfully. |
Collaborator
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes a random segfault of the OpenCL CPU runtime in the coverage build, which fails the coverage job of PRs, such as #3157 in
test_sumprod.py, and makes it rely on the retry elsewhere. The crash is always atlibintelocl.so+0x2d5f026: a TBB worker inNDRange::InitcallsclDevCmdStatusChangedon a freed command.In a coverage build,
-fprofile-instr-generate -fcoverage-mappingalso instrument device code, and the device compiler puts every__profc_counter of a module into each of its kernel images, where SYCL treats them as device globals. This happens for the dpnp sources compiled into the libtensor modules (choose.cppandputmask.cppgive each_tensor_implimage about 800 counters), and for every target that keeps the global flags: the pybind11 extensions,dpnp_backend_cand the Cython modules. With them, a fewdpnp.tensor.concatcalls onopencl:cpusegfault in about half the runs;concatcopies with_copy_usm_ndarray_into_usm_ndarrayof_tensor_impl.Device counts never reach the profile, so this PR instruments the host side only (
-Xarch_host), as the static libtensor sources already are: in the globalPROFILE_FLAGS, for the dpnp sources of the libtensor modules and underDPNP_TENSOR_GENERATE_COVERAGE_FOR_PYBIND11_EXTENSIONS.tensor_build_flags.cmakenow stripsPROFILE_FLAGSas a whole. Release builds are unchanged.Checked on a local coverage build of master, on
opencl:cpu(Core Ultra 7 265U):concatcalls) crashes 0 times with the change and 5 times without it;third_party/cupy/math_tests/test_sumprod.pypasses 4 runs out of 4, against 3 crashes out of 3 without it.__profc_counter left (13261 of 23417 images had them before), and host coverage is still recorded, e.g. 78% of the regions ofchoose.cppafter onedpnp.choosecall.DPNP_TENSOR_GENERATE_COVERAGE_FOR_PYBIND11_EXTENSIONS=ON, the libtensor sources get only the-Xarch_hostforms; that configuration was not built.The full coverage job wasn't run locally; the CI run of this PR is the check for that.