Skip to content

[WIP] Add new Scalar type for GPU performance - #60

Open
kshyatt wants to merge 29 commits into
mainfrom
ksh/scalar
Open

kshyatt wants to merge 29 commits into
mainfrom
ksh/scalar

Conversation

@kshyatt

@kshyatt kshyatt commented Nov 13, 2026

Copy link
Copy Markdown
Member

Idea is to allow us to take advantage of JuliaGPU/CUDA.jl#2616 and the AMDGPU equivalent by avoiding round-trip copies when using VI with GPU arrays.

@kshyatt

kshyatt commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

I think a DenseArray can be N-dimensional which isn't really a Vector, so I've updated that and created a BLASMatrix and BLASArray pair to encapsulate things that should be allowed.

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.38462% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/scalar.jl 80.85% 9 Missing ⚠️
ext/VectorInterfacecuBLASExt.jl 66.66% 6 Missing ⚠️
ext/VectorInterfaceMooncakeExt.jl 90.47% 2 Missing ⚠️
src/abstractarray.jl 91.66% 2 Missing ⚠️
Files with missing lines Coverage Δ
ext/VectorInterfaceCUDACoreExt.jl 100.00% <100.00%> (ø)
ext/VectorInterfaceEnzymeExt.jl 84.56% <100.00%> (+84.56%) ⬆️
src/VectorInterface.jl 100.00% <100.00%> (+91.66%) ⬆️
ext/VectorInterfaceMooncakeExt.jl 88.23% <90.47%> (+88.23%) ⬆️
src/abstractarray.jl 96.05% <91.66%> (+96.05%) ⬆️
ext/VectorInterfacecuBLASExt.jl 66.66% <66.66%> (ø)
src/scalar.jl 80.85% <80.85%> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kshyatt

kshyatt commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

So currently this doesn't make use of the device-side storage in cuBLAS and rocBLAS. If we like the overall approach, I will then add some specializations via extensions here and elsewhere that will use them.

@kshyatt
kshyatt requested a review from Jutho November 13, 2026 14:20
Comment thread src/scalar.jl Resolved
Comment thread src/abstractarray.jl Resolved
Comment thread src/abstractarray.jl Resolved
@kshyatt

kshyatt commented Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

Fail looks like an Enzyme bug, EnzymeAD/Enzyme.jl#4051

@wsmoses

wsmoses commented Nov 14, 2026

Copy link
Copy Markdown

or rather actually a julia problem, where 1.13.4 broke otherwise functional code in 1.13.3. I'd rec to pinning 1.13.3 for now

@kshyatt

kshyatt commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Added the CUDA extension I think we'll need, it's quite short so far but we'll probably want to add a cuBLAS one as well to overrride inner! and norm!

@kshyatt

kshyatt commented Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Did some benchmarking of norm and norm! on 10,000 CuVectors on this branch vs main:

eltype length main norm(x) syncs D→H branch norm!(r, x) syncs D→H speedup
Float32 1e3 473 ms 10,000 10,000 132 ms 0 0 3.6x
Float32 1e5 496 ms 10,000 10,000 132 ms 0 0 3.7x
Float32 1e7 572 ms 10,008 10,000 175 ms 0 0 3.3x
Float64 1e3 665 ms 19,896 10,000 278 ms 0 0 2.4x
Float64 1e5 870 ms 19,984 10,000 484 ms 0 0 1.8x
Float64 1e7 1828 ms 19,999 10,000 1402 ms 0 0 1.3x
ComplexF64 1e3 650 ms 19,836 10,000 253 ms 0 0 2.6x
ComplexF64 1e5 894 ms 19,943 10,000 474 ms 0 0 1.9x
ComplexF64 1e7 2630 ms 19,997 10,000 2192 ms 0 0 1.2x

The fp64 results being much worse are a result of me running this on a consumer CUDA card. This removes the copies and synchronizations also 🎉

@kshyatt

kshyatt commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

We can handle support for AMDGPU in a separate PR once they release a new version

@kshyatt
kshyatt force-pushed the ksh/scalar branch 2 times, most recently from 6d4438a to 36277ab Compare November 16, 2026 17:40
Comment thread src/abstractarray.jl Resolved
Comment thread src/abstractarray.jl Resolved
Comment thread ext/VectorInterfaceEnzymeExt.jl Resolved
@kshyatt

kshyatt commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

After talking with @Jutho we're going to try having the GPU inner and norm return Scalar, and get rid of the in-place (bang !) methods. I'll try to make a PR to KrylovKit with the new changes and see what we think of it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants