Repository navigation
29-rc: add checksum verification - #591
Conversation
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
f8f8eb2 to
445ecd8
Compare
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
6c21918 to
ed5ff0d
Compare
Pushed some commits to make it conditional, and CI looks to like it, so can squash if we want to keep that. |
| 'x86_64') \ | ||
| url='https://download.docker.com/linux/static/test/x86_64/docker-29.10.0-rc.1.tgz'; \ | ||
| sha256='d023bcab935f045548c5df79ea96d1f3fccbc091cf9a8347139154859f4767b8'; \ |
There was a problem hiding this comment.
We do have separate downloads now for the CLI and Engine, so we could use the CLI packages here BUT the engine package doesn't contain the containerd binaries, and we do not (yet) have a separate download for containerd (we still need the ctr and shim binaries); https://download.docker.com/linux/static/test/x86_64/
tianon
left a comment
There was a problem hiding this comment.
Niiiice, I'm so excited to let this TODO finally die ❤️
I've made some code suggestions that should hopefully let us get rid of all changes to 29/ so it stays as-is until we swap over to GA -- I've also done some code golfing suggestions that I wouldn't blame you one bit for not following, and I'm happy to take over from here if you'd rather. 🙇 👍
| @@ -58,7 +58,9 @@ def download(opts): | |||
| wget -O \(opts.target | @sh) \"$url\";" | |||
| , | |||
| if $notSha256 then "" else " | |||
There was a problem hiding this comment.
| if $notSha256 then "" else " | |
| if $notSha256 or ($sha256 == null) then "" else " |
There was a problem hiding this comment.
This didn't work; not exactly sure why; it complained about the $sha255 not being defined I think 🤔 (need to look closer)
edit; error was this;
./apply-templates.sh 29
processing 29/cli ...
jq: error: $sha256 is not defined at shared.jq, line 60:
if $notSha256 or ($sha256 == null) then "" else "
jq: 1 compile errorThere was a problem hiding this comment.
Oh right, down here it would need to be .[opts.sha256Key]
There was a problem hiding this comment.
Oh shoot, or actually more complicated than that because that's per-arch. Give me a sec.
There was a problem hiding this comment.
Ok, it's not perfect, but this is the right idea:
diff --git a/shared.jq b/shared.jq
index 372d77d..ed950bb 100644
--- a/shared.jq
+++ b/shared.jq
@@ -57,11 +57,9 @@ def download(opts):
wget -O \(opts.target | @sh) \"$url\";"
,
- if $notSha256 then "" else "
- if [ -n \"${sha256:-}\" ]; then
- echo \"$sha256 *\"\(opts.target | @sh) | sha256sum -c -;
- fi;"
- end
+ if opts.sha256Key and all(opts.arches[]; has(opts.sha256Key)) then "
+ echo \"$sha256 *\"\(opts.target | @sh) | sha256sum -c -;"
+ else "" end
] | add
| rtrimstr(";")
| gsub("(?<=[^[:space:]])\n"; " \\\n")There was a problem hiding this comment.
(we should probably also update versions.sh to make it so that if any() and all() don't agree here it's an error, since it doesn't make sense for us to have checksums for one arch but not all of them, but that's a separate thing and is "belt-and-suspenders" so I'm not too worried about it)
| if checksum="$(_curl "$url.sha256")"; then | ||
| read -r sha256 filename <<< "$checksum" | ||
| if [[ ! "$sha256" =~ ^[[:xdigit:]]{64}$ ]] || [[ "$filename" != "${url##*/}" ]]; then |
There was a problem hiding this comment.
I'm honestly not super worried about the filename (and if the format ever changes to be just the checksum, as I'd personally expect from file.sha256, we don't want to come back to this code):
| if checksum="$(_curl "$url.sha256")"; then | |
| read -r sha256 filename <<< "$checksum" | |
| if [[ ! "$sha256" =~ ^[[:xdigit:]]{64}$ ]] || [[ "$filename" != "${url##*/}" ]]; then | |
| if sha256="$(_curl "$url.sha256")"; then | |
| sha256="${sha256%% *}" | |
| if [[ ! "$sha256" =~ ^[[:xdigit:]]{64}$ ]]; then |
There was a problem hiding this comment.
I'm honestly not super worried about the filename
This part definitely was my AI buddy nagging me about "should validate that as well";
and if the format ever changes to be just the checksum, as I'd personally expect from file.sha256, we don't want to come back to this code:
That's actually something I wondered while working on this; what's more common if a checksum is provided per-file? Still follow the sha256sum format (<hash> <filename>), or just a bare hash?
I could see the advantage of the current format be that a sha256sum -c <filename>.sha256 would "just work". But yeah, to capture the hash, you'd have to trim the content.
There was a problem hiding this comment.
Actually; one thing we should fix is that currently it doesn't use binary format;
curl -fsSL https://download.docker.com/linux/static/test/x86_64/docker-engine_29.10.0-rc.1.tgz.sha256
91a4274c4c080076719bcf4633259eaccb26c6ac733f3d3990b88c9323ac525d docker-engine_29.10.0-rc.1.tgzI'd expect that to be
-91a4274c4c080076719bcf4633259eaccb26c6ac733f3d3990b88c9323ac525d docker-engine_29.10.0-rc.1.tgz
+91a4274c4c080076719bcf4633259eaccb26c6ac733f3d3990b88c9323ac525d *docker-engine_29.10.0-rc.1.tgzThere was a problem hiding this comment.
☝️ opened a PR to use binary mode;
There was a problem hiding this comment.
Yeah, IMO it's weird if foo.sha256 contains XXXX foo or XXXX *foo and would expect it to be just XXXX, but both ways are pretty common. Having the checksum available is what's really important.
There was a problem hiding this comment.
Yeah, agreed that .. it's a bit double to have both filename.shaXX and the filename to be repeated in the file.
I guess there's something to be said for being able to use sha256sum and it to "just work" (both when creating and consuming). Static binaries all seem to be a bit under defined in conventions 😞
|
Thanks! I'll probably take most (or all) of your suggestions. My initial motivation of this PR was as a bit of a reminder ("don't forget to update the official images now that we can") and to make sure this could all work (and no fixes needed in our packaging). Admitted; a bunch was "paint by numbers" and "copy by example", with some of my AI buddy mixed in. |
Co-authored-by: Tianon Gravi <admwiggin@gmail.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
7cddf89 to
1884016
Compare
Changes: - docker-library/docker@c879009: 29-rc: add checksum verification (docker-library/docker#591) - docker-library/docker@f9a1f03: Update 29-rc to 29.10.0-rc.1, buildx 0.38.2, compose 5.7.0
Changes: - docker-library/docker@c879009: 29-rc: add checksum verification (docker-library/docker#591) - docker-library/docker@f9a1f03: Update 29-rc to 29.10.0-rc.1, buildx 0.38.2, compose 5.7.0 Co-authored-by: Docker Library Bot <doi+docker-library-bot@docker.com>
|
Thanks for carrying it! |
This probably fails because we only have the checksums for v29.10.0-rc.1, not for the non-rc releases, but let's see how it goes.