Repository navigation
xrootd: take firefly flow-id of third-party copies from the source connection - #8309
Open
ShawnMcKee wants to merge 2 commits into
Open
ShawnMcKee wants to merge 2 commits into
ShawnMcKee wants to merge 2 commits into
Conversation
Motivation: A firefly ties a network flow to an experiment and an activity. The flow-id must therefore name the TCP connection that carries the data. For xrootd and http(s) transfers that a door redirects to a pool, it does not (dCache#8307). NettyTransferService sends the start marker when the mover is created, before the client has connected to the pool, and takes the remote endpoint from the protocol info. That is the client's connection to the door. The client then connects to the pool from a different port, so the marker carries one port from each connection and matches no flow. The end marker repeats the same endpoint. On a production pool the client port was wrong in 54 of 55 transfers and right in none. Modification: The start marker is no longer sent from executeMover. NettyMoverChannel gains connected(remote, local), which the protocol handlers call with the addresses of the Netty channel once a client has attached to the mover: HttpPoolRequestHandler for GET and PUT, XrootdPoolRequestHandler when a file is opened for reading or writing. The first such connection is recorded on the mover and announced with the start marker. Mover gains getRemoteEndpoint() (as in dCache#8306) and isPassive(). DefaultPostTransferService uses the recorded remote endpoint for the end marker, so both markers of a transfer carry the same flow-id. For a passive mover that no client connected to, no end marker is sent, as there was no flow and no start marker. Not covered: when a pool is the destination of an xrootd third-party copy, the file arrives over a connection that the pool's third-party client (TpcWriteDescriptor) opens to the source. The requesting client's connection carries no file data and is not reported, so no marker is sent for such transfers until the third-party client reports its own connection. Result: Fireflies for xrootd and http(s) client transfers name the connection that carries the data. The start marker is sent when the client has connected to the pool instead of before. Movers that no client connects to emit no markers. Testing: - New NettyTransferServiceFlowMarkerTest. - PR pipeline unit-test job run locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5178 tests, 0 failures. - Backported to 11.2.10 and run on a production pool. TCP connections to the mover ports were captured for 10 minutes and compared with the endpoints the pool logged. Before: client port right in 0 of 55 transfers. After: right in 65 of 65 (64 xrootd, 1 https). Start and end markers carried the same flow-id in 67 of 67 transfers. Fixes: dCache#8307 Acked-by: Target: master Require-book: no Require-notes: yes Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Shawn McKee <smckee@umich.edu>
…nnection Motivation: When a pool is the destination of an xrootd third-party copy, the file arrives over a connection that the pool's third-party client opens to the source. The client that requested the copy connects to the pool too, but that connection carries no file data. Since the change for dCache#8307, the requesting client's connection is not reported to the mover, so such transfers emit no flow markers. Before that change, the start marker named the requesting client's connection to the door, and the end marker named the source host and port from the request. Neither is the connection that carries the data, and the source may redirect the third-party client to another server. Modification: TpcWriteDescriptor reports the third-party client's connection to the mover (NettyMoverChannel#connected) when the first data arrives from the source. By then any redirect has been followed, so the connection reported is the one that carries the file. The start marker is sent then, and the end marker reuses the same endpoints. Result: Fireflies for xrootd third-party copies into dCache name the connection from the pool to the server that sends the file. A copy of a zero-length file, for which no data arrives, emits no markers. Testing: - New TpcWriteDescriptorFlowMarkerTest (5 tests): the connection is reported when the first data arrives, only once, and for the client that receives the data after a redirect. Three of the tests fail without the change. - PR pipeline unit-test job run locally in maven:3.9.12-eclipse-temurin-21: 5183 tests, 0 failures. - Backported to 11.2.10 and run on a production pool as the destination of `xrdcp --tpc delegate only` from a dCache source (MWT2), with the pool's connections captured. The source door redirected the pool to a source pool; both markers named that data connection (port for port) and the end marker reported the full file size as received. The requesting client appeared in no marker. Acked-by: Target: master Require-book: no Require-notes: yes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shawn McKee <smckee@umich.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #8308. This branch contains #8308's commit (
6b55d9790c) followed by one new commit (eafe9b2634). Only the second commit is for review here; once #8308 is merged, this PR will show just that commit.Follow-up to #8307 / #8308.
Problem
When a pool is the destination of an xroot third-party copy, the file arrives over a connection that the pool's third-party client (
TpcWriteDescriptor) opens to the source. The client that requested the copy also connects to the pool, but that connection carries no file data.startmarker named the requesting client's connection to the door, and theendmarker named the source host and port from the request. Neither was the connection that carries the data, and the source may redirect the third-party client to another server.Fix
TpcWriteDescriptorreports its client's connection to the mover (NettyMoverChannel#connected, added in #8308) when the first data arrives from the source. By then any redirect has been followed, so the reported connection is the one that carries the file. Thestartmarker is sent then, and theendmarker reuses the same endpoints.Behaviour to be aware of
scitags lifecycle=end skip reason=no-data-connection, and no firefly was sent.Testing
New
TpcWriteDescriptorFlowMarkerTest(5 tests): the connection is reported when the first data arrives, only once, and for the client that receives the data after a redirect. 3 of the tests fail without the change.Ran the PR pipeline's unit-test job locally in
maven:3.9.12-eclipse-temurin-21(mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests): 5183 tests, 0 failures, 0 errors.Tested on a production pool. The change was backported to 11.2.10 and run on an AGLT2 pool, as the destination of
xrdcp --tpc delegate onlyfrom a dCache 11.2.11 source (MWT2, 448 MB). The pool's connections were captured alongside the fireflies:pool:51790→door:1094pool:47282→source pool:33140startandendmarkerssource pool:33140↔pool:47282The
endmarker reportedreceivedequal to the file size. The requesting client appeared in no marker.The source pool (stock 11.2.11) also sent fireflies for the same transfer. They named our connection to its door, port 51790: the same fault as pool: fireflies for client transfers (xrootd, http) do not name the connection that carries the data #8307, seen from the other side.
Signed-off-by: Shawn McKee smckee@umich.edu
🤖 Generated with Claude Code