Repository navigation
pool: use pool host name as firefly syslog HOSTNAME - #8303
Open
ShawnMcKee wants to merge 1 commit into
Open
ShawnMcKee wants to merge 1 commit into
ShawnMcKee wants to merge 1 commit into
Conversation
ShawnMcKee
force-pushed
the
scitags/firefly-syslog-hostname
branch
from
November 13, 2026 17:03
1d0164c to
9bb016e
Compare
Motivation:
FlowMarker builds the RFC 5424 syslog header of each firefly using the
flow's src-ip as HOSTNAME. TransferLifeCycle always passes the remote
peer as 'src' and the local pool endpoint as 'dst', so every dCache
firefly claims to originate from the remote host rather than the pool
that emitted it.
This was observed in the ESnet/SciTags collector for an HTTP-TPC pull
from KEK (StoRM) into DESY (dCache): the firefly carried
app-name=dCache and application=remotehttpsdatatransfer, but its syslog
host was KEK's address. Collectors and tools that attribute fireflies
to an emitting host/site by syslog HOSTNAME (e.g. firefly-explorer's
'shost' filter) therefore misattribute all dCache fireflies to the
remote party.
Modification:
Add FlowMarkerBuilder#withHostname and use it for the syslog HOSTNAME
instead of deriving it from src-ip. If unset, the RFC 5424 NILVALUE
("-") is used instead of "localhost".
TransferLifeCycle sets the hostname to the FQDN of the pool node, as
returned by NetworkUtils#getCanonicalHostName. The name is resolved
once and does not depend on the flow's src/dst fields, so it is the
same for start and end markers and for transfers in either direction
(client read/write, HTTP-TPC pull/push). RFC 5424 prefers the FQDN for
HOSTNAME.
The flow-id src/dst fields are unchanged.
Result:
Fireflies emitted by dCache pools now identify the pool node by its
FQDN as the syslog HOSTNAME, so collectors can attribute them to the
correct host and site.
Testing:
- New unit tests in FlowMarkerTest and TransferLifeCycleTest check the
syslog HOSTNAME for start markers, for end markers with data
entering and leaving the pool, and the NILVALUE fallback.
Acked-by:
Target: master, 12.1, 12.0, 11.2
Require-book: no
Require-notes: yes
Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ShawnMcKee
force-pushed
the
scitags/firefly-syslog-hostname
branch
from
November 13, 2026 17:55
9bb016e to
1e38f24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FlowMarkerBuilder.build()fills the RFC 5424 syslog HOSTNAME of every firefly with the flow'ssrc-ip:TransferLifeCyclealways passes the remote peer assrcand the local pool endpoint asdst, so every firefly a pool emits names the remote host as its origin, never the pool.Impact
Collectors and tools that attribute a firefly to an emitting host or site by its syslog HOSTNAME attribute all dCache fireflies to the remote party.
This was reported by Marian Babik from the ESnet/SciTags collector: an HTTP-TPC pull from KEK (StoRM) into DESY (dCache) produced a firefly with app-name
dCacheandapplication=remotehttpsdatatransfer, but with KEK's address as syslog host, i.e. it appeared to be sent by a site that does not run dCache.Reproduced on an AGLT2 pool running stock 11.2.10 (remote address shortened):
The packet leaves the pool (
…:231:128), but the header carries the remote client's address.Fix
FlowMarkerBuilder: addwithHostname(String)and use it for the syslog HOSTNAME instead of deriving it fromsrc-ip. If no hostname is set, the RFC 5424 NILVALUE (-) is used instead oflocalhost.TransferLifeCycle: set the hostname to the FQDN of the pool node, fromNetworkUtils.getCanonicalHostName().The name is resolved once and cached, so there is no DNS lookup per transfer. It does not depend on the flow's
src/dstfields, so it is the same for start and end markers and for transfers in either direction. RFC 5424 prefers the FQDN for HOSTNAME.The JSON payload is unchanged. In particular, the
flow-idsrc-ip/dst-ipassignment (remote peer / local pool) is not touched by this patch.Changed since the first version of this PR
The first version used the pool's local endpoint IP address (
dst) as HOSTNAME. Following feedback from Marian Babik, it now uses the pool node's FQDN, which is what collectors expect and which does not rely ondstalways being the local endpoint.Testing
FlowMarkerTestandTransferLifeCycleTestcover the HOSTNAME of start markers, of end markers with data entering and with data leaving the pool, and the NILVALUE fallback.maven:3.9.12-eclipse-temurin-21(mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests) on this version: 5177 tests, 0 failures, 0 errors.tcpdumpon UDP 10514 after the restart: 64 of 64 captured fireflies carryumfs19.aglt2.orgas HOSTNAME, withsrc-ip/dst-ipunchanged. They coverxrootd,httpsandremotehttpsdatatransfer,startandend, data entering and leaving the pool, IPv4 and IPv6 flows, and both the copy sent to the remote peer and the copy sent to the configured collector:dst-ipare of different families. The FQDN identifies the pool node in both cases.gfal-copy --copy-mode push, 536 MB from this pool to MWT2): the start and end fireflies carryumfs19.aglt2.orgas HOSTNAME; the end firefly reportssent=535965594,received=0.The commit cherry-picks cleanly onto 11.2 (tested on the 11.2.10 tag). Suggested targets: master, 12.1, 12.0, 11.2.
Signed-off-by: Shawn McKee smckee@umich.edu
🤖 Generated with Claude Code