Skip to content

pool: use pool host name as firefly syslog HOSTNAME - #8303

Open
ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-syslog-hostname
Open

ShawnMcKee wants to merge 1 commit into
dCache:masterfrom
ShawnMcKee:scitags/firefly-syslog-hostname

Conversation

@ShawnMcKee

@ShawnMcKee ShawnMcKee commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

FlowMarkerBuilder.build() fills the RFC 5424 syslog HOSTNAME of every firefly with the flow's src-ip:

String host = flow.has("src-ip") ? flow.getString("src-ip") : "localhost";
String syslogHeader = String.format("<134>1 %s %s dCache - firefly-json - ", time, host);

TransferLifeCycle always passes the remote peer as src and the local pool endpoint as dst, so every firefly a pool emits names the remote host as its origin, never the pool.

Impact

Collectors and tools that attribute a firefly to an emitting host or site by its syslog HOSTNAME attribute all dCache fireflies to the remote party.

This was reported by Marian Babik from the ESnet/SciTags collector: an HTTP-TPC pull from KEK (StoRM) into DESY (dCache) produced a firefly with app-name dCache and application=remotehttpsdatatransfer, but with KEK's address as syslog host, i.e. it appeared to be sent by a site that does not run dCache.

Reproduced on an AGLT2 pool running stock 11.2.10 (remote address shortened):

udp  2001:48a8:68f7:1:192:41:231:128 -> <collector>:10514
<134>1 2026-11-13T16:08:43.579Z 2001:660:5009:17:… dCache - firefly-json - {... "src-ip":"2001:660:5009:17:…","dst-ip":"2001:48a8:68f7:1:192:41:231:128" ...}

The packet leaves the pool (…:231:128), but the header carries the remote client's address.

Fix

  • FlowMarkerBuilder: add withHostname(String) and use it for the syslog HOSTNAME instead of deriving it from src-ip. If no hostname is set, the RFC 5424 NILVALUE (-) is used instead of localhost.
  • TransferLifeCycle: set the hostname to the FQDN of the pool node, from NetworkUtils.getCanonicalHostName().

The name is resolved once and cached, so there is no DNS lookup per transfer. It does not depend on the flow's src/dst fields, so it is the same for start and end markers and for transfers in either direction. RFC 5424 prefers the FQDN for HOSTNAME.

The JSON payload is unchanged. In particular, the flow-id src-ip/dst-ip assignment (remote peer / local pool) is not touched by this patch.

Changed since the first version of this PR

The first version used the pool's local endpoint IP address (dst) as HOSTNAME. Following feedback from Marian Babik, it now uses the pool node's FQDN, which is what collectors expect and which does not rely on dst always being the local endpoint.

Testing

  • New unit tests in FlowMarkerTest and TransferLifeCycleTest cover the HOSTNAME of start markers, of end markers with data entering and with data leaving the pool, and the NILVALUE fallback.
  • Ran the PR pipeline's unit-test job locally in maven:3.9.12-eclipse-temurin-21 (mvn -P code-coverage clean verify -DskipTests=false -Drun.slow.tests) on this version: 5177 tests, 0 failures, 0 errors.
  • Deployed this version on a production AGLT2 pool node (two pools) as 11.2.10 plus this commit. tcpdump on UDP 10514 after the restart: 64 of 64 captured fireflies carry umfs19.aglt2.org as HOSTNAME, with src-ip/dst-ip unchanged. They cover xrootd, https and remotehttpsdatatransfer, start and end, data entering and leaving the pool, IPv4 and IPv6 flows, and both the copy sent to the remote peer and the copy sent to the configured collector:
udp  2001:48a8:68f7:1:192:41:231:128 -> <collector>:10514
<134>1 2026-11-13T17:28:21.276Z umfs19.aglt2.org dCache - firefly-json - {...}
  • For IPv4 transfers the copy to the collector went out over IPv6, so the packet's source address and the flow's dst-ip are of different families. The FQDN identifies the pool node in both cases.
  • The first version of this PR (IP address as HOSTNAME) ran on the same node earlier the same day: 12 of 12 captured fireflies carried the pool's address.
  • HTTP-TPC push, triggered deliberately (gfal-copy --copy-mode push, 536 MB from this pool to MWT2): the start and end fireflies carry umfs19.aglt2.org as HOSTNAME; the end firefly reports sent=535965594, received=0.

The commit cherry-picks cleanly onto 11.2 (tested on the 11.2.10 tag). Suggested targets: master, 12.1, 12.0, 11.2.

Signed-off-by: Shawn McKee smckee@umich.edu

🤖 Generated with Claude Code

@ShawnMcKee ShawnMcKee changed the title pool: use local pool address as firefly syslog HOSTNAME pool: use pool host name as firefly syslog HOSTNAME Nov 13, 2026
@ShawnMcKee
ShawnMcKee force-pushed the scitags/firefly-syslog-hostname branch from 1d0164c to 9bb016e Compare November 13, 2026 17:03
Motivation:
FlowMarker builds the RFC 5424 syslog header of each firefly using the
flow's src-ip as HOSTNAME. TransferLifeCycle always passes the remote
peer as 'src' and the local pool endpoint as 'dst', so every dCache
firefly claims to originate from the remote host rather than the pool
that emitted it.

This was observed in the ESnet/SciTags collector for an HTTP-TPC pull
from KEK (StoRM) into DESY (dCache): the firefly carried
app-name=dCache and application=remotehttpsdatatransfer, but its syslog
host was KEK's address. Collectors and tools that attribute fireflies
to an emitting host/site by syslog HOSTNAME (e.g. firefly-explorer's
'shost' filter) therefore misattribute all dCache fireflies to the
remote party.

Modification:
Add FlowMarkerBuilder#withHostname and use it for the syslog HOSTNAME
instead of deriving it from src-ip. If unset, the RFC 5424 NILVALUE
("-") is used instead of "localhost".

TransferLifeCycle sets the hostname to the FQDN of the pool node, as
returned by NetworkUtils#getCanonicalHostName. The name is resolved
once and does not depend on the flow's src/dst fields, so it is the
same for start and end markers and for transfers in either direction
(client read/write, HTTP-TPC pull/push). RFC 5424 prefers the FQDN for
HOSTNAME.

The flow-id src/dst fields are unchanged.

Result:
Fireflies emitted by dCache pools now identify the pool node by its
FQDN as the syslog HOSTNAME, so collectors can attribute them to the
correct host and site.

Testing:
- New unit tests in FlowMarkerTest and TransferLifeCycleTest check the
  syslog HOSTNAME for start markers, for end markers with data
  entering and leaving the pool, and the NILVALUE fallback.

Acked-by:
Target: master, 12.1, 12.0, 11.2
Require-book: no
Require-notes: yes

Signed-off-by: Shawn McKee <smckee@umich.edu>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant