A residual of 2⁻⁵⁶ is enough to tilt a joint axis

In May the gz_sim-ci-main-homebrew-arm64 job went red on a string comparison. ModelCommandAPI.Commands runs gz model -m vehicle_blue --joint against a test world and compares the whole printed block against an expected string. All but two lines matched.

The same test passed on macOS amd64 and on every Linux builder. It was filed as a numerical tolerance issue, and a few hours later the change that unblocks everybody else’s pull requests was merged: two EXPECT_EQ calls behind a preprocessor guard.

That is a reasonable thing to do to a red board, and it is also how a bug becomes permanent. This was the honest version of it, since the guard names the issue rather than deleting the expectation. But two assertions on a command-line tool’s output were now switched off across an entire architecture, and nobody had said where 1.26714e-17 came from.

I have an arm64 Mac. This is what was under it.

The first thing to explain is not the residual. It is why exactly one line in that whole printed block moved.

Every numeric field in gz model’s output goes through a poseInfo helper that builds its strings with std::to_string, which prints six decimal places. The joint axis is the one field streamed straight to std::cout at the default stream precision:

So a residual that prints as 1.26714e-17 in the axis line prints as 0.000000 in every pose, every inertial pose and every mass. When I removed the two guards on a macos-15 runner and re-ran the comparison, the axis line was the only divergence in the entire output; poses, inertial matrices and masses all matched byte for byte. The axis is not computed differently from everything around it. It is simply the only field printed at enough precision to show what the rest of the block is carrying too.

It also handed me a cheap fix I never shipped: route the axis through the same formatter and the number becomes unprintable. I wrote it, verified it red to green on arm64, and then said so in the issue while asking for something else. A change that makes a symptom invisible is worth having on the table and worth naming as exactly that.

sdformat resolves poses on a directed graph of frames. resolvePose computes the pose of a frame relative to a resolve-to frame by walking each of them up to the graph root and composing the two results:

When the frame vertex and the resolve-to vertex are the same vertex, poseR and _pose are the product of the same edge chain. That last line is then a value composed with its own inverse: the identity in real arithmetic, and in doubles a quaternion inverse followed by a quaternion multiply whose cross terms are under no obligation to cancel.

The case is not exotic. It is reached by an axis declared the ordinary way, which is how almost every joint in almost every world file declares one:

With no xyz_expressed_in given, JointAxis::ResolveXyz takes both the frame the axis is expressed in and the frame it resolves to from the same parent name. So the axis is resolved against its own frame, rotated by a quaternion that should be exactly (1, 0, 0, 0), and comes back tilted by however much that quaternion is not.

How much depends on the rotation. The world in the failing test puts the wheel at -1.5707 radians, a hand-written approximation of minus π/2 that is not exactly representable in binary. An exact π/2 round trips cleanly on the same machine in the same binary. Composed with its own inverse, -1.5707 leaves 1.2671444049156495e-17 in the y component of the rotated axis, which at the stream’s default six significant figures prints as 1.26714e-17.

I wrote the issue up as three possible fixes rather than opening a pull request. Test-only: extend the existing helper that normalizes negative zeros to also normalize near-zero scientific notation, in line with what the physics library had already done for arm64 drift, and drop the guards. Formatting: print the axis the way the poses are printed, about twenty lines, at the cost of changing user-visible output from [0 0 1] to [0.000000 0.000000 1.000000]. Upstream: stop producing the residual at all, then drop the guards.

The maintainer took the third, and added something I had not seen. resolvePose walking both frames all the way to the graph root is wasteful in the general case; resolving to the lowest common ancestor instead would be cheaper, and it would make the same-vertex case fall out for free, since the lowest common ancestor of a vertex and itself is that vertex and both walks become zero length.

That is the better change and a much larger one. Resolving via the ancestor is mathematically equivalent but not bit-identical, so it moves the numbers under every hardcoded pose expectation in the test suite. I asked to split it: land the same-vertex guard as a small correctness fix that can be backported to the release branches, and take the rework separately with benchmarks behind it, since the ancestor helper allocates an unordered_set and a vector per call and I would rather measure that trade than assume it. He agreed, and wanted both changes eventually.

Then I tried to reproduce it inside sdformat and could not.

The arithmetic was lossy exactly as advertised. Building a pose directly and composing it with its own inverse, measured from inside sdformat’s own unit test binary on an arm64 runner:

There is the number, in isolation, from the same rotation the failing world uses. But in the same binary on the same machine, resolvePose called with a frame as both arguments returned bitwise identity for every frame I tried, and JointAxis::ResolveXyz returned exactly (0, 0, 1) against a fixture that mirrored the gz-sim world down to the -1.5707.

So the operation was lossy, the path that performs it was not, and I was holding a proposed fix to a function I could not show was wrong. I withdrew it in the issue and asked the maintainer to hold off: the guard would remove a genuinely lossy operation, but on that evidence it would not fix the downstream failure, and I was not going to open an upstream pull request on that basis.

Two things from the dead end were worth keeping either way, because both are traps for whoever eventually writes the regression test.

EXPECT_EQ cannot see this. Pose3d and Vector3d compare through a tolerance-based operator==, so a residual near 1e-17 compares equal to zero and the assertion passes. Each component has to be checked on its own with EXPECT_DOUBLE_EQ.

The obvious fixture cannot see it either. The existing test file for frames relative to joints uses only 0 and exact π/2 rotations, and those round trip exactly. A regression test written against that file passes whether or not the behavior is present. That is the worst kind of test, one that reports on nothing while looking like coverage.

The maintainer settled it in one message. He took the branch, reverted the FrameSemantics.cc hunk by hand, rebuilt, and ran the new test on his own arm64 Mac:

Those are the values from the isolated round trip, arriving through resolvePose and coming out through ResolveXyz, and the second is the 1.26714e-17 from the continuous integration diff. The path was where I had said it was. My measurement of it was what had been wrong.

The difference was the runner. My runs used sdformat’s existing macOS workflow, which is runs-on: macos-latest, and macos-latest now resolves to the macos-26-arm64 image. Same architecture as his machine, same source, opposite result. What is left is the toolchain: the newer Apple clang on that image evidently contracts the quaternion multiply so the round trip cancels exactly, while the clang on macOS 15 does not.

That reframes the bug. Fails on arm64, passes on amd64 is the observation, not the mechanism. The mechanism is floating-point contraction in code generation: whether the compiler is allowed to fuse a multiply and an add into one instruction with a single rounding step, which changes the result by exactly this order of magnitude. It correlates with the instruction set, because arm64 has had the fused instruction all along. It is not caused by it.

The practical consequence deserves its own line, because it costs a day if you do not know it: this cannot be reproduced on GitHub’s macos-latest runners. Not because they are the wrong architecture; they are arm64. Because they are the wrong compiler. Reproducing it needs a machine whose toolchain does not fold the round trip away.

I corrected the pull request description and both issue threads rather than quietly re-pushing over it. Somebody reading that thread in a year needs the correction more than they need me to have been right the first time.

The fix is a guard:

It sits after the first resolve-to-root call, not before it. That call is what validates the frame vertex and produces the graph diagnostics for a disconnected or cyclic frame, so short-circuiting above it would have made the identity case quietly succeed where it used to report an error. Fourteen lines with the comment that explains why they are there.

The test is sixty-five lines, and it needs a fixture of its own: a wheel link at -1.5707 with an axis declared without xyz_expressed_in, mirroring the world that exposed it, plus a second joint at an exactly representable angle as a control, one that resolves cleanly either way, so a failure on the first says something the second does not.

Then the amd64 Jenkins job went red, with all 1341 tests passing.

Two compiler warnings, both mine, both in the new test. I had bound string literals to const std::string & in two range-for loops, so GCC 15 reported -Wrange-loop-construct for the temporary constructed on every iteration. Clang does not warn about it, which is why the arm64 job stayed green and only the Linux one moved. Letting the loop variable deduce its own type fixes it, and matches the idiom already used elsewhere in the parser.

That is the real shape of what this cost, and it is not the fourteen lines. A retraction in public, a run on the wrong image that made me doubt a correct diagnosis overnight, and a red board that turned out to be a style rule rather than a failure. A quality gate that fails a build at two warnings will report a passing suite as broken, and the only way to tell which one you are looking at is to read the console rather than the badge.

With the cause merged, the maintainer asked the question that makes the whole exercise worth doing: would reverting the skip be enough?

The downstream pull request is an exact revert: six lines deleted, one file, nothing added. But a revert is only interesting if the restored expectations actually pass, and a green run proves nothing at all if it happened on a toolchain that never produced the residual in the first place. So the verification job checks the toolchain before it trusts its own result: it compiles a probe that composes the wheel-link pose with its own inverse and rotates the unit z axis by it, and fails the job outright if the round trip cancels exactly.

The same values the maintainer measured, on a runner that does produce the residual, with the sdformat fix installed, and the test passing anyway. Because the restored expectation compares against a string containing [0 0 1], that run is also the first direct observation of gz model --joint printing the declared axis on an affected toolchain. Until then only the sdformat unit test had been confirmed.

What the run is not: the project’s own builder. It is GitHub Actions on a fork, building against head-only Homebrew formulas, with no red baseline in the same run. I could not build the same tree against a pre-fix sdformat without pinning the formula, so the before-and-after pairing still rests on the maintainer’s local revert rather than on my job. Writing that into the pull request is cheaper than having a reviewer find it.

Both changes are merged. Neither issue is closed, and that is correct. The sdformat issue stays open for the lowest-common-ancestor rework and the benchmarks it needs. The gz-sim issue was reopened until the backports land: Mergify opened backports to four sdformat release branches, but gz-sim’s release branches build against released sdformat, so the same guard on three of them has to stay until those releases ship. The fix is on main in both repositories and nowhere else yet.

The residual was in every pose in that output the whole time. What made it a bug was one field printed at full precision. What made it fixable eleven weeks later was that whoever switched the assertions off left the issue number in the source instead of deleting the line.