Flaky test_annotationlib test due to non-deterministic set iteration ordering

[colesbury]

Seen in a 3.14 buildbot

test_type_repr (test.test_annotationlib.TestTypeRepr.test_type_repr) ... ok

======================================================================
FAIL: test_nonexistent_attribute (test.test_annotationlib.TestForwardRefFormat.test_nonexistent_attribute)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/3.14.ware-debian-x86.installed/build/target/lib/python3.14/test/test_annotationlib.py", line 161, in test_nonexistent_attribute
    self.assertEqual(epsilon_anno, support.EqualToForwardRef("some | {obj, module}", owner=f))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ForwardRef('some | {module, obj}', owner=[79 chars]810>) != EqualToForwardRef('some | {obj, module}',[86 chars]810>)

----------------------------------------------------------------------
Ran 110 tests in 0.079s

FAILED (failures=1)

https://buildbot.python.org/#/builders/1793/builds/879/steps/8/logs/stdio

I stuck this into Claude and it the cause was due to non-deterministic set ordering, which I think is correct. I stuck a random.shuffle(elts) below and was able to repro the issue locally:

cpython/Lib/annotationlib.py

Lines 390 to 399 in a01694d

	elif type(other) in (list, tuple, set):
	extra_names = {}
	elts = []
	for elt in other:
	new_elt, new_extra_names = self.__convert_to_ast(elt)
	if new_extra_names is not None:
	extra_names.update(new_extra_names)
	elts.append(new_elt)
	ast_class = {list: ast.List, tuple: ast.Tuple, set: ast.Set}[type(other)]
	return ast_class(elts), extra_names

Looks related to:

• annotationlib: Improve test coverage #141174

cc @JelleZijlstra

Linked PRs

• gh-144348: annotationlib: fix test that relies on set ordering #144359
• [3.14] gh-144348: annotationlib: fix test that relies on set ordering (GH-144359) #144375

[Muawiya-contact]

Hi @colesbury, this flaky test makes sense due to the non-deterministic set iteration ordering. I’d like to help investigate and possibly fix the test to make it deterministic. Could you please assign me so I can start working on it?

[StanFromIreland]

@Muawiya-contact please stop posting the same request under multiple issues, and furthermore, Jelle has already assigned himself here.

[JelleZijlstra]

Thanks for reporting and sorry for the trouble. I'm sort of surprised this hasn't come up more, shouldn't this fail 50% of the time?

[Muawiya-contact]

@StanFromIreland Understood, thanks for the clarification. I’ll avoid cross-posting requests and won’t pursue assignment here since it’s already taken.

[colesbury]

I'm sort of surprised this hasn't come up more, shouldn't this fail 50% of the time?

I was also surprised it didn't fail more often. The elements are _Stringifier, which uses id(self) as the hash. From testing, it seems like the object address / id() are generally in order, so we nearly always get the same order, but that depends on the memory allocator.

It looks like the fixes are already merged, so I'll close this issue.