gh-144995: Optimize memoryview == memoryview by vstinner · Pull Request #144996 · python/cpython

vstinner · 2026-02-19T11:50:58Z

Issue: Optimize memoryview comparison: a memoryview is equal to itself #144995

vstinner · 2026-02-19T11:52:10Z

Results of the benchmark from the issue:

bytes 0.000122 seconds
mview 0.000146 seconds
⇒ 1.197965 time slower

memoryview comparison complexity is no longer O(n) but O(1): values are no longer compared.

eendebakpt · 2026-02-19T14:51:56Z

 }

+static int
+is_float_format(const char *format)


Does this cover the complex types?

import numpy as np a = np.array([1+2j, 3+4j, float('nan')], dtype=np.complex128) mv = memoryview(a) mv == mv # False

This memory format is Zd. Oh, my change doesn't work for this memoryview. I should replace the blocklist with an allowlist. I'm not a memoryview/buffer expert. I didn't know that 3rd party projects can have their own format.

vstinner · 2026-02-19T15:53:04Z

@eendebakpt: I updated the PR to allow formats known to be safe for pointer comparison (integer types), instead of blocking formats known to use floats.

I excluded the format P since I don't know well this format. Or can we allow it?

eendebakpt · 2026-02-19T21:55:48Z

@eendebakpt: I updated the PR to allow formats known to be safe for pointer comparison (integer types), instead of blocking formats known to use floats.

I excluded the format P since I don't know well this format. Or can we allow it?

I think adding the P is fine (but I am no expert either). Leaving it out is the safe option, we can reconsider if this turns out to be a performance bottleneck.

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>

serhiy-storchaka · 2026-02-20T18:49:42Z

+        # A memoryview is equal to itself: there is no need to compare
+        # individual values. This is not true for float values since they can
+        # be NaN, and NaN is not equal to itself.
+        for int_format in 'bBhHiIlLqQ':


Can "?" be tested? Can format starting with "@" be tested? Can the null format be tested?

I don't know how to test these formats. array.array doesn't support "P" and "?" formats and it doesn't support "@" byte order. Do you have an idea how to test these cases?

memoryview.cast() supports them.

Surprisingly:

>>> memoryview(b'\0\1').cast('?') == memoryview(b'\0\2').cast('?') False

even if

>>> list(memoryview(b'\0\1').cast('?')) == list(memoryview(b'\0\2').cast('?')) True

But this may be platform depending, so I would not test values different than 0 and 1. Or 1 is also not safe?

It may be undefined behavior to interpret random values except 0 as void* (even if it works on x86). Maybe there is a way to create an array of pointers in ctypes? Or it is not worth to bother?

@b

* Optimize also "P" format * Test also "m != m" * Handle native formats such as "@b"

vstinner · 2026-02-21T12:34:57Z

I updated the PR to address @serhiy-storchaka's review:

Optimize also "P" format
Test also "m != m"
Handle native formats such as "@B"

vstinner · 2026-02-21T15:25:35Z

I added tests on 4 more formats: @b, @b, P and ?.

serhiy-storchaka

I have some doubts about the 'P' test. It may be an operation with undefined behavior (although CPython may be never run on platforms were this does not work, but I am not sure). It would be safer to omit that test. There are no other tests for 'P' format. But the optimization should work for it (if we exclude undefined behavior).

vstinner · 2026-02-21T19:05:51Z

I modified the tests to check that the result with optimization is the same as the result without optimization:

        def check_equal(view, is_equal):
            self.assertEqual(view == view, is_equal)
            self.assertEqual(view != view, not is_equal)

            # Comparison with a different memoryview doesn't use
            # the optimization and should give the same result.
            view2 = memoryview(view)
            self.assertEqual(view2 == view, is_equal)
            self.assertEqual(view2 != view2, not is_equal)

I have some doubts about the 'P' test. It may be an operation with undefined behavior (although CPython may be never run on platforms were this does not work, but I am not sure). It would be safer to omit that test. There are no other tests for 'P' format. But the optimization should work for it (if we exclude undefined behavior).

For boolean (? format), I use memoryview(b'\0\1\2').cast('?') in the test. While m.tolist() == m.tolist() gives a different result than m == m, the important part here is that the can_compare_ptr optimization doesn't change m == m result.

If you are not confident with my P test and would prefer to remove the test, I would prefer removing the optimization for this type.

Disable also the optimization if the format string is NULL.

vstinner · 2026-03-02T21:47:53Z

I have some doubts about the 'P' test. (...). It would be safer to omit that test.

Ok, I disabled the optimization for the "P" format.

I also disabled the optimization if the format string is NULL.

@serhiy-storchaka: Would you mind to review the updated PR?

vstinner · 2026-03-02T21:55:02Z

I added tests on "c", "n" and "N" formats. Now all optimized formats have tests.

serhiy-storchaka · 2026-03-03T09:42:27Z

+            // "d" (double), "f" (float) and "e" (16-bit float).
+            // Do not optimize "P" format.
+            can_compare_ptr = (format[0] != 0
+                               && strchr("bBchHiIlLnNqQ?", format[0]) != NULL


For very small memoryviews, what is faster, strchr() or standard path?

* Skip "n" and "N" test if there is no struct format, instead of failing. * Remove can_compare_ptr variable.

vstinner · 2026-03-03T10:35:53Z

I updated the PR to skip "n" and "N" tests if there is no struct format, instead of failing. I also removed the can_compare_ptr variable (in the C code).

vstinner · 2026-03-03T10:51:55Z

For very small memoryviews, what is faster, strchr() or standard path?

I don't know. I ran a benchmark. It seems like even for an empty memoryview, it's faster with the optimization. Benchmark run on Linux (Fedora 43) with CPU isolation:

Benchmark	ref	optim
0-byte bytes view	19.9 ns	18.0 ns: 1.11x faster
1-byte bytes view	21.5 ns	18.7 ns: 1.15x faster
5-byte bytes view	26.2 ns	17.8 ns: 1.47x faster
10-byte bytes view	31.7 ns	17.8 ns: 1.78x faster
25-byte bytes view	56.0 ns	18.1 ns: 3.10x faster
100-byte bytes view	142 ns	18.1 ns: 7.83x faster
Geometric mean	(ref)	2.08x faster

Benchmark:

import pyperf
import array
runner = pyperf.Runner()
for length in (0, 1, 5, 10, 25, 100):
    runner.timeit(
        f'{length}-byte bytes view',
        setup=f'import array; view=memoryview(array.array("B", [0] * {length}))',
        stmt='view == view')

serhiy-storchaka

LGTM. 👍

serhiy-storchaka · 2026-03-03T10:57:13Z

-        m = memoryview(a.tobytes()).cast('n')
-        check_equal(m, True)
+            int_format = None
+        if int_format:


And since we already use subTest() in this test you can wrap this in a subtest and use skipTest() which will only skip a subtest and report this.

vstinner · 2026-03-03T11:15:55Z

Merged. Thanks for the review @eendebakpt and @serhiy-storchaka.

Optimize memoryview comparison: a memoryview is equal to itself, there is no need to compare values, except if it uses float format. Benchmark comparing 1 MiB: from timeit import timeit with open("/dev/random", 'br') as fp: data = fp.read(2**20) view = memoryview(data) LOOPS = 1_000 b = timeit('x == x', number=LOOPS, globals={'x': data}) m = timeit('x == x', number=LOOPS, globals={'x': view}) print("bytes %f seconds" % b) print("mview %f seconds" % m) print("=> %f time slower" % (m / b)) Result before the change: bytes 0.000026 seconds mview 1.445791 seconds => 55660.873940 time slower Result after the change: bytes 0.000026 seconds mview 0.000028 seconds => 1.104382 time slower This missed optimization was discovered by Pierre-Yves David while working on Mercurial. Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>

pythongh-144995: Optimize memoryview == memoryview

2294542

bedevere-app Bot added the awaiting core review label Feb 19, 2026

bedevere-app Bot mentioned this pull request Feb 19, 2026

Optimize memoryview comparison: a memoryview is equal to itself #144995

Closed

eendebakpt reviewed Feb 19, 2026

View reviewed changes

vstinner added 2 commits February 19, 2026 16:45

Replace blocklist with allowlist

0dd7911

Dummy change to update GitHub

61e37e4

eendebakpt reviewed Feb 19, 2026

View reviewed changes

Comment thread Objects/memoryobject.c Outdated

eendebakpt reviewed Feb 19, 2026

View reviewed changes

Comment thread Objects/memoryobject.c Outdated

Apply suggestions from code review

102f26d

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>

serhiy-storchaka reviewed Feb 20, 2026

View reviewed changes

Address review

2316fab

* Optimize also "P" format * Test also "m != m" * Handle native formats such as "@b"

Test more formats: @b, @b, P and ?

800e85f

serhiy-storchaka reviewed Feb 21, 2026

View reviewed changes

Comment thread Objects/memoryobject.c Outdated

vstinner added 2 commits February 21, 2026 19:52

Fix for empty format string

a754be4

Add tests with optimization disabled

1d72ed2

Do not optimize "P" format

134f0de

Disable also the optimization if the format string is NULL.

Add tests on "c", "n" and "N" formats

3272520

serhiy-storchaka reviewed Mar 3, 2026

View reviewed changes

Address Serhiy's review

83ad213

* Skip "n" and "N" test if there is no struct format, instead of failing. * Remove can_compare_ptr variable.

serhiy-storchaka approved these changes Mar 3, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Mar 3, 2026

vstinner merged commit c9d1234 into python:main Mar 3, 2026
51 checks passed

vstinner deleted the memoryview_equal branch March 3, 2026 11:15

bedevere-app Bot removed the awaiting merge label Mar 3, 2026

Uh oh!

Conversation

vstinner commented Feb 19, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Feb 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

eendebakpt commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented Feb 21, 2026

Uh oh!

vstinner commented Feb 21, 2026

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Feb 21, 2026

Uh oh!

vstinner commented Mar 2, 2026

Uh oh!

vstinner commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented Mar 3, 2026

Uh oh!

vstinner commented Mar 3, 2026

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vstinner commented Feb 19, 2026 •

edited by bedevere-app Bot

Loading