Use Python 3.15 lazy import

[vstinner]

Make tomli import time up to 10x faster on Python 3.15. Use Python 3.15 lazy imports to only load regular expressions to parse datetime, localtime or non-trivial numbers (others than just decimal digits). Add try_simple_decimal() parser for simple decimal numbers.

On Python 3.15, use also the new built-in frozendict type (for BASIC_STR_ESCAPE_REPLACEMENTS), instead of types.MappingProxyType.

[vstinner]

Hi! I adapted my change for Python tomllib to tomli. I replaced tomllib with tomli (except in test_try_simple_decimal()), replaced script_helper.assert_python_ok with subprocess in tests, and made some other minor changes to adapt the code to tomli.

[vstinner]

"Tests / Binary wheels for macOS (pull_request)" failed with: ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel. But I don't see the exact error which would explain the failure.

"Tests / coverage (pull_request)" failed because tomli/_parser.py only has a coverage of 99%. How can I know which lines are not covered?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (bbca44b) to head (c80a0bd).

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #292   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            4         4           
  Lines          535       558   +23     
  Branches       100       106    +6     
=========================================
+ Hits           535       558   +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hukkin]

Hi!

With what I currently know, I wouldn't merge the try_simple_decimal optimization.

• It seems to slow down tox -e benchmark by about 2 or 3 %.
• Added code and complexity. Tomli maintains two competing ways to parse a subset of integers.
• .site.toml PEP (number 829) is currently not accepted.
• If the PEP is accepted, not every .site.toml will contain integers. It's not clear to me if the majority will.
• My time is quite limited so I haven't been able to familiarize myself with the PEP. It's not clear to me how large proportion of Python installations/interpreter startups will read a .site.toml.

If CPython codeowners consider this optimization more important than what it currently appears to me, I'll merge anyway just to keep the tomli/tomllib diff down. Let me know @encukou @hauntsaninja!

[vstinner]

Added code and complexity. Tomli maintains two competing ways to parse a subset of integers.

I understand that. It's your call to decide if the startup time optimization is worth it.

.site.toml PEP (number 829) is currently not accepted.

I consider that improving startup time is worth it even if the PEP is rejected, so I proposed this change.

It seems to slow down tox -e benchmark by about 2 or 3 %.

I made a tiny change (add USE_SIMPLE_DECIMAL) to fix this performance issue (see below).

---

I pushed a change adding USE_SIMPLE_DECIMAL variable to disable try_simple_decimal() when the first non-simple number is found (ex: datetime, time or float): switch to regular expressions.

I wrote a benchmark using pyperf:

import pyperf
import os.path
import tomli

with open('pyproject.toml', 'rb') as fp:
    PYPROJECT_TOML = fp.read().decode()
with open('benchmark/data.toml', 'rb') as fp:
    BENCHMARK_TOML = fp.read().decode()

def parse_toml(data):
    tomli.loads(data)

runner = pyperf.Runner()
runner.bench_func(f'parse benchmark/data.toml', parse_toml, BENCHMARK_TOML)
runner.bench_func(f'parse pyproject.toml', parse_toml, PYPROJECT_TOML)

Results on my Fedora 43 laptop with CPU isolation:

$ python -m pyperf compare_to ref.json lazy.json --table
Benchmark hidden because not significant (2): parse benchmark/data.toml, parse pyproject.toml

So this change has no significant impact on performance.

• benchmark/data.toml tests USE_SIMPLE_DECIMAL=False case
• pyproject.toml tests USE_SIMPLE_DECIMAL=True case

Note: pyperf spawns fresh worker processes for each benchmark, so each benchmark can test a different USE_SIMPLE_DECIMAL case.

[encukou]

I agree that the speedup doesn't look like it's worth the implementation complexity, especially before PEP 829 is accepted.

[vstinner]

I extracted the frozendict change in a separated PR: #294.

[hukkin]

Would you like to extract the lazy import change in a separate PR too? Integers seem rarely used in TOML so I think it makes sense even without try_simple_decimal.

[vstinner]

Ok, I abandon this change and I wrote #295 to only add __lazy_modules__ (without try_simple_decimal()).