Misbehavior of `f.__code__.co_firstlineno` for decorated functions

[picnixz]

Bug report

Bug description:

See also: #139783

Compare:

dummy = lambda f: f                 # L1
@dummy                              # L2
def f():                            # L3
    return                          # L4
print(f.__code__.co_firstlineno)    # L5  # result = 2

with

dummy = lambda f: f                 # L1
pass                                # L2
def f():                            # L3
    return                          # L4
print(f.__code__.co_firstlineno)    # L5  # result = 3

and

dummy = lambda f: f                 # L1
@\
dummy                               # L3
def f():                            # L4
    return                          # L5
print(f.__code__.co_firstlineno)    # L6  # result = 3

In the last example, the line continuation after @ is not handled correctly and the first line for f.__code__ then becomes the line at dummy. Because of that, the values returned by inspect.findsource(f) will not be consistent and inspect.getsourcelines will not use the correct source (it will only start at "dummy" and will not contain the "@" part).

@serhiy-storchaka suggested that this is an issue in the AST parser. I however don't know how we should actually fix this.

Consequences to inspect.getsource and other introspection functions:

import inspect

def outer(*args, **kwargs):
    def inner(f):
        return f
    return deco

@(
    outer(
        ...
    )
)
def f():
    return "hello"

print(inspect.getsource(f))

On Python 3.12, the output is expected:

@(
    outer(
        ...
    )
)
def func():
    return "hello"

On Python 3.13.7, the output is missing the first line @(:

    outer(
        ...
    )
)
def func():
    return "hello"

On Python 3.13.9 and main, we are missing @ and outer parentheses:

    outer(
        ...
    )

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[serhiy-storchaka]

This is a complex issue. The range of FunctionDef starts with the def line, it does not include decorators. The decorator_list attribute is a list of decorator expressions. Expression does not include @, so its range does not include the line with a single @\. co_firstlineno is calculated as minimum of lineno of the function and all decorator expressions.

Decorated functions and classes are the only nodes whose range do not include ranges of all its subnodes. We can change this by making their range including decorators (with @), this is relatively easy. This can only be done in a new release. We will lost information about the line containing def or class. I do not know what will be an impact of this.

Other option -- change AST even more. Instead of the decorator_list attribute of function and class nodes containing the list of decorators, introduce a Decorated node that wraps the function or class node and contains the list of decorators. Then we will have the position of def or class as well as the position of the first @, and the ranges will children will be always included in the range of the parent. But this will require changes in the code generator and some other code.

There may be old issues for this problem.

[picnixz]

Expression does not include @, so its range does not include the line with a single @. co_firstlineno is calculated as minimum of lineno of the function and all decorator expressions.

Yes that's what I observed.

We will lost information about the line containing def or class

I think it will have an impact for analysis tools. Especially, static ones that could only rely on reading the file. An alternative to keep the current behavior could be to have an additional attribute that includes the entire range. That way we know the effective location of def and its surrounding.

[picnixz]

Actually, it may not necessarily be that bad. In codegen_function:

    firstlineno = s->lineno;
    if (asdl_seq_LEN(decos)) {
        firstlineno = ((expr_ty)asdl_seq_GET(decos, 0))->lineno;
    }

So, we're already considering firstlineno to be the lineno of the first decorator. The decorators are arbitrary nodes but we could add where the decorator really started. And here, firstlineno wouldn't be lineno but atsymbol_lineno for instance.

[picnixz]

I am trying to modify the parser but I never touched the grammar file and thus I'm not sure how I can make @ as part of the line being parsed. AFAIK, this is the only case where we have a non-keyword word which produces an AST node.

[serhiy-storchaka]

See also #77392 and #9731. So the first idea would return the old behavior which had its own issues.

[picnixz]

Arf, I see. I don't have much time today but I'll try to have a look tomorrow.

[dmadisetti]

On closer look, #140268 is not a duplicate, but likely stemming from the same change.

edit: Confirmed this bug was introduced between 13.7 and 13.8

edit again: nevermind, I isolated it to an inspect.py change and no parse differences. Marking this off topic. and already fixed: f4104f5