Skip to content

Use nested subscript line numbers in fastparse#21500

Open
Sean-Kenneth-Doherty wants to merge 3 commits into
python:masterfrom
Sean-Kenneth-Doherty:fix-fastparse-nested-type-lines
Open

Use nested subscript line numbers in fastparse#21500
Sean-Kenneth-Doherty wants to merge 3 commits into
python:masterfrom
Sean-Kenneth-Doherty:fix-fastparse-nested-type-lines

Conversation

@Sean-Kenneth-Doherty
Copy link
Copy Markdown

@Sean-Kenneth-Doherty Sean-Kenneth-Doherty commented May 17, 2026

Fixes #21480

When fastparse converts a type annotation subscript into an UnboundType, it used the outer annotation line captured when TypeConverter was created. For nested multi-line subscripts, that made wrong-arity errors point at the outer annotation instead of the nested offending subscript.

Use the AST subscript node's own line number for real source annotations while preserving source offsets for annotations parsed out of type comments. This matches the native parser behavior for nested annotations without collapsing type-comment errors to line 1.

Verification:

  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testNestedSubscriptWrongArityLineNumber testClassVarTooManyArguments testInvalidNumberOfGenericArgsInTypeDecl
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testInvalidNumberOfGenericArgsInUndefinedArg testInvalidNumberOfGenericArgsInNestedBlock testInvalidNumberOfGenericArgsInTupleType testInvalidNumberOfGenericArgsInFunctionType testInvalidFunctionType testInvalidOptionalType testClassVarInClassVar
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testPreviousErrorInMethodSemanalPass3
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testTypeCommentNestedSubscriptWrongArityLineNumber testNestedSubscriptWrongArityLineNumber testClassVarTooManyArguments
  • real AST check on local python3.11 and python3.12: nested source annotations report line 3 while parsed type comments report relative line 1
  • TMPDIR=/home/sean/.cache/codex-tmp/go-build GOTMPDIR=/home/sean/.cache/codex-tmp/go-build PATH=.venv/bin:$PATH .venv/bin/pre-commit run --files mypy/fastparse.py test-data/unit/check-errorcodes.test
  • .venv/bin/python -m compileall -q mypy/fastparse.py
  • TMPDIR=/home/sean/.cache/codex-tmp/go-build GOTMPDIR=/home/sean/.cache/codex-tmp/go-build PATH=.venv/bin:$PATH .venv/bin/python -m mypy --config-file mypy_self_check.ini --no-incremental mypy/fastparse.py

Note: the full local self-check over -p mypy -p mypyc was attempted but did not complete in a useful time locally after a long silent low-CPU run, so I replaced it with the narrower parser-file self-check plus the failing fixture coverage above.

@github-actions

This comment has been minimized.

@Sean-Kenneth-Doherty
Copy link
Copy Markdown
Author

Follow-up pushed at 8dee2d2 to fix the CI line-number regressions.

Root cause: the PR started using nested Subscript AST line numbers, which is correct for normal annotations, but type comments and parsed string annotations are parsed from synthetic snippets where AST line numbers are relative to the snippet. Those paths were reporting main:1 instead of the source line.

The follow-up gives synthetic parses an explicit source-line offset and applies it only when copying nested subscript line/end-line positions.

Local validation:

.venv/bin/python -m pytest -n0 -q mypy/test/testcheck.py::TypeCheckSuite::check-errorcodes.test::testNestedSubscriptWrongArityLineNumber
# 1 passed

.venv/bin/python -m pytest -n0 -q <14 CI-failing node ids>
# 14 passed

uvx black --check mypy/fastparse.py
uvx ruff check mypy/fastparse.py
git diff --check
# all passed

@Sean-Kenneth-Doherty
Copy link
Copy Markdown
Author

I pushed follow-up commit 8dee2d297 after the first CI run exposed line-number regressions in parsed type comments. The initial fix used raw AST lineno for every subscript; that was right for real source annotations but wrong for synthetic parses where AST lines start at 1.

The follow-up keeps normal source annotations at offset 0 and passes line_offset=line - 1 for parsed type comments / function type comments, so nested source annotations preserve their own line while type comments keep their source-file line numbers.

Fresh local validation on the updated head:

  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testNestedSubscriptWrongArityLineNumber
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testErrorCodeTypeIgnoreMisspelled1
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py testInvalidNumberOfGenericArgsInTypeDecl
  • PATH=.venv/bin:$PATH .venv/bin/python runtests.py check-errorcodes.test check-semanal-error.test check-literal.test -> 132 passed, 13 passed, 121 passed, 1 skipped
  • PATH=.venv/bin:$PATH .venv/bin/python -m pytest -n0 mypy/test/testsemanal.py::SemAnalSuite::semanal-types.test mypy/test/testsemanal.py::SemAnalSuite::semanal-classvar.test mypy/test/testsemanal.py::SemAnalErrorSuite::semanal-errors.test -> 288 passed
  • PATH=.venv/bin:$PATH .venv/bin/python -m pytest -n0 mypy/test/testfinegrained.py::FineGrainedSuite::fine-grained.test -> 484 passed, 7 skipped
  • TMPDIR=/home/sean/.cache/codex-tmp/go-build GOTMPDIR=/home/sean/.cache/codex-tmp/go-build PATH=.venv/bin:$PATH .venv/bin/python -m mypy --config-file mypy_self_check.ini --no-incremental --num-workers 0 mypy/fastparse.py -> no issues
  • TMPDIR=/home/sean/.cache/codex-tmp/go-build GOTMPDIR=/home/sean/.cache/codex-tmp/go-build PATH=.venv/bin:$PATH .venv/bin/pre-commit run --files mypy/fastparse.py test-data/unit/check-errorcodes.test -> passed
  • git diff --check -> clean

The refreshed GitHub Actions matrix is running now.

@Sean-Kenneth-Doherty
Copy link
Copy Markdown
Author

Follow-up after the first CI run exposed line-number regressions in synthetic type parses:

  • cf126cd adds a regression for nested subscripts inside # type: comments, covering the offset path that previously produced main:1 in unrelated fixture cases.
  • PATH=.venv/bin:$PATH .venv/bin/python -m pytest -q on the 16-case set covering the new annotation/type-comment regressions plus the CI failure cases -> 16 passed
  • TMPDIR=/home/sean/.cache/codex-tmp/go-build GOTMPDIR=/home/sean/.cache/codex-tmp/go-build PATH=.venv/bin:$PATH .venv/bin/pre-commit run --files mypy/fastparse.py test-data/unit/check-errorcodes.test -> passed
  • .venv/bin/python -m compileall -q mypy/fastparse.py -> passed
  • git diff --check origin/master...HEAD -> clean

The refreshed GitHub Actions matrix is still running on the current head (cf126cd).

@github-actions
Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/_internal/concurrency/api.py:181: error: "Call" expects no type arguments, but 1 given  [type-arg]
+ src/prefect/_internal/concurrency/api.py:183: error: "Call" expects no type arguments, but 1 given  [type-arg]

antidote (https://github.com/Finistere/antidote)
- src/antidote/lib/interface_ext/__init__.py:1359: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
- src/antidote/lib/interface_ext/__init__.py:1359: error: Missing type arguments for generic type "staticmethod"  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: Missing type arguments for generic type "staticmethod"  [type-arg]

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:434: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
+ ibis/selectors.py:437: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
- ibis/selectors.py:434: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ ibis/selectors.py:437: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- ibis/expr/types/generic.py:152: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:164: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:152: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:164: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/generic.py:170: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:171: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:170: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:171: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/temporal.py:428: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:431: error: "Value" expects no type arguments, but 1 given  [type-arg]
- ibis/expr/types/temporal.py:1161: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:1174: error: "Value" expects no type arguments, but 1 given  [type-arg]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fastparse and native parser assign different line numbers to nested subscripts in multi-line type annotations

1 participant