Add async support for Dataverse SDK#171
Conversation
…o v1 - Implemented unit tests for the `list_pages` method in `TestListPages` class, covering various scenarios including iterator return, page content validation, and parameter passing. - Added checks for deprecation warnings to ensure no warnings are raised during the usage of `list_pages`. - Introduced a new migration script `migrate_v0_to_v1.py` to automate the transition from beta (v0) to GA (v1) API calls, including method renaming and argument adjustments. - Created a new `tools` directory to house the migration script.
…pdate deprecated methods - Added simple and advanced streaming options in SKILL.md for records.list_pages() and execute_pages(). - Updated QueryBuilder to replace records.get() with records.list() in documentation and method calls. - Improved unit tests to validate new streaming functionality and ensure correct method delegation.
- Changed method name from `fetch_xml` to `fetchxml` across the codebase for consistency. - Updated relevant documentation to reflect the new method name. - Added a new example script for FetchXML usage demonstrating various scenarios. - Adjusted unit tests to accommodate the method name change and ensure proper functionality.
…aging improvements
…f QueryResult indexing
…ovals; modify prodev_quick_start.py for additional parameter; enhance pyproject.toml for migration tool; refine migrate_v0_to_v1.py documentation and usage instructions.
…ng and warnings; update migration tool for client variable support and manual review detection.
…agination options - Added `include_annotations` parameter to `_RecordGet` and `_RecordList` classes for OData requests. - Updated `_BatchClient` to handle new parameters in batch operations. - Enhanced `_ODataClient` methods to support `include_annotations`, `expand`, `page_size`, and `count` parameters. - Modified `BatchRecordOperations` to pass new parameters in batch record retrieval and listing methods. - Updated `RecordOperations` to include new parameters for retrieving and listing records. - Added unit tests to validate the new functionality for batch operations and record retrieval. - Implemented migration tool updates to handle changes in method signatures and ensure backward compatibility.
Extract all I/O-free methods from _ODataClient and _BatchClient into new shared base classes (_ODataBase, _BatchBase) so that a future async sibling can inherit the same pure logic without duplicating code. - Add data/_odata_base.py: _ODataBase with URL builders, payload constructors, cache helpers, _RequestContext, _USER_AGENT, _DEFAULT_EXPECTED_STATUSES, _extract_pagingcookie, _GUID_RE, _CALL_SCOPE_CORRELATION_ID, and _http_logger initialisation - Add data/_batch_base.py: _BatchBase with all intent dataclasses, multipart serialisation, response parsing, and the pure table resolvers - _ODataClient and _BatchClient now inherit from the respective base classes and retain only the I/O-dependent methods - Update test patch target for urlparse to _odata_base module - All 1223 unit tests pass; overall coverage remains at 94% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The async batch client will pass an async OData client that inherits from _ODataBase, not _ODataClient. _BatchBase only calls pure _build_* methods defined on _ODataBase, so the broader base type is correct. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add _ODataBase.close() to own cache clearing and logger teardown; _ODataClient.close() now delegates via super() then closes _http only - Reorder _ODataClient bases to (_FileUploadMixin, _RelationshipOperationsMixin, _ODataBase) so mixins are searched before the base, matching Python MRO convention Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…base Helpers moved to _BatchBase (_CRLF, _BOUNDARY_RE, _extract_boundary, etc.) are no longer re-imported by _BatchClient. Two test files updated to import these helpers directly from _batch_base where they now live. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… missing imports to _ODataBase - Remove _SQL_* patterns and _sql_guardrails from _ODataClient (now inherited from _ODataBase) - Add warnings import to _odata_base.py (needed by _sql_guardrails) - Add VALIDATION_SQL_WRITE_BLOCKED and VALIDATION_SQL_UNSUPPORTED_SYNTAX imports to _odata_base.py - Add missing table name lowercasing logic to _ODataBase._build_lookup_field_models
e491241 to
b848324
Compare
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add _BatchContext Protocol to _batch_base.py; re-type BatchRecordOperations, BatchTableOperations, BatchQueryOperations, BatchDataFrameOperations __init__ from BatchRequest to _BatchContext — removes type: ignore on AsyncBatchRequest - Extract _QueryBuilderBase from QueryBuilder with all fluent methods and build(); QueryBuilder inherits base and keeps execute/execute_pages/to_dataframe; AsyncQueryBuilder will inherit base directly, eliminating deprecated sync surface Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
b32987a to
ad9297e
Compare
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
Self (from typing, under TYPE_CHECKING) is the idiomatic replacement for the self-referential TypeVar pattern. With `from __future__ import annotations` already in place, Self is never evaluated at runtime — only used by type checkers. Method signatures in docs now show `Self` instead of the private `_QB` TypeVar. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces AsyncDataverseClient and the complete aio/ async stack mirroring the sync client: _AsyncODataClient (CRUD, SQL, metadata, file upload, relationships), _AsyncBatchClient with _SyncResponseWrapper bridge, _AsyncHttpClient with aiohttp and identical retry/timeout logic, and all async operation namespaces (records, tables, query, files, batch, dataframe). Fixes a cold-start race in _bulk_fetch_picklists by adding asyncio.Lock (double-checked locking pattern) -- concurrent coroutines no longer issue redundant metadata fetches for the same table. Adds pytest-asyncio (asyncio_mode=auto) and aiohttp optional dependency. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AsyncDataverseClient is imported from the module directly: from PowerPlatform.Dataverse.aio.async_client import AsyncDataverseClient Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AsyncBatchRecordOperations, AsyncBatchTableOperations, AsyncBatchQueryOperations, AsyncBatchDataFrameOperations, and AsyncChangeSetRecordOperations were byte-for-byte copies of their sync counterparts with no async methods or I/O. They are removed and the sync classes from operations.batch are imported and used directly. AsyncChangeSet is kept — it needs __aenter__/__aexit__ for `async with batch.changeset()`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ions support - Add retrieve() and list() and list_pages() to AsyncRecordOperations (GA replacements for deprecated get()) - Add expand and include_annotations parameters to _AsyncODataClient._get() and _build_get() to match sync _ODataClient - Add _build_list() to _AsyncODataClient for batch list support - Add _RecordList handling to _AsyncBatchClient._resolve_item() and _resolve_record_list(); fix _resolve_record_get to pass expand and include_annotations - Update test for _resolve_record_get to match new call signature Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dList - TestAsyncRecordRetrieve (11 tests): return type, all params forwarded, 404 → None, non-404 re-raised, ValueError re-raised, no DeprecationWarning, record.id and record.table set correctly - TestAsyncRecordList (15 tests): QueryResult return, multi-page collection, all query params forwarded, FilterExpression → str conversion, to_dataframe(), no DeprecationWarning - TestAsyncRecordListPages (12 tests): async generator type, QueryResult per page, page contents, all params forwarded, no DeprecationWarning - TestResolveRecordGet: 3 new tests covering expand, include_annotations, and combined forwarding to _build_get - TestResolveRecordList (11 tests): dispatch, all _RecordList fields forwarded to _build_list - Add _build_list AsyncMock to _make_batch_client() helper - Add _RecordList import to test file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…atch tests pyproject.toml: - Omit _skill_installer.py (CLI installer, not SDK logic) and extensions/__init__.py (empty placeholder) from coverage measurement tests/unit/aio/data/test_async_batch_internal.py: - TestSyncResponseWrapper: json() returns payload, None, status_code, text - TestExecuteEdgeCases: batch size exceeded raises, json parse failure falls back to empty dict - TestResolveAllEdgeCases: empty changeset silently skipped, non-changeset item resolved and extended - TestResolveItemDispatch: one test per intent type exercising every branch of _resolve_item() (record update/upsert, all table types, sql, unknown) - Add _build_list, _build_create_entity, _build_get_entity, _build_list_entities, _build_create_relationship, _build_delete_relationship, _build_get_relationship, _build_lookup_field_models to _make_batch_client() - Add imports for _SyncResponseWrapper, _MAX_BATCH_SIZE, and all intent types Result: 2091 tests, 96.63% coverage (was 93.64%) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ecated surface - Remove deprecated filter_eq/filter_ne/filter_in methods from async_records and async_query (not in GA API) - Remove stale tests for removed methods - Use asyncio.gather() for independent I/O: relationship sub-requests, batch delete/remove-columns, multi-record label conversion, delete_columns metadata lookups - Fix blocking file I/O in _async_upload: asyncio.to_thread() for reads, Path instead of os.path - Remove lazy imports in _async_relationships; move to module level - Fix test_async_batch: wrap deprecated records.get() call in pytest.warns(DeprecationWarning) - Remove unused Optional import from _async_batch - Remove redundant return None statements in _async_odata Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l() on AsyncQueryOperations - New aio/models/async_fetchxml_query.py: async paging with cookie parsing, max-pages guard, URL-length guard - New aio/models/async_query_builder.py: async execute()/execute_pages() over QueryBuilder fluent interface - aio/operations/async_query.py: add builder() and fetchxml() factory methods with full validation - tests/unit/aio/test_async_query.py: 14 new tests covering builder, fetchxml, execute, execute_pages, and all error paths Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add examples/aio/_auth.py: AsyncInteractiveBrowserCredential wrapper that delegates get_token() to the sync InteractiveBrowserCredential via ThreadPoolExecutor, satisfying the AsyncTokenCredential protocol - Add examples/aio/basic/: installation_example.py, functional_testing.py - Add examples/aio/advanced/: walkthrough, batch, dataframe_operations, relationships, alternate_keys_upsert, fetchxml, sql_examples, prodev_quick_start, datascience_risk_assessment, file_upload - Fix prodev_quick_start: create tables sequentially (Dataverse holds a metadata customization lock per request; concurrent creates fail); retry cleanup loop to handle transient SQL deadlocks All 12 scripts validated end-to-end against a live Dataverse environment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…coverage to 98% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…syncBatchRequest - AsyncQueryBuilder now inherits _QueryBuilderBase instead of QueryBuilder, eliminating deprecated sync execution surface (execute(by_page=...), to_dataframe()) - AsyncBatchRequest drops 4 type: ignore[arg-type] now that batch operation classes accept _BatchContext instead of the concrete BatchRequest Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors sync _ODataClient: reads config.operation_context in __init__ and appends user_agent_context as a parenthesized comment to the User-Agent header. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…) content_type - Add `context: Optional[OperationContext] = None` keyword argument to AsyncDataverseClient.__init__ with the same conflict validation and three-branch config initialization as the sync DataverseClient - Update AsyncDataverseClient docstring to document the context parameter - Add content_type=None to all await r.json() calls in _async_relationships.py for consistency with _async_odata.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirror TestOperationContextClient from test_operation_context.py: - context= kwarg stores OperationContext in _config - no context= leaves operation_context as None - config= + context= together raise ValueError - config= alone wires operation_context correctly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds _HttpResponse to _async_http.py — a materialized response that buffers the body bytes in _request, exposing sync .text and .json() so callers need no await. Eliminates _SyncResponseWrapper from _async_batch.py and removes all remaining await r.json() calls throughout the async layer (_async_odata.py, _async_relationships.py). Also removes aiohttp.ContentTypeError from except clauses now that json() is a plain json.loads call (ValueError covers it). All 2155 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- async_fetchxml_query.py: fix leftover await r.json() -> r.json() - _async_odata.py: add TYPE_CHECKING guard for aiohttp annotation - test helpers: replace AsyncMock with MagicMock for .json/.text on _AsyncResponse-compatible mocks (body already materialized, no await) All 2155 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ase) Follows the same cleanup applied to the sync _ODataClient on the refactoring branch. _operation_context is now initialized once in _ODataBase.__init__ and inherited by both sync and async subclasses. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
500c19f to
55e3359
Compare
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Warning
ADO PR pipeline YAML change detected
This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.
Action required (post-merge): Re-enable / approve the updated YAML for:
- DV-Python-SDK-PullRequest (definitionId=29922)
- https://dev.azure.com/dynamicscrm/OneCRM/_build?definitionId=29922
Please resolve this comment after completing the post-merge steps.
aio/async package mirroring the sync SDK:_AsyncHttpClientwrappingaiohttpwith identical retry, backoff, and timeout logic_AsyncAuthManagerfor async Azure Identity token acquisition_AsyncODataClient— full CRUD, SQL-over-API, table/column metadata, file upload, and relationship operations_AsyncBatchClientwith_SyncResponseWrapperbridging the async HTTP response to the shared sync multipart parser in_BatchBaserecords,tables,query,files,batch,dataframe— all mirroring their sync counterpartsAsyncDataverseClientwith lazy init, async context manager, and session lifecycle managementpytest-asyncio(asyncio_mode = auto) andaiohttpas an optional dependency (pip install PowerPlatform-Dataverse-Client[async]).Test plan
black --checkpasses on all files