Feature Request: Streaming Response

[x2764tech]

As part of a project I'm working on, I'd like to use this library to download form data to files.
I really like that the API adds a load of useful fields to the export_records method, and I'd like to continue to use that.
However, I'd also like not to have to load the entire response into memory, as I don't know how large the responses will get for our 20,000+ cohort.
Is there a way to implement the Body Content Workflow, or would this require a lot of work?

[pwildenhain]

I think this could be a great contribution. I don't currently have the bandwidth, but you're welcome to take a crack at it 👍🏻

A potential hack in the meantime could be using the fields parameter in export_records to export just the record_id field for the whole project. Then from there you can manually chunk your requests to, say, 1000 records at a time for a 20,000 record cohort, by passing a list of records to the records parameter in export_records.

[njvack]

If your request is big enough to clobber your client, it's also going to do a number on the server.

I would recommend using date_begin and date_end to page through the data in more manageable chunks. We're currently testing out an incremental exporter that should just nab records added / updated since the previous run (plus some buffer) and then merging that into the main dataset.

[x2764tech]

@njvack What volume of data are you looking at? How do you handle data changes that might happen during the download?

I'm looking at using FaaS to download data, so making the compute requirements as small as possible (512MB instances) is still a consideration.
Also, providing flexibile transport implmentation might provide a route to asyncio... again not a major concern for the export situation, but a consideration in an interactive system that uses 'small' queries to update a bulk download...

[njvack]

• ~100k rows across ~1500 columns?
• We'll handle data changes that might happen during download by overlapping downloads and merging everything. I don't have any particular confidence that a bulk export happens in a transaction in REDCap anyhow

But really, if you want streaming... pycap is not a particularly heavy wrapper around the API and just using httpx or something is maybe what you want. pycap is great because it'll give you, like, a dataframe and handle nice data types but if you're doing something close to the metal it's okay to be close to the metal

[njvack]

For what it's worth: We're going to switch to the incremental downloads at the request of our admins. The full dump of the project is currently 165M so yeah I can see why they might not be excited about us nabbing the whole thing every day