Batch processing

[0passion0]

Does Pathway support batch writing of data? When dealing with large volumes of data, using on_change affects performance. Is there a way to retrieve complete multi-row data at once?

[szymondudycz]

We do not currently support batch writing of data. In your case, is the on_change method itself causing issues with performance or is it writing data one by one? If the latter, you can consider adding buffers yourself. (And if the former, please let us know - that will help us deciding that batching would be an important addition in the future)

Also, what are you writing data to? We are constantly adding new connectors (integrated in rust, which makes them faster), maybe the one you want is on the roadmap, and if not we'll consider adding it to the roadmap.

[0passion0]

Thank you. We are currently using MySQL and MongoDB as our main databases.
I was wondering, since MySQL is relatively common, why isn’t there a connector for it?
Also, when storing relational data with frequent writes, how is transactionality ensured?
Is this something you are considering implementing?

[zxqfd555-pw]

Hi @0passion0,

Let me break the response into several parts.

1. MongoDB connector for output:
   We already have a connector for MongoDB output. You can check it out at this link.

2. MySQL connector:
   We don’t have a MySQL connector yet because no users or clients have requested it. However, adding it is straightforward, and I’ve created an issue to track this feature. You can track the progress in the issue here. It will likely be added in a matter of weeks, in one of the next releases.

3. Transactional consistency of output data:
   This is a bit more complex. While databases support transactions across multiple tables, most storage backends where we output data do not - as an example, a file system or lightweight queues like NATS. Therefore, on output, each table is independent, and transactions are handled separately for each one. To ensure data consistency across tables, you might want to concatenate them or otherwise make them a part of a single table. Then, there is no need to synchronize writes to several tables.

4. Default commit behavior:
   When we write a table to MongoDB, or PostgreSQL (not exactly what's requested, but similar in behaviour), the default behaviour is that all data corresponding to the same minibatch, that is, with the same time, is written in a single transaction. That is, the output for a single table is always consistent. You can also assume that if you have several tables and all of them are written up to a certain time T, then all data with times up to T must be consistent between these tables, too.