What actually happens when you send a folder from a browser

When you pick a folder with Add folder, the browser hands us a directory handle, not a list of files. Walking it is I/O, and doing that on the main thread for hundreds of thousands of entries freezes the page. So the walk runs in four Web Workers.

The first version split top-level folders between workers by position in the listing. That dropped a folder of a thousand files in one test, because listing order is not stable between calls. Workers now claim top-level folders by a hash of the folder name, which is stable, so every folder belongs to exactly one worker and every file is read once.

A request per file is the classic small-file mistake: each one pays connection setup, headers, a round trip and a database touch on the node. Files below a size threshold are instead packed into a batch of up to a thousand, assembled in the worker as a single multipart body from one buffer, and sent as one request. The node looks up all paths in the batch with one query and writes them into the package bundle at their pre-assigned offsets.

The browser keeps six batches in flight. Three was too few to fill a typical uplink when each batch spent time waiting on the node; twelve gained nothing and made the progress display jumpy. Six was the point where the uplink stayed busy on the connections we tested.

Before we looked at the node side, the upload of a 420,000-file folder ran at under 1 MB/s on a connection whose uplink is about 7 MB/s. The bytes were not the problem. The node's per-request origin check, used to answer the browser's cross-origin preflight, hydrated the whole session, all 420,000 paths, to answer a question that needed four fields. Reading a session summary instead removed most of the per-batch cost.

The progress display, incidentally, now reports a rate over an eight-second window once a second. The old display updated several times a second from a two-second window and looked like it was flickering between two different uploads.