A browser robot for thousands of records on someone else's shifting sites

By the hundredth form, your attention is not what it was on the first

On an external web portal you have to open an account, fill in a dozen fields, wait for a check, upload a document, and carry the answer back. There is almost no decision-making here: the same actions repeat for every record.

A single form takes a few minutes. When there are thousands of profiles and tens of thousands of documents, manual work stretches into months. Toward the end of a long queue, typos, skipped fields, and mismatches between systems creep in. Hiring a person just to copy data between browser windows would mean scaling the drudgery itself.

We handed this route over to a controlled browser built on Python and Playwright. The data is entered once, the robot goes to the external site, fills in the form, gets the result, and returns a status. One record and a thousand records run the same scenario, but behind the large volume there is no operator clicking "next" by hand.

A single request goes through without a human at the screen

The request comes in through the customer's interface. Before the browser launches, the system checks the required fields, formats, and dates: there is no point sending an empty or clearly invalid record to someone else's portal.

After the check, the task lands in a Celery and Redis queue. Workers pick up jobs as they become available and run them in the background. The user does not need to keep a page open or watch each step; they submit the data and get the result.

The queue separates the incoming flow from the speed of the external sites. If a portal responds slowly, jobs wait their turn to be processed without tying up a workstation or blocking the intake of the next records. A slowdown in the external service does not lead to a lost record: it stays in the queue until launch. The robot can run in batches, at night, without breaks, adapting to the pace of a service we do not control.

The simplest step is filling in the known fields

Describing the smooth scenario looks short: open the page, find the field, enter the value, press the button, read the response. The trouble starts after the form is submitted.

Someone else's site may spend a long time rendering the page, show an unexpected window, return an error, return nothing, or change the interface structure without warning. Two consecutive requests sometimes get different responses, even though the robot performs identical actions.

That is why the scenario describes more than just the happy path. For every expected state a reaction is defined: wait for an element, extract the needed result, recognize a known error, retry a transient failure, or stop the task for review. Before going to a large volume, these branches are checked with many runs.

A live external site can still show a variant that was never there before. The goal of the system is not to promise "it never fails," but to make sure a single non-standard response does not lose the task or stop the other records.

A transient failure gets three attempts

If a run does not finish, the task is launched again. After the second failure the robot tries a third time. This closes out transient problems: the page did not load in time, the service hung for a minute, or the response arrived with a delay. Most often the second or third launch goes through without intervention, because by then the external site is responding normally again.

Manual review is rarely reached. But it is exactly this exit that keeps the automation from turning an error into a silent lost record: the task neither disappears nor hangs in the queue forever.

Every run is saved in full. The team opens its history and sees at which step, field, and page state the robot stopped. The next fix rests on a record of the real failure, not on an attempt to reproduce the problem from the user's retelling.

If a site has rebuilt a page, the scenario usually adapts within a single day. The record shows the point of breakage, so there is no need to rewrite the whole system or guess where the interface changed.

A new portal gets its own scenario, but not a new platform

The browser drudgery consists of repetitive operations: create a record, update data, attach a document, get a result, sync a status, resolve a mismatch. These operations run on top of a shared foundation with a queue, data validation, retries, and a record of the runs.

For the next site, its sequence of pages and fields is described. The timeline depends on the complexity of the scenario: the robot masters a simple site with a few predictable screens in a few days. The general task handling and result return do not need to be built again.

The approach applies where manual work can be broken down into stable steps. If a process requires a professional judgment on every screen, the robot should not pretend to be a human: the disputable case stays with a specialist.

Five thousand profiles show the difference better than a demo

The solution ran through a live flow of roughly 5000 profiles and tens of thousands of documents with supporting data, without stopping the main process. The flows of different customers stay isolated and do not mix.

Updating and checking one profile by hand takes about three minutes. Uploading a document together with its status is another two or three. Across the whole volume this is more than 2000 person-hours: over a year for a single operator, or several months for a small team.

The robot does not spend an instant on an operation either: the external site still sets the speed. A profile is processed in about two minutes, a document in about one. The gain comes from working in the background, a parallel flow, no breaks, and the same attention paid to the first record and the thousandth. The entire volume is closed out in a matter of days.

The difference shows not on a single operation but across the whole volume:

The operator stops carrying data between windows and steps in where a new response, a disputable result, or a changed site needs review.

Sensitive data is kept separate between customers

Personal data is being processed, so the flows have separate scopes of visibility. Each customer gets access only to their own records, and the data of different projects does not mix.

The system also does not store anything extra. The task and the run record hold the information needed for the specific scenario and the later review. The customer is not shown the internal queue and technical retries: their interface stays deliberately simple - submit the data and get the result. Watching the course of the runs remains the team's internal work.

Someone else's site stays someone else's

The scenario needs maintenance, because the portal owner can change a page without notice. That kind of adaptation is built into operations, not treated as an unexpected defect of the robot.

The robot also does not guarantee the availability of the external service. The queue and retries survive a transient failure, but they cannot force someone else's site to respond or to keep its previous markup. After a persistent error, a human still makes the decision.

The boundary of the project is therefore clear: the machine performs a formalized browser routine, handles a large volume the same way, and leaves a detailed trail. The human stays where the external site has changed the rules or where a response cannot be interpreted in a predefined way.