A forecasting model made decisions based on series from roughly 10,000 external sources in different formats (XML, Excel, CSV), where data arrived with gaps, outliers and no context for its origin, with record counts running into the billions.
We built a pipeline of independent Python parser microservices that normalize the mixed formats in parallel, load them into a time-series store without duplicates alongside each object's passport, and assemble model-ready datasets from the cleaned series.
Parsing stopped being a sequential bottleneck - sources are processed in parallel, and the forecast comes back into the system as a ready machine-readable order, reports and charts instead of manually reconciling files.
Tomorrow's decision starts far from the model
Every day the organization decides whether to buy or sell an asset for the next day or two. An error in the forecast quickly turns into real money, so the model's output has to rest on data you can trust.
Before it reaches the model, this data travels a long road. It arrives from roughly 10,000 external sources, and the record count runs into the billions from the very start. One object sends XML, another Excel or CSV. The measurement frequency varies, the series contain gaps and outliers, and the number alone, without context, does not explain where and what was measured.
Our part of the project sat exactly here: between the raw export and the finished dataset. We had to gather dozens of gigabytes of heterogeneous history, bring it to a common form, and not lose its meaning along the way.
Let's take a single measurement and run it through the whole pipeline
An incoming file or a portion of data arrives from an external object. First a specialized parser recognizes the format, extracts the time series, and links it to the source. Then the data goes through cleaning, lands in the time-series storage, and turns into vectors you can compute formulas over. From the ready vectors a specialist assembles a dataset for the model.
After the computation the road continues: the forecast returns to the system, appears in reports and charts, and then becomes a structured order file for an external receiving system. Value arises only when the number reaches a concrete action.
Ten thousand sources cannot be parsed sequentially
Processing such a volume in sequence would take hours, so parsing is split into independent Python microservices distributed across different servers. Sources are processed in parallel, and the failure of one parser does not require stopping the whole flow.
Each format is isolated from the rest of the system. If one supplier changes the file structure, its parser is adjusted; the storage, the cleaning, the vector layer, and the models keep working with the same internal representation.
Loading is idempotent. A resent file or a repeated run does not create a second copy of the same records. Before loading, the system checks what has already been processed and does not pull parsed data into the database again. This makes it safe to restart jobs after a failure without clearing the result by hand.
The number gets an object passport
A time series stores a sequence of values, but the meaning lives right next to it. For every object there is a passport: name, coordinates, and attributes that explain where the measurement came from and its place in the overall structure.
Without such a passport, two identical numbers may belong to different objects and describe different processes. That is why the storage connects the history of measurements to reference tables, rather than piling everything into one wide table without context.
A specialized time-series database keeps years of history with daily chunks and compression. Aggregates for charts are built on request, so there is no need to create yet another copy of the data in advance for every new period.
| Layer | What it preserves | Why it matters to the next stage |
|---|---|---|
| Object passport | Name, coordinates, and attributes | The series does not lose its origin and meaning |
| Time series | Values over time | History can be analyzed and aggregates built |
| Clean vector | A comparable series free of technical noise | Formulas and models get a single representation |
| Dataset | Selected source and derived features | The model trains on a set tailored to a specific task |
Cleaning has to tell an error from a meaningful spike
Measurements inevitably contain gaps and values that stand out from the rest of the series. Before the computations, the pipeline fills the gaps and removes outliers so that a random dropout or a technical jump does not distort the training.
But aggressive cleaning is no less dangerous than dirty data. A real spike may be exactly the event that affects the forecast. That is why preparing a series has to remove technical noise and at the same time preserve meaningful changes in the underlying process.
A clean series does not mean a perfectly smooth one. If you flatten every sharp change, the events the forecast is built for will disappear along with the errors.
This work happens before the model ever sees the data. As a result, the specialist compares comparable series rather than explaining to the algorithm why one source is missing an hour while another accidentally shows an impossible value.
A formula assembles a new series without a manual export
The cleaned series are brought to a single vector representation. You can perform arithmetic over the vectors: add, subtract, multiply, and divide one series by another. This is how derived features emerge from the source measurements.
The calculator stores the formula and runs it over the series right inside the data loop. The analyst does not have to export dozens of files, align columns by hand, and repeat the same work for the next period. They pick the source and derived vectors they need and assemble a dataset for a specific model.
The machine-learning layer was handled by a separate specialist. The pipeline handed over the prepared datasets and took the forecast back. This separation matters: the quality of preparation can be developed independently of any specific model, and the model does not have to solve the collection and cleaning problem from scratch every time.
The forecast comes back as a ready-made order
The result of the computation is used in reports and charts, but the process does not end there. The system builds a machine-readable order - a structured file that an external system accepts. The forecast does not stay a cell that a person has to move by hand and reinterpret from scratch.
The manual chain of "gather files, parse, clean, combine, compute" turns into a repeatable pipeline. The specialist works with a library of series and formulas, the model receives a prepared dataset, and the outcome reaches the decision-making loop.
Beyond a day or two the uncertainty grows
The working horizon of the forecast is the next day or two. The system can compute a more distant period too, but accuracy naturally declines along with the horizon. This is a property of the original task, not a defect of the storage or the pipeline.
The pipeline itself also does not turn a bad source into a good one automatically. Its result depends on the completeness of the data, the correctness of the passports, and the cleaning rules. It surfaces and prepares the available history, but it cannot restore meaning that the source never conveyed in the first place.
At the same time, the architecture allows growth without replacing the foundation: connecting new sources and formats, expanding the library of formulas, new datasets, and a tighter link with models and external systems. Every new format gets its own parser, while the already-cleaned series keep traveling the common road.
Oleksandr
This is a real project, presented anonymized. We deliberately change the identifying details - the industry, the specifics, anything that could point to the client - and we don't reveal the client's name or trade secrets. What stays intact is the substance: the problem we solved and the engineering approach behind it. This case study is here to show the problem and its solution - what we do and how.