Data preparation services
Overview
This service provides a web interface for configuration and scheduling of automated data scrapers (containers) capable of accessing data from known or custom sources such as Copernicus, ARSO, SURS, MKGP, and GURS. The service allows a user to configure time‑based file retention policy for each configured scrapper and configuration of the destination directory within the user’s workspace. The service provides tools for live monitoring of the scrapper container execution process. Implemented scrappers utilize SLAIF storage service and database APIs for data storage and housekeeping of already collected files to avoid scrapping duplicate files. The configuration and scheduling are based on top of the existing kubectl tools.
Outcome
A correctly configured scrapper is executed at scheduled intervals where it downloads the data from a public source and transfers it into the user’s workspace.
Service Card
| Category | Infrastructure and data-related services |
| Domain | Automated data acquisition and preparation |
| Vertical | Cross-domain |
| Target users | Data engineers |
| Service type | Platform / Library / Service |
| Outputs | — |
| Requirements | — |
| Limitations | — |
| Availability | Q4/2026 |
| Licensing | — |
| Visual content | assets/service-visuals/data-preparation-services-real.png |
| Visual caption |