Data Processing Services

Opis

The Data Processing Service provides scalable and distributed capabilities for executing large-scale data processing workloads within the SLAIF AI infrastructure. Users can process datasets stored in SLAIF storage systems, data lakes, or external sources connected through data access services, within a unified environment combining cloud-native compute resources and HPC infrastructure. Batch processing supports large-scale transformations, aggregation, and feature engineering workflows required for machine learning training. Real-time and streaming capabilities enable analysis of data generated by industrial systems, sensors, or IoT platforms, supporting near-real-time decision-making. The service integrates with SLAIF storage and AI pipelines, enabling seamless transitions between data ingestion, transformation, model training, and inference workflows. Key technologies include OpenStack compute and storage services for scalable compute resources and persistent storage, Kubernetes for scalable orchestration of processing workloads, Kubeflow Pipelines for integrating data processing tasks within AI training pipelines, and S3-compatible object storage systems (such as Ceph) for data lake integration.

Pričakovani rezultati

Users can execute scalable data processing pipelines capable of performing large-scale batch processing of datasets and integrating processed data directly into machine learning pipelines. The service enables efficient preparation and transformation of datasets required for AI model training and analytics applications within SLAIF.

Kartica storitve

Kategorija	Infrastructure and data-related services
Domena	Data Infrastructure, Scalable Data Processing
Vertikala	Cross-domain
Ciljni uporabniki	Data Engineers, Data Scientists, Platform Administrators, SMEs, Research Institutions
Tip storitve	Data Processing Platform
Rezultat
Zahteve
Omejitve
Na voljo od	Q3/2026
Licenca
Slika	assets/service-visuals/data-processing-services-real.png
Oznaka slike