Situation

A client in public administration developed multiple distributed AI applications in PySpark, initially running in local mode.

Task

The client requested the deployment of these AI applications to a production environment on a dedicated compute cluster running Kubernetes.

Action

I containerized the local PySpark AI applications and configured the necessary Kubernetes resources for a successful production environment. This included:

• Creation of deployments, external secrets, network policies, and helm charts.

• Model monitoring and model serving via mlflow.

• Optimization of compute resources through automatic pod autoscaling.

• Scheduling of batch jobs with Apache Airflow.

Result

PySpark AI-Applications were successfully deployed to the production environment and became available for end-user consumption, ensuring fault tolerance, scalability and reliability.