Deployment
Platforma consists of two main software components: a desktop application for end-users and a backend system that persists user data, orchestrates calculations and manages the data storage. It is common for an organization to have multiple users and multiple desktop applications installed, while maintaining a single instance of the backend deployed within the organization's infrastructure.
This section provides comprehensive guidance on the installation, configuration, and maintenance of the pl
-backend service, as well as block and software registries.
What is pl
-backend?
The core of Platforma is its backend, known as pl
. It performs essential functions such as running pipelines, managing files, and storing projects created by users, among many other tasks. The desktop application continuously interacts with pl
, allowing users to perform calculations and store data remotely while managing these processes locally from their PCs. The pl
-backend is composed of several modules, each with distinct responsibilities.
Core
The core of the backend serves as a persistence layer for Platforma's internal state. It maintains all information about projects, blocks, active pipelines, files, and more. It also manages the lifecycle of resources such as working directories, software executions, temporary files, and data in object storage. Under the hood, the core is a high-performance graph database built on top of RocksDB, designed to efficiently track and manage the computational and storage resources critical to modern computational biology applications. For each hardware resource in user projects, the core maintains a corresponding database record linked to that resource. Consistency between the actual hardware resources and the records in the core database is ensured by the controllers.
Controllers
Controllers are relatively straightforward components that connect real-world resources with the backend's core state. The are two key controllers:
-
Data Controller: Responsible for interactions with data storage systems such as S3, local file systems, scratch space, and others. The Data Controller prepares working directories for software runs, indexes files, imports data from external sources, and more.
-
Runner Controller: Manages the execution of software within computational workflows. It provides the standard interface for running "some command" in "some execution environment". When a user runs a block in the desktop app and the underlying workflow is rendered, the Runner Controller actually schedules the individual software jobs described in the workflow.
-
Packages Controller: Retrieves packages with assets and software for execution. It provides the standard interface for downloading and installing packages from remote registries. These packages can contain JAVA runtime or Python interpreter, that are used in worflows, or genome references for genetic data analysis.
Both the core and controllers are "passive," meaning they primarily react to changes initiated externally. When a user performs an action in the app, these changes are sent to the core, which then communicates them to the controllers to carry out the necessary tasks.
Deployment environments
While there are no specific restrictions on how the backend can be deployed, several common deployment environments include:
-
Single Server: The backend runs on a remote server, with all calculations performed locally on that server.
-
Kubernetes: The backend operates within a Kubernetes cluster, utilizing available object storage and the Kubernetes scheduler to store data and perform calculations.
-
Cloud: The backend runs in a cloud environment, using cloud-native object storage, scratch space, and batch systems for data storage and calculations. Configuration can vary significantly depending on the cloud provider.
-
HPC Cluster: The backend operates on a remote server with access to an HPC cluster, performing calculations within the cluster managed by a batch system such as Slurm.
- Local: backend runs on the same host, where Platforma's desktop app is used. This is what desktop app does in the 'local' mode: it starts the instance of backend, configured for this mode and uses that instance.
See our Deployment Guides for real examples.
Registries
There are two external components used by the backend to perform calculations:
-
Block registry: is an external registry with the source code of the blocks. When user adds a block into the project, block's workflow, user interface and model are downloaded from the corresponding block registry.
-
Software registry: is the place from where the software used in the workflow is stored in either binary or container format. When the workflow is executed, the backend downloads software used in the workflows from the corresponding registry.
By default Platforma uses pl-open
block & software registries. One can configure to use multiple registries or setup a private registries to keep proprietary blocks.
Next steps
📄️ Getting Started
Getting Started with deployment configuration
🗃️ Deployment Guides
3 items
🗃️ Advanced topics on backend configuration
5 items
📄️ Backups
Setting up backups of Platforma state