CLIO Runtime Overview
This section is aimed at developers building modules (ChiMods) or integrating with the CLIO Runtime programmatically.
What is the CLIO Runtime?
The CLIO Runtime is the task-execution engine at the core of IOWarp. It provides:
- A shared-memory task queue that clients push work into.
- A pool of worker threads that poll for and execute tasks.
- A module system (ChiMods) for extending the runtime with custom logic.
- Distributed routing so tasks can transparently execute on remote nodes.
Applications never call Module code directly. Instead they submit tasks via a client library; the runtime schedules and executes them.
Core Abstractions
Pools and Containers
A pool is a named, addressable group of containers spread across one or more nodes. Every Module instance is backed by a pool. Pools are identified by a PoolId (e.g., "512.0") and a human-readable pool_name.
Each pool is made up of one or more containers, numbered sequentially from 0 to N-1. A container is a single instance of a Module on a specific node. Each container maps to exactly one node, but a single node may host multiple containers from the same pool.
Pool "cte_main" (id 512.0)
┌──────────────────────────────────────────┐
│ │
Client A ──task──> Container 0 Container 1 Container 2│
│ │ │ │ │
Client B ──task──> │ │ │ │
│ │ │ │ │
Client C ──task──> │ │ │ │
└──────┼──────────────┼──────────────┼──────┘
│ │ │
┌────▼────┐ ┌─────▼────┐ ┌─────▼────┐
│ Node 0 │ │ Node 1 │ │ Node 2 │
└─────────┘ └──────────┘ └──────────┘
In this example, each node hosts one container. In a configuration where there are more containers than nodes, multiple containers map to the same node:
Pool "my_pool" (4 containers, 2 nodes)
┌─────────────────────────────────────────────────────┐
│ Container 0 Container 1 Container 2 Container 3│
└───────┼────────────┼──────────────┼───────────┼─────┘
│ │ │ │
┌────▼────────────▼───┐ ┌─────▼───────────▼──┐
│ Node 0 │ │ Node 1 │
│ (containers 0, 1) │ │ (containers 2, 3) │
└─────────────────────┘ └────────────────────┘
Container numbering is sequential: containers are assigned IDs 0, 1, 2, ... in the order they are created. This means PoolQuery::DirectHash(key) routes to container = key % num_containers, which in turn maps to a known physical node via the pool's address table. This is how hash-based load balancing works across a cluster.
When you add an entry to the compose section of the configuration file, the runtime creates a pool (and its containers) for that module at startup.
Containers provide a set of callbacks:
| Callback | Trigger |
|---|---|
Create | Pool creation — initialize module state. |
Destroy | Pool teardown — clean up resources. |
| User-defined methods | Client tasks (e.g., Put, Get). |
Monitor | Periodic or event-driven scheduling hooks. |
| Recovery callbacks | Automatic crash recovery and container migration. |
Tasks
A task is a unit of work submitted by a client to a pool. Tasks carry:
- A target pool (identified by
PoolId). - A pool query that determines which container handles the task (see below).
- Serialized input parameters and space for output results.
Tasks are placed into shared-memory queues and picked up by worker threads. From the client's perspective, task submission is asynchronous — you get back a Future that you can Wait() on.
Pool Query (Routing)
The PoolQuery attached to a task controls how the runtime routes it to a container:
| Mode | Description |
|---|---|
| Local | Execute on the container local to the submitting node. |
| DirectId | Route to a specific container by its ID. |
| DirectHash | Hash a value to select a container (consistent-hashing style). |
| Range | Fan out to a contiguous range of containers. |
| Broadcast | Send to every container in the pool (all nodes). |
| Physical | Route to a specific physical node by node ID. |
| Dynamic | Routes through the Monitor for cache-optimized scheduling. Recommended for Create operations. |
In compose files, pool_query is typically set to local (single-node) or dynamic (multi-node).
Internal Architecture
Workers
Internally, the runtime runs a fixed number of worker threads (configured by runtime.num_threads). Each worker:
- Is pinned to a CPU core by the scheduler.
- Polls its task queue for pending work.
- Executes tasks by dispatching to the target container's method.
- If no work is available, busy-waits for a configurable period (
first_busy_wait) before sleeping.
Workers use cooperative multitasking: a long-running task can yield (via task->Yield()) to let the worker process other tasks, and will be resumed later from the blocked queue.
Task Lifecycle
Client Runtime
│ │
│ AsyncCreate(pool, ...) │
│ ─────────────────────────>│ Task placed in shared-memory queue
│ Future<T> │
│ │ Worker picks up task
│ │ Dispatches to Container method
│ │ Container executes & writes results
│ │
│ future.Wait() │ Client busy-waits on completion flag
│ <─────────────────────────│
│ Results available │
Distributed Execution
When a task's PoolQuery resolves to a remote node, the runtime serializes the task and sends it over ZeroMQ (or Unix domain socket, depending on CLIO_IPC_MODE). The remote node executes the task and returns the result. This is transparent to the client.
Compose Files
The compose section of the configuration file tells the runtime which modules to load and how to initialize them at startup. Each entry maps to a GetOrCreatePool call.
compose:
- mod_name: clio_cae_core
pool_name: clio_cae_core_pool
pool_query: local
pool_id: "400.0"
| Field | Description |
|---|---|
mod_name | Name of the shared library to load. The runtime looks for lib<mod_name>.so on the library search path. For example, clio_cae_core loads libclio_cae_core.so. |
pool_name | A user-chosen name for the pool. Must be unique within the runtime. |
pool_query | How the pool should be created — typically local for single-node or dynamic for multi-node. |
pool_id | A unique identifier for the pool in "<major>.<minor>" format. |
Any additional YAML keys in the entry (e.g., storage, dpe, targets for CTE) are passed as opaque configuration to the module's Create method.
Custom Modules
You can compose your own ChiMods the same way. Build a shared library following the Module Development Guide, install it, and add an entry to compose:
compose:
- mod_name: my_custom_module
pool_name: my_pool
pool_query: local
pool_id: "600.0"
# any module-specific keys here
The runtime will dlopen libmy_custom_module.so, call its Create method with the YAML block, and the module is ready to receive tasks.
Next Steps
- Configuration Reference — Full parameter reference.
- Module Development Guide — Build your own Module.