Module Developer Guide

Table of Contents

1. Overview
2. Architecture
3. Coding Style
4. Module Structure
   • Task Definition
   • Client Implementation
   • Runtime Container
   • Dynamic Scheduling (ScheduleTask)
5. Configuration and Code Generation
6. Task Development
7. Synchronization Primitives
8. Pool Query and Task Routing
9. Client-Server Communication
10. Memory Management
    • CLIO_IPC Buffer Allocation
    • Task Data Structures
11. Build System Integration
12. External Module Development
13. Example Module

Overview

CLIO Runtime modules (ChiMods) are dynamically loadable components that extend the runtime with new functionality. Each module consists of:

• Client library: Minimal code for task submission from user processes
• Runtime library: Server-side execution logic
• Task definitions: Shared structures for client-server communication
• Configuration: YAML metadata describing the module

Header Organization: All Module headers are organized under the namespace directory structure (include/[namespace]/[module_name]/) to provide clear namespace separation and prevent header conflicts.

Architecture

Core Principles

1. Client-Server Separation: Clients only submit tasks; runtime handles all logic
2. Shared Memory Communication: Tasks are allocated in shared memory segments
3. Task-Based Processing: All operations are expressed as tasks with methods
4. Zero-Copy Design: Data stays in shared memory; only pointers are passed

Key Components

Module/
├── include/
│   └── [namespace]/
│       └── MOD_NAME/
│           ├── MOD_NAME_client.h     # Client API
│           ├── MOD_NAME_runtime.h    # Runtime container
│           ├── MOD_NAME_tasks.h      # Task definitions
│           └── autogen/
│               └── MOD_NAME_methods.h    # Method constants
├── src/
│   ├── MOD_NAME_client.cc        # Client implementation
│   ├── MOD_NAME_runtime.cc       # Runtime implementation
│   └── autogen/
│       └── MOD_NAME_lib_exec.cc  # Auto-generated virtual method implementations
├── clio_mod.yaml              # Module configuration
└── CMakeLists.txt                 # Build configuration

Include Directory Structure:

• All Module headers are organized under the namespace directory
• Structure: include/[namespace]/[module_name]/
• Example: Admin headers are in include/[namespace]/admin/ (where [namespace] is the namespace from clio_repo.yaml)
• Headers follow naming pattern: [module_name]_[type].h
• Auto-generated headers are in the autogen/ subdirectory
• Note: The namespace comes from clio_repo.yaml and the chimod directory name doesn't need to match the namespace

Coding Style

General Guidelines

1. Namespace: All module code under clio::run::MOD_NAME

2. Naming Conventions:

   • Classes: PascalCase (e.g., CustomTask)
   • Methods: PascalCase for public, camelCase for private
   • Variables: snake_case_ with trailing underscore for members
   • Constants: kConstantName
   • Enums: kEnumValue

3. Header Guards: Use #ifndef MOD_NAME_COMPONENT_H_

4. Includes: System headers first, then library headers, then local headers

5. Comments: Use Doxygen-style comments for public APIs

Code Formatting

#include <clio_runtime/clio_runtime.h>

// Real module code lives in `namespace clio::run::MOD_NAME { ... }`.
/**
 * Brief description
 *
 * Detailed description if needed
 * @param param_name Parameter description
 * @return Return value description
 */
class ExampleClass {
 public:
  // Public methods
  void PublicMethod();

 private:
  // Private members with trailing underscore
  clio::run::u32 member_variable_;
};

Module Structure

Task Definition (MOD_NAME_tasks.h)

Task definition patterns:

1. CreateParams Structure: Define configuration parameters for pool creation
   • Uses cereal serialization for programmatic creation (direct API calls)
   • Implements LoadConfig() for declarative creation (compose mode from YAML)
   • Must define chimod_lib_name static string for module loading
2. CreateTask Template: Use GetOrCreatePoolTask template for container creation (non-admin modules)
3. Custom Tasks: Define custom tasks with SHM/Emplace constructors and HSHM data members

// MOD_NAME_tasks.h — contents shown at `namespace clio::run::MOD_NAME` scope.
#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/autogen/MOD_NAME_methods.h>
#include <clio_runtime/config_manager.h>  // clio::run::PoolConfig
#include <yaml-cpp/yaml.h>
// Include admin tasks for GetOrCreatePoolTask
#include <clio_runtime/admin/admin_tasks.h>

/**
 * CreateParams for MOD_NAME chimod
 * Contains configuration parameters for MOD_NAME container creation
 */
struct CreateParams {
  // MOD_NAME-specific parameters
  std::string config_data_;
  clio::run::u32 worker_count_;

  // Required: chimod library name for module manager
  static constexpr const char* chimod_lib_name = "clio_MOD_NAME";

  // Default constructor
  CreateParams() : worker_count_(1) {}

  // Constructor with parameters
  CreateParams(const std::string& config_data,
               clio::run::u32 worker_count = 1)
      : config_data_(config_data), worker_count_(worker_count) {}

  // Serialization support for cereal
  template<class Archive>
  void serialize(Archive& ar) {
    ar(config_data_, worker_count_);
  }

  /**
   * Load configuration from PoolConfig (for compose mode)
   * Required for compose feature support
   * @param pool_config Pool configuration from compose section
   */
  void LoadConfig(const clio::run::PoolConfig& pool_config) {
    // Parse YAML config string
    YAML::Node config = YAML::Load(pool_config.config_);

    // Load module-specific parameters from YAML
    if (config["config_data"]) {
      config_data_ = config["config_data"].as<std::string>();
    }
    if (config["worker_count"]) {
      worker_count_ = config["worker_count"].as<clio::run::u32>();
    }
  }
};

/**
 * CreateTask - Initialize the MOD_NAME container
 * Type alias for GetOrCreatePoolTask with CreateParams (uses kGetOrCreatePool method)
 * Non-admin modules should use GetOrCreatePoolTask instead of BaseCreateTask
 */
using CreateTask = clio::run::admin::GetOrCreatePoolTask<CreateParams>;

CreateParams Dual-Mode System

CreateParams supports two modes of pool creation, controlled internally by BaseCreateTask::GetParams():

1. Programmatic mode (default): When you call AsyncCreate() from the client API, CreateParams is serialized with cereal and sent as binary data. GetParams() deserializes it directly.

2. Compose mode: When pools are created from YAML configuration (via the compose feature), GetParams() detects the compose flag and instead deserializes a clio::run::PoolConfig struct, then calls your LoadConfig() method to populate the parameters from the YAML config string.

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/admin/admin_tasks.h>
#include <clio_runtime/config_manager.h>
// Internal logic in BaseCreateTask::GetParams() (admin_tasks.h). do_compose_ and
// chimod_params_ are BaseCreateTask members; shown here as parameters.
template <typename CreateParamsT>
CreateParamsT GetParamsExample(bool do_compose,
                               const clio::run::priv::string& chimod_params) {
  if (do_compose) {
    // Compose mode: deserialize PoolConfig, then call LoadConfig
    clio::run::PoolConfig pool_config =
        clio::run::Task::Deserialize<clio::run::PoolConfig>(chimod_params);
    CreateParamsT params;
    params.LoadConfig(pool_config);
    return params;
  } else {
    // Programmatic mode: direct cereal deserialization
    return clio::run::Task::Deserialize<CreateParamsT>(chimod_params);
  }
}

The clio::run::PoolConfig struct carries the YAML compose entry:

• mod_name_ - Module library name (e.g., "clio_bdev")
• pool_name_ - Pool identifier or file path
• pool_id_ - Pool ID
• pool_query_ - Scheduling query (dynamic or local)
• config_ - Remaining YAML fields as a raw string for LoadConfig() to parse
• restart_ - Whether the pool should be restarted

This means every CreateParams must implement both serialize() (for programmatic mode) and LoadConfig() (for compose mode). See the Compose Configuration Feature section for full details on the YAML format and usage.

#include <clio_runtime/clio_runtime.h>

// Method IDs live in autogen/MOD_NAME_methods.h; shown locally for this example.
namespace Method {
GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10;
}  // namespace Method

/**
 * Custom operation task
 */
struct CustomTask : public clio::run::Task {
  // Task-specific data using standard types
  INOUT std::string data_;              // Input/output string
  IN clio::run::u32 operation_id_;      // Input parameter
  OUT clio::run::u32 result_code_;      // Output result

  // Default constructor
  CustomTask()
      : clio::run::Task(),
        operation_id_(0),
        result_code_(0) {}

  // Emplace constructor
  explicit CustomTask(
      const clio::run::TaskId &task_id,
      const clio::run::PoolId &pool_id,
      const clio::run::PoolQuery &pool_query,
      const std::string &data,
      clio::run::u32 operation_id)
      : clio::run::Task(task_id, pool_id, pool_query, Method::kCustom),
        data_(data),
        operation_id_(operation_id),
        result_code_(0) {
    task_id_ = task_id;
    pool_id_ = pool_id;
    method_ = Method::kCustom;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }

  // Serialize IN/INOUT params (input) and OUT/INOUT params (output)
  template <typename Archive>
  void SerializeIn(Archive& ar) {
    Task::SerializeIn(ar);
    ar(data_, operation_id_);
  }
  template <typename Archive>
  void SerializeOut(Archive& ar) {
    Task::SerializeOut(ar);
    ar(data_, result_code_);
  }
};

Client Implementation (MOD_NAME_client.h/cc)

The client provides an async-only API for task submission. All operations return clio::run::Future<TaskType> objects:

// MOD_NAME_client.h — contents shown at `namespace clio::run::MOD_NAME` scope.
#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;  // CreateTask, CustomTask, CreateParams

class Client : public ContainerClient {
 public:
  Client() = default;
  explicit Client(const PoolId& pool_id) { Init(pool_id); }

  /**
   * Async Create operation - returns Future for task completion
   * Caller must call task.Wait() and check GetReturnCode()
   */
  Future<CreateTask> AsyncCreate(
      const PoolQuery& pool_query,
      const std::string& pool_name,
      const PoolId& custom_pool_id) {
    auto* ipc_manager = CLIO_CPU_IPC;

    // CRITICAL: CreateTask MUST use admin pool for GetOrCreatePool processing
    auto task = ipc_manager->NewTask<CreateTask>(
        CreateTaskId(),
        kAdminPoolId,  // Always use admin pool for CreateTask
        pool_query,
        CreateParams::chimod_lib_name,  // Module name from CreateParams
        pool_name,              // User-provided pool name
        custom_pool_id,         // Target pool ID to create
        this);                  // Client pointer for PostWait callback

    // Submit to runtime and return Future
    return ipc_manager->Send(task);
  }

  /**
   * Async Custom operation - example of a typical async method
   */
  Future<CustomTask> AsyncCustom(
      const PoolQuery& pool_query,
      const std::string& input_data,
      u32 operation_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager->NewTask<CustomTask>(
        CreateTaskId(),
        pool_id_,           // Use client's pool_id_ for non-Create operations
        pool_query,
        input_data,
        operation_id);
    return ipc_manager->Send(task);
  }
};

Usage Pattern:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <iostream>

void example() {
  using namespace clio::run;
  // Create client and initialize
  MOD_NAME::Client client;
  const PoolId pool_id(7000, 0);

  // Async create
  auto create_task = client.AsyncCreate(PoolQuery::Dynamic(), "my_pool", pool_id);
  create_task.Wait();

  if (create_task->GetReturnCode() != 0) {
    std::cerr << "Create failed" << std::endl;
    return;
  }

  // Perform operations
  auto op_task = client.AsyncCustom(PoolQuery::Local(), "data", 1);
  op_task.Wait();

  // Access results
  std::cout << "Result: " << op_task->GetReturnCode() << std::endl;
}

Module CreateTask Pool Assignment Requirements

CRITICAL: All Module clients implementing Create functions MUST use the explicit clio::run::kAdminPoolId variable when constructing CreateTask operations. You CANNOT use pool_id_ for CreateTask operations.

Why This is Required

CreateTask operations are actually GetOrCreatePoolTask operations that must be processed by the admin Module to create or find the target pool. The pool_id_ variable is not initialized until after the Create operation completes successfully.

Correct Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  Client client;
  PoolQuery pool_query = PoolQuery::Dynamic();
  std::string pool_name = "my_pool";
  PoolId pool_id(7000, 0);
  // CORRECT: Always use kAdminPoolId for CreateTask
  auto task = ipc_manager->NewTask<CreateTask>(
      CreateTaskId(),
      kAdminPoolId,          // REQUIRED: Use admin pool for CreateTask
      pool_query,
      CreateParams::chimod_lib_name,
      pool_name,
      pool_id,               // Target pool ID to create
      &client);              // Client pointer for PostWait callback
  (void)task;
}

Incorrect Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  Client client;
  PoolQuery pool_query = PoolQuery::Dynamic();
  std::string pool_name = "my_pool";
  // WRONG: Never use pool_id_ for CreateTask operations
  auto task = ipc_manager->NewTask<CreateTask>(
      CreateTaskId(),
      client.pool_id_,        // WRONG: pool_id_ is not initialized yet
      pool_query,
      CreateParams::chimod_lib_name,
      pool_name,
      client.pool_id_,
      &client);
  (void)task;
}

Key Points

1. Admin Pool Processing: CreateTask is a GetOrCreatePoolTask that must be handled by the admin pool
2. Uninitialized Variable: pool_id_ is not set until after Create completes
3. Universal Requirement: This applies to ALL Module clients, including admin, bdev, and custom modules
4. Create Responsibility: Create operations are responsible for allocating new pool IDs using the admin pool

Module Name Requirements

CRITICAL: All Module clients MUST use CreateParams::chimod_lib_name instead of hardcoding module names.

Correct Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  Client client;
  PoolQuery pool_query = PoolQuery::Dynamic();
  std::string pool_name = "my_pool";
  PoolId pool_id(7000, 0);
  // CORRECT: Use CreateParams::chimod_lib_name
  auto task = ipc_manager->NewTask<CreateTask>(
      CreateTaskId(),
      kAdminPoolId,
      pool_query,
      CreateParams::chimod_lib_name,  // Dynamic reference to CreateParams
      pool_name,
      pool_id,
      &client);
  (void)task;
}

Incorrect Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  Client client;
  PoolQuery pool_query = PoolQuery::Dynamic();
  std::string pool_name = "my_pool";
  PoolId pool_id(7000, 0);
  // WRONG: Never hardcode module names
  auto task = ipc_manager->NewTask<CreateTask>(
      CreateTaskId(),
      kAdminPoolId,
      pool_query,
      "clio_MOD_NAME",  // Hardcoded name breaks flexibility
      pool_name,
      pool_id,
      &client);
  (void)task;
}

Why This is Required

1. Namespace Flexibility: Allows ChiMods to work with different namespace configurations
2. Single Source of Truth: The module name is defined once in CreateParams
3. External ChiMods: Essential for external ChiMods using custom namespaces
4. Maintainability: Changes to module names only require updating CreateParams

Implementation Pattern

All non-admin ChiMods using GetOrCreatePoolTask must follow this pattern:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  Client client;
  PoolQuery pool_query = PoolQuery::Dynamic();
  std::string pool_name = "my_pool";
  PoolId pool_id(7000, 0);
  // In AsyncCreate method
  auto task = ipc_manager->NewTask<CreateTask>(
      CreateTaskId(),
      kAdminPoolId,                         // Always use admin pool
      pool_query,
      CreateParams::chimod_lib_name,        // REQUIRED: Use static member
      pool_name,                            // Pool identifier
      pool_id,                              // Target pool ID
      &client);                             // Client pointer for PostWait
  (void)task;
}

Note: The admin Module uses BaseCreateTask directly and doesn't require the chimod name parameter.

Runtime Container (MOD_NAME_runtime.h/cc)

The runtime container executes tasks server-side:

// MOD_NAME_runtime.h — contents shown at `namespace clio::run::MOD_NAME` scope.
#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>  // Client, CreateTask, CustomTask
using namespace clio::run;
using namespace clio::run::MOD_NAME;

class RuntimeExample : public clio::run::Container {
 public:
  RuntimeExample() = default;
  ~RuntimeExample() override = default;

  /**
   * Initialize container with pool information (REQUIRED)
   * This is called by the framework before Create is called
   */
  void Init(const PoolId& pool_id, const std::string& pool_name,
            u32 container_id = 0) override {
    // Call base class initialization
    clio::run::Container::Init(pool_id, pool_name, container_id);

    // Initialize the client for this Module
    client_ = Client(pool_id);
  }

  /**
   * Create the container (Method::kCreate). Task methods are coroutines that
   * return TaskResume. Container is already initialized via Init().
   */
  TaskResume Create(shared_ptr<CreateTask>& task) {
    CLIO_TASK_BODY_BEGIN
    // Additional container-specific initialization logic here
    (void)task;
    CLIO_CO_RETURN;
    CLIO_TASK_BODY_END
  }

  /**
   * Custom operation (Method::kCustom)
   */
  TaskResume Custom(shared_ptr<CustomTask>& task) {
    CLIO_TASK_BODY_BEGIN
    std::string result = ProcessData(task->data_.str(), task->operation_id_);
    (void)result;
    task->SetReturnCode(0);
    CLIO_CO_RETURN;
    CLIO_TASK_BODY_END
  }

 private:
  std::string ProcessData(const std::string& input, u32 op_id) {
    (void)op_id;
    // Business logic here
    return input + "_processed";
  }

  Client client_;
};

// The real runtime .cc registers the concrete container with:
//   CLIO_TASK_CC(clio::run::MOD_NAME::Runtime)

Dynamic Scheduling (ScheduleTask)

The CLIO Runtime lets a container decide where each task runs by overriding Container::ScheduleTask(). This is the single hook for dynamic routing: the worker calls it before executing the task, and the returned PoolQuery determines the route. The default implementation returns the task's existing pool_query_ unchanged.

ScheduleTask Overview

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
using namespace clio::run;

// A container overrides ScheduleTask to pick a route for each task. Only the
// Container virtuals relevant to routing are shown here.
class RoutingExample : public clio::run::Container {
 public:
  PoolQuery ScheduleTask(const shared_ptr<Task>& task) override {
    // Only remap dynamic requests; everything else keeps its query.
    if (task->pool_query_.IsDynamicMode()) {
      // Dynamic scheduling logic - choose a concrete route.
      return PoolQuery::Broadcast();
    }
    return task->pool_query_;
  }
};

ScheduleTask() returns the PoolQuery to route with. Inspect task->pool_query_ (e.g. IsDynamicMode()) and return PoolQuery::Local(), PoolQuery::Broadcast(), or any other factory to redirect the task.

Normal Routing (Default)

When a container does not override ScheduleTask(), tasks route using the PoolQuery the client supplied (Local(), Broadcast(), DirectHash(), ...). The task then executes to completion and is cleaned up. This is the default for all standard task processing.

Dynamic Routing

Override ScheduleTask() when a container needs a runtime-dependent routing decision, for example:

• Cache optimization patterns (check local, then broadcast if not found)
• Conditional distributed execution based on state
• Hashing a key/name to a specific container

The hook runs before execution, is expected to be lightweight (fast checks only), and must not perform the task's real work or spawn subtasks. It simply returns the PoolQuery to route with.

Example: Route a Create by Local Pool Existence

An admin-style container can route a Dynamic() create to Local() when the pool already exists locally, and Broadcast() otherwise:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
using namespace clio::run;

class AdminRoutingExample : public clio::run::Container {
 public:
  PoolQuery ScheduleTask(const shared_ptr<Task>& task) override {
    if (task->pool_query_.IsDynamicMode()) {
      // Fast local check (e.g. pool_manager->FindPoolByName(name)).
      bool pool_exists_locally = false;
      // Pool exists locally -> run local; otherwise broadcast creation.
      return pool_exists_locally ? PoolQuery::Local() : PoolQuery::Broadcast();
    }
    return task->pool_query_;
  }
};

Using Dynamic() PoolQuery

The PoolQuery::Dynamic() factory marks a request as needing dynamic routing; the container's ScheduleTask() override then resolves it to a concrete route:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Client code - request dynamic routing
  auto pool_query = PoolQuery::Dynamic();
  client.AsyncCreate(pool_query, "my_pool_name", PoolId(7000, 0)).Wait();
}

What happens internally:

1. The worker sees the Dynamic() pool query
2. It calls the container's ScheduleTask() override
3. ScheduleTask() inspects local state and returns a concrete PoolQuery
4. The task is routed using that query and executes normally

Benefits of Dynamic Scheduling

Performance Optimization:

• Avoids redundant operations (e.g., pool creation when pool already exists)
• Reduces network overhead by checking local state first
• Enables intelligent routing based on runtime conditions

Cache Optimization Pattern:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
using namespace clio::run;

class CacheRoutingExample : public clio::run::Container {
 public:
  PoolQuery ScheduleTask(const shared_ptr<Task>& task) override {
    if (task->pool_query_.IsDynamicMode()) {
      bool have_locally = false;  // e.g. LocalCacheHas(resource_id)
      return have_locally ? PoolQuery::Local()       // Found locally
                          : PoolQuery::Broadcast();   // Need to fetch
    }
    return task->pool_query_;
  }
};

State-Dependent Routing:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
using namespace clio::run;

class StateRoutingExample : public clio::run::Container {
 public:
  PoolQuery ScheduleTask(const shared_ptr<Task>& task) override {
    if (task->pool_query_.IsDynamicMode()) {
      bool should_execute_distributed = true;  // runtime condition
      return should_execute_distributed ? PoolQuery::Broadcast()
                                        : PoolQuery::Local();
    }
    return task->pool_query_;
  }
};

Implementation Guidelines

DO:

• ✅ Inspect task->pool_query_ (e.g. IsDynamicMode()) inside ScheduleTask()
• ✅ Return a concrete PoolQuery (Local(), Broadcast(), ...) from ScheduleTask()
• ✅ Keep ScheduleTask() logic lightweight (fast checks only)
• ✅ Use ScheduleTask() for cache optimization patterns

DON'T:

• ❌ Perform expensive operations in ScheduleTask()
• ❌ Modify task output parameters in ScheduleTask()
• ❌ Call Wait() or spawn subtasks in ScheduleTask()
• ❌ Override ScheduleTask() for simple operations that don't need routing optimization

Worker Implementation Details

The worker calls ScheduleTask() to obtain the route, sends the task to the resolved location, and then runs it to completion. Overriding ScheduleTask() is the only step a module needs; the worker handles the rest.

Configuration and Code Generation

Overview

CLIO Runtime uses a two-level configuration system with automated code generation:

1. clio_repo.yaml: Repository-wide configuration (namespace, version, etc.)
2. clio_mod.yaml: Module-specific configuration (method IDs, metadata)
3. clio_run repo refresh: Utility script that generates autogen files from YAML configurations

clio_repo.yaml

Located at chimods/clio_repo.yaml, this file defines repository-wide settings:

# Repository Configuration
namespace: clio::run       # MUST match namespace in all clio_mod.yaml files
version: 1.0.0
description: "CLIO Runtime Module Repository"

# Module discovery - directories to scan for ChiMods
modules:
  - MOD_NAME
  - admin  
  - bdev

Key Requirements:

• The namespace field MUST be identical in both clio_repo.yaml and all clio_mod.yaml files
• Used by build system for CMake package generation and installation paths
• Determines export target names: $\{namespace\}::$\{module\}_runtime, $\{namespace\}::$\{module\}_client

clio_mod.yaml

Each Module must have its own configuration file specifying methods and metadata:

# MOD_NAME Module Configuration
module_name: MOD_NAME
namespace: clio::run       # MUST match clio_repo.yaml namespace
version: 1.0.0

# Inherited Methods (fixed IDs)
kCreate: 0        # Container creation (required)
kDestroy: 1       # Container destruction (required)
kNodeFailure: -1  # Not implemented (-1 means disabled)
kRecover: -1      # Not implemented 
kMigrate: -1      # Not implemented
kUpgrade: -1      # Not implemented

# Custom Methods (start from 10, use sequential IDs)
kCustom: 10       # Custom operation method
kCoMutexTest: 20  # CoMutex synchronization testing method
kCoRwLockTest: 21 # CoRwLock reader-writer synchronization testing method

Method ID Assignment Rules:

• 0-9: Reserved for system methods (kCreate=0, kDestroy=1, etc.)
• 10+: Custom methods (assign sequential IDs starting from 10)
• Disabled methods: Use -1 to disable inherited methods not implemented
• Consistency: Once assigned, never change method IDs (breaks compatibility)

clio_run repo refresh Utility

The clio_run repo refresh utility automatically generates autogen files from YAML configurations.

Usage

# From project root, regenerate all autogen files
./build/bin/clio_run repo refresh chimods

# The utility will:
# 1. Read clio_repo.yaml for global settings
# 2. Scan each module's clio_mod.yaml 
# 3. Generate MOD_NAME_methods.h with method constants
# 4. Generate MOD_NAME_lib_exec.cc with virtual method dispatch

Generated Files

For each Module, the utility generates:

1. include/[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h:

   #include <clio_runtime/clio_runtime.h>
   // Shown at `namespace clio::run::MOD_NAME` scope in the real header.
   namespace Method {
   GLOBAL_CROSS_CONST clio::run::u32 kCreate = 0;
   GLOBAL_CROSS_CONST clio::run::u32 kDestroy = 1;
   GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10;
   GLOBAL_CROSS_CONST clio::run::u32 kCoMutexTest = 20;
   }  // namespace Method

2. src/autogen/MOD_NAME_lib_exec.cc:

   • Virtual method dispatch (Runtime::Run, etc.)
   • Task serialization support (SaveIn/Out, LoadIn/Out)
   • Memory management (Del, NewCopy, Aggregate)

When to Run clio_run repo refresh

ALWAYS run clio_run repo refresh when:

• Adding new methods to clio_mod.yaml
• Changing method IDs or names
• Adding new ChiMods to the repository
• Modifying namespace or version information

Important Notes

• Never manually edit autogen files - they are overwritten by clio_run repo refresh
• Run clio_run repo refresh before building after YAML changes
• Commit autogen files to git so other developers don't need to regenerate
• Method IDs are permanent - changing them breaks binary compatibility

Workflow Summary

1. Define methods in clio_mod.yaml with sequential IDs
2. Implement corresponding methods in MOD_NAME_runtime.h/MOD_NAME_runtime.cc
3. Run ./build/bin/clio_run repo refresh chimods to generate autogen files
4. Build project with make - autogen files provide the dispatch logic
5. Autogen files handle virtual method routing, serialization, and memory management

This automated approach ensures consistency across all ChiMods and reduces boilerplate code maintenance.

Task Development

Task Requirements

1. Inherit from clio::run::Task: All tasks must inherit the base Task class
2. Two Constructors: SHM and emplace constructors are mandatory
3. Serializable Types: Use standard types (std::string, std::vector, etc.) for member variables
4. Method Assignment: Set the method_ field to identify the operation
5. shared_ptr Usage: All task handlers take the task as clio::run::shared_ptr<TaskType>&
6. Copy Method: Required - copies task data for network transport and replication
7. Aggregate Method: Optional - combines results from task replicas when needed

Task Methods: Copy and Aggregate

All tasks must implement a Copy() method. The network adapter calls Copy() when tasks are sent to remote nodes — without it, your task cannot be transported across the network. Aggregate() is optional and is used to combine results from task replicas after distributed execution.

The autogenerated NewCopy dispatcher allocates a new task and calls your Copy() method, but it does not call Task::Copy() for you. You must call the base method yourself.

Copy Method

The Copy() method creates a deep copy of a task for network transport and replication. You must call Task::Copy() first to copy the base task fields (pool_id, task_id, pool_query, method, task_flags, period, return_code, completer, stat).

Signature:

#include <clio_runtime/clio_runtime.h>
struct YourTask : public clio::run::Task {
  // Deep-copy for network transport / replication.
  void Copy(const ctp::ipc::FullPtr<YourTask> &other);
};

Implementation Pattern:

#include <clio_runtime/clio_runtime.h>
#include <vector>
#include <cstdint>
struct WriteTask : public clio::run::Task {
  IN clio::run::u64 offset_;
  IN std::vector<uint8_t> data_;
  IN clio::run::u64 length_;
  OUT clio::run::u64 bytes_written_;

  void Copy(const ctp::ipc::FullPtr<WriteTask> &other) {
    // REQUIRED: Copy base Task fields first
    Task::Copy(other.template Cast<Task>());
    // Copy task-specific fields
    offset_ = other->offset_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_written_ = other->bytes_written_;
  }
};

Key Points:

• Always call Task::Copy(other.template Cast<Task>()) as the first line
• Then copy all task-specific fields from the source task
• The destination task (this) is already constructed - don't call constructors
• For pointer fields, decide if you need deep or shallow copy based on ownership

Aggregate Method (Optional)

The Aggregate() method combines results from a task replica back into the original task. It is optional — implement it when your task is distributed across multiple nodes and you need to merge replica results. When implemented, you must call Task::Aggregate() first to propagate the return code and completer from the replica.

Signature: the replica arrives typed as the base Task; cast it to your task.

#include <clio_runtime/clio_runtime.h>
struct YourTask : public clio::run::Task {
  // Merge a replica's OUT fields back into this task (optional).
  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica);
};

Implementation Patterns:

Pattern 1: Last-Writer-Wins (Simple Override)

#include <clio_runtime/clio_runtime.h>
struct WriteTask : public clio::run::Task {
  OUT clio::run::u64 bytes_written_;

  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica_base) {
    // REQUIRED: AggregateOut base Task fields first (propagates return code)
    Task::AggregateOut(replica_base);
    auto replica = replica_base.template Cast<WriteTask>();
    // Copy the result from the completed replica
    bytes_written_ = replica->bytes_written_;
  }
};

Pattern 2: Accumulation (Sum/Max/Min)

#include <clio_runtime/clio_runtime.h>
#include <algorithm>
struct GetStatsTask : public clio::run::Task {
  OUT clio::run::u64 total_bytes_;
  OUT clio::run::u64 operation_count_;
  OUT clio::run::u64 max_latency_us_;

  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica_base) {
    // REQUIRED: AggregateOut base Task fields first
    Task::AggregateOut(replica_base);
    auto replica = replica_base.template Cast<GetStatsTask>();
    // Sum cumulative metrics
    total_bytes_ += replica->total_bytes_;
    operation_count_ += replica->operation_count_;
    // Take maximum for latency
    max_latency_us_ = std::max(max_latency_us_, replica->max_latency_us_);
  }
};

Pattern 3: Full Copy Aggregate

#include <clio_runtime/clio_runtime.h>
#include <vector>
#include <cstdint>
struct ReadTask : public clio::run::Task {
  IN clio::run::u64 offset_;
  OUT std::vector<uint8_t> data_;
  OUT clio::run::u64 bytes_read_;

  void Copy(const ctp::ipc::FullPtr<ReadTask> &other) {
    Task::Copy(other.template Cast<Task>());
    offset_ = other->offset_;
    data_ = other->data_;
    bytes_read_ = other->bytes_read_;
  }

  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica_base) {
    // REQUIRED: AggregateOut base Task fields first
    Task::AggregateOut(replica_base);
    // For reads, take all results from the replica
    Copy(replica_base.template Cast<ReadTask>());
  }
};

Copy/Aggregate Usage in Networking

When tasks are sent across nodes, the network adapter uses Copy and Aggregate:

1. Send Phase: Copy() creates a replica of the origin task for network transport

   NewCopyTask(method, origin_task, deep_copy);
   // Internally allocates a new task and calls: replica->Copy(origin_task)

2. Recv Phase: AggregateOut() merges replica results back into the origin task

   AggregateOut(method, origin_task, replica);
   // Internally calls: origin_task->AggregateOut(replica)

3. Autogeneration: The code generator creates dispatcher methods that allocate a new task and call your Copy(). Note that NewCopyTask does not call Task::Copy() for you — your Copy() implementation must do that itself:

   // In autogen/MOD_NAME_lib_exec.cc (conceptual)
   shared_ptr<Task> Runtime::NewCopyTask(u32 method,
                                         shared_ptr<Task>& orig_task, bool deep) {
     switch (method) {
       case Method::kWrite: {
         auto new_task = CLIO_IPC->NewTask<WriteTask>();
         auto& orig = orig_task.Cast<WriteTask>();
         // Calls YOUR Copy(), which must call Task::Copy() internally
         new_task->Copy(ctp::ipc::FullPtr<WriteTask>(orig.get()));
         return new_task.Cast<Task>();
       }
     }
   }

Complete Example: ReadTask with Copy and Aggregate

#include <clio_runtime/clio_runtime.h>
#include <vector>
#include <cstdint>

// Method IDs live in autogen/MOD_NAME_methods.h; shown locally for this example.
namespace Method {
GLOBAL_CROSS_CONST clio::run::u32 kRead = 11;
}  // namespace Method

struct ReadTask : public clio::run::Task {
  IN clio::run::u64 offset_;
  OUT std::vector<uint8_t> data_;
  INOUT clio::run::u64 length_;
  OUT clio::run::u64 bytes_read_;

  ReadTask()
      : clio::run::Task(), offset_(0), length_(0), bytes_read_(0) {}

  explicit ReadTask(const clio::run::TaskId &task_node,
                    const clio::run::PoolId &pool_id,
                    const clio::run::PoolQuery &pool_query,
                    clio::run::u64 offset,
                    clio::run::u64 length)
      : clio::run::Task(task_node, pool_id, pool_query, Method::kRead),
        offset_(offset), length_(length), bytes_read_(0) {
    task_id_ = task_node;
    pool_id_ = pool_id;
    method_ = Method::kRead;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }

  void Copy(const ctp::ipc::FullPtr<ReadTask> &other) {
    // REQUIRED: Copy base Task fields first
    Task::Copy(other.template Cast<Task>());
    // Copy task-specific fields
    offset_ = other->offset_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_read_ = other->bytes_read_;
  }

  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica_base) {
    // REQUIRED: AggregateOut base Task fields first
    Task::AggregateOut(replica_base);
    // For reads, take the result from the replica
    Copy(replica_base.template Cast<ReadTask>());
  }
};

Task Naming Conventions

CRITICAL: All task names MUST follow consistent naming patterns to ensure proper code generation and maintenance.

Required Naming Pattern

The naming convention enforces consistency across function names, task types, and method constants:

Function Name → Task Name → Method Constant
FunctionName()  → FunctionNameTask  → kFunctionName

Examples

Correct Naming: AsyncGetStats() → GetStatsTask → kGetStats → GetStats().

#include <clio_runtime/clio_runtime.h>

// In autogen/bdev_methods.h
namespace Method {
GLOBAL_CROSS_CONST clio::run::u32 kGetStats = 14;  // Get performance statistics
}  // namespace Method

// In bdev_tasks.h
struct GetStatsTask : public clio::run::Task {
  OUT clio::run::u64 total_bytes_;
  OUT clio::run::u64 remaining_size_;
  // ... constructors and Copy/AggregateOut
};

// In bdev_client.h (async-only API)
clio::run::Future<GetStatsTask> AsyncGetStats(const clio::run::PoolQuery& pool_query);

// In bdev_runtime.h — runtime method returns TaskResume
clio::run::TaskResume GetStats(clio::run::shared_ptr<GetStatsTask>& task);

Incorrect Naming Examples (names don't line up — do not do this):

// WRONG: Function and task names don't match
Future<StatTask> AsyncGetStats(...);   // Task name doesn't match function
struct StatTask { ... };               // Task name doesn't match function

// WRONG: Method constant doesn't match function
GLOBAL_CROSS_CONST u32 kStat = 14;     // Method doesn't match function name

// WRONG: Runtime method doesn't match function
TaskResume Stat(shared_ptr<StatTask>& task);  // Runtime method doesn't match

Naming Rules

1. Client Function Names: Use Async prefix with descriptive verbs (e.g., AsyncGetStats, AsyncAllocateBlocks, AsyncWriteData)
2. Task Names: Remove Async prefix and append "Task" (e.g., GetStatsTask, AllocateBlocksTask)
3. Method Constants: Prefix with "k" and match the base function name (e.g., kGetStats, kAllocateBlocks)
4. Runtime Methods: Match the base function name without Async prefix (e.g., GetStats())

Backward Compatibility

When renaming tasks, provide backward compatibility aliases:

#include <clio_runtime/clio_runtime.h>
struct GetStatsTask : public clio::run::Task {};

// In bdev_tasks.h - provide alias for old name
using StatTask = GetStatsTask;  // Backward compatibility

// In autogen/bdev_methods.h - provide constant alias
namespace Method {
GLOBAL_CROSS_CONST clio::run::u32 kGetStats = 14;
GLOBAL_CROSS_CONST clio::run::u32 kStat = kGetStats;  // Backward compatibility
}  // namespace Method

// In bdev_runtime.h - primary method + backward-compat wrapper declaration
clio::run::TaskResume GetStats(clio::run::shared_ptr<GetStatsTask>& task);  // Primary
clio::run::TaskResume Stat(clio::run::shared_ptr<StatTask>& task);          // Wrapper

Benefits of Consistent Naming

1. Code Generation: Automated tools can reliably generate method dispatch code
2. Maintenance: Clear correlation between client functions and runtime implementations
3. Documentation: Self-documenting code with predictable naming patterns
4. Debugging: Easy to trace from client calls to runtime execution
5. Testing: Consistent patterns make it easier to write comprehensive tests

Method System and Auto-Generated Files

Method Definitions (autogen/MOD_NAME_methods.h)

Method IDs are now defined as namespace constants instead of enum class values. This eliminates the need for static casting:

#include <clio_runtime/clio_runtime.h>

// autogen/MOD_NAME_methods.h — shown at `namespace clio::run::MOD_NAME` scope.
namespace Method {
  // Inherited methods
  GLOBAL_CROSS_CONST clio::run::u32 kCreate = 0;
  GLOBAL_CROSS_CONST clio::run::u32 kDestroy = 1;
  GLOBAL_CROSS_CONST clio::run::u32 kNodeFailure = 2;
  GLOBAL_CROSS_CONST clio::run::u32 kRecover = 3;
  GLOBAL_CROSS_CONST clio::run::u32 kMigrate = 4;
  GLOBAL_CROSS_CONST clio::run::u32 kUpgrade = 5;

  // Module-specific methods
  GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10;
}  // namespace Method

Key Changes:

• Namespace instead of enum class: Use Method::kMethodName directly
• GLOBAL_CROSS_CONST values: No more static casting required
• Include clio_runtime.h: Required for GLOBAL_CROSS_CONST macro
• Direct assignment: method_ = Method::kCreate; (no casting)

BaseCreateTask Template System

For modules that need container creation functionality, use the BaseCreateTask template instead of implementing custom CreateTask. However, there are different approaches depending on whether your module is the admin module or a regular Module:

GetOrCreatePoolTask vs BaseCreateTask Usage

For Non-Admin Modules (Recommended Pattern):

All non-admin ChiMods should use GetOrCreatePoolTask which is a specialized version of BaseCreateTask designed for external pool creation:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/admin/admin_tasks.h>  // Include admin templates
// Shown at `namespace clio::run::MOD_NAME` scope.

/**
 * CreateParams for MOD_NAME container creation
 */
struct CreateParams {
  // Module-specific configuration
  std::string config_data_;
  clio::run::u32 worker_count_;

  // Required: chimod library name
  static constexpr const char* chimod_lib_name = "clio_MOD_NAME";

  // Constructors
  CreateParams() : worker_count_(1) {}

  CreateParams(const std::string& config_data, clio::run::u32 worker_count = 1)
      : config_data_(config_data), worker_count_(worker_count) {}

  // Cereal serialization
  template<class Archive>
  void serialize(Archive& ar) {
    ar(config_data_, worker_count_);
  }
};

/**
 * CreateTask - Non-admin modules should use GetOrCreatePoolTask
 * This uses Method::kGetOrCreatePool and is designed for external pool creation
 */
using CreateTask = clio::run::admin::GetOrCreatePoolTask<CreateParams>;

For Admin Module Only:

The admin module itself uses BaseCreateTask directly with Method::kCreate:

#include <clio_runtime/admin/admin_tasks.h>
// Shown at `namespace clio::run::admin` scope. Admin uses BaseCreateTask with
// Method::kCreate (0) and IS_ADMIN=true.
using AdminCreateTask =
    clio::run::admin::BaseCreateTask<clio::run::admin::CreateParams,
                                     /*MethodId=*/0, /*IS_ADMIN=*/true>;

BaseCreateTask Template Parameters

The BaseCreateTask template has three parameters with smart defaults:

template <typename CreateParamsT,
          clio::run::u32 MethodId = Method::kGetOrCreatePool,
          bool IS_ADMIN = false>
struct BaseCreateTask : public clio::run::Task

Template Parameters:

1. CreateParamsT: Your module's parameter structure (required)
2. MethodId: Method ID for the task (default: kGetOrCreatePool)
3. IS_ADMIN: Whether this is an admin operation (default: false)

GetOrCreatePoolTask Template:

The GetOrCreatePoolTask template is a convenient alias that uses the optimal defaults for non-admin modules:

template<typename CreateParamsT>
using GetOrCreatePoolTask = BaseCreateTask<CreateParamsT, Method::kGetOrCreatePool, false>;

When to Use Each Pattern:

• GetOrCreatePoolTask: For all non-admin ChiMods (recommended)
• BaseCreateTask with Method::kCreate: Only for admin module internal operations
• BaseCreateTask with Method::kGetOrCreatePool: Same as GetOrCreatePoolTask (not typically used directly)

BaseCreateTask Structure

BaseCreateTask provides a unified structure for container creation and pool operations:

template <typename CreateParamsT, clio::run::u32 MethodId, bool IS_ADMIN>
struct BaseCreateTask : public clio::run::Task {
  // Pool operation parameters
  INOUT clio::run::priv::string chimod_name_;   // Module name for loading
  IN    clio::run::priv::string pool_name_;     // Target pool name
  INOUT clio::run::priv::string chimod_params_; // Serialized CreateParamsT
  INOUT clio::run::PoolId       new_pool_id_;   // In: requested ID, Out: actual ID

  // Results
  OUT clio::run::u32            return_code_;    // 0 = success, non-zero = error
  OUT clio::run::priv::string   error_message_;  // Error description if failed

  bool is_admin_;                                // Set to IS_ADMIN template value

  // Serialization / accessors
  template<typename... Args> void SetParams(Args &&...args);
  CreateParamsT GetParams() const;
};

Key Features:

• Single pool_id: Serves as both input (requested) and output (result)
• Serialized parameters: chimod_params_ stores serialized CreateParamsT
• Error checking: Use result_code_ != 0 to check for failures
• Template-driven behavior: IS_ADMIN template parameter sets volatile variable
• No static casting: Direct method assignment using namespace constants

Usage Examples

Non-Admin Module Container Creation (Recommended):

#include <clio_runtime/admin/admin_tasks.h>
struct MyCreateParams {
  static constexpr const char* chimod_lib_name = "clio_my_module";
  template <class Archive> void serialize(Archive&) {}
};
// Use GetOrCreatePoolTask for all non-admin modules
using CreateTask = clio::run::admin::GetOrCreatePoolTask<MyCreateParams>;

Admin Module Container Creation:

#include <clio_runtime/admin/admin_tasks.h>
// Admin module uses BaseCreateTask with Method::kCreate (0) and IS_ADMIN=true
using AdminCreateTask =
    clio::run::admin::BaseCreateTask<clio::run::admin::CreateParams, 0, true>;

Data Annotations

• IN: Input-only parameters (read by runtime)
• OUT: Output-only parameters (written by runtime)
• INOUT: Bidirectional parameters

Task Lifecycle

1. Client allocates task in shared memory using ipc_manager->NewTask()
2. Client enqueues task pointer to IPC queue
3. Worker dequeues and executes task
4. The task's clio::run::shared_ptr<Task> handles are released as they go out of scope
5. Task memory is reclaimed (RAII) once the last handle is destroyed

Note: There is no Del/DelTask step. Tasks are clio::run::shared_ptr handles, so cleanup is automatic (RAII).

C++20 Coroutines for Module Development

CLIO Runtime uses C++20 coroutines to enable cooperative task execution within runtime methods. This section covers the coroutine primitives and patterns used in Module development.

Overview

Coroutines allow runtime methods to suspend execution (co_await) and resume later without blocking the worker thread. This is essential for:

• Nested Pool Creation: Create methods that need to create sub-pools
• Subtask Execution: Runtime methods that spawn and wait for child tasks
• I/O Operations: Async I/O that needs to yield while waiting

TaskResume Return Type

All runtime task handlers return clio::run::TaskResume and take the task as a clio::run::shared_ptr<TaskT>& (the RunContext is reached via clio::run::GetCurrentRunContext() when needed):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

class RuntimeExample : public clio::run::Container {
 public:
  // Handler that suspends: uses CLIO_CO_AWAIT before CLIO_CO_RETURN
  TaskResume Create(shared_ptr<CreateTask>& task);

  // Simple handler: still returns TaskResume (body just does CLIO_CO_RETURN)
  TaskResume Custom(shared_ptr<CustomTask>& task);
};

Key Points:

• All handlers return TaskResume; wrap the body in CLIO_TASK_BODY_BEGIN / CLIO_TASK_BODY_END
• Use CLIO_CO_AWAIT(subtask) to suspend on a subtask, and CLIO_CO_RETURN to finish
• The TaskResume type integrates with CLIO Runtime's task scheduling system

co_await - Suspending Execution

Use co_await to suspend the current coroutine until an operation completes:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

// A Create handler that suspends on a subtask.
TaskResume CreateWithSubtask(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  Client client;
  // Create a subtask (e.g. initialize a dependent resource)
  auto sub = client.AsyncCustom(PoolQuery::Local(), "storage_device", 1);

  // Suspend until subtask completes - worker can process other tasks
  CLIO_CO_AWAIT(sub);

  // Execution resumes here after sub completes
  if (sub->GetReturnCode() != 0) {
    task->SetReturnCode(1);
    task->error_message_ = "Failed to create storage device";
    CLIO_CO_RETURN;
  }

  // Continue with remaining initialization
  task->SetReturnCode(0);
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

What Happens During co_await:

1. Coroutine state is saved
2. Worker thread is released to process other tasks
3. When awaited operation completes, coroutine is scheduled for resumption
4. Execution continues from the point after co_await

co_return - Completing Coroutines

Use co_return to complete a coroutine. For TaskResume coroutines, co_return takes no value:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume CreateWithValidation(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  // Early return on error (e.g. ValidateParams(task))
  if (task->pool_name_.empty()) {
    task->SetReturnCode(1);
    CLIO_CO_RETURN;  // Complete coroutine immediately
  }

  // Normal completion
  task->SetReturnCode(0);
  CLIO_CO_RETURN;  // Complete coroutine
  CLIO_TASK_BODY_END
}

Important Notes:

• Always use co_return (not return) in coroutine methods
• co_return takes no arguments for TaskResume coroutines
• Ensure all code paths end with co_return

Common Coroutine Patterns

Pattern 1: Nested Pool Creation

The most common use case is creating dependent pools during container initialization:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume CreateNestedPool(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  Client client;
  // Create a dependent subtask (e.g. a storage pool)
  auto sub = client.AsyncCustom(PoolQuery::Local(), "storage", 1);

  CLIO_CO_AWAIT(sub);

  if (sub->GetReturnCode() != 0) {
    task->SetReturnCode(2);
    task->error_message_ = "Storage initialization failed";
    CLIO_CO_RETURN;
  }

  task->SetReturnCode(0);
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

Pattern 2: Sequential Subtask Execution

Execute multiple subtasks in sequence:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume InitializeSequential(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  Client client;
  // Step 1: Initialize storage
  auto storage_task = client.AsyncCustom(PoolQuery::Local(), "storage", 1);
  CLIO_CO_AWAIT(storage_task);

  if (storage_task->GetReturnCode() != 0) {
    task->SetReturnCode(1);
    CLIO_CO_RETURN;
  }

  // Step 2: Initialize network (depends on storage)
  auto network_task = client.AsyncCustom(PoolQuery::Local(), "network", 2);
  CLIO_CO_AWAIT(network_task);

  if (network_task->GetReturnCode() != 0) {
    task->SetReturnCode(2);
    CLIO_CO_RETURN;
  }

  // Step 3: Complete initialization
  task->SetReturnCode(0);
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

Pattern 3: Parallel Subtask Execution

For independent subtasks, launch them all first, then await:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume ParallelInit(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  Client client;
  // Launch multiple independent subtasks
  auto task1 = client.AsyncCustom(PoolQuery::Local(), "a", 1);
  auto task2 = client.AsyncCustom(PoolQuery::Local(), "b", 2);
  auto task3 = client.AsyncCustom(PoolQuery::Local(), "c", 3);

  // Await all tasks (order doesn't matter for independent tasks)
  CLIO_CO_AWAIT(task1);
  CLIO_CO_AWAIT(task2);
  CLIO_CO_AWAIT(task3);

  // Check all results
  if (task1->GetReturnCode() != 0 ||
      task2->GetReturnCode() != 0 ||
      task3->GetReturnCode() != 0) {
    task->SetReturnCode(1);
    CLIO_CO_RETURN;
  }

  task->SetReturnCode(0);
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

When to Use Coroutines

Use coroutines (TaskResume return type) when:

• ✅ Creating nested/dependent pools in Create methods
• ✅ Spawning and waiting for subtasks
• ✅ Performing async I/O operations that need to yield
• ✅ Any operation that might block and should yield to other tasks

Simple handlers still return TaskResume (the body just does CLIO_CO_RETURN without awaiting) when:

• ✅ The operation is synchronous
• ✅ It completes quickly without waiting
• ✅ It does not spawn subtasks or wait for external events

PoolManager Coroutine Integration

The PoolManager methods that create/destroy pools are coroutines:

#include <clio_runtime/clio_runtime.h>
using namespace clio::run;
// In PoolManager (internal usage)
TaskResume CreatePool(shared_ptr<Task> task, RunContext* rctx);
TaskResume DestroyPool(shared_ptr<Task> task, RunContext* rctx);

Why These Are Coroutines:

• CreatePool co_awaits the container's Create method
• Create methods may need to co_await nested pool creations
• The coroutine chain allows proper suspension and resumption

Admin Runtime Coroutine Methods

The admin runtime has coroutine methods for pool management:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/admin/admin_tasks.h>
using namespace clio::run;
// Admin runtime coroutine methods (task passed as shared_ptr<TaskT>&)
template <typename T>
TaskResume GetOrCreatePool(shared_ptr<admin::GetOrCreatePoolTask<T>>& task);
TaskResume DestroyPool(shared_ptr<admin::DestroyTask>& task);

These methods co_await PoolManager operations internally.

Best Practices

DO:

• ✅ Use co_return (not return) in coroutine methods
• ✅ Check return codes after co_await
• ✅ Use TaskResume return type for methods that need co_await
• ✅ Handle errors before continuing after co_await

DON'T:

• ❌ Mix return and co_return in the same method
• ❌ Use co_await in non-coroutine (void) methods
• ❌ Forget to co_return at the end of coroutine methods
• ❌ Ignore return codes from awaited tasks

Migration from Non-Coroutine Methods

To convert a regular method to a coroutine:

1. Change return type: void → clio::run::TaskResume
2. Change all return;: → co_return;
3. Add co_await: For any async operations that need waiting
4. Update autogen: Ensure dispatch code handles the new return type

Framework Task Cleanup

Tasks are held as clio::run::shared_ptr<Task> handles, so cleanup is automatic (RAII) — there is no Del dispatcher to write. When the last handle to a task goes out of scope, the task's memory is reclaimed from its allocator. Module code never calls a delete/free routine for tasks.

Synchronization Primitives

CLIO Runtime provides specialized cooperative synchronization primitives designed for the runtime's task-based architecture. These should be used instead of standard synchronization primitives like std::mutex, std::shared_mutex, or pthread_mutex when synchronizing access to module data structures.

Why Use CLIO Runtime Synchronization Primitives?

Critical: Always use CoMutex and CoRwLock for module synchronization:

1. Cooperative Design: Compatible with CLIO Runtime's fiber-based task execution
2. TaskId Grouping: Tasks sharing the same TaskId can proceed together (bypassing locks)
3. Deadlock Prevention: Designed to prevent deadlocks in the runtime environment
4. Runtime Integration: Automatically integrate with CHI_CUR_WORKER and task context
5. Performance: Optimized for the runtime's execution model

Do NOT use these standard synchronization primitives in module code:

• ❌ std::mutex - Can cause fiber blocking issues
• ❌ std::shared_mutex - Not compatible with task execution model
• ❌ pthread_mutex_t - Can deadlock with runtime scheduling
• ❌ std::condition_variable - Incompatible with cooperative scheduling

CoMutex: Cooperative Mutual Exclusion

CoMutex provides mutual exclusion with TaskId grouping support. Tasks sharing the same TaskId can bypass the lock and execute concurrently.

Basic Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
#include <clio_runtime/comutex.h>
using namespace clio::run;

struct SomeTaskType : public clio::run::Task {};

class SyncExample : public clio::run::Container {
private:
  // Static member for shared synchronization across all container instances
  static clio::run::CoMutex shared_mutex_;

  // Instance member for per-container synchronization
  clio::run::CoMutex instance_mutex_;

public:
  void SomeTask(shared_ptr<SomeTaskType>& task) {
    (void)task;
    // Manual lock/unlock
    shared_mutex_.Lock();
    // ... critical section ...
    shared_mutex_.Unlock();

    // OR use RAII scoped lock (recommended)
    clio::run::ScopedCoMutex lock(instance_mutex_);
    // ... critical section ...
    // Automatically unlocks when leaving scope
  }
};

// Static member definition (required)
clio::run::CoMutex SyncExample::shared_mutex_;

Key Features

1. Automatic Task Context: Uses CHI_CUR_WORKER internally - no task parameters needed
2. TaskId Grouping: Tasks with the same TaskId bypass the mutex
3. RAII Support: ScopedCoMutex for automatic lock management
4. Try-Lock Support: Non-blocking lock attempts

API Reference

namespace chi {
  class CoMutex {
  public:
    // Blocking operations
    void Lock();                    // Block until lock acquired
    void Unlock();                  // Release the lock
    bool TryLock();                 // Non-blocking lock attempt
    
    // No task parameters needed - uses CHI_CUR_WORKER automatically
  };
  
  // RAII wrapper (recommended)
  class ScopedCoMutex {
  public:
    explicit ScopedCoMutex(CoMutex& mutex);  // Locks in constructor
    ~ScopedCoMutex();                        // Unlocks in destructor
  };
}

CoRwLock: Cooperative Reader-Writer Lock

CoRwLock provides reader-writer semantics with TaskId grouping. Multiple readers can proceed concurrently, but writers have exclusive access.

Basic Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
#include <clio_runtime/corwlock.h>
using namespace clio::run;

struct ReadTaskType : public clio::run::Task {};
struct WriteTaskType : public clio::run::Task {};

class RwSyncExample : public clio::run::Container {
private:
  static clio::run::CoRwLock data_lock_;  // Protect shared data structures

public:
  void ReadTask(shared_ptr<ReadTaskType>& task) {
    (void)task;
    // Manual reader lock
    data_lock_.ReadLock();
    // ... read operations ...
    data_lock_.ReadUnlock();

    // OR use RAII scoped reader lock (recommended)
    clio::run::ScopedCoRwReadLock lock(data_lock_);
    // ... read operations ...
    // Automatically unlocks when leaving scope
  }

  void WriteTask(shared_ptr<WriteTaskType>& task) {
    (void)task;
    // RAII scoped writer lock (recommended)
    clio::run::ScopedCoRwWriteLock lock(data_lock_);
    // ... write operations ...
    // Automatically unlocks when leaving scope
  }
};

// Static member definition
clio::run::CoRwLock RwSyncExample::data_lock_;

Key Features

1. Multiple Readers: Concurrent read access when no writers are active
2. Exclusive Writers: Writers get exclusive access, blocking all other operations
3. TaskId Grouping: Tasks with same TaskId can bypass reader locks
4. Automatic Context: Uses CHI_CUR_WORKER for task identification
5. RAII Support: Scoped locks for both readers and writers

API Reference

namespace chi {
  class CoRwLock {
  public:
    // Reader operations
    void ReadLock();                // Acquire reader lock
    void ReadUnlock();              // Release reader lock
    bool TryReadLock();             // Non-blocking reader lock attempt
    
    // Writer operations  
    void WriteLock();               // Acquire exclusive writer lock
    void WriteUnlock();             // Release writer lock
    bool TryWriteLock();            // Non-blocking writer lock attempt
  };
  
  // RAII wrappers (recommended)
  class ScopedCoRwReadLock {
  public:
    explicit ScopedCoRwReadLock(CoRwLock& lock);  // Acquire read lock
    ~ScopedCoRwReadLock();                        // Release read lock
  };
  
  class ScopedCoRwWriteLock {
  public:
    explicit ScopedCoRwWriteLock(CoRwLock& lock); // Acquire write lock
    ~ScopedCoRwWriteLock();                       // Release write lock
  };
}

TaskId Grouping Behavior

Both CoMutex and CoRwLock support TaskId grouping, which allows related tasks to bypass synchronization:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  PoolId pool_id(7000, 0);
  auto pool_query = PoolQuery::Local();
  // Tasks created with the same TaskId can proceed together
  auto task_id = CreateTaskId();

  // These tasks share the same TaskId - they can bypass CoMutex/CoRwLock
  auto task1 = ipc_manager->NewTask<CustomTask>(task_id, pool_id, pool_query, "a", 1);
  auto task2 = ipc_manager->NewTask<CustomTask>(task_id, pool_id, pool_query, "b", 2);

  // This task has a different TaskId - must respect locks normally
  auto task3 = ipc_manager->NewTask<CustomTask>(CreateTaskId(), pool_id, pool_query, "c", 3);
  (void)task1; (void)task2; (void)task3;
}

Key Points:

• Tasks with the same TaskId are considered "grouped" and can bypass locks
• Use TaskId grouping for logically related operations that don't need mutual exclusion
• Different TaskIds must respect normal lock semantics

Best Practices

1. Use RAII Wrappers: Always prefer ScopedCoMutex and ScopedCoRw*Lock over manual lock/unlock
2. Static vs Instance: Use static members for cross-container synchronization, instance members for per-container data
3. Member Definition: Don't forget to define static members in your .cc file
4. Choose Appropriate Lock: Use CoRwLock for read-heavy workloads, CoMutex for simple mutual exclusion
5. Minimal Critical Sections: Keep locked sections as small as possible
6. TaskId Design: Group related tasks that can safely bypass locks

Example: Module with Synchronized Data Structure

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
#include <clio_runtime/comutex.h>
#include <clio_runtime/corwlock.h>
#include <unordered_map>
using namespace clio::run;

struct ModuleData { clio::run::u64 value_ = 0; };
struct ReadDataTask : public clio::run::Task {
  IN clio::run::u32 key_; OUT ModuleData result_data_; OUT clio::run::u32 result_;
};
struct WriteDataTask : public clio::run::Task {
  IN clio::run::u32 key_; IN ModuleData new_data_; OUT clio::run::u32 result_;
};
struct ExclusiveTask : public clio::run::Task { OUT clio::run::u32 result_; };

// In MOD_NAME_runtime.h
class SyncDataExample : public clio::run::Container {
private:
  // Synchronized data structure (container-private in-memory state)
  std::unordered_map<clio::run::u32, ModuleData> data_map_;

  // Synchronization primitives
  static clio::run::CoRwLock data_lock_;        // For data_map_ access
  static clio::run::CoMutex operation_mutex_;   // For exclusive operations

public:
  void ReadData(shared_ptr<ReadDataTask>& task);
  void WriteData(shared_ptr<WriteDataTask>& task);
  void ExclusiveOperation(shared_ptr<ExclusiveTask>& task);
};

// In MOD_NAME_runtime.cc
clio::run::CoRwLock SyncDataExample::data_lock_;
clio::run::CoMutex SyncDataExample::operation_mutex_;

void SyncDataExample::ReadData(shared_ptr<ReadDataTask>& task) {
  clio::run::ScopedCoRwReadLock lock(data_lock_);  // Multiple readers allowed

  // Safe to read data_map_ concurrently
  auto it = data_map_.find(task->key_);
  if (it != data_map_.end()) {
    task->result_data_ = it->second;
    task->result_ = 0;  // Success
  } else {
    task->result_ = 1;  // Not found
  }
}

void SyncDataExample::WriteData(shared_ptr<WriteDataTask>& task) {
  clio::run::ScopedCoRwWriteLock lock(data_lock_);  // Exclusive writer access

  // Safe to modify data_map_ exclusively
  data_map_[task->key_] = task->new_data_;
  task->result_ = 0;  // Success
}

void SyncDataExample::ExclusiveOperation(shared_ptr<ExclusiveTask>& task) {
  clio::run::ScopedCoMutex lock(operation_mutex_);  // Exclusive operation

  // Perform operation that requires complete exclusivity
  // ... complex operation ...
  task->result_ = 0;  // Success
}

This synchronization model ensures thread-safe access to module data structures while maintaining compatibility with CLIO Runtime's cooperative task execution system.

Pool Query and Task Routing

Overview of PoolQuery

PoolQuery is a fundamental component of CLIO Runtime's task routing system that determines where and how tasks are executed across the distributed runtime. It provides flexible routing strategies for load balancing, locality optimization, and distributed execution patterns.

PoolQuery Types

The clio::run::PoolQuery class provides six different routing modes through static factory methods:

1. Local Mode

clio::run::PoolQuery::Local()

• Purpose: Routes tasks to the local node only
• Use Case: Operations that must execute on the calling node
• Example: MPI-based container creation, node-specific diagnostics

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Client usage in MPI environment
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Local(), "my_pool", custom_pool_id).Wait();
}

2. Direct ID Mode

clio::run::PoolQuery::DirectId(ContainerId container_id)

• Purpose: Routes to a specific container by its unique ID
• Use Case: Targeted operations on known containers
• Example: Container-specific configuration changes

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Route to container with ID 42
  auto query = PoolQuery::DirectId(ContainerId(42));
  client.AsyncCustom(query, "new_config", 1).Wait();
}

3. Direct Hash Mode

clio::run::PoolQuery::DirectHash(u32 hash)

• Purpose: Routes using consistent hash-based load balancing
• Use Case: Distributing operations across containers deterministically
• Example: Key-value store operations where keys map to specific containers

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <functional>
#include <string>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  std::string key = "user:42";
  std::string value = "payload";
  // Hash-based routing for a key
  u32 hash = static_cast<u32>(std::hash<std::string>{}(key));
  auto query = PoolQuery::DirectHash(hash);
  client.AsyncCustom(query, value, 1).Wait();
}

4. Range Mode

clio::run::PoolQuery::Range(u32 offset, u32 count)

• Purpose: Routes to a range of containers
• Use Case: Batch operations across multiple containers
• Example: Parallel scan operations, bulk updates

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Process containers 10-19 (10 containers starting at offset 10)
  auto query = PoolQuery::Range(10, 10);
  client.AsyncCustom(query, "update_data", 1).Wait();
}

5. Broadcast Mode

clio::run::PoolQuery::Broadcast()

• Purpose: Routes to all containers in the pool
• Use Case: Global operations affecting all containers
• Example: Configuration updates, global cache invalidation

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Broadcast configuration change to all containers
  auto query = PoolQuery::Broadcast();
  client.AsyncCustom(query, "invalidate_cache", 1).Wait();
}

6. Physical Mode

clio::run::PoolQuery::Physical(u32 node_id)

• Purpose: Routes to a specific physical node by ID
• Use Case: Node-specific operations in distributed deployments
• Example: Remote node administration, cross-node data migration

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Execute on physical node 3
  auto query = PoolQuery::Physical(3);
  client.AsyncCustom(query, "node_diagnostics", 1).Wait();
}

7. Dynamic Mode (Recommended for Create Operations)

clio::run::PoolQuery::Dynamic()

• Purpose: Intelligent routing with automatic caching optimization
• Use Case: Create operations that benefit from local cache checking
• Behavior: Resolved by the container's ScheduleTask() override for cache optimization
  1. Check if pool exists locally using PoolManager
  2. If pool exists: change pool_query to Local (execute locally using existing pool)
  3. If pool doesn't exist: change pool_query to Broadcast (create pool on all nodes)
• Benefits:
  • Avoids redundant pool creation attempts
  • Eliminates unnecessary network overhead for existing pools
  • Automatic fallback to broadcast creation when needed
• Example: Container creation with automatic caching

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Recommended: Use Dynamic() for Create operations
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Dynamic(), "my_pool", custom_pool_id).Wait();

  // Dynamic scheduling will:
  // - Check local cache for "my_pool"
  // - If found: switch to Local mode (fast path)
  // - If not found: switch to Broadcast mode (creation path)
}

PoolQuery Usage Guidelines

Best Practices

1. Never use null queries: Always specify an explicit PoolQuery type
2. Default to Dynamic for Create: Use PoolQuery::Dynamic() for container creation to enable automatic caching optimization
3. Alternative: Use Broadcast or Local explicitly:
   • Use Broadcast() when you want to force distributed creation regardless of cache
   • Use Local() in MPI jobs when you want node-local containers only
4. Consider locality: Prefer local execution to minimize network overhead for regular operations
5. Use appropriate granularity: Match routing mode to operation scope

Common Patterns

Container Creation Pattern (Recommended):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Recommended: Use Dynamic for automatic cache optimization
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Dynamic(), "my_pool_name", custom_pool_id).Wait();
}

Container Creation Pattern (Explicit Broadcast):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Alternative: Use Broadcast to force distributed creation regardless of cache
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Broadcast(), "my_pool_name", custom_pool_id).Wait();
}

Container Creation Pattern (MPI Environments):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // In MPI jobs, Local may be more efficient for node-local containers
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Local(), "my_pool_name", custom_pool_id).Wait();
}

Load-Balanced Operations:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <functional>
#include <string>
#include <vector>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  std::vector<std::string> items = {"a", "b", "c"};
  // Use hash-based routing for even distribution
  for (const auto& item : items) {
    u32 hash = static_cast<u32>(std::hash<std::string>{}(item));
    auto query = PoolQuery::DirectHash(hash);
    client.AsyncCustom(query, item, 1).Wait();
  }
}

Batch Processing:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <algorithm>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // Process containers in chunks
  const u32 total_containers = 100;
  const u32 batch_size = 10;
  for (u32 offset = 0; offset < total_containers; offset += batch_size) {
    u32 count = std::min(batch_size, total_containers - offset);
    auto query = PoolQuery::Range(offset, count);
    client.AsyncCustom(query, "batch", count).Wait();
  }
}

Runtime Routing Implementation

The runtime uses PoolQuery to determine task routing through several stages:

1. Query Validation: Ensures the query parameters are valid
2. Container Resolution: Maps query to specific container(s)
3. Task Distribution: Routes task to appropriate worker queues
4. Load Balancing: Applies distribution strategies for multi-container queries

PoolQuery in Task Definitions

Tasks must include PoolQuery in their constructors (no allocator parameter needed):

#include <clio_runtime/clio_runtime.h>
namespace Method { GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10; }

class CustomTask : public clio::run::Task {
 public:
  CustomTask(const clio::run::TaskId &task_id,
             const clio::run::PoolId &pool_id,
             const clio::run::PoolQuery &pool_query  /* Required parameter */)
      : clio::run::Task(task_id, pool_id, pool_query, Method::kCustom) {
    // Task initialization
  }
};

Advanced PoolQuery Features

Query Introspection

#include <clio_runtime/clio_runtime.h>
void example() {
  using namespace clio::run;
  PoolQuery query = PoolQuery::Range(0, 10);

  // Check routing mode
  if (query.IsRangeMode()) {
    u32 offset = query.GetRangeOffset();
    u32 count = query.GetRangeCount();
    (void)offset; (void)count;  // Process range parameters
  }

  // Get routing mode enum
  RoutingMode mode = query.GetRoutingMode();
  switch (mode) {
    case RoutingMode::Local:
      // Handle local routing
      break;
    case RoutingMode::Broadcast:
      // Handle broadcast
      break;
    default:
      break;  // ... other cases
  }
}

Combining with Task Priorities

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  PoolId pool_id(7000, 0);
  // High-priority broadcast
  auto query = PoolQuery::Broadcast();
  auto task = ipc_manager->NewTask<CustomTask>(
      CreateTaskId(), pool_id, query, "update_data", 1);
  ipc_manager->Send(task).Wait();
}

Troubleshooting PoolQuery Issues

Common Errors:

1. Null Query Error: "NEVER use a null pool query"

   • Solution: Always use a factory method like PoolQuery::Local()

2. Invalid Container ID: Container not found for DirectId query

   • Solution: Verify container exists before using DirectId

3. Range Out of Bounds: Range exceeds available containers

   • Solution: Check pool size before creating Range queries

4. Node ID Invalid: Physical node ID doesn't exist

   • Solution: Validate node IDs against cluster configuration

Client-Server Communication

Client Implementation Patterns

CLIO Runtime uses an async-only client API pattern. All client operations are asynchronous, returning clio::run::Future<TaskType> objects. This design:

• Enables parallel task submission for better performance
• Provides consistent API across all operations
• Allows fine-grained control over task completion timing
• Simplifies the codebase by eliminating duplicate sync/async methods

Async Create Pattern

IMPORTANT: All Module clients MUST update their pool_id_ field with the actual pool ID returned from completed CreateTask operations. This is essential because:

1. CreateTask operations may return a different pool ID than initially specified
2. Pool creation may reuse existing pools with different IDs
3. Subsequent client operations depend on the correct pool ID

Required Pattern for All Client AsyncCreate Methods:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

struct MyClient : public ContainerClient {
  Future<CreateTask> AsyncCreate(const PoolQuery& pool_query,
                                 const std::string& pool_name,
                                 const PoolId& custom_pool_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager->NewTask<CreateTask>(
        CreateTaskId(),
        kAdminPoolId,  // Always use admin pool for CreateTask
        pool_query,
        CreateParams::chimod_lib_name,
        pool_name,
        custom_pool_id,
        this);  // Client pointer for PostWait callback
    return ipc_manager->Send(task);
  }
};

Required Parameters for All AsyncCreate Methods:

1. pool_query: Task routing strategy (use Dynamic() recommended, Broadcast() for non-MPI, Local() for MPI)
2. pool_name: User-provided name for the pool (must be unique, used as file path for file-based modules)
3. custom_pool_id: Explicit pool ID for the container being created (must not be null)
4. Module-specific parameters: Additional parameters specific to the Module (e.g., BDev type, size)

Why Pool ID Update Is Required:

• Pool Reuse: CreateTask is actually a GetOrCreatePoolTask that may return an existing pool
• ID Assignment: The admin Module may assign a different pool ID than requested
• Client Consistency: All subsequent operations must use the correct pool ID
• Distributed Operation: Pool IDs must be consistent across all client instances

Usage Pattern (Caller Side):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <iostream>
void example() {
  using namespace clio::run;
  // Create a client and async create the pool
  MOD_NAME::Client client;
  const PoolId pool_id(7000, 0);

  auto task = client.AsyncCreate(PoolQuery::Dynamic(), "my_pool", pool_id);

  // Wait for completion
  task.Wait();

  // Check result
  if (task->GetReturnCode() != 0) {
    std::cerr << "Create failed: " << task->error_message_.str() << std::endl;
    return;
  }

  // The client's pool_id_ is updated via PostWait callback
  client.pool_id_ = task->new_pool_id_;
  // Now can use the client for operations
  auto op_task = client.AsyncCustom(PoolQuery::Local(), "payload", 1);
  op_task.Wait();
}

Examples of Correct AsyncCreate Implementation:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

struct MyClient : public ContainerClient {
  // Simple AsyncCreate method (admin-style: no extra CreateParams fields)
  Future<CreateTask> AsyncCreate(const PoolQuery& pool_query,
                                 const std::string& pool_name,
                                 const PoolId& custom_pool_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager->NewTask<CreateTask>(
        CreateTaskId(), kAdminPoolId, pool_query,
        CreateParams::chimod_lib_name, pool_name, custom_pool_id, this);
    return ipc_manager->Send(task);
  }
  // A module with extra CreateParams fields (e.g. BDev's size/io_depth/alignment)
  // appends them as trailing NewTask<CreateTask>(...) arguments, forwarded to the
  // task's emplace constructor, before the `this` client pointer.
};

Common Mistakes to Avoid:

• ❌ Using null PoolId for custom_pool_id: Create operations REQUIRE explicit, non-null pool IDs
• ❌ Forgetting PostWait callback: Ensure client pointer is passed to NewTask for pool_id_ update
• ❌ Using original pool_id_: The task may return a different pool ID than initially specified
• ❌ Accessing results before Wait(): Always call task.Wait() before reading task fields
• ❌ Implementing synchronous wrappers: Use async-only pattern, let callers handle waiting
• ❌ Using Local instead of Dynamic/Broadcast: Use Dynamic() (recommended) or Broadcast() for distributed container creation

Critical Validation:

The runtime validates that custom_pool_id is not null during Create operations. If a null PoolId is provided, the Create operation will fail with an error:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  // WRONG - This will fail with error
  PoolId null_id;  // Null pool ID
  client.AsyncCreate(PoolQuery::Broadcast(), "my_pool", null_id).Wait();
  // Error: "Cannot create pool with null PoolId. Explicit pool IDs are required."

  // CORRECT - Always provide explicit pool IDs
  const PoolId custom_pool_id(7000, 0);
  client.AsyncCreate(PoolQuery::Broadcast(), "my_pool", custom_pool_id).Wait();
}

This pattern is mandatory for all Module clients and ensures correct pool ID management throughout the client lifecycle.

Memory Segments

Three shared memory segments are used:

1. Main Segment: Tasks and control structures
2. Client Data Segment: User data buffers
3. Runtime Data Segment: Runtime-only data

IPC Queue

Tasks are submitted via the IPC manager:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  PoolId pool_id(7000, 0);
  // Client side - create and submit task, returns Future
  auto task = ipc_manager->NewTask<CustomTask>(
      CreateTaskId(), pool_id, PoolQuery::Local(), "data", 1);
  auto future = ipc_manager->Send(task);

  // Wait for completion
  future.Wait();

  // Access results
  auto result = future->GetReturnCode();
  (void)result;
}

Memory Management

Task Memory Allocation

Tasks are allocated in private memory using standard new/delete. Use standard C++ types (std::string, std::vector) for task data fields:

// Standard C++ types work in task definitions
std::string my_string = "initial value";
std::vector<int> my_vec = {1, 2, 3};

Best Practices

1. Use standard C++ types (std::string, std::vector) for task data fields
2. Use FullPtr for cross-process references
3. Let RAII handle task cleanup (tasks are clio::run::shared_ptr handles)

Task Allocation Pattern

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  using namespace clio::run::MOD_NAME;
  auto* ipc_manager = CLIO_CPU_IPC;
  PoolId pool_id(7000, 0);
  auto pool_query = PoolQuery::Local();
  std::string input_data = "data";
  u32 operation_id = 1;
  // Client side - allocation (NewTask uses main allocator automatically)
  auto task = ipc_manager->NewTask<CustomTask>(
      CreateTaskId(), pool_id, pool_query, input_data, operation_id);

  // Cleanup is automatic: NewTask returns a shared_ptr handle (RAII); the task
  // is reclaimed when the last handle is destroyed. No DelTask call needed.
  (void)task;
}

CLIO_IPC Buffer Allocation

The CLIO_IPC singleton provides centralized buffer allocation for shared memory operations in client code. Use this for allocating temporary buffers that need to be shared between client and runtime processes.

Important: AllocateBuffer returns ctp::ipc::FullPtr<char>, not ctp::ipc::ShmPtr<>. It is NOT a template function.

Basic Usage

#include <clio_runtime/clio_runtime.h>
#include <cstring>
void example() {
  using namespace clio::run;
  const char* source_data = "example data";
  size_t data_size = 12;

  // Get the IPC manager singleton
  auto* ipc_manager = CLIO_CPU_IPC;

  // Allocate a buffer in shared memory (returns FullPtr<char>)
  size_t buffer_size = 1024;
  ctp::ipc::FullPtr<char> buffer_ptr = ipc_manager->AllocateBuffer(buffer_size);

  // Use the buffer (example: copy data into it)
  char* buffer_data = buffer_ptr.ptr_;
  memcpy(buffer_data, source_data, data_size);

  // Alternative: Use directly
  strncpy(buffer_ptr.ptr_, "example data", buffer_size);

  // The buffer is reclaimed when the framework deallocates it.
}

Use Cases for CLIO_IPC Buffers

• Temporary data transfer: When passing large data to tasks
• Intermediate storage: For computations that need shared memory
• I/O operations: Reading/writing data that needs to be accessible by runtime

Best Practices

#include <clio_runtime/clio_runtime.h>
#include <string>
void example() {
  using namespace clio::run;
  size_t data_size = 1024;
  // ✅ Good: Use CLIO_CPU_IPC for temporary shared buffers
  auto* ipc_manager = CLIO_CPU_IPC;
  ctp::ipc::FullPtr<char> temp_buffer = ipc_manager->AllocateBuffer(data_size);
  (void)temp_buffer;

  // ✅ Good: Use std::string/std::vector for task data fields
  std::string task_string = "persistent data";
  (void)task_string;

  // ❌ Avoid: Don't use CLIO_CPU_IPC for small, simple task parameters
  // Use standard types directly in task definitions instead
}

Task Data Structures

Use standard C++ types for task data fields. The framework handles serialization automatically.

Strings and Vectors

#include <clio_runtime/clio_runtime.h>
namespace Method { GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10; }

// Task definition using standard types
struct CustomTask : public clio::run::Task {
  INOUT std::string input_data_;
  INOUT std::string output_data_;

  // Default constructor
  CustomTask() : clio::run::Task() {}

  // Emplace constructor
  explicit CustomTask(const clio::run::TaskId& task_id,
                      const clio::run::PoolId& pool_id,
                      const clio::run::PoolQuery& pool_query,
                      const std::string& input)
      : clio::run::Task(task_id, pool_id, pool_query, Method::kCustom),
        input_data_(input) {}

  std::string getResult() const {
    return output_data_;
  }
};

#include <clio_runtime/clio_runtime.h>
#include <vector>
namespace Method { GLOBAL_CROSS_CONST clio::run::u32 kProcessArray = 12; }

// Task definition using standard vector
struct ProcessArrayTask : public clio::run::Task {
  INOUT std::vector<clio::run::u32> data_array_;
  INOUT std::vector<float> result_array_;

  // Default constructor
  ProcessArrayTask() : clio::run::Task() {}

  // Emplace constructor
  explicit ProcessArrayTask(const clio::run::TaskId& task_id,
                            const clio::run::PoolId& pool_id,
                            const clio::run::PoolQuery& pool_query,
                            const std::vector<clio::run::u32>& input_data)
      : clio::run::Task(task_id, pool_id, pool_query, Method::kProcessArray),
        data_array_(input_data) {}
};

Serialization Support

Standard types automatically support serialization for task communication:

// Task definition - no manual serialization needed
struct SerializableTask : public clio::run::Task {
  INOUT std::string message_;
  INOUT std::vector<clio::run::u64> timestamps_;

  // Cereal automatically handles standard types
  template<class Archive>
  void serialize(Archive& ar) {
    ar(message_, timestamps_);  // Works automatically
  }
};

GPU-Compatible Data Structures

For GPU-compatible modules, HSHM provides shared-memory data structures (ctp::string, ctp::ipc::vector, ctp::ipc::ring_buffer) with cross-platform annotations. See the Vector Guide, Ring Buffer Guide, and String Guide for details.

Bulk Transfer Support with ar.bulk

For tasks that involve large data transfers (such as I/O operations), CLIO Runtime provides ar.bulk() for efficient bulk data serialization. This feature integrates with the Lightbeam networking layer to enable zero-copy data transfer and RDMA optimization.

Overview

The ar.bulk() method marks data pointers for bulk transfer during task serialization. This is essential for:

• Large I/O Operations: Read/write tasks with multi-megabyte payloads
• Zero-Copy Transfer: Avoiding unnecessary data copies during serialization
• RDMA Optimization: Preparing data for remote direct memory access
• Distributed Execution: Sending tasks with large data buffers across nodes

Bulk Transfer Flags

Two flags control bulk transfer behavior:

// Defined in clio_ctp/lightbeam/lightbeam.h
BULK_EXPOSE  // Metadata only - no data transfer (receiver allocates)
BULK_XFER    // Marks bulk for actual data transmission

Flag Usage:

• BULK_EXPOSE: Sender exposes metadata (size, pointer info) but doesn't transfer data
  • Receiver sees the bulk size and allocates local buffer
  • Useful when receiver will write data (e.g., Read operations)
• BULK_XFER: Marks bulk for actual data transmission
  • Data is transferred over network
  • Used when sender has data to send (e.g., Write operations)

Basic Usage Pattern

Write Operation (Sender Has Data)

For write operations, the sender has data to transfer:

#include <clio_runtime/clio_runtime.h>
#include <clio_ctp/lightbeam/lightbeam.h>  // BULK_EXPOSE / BULK_XFER

struct Block { clio::run::u64 offset_ = 0; clio::run::u64 size_ = 0; };

struct WriteTask : public clio::run::Task {
  IN Block block_;                    // Block to write to
  IN ctp::ipc::ShmPtr<> data_;        // Data buffer pointer
  IN size_t length_;                  // Data length
  OUT clio::run::u64 bytes_written_;  // Result

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // Use BULK_XFER to transfer data from sender to receiver
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(bytes_written_);
    // No bulk transfer needed for output (just metadata)
  }
};

Workflow:

1. Client Side (SerializeIn):
   • Serializes block_ and length_ metadata
   • Marks data_ buffer with BULK_XFER flag
   • Lightbeam transmits the data buffer to receiver
2. Runtime Side (Execute):
   • Receives metadata and data buffer
   • Executes write operation using transferred data
   • Sets bytes_written_ result
3. Client Side (SerializeOut):
   • Receives bytes_written_ result
   • No bulk transfer needed for small output values

Read Operation (Receiver Needs Data)

For read operations, the receiver allocates buffer for incoming data:

#include <clio_runtime/clio_runtime.h>
#include <clio_ctp/lightbeam/lightbeam.h>  // BULK_EXPOSE / BULK_XFER

struct Block { clio::run::u64 offset_ = 0; clio::run::u64 size_ = 0; };

struct ReadTask : public clio::run::Task {
  IN Block block_;                 // Block to read from
  OUT ctp::ipc::ShmPtr<> data_;    // Data buffer pointer (allocated by receiver)
  INOUT size_t length_;            // Requested/actual length
  OUT clio::run::u64 bytes_read_;  // Result

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // Use BULK_EXPOSE - metadata only, receiver will allocate buffer
    ar.bulk(data_, length_, BULK_EXPOSE);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_, bytes_read_);
    // Use BULK_XFER to transfer read data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }
};

Workflow:

1. Client Side (SerializeIn):
   • Serializes block_ and length_ metadata
   • Marks data_ with BULK_EXPOSE (no data sent yet)
   • Receiver sees buffer size needed
2. Runtime Side (Execute):
   • Receives metadata including buffer size
   • Allocates local buffer for data_
   • Executes read operation filling the buffer
   • Sets length_ and bytes_read_ results
3. Client Side (SerializeOut):
   • Marks data_ with BULK_XFER flag
   • Lightbeam transfers read data back to client
   • Client receives length_, bytes_read_, and data buffer

API Reference

// Method on the task archive (Save/Load), not a free function:
void bulk(ctp::ipc::ShmPtr<> ptr, size_t size, uint32_t flags);

Parameters:

• ptr: Pointer to data buffer (ctp::ipc::ShmPtr<>, ctp::ipc::FullPtr, or raw pointer)
• size: Size of data in bytes
• flags: Transfer flags (BULK_EXPOSE or BULK_XFER)

Behavior:

• Records bulk transfer metadata in the archive
• For BULK_XFER: Prepares data for network transmission
• For BULK_EXPOSE: Records metadata only (size and pointer info)
• Integrates with Lightbeam networking for actual data transfer

Advanced Pattern: Bidirectional Transfer

Some operations require data transfer in both directions:

#include <clio_runtime/clio_runtime.h>
#include <clio_ctp/lightbeam/lightbeam.h>  // BULK_XFER

struct ProcessTask : public clio::run::Task {
  INOUT ctp::ipc::ShmPtr<> data_;    // Data buffer (modified in-place)
  INOUT size_t length_;              // Buffer length

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(length_);
    // Send input data to runtime
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_);
    // Send modified data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }
};

Integration with Lightbeam

The ar.bulk() calls integrate seamlessly with the Lightbeam networking layer:

1. Archive Records Bulks:

   • TaskOutputArchive stores bulk metadata in bulk_transfers_ vector
   • Each bulk includes pointer, size, and flags

2. Lightbeam Transmission:

   • Bulks marked BULK_XFER are transmitted via Send() and RecvBulks()
   • Bulks marked BULK_EXPOSE provide metadata only
   • Receiver inspects all bulks to determine buffer sizes

3. Zero-Copy Optimization:

   • Data stays in original buffers during serialization
   • Only pointers and metadata are serialized
   • Actual data transfer handled separately by Lightbeam

Complete Example: BDev Read Task

#include <clio_runtime/clio_runtime.h>
#include <clio_ctp/lightbeam/lightbeam.h>  // BULK_EXPOSE / BULK_XFER

struct Block { clio::run::u64 offset_ = 0; clio::run::u64 size_ = 0; };
namespace Method { GLOBAL_CROSS_CONST clio::run::u32 kRead = 11; }

struct ReadTask : public clio::run::Task {
  IN Block block_;                 // Block descriptor
  OUT ctp::ipc::ShmPtr<> data_;    // Data buffer
  INOUT size_t length_;            // Buffer length
  OUT clio::run::u64 bytes_read_;  // Bytes actually read

  /** SHM default constructor */
  ReadTask()
      : clio::run::Task(), length_(0), bytes_read_(0) {}

  /** Emplace constructor */
  explicit ReadTask(const clio::run::TaskId &task_node,
                    const clio::run::PoolId &pool_id,
                    const clio::run::PoolQuery &pool_query,
                    const Block &block,
                    ctp::ipc::ShmPtr<> data,
                    size_t length)
      : clio::run::Task(task_node, pool_id, pool_query, Method::kRead),
        block_(block), data_(data), length_(length), bytes_read_(0) {}

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // BULK_EXPOSE: Tell receiver the buffer size, but don't send data yet
    // Receiver will allocate local buffer
    ar.bulk(data_, length_, BULK_EXPOSE);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_, bytes_read_);
    // BULK_XFER: Transfer the read data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Copy from another ReadTask */
  void Copy(const ctp::ipc::FullPtr<ReadTask> &other) {
    // REQUIRED: Copy base Task fields first
    Task::Copy(other.template Cast<Task>());
    block_ = other->block_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_read_ = other->bytes_read_;
  }

  /** Aggregate replica OUT fields into this task */
  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &replica_base) {
    Task::AggregateOut(replica_base);
    // For reads, just copy the result from the replica
    Copy(replica_base.template Cast<ReadTask>());
  }
};

Best Practices

DO:

• ✅ Use BULK_XFER when sender has data to transmit (Write operations)
• ✅ Use BULK_EXPOSE when receiver needs to allocate buffer (Read operations)
• ✅ Always specify both SerializeIn() and SerializeOut() for consistency
• ✅ Use ar.bulk() for data buffers larger than a few KB
• ✅ Ensure data buffer lifetime extends until serialization completes

DON'T:

• ❌ Don't use ar.bulk() for small data (< 4KB) - serialize directly instead
• ❌ Don't forget to specify bulk size - it determines receiver buffer allocation
• ❌ Don't mix ar() and ar.bulk() for the same data - choose one approach
• ❌ Don't use BULK_EXPOSE for write operations (sender has data to send)
• ❌ Don't use BULK_XFER in SerializeIn for read operations (no data to send yet)

Performance Considerations

1. Buffer Alignment: Ensure buffers are properly aligned (typically 4KB for I/O operations)
2. Size Thresholds: Use bulk transfer for data > 4KB; use regular serialization for smaller data
3. Zero-Copy: Lightbeam can use zero-copy techniques when data is in shared memory
4. RDMA Ready: The bulk transfer API is designed for future RDMA transport integration

Troubleshooting

Common Issues:

1. Missing Data Transfer:

   • Ensure BULK_XFER flag is used when data should be transmitted
   • Check that SerializeOut uses BULK_XFER for read operations

2. Buffer Size Mismatch:

   • Verify length_ parameter matches actual buffer size
   • Ensure receiver allocates buffer matching the exposed size

3. Serialization Order:

   • Serialize metadata (block, length) before ar.bulk() call
   • This ensures receiver knows buffer size before allocating

Build System Integration

CMakeLists.txt Template

Module CMakeLists.txt files should use the standardized ClioCoreCommon.cmake functions for consistency and proper configuration:

cmake_minimum_required(VERSION 3.10)

# Create both client and runtime libraries for your module
# This creates targets: ${NAMESPACE}_${CHIMOD_NAME}_runtime and ${NAMESPACE}_${CHIMOD_NAME}_client
# CMake aliases: ${NAMESPACE}::${CHIMOD_NAME}_runtime and ${NAMESPACE}::${CHIMOD_NAME}_client
add_chimod_client(
  CHIMOD_NAME YOUR_MODULE_NAME
  SOURCES src/YOUR_MODULE_NAME_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME YOUR_MODULE_NAME
  SOURCES src/YOUR_MODULE_NAME_runtime.cc src/autogen/YOUR_MODULE_NAME_lib_exec.cc
)

# Installation is automatic - no separate install_chimod() call required

CMakeLists.txt Guidelines

DO:

• Use add_chimod_client() and add_chimod_runtime() utility functions (installation is automatic)
• Set CHIMOD_NAME to your module's name
• List source files explicitly in SOURCES parameters
• Include autogen source files in runtime SOURCES
• Keep the CMakeLists.txt minimal and consistent

DON'T:

• Use manual add_library() calls - use the utilities instead
• Call install_chimod() separately - it's handled automatically
• Include relative paths like ../include/* - use proper include directories
• Set custom compile definitions - the utilities handle this
• Manually configure target properties - the utilities provide standard settings

Module Build Functions Reference

add_chimod_client() Function

Creates a Module client library target with automatic dependency management.

add_chimod_client(
  SOURCES source_file1.cc source_file2.cc ...
  [COMPILE_DEFINITIONS definition1 definition2 ...]
  [LINK_LIBRARIES library1 library2 ...]
  [LINK_DIRECTORIES directory1 directory2 ...]
  [INCLUDE_LIBRARIES target1 target2 ...]
  [INCLUDE_DIRECTORIES directory1 directory2 ...]
)

Parameters:

• SOURCES (required): List of source files for the client library
• COMPILE_DEFINITIONS (optional): Additional preprocessor definitions beyond automatic ones
• LINK_LIBRARIES (optional): Additional libraries to link beyond automatic dependencies
• LINK_DIRECTORIES (optional): Additional library search directories
• INCLUDE_LIBRARIES (optional): Target libraries whose include directories should be inherited
• INCLUDE_DIRECTORIES (optional): Additional include directories beyond automatic ones

Automatic Behavior:

• Creates target: ${PACKAGE_NAME}_${CHIMOD_NAME}_client
• Creates alias: ${NAMESPACE}::${CHIMOD_NAME}_client
• Automatically links the core CLIO Runtime library (clio::run::cxx, falling back to ctp::cxx for external builds)
• For non-admin ChiMods: automatically links clio_admin_client and clio_bdev_client
• Automatically includes module headers from include/ directory
• Installs library and headers with proper CMake export configuration

add_chimod_client() vs add_clio_module_client()

add_chimod_client() / add_chimod_runtime() are thin macro wrappers that forward to the canonical add_clio_module_client() / add_clio_module_runtime(). Both spellings work and take identical arguments.

Example:

add_chimod_client(
  SOURCES src/my_module_client.cc
  COMPILE_DEFINITIONS MY_MODULE_DEBUG=1
  LINK_LIBRARIES additional_lib
  INCLUDE_DIRECTORIES ${EXTERNAL_INCLUDE_DIR}
)

add_chimod_runtime() Function

Creates a Module runtime library target with automatic dependency management.

add_chimod_runtime(
  SOURCES source_file1.cc source_file2.cc ...
  [COMPILE_DEFINITIONS definition1 definition2 ...]
  [LINK_LIBRARIES library1 library2 ...]
  [LINK_DIRECTORIES directory1 directory2 ...]
  [INCLUDE_LIBRARIES target1 target2 ...]
  [INCLUDE_DIRECTORIES directory1 directory2 ...]
)

Parameters:

• SOURCES (required): List of source files for the runtime library (include autogen files)
• COMPILE_DEFINITIONS (optional): Additional preprocessor definitions beyond automatic ones
• LINK_LIBRARIES (optional): Additional libraries to link beyond automatic dependencies
• LINK_DIRECTORIES (optional): Additional library search directories
• INCLUDE_LIBRARIES (optional): Target libraries whose include directories should be inherited
• INCLUDE_DIRECTORIES (optional): Additional include directories beyond automatic ones

Automatic Behavior:

• Creates target: ${PACKAGE_NAME}_${CHIMOD_NAME}_runtime
• Creates alias: ${NAMESPACE}::${CHIMOD_NAME}_runtime
• Automatically defines CLIO_RUNTIME=1 for runtime code
• Automatically links the core CLIO Runtime library (clio::run::cxx, falling back to ctp::cxx for external builds)
• Automatically links rt library for POSIX real-time support (Linux only — Windows ships its AIO support in kernel32/winsock)
• For non-admin ChiMods: automatically links both clio_admin_runtime and clio_admin_client
• Automatically includes module headers from include/ directory
• Links to client library if it exists
• Installs library and headers with proper CMake export configuration

Example:

add_chimod_runtime(
  SOURCES 
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
  COMPILE_DEFINITIONS MY_MODULE_RUNTIME_DEBUG=1
  LINK_LIBRARIES libaio
  INCLUDE_DIRECTORIES ${LIBAIO_INCLUDE_DIR}
)

Configuration Requirements

Before using these functions, ensure your Module directory contains:

1. clio_mod.yaml: Module configuration file defining the module name

   module_name: my_module

2. Include structure: Headers organized as include/[namespace]/[module_name]/

3. Source files: Client and runtime implementations with autogen files for runtime

Typical Usage Pattern

Most ChiMods use both functions together:

# Create client library
add_chimod_client(
  SOURCES src/my_module_client.cc
)

# Create runtime library
add_chimod_runtime(
  SOURCES 
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
)

Function Dependencies: Both functions automatically handle common dependencies:

• Core Library: Automatically links appropriate CLIO Runtime core library
• Runtime Libraries: add_chimod_runtime() automatically links rt library for async I/O operations
• Admin Module Integration: For non-admin chimods, both functions automatically link admin libraries and include admin headers
• Client-Runtime Linking: Runtime automatically links to client library when both exist

This eliminates the need for manual dependency configuration in individual Module CMakeLists.txt files.

Target Naming and Linking

Target Format

The C++ namespace comes from clio_repo.yaml (e.g. clio::run, clio::cte). Because :: is illegal in filenames, a package name is derived from it by replacing :: with _ — clio::run → clio_run, clio::cte → clio_cte. Raw target names and install paths use the package name; CMake aliases use the namespace.

Target Names (also the library filename, lib<target>.so):

• Runtime: ${PACKAGE_NAME}_${CHIMOD_NAME}_runtime (e.g., clio_run_myMod_runtime)
• Client: ${PACKAGE_NAME}_${CHIMOD_NAME}_client (e.g., clio_run_myMod_client)

CMake Aliases (Recommended):

• Runtime: ${NAMESPACE}::${CHIMOD_NAME}_runtime (e.g., clio::run::admin_runtime)
• Client: ${NAMESPACE}::${CHIMOD_NAME}_client (e.g., clio::run::admin_client)

Package Names:

• Per-module package: ${PACKAGE_NAME}_${CHIMOD_NAME} (e.g., clio_run_admin), installed under lib/cmake/
• Umbrella package: clio-core (legacy alias: iowarp-core). A single find_package(clio-core CONFIG REQUIRED) provides all ctp::* targets, clio::run::cxx, the admin client/runtime, and the ChiMod build utilities — external projects normally need nothing else.

LIB_NAME overrides the derived name

Passing LIB_NAME foo to add_chimod_client() pins the target and library filename to foo_client instead of the derived form. The in-tree admin and bdev modules use this, which is why their targets are clio_admin_client and clio_bdev_client rather than clio_run_admin_client / clio_run_bdev_client. The :: aliases are unaffected.

External Application Linking:

# External applications typically only need Module client libraries.
# The umbrella package provides all ctp::* targets, clio::run::cxx,
# clio::run::admin_client / admin_runtime, and the ChiMod build utilities.
find_package(clio-core CONFIG REQUIRED)
find_package(Threads REQUIRED)

target_link_libraries(my_external_app
  clio::run::admin_client           # Admin client (includes all dependencies)
  ${CMAKE_THREAD_LIBS_INIT}         # Threading support
)
# Note: clio::run::cxx is automatically included by Module client libraries

Internal Development Linking:

# For internal development within the CLIO Runtime project
target_link_libraries(internal_app
  clio::run::admin_client           # Module client
  clio::run::bdev_client            # BDev client
  # Core dependencies are automatically linked by Module libraries
)

Compatibility aliases. Installed module packages also emit the historical clio_<suffix>:: alias namespace (e.g. clio_cte::core_client alongside clio::cte::core_client), plus wrp_<suffix>:: for non-run namespaces, so existing downstream target_link_libraries lines keep resolving.

Automatic Dependencies

The Module build functions automatically handle common dependencies:

For Runtime Code:

• rt library: Automatically linked for POSIX real-time library support (async I/O operations)
• Admin Module: Automatically linked for all non-admin ChiMods (both runtime and client)
• Admin includes: Automatically added to include directories for non-admin ChiMods

For All ChiMods:

• Creates both client and runtime shared libraries
• Sets proper include directories (include/, ${CMAKE_SOURCE_DIR}/include)
• Automatically links core CLIO Runtime dependencies
• Sets required compile definitions (CLIO_RUNTIME=1 for runtime targets, plus DEBUG / NDEBUG per build config)
• Configures proper build flags and settings

Simplified Development: Module developers no longer need to manually specify:

• rt library dependencies
• Admin Module dependencies (clio_admin_runtime, clio_admin_client)
• Admin include directories
• Core CLIO Runtime library dependencies
• Common linking patterns

Important Note for External Applications: External applications linking against Module libraries receive all necessary dependencies automatically. The Module client libraries include the core CLIO Runtime library as a transitive dependency.

Automatic Installation: The Module build functions automatically handle installation:

• Installs libraries to the correct destination
• Sets up proper runtime paths
• Configures installation properties
• Includes automatic dependencies in exported CMake packages
• No separate install_chimod() call required

Targets Created by Module Functions

When you call add_chimod_client() and add_chimod_runtime() with CHIMOD_NAME YOUR_MODULE_NAME, they create the following CMake targets using the underscore-based naming format:

Target Naming System

• Actual Target Names: ${PACKAGE_NAME}_${CHIMOD_NAME}_runtime and ${PACKAGE_NAME}_${CHIMOD_NAME}_client
• CMake Aliases: ${NAMESPACE}::${CHIMOD_NAME}_runtime and ${NAMESPACE}::${CHIMOD_NAME}_client (recommended)
• Package Names: ${PACKAGE_NAME}_${CHIMOD_NAME} (for find_package())

Runtime Target: ${PACKAGE_NAME}_${CHIMOD_NAME}_runtime

• Target Name: clio_run_YOUR_MODULE_NAME_runtime (or clio_admin_runtime when LIB_NAME pins it)
• CMake Alias: clio::run::YOUR_MODULE_NAME_runtime (e.g., clio::run::admin_runtime) - recommended for linking
• Type: Shared library (.so file)
• Purpose: Contains server-side execution logic, runs in the CLIO Runtime process
• Compile Definitions:
  • CLIO_RUNTIME=1 - Selects the runtime-side code paths
  • DEBUG / NDEBUG - Per build configuration
• Include Directories:
  • include/ - Local module headers
  • $\{CMAKE_SOURCE_DIR\}/include - Clio framework headers
• Dependencies: Links against clio::run::cxx, rt library (automatic, Linux), admin dependencies (automatic)

Client Target: ${PACKAGE_NAME}_${CHIMOD_NAME}_client

• Target Name: clio_run_YOUR_MODULE_NAME_client (or clio_admin_client when LIB_NAME pins it)
• CMake Alias: clio::run::YOUR_MODULE_NAME_client (e.g., clio::run::admin_client) - recommended for linking
• Type: Shared library (.so file)
• Purpose: Contains client-side API, runs in user processes
• Compile Definitions:
  • DEBUG / NDEBUG - Per build configuration (no CLIO_RUNTIME)
• Include Directories:
  • include/ - Local module headers
  • $\{CMAKE_SOURCE_DIR\}/include - Clio framework headers
• Dependencies: Links against clio::run::cxx, admin dependencies (automatic)

Namespace Configuration

The namespace is automatically read from clio_repo.yaml files. The system searches up the directory tree from the CMakeLists.txt location to find the first clio_repo.yaml file:

Main project clio_repo.yaml:

namespace: clio::run  # Main project namespace

Module repository chimods/clio_repo.yaml:

namespace: clio::mods   # Modules get this namespace

This means modules in the chimods/ directory will use the clio::mods namespace, creating targets like clio_mods_admin_runtime (the package name is the namespace with :: replaced by _), while other components use the main project namespace.

Example Output Files

For a module named "admin" with namespace clio::mods (from chimods/clio_repo.yaml), the build produces:

build/bin/libclio_mods_admin_runtime.so    # Runtime library
build/bin/libclio_mods_admin_client.so     # Client library

Using the Targets

You can reference these targets in your CMakeLists.txt using the full target name:

# Add custom properties to the runtime target
set_target_properties(${CLIO_RUN_PACKAGE_NAME}_${CHIMOD_NAME}_runtime PROPERTIES
  VERSION 1.0.0
  SOVERSION 1
)

# Add additional dependencies if needed
target_link_libraries(${CLIO_RUN_PACKAGE_NAME}_${CHIMOD_NAME}_runtime PRIVATE some_external_lib)

# Or use the global property to get the actual target name
get_property(RUNTIME_TARGET GLOBAL PROPERTY ${CHIMOD_NAME}_RUNTIME_TARGET)
target_link_libraries(${RUNTIME_TARGET} PRIVATE some_external_lib)

Module Configuration (clio_mod.yaml)

name: MOD_NAME
version: 1.0.0
description: "Module description"
author: "Author Name"
methods:
  - kCreate
  - kCustom
dependencies: []

Auto-Generated Method Files

Each module requires an auto-generated methods file at include/[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h. This file must:

1. Include clio_runtime.h: Required for GLOBAL_CROSS_CONST macro
2. Use namespace constants: Define methods as GLOBAL_CROSS_CONST clio::run::u32 values
3. Follow naming convention: Method names should start with k (e.g., kCreate, kCustom)

Required Template:

#include <clio_runtime/clio_runtime.h>

// autogen/MOD_NAME_methods.h — shown at `namespace clio::run::MOD_NAME` scope.
namespace Method {
  // Standard inherited methods (always include these)
  GLOBAL_CROSS_CONST clio::run::u32 kCreate = 0;
  GLOBAL_CROSS_CONST clio::run::u32 kDestroy = 1;
  GLOBAL_CROSS_CONST clio::run::u32 kNodeFailure = 2;
  GLOBAL_CROSS_CONST clio::run::u32 kRecover = 3;
  GLOBAL_CROSS_CONST clio::run::u32 kMigrate = 4;
  GLOBAL_CROSS_CONST clio::run::u32 kUpgrade = 5;

  // Module-specific methods (customize these)
  GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10;
  // Add more module-specific methods starting from 10+
}  // namespace Method

Important Notes:

• GLOBAL_CROSS_CONST is required: Do not use const or constexpr - use GLOBAL_CROSS_CONST
• Include clio_runtime.h: This header defines the GLOBAL_CROSS_CONST macro
• Standard methods 0-5: Always include the inherited methods (kCreate through kUpgrade)
• Custom methods 10+: Start custom methods from ID 10 to avoid conflicts
• No static casting needed: Use method values directly (e.g., method_ = Method::kCreate;)

Runtime Entry Points

Use the CLIO_TASK_CC macro to define module entry points:

// At the end of your runtime source file (_runtime.cc)
CLIO_TASK_CC(your_namespace::YourContainerClass)

This macro automatically generates all required extern "C" functions and gets the module name from YourContainerClass::CreateParams::chimod_lib_name:

• alloc_chimod() - Creates container instance
• new_chimod() - Creates and initializes container
• get_chimod_name() - Returns module name
• destroy_chimod() - Destroys container instance

Requirements for CLIO_TASK_CC to work:

1. Your runtime class must define a public typedef: using CreateParams = your_namespace::CreateParams;
2. Your CreateParams struct must have: static constexpr const char* chimod_lib_name = "your_module_name";

IMPORTANT: The chimod_lib_name should NOT include the _runtime suffix. The module manager automatically appends _runtime when loading the library. For example, use "clio_run_mymodule" not "clio_run_mymodule_runtime".

Example:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
// Shown at `namespace clio::run::your_module` scope.
struct YourCreateParams {
  static constexpr const char* chimod_lib_name = "clio_your_module";
  template <class Archive> void serialize(Archive&) {}  // other parameters
};

class Runtime : public clio::run::Container {
public:
  using CreateParams = YourCreateParams;  // Required for CLIO_TASK_CC
  // ... rest of class (implements the Container virtuals)
};

• is_clio_chimod_ - Module identification flag

Auto-Generated Code Pattern

Overview

ChiMods use auto-generated source files to implement the Container virtual APIs (Run, Monitor, Del, SaveIn, LoadIn, SaveOut, LoadOut, NewCopy). This approach provides consistent dispatch logic and reduces boilerplate code.

New Pattern: Auto-Generated Source Files

Instead of using inline functions in headers, ChiMods now use auto-generated .cc source files that implement the virtual methods directly. This pattern:

• Eliminates inline dispatchers: Virtual methods are implemented directly in auto-generated source
• Reduces header dependencies: No need to include autogen headers in runtime files
• Improves compilation: Source files compile once, not in every including file
• Maintains consistency: All ChiMods use the same dispatch pattern

File Structure

src/
└── autogen/
    └── MOD_NAME_lib_exec.cc    # Auto-generated virtual method implementations

The auto-generated source file contains:

• Container virtual method implementations (Run, Del, etc.)
• Switch-case dispatch based on method IDs
• Proper task type casting and method invocation
• IPC manager integration for task lifecycle management

Auto-Generated Source Template

/**
 * Auto-generated execution implementation for MOD_NAME Module (conceptual).
 * The real dispatcher lives in src/autogen/MOD_NAME_lib_exec.cc; today Run has
 * the signature `TaskResume Run(u32 method, shared_ptr<Task> task_ptr)` and
 * there is no Del (tasks are shared_ptr RAII handles). Shown here to illustrate
 * the switch-case dispatch shape.
 */

#include <clio_runtime/MOD_NAME/MOD_NAME_runtime.h>
#include <clio_runtime/MOD_NAME/autogen/MOD_NAME_methods.h>
#include <clio_runtime/clio_runtime.h>

namespace clio::run::MOD_NAME {

//==============================================================================
// Runtime Virtual API Implementations
//==============================================================================

void Runtime::Run(clio::run::u32 method, ctp::ipc::FullPtr<clio::run::Task> task_ptr, clio::run::RunContext& rctx) {
  switch (method) {
    case Method::kCreate: {
      Create(task_ptr.Cast<CreateTask>(), rctx);
      break;
    }
    case Method::kDestroy: {
      Destroy(task_ptr.Cast<DestroyTask>(), rctx);
      break;
    }
    case Method::kCustom: {
      Custom(task_ptr.Cast<CustomTask>(), rctx);
      break;
    }
    default: {
      // Unknown method - do nothing
      break;
    }
  }
}

void Runtime::Del(clio::run::u32 method, ctp::ipc::FullPtr<clio::run::Task> task_ptr) {
  // Use IPC manager to deallocate task from shared memory
  auto* ipc_manager = CLIO_CPU_IPC;
  
  switch (method) {
    case Method::kCreate: {
      ipc_manager->DelTask(task_ptr.Cast<CreateTask>());
      break;
    }
    case Method::kDestroy: {
      ipc_manager->DelTask(task_ptr.Cast<DestroyTask>());
      break;
    }
    case Method::kCustom: {
      ipc_manager->DelTask(task_ptr.Cast<CustomTask>());
      break;
    }
    default: {
      // For unknown methods, still try to delete from main segment
      ipc_manager->DelTask(task_ptr);
      break;
    }
  }
}

// SaveIn, LoadIn, SaveOut, LoadOut, and NewCopy follow similar patterns...

} // namespace clio::run::MOD_NAME

CMake Integration

The auto-generated source file must be included in the RUNTIME_SOURCES:

add_chimod_client(
  CHIMOD_NAME MOD_NAME
  SOURCES src/MOD_NAME_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME MOD_NAME
  SOURCES src/MOD_NAME_runtime.cc src/autogen/MOD_NAME_lib_exec.cc
)

Benefits

1. Cleaner Runtime Code: Runtime implementations focus on business logic, not dispatching
2. Better Compilation: Source files compile once instead of being inlined in every header include
3. Consistent Pattern: All ChiMods use identical dispatch logic
4. Header Simplification: No need to include complex autogen headers
5. Better IDE Support: Proper source files work better with IDEs than inline templates

Important Notes

• Auto-generated files: These files should be generated by tools, not hand-written
• Do not edit: Manual changes to autogen files will be lost when regenerated
• Template consistency: All ChiMods should follow the same autogen template pattern
• Build integration: Autogen source files must be included in CMake build

External Module Development

When developing ChiMods in external repositories (outside the main CLIO Runtime project), you need to link against the installed CLIO Runtime libraries and use the CMake package discovery system.

Prerequisites

Before developing external ChiMods, ensure CLIO Runtime is properly installed:

# Configure and build CLIO Runtime
cmake --preset debug
cmake --build build

# Install CLIO Runtime libraries and CMake configs  
cmake --install build --prefix /usr/local

This installs:

• Core CLIO Runtime library (libcxx.so)
• Module libraries (libclio_admin_runtime.so, libclio_admin_client.so, etc.)
• CMake package configuration files for external discovery
• Header files for development

External Module Repository Structure

Your external Module repository should follow this structure:

my_external_chimod/
├── clio_repo.yaml          # Repository namespace configuration
├── CMakeLists.txt              # Root CMake configuration
├── modules/                    # Module modules directory (name is flexible)
│   └── my_module/
│       ├── clio_mod.yaml   # Module configuration
│       ├── CMakeLists.txt      # Module build configuration  
│       ├── include/
│       │   └── [namespace]/
│       │       └── my_module/
│       │           ├── my_module_client.h
│       │           ├── my_module_runtime.h
│       │           ├── my_module_tasks.h
│       │           └── autogen/
│       │               └── my_module_methods.h
│       └── src/
│           ├── my_module_client.cc
│           ├── my_module_runtime.cc
│           └── autogen/
│               └── my_module_lib_exec.cc

Note: The directory name for modules (shown here as modules/) is flexible. You can use chimods/, components/, plugins/, or any other name that fits your project structure. The directory name doesn't need to match the namespace.

Repository Configuration (clio_repo.yaml)

Create a clio_repo.yaml file in your repository root to define the namespace:

# Repository-level configuration
namespace: myproject      # Your custom namespace (replaces "clio::run")

This namespace will be used for:

• CMake target names: myproject_my_module_runtime, myproject_my_module_client
• Library file names: libmyproject_my_module_runtime.so, libmyproject_my_module_client.so
• C++ namespaces: myproject::my_module

Root CMakeLists.txt

Your repository's root CMakeLists.txt must find and link to the installed CLIO Runtime packages:

cmake_minimum_required(VERSION 3.20)
project(my_external_chimod)

set(CMAKE_CXX_STANDARD_20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Find required CLIO Runtime packages
# These packages are installed by 'cmake --install build --prefix /usr/local'
# `clio-core` is the canonical umbrella package; `iowarp-core` still resolves
# to the same Config. It brings in clio::run::cxx, the admin client/runtime,
# every ctp::* target, and the ChiMod build utilities.
find_package(clio-core CONFIG REQUIRED)

# Set CMAKE_PREFIX_PATH if CLIO Runtime is installed in a custom location
# set(CMAKE_PREFIX_PATH "/path/to/install" ${CMAKE_PREFIX_PATH})

# ClioCoreCommon.cmake is included automatically by find_package(clio-core).
# This provides add_chimod_client(), add_chimod_runtime(), and other build functions

# Add subdirectories containing your ChiMods
add_subdirectory(modules/my_module)  # Use your actual directory name

Module CMakeLists.txt

Each Module's CMakeLists.txt uses the standard CLIO Runtime build utilities:

cmake_minimum_required(VERSION 3.20)

# Create both client and runtime libraries using standard CLIO Runtime utilities
# These functions are provided by ClioCoreCommon.cmake (included via find_package(clio-core))
# Creates targets: my_namespace_my_module_client, my_namespace_my_module_runtime
# Creates aliases: my_namespace::my_module_client, my_namespace::my_module_runtime
add_chimod_client(
  CHIMOD_NAME my_module
  SOURCES src/my_module_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME my_module
  SOURCES
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
)

# Installation is automatic - no separate install_chimod() call required
# Package name: my_namespace_my_module (for find_package)

# Optional: Add additional dependencies if your Module needs external libraries
# get_property(RUNTIME_TARGET GLOBAL PROPERTY my_module_RUNTIME_TARGET)
# get_property(CLIENT_TARGET GLOBAL PROPERTY my_module_CLIENT_TARGET)
# target_link_libraries(${RUNTIME_TARGET} PRIVATE some_external_lib)
# target_link_libraries(${CLIENT_TARGET} PRIVATE some_external_lib)

External Applications Using Your Module

Once installed, external applications can find and link to your Module. Based on our external unit test patterns (see test/unit/external-chimod/CMakeLists.txt):

# External application CMakeLists.txt
find_package(clio-core CONFIG REQUIRED)        # Core CLIO Runtime + utilities + admin
find_package(my_namespace_my_module REQUIRED)  # Your Module package

# Simple linking pattern - Module libraries include all dependencies
target_link_libraries(my_external_app
  my_namespace::my_module_client    # Your Module client
  clio::run::admin_client           # Admin client (if needed)
  ${CMAKE_THREAD_LIBS_INIT}         # Threading support
)
# Core CLIO Runtime library is automatically included by Module dependencies

External Module Implementation

Your external Module implementation follows the same patterns as internal ChiMods:

CreateParams Configuration

In your my_module_tasks.h, the CreateParams must reference your custom namespace:

struct CreateParams {
  // Your module-specific parameters
  std::string config_data_;
  clio::run::u32 worker_count_;
  
  // CRITICAL: Library name must match your namespace
  static constexpr const char* chimod_lib_name = "myproject_my_module";
  
  // Constructors and serialization...
};

C++ Namespace

Use your custom namespace throughout your implementation:

// In all header and source files
namespace myproject::my_module {

// Your Module implementation...
class Runtime : public clio::run::Container {
  // Implementation...
};

class Client : public clio::run::ContainerClient {  
  // Implementation...
};

} // namespace myproject::my_module

Building External ChiMods

# Configure your external Module project
mkdir build && cd build
cmake ..

# Build your ChiMods
make

# Optional: Install your ChiMods
make install

The build system will automatically:

• Link all necessary core CLIO Runtime dependencies
• Link against clio::run::admin_client and clio::run::admin_runtime (for non-admin modules)
• Generate libraries with your custom namespace: libmyproject_my_module_runtime.so
• Configure proper include paths and dependencies

Usage in Applications

Applications using your external Module would reference it as:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <clio_runtime/admin/admin_client.h>

void example() {
  using namespace clio::run;
  // Initialize CLIO Runtime (client mode with embedded runtime)
  CLIO_INIT(RuntimeMode::kClient, true);

  // Create your Module client
  const PoolId pool_id = PoolId(7000, 0);
  MOD_NAME::Client client(pool_id);

  // Use your Module (async-only API returns a Future)
  auto create = client.AsyncCreate(PoolQuery::Dynamic(), "my_pool", pool_id);
  create.Wait();
}

CLIO_INIT Initialization Modes

CLIO Runtime provides a unified initialization function CLIO_INIT() that supports different operational modes:

Client Mode with Embedded Runtime (Most Common):

// Initialize both client and runtime in single process
// Recommended for: Applications, unit tests, and benchmarks
clio::run::CLIO_INIT(clio::run::RuntimeMode::kClient, true);

Client-Only Mode (Advanced):

// Initialize client only - connects to external runtime
// Recommended for: Production deployments with separate runtime process
clio::run::CLIO_INIT(clio::run::RuntimeMode::kClient, false);

Runtime/Server Mode (Advanced):

// Initialize runtime/server only - no client
// Recommended for: Standalone runtime processes
clio::run::CLIO_INIT(clio::run::RuntimeMode::kServer, false);

Usage Example (Unit Tests/Benchmarks):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>

void example() {
  using namespace clio::run;
  // Initialize both client and runtime in single process
  CLIO_INIT(RuntimeMode::kClient, true);

  // Create your Module client
  const PoolId pool_id = PoolId(7000, 0);
  MOD_NAME::Client client(pool_id);

  // Test your Module functionality
  auto create = client.AsyncCreate(PoolQuery::Dynamic(), "test_pool", pool_id);
  create.Wait();

  // Assertions and test logic...
}

When to Use Each Mode:

• Client with Embedded Runtime (kClient, true): Unit tests, benchmarks, and standalone applications
• Client Only (kClient, false): Production applications connecting to existing external runtime
• Server/Runtime Only (kServer, false): Dedicated runtime processes

Dependencies and Installation Paths

External Module development requires these components to be installed:

1. Core Package: clio-core (includes the main library and the ClioCoreCommon.cmake utilities)
2. Admin Module: clio::run::admin_client and clio::run::admin_runtime (required for most modules)
3. CMake Configs: Package discovery files (automatically installed with packages)
4. Headers: All Clio framework headers (installed with packages)

All build utilities (add_chimod_client(), add_chimod_runtime()) are automatically available via find_package(clio-core CONFIG).

If CLIO Runtime is installed in a custom location, set CMAKE_PREFIX_PATH:

export CMAKE_PREFIX_PATH="/path/to/[namespace]/install:$CMAKE_PREFIX_PATH"

Common External Development Issues

ClioCoreCommon.cmake Not Found:

• Ensure CLIO Runtime was installed with cmake --install build --prefix <path>
• Verify CMAKE_PREFIX_PATH includes the CLIO Runtime installation directory
• Check that find_package(clio-core CONFIG REQUIRED) succeeded (ClioCoreCommon.cmake is included automatically)

Library Name Mismatch:

• Ensure CreateParams::chimod_lib_name exactly matches your namespace and module name
• For namespace "myproject" and module "my_module": chimod_lib_name = "myproject_my_module"
• The system automatically appends "_runtime" to find the runtime library
• Target names use format: myproject_my_module_runtime and myproject_my_module_client

Missing Dependencies:

• The Module build functions automatically link admin and rt library dependencies
• Ensure all external dependencies (Boost, MPI, etc.) are available in your build environment
• Use the same dependency versions that CLIO Runtime was built with
• For runtime code, rt library is automatically included for async I/O support

External Module Checklist

• Repository Configuration: clio_repo.yaml with custom namespace
• CMake Setup: Root CMakeLists.txt finds the clio-core package
• Module Configuration: clio_mod.yaml with method definitions
• Library Name: CreateParams::chimod_lib_name matches namespace pattern
• C++ Namespace: All code uses custom namespace consistently
• Build Integration: Module CMakeLists.txt uses add_chimod_client() and add_chimod_runtime() (installation is automatic)
• Dependencies: All required external libraries available at build time
• Automatic Linking: Rely on Module build functions for rt and admin dependencies

Example Module

See the chimods/MOD_NAME directory for a complete working example that demonstrates:

• Task definition with proper constructors
• Client API with async-only methods
• Runtime container with execution logic
• Build system integration
• YAML configuration

Creating a New Module

1. Copy the MOD_NAME template directory
2. Rename all MOD_NAME occurrences to your module name
3. Update the clio_mod.yaml configuration
4. Define your tasks in the _tasks.h file
5. Implement client API in _client.h/cc
6. Implement runtime logic in _runtime.h/cc
7. Add CLIO_TASK_CC(YourContainerClass) at the end of runtime source
8. Add to the build system
9. Test with client and runtime

Recent Changes and Best Practices

Container Initialization Pattern

Starting with the latest version, container initialization has been simplified:

1. No Separate Init Method: The Init method has been merged with Create
2. Create Does Everything: The Create method now handles both container creation and initialization
3. Access to Task Data: Since Create receives the CreateTask, you have access to pool_id and pool_query from the task

RAII Task Cleanup

Task cleanup is automatic — tasks are clio::run::shared_ptr<Task> handles:

1. No Custom Del Methods Required: There is no Del/DelTask step to implement
2. RAII Handles Cleanup: The task is reclaimed when the last shared_ptr handle is destroyed
3. Allocator Reclamation: Task memory returns to its allocator automatically

Simplified Module Entry Points

Module entry points are now hidden behind the CLIO_TASK_CC macro:

1. Single Macro Call: Replace complex extern "C" blocks with one macro
2. Automatic Container Integration: Works seamlessly with clio::run::Container base class
3. Cleaner Module Code: Eliminates boilerplate entry point code

// Old approach (complex extern "C" block)
extern "C" {
  clio::run::Container* alloc_chimod() { /* ... */ }
  clio::run::Container* new_chimod(/*...*/) { /* ... */ }
  const char* get_chimod_name() { /* ... */ }
  void destroy_chimod(/*...*/) { /* ... */ }
  bool is_clio_chimod_ = true;
}

// New approach (simple macro)
CLIO_TASK_CC(clio::run::MOD_NAME::Runtime)

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume Create(shared_ptr<CreateTask>& task) {
  CLIO_TASK_BODY_BEGIN
  // Container is already initialized via Init() before Create is called
  // Do NOT call Init() here

  // Container-specific initialization logic
  // All tasks will be routed through the external queue lanes
  // which are automatically mapped to workers at runtime startup
  (void)task;
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

shared_ptr Parameter Pattern

All runtime task handlers take the task as clio::run::shared_ptr<TaskType>&:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume Custom(shared_ptr<CustomTask>& task) {
  CLIO_TASK_BODY_BEGIN
  (void)task;
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

Benefits of shared_ptr handles:

• Shared Memory Safety: Provides safe access across process boundaries
• Automatic Dereferencing: Use task->field just like raw pointers
• Memory Management: RAII reclaims the task when the last handle is destroyed
• Null Checking: Use task.IsNull() to check validity

Custom Namespace Configuration

Overview

While the default namespace is clio::run, you can customize the namespace for your Module modules. This is useful for:

• Project Branding: Use your own project or company namespace
• Avoiding Conflicts: Prevent naming conflicts with other Module collections
• Module Organization: Group related modules under a custom namespace

Configuring Custom Namespace

The namespace is controlled by the clio_repo.yaml file in your project root:

namespace: your_custom_namespace

For example:

namespace: mycompany

Required Changes for Custom Namespace

When using a custom namespace, you must update several components:

1. CreateParams chimod_lib_name

The most critical change is updating the chimod_lib_name in your CreateParams:

#include <clio_runtime/clio_runtime.h>
// Default clio namespace
struct DefaultCreateParams {
  static constexpr const char* chimod_lib_name = "clio_your_module";
};

// Custom namespace example
struct CustomCreateParams {
  static constexpr const char* chimod_lib_name = "mycompany_your_module";
};

2. Module Namespace Declaration

Update your module's C++ namespace:

// Default
namespace clio::run::your_module {
  // module code
}

// Custom
namespace mycompany::your_module {
  // module code
}

3. CMake Library Names

The CMake system automatically uses your custom namespace. Libraries will be named:

• Default: libclio_run_module_runtime.so, libclio_run_module_client.so
• Custom: libmycompany_module_runtime.so, libmycompany_module_client.so

4. Runtime Integration

If your runtime code references the admin module or other system modules, update the references:

// Default admin module reference
auto* admin_chimod = module_manager->GetChiMod("clio_admin");

// Custom namespace admin module
auto* admin_chimod = module_manager->GetChiMod("mycompany_admin");

Checklist for Custom Namespace

• Update clio_repo.yaml with your custom namespace
• Update CreateParams::chimod_lib_name to use custom namespace prefix
• Update C++ namespace declarations in all module files
• Update runtime references to admin module and other system modules
• Update any hardcoded module names in configuration or startup code
• Rebuild all modules after namespace changes
• Update library search paths if needed for deployment

Example: Complete Custom Namespace Module

# clio_repo.yaml
namespace: mycompany

#include <clio_runtime/admin/admin_tasks.h>
// mymodule_tasks.h
namespace mycompany::mymodule {

struct CreateParams {
  static constexpr const char* chimod_lib_name = "mycompany_mymodule";
  template <class Archive> void serialize(Archive&) {}  // other parameters
};

// Admin's BaseCreateTask with Method::kCreate (0)
using CreateTask = clio::run::admin::BaseCreateTask<CreateParams, /*MethodId=*/0>;

}  // namespace mycompany::mymodule

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/container.h>
// mymodule_runtime.h
namespace mycompany::mymodule {

struct CreateParams {
  static constexpr const char* chimod_lib_name = "mycompany_mymodule";
};

class Runtime : public clio::run::Container {
public:
  using CreateParams = mycompany::mymodule::CreateParams;  // Required for CLIO_TASK_CC
  // ... rest of class (implements the Container virtuals)
};

}  // namespace mycompany::mymodule

// mymodule_runtime.cc
CLIO_TASK_CC(mycompany::mymodule::Runtime)

Important Notes

• Library Name Consistency: The chimod_lib_name must exactly match what the CMake system generates
• Admin Module: If you customize the namespace, you may also want to rebuild the admin module with your custom namespace
• Backward Compatibility: Changing namespace breaks compatibility with existing deployments using default namespace
• Documentation: Update any module-specific documentation to reflect the new namespace

Advanced Topics

Task Scheduling

Tasks can be scheduled with different priorities:

• kLowLatency: For time-critical operations
• kHighLatency: For batch processing

Automatic Routing Architecture

The framework handles all task routing automatically:

1. Client-Side Enqueuing:

   • Tasks are enqueued via IpcManager::Enqueue() from client code
   • Lane selection uses PID+TID hash for automatic distribution across lanes
   • Formula: lane_id = hash(PID, TID) % num_lanes

2. Worker-Lane Mapping (1:1 Direct Mapping):

   • Number of lanes automatically equals number of sched workers (default: 8)
   • Each worker assigned exactly one lane: worker i → lane i
   • No round-robin needed - perfect 1:1 correspondence
   • Lane headers track assigned worker ID

3. No Configuration Required:

   • Lane count automatically matches sched worker count from config
   • No separate task_queue_lanes configuration needed
   • Change worker count → lane count adjusts automatically

Example: With 8 sched workers (default):

• 8 lanes created automatically in external queue
• Worker 0 → Lane 0, Worker 1 → Lane 1, ..., Worker 7 → Lane 7
• Client tasks distributed via hash to lanes 0-7
• Each worker processes tasks from its dedicated lane

Error Handling

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;

TaskResume Custom(shared_ptr<CustomTask>& task) {
  CLIO_TASK_BODY_BEGIN
  try {
    // Operation logic
    task->SetReturnCode(0);
  } catch (const std::exception& e) {
    task->SetReturnCode(1);
    task->data_ = e.what();
  }
  // Framework handles task completion automatically
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

Debugging Tips

1. Check Shared Memory: Use ipcs -m to view segments
2. Verify Task State: Check task completion status
3. Monitor Queue Depth: Use GetProcessQueue() to inspect queues
4. Enable Debug Logging: Set CHI_DEBUG environment variable
5. Use GDB: Attach to runtime process for debugging

Common Issues and Solutions

Tasks Not Being Executed:

• Cause: Tasks not being routed to worker queues
• Solution: Verify task pool_query is set correctly and pool exists
• Debug: Add logging in task execution methods to verify they're being called

Queue Overflow or Deadlocks:

• Cause: Tasks being enqueued but not dequeued from lanes
• Solution: Verify lane creation in Create() method and proper task routing
• Debug: Check lane sizes with lane->Size() and lane->IsEmpty()

Memory Leaks in Shared Memory:

• Cause: Tasks not being properly cleaned up
• Solution: Ensure framework Del dispatcher is working correctly
• Debug: Monitor shared memory usage with ipcs -m

Performance Considerations

1. Minimize Allocations: Reuse buffers when possible
2. Batch Operations: Submit multiple tasks together
3. Use Appropriate Segments: Put large data in client_data_segment
4. Avoid Blocking: Use async operations when possible
5. Profile First: Measure before optimizing

Unit Testing

Unit testing for ChiMods is covered in the separate Module Test Guide. This guide provides comprehensive information on:

• Test environment setup and configuration
• Environment variables and module discovery
• Test framework integration patterns
• Complete test examples with fixtures
• CMake integration and build setup
• Best practices for Module testing

The test guide demonstrates how to test both runtime and client components in the same process, enabling comprehensive integration testing without complex multi-process coordination.

Quick Reference Checklist

When creating a new CLIO Runtime module, ensure you have:

Task Definition Checklist (_tasks.h)

• Tasks inherit from clio::run::Task or use GetOrCreatePoolTask template (recommended for non-admin modules)
• Use GetOrCreatePoolTask: For non-admin modules instead of BaseCreateTask directly
• Use BaseCreateTask with IS_ADMIN=true: Only for admin module
• SHM default constructor (if custom task)
• Emplace constructor with all required parameters (if custom task)
• Uses serializable types (std::string, std::vector, etc.)
• Method constant assigned in constructor (e.g., method_ = Method::kCreate;)
• No static casting: Use Method namespace constants directly
• Include auto-generated methods file for Method constants

Runtime Container Checklist (_runtime.h/cc)

• Inherits from clio::run::Container
• Init() method overridden - calls base class Init() then initializes client for this Module
• Create() method does NOT call clio::run::Container::Init() (container is already initialized before Create is called)
• All task handlers take clio::run::shared_ptr<TaskType>& and return TaskResume
• NO custom Del methods needed - tasks are shared_ptr handles (RAII cleanup)
• Uses CLIO_TASK_CC(ClassName) macro for entry points
• Routing is automatic - tasks are routed through external queue lanes mapped to workers (1:1 worker-to-lane mapping)

Client API Checklist (_client.h/cc)

• Inherits from clio::run::ContainerClient
• Uses CLIO_IPC->NewTask<TaskType>() for allocation
• Uses CLIO_IPC->Send() for task submission (returns Future)
• Async-only API: All methods return clio::run::Future<TaskType>
• CRITICAL: AsyncCreate passes this pointer for PostWait callback to update pool_id_

Build System Checklist

• CMakeLists.txt creates both client and runtime libraries
• clio_mod.yaml defines module metadata
• Auto-generated methods file: autogen/MOD_NAME_methods.h with Method namespace
• Include clio_runtime.h: In methods file for GLOBAL_CROSS_CONST macro
• GLOBAL_CROSS_CONST constants: Use namespace constants, not enum class
• Proper install targets configured
• Links against the clio::run::cxx library

Common Pitfalls to Avoid

• ❌ CRITICAL: Not updating pool_id_ in Create methods (leads to incorrect pool ID for subsequent operations)
• ❌ Using raw pointers instead of shared_ptr<TaskType>& in runtime methods
• ❌ Calling clio::run::Container::Init() in Create method (container is already initialized by framework before Create is called)
• ❌ Not overriding Init() method (required to initialize the client member)
• ❌ Using non-serializable types in task data members
• ❌ Implementing custom Del methods (tasks are shared_ptr handles, RAII cleanup)
• ❌ Writing complex extern "C" blocks (use CLIO_TASK_CC macro instead)
• ❌ Using static_cast with Method values (use Method::kName directly)
• ❌ Attempting to manually manage task routing (framework handles automatically)
• ❌ Missing clio_runtime.h include in methods file (GLOBAL_CROSS_CONST won't work)
• ❌ Using enum class for methods (use namespace with GLOBAL_CROSS_CONST instead)
• ❌ Using BaseCreateTask directly for non-admin modules (use GetOrCreatePoolTask instead)
• ❌ Forgetting GetOrCreatePoolTask template for container creation (reduces boilerplate)

Pool Name Requirements

CRITICAL: All Module Create functions MUST require a user-provided pool_name parameter. Never auto-generate pool names using pool_id_ during Create operations.

Why Pool Names Are Required

1. pool_id_ Not Available: pool_id_ is not set until after Create completes
2. User Intent: Users should explicitly name their pools for better organization
3. Uniqueness: Users can ensure uniqueness better than auto-generation
4. Debugging: Named pools are easier to identify during debugging

Pool Naming Guidelines

• Descriptive Names: Use names that identify purpose or content
• File-based Devices: For BDev file devices, pool_name serves as the file path
• RAM-based Devices: For BDev RAM devices, pool_name should be unique identifier
• Unique Identifiers: Consider timestamp + PID combinations when needed

Correct Pool Naming Usage

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
#include <string>
void example() {
  using namespace clio::run;
  MOD_NAME::Client client;
  auto pool_query = PoolQuery::Dynamic();
  u64 timestamp = 12345;
  std::string user_identifier = "u1";

  // File-based device - pool_name is the file path
  std::string file_path = "/path/to/device.dat";
  const PoolId bdev_pool_id(7000, 0);
  client.AsyncCreate(pool_query, file_path, bdev_pool_id).Wait();

  // RAM-based device - pool_name is unique identifier
  std::string ram_name = "my_ram_device_" + std::to_string(timestamp);
  const PoolId ram_pool_id(7001, 0);
  client.AsyncCreate(pool_query, ram_name, ram_pool_id).Wait();

  // Other ChiMods - pool_name is descriptive identifier
  std::string mod_name = "my_container_" + user_identifier;
  const PoolId mod_pool_id(7002, 0);
  client.AsyncCreate(pool_query, mod_name, mod_pool_id).Wait();
}

Incorrect Pool Naming Usage

// WRONG: Using pool_id_ before it's set (will be 0 or garbage)
std::string bad_name = "pool_" + std::to_string(pool_id_.ToU64());

// WRONG: Using empty strings
client.AsyncCreate(pool_query, "", pool_id);

// WRONG: Auto-generating inside Create function
// Create functions should not auto-generate names
void Create(pool_query) {
    std::string auto_name = "pool_" + generate_id();  // Wrong approach
}

Client Interface Pattern (Async-Only)

All Module clients should follow the async-only interface pattern:

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_tasks.h>
using namespace clio::run;
using namespace clio::run::MOD_NAME;  // CreateTask, CustomTask, CreateParams

class Client : public clio::run::ContainerClient {
 public:
  // Async-only Create with required pool_name - returns Future
  Future<CreateTask> AsyncCreate(
      const PoolQuery& pool_query,
      const std::string& pool_name,
      const PoolId& custom_pool_id /* user-provided name */) {
    auto* ipc_manager = CLIO_CPU_IPC;
    // Use pool_name directly, never generate internally
    auto task = ipc_manager->NewTask<CreateTask>(
        CreateTaskId(),
        kAdminPoolId,  // Always use admin pool
        pool_query,
        CreateParams::chimod_lib_name,  // Never hardcode
        pool_name,          // User-provided name
        custom_pool_id,     // Target pool ID
        this);              // Client pointer for PostWait callback
    return ipc_manager->Send(task);
  }

  // Example of async-only operation pattern
  Future<CustomTask> AsyncCustom(
      const PoolQuery& pool_query,
      const std::string& input_data,
      u32 operation_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager->NewTask<CustomTask>(
        CreateTaskId(),
        pool_id_,           // Use client's pool_id_ for non-Create operations
        pool_query,
        input_data,
        operation_id);
    return ipc_manager->Send(task);
  }
};

Usage Pattern (Caller Side):

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/MOD_NAME/MOD_NAME_client.h>
void example() {
  using namespace clio::run;
  // Initialize and create
  MOD_NAME::Client client;
  const PoolId pool_id(7000, 0);

  auto create_task = client.AsyncCreate(PoolQuery::Dynamic(), "my_pool", pool_id);
  create_task.Wait();

  if (create_task->GetReturnCode() != 0) {
    // Handle error
    return;
  }

  // Perform operations
  auto op_task = client.AsyncCustom(PoolQuery::Local(), "data", 1);
  op_task.Wait();

  // Access results
  auto result = op_task->data_.str();
  (void)result;
}

BDev-Specific Requirements

• Single Interface: Use only one AsyncCreate() method (no multiple overloads)
• File Devices: pool_name parameter serves as the file path
• RAM Devices: pool_name parameter serves as unique identifier
• Method Signature: AsyncCreate(pool_query, pool_name, custom_pool_id, bdev_type, total_size=0, io_depth=32, alignment=4096)

Compose Configuration Feature

The compose feature allows automatic pool creation from YAML configuration files. This enables declarative infrastructure setup where all required pools can be defined in configuration and created during runtime initialization or via utility script.

CreateParams LoadConfig Requirement

CRITICAL: All Module CreateParams structures MUST implement a LoadConfig() method to support compose feature.

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/config_manager.h>
#include <clio_ctp/util/config_parse.h>
#include <yaml-cpp/yaml.h>

struct CreateParams {
  std::string parameter_name_;
  clio::run::u64 total_size_ = 0;

  /**
   * Load configuration from PoolConfig (for compose mode)
   * Required for compose feature support
   * @param pool_config Pool configuration from compose section
   */
  void LoadConfig(const clio::run::PoolConfig& pool_config) {
    // Parse YAML config string
    YAML::Node config = YAML::Load(pool_config.config_);

    // Load module-specific parameters from YAML
    if (config["parameter_name"]) {
      parameter_name_ = config["parameter_name"].as<std::string>();
    }

    // Parse size strings (e.g., "2GB", "512MB")
    if (config["capacity"]) {
      std::string capacity_str = config["capacity"].as<std::string>();
      total_size_ = ctp::ConfigParse::ParseSize(capacity_str);
    }
  }
};

Compose Configuration Format

compose:
- mod_name: clio_bdev        # Module library name
  pool_name: ram://test          # Pool name (or file path for BDev)
  pool_query: dynamic            # Either "dynamic" or "local"
  pool_id: 200.0                 # Pool ID in "major.minor" format
  capacity: 2GB                  # Module-specific parameters
  bdev_type: ram                 # Additional parameters as needed
  io_depth: 32
  alignment: 4096

- mod_name: clio_another_mod
  pool_name: my_pool
  pool_query: local
  pool_id: 201.0
  custom_param: value

Usage Modes

1. Automatic During Runtime Init: Pools are automatically created when runtime initializes if compose section is present in configuration:

export CLIO_SERVER_CONF=/path/to/config_with_compose.yaml
clio_run start

2. Manual via clio_run compose Utility: Create pools using compose configuration against running runtime:

clio_run compose /path/to/compose_config.yaml

Implementation Checklist

When adding compose support to a Module:

• Add LoadConfig(const clio::run::PoolConfig& pool_config) method to CreateParams
• Parse all module-specific parameters from YAML config
• Handle optional parameters with defaults
• Use ctp::ConfigParse::ParseSize() for size strings
• Include <yaml-cpp/yaml.h> and <clio_runtime/config_manager.h> in tasks header
• Test with compose configuration before release

Example Admin Module LoadConfig

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/config_manager.h>

struct AdminCreateParams {
  void LoadConfig(const clio::run::PoolConfig& pool_config) {
    // Admin doesn't have additional configuration fields
    // YAML config parsing would go here for modules with config fields
    (void)pool_config;  // Suppress unused parameter warning
  }
};

Example BDev Module LoadConfig

#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/config_manager.h>
#include <clio_ctp/util/config_parse.h>
#include <yaml-cpp/yaml.h>

struct BDevCreateParams {
  enum class BdevType { kFile, kRam };
  BdevType bdev_type_ = BdevType::kRam;
  clio::run::u64 total_size_ = 0;
  clio::run::u32 io_depth_ = 32;
  clio::run::u32 alignment_ = 4096;

  void LoadConfig(const clio::run::PoolConfig& pool_config) {
    YAML::Node config = YAML::Load(pool_config.config_);

    // Load BDev type
    if (config["bdev_type"]) {
      std::string type_str = config["bdev_type"].as<std::string>();
      if (type_str == "file") {
        bdev_type_ = BdevType::kFile;
      } else if (type_str == "ram") {
        bdev_type_ = BdevType::kRam;
      }
    }

    // Load capacity (parse size strings)
    if (config["capacity"]) {
      std::string capacity_str = config["capacity"].as<std::string>();
      total_size_ = ctp::ConfigParse::ParseSize(capacity_str);
    }

    // Load optional parameters
    if (config["io_depth"]) {
      io_depth_ = config["io_depth"].as<clio::run::u32>();
    }
    if (config["alignment"]) {
      alignment_ = config["alignment"].as<clio::run::u32>();
    }
  }
};