Skip to main content

Chimaera Module Development Guide

Linking​

find_package(iowarp-core CONFIG)

Table of Contents​

  1. Overview
  2. Architecture
  3. Coding Style
  4. Module Structure
  5. Configuration and Code Generation
  6. Task Development
  7. Synchronization Primitives
  8. Pool Query and Task Routing
  9. Client-Server Communication
  10. Memory Management
  11. Build System Integration
  12. External ChiMod Development
  13. Example Module

Overview​

Chimaera modules (ChiMods) are dynamically loadable components that extend the runtime with new functionality. Each module consists of:

  • Client library: Minimal code for task submission from user processes
  • Runtime library: Server-side execution logic
  • Task definitions: Shared structures for client-server communication
  • Configuration: YAML metadata describing the module

Header Organization: All ChiMod headers are organized under the namespace directory structure (include/[namespace]/[module_name]/) to provide clear namespace separation and prevent header conflicts.

Architecture​

Core Principles​

  1. Client-Server Separation: Clients only submit tasks; runtime handles all logic
  2. Shared Memory Communication: Tasks are allocated in shared memory segments
  3. Task-Based Processing: All operations are expressed as tasks with methods
  4. Zero-Copy Design: Data stays in shared memory; only pointers are passed

Key Components​

ChiMod/
├── include/
│   └── [namespace]/
│       └── MOD_NAME/
│           ├── MOD_NAME_client.h     # Client API
│           ├── MOD_NAME_runtime.h    # Runtime container
│           ├── MOD_NAME_tasks.h      # Task definitions
│           └── autogen/
│               └── MOD_NAME_methods.h    # Method constants
├── src/
│   ├── MOD_NAME_client.cc        # Client implementation
│   ├── MOD_NAME_runtime.cc       # Runtime implementation
│   └── autogen/
│       └── MOD_NAME_lib_exec.cc  # Auto-generated virtual method implementations
├── chimaera_mod.yaml              # Module configuration
└── CMakeLists.txt                 # Build configuration

Include Directory Structure:

  • All ChiMod headers are organized under the namespace directory
  • Structure: include/[namespace]/[module_name]/
  • Example: Admin headers are in include/[namespace]/admin/ (where [namespace] is the namespace from chimaera_repo.yaml)
  • Headers follow naming pattern: [module_name]_[type].h
  • Auto-generated headers are in the autogen/ subdirectory
  • Note: The namespace comes from chimaera_repo.yaml and the chimod directory name doesn't need to match the namespace

Coding Style​

General Guidelines​

  1. Namespace: All module code under chimaera::MOD_NAME

  2. Naming Conventions:

    • Classes: PascalCase (e.g., CustomTask)
    • Methods: PascalCase for public, camelCase for private
    • Variables: snake_case_ with trailing underscore for members
    • Constants: kConstantName
    • Enums: kEnumValue

  3. Header Guards: Use #ifndef MOD_NAME_COMPONENT_H_

  4. Includes: System headers first, then library headers, then local headers

  5. Comments: Use Doxygen-style comments for public APIs

Code Formatting​

namespace chimaera::MOD_NAME {

/**
 * Brief description
 * 
 * Detailed description if needed
 * @param param_name Parameter description
 * @return Return value description
 */
class ExampleClass {
 public:
  // Public methods
  void PublicMethod();
  
 private:
  // Private members with trailing underscore
  u32 member_variable_;
};

}  // namespace chimaera::MOD_NAME

Module Structure​

Task Definition (MOD_NAME_tasks.h)​

Task definition patterns:

  1. CreateParams Structure: Define configuration parameters for container creation
    • CreateParams use cereal serialization and do NOT require allocator-based constructors
    • Only need default constructor and parameter-based constructors
    • Allocator is NOT passed to CreateParams - it's handled internally by BaseCreateTask
  2. CreateTask Template: Use GetOrCreatePoolTask template for container creation (non-admin modules)
  3. Custom Tasks: Define custom tasks with SHM/Emplace constructors and HSHM data members
#ifndef MOD_NAME_TASKS_H_
#define MOD_NAME_TASKS_H_

#include <chimaera/chimaera.h>
#include <[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h>
// Include admin tasks for GetOrCreatePoolTask
#include <[namespace]/admin/admin_tasks.h>

namespace chimaera::MOD_NAME {

/**
 * CreateParams for MOD_NAME chimod
 * Contains configuration parameters for MOD_NAME container creation
 */
struct CreateParams {
  // MOD_NAME-specific parameters
  std::string config_data_;
  chi::u32 worker_count_;

  // Required: chimod library name for module manager
  static constexpr const char* chimod_lib_name = "chimaera_MOD_NAME";

  // Default constructor
  CreateParams() : worker_count_(1) {}

  // Constructor with parameters
  CreateParams(const std::string& config_data = "",
               chi::u32 worker_count = 1)
      : config_data_(config_data), worker_count_(worker_count) {}

  // Serialization support for cereal
  template<class Archive>
  void serialize(Archive& ar) {
    ar(config_data_, worker_count_);
  }

  /**
   * Load configuration from PoolConfig (for compose mode)
   * Required for compose feature support
   * @param pool_config Pool configuration from compose section
   */
  void LoadConfig(const chi::PoolConfig& pool_config) {
    // Parse YAML config string
    YAML::Node config = YAML::Load(pool_config.config_);

    // Load module-specific parameters from YAML
    if (config["config_data"]) {
      config_data_ = config["config_data"].as<std::string>();
    }
    if (config["worker_count"]) {
      worker_count_ = config["worker_count"].as<chi::u32>();
    }
  }
};

/**
 * CreateTask - Initialize the MOD_NAME container
 * Type alias for GetOrCreatePoolTask with CreateParams (uses kGetOrCreatePool method)
 * Non-admin modules should use GetOrCreatePoolTask instead of BaseCreateTask
 */
using CreateTask = chimaera::admin::GetOrCreatePoolTask<CreateParams>;

/**
 * Custom operation task
 */
struct CustomTask : public chi::Task {
  // Task-specific data using HSHM macros
  INOUT chi::string data_;      // Input/output string
  IN chi::u32 operation_id_;     // Input parameter
  OUT chi::u32 result_code_;     // Output result

  // SHM constructor
  explicit CustomTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc)
      : chi::Task(alloc), 
        data_(alloc), 
        operation_id_(0), 
        result_code_(0) {}

  // Emplace constructor
  explicit CustomTask(
      const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc,
      const chi::TaskId &task_id,
      const chi::PoolId &pool_id,
      const chi::PoolQuery &pool_query,
      const std::string &data,
      chi::u32 operation_id)
      : chi::Task(alloc, task_id, pool_id, pool_query, 10),
        data_(alloc, data),
        operation_id_(operation_id),
        result_code_(0) {
    task_id_ = task_id;
    pool_id_ = pool_id;
    method_ = Method::kCustom;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }
};

}  // namespace chimaera::MOD_NAME

#endif  // MOD_NAME_TASKS_H_

Client Implementation (MOD_NAME_client.h/cc)​

The client provides a simple API for task submission:

#ifndef MOD_NAME_CLIENT_H_
#define MOD_NAME_CLIENT_H_

#include <chimaera/chimaera.h>
#include <[namespace]/MOD_NAME/MOD_NAME_tasks.h>

namespace chimaera::MOD_NAME {

class Client : public chi::ContainerClient {
 public:
  Client() = default;
  explicit Client(const chi::PoolId& pool_id) { Init(pool_id); }

  /**
   * Synchronous operation - waits for completion
   */
  void Create(const hipc::MemContext& mctx, 
              const chi::PoolQuery& pool_query,
              const CreateParams& params = CreateParams()) {
    auto task = AsyncCreate(mctx, pool_query, params);
    task->Wait();
    
    // CRITICAL: Update client pool_id_ with the actual pool ID from the task
    pool_id_ = task->new_pool_id_;
    
    CHI_IPC->DelTask(task);
  }

  /**
   * Asynchronous operation - returns immediately
   */
  hipc::FullPtr<CreateTask> AsyncCreate(
      const hipc::MemContext& mctx,
      const chi::PoolQuery& pool_query,
      const CreateParams& params = CreateParams()) {
    auto* ipc_manager = CHI_IPC;
    
    // CRITICAL: CreateTask MUST use admin pool for GetOrCreatePool processing
    auto task = ipc_manager->NewTask<CreateTask>(
        chi::CreateTaskId(),
        chi::kAdminPoolId,  // Always use admin pool for CreateTask
        pool_query,
        CreateParams::chimod_lib_name,  // ChiMod name from CreateParams
        pool_name_,             // Pool name from base client
        params);                // CreateParams with configuration
    
    // Submit to runtime
    ipc_manager->Enqueue(task);
    return task;
  }
};

}  // namespace chimaera::MOD_NAME

#endif  // MOD_NAME_CLIENT_H_

ChiMod CreateTask Pool Assignment Requirements​

CRITICAL: All ChiMod clients implementing Create functions MUST use the explicit chi::kAdminPoolId variable when constructing CreateTask operations. You CANNOT use pool_id_ for CreateTask operations.

Why This is Required​

CreateTask operations are actually GetOrCreatePoolTask operations that must be processed by the admin ChiMod to create or find the target pool. The pool_id_ variable is not initialized until after the Create operation completes successfully.

Correct Usage​

// CORRECT: Always use chi::kAdminPoolId for CreateTask
auto task = ipc_manager->NewTask<CreateTask>(
    chi::CreateTaskId(),
    chi::kAdminPoolId,          // REQUIRED: Use admin pool for CreateTask
    pool_query,
    CreateParams::chimod_lib_name,
    pool_name,
    pool_id_,                   // Target pool ID to create
    params);

Incorrect Usage​

// WRONG: Never use pool_id_ for CreateTask operations
auto task = ipc_manager->NewTask<CreateTask>(
    chi::CreateTaskId(),
    pool_id_,                   // WRONG: pool_id_ is not initialized yet
    pool_query,
    CreateParams::chimod_lib_name,
    pool_name,
    pool_id_,
    params);

Key Points​

  1. Admin Pool Processing: CreateTask is a GetOrCreatePoolTask that must be handled by the admin pool
  2. Uninitialized Variable: pool_id_ is not set until after Create completes
  3. Universal Requirement: This applies to ALL ChiMod clients, including admin, bdev, and custom modules
  4. Create Responsibility: Create operations are responsible for allocating new pool IDs using the admin pool

ChiMod Name Requirements​

CRITICAL: All ChiMod clients MUST use CreateParams::chimod_lib_name instead of hardcoding module names.

Correct Usage​

// CORRECT: Use CreateParams::chimod_lib_name
auto task = ipc_manager->NewTask<CreateTask>(
    chi::CreateTaskId(),
    chi::kAdminPoolId,
    pool_query,
    CreateParams::chimod_lib_name,  // Dynamic reference to CreateParams
    pool_name,
    pool_id,
    params);

Incorrect Usage​

// WRONG: Never hardcode module names
auto task = ipc_manager->NewTask<CreateTask>(
    chi::CreateTaskId(),
    chi::kAdminPoolId,
    pool_query,
    "chimaera_MOD_NAME",  // Hardcoded name breaks flexibility
    pool_name,
    pool_id,
    params);

Why This is Required​

  1. Namespace Flexibility: Allows ChiMods to work with different namespace configurations
  2. Single Source of Truth: The module name is defined once in CreateParams
  3. External ChiMods: Essential for external ChiMods using custom namespaces
  4. Maintainability: Changes to module names only require updating CreateParams

Implementation Pattern​

All non-admin ChiMods using GetOrCreatePoolTask must follow this pattern:

// In AsyncCreate method
auto task = ipc_manager->NewTask<CreateTask>(
    chi::CreateTaskId(),
    chi::kAdminPoolId,                    // Always use admin pool
    pool_query,
    CreateParams::chimod_lib_name,        // REQUIRED: Use static member
    pool_name,                            // Pool identifier
    pool_id,                              // Target pool ID
    /* ...CreateParams arguments... */);

Note: The admin ChiMod uses BaseCreateTask directly and doesn't require the chimod name parameter.

Runtime Container (MOD_NAME_runtime.h/cc)​

The runtime container executes tasks server-side:

#ifndef MOD_NAME_RUNTIME_H_
#define MOD_NAME_RUNTIME_H_

#include <chimaera/chimaera.h>
#include <[namespace]/MOD_NAME/MOD_NAME_tasks.h>

namespace chimaera::MOD_NAME {

class Container : public chi::Container {
 public:
  Container() = default;
  ~Container() override = default;

  /**
   * Initialize container with pool information (REQUIRED)
   * This is called by the framework before Create is called
   */
  void Init(const chi::PoolId& pool_id, const std::string& pool_name) override {
    // Call base class initialization
    chi::Container::Init(pool_id, pool_name);

    // Initialize the client for this ChiMod
    client_ = Client(pool_id);
  }

  /**
   * Create the container (Method::kCreate)
   * This method creates queues and sets up container resources
   * NOTE: Container is already initialized via Init() before Create is called
   */
  void Create(hipc::FullPtr<CreateTask> task, chi::RunContext& ctx) {
    // Container is already initialized via Init() before Create is called
    // Do NOT call Init() here

    // Additional container-specific initialization logic here
    std::cout << "Container created and initialized for pool: " << pool_name_
              << " (ID: " << pool_id_ << ")" << std::endl;
  }

  /**
   * Custom operation (Method::kCustom)
   */
  void Custom(hipc::FullPtr<CustomTask> task, chi::RunContext& ctx) {
    // Process the operation
    std::string result = processData(task->data_.str(),
                                    task->operation_id_);
    task->data_ = hipc::string(main_allocator_, result);
    task->result_code_ = 0;
    // Task completion is handled by the framework
  }

 private:
  std::string processData(const std::string& input, u32 op_id) {
    // Business logic here
    return input + "_processed";
  }
};

}  // namespace chimaera::MOD_NAME

// Define ChiMod entry points using CHI_TASK_CC macro
CHI_TASK_CC(chimaera::MOD_NAME::Container)

#endif  // MOD_NAME_RUNTIME_H_

Execution Modes and Dynamic Scheduling​

The Chimaera runtime supports two execution modes through the ExecMode enum in RunContext, enabling sophisticated task routing patterns:

ExecMode Overview​

/**
 * Execution mode for task processing
 */
enum class ExecMode : u32 {
  kExec = 0,              /**< Normal task execution (default) */
  kDynamicSchedule = 1    /**< Dynamic scheduling - route after execution */
};

The execution mode is accessible through the RunContext parameter passed to all task methods:

void YourMethod(hipc::FullPtr<YourTask> task, chi::RunContext& rctx) {
  // Check execution mode
  if (rctx.exec_mode == chi::ExecMode::kDynamicSchedule) {
    // Dynamic scheduling logic - modify task routing
    task->pool_query_ = chi::PoolQuery::Broadcast();
    return;  // Return early - task will be re-routed
  }

  // Normal execution logic (kExec mode)
  // ... perform actual work ...
}

kExec Mode (Default)​

Purpose: Normal task execution mode where tasks are processed completely and then marked as finished.

Behavior:

  • Tasks execute their full logic
  • Results are written to task output parameters
  • Worker calls EndTask() after execution completes
  • Task is marked as completed and cleaned up

When to use: The default mode for all standard task processing.

kDynamicSchedule Mode​

Purpose: Two-phase execution where the first execution determines routing, then the task is re-routed and executed again.

Behavior:

  1. Worker sets exec_mode = kDynamicSchedule before first execution
  2. Task method examines state and modifies task->pool_query_ for routing
  3. Task returns early without performing full execution
  4. Worker calls RerouteDynamicTask() instead of EndTask()
  5. Task is re-routed using the updated pool_query_
  6. Task executes again in normal kExec mode with the new routing

When to use:

  • Tasks that need runtime-dependent routing decisions
  • Cache optimization patterns (check local, then broadcast if not found)
  • Conditional distributed execution based on state

Example: GetOrCreatePool with Dynamic Scheduling​

The admin ChiMod's GetOrCreatePool method demonstrates the canonical dynamic scheduling pattern:

void Runtime::GetOrCreatePool(
    hipc::FullPtr<chimaera::admin::GetOrCreatePoolTask<chimaera::admin::CreateParams>> task,
    chi::RunContext &rctx) {

  auto *pool_manager = CHI_POOL_MANAGER;
  std::string pool_name = task->pool_name_.str();

  // PHASE 1: Dynamic scheduling - determine routing
  if (rctx.exec_mode == chi::ExecMode::kDynamicSchedule) {
    // Check if pool exists locally first
    chi::PoolId existing_pool_id = pool_manager->FindPoolByName(pool_name);

    if (!existing_pool_id.IsNull()) {
      // Pool exists locally - route to local execution only
      HILOG(kDebug, "Admin: Pool '{}' found locally (ID: {}), using Local query",
            pool_name, existing_pool_id);
      task->pool_query_ = chi::PoolQuery::Local();
    } else {
      // Pool doesn't exist - broadcast creation to all nodes
      HILOG(kDebug, "Admin: Pool '{}' not found locally, broadcasting creation",
            pool_name);
      task->pool_query_ = chi::PoolQuery::Broadcast();
    }
    return;  // Return early - worker will re-route task
  }

  // PHASE 2: Normal execution - actually create/get the pool
  HILOG(kDebug, "Admin: Executing GetOrCreatePool task - ChiMod: {}, Pool: {}",
        task->chimod_name_.str(), pool_name);

  task->return_code_ = 0;
  task->error_message_ = "";

  try {
    if (!pool_manager->CreatePool(task.Cast<chi::Task>(), &rctx)) {
      task->return_code_ = 2;
      task->error_message_ = "Failed to create or get pool via PoolManager";
      return;
    }

    task->return_code_ = 0;
    pools_created_++;

    HILOG(kDebug, "Admin: Pool operation completed successfully - ID: {}, Name: {}",
          task->new_pool_id_, pool_name);

  } catch (const std::exception &e) {
    task->return_code_ = 99;
    task->error_message_ = hipc::string(
        task->GetCtxAllocator(),
        std::string("Exception during pool creation: ") + e.what());
    HELOG(kError, "Admin: Pool creation failed with exception: {}", e.what());
  }
}

Using Dynamic() PoolQuery​

The PoolQuery::Dynamic() factory method triggers dynamic scheduling:

// Client code - request dynamic routing
auto pool_query = chi::PoolQuery::Dynamic();
client.Create(mctx, pool_query, "my_pool_name", pool_id);

What happens internally:

  1. Worker recognizes Dynamic() pool query
  2. Sets rctx.exec_mode = ExecMode::kDynamicSchedule
  3. Routes task to local node first
  4. Task method checks cache and updates pool_query_
  5. Worker re-routes with updated query
  6. Task executes again in normal mode with correct routing

Benefits of Dynamic Scheduling​

Performance Optimization:

  • Avoids redundant operations (e.g., pool creation when pool already exists)
  • Reduces network overhead by checking local state first
  • Enables intelligent routing based on runtime conditions

Cache Optimization Pattern:

// Check local cache first
if (rctx.exec_mode == chi::ExecMode::kDynamicSchedule) {
  if (LocalCacheHas(resource_id)) {
    task->pool_query_ = chi::PoolQuery::Local();  // Found locally
  } else {
    task->pool_query_ = chi::PoolQuery::Broadcast();  // Need to fetch
  }
  return;
}

// Normal execution with optimized routing
auto resource = GetResource(resource_id);

State-Dependent Routing:

// Route based on runtime conditions
if (rctx.exec_mode == chi::ExecMode::kDynamicSchedule) {
  if (ShouldExecuteDistributed(task)) {
    task->pool_query_ = chi::PoolQuery::Broadcast();
  } else {
    task->pool_query_ = chi::PoolQuery::Local();
  }
  return;
}

Implementation Guidelines​

DO:

  • ✅ Check rctx.exec_mode at the start of your method
  • ✅ Return early after modifying pool_query_ in dynamic mode
  • ✅ Keep dynamic scheduling logic lightweight (fast checks only)
  • ✅ Use dynamic scheduling for cache optimization patterns

DON'T:

  • ❌ Perform expensive operations in dynamic scheduling mode
  • ❌ Modify task output parameters in dynamic scheduling mode
  • ❌ Call Wait() or spawn subtasks in dynamic scheduling mode
  • ❌ Use dynamic scheduling for simple operations that don't need routing optimization

Worker Implementation Details​

The worker automatically handles dynamic scheduling:

// Worker::ExecTask() logic
if (run_ctx->exec_mode == ExecMode::kDynamicSchedule) {
  // After task returns, call RerouteDynamicTask instead of EndTask
  RerouteDynamicTask(task_ptr, run_ctx);
  return;
}

// Normal mode - end task after execution
EndTask(task_ptr, run_ctx);

The RerouteDynamicTask() method:

  1. Resets task flags (TASK_STARTED, TASK_ROUTED)
  2. Re-routes task using updated pool_query_
  3. Sets exec_mode = kExec for next execution
  4. Schedules task for execution again

Configuration and Code Generation​

Overview​

Chimaera uses a two-level configuration system with automated code generation:

  1. chimaera_repo.yaml: Repository-wide configuration (namespace, version, etc.)
  2. chimaera_mod.yaml: Module-specific configuration (method IDs, metadata)
  3. chi_refresh_repo: Utility script that generates autogen files from YAML configurations

chimaera_repo.yaml​

Located at chimods/chimaera_repo.yaml, this file defines repository-wide settings:

# Repository Configuration
namespace: chimaera        # MUST match namespace in all chimaera_mod.yaml files
version: 1.0.0
description: "Chimaera Runtime ChiMod Repository"

# Module discovery - directories to scan for ChiMods
modules:
  - MOD_NAME
  - admin  
  - bdev

Key Requirements:

  • The namespace field MUST be identical in both chimaera_repo.yaml and all chimaera_mod.yaml files
  • Used by build system for CMake package generation and installation paths
  • Determines export target names: ${namespace}::${module}_runtime, ${namespace}::${module}_client

chimaera_mod.yaml​

Each ChiMod must have its own configuration file specifying methods and metadata:

# MOD_NAME ChiMod Configuration
module_name: MOD_NAME
namespace: chimaera        # MUST match chimaera_repo.yaml namespace
version: 1.0.0

# Inherited Methods (fixed IDs)
kCreate: 0        # Container creation (required)
kDestroy: 1       # Container destruction (required)
kNodeFailure: -1  # Not implemented (-1 means disabled)
kRecover: -1      # Not implemented 
kMigrate: -1      # Not implemented
kUpgrade: -1      # Not implemented

# Custom Methods (start from 10, use sequential IDs)
kCustom: 10       # Custom operation method
kCoMutexTest: 20  # CoMutex synchronization testing method
kCoRwLockTest: 21 # CoRwLock reader-writer synchronization testing method

Method ID Assignment Rules:

  • 0-9: Reserved for system methods (kCreate=0, kDestroy=1, etc.)
  • 10+: Custom methods (assign sequential IDs starting from 10)
  • Disabled methods: Use -1 to disable inherited methods not implemented
  • Consistency: Once assigned, never change method IDs (breaks compatibility)

chi_refresh_repo Utility​

The chi_refresh_repo utility automatically generates autogen files from YAML configurations.

Usage​

# From project root, regenerate all autogen files
./build/bin/chi_refresh_repo chimods

# The utility will:
# 1. Read chimaera_repo.yaml for global settings
# 2. Scan each module's chimaera_mod.yaml 
# 3. Generate MOD_NAME_methods.h with method constants
# 4. Generate MOD_NAME_lib_exec.cc with virtual method dispatch

Generated Files​

For each ChiMod, the utility generates:

  1. include/[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h:

    namespace chimaera::MOD_NAME {
    namespace Method {
    GLOBAL_CONST chi::u32 kCreate = 0;
    GLOBAL_CONST chi::u32 kDestroy = 1;
    GLOBAL_CONST chi::u32 kCustom = 10;
    GLOBAL_CONST chi::u32 kCoMutexTest = 20;
    }  // namespace Method
    }  // namespace chimaera::MOD_NAME

  2. src/autogen/MOD_NAME_lib_exec.cc:

    • Virtual method dispatch (Runtime::Run, etc.)
    • Task serialization support (SaveIn/Out, LoadIn/Out)
    • Memory management (Del, NewCopy, Aggregate)

When to Run chi_refresh_repo​

ALWAYS run chi_refresh_repo when:

  • Adding new methods to chimaera_mod.yaml
  • Changing method IDs or names
  • Adding new ChiMods to the repository
  • Modifying namespace or version information

Important Notes​

  • Never manually edit autogen files - they are overwritten by chi_refresh_repo
  • Run chi_refresh_repo before building after YAML changes
  • Commit autogen files to git so other developers don't need to regenerate
  • Method IDs are permanent - changing them breaks binary compatibility

Workflow Summary​

  1. Define methods in chimaera_mod.yaml with sequential IDs
  2. Implement corresponding methods in MOD_NAME_runtime.h/cc
  3. Run ./build/bin/chi_refresh_repo chimods to generate autogen files
  4. Build project with make - autogen files provide the dispatch logic
  5. Autogen files handle virtual method routing, serialization, and memory management

This automated approach ensures consistency across all ChiMods and reduces boilerplate code maintenance.

Task Development​

Task Requirements​

  1. Inherit from chi::Task: All tasks must inherit the base Task class
  2. Two Constructors: SHM and emplace constructors are mandatory
  3. Serializable Types: Use HSHM types (chi::string, chi::vector, etc.) for member variables
  4. Method Assignment: Set the method_ field to identify the operation
  5. FullPtr Usage: All task method signatures use hipc::FullPtr<TaskType> instead of raw pointers
  6. Copy Method: Optional - implement for tasks that need to be replicated across nodes
  7. Aggregate Method: Optional - implement for tasks that need to combine results from replicas

Optional Task Methods: Copy and Aggregate​

For tasks that will be distributed across multiple nodes or need to combine results from multiple executions, you can optionally implement Copy() and Aggregate() methods.

Copy Method​

The Copy() method is used to create a deep copy of a task, typically when distributing work across multiple nodes. This is useful for:

  • Remote task execution via networking
  • Task replication for fault tolerance
  • Creating independent task replicas with separate data

Signature:

void Copy(const hipc::FullPtr<YourTask> &other);

Implementation Pattern:

struct WriteTask : public chi::Task {
  IN Block block_;
  IN hipc::Pointer data_;
  IN size_t length_;
  OUT chi::u64 bytes_written_;

  /**
   * Copy from another WriteTask (assumes this task is already constructed)
   * @param other Pointer to the source task to copy from
   */
  void Copy(const hipc::FullPtr<WriteTask> &other) {
    // Copy task-specific fields only
    // Base Task fields are copied automatically by NewCopy
    block_ = other->block_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_written_ = other->bytes_written_;
  }
};

Key Points:

  • DO NOT call chi::Task::Copy() - base fields are copied automatically by autogenerated NewCopy
  • Copy only task-specific fields from the source task
  • The destination task (this) is already constructed - don't call constructors
  • For pointer fields, decide if you need deep or shallow copy based on ownership
  • NewCopy in autogen code handles calling base Task::Copy before your Copy method

Aggregate Method​

The Aggregate() method combines results from multiple task replicas into a single result. This is commonly used for:

  • Combining results from distributed task execution
  • Merging partial results from parallel operations
  • Accumulating metrics from multiple nodes

Signature:

void Aggregate(const hipc::FullPtr<YourTask> &other);

Implementation Patterns:

Pattern 1: Last-Writer-Wins (Simple Override)

struct WriteTask : public chi::Task {
  IN Block block_;
  IN hipc::Pointer data_;
  OUT chi::u64 bytes_written_;

  /**
   * Aggregate results from another WriteTask
   * For write operations, we typically just copy the result from the completed replica
   */
  void Aggregate(const hipc::FullPtr<WriteTask> &other) {
    // Simply copy the result - last writer wins
    Copy(other);
  }
};

Pattern 2: Accumulation (Sum/Max/Min)

struct GetStatsTask : public chi::Task {
  OUT chi::u64 total_bytes_;
  OUT chi::u64 operation_count_;
  OUT chi::u64 max_latency_us_;

  /**
   * Aggregate statistics from multiple replicas
   * Accumulate totals and find maximum values
   */
  void Aggregate(const hipc::FullPtr<GetStatsTask> &other) {
    // Sum cumulative metrics
    total_bytes_ += other->total_bytes_;
    operation_count_ += other->operation_count_;

    // Take maximum for latency
    max_latency_us_ = std::max(max_latency_us_, other->max_latency_us_);
  }
};

Pattern 3: List/Vector Merging

struct AllocateBlocksTask : public chi::Task {
  OUT chi::ipc::vector<Block> blocks_;

  /**
   * Aggregate block allocations from multiple replicas
   * Combine all allocated blocks into a single list
   */
  void Aggregate(const hipc::FullPtr<AllocateBlocksTask> &other) {
    // Append blocks from other task to this task's list
    blocks_.insert(blocks_.end(),
                   other->blocks_.begin(),
                   other->blocks_.end());
  }
};

Pattern 4: Custom Logic

struct QueryTask : public chi::Task {
  OUT chi::ipc::vector<Result> results_;
  OUT chi::u32 error_count_;

  /**
   * Aggregate query results with custom deduplication
   */
  void Aggregate(const hipc::FullPtr<QueryTask> &other) {
    // Merge results with deduplication
    for (const auto &result : other->results_) {
      if (!ContainsResult(results_, result)) {
        results_.push_back(result);
      }
    }

    // Accumulate error counts
    error_count_ += other->error_count_;
  }

private:
  bool ContainsResult(const chi::ipc::vector<Result> &vec, const Result &r) {
    // Custom deduplication logic
    return std::find(vec.begin(), vec.end(), r) != vec.end();
  }
};

When to Implement Copy and Aggregate​

Implement Copy when:

  • Your task will be sent to remote nodes for execution
  • Task data needs to be replicated for fault tolerance
  • You need independent copies with separate data ownership

Implement Aggregate when:

  • Your task returns results that can be combined (sums, lists, statistics)
  • You're using distributed execution patterns (e.g., map-reduce)
  • Multiple replicas produce partial results that need merging

Skip Copy and Aggregate when:

  • Tasks are only executed locally on a single node
  • Results don't need to be combined across executions
  • Tasks have no output parameters (side-effects only)
  • Default shallow copy behavior is sufficient

Copy/Aggregate Usage in Networking​

When tasks are sent across nodes using Send/Recv:

  1. Send Phase: The Copy() method creates a replica of the origin task

    hipc::FullPtr<Task> replica;
    container->NewCopy(task->method_, origin_task, replica, /* replica_flag */);
    // Internally calls task->Copy(origin_task)

  2. Recv Phase: The Aggregate() method combines replica results back into the origin

    container->Aggregate(task->method_, origin_task, replica);
    // Internally calls origin_task->Aggregate(replica)

  3. Autogeneration: The code generator creates dispatcher methods that call your Copy/Aggregate implementations:

    // In autogen/MOD_NAME_lib_exec.cc
    void NewCopy(Runtime* runtime, chi::u32 method,
                 hipc::FullPtr<chi::Task> orig_task,
                 hipc::FullPtr<chi::Task>& new_task,
                 bool deep_copy) {
      switch (method) {
        case Method::kWrite: {
          auto orig = orig_task.Cast<WriteTask>();
          new_task = CHI_IPC->NewTask<WriteTask>(...);
          new_task.Cast<WriteTask>()->Copy(orig);
          break;
        }
      }
    }

    void Aggregate(Runtime* runtime, chi::u32 method,
                   hipc::FullPtr<chi::Task> task,
                   const hipc::FullPtr<chi::Task> &replica) {
      switch (method) {
        case Method::kWrite: {
          task.Cast<WriteTask>()->Aggregate(replica.Cast<WriteTask>());
          break;
        }
      }
    }

Complete Example: ReadTask with Copy and Aggregate​

struct ReadTask : public chi::Task {
  IN Block block_;
  OUT hipc::Pointer data_;
  INOUT size_t length_;
  OUT chi::u64 bytes_read_;

  /** SHM constructor */
  explicit ReadTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc)
      : chi::Task(alloc), length_(0), bytes_read_(0) {}

  /** Emplace constructor */
  explicit ReadTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc,
                    const chi::TaskId &task_node,
                    const chi::PoolId &pool_id,
                    const chi::PoolQuery &pool_query,
                    const Block &block,
                    hipc::Pointer data,
                    size_t length)
      : chi::Task(alloc, task_node, pool_id, pool_query, 10),
        block_(block), data_(data), length_(length), bytes_read_(0) {
    task_id_ = task_node;
    pool_id_ = pool_id;
    method_ = Method::kRead;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }

  /**
   * Copy from another ReadTask
   * Used when creating replicas for remote execution
   */
  void Copy(const hipc::FullPtr<ReadTask> &other) {
    // Copy task-specific fields only
    // Base Task fields are copied automatically by NewCopy
    block_ = other->block_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_read_ = other->bytes_read_;
  }

  /**
   * Aggregate results from replica
   * For read operations, simply copy the data from the completed replica
   */
  void Aggregate(const hipc::FullPtr<ReadTask> &other) {
    // For reads, we just take the result from the replica
    Copy(other);
  }
};

Task Naming Conventions​

CRITICAL: All task names MUST follow consistent naming patterns to ensure proper code generation and maintenance.

Required Naming Pattern​

The naming convention enforces consistency across function names, task types, and method constants:

Function Name → Task Name → Method Constant
FunctionName()  → FunctionNameTask  → kFunctionName

Examples​

Correct Naming:

// Function: GetStats() and AsyncGetStats()
// Task: GetStatsTask  
// Method: kGetStats

// In bdev_client.h
PerfMetrics GetStats(const hipc::MemContext& mctx, chi::u64& remaining_size);
hipc::FullPtr<GetStatsTask> AsyncGetStats(const hipc::MemContext& mctx);

// In bdev_tasks.h  
struct GetStatsTask : public chi::Task {
  OUT PerfMetrics metrics_;
  OUT chi::u64 remaining_size_;
  // ... constructors and methods
};

// In chimaera_mod.yaml
kGetStats: 14    # Get performance statistics

// In bdev_runtime.h
void GetStats(hipc::FullPtr<GetStatsTask> task, chi::RunContext& ctx);

Incorrect Naming Examples:

// WRONG: Function and task names don't match
PerfMetrics GetStats(...);           // Function name
struct StatTask { ... };             // Task name doesn't match function

// WRONG: Method constant doesn't match function  
GLOBAL_CONST chi::u32 kStat = 14;    // Method doesn't match function name

// WRONG: Runtime method doesn't match function
void Stat(hipc::FullPtr<StatTask> task, ...);  // Runtime method doesn't match

Naming Rules​

  1. Function Names: Use descriptive verbs (e.g., GetStats, AllocateBlocks, WriteData)
  2. Task Names: Always append "Task" to the function name (e.g., GetStatsTask, AllocateBlocksTask)
  3. Method Constants: Prefix with "k" and match the function name exactly (e.g., kGetStats, kAllocateBlocks)
  4. Runtime Methods: Must match the function name exactly (e.g., GetStats())

Backward Compatibility​

When renaming tasks, provide backward compatibility aliases:

// In bdev_tasks.h - provide alias for old name
using StatTask = GetStatsTask;  // Backward compatibility

// In autogen/bdev_methods.h - provide constant alias
GLOBAL_CONST chi::u32 kGetStats = 14;
GLOBAL_CONST chi::u32 kStat = kGetStats;  // Backward compatibility

// In bdev_runtime.h - provide wrapper methods
void GetStats(hipc::FullPtr<GetStatsTask> task, chi::RunContext& ctx);  // Primary
void Stat(hipc::FullPtr<StatTask> task, chi::RunContext& ctx) {         // Wrapper
  GetStats(task, ctx);
}

Benefits of Consistent Naming​

  1. Code Generation: Automated tools can reliably generate method dispatch code
  2. Maintenance: Clear correlation between client functions and runtime implementations
  3. Documentation: Self-documenting code with predictable naming patterns
  4. Debugging: Easy to trace from client calls to runtime execution
  5. Testing: Consistent patterns make it easier to write comprehensive tests

Method System and Auto-Generated Files​

Method Definitions (autogen/MOD_NAME_methods.h)​

Method IDs are now defined as namespace constants instead of enum class values. This eliminates the need for static casting:

#ifndef MOD_NAME_AUTOGEN_METHODS_H_
#define MOD_NAME_AUTOGEN_METHODS_H_

#include <chimaera/chimaera.h>

namespace chimaera::MOD_NAME {

namespace Method {
  // Inherited methods
  GLOBAL_CONST chi::u32 kCreate = 0;
  GLOBAL_CONST chi::u32 kDestroy = 1;
  GLOBAL_CONST chi::u32 kNodeFailure = 2;
  GLOBAL_CONST chi::u32 kRecover = 3;
  GLOBAL_CONST chi::u32 kMigrate = 4;
  GLOBAL_CONST chi::u32 kUpgrade = 5;
  
  // Module-specific methods
  GLOBAL_CONST chi::u32 kCustom = 10;
}

} // namespace chimaera::MOD_NAME

#endif // MOD_NAME_AUTOGEN_METHODS_H_

Key Changes:

  • Namespace instead of enum class: Use Method::kMethodName directly
  • GLOBAL_CONST values: No more static casting required
  • Include chimaera.h: Required for GLOBAL_CONST macro
  • Direct assignment: method_ = Method::kCreate; (no casting)

BaseCreateTask Template System​

For modules that need container creation functionality, use the BaseCreateTask template instead of implementing custom CreateTask. However, there are different approaches depending on whether your module is the admin module or a regular ChiMod:

GetOrCreatePoolTask vs BaseCreateTask Usage​

For Non-Admin Modules (Recommended Pattern):

All non-admin ChiMods should use GetOrCreatePoolTask which is a specialized version of BaseCreateTask designed for external pool creation:

#include <[namespace]/admin/admin_tasks.h>  // Include admin templates

namespace chimaera::MOD_NAME {

/**
 * CreateParams for MOD_NAME container creation
 */
struct CreateParams {
  // Module-specific configuration
  std::string config_data_;
  chi::u32 worker_count_;
  
  // Required: chimod library name
  static constexpr const char* chimod_lib_name = "chimaera_MOD_NAME";

  // Constructors
  CreateParams() : worker_count_(1) {}

  CreateParams(const std::string& config_data = "",
               chi::u32 worker_count = 1)
      : config_data_(config_data), worker_count_(worker_count) {}

  // Cereal serialization
  template<class Archive>
  void serialize(Archive& ar) {
    ar(config_data_, worker_count_);
  }
};

/**
 * CreateTask - Non-admin modules should use GetOrCreatePoolTask
 * This uses Method::kGetOrCreatePool and is designed for external pool creation
 */
using CreateTask = chimaera::admin::GetOrCreatePoolTask<CreateParams>;

}  // namespace chimaera::MOD_NAME

For Admin Module Only:

The admin module itself uses BaseCreateTask directly with Method::kCreate:

namespace chimaera::admin {

/**
 * CreateTask - Admin uses BaseCreateTask with Method::kCreate and IS_ADMIN=true
 */
using CreateTask = BaseCreateTask<CreateParams, Method::kCreate, true>;

}  // namespace chimaera::admin

BaseCreateTask Template Parameters​

The BaseCreateTask template has three parameters with smart defaults:

template <typename CreateParamsT, 
          chi::u32 MethodId = Method::kGetOrCreatePool, 
          bool IS_ADMIN = false>
struct BaseCreateTask : public chi::Task

Template Parameters:

  1. CreateParamsT: Your module's parameter structure (required)
  2. MethodId: Method ID for the task (default: kGetOrCreatePool)
  3. IS_ADMIN: Whether this is an admin operation (default: false)

GetOrCreatePoolTask Template:

The GetOrCreatePoolTask template is a convenient alias that uses the optimal defaults for non-admin modules:

template<typename CreateParamsT>
using GetOrCreatePoolTask = BaseCreateTask<CreateParamsT, Method::kGetOrCreatePool, false>;

When to Use Each Pattern:

  • GetOrCreatePoolTask: For all non-admin ChiMods (recommended)
  • BaseCreateTask with Method::kCreate: Only for admin module internal operations
  • BaseCreateTask with Method::kGetOrCreatePool: Same as GetOrCreatePoolTask (not typically used directly)

BaseCreateTask Structure​

BaseCreateTask provides a unified structure for container creation and pool operations:

template <typename CreateParamsT, chi::u32 MethodId, bool IS_ADMIN>
struct BaseCreateTask : public chi::Task {
  // Pool operation parameters
  INOUT chi::string chimod_name_;     // ChiMod name for loading
  IN chi::string pool_name_;          // Target pool name
  INOUT chi::string chimod_params_;   // Serialized CreateParamsT
  INOUT chi::PoolId pool_id_;          // Input: requested ID, Output: actual ID
  
  // Results
  OUT chi::u32 result_code_;           // 0 = success, non-zero = error
  OUT chi::string error_message_;     // Error description if failed
  
  // Runtime flag set by template parameter
  volatile bool is_admin_;             // Set to IS_ADMIN template value
  
  // Serialization methods
  template<typename... Args>
  void SetParams(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc, Args &&...args);
  
  CreateParamsT GetParams(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc) const;
};

Key Features:

  • Single pool_id: Serves as both input (requested) and output (result)
  • Serialized parameters: chimod_params_ stores serialized CreateParamsT
  • Error checking: Use result_code_ != 0 to check for failures
  • Template-driven behavior: IS_ADMIN template parameter sets volatile variable
  • No static casting: Direct method assignment using namespace constants

Usage Examples​

Non-Admin ChiMod Container Creation (Recommended):

// Use GetOrCreatePoolTask for all non-admin modules
using CreateTask = chimaera::admin::GetOrCreatePoolTask<MyCreateParams>;

Admin Module Container Creation:

// Admin module uses BaseCreateTask with Method::kCreate and IS_ADMIN=true
using CreateTask = chimaera::admin::BaseCreateTask<AdminCreateParams, Method::kCreate, true>;

Alternative (Not Recommended):

// Direct BaseCreateTask usage - GetOrCreatePoolTask is cleaner
using CreateTask = chimaera::admin::BaseCreateTask<MyCreateParams, Method::kGetOrCreatePool, false>;

Migration from Custom CreateTask​

If you have existing custom CreateTask implementations, migrate to BaseCreateTask:

Before (Custom Implementation):

struct CreateTask : public chi::Task {
  // Custom constructor implementations
  explicit CreateTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc,
                      const chi::TaskId &task_id,
                      const chi::PoolId &pool_id,
                      const chi::PoolQuery &pool_query)
      : chi::Task(alloc, task_id, pool_id, pool_query, 0) {
    method_ = Method::kCreate;  // Static casting required
    // ... initialization code ...
  }
};

After (GetOrCreatePoolTask - Recommended for Non-Admin Modules):

// Create params structure
struct CreateParams {
  static constexpr const char* chimod_lib_name = "chimaera_mymodule";
  // ... other params ...
  template<class Archive> void serialize(Archive& ar) { /* ... */ }
};

// Simple type alias using GetOrCreatePoolTask - no custom implementation needed
using CreateTask = chimaera::admin::GetOrCreatePoolTask<CreateParams>;

Benefits of Migration:

  • No static casting: Direct use of Method::kCreate
  • Standardized structure: Consistent across all modules
  • Built-in serialization: SetParams/GetParams methods included
  • Error handling: Standardized result_code and error_message
  • Less boilerplate: No need to implement constructors manually

Data Annotations​

  • IN: Input-only parameters (read by runtime)
  • OUT: Output-only parameters (written by runtime)
  • INOUT: Bidirectional parameters

Task Lifecycle​

  1. Client allocates task in shared memory using ipc_manager->NewTask()
  2. Client enqueues task pointer to IPC queue
  3. Worker dequeues and executes task
  4. Framework calls ipc_manager->DelTask() to deallocate task from shared memory
  5. Task memory is properly reclaimed from the appropriate memory segment

Note: Individual DelTaskType methods are no longer required. The framework's autogenerated Del dispatcher automatically calls ipc_manager->DelTask() for proper shared memory deallocation.

Framework Del Implementation​

The autogenerated Del dispatcher handles task cleanup:

inline void Del(Runtime* runtime, chi::u32 method, hipc::FullPtr<chi::Task> task_ptr) {
  auto* ipc_manager = CHI_IPC;
  Method method_enum = static_cast<Method>(method);

  switch (method_enum) {
    case Method::kCreate: {
      ipc_manager->DelTask(task_ptr.Cast<CreateTask>());
      break;
    }
    case Method::kCustom: {
      ipc_manager->DelTask(task_ptr.Cast<CustomTask>());
      break;
    }
    default:
      ipc_manager->DelTask(task_ptr);
      break;
  }
}

This ensures proper shared memory deallocation without requiring module-specific cleanup code.

Synchronization Primitives​

Chimaera provides specialized cooperative synchronization primitives designed for the runtime's task-based architecture. These should be used instead of standard synchronization primitives like std::mutex, std::shared_mutex, or pthread_mutex when synchronizing access to module data structures.

Why Use Chimaera Synchronization Primitives?​

Critical: Always use CoMutex and CoRwLock for module synchronization:

  1. Cooperative Design: Compatible with Chimaera's fiber-based task execution
  2. TaskId Grouping: Tasks sharing the same TaskId can proceed together (bypassing locks)
  3. Deadlock Prevention: Designed to prevent deadlocks in the runtime environment
  4. Runtime Integration: Automatically integrate with CHI_CUR_WORKER and task context
  5. Performance: Optimized for the runtime's execution model

Do NOT use these standard synchronization primitives in module code:

  • ❌ std::mutex - Can cause fiber blocking issues
  • ❌ std::shared_mutex - Not compatible with task execution model
  • ❌ pthread_mutex_t - Can deadlock with runtime scheduling
  • ❌ std::condition_variable - Incompatible with cooperative scheduling

CoMutex: Cooperative Mutual Exclusion​

CoMutex provides mutual exclusion with TaskId grouping support. Tasks sharing the same TaskId can bypass the lock and execute concurrently.

Basic Usage​

#include <chimaera/comutex.h>

class Runtime : public chi::Container {
private:
  // Static member for shared synchronization across all container instances
  static chi::CoMutex shared_mutex_;
  
  // Instance member for per-container synchronization
  chi::CoMutex instance_mutex_;

public:
  void SomeTask(hipc::FullPtr<SomeTaskType> task, chi::RunContext& rctx) {
    // Manual lock/unlock
    shared_mutex_.Lock();
    // ... critical section ...
    shared_mutex_.Unlock();
    
    // OR use RAII scoped lock (recommended)
    chi::ScopedCoMutex lock(instance_mutex_);
    // ... critical section ...
    // Automatically unlocks when leaving scope
  }
};

// Static member definition (required)
chi::CoMutex Runtime::shared_mutex_;

Key Features​

  1. Automatic Task Context: Uses CHI_CUR_WORKER internally - no task parameters needed
  2. TaskId Grouping: Tasks with the same TaskId bypass the mutex
  3. RAII Support: ScopedCoMutex for automatic lock management
  4. Try-Lock Support: Non-blocking lock attempts

API Reference​

namespace chi {
  class CoMutex {
  public:
    // Blocking operations
    void Lock();                    // Block until lock acquired
    void Unlock();                  // Release the lock
    bool TryLock();                 // Non-blocking lock attempt
    
    // No task parameters needed - uses CHI_CUR_WORKER automatically
  };
  
  // RAII wrapper (recommended)
  class ScopedCoMutex {
  public:
    explicit ScopedCoMutex(CoMutex& mutex);  // Locks in constructor
    ~ScopedCoMutex();                        // Unlocks in destructor
  };
}

CoRwLock: Cooperative Reader-Writer Lock​

CoRwLock provides reader-writer semantics with TaskId grouping. Multiple readers can proceed concurrently, but writers have exclusive access.

Basic Usage​

#include <chimaera/corwlock.h>

class Runtime : public chi::Container {
private:
  static chi::CoRwLock data_lock_;  // Protect shared data structures

public:
  void ReadTask(hipc::FullPtr<ReadTaskType> task, chi::RunContext& rctx) {
    // Manual reader lock
    data_lock_.ReadLock();
    // ... read operations ...
    data_lock_.ReadUnlock();
    
    // OR use RAII scoped reader lock (recommended)
    chi::ScopedCoRwReadLock lock(data_lock_);
    // ... read operations ...
    // Automatically unlocks when leaving scope
  }
  
  void WriteTask(hipc::FullPtr<WriteTaskType> task, chi::RunContext& rctx) {
    // RAII scoped writer lock (recommended)
    chi::ScopedCoRwWriteLock lock(data_lock_);
    // ... write operations ...
    // Automatically unlocks when leaving scope
  }
};

// Static member definition
chi::CoRwLock Runtime::data_lock_;

Key Features​

  1. Multiple Readers: Concurrent read access when no writers are active
  2. Exclusive Writers: Writers get exclusive access, blocking all other operations
  3. TaskId Grouping: Tasks with same TaskId can bypass reader locks
  4. Automatic Context: Uses CHI_CUR_WORKER for task identification
  5. RAII Support: Scoped locks for both readers and writers

API Reference​

namespace chi {
  class CoRwLock {
  public:
    // Reader operations
    void ReadLock();                // Acquire reader lock
    void ReadUnlock();              // Release reader lock
    bool TryReadLock();             // Non-blocking reader lock attempt
    
    // Writer operations  
    void WriteLock();               // Acquire exclusive writer lock
    void WriteUnlock();             // Release writer lock
    bool TryWriteLock();            // Non-blocking writer lock attempt
  };
  
  // RAII wrappers (recommended)
  class ScopedCoRwReadLock {
  public:
    explicit ScopedCoRwReadLock(CoRwLock& lock);  // Acquire read lock
    ~ScopedCoRwReadLock();                        // Release read lock
  };
  
  class ScopedCoRwWriteLock {
  public:
    explicit ScopedCoRwWriteLock(CoRwLock& lock); // Acquire write lock
    ~ScopedCoRwWriteLock();                       // Release write lock
  };
}

TaskId Grouping Behavior​

Both CoMutex and CoRwLock support TaskId grouping, which allows related tasks to bypass synchronization:

// Tasks created with the same TaskId can proceed together
auto task_id = chi::CreateTaskId();

// These tasks share the same TaskId - they can bypass CoMutex/CoRwLock
auto task1 = ipc_manager->NewTask<Task1>(task_id, pool_id, pool_query, ...);
auto task2 = ipc_manager->NewTask<Task2>(task_id, pool_id, pool_query, ...);

// This task has a different TaskId - must respect locks normally
auto task3 = ipc_manager->NewTask<Task3>(chi::CreateTaskId(), pool_id, pool_query, ...);

Key Points:

  • Tasks with the same TaskId are considered "grouped" and can bypass locks
  • Use TaskId grouping for logically related operations that don't need mutual exclusion
  • Different TaskIds must respect normal lock semantics

Best Practices​

  1. Use RAII Wrappers: Always prefer ScopedCoMutex and ScopedCoRw*Lock over manual lock/unlock
  2. Static vs Instance: Use static members for cross-container synchronization, instance members for per-container data
  3. Member Definition: Don't forget to define static members in your .cc file
  4. Choose Appropriate Lock: Use CoRwLock for read-heavy workloads, CoMutex for simple mutual exclusion
  5. Minimal Critical Sections: Keep locked sections as small as possible
  6. TaskId Design: Group related tasks that can safely bypass locks

Example: Module with Synchronized Data Structure​

// In MOD_NAME_runtime.h
class Runtime : public chi::Container {
private:
  // Synchronized data structure
  chi::hash_map<chi::u32, ModuleData> data_map_;
  
  // Synchronization primitives
  static chi::CoRwLock data_lock_;        // For data_map_ access
  static chi::CoMutex operation_mutex_;   // For exclusive operations

public:
  void ReadData(hipc::FullPtr<ReadDataTask> task, chi::RunContext& rctx);
  void WriteData(hipc::FullPtr<WriteDataTask> task, chi::RunContext& rctx);
  void ExclusiveOperation(hipc::FullPtr<ExclusiveTask> task, chi::RunContext& rctx);
};

// In MOD_NAME_runtime.cc  
chi::CoRwLock Runtime::data_lock_;
chi::CoMutex Runtime::operation_mutex_;

void Runtime::ReadData(hipc::FullPtr<ReadDataTask> task, chi::RunContext& rctx) {
  chi::ScopedCoRwReadLock lock(data_lock_);  // Multiple readers allowed
  
  // Safe to read data_map_ concurrently
  auto it = data_map_.find(task->key_);
  if (it != data_map_.end()) {
    task->result_data_ = it->second;
    task->result_ = 0;  // Success
  } else {
    task->result_ = 1;  // Not found
  }
}

void Runtime::WriteData(hipc::FullPtr<WriteDataTask> task, chi::RunContext& rctx) {
  chi::ScopedCoRwWriteLock lock(data_lock_);  // Exclusive writer access
  
  // Safe to modify data_map_ exclusively
  data_map_[task->key_] = task->new_data_;
  task->result_ = 0;  // Success
}

void Runtime::ExclusiveOperation(hipc::FullPtr<ExclusiveTask> task, chi::RunContext& rctx) {
  chi::ScopedCoMutex lock(operation_mutex_);  // Exclusive operation
  
  // Perform operation that requires complete exclusivity
  // ... complex operation ...
  task->result_ = 0;  // Success
}

This synchronization model ensures thread-safe access to module data structures while maintaining compatibility with Chimaera's cooperative task execution system.

Pool Query and Task Routing​

Overview of PoolQuery​

PoolQuery is a fundamental component of Chimaera's task routing system that determines where and how tasks are executed across the distributed runtime. It provides flexible routing strategies for load balancing, locality optimization, and distributed execution patterns.

PoolQuery Types​

The chi::PoolQuery class provides six different routing modes through static factory methods:

1. Local Mode​

chi::PoolQuery::Local()
  • Purpose: Routes tasks to the local node only
  • Use Case: Operations that must execute on the calling node
  • Example: MPI-based container creation, node-specific diagnostics
// Client usage in MPI environment
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Local(), "my_pool", custom_pool_id);

2. Direct ID Mode​

chi::PoolQuery::DirectId(ContainerId container_id)
  • Purpose: Routes to a specific container by its unique ID
  • Use Case: Targeted operations on known containers
  • Example: Container-specific configuration changes
// Route to container with ID 42
auto query = chi::PoolQuery::DirectId(ContainerId(42));
client.UpdateConfig(HSHM_MCTX, query, new_config);

3. Direct Hash Mode​

chi::PoolQuery::DirectHash(u32 hash)
  • Purpose: Routes using consistent hash-based load balancing
  • Use Case: Distributing operations across containers deterministically
  • Example: Key-value store operations where keys map to specific containers
// Hash-based routing for a key
u32 hash = std::hash<std::string>{}(key);
auto query = chi::PoolQuery::DirectHash(hash);
client.Put(HSHM_MCTX, query, key, value);

4. Range Mode​

chi::PoolQuery::Range(u32 offset, u32 count)
  • Purpose: Routes to a range of containers
  • Use Case: Batch operations across multiple containers
  • Example: Parallel scan operations, bulk updates
// Process containers 10-19 (10 containers starting at offset 10)
auto query = chi::PoolQuery::Range(10, 10);
client.BulkUpdate(HSHM_MCTX, query, update_data);

5. Broadcast Mode​

chi::PoolQuery::Broadcast()
  • Purpose: Routes to all containers in the pool
  • Use Case: Global operations affecting all containers
  • Example: Configuration updates, global cache invalidation
// Broadcast configuration change to all containers
auto query = chi::PoolQuery::Broadcast();
client.InvalidateCache(HSHM_MCTX, query);

6. Physical Mode​

chi::PoolQuery::Physical(u32 node_id)
  • Purpose: Routes to a specific physical node by ID
  • Use Case: Node-specific operations in distributed deployments
  • Example: Remote node administration, cross-node data migration
// Execute on physical node 3
auto query = chi::PoolQuery::Physical(3);
client.NodeDiagnostics(HSHM_MCTX, query);
chi::PoolQuery::Dynamic()
  • Purpose: Intelligent routing with automatic caching optimization
  • Use Case: Create operations that benefit from local cache checking
  • Behavior: Uses dynamic scheduling (ExecMode::kDynamicSchedule) for cache optimization
    1. Check if pool exists locally using PoolManager
    2. If pool exists: change pool_query to Local (execute locally using existing pool)
    3. If pool doesn't exist: change pool_query to Broadcast (create pool on all nodes)
  • Benefits:
    • Avoids redundant pool creation attempts
    • Eliminates unnecessary network overhead for existing pools
    • Automatic fallback to broadcast creation when needed
  • Example: Container creation with automatic caching
// Recommended: Use Dynamic() for Create operations
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Dynamic(), "my_pool", custom_pool_id);

// Dynamic scheduling will:
// - Check local cache for "my_pool"
// - If found: switch to Local mode (fast path)
// - If not found: switch to Broadcast mode (creation path)

PoolQuery Usage Guidelines​

Best Practices​

  1. Never use null queries: Always specify an explicit PoolQuery type
  2. Default to Dynamic for Create: Use PoolQuery::Dynamic() for container creation to enable automatic caching optimization
  3. Alternative: Use Broadcast or Local explicitly:
    • Use Broadcast() when you want to force distributed creation regardless of cache
    • Use Local() in MPI jobs when you want node-local containers only
  4. Consider locality: Prefer local execution to minimize network overhead for regular operations
  5. Use appropriate granularity: Match routing mode to operation scope

Common Patterns​

Container Creation Pattern (Recommended):

// Recommended: Use Dynamic for automatic cache optimization
// This checks local cache first and falls back to broadcast creation if needed
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Dynamic(), "my_pool_name", custom_pool_id);

Container Creation Pattern (Explicit Broadcast):

// Alternative: Use Broadcast to force distributed creation regardless of cache
// This ensures the container is created across all nodes in distributed environments
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Broadcast(), "my_pool_name", custom_pool_id);

Container Creation Pattern (MPI Environments):

// In MPI jobs, Local may be more efficient for node-local containers
// Use Local when you want node-local containers only
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Local(), "my_pool_name", custom_pool_id);

Load-Balanced Operations:

// Use hash-based routing for even distribution
for (const auto& item : items) {
  u32 hash = ComputeHash(item.id);
  auto query = chi::PoolQuery::DirectHash(hash);
  client.Process(HSHM_MCTX, query, item);
}

Batch Processing:

// Process containers in chunks
const u32 total_containers = pool_info->num_containers_;
const u32 batch_size = 10;
for (u32 offset = 0; offset < total_containers; offset += batch_size) {
  u32 count = std::min(batch_size, total_containers - offset);
  auto query = chi::PoolQuery::Range(offset, count);
  client.BatchProcess(HSHM_MCTX, query, batch_data);
}

Runtime Routing Implementation​

The runtime uses PoolQuery to determine task routing through several stages:

  1. Query Validation: Ensures the query parameters are valid
  2. Container Resolution: Maps query to specific container(s)
  3. Task Distribution: Routes task to appropriate worker queues
  4. Load Balancing: Applies distribution strategies for multi-container queries

PoolQuery in Task Definitions​

Tasks must include PoolQuery in their constructors:

class CustomTask : public chi::Task {
 public:
  CustomTask(hipc::Allocator *alloc,
             const chi::TaskId &task_id,
             const chi::PoolId &pool_id,
             const chi::PoolQuery &pool_query,  // Required parameter
             /* custom parameters */)
      : chi::Task(alloc, task_id, pool_id, pool_query, method_id) {
    // Task initialization
  }
};

Advanced PoolQuery Features​

Query Introspection​

PoolQuery query = PoolQuery::Range(0, 10);

// Check routing mode
if (query.IsRangeMode()) {
  u32 offset = query.GetRangeOffset();
  u32 count = query.GetRangeCount();
  // Process range parameters
}

// Get routing mode enum
RoutingMode mode = query.GetRoutingMode();
switch (mode) {
  case RoutingMode::Local:
    // Handle local routing
    break;
  case RoutingMode::Broadcast:
    // Handle broadcast
    break;
  // ... other cases
}

Combining with Task Priorities​

// High-priority broadcast
auto query = chi::PoolQuery::Broadcast();
auto task = ipc_manager->NewTask<UpdateTask>(
    chi::CreateTaskId(),
    pool_id,
    query,
    update_data
);
ipc_manager->Enqueue(task, chi::kHighPriority);

Troubleshooting PoolQuery Issues​

Common Errors:

  1. Null Query Error: "NEVER use a null pool query"

    • Solution: Always use a factory method like PoolQuery::Local()

  2. Invalid Container ID: Container not found for DirectId query

    • Solution: Verify container exists before using DirectId

  3. Range Out of Bounds: Range exceeds available containers

    • Solution: Check pool size before creating Range queries

  4. Node ID Invalid: Physical node ID doesn't exist

    • Solution: Validate node IDs against cluster configuration

Client-Server Communication​

Client Implementation Patterns​

Critical Pool ID Update Pattern​

IMPORTANT: All ChiMod clients that implement Create methods MUST update their pool_id_ field with the actual pool ID returned from completed CreateTask operations. This is essential because:

  1. CreateTask operations may return a different pool ID than initially specified
  2. Pool creation may reuse existing pools with different IDs
  3. Subsequent client operations depend on the correct pool ID

Required Pattern for All Client Create Methods:

void Create(const hipc::MemContext& mctx,
            const chi::PoolQuery& pool_query,
            const std::string& pool_name,
            const chi::PoolId& custom_pool_id,
            /* other module-specific parameters */) {
    auto task = AsyncCreate(mctx, pool_query, pool_name, custom_pool_id, /* other params */);
    task->Wait();

    // CRITICAL: Update client pool_id_ with the actual pool ID from the task
    pool_id_ = task->new_pool_id_;

    CHI_IPC->DelTask(task);
}

Required Parameters for All Create Methods:

  1. mctx: Memory context for shared memory allocations
  2. pool_query: Task routing strategy (use Broadcast() for non-MPI, Local() for MPI)
  3. pool_name: User-provided name for the pool (must be unique, used as file path for file-based modules)
  4. custom_pool_id: Explicit pool ID for the container being created (must not be null)
  5. Module-specific parameters: Additional parameters specific to the ChiMod (e.g., BDev type, size)

Why This Is Required:

  • Pool Reuse: CreateTask is actually a GetOrCreatePoolTask that may return an existing pool
  • ID Assignment: The admin ChiMod may assign a different pool ID than requested
  • Client Consistency: All subsequent operations must use the correct pool ID
  • Distributed Operation: Pool IDs must be consistent across all client instances

Examples of Correct Implementation:

// Admin client Create method
void Create(const hipc::MemContext& mctx,
            const chi::PoolQuery& pool_query,
            const std::string& pool_name,
            const chi::PoolId& custom_pool_id) {
    auto task = AsyncCreate(mctx, pool_query, pool_name, custom_pool_id);
    task->Wait();

    // CRITICAL: Update client pool_id_ with the actual pool ID from the task
    pool_id_ = task->new_pool_id_;

    auto* ipc_manager = CHI_IPC;
    ipc_manager->DelTask(task);
}

// BDev client Create method (with module-specific parameters)
void Create(const hipc::MemContext& mctx,
            const chi::PoolQuery& pool_query,
            const std::string& pool_name,
            const chi::PoolId& custom_pool_id,
            BdevType bdev_type,
            chi::u64 total_size = 0,
            chi::u32 io_depth = 128,
            chi::u32 alignment = 4096) {
    auto task = AsyncCreate(mctx, pool_query, pool_name, custom_pool_id,
                           bdev_type, total_size, io_depth, alignment);
    task->Wait();

    // CRITICAL: Update client pool_id_ with the actual pool ID from the task
    pool_id_ = task->new_pool_id_;

    CHI_IPC->DelTask(task);
}

// MOD_NAME client Create method (simple case)
void Create(const hipc::MemContext& mctx,
            const chi::PoolQuery& pool_query,
            const std::string& pool_name,
            const chi::PoolId& custom_pool_id) {
    auto task = AsyncCreate(mctx, pool_query, pool_name, custom_pool_id);
    task->Wait();

    // CRITICAL: Update client pool_id_ with the actual pool ID from the task
    pool_id_ = task->new_pool_id_;

    CHI_IPC->DelTask(task);
}

Common Mistakes to Avoid:

  • ❌ Using null PoolId for custom_pool_id: Create operations REQUIRE explicit, non-null pool IDs
  • ❌ Forgetting to update pool_id_: Leads to incorrect pool ID for subsequent operations
  • ❌ Using original pool_id_: The task may return a different pool ID than initially specified
  • ❌ Updating before task completion: Always wait for task completion before reading new_pool_id_
  • ❌ Not implementing this pattern: All synchronous Create methods must follow this pattern
  • ❌ Using Local instead of Broadcast: In non-MPI environments, use Broadcast() for distributed container creation

Critical Validation:

The runtime validates that custom_pool_id is not null during Create operations. If a null PoolId is provided, the Create operation will fail with an error:

// WRONG - This will fail with error
chi::PoolId null_id;  // Null pool ID
client.Create(HSHM_MCTX, chi::PoolQuery::Broadcast(), "my_pool", null_id);
// Error: "Cannot create pool with null PoolId. Explicit pool IDs are required."

// CORRECT - Always provide explicit pool IDs
const chi::PoolId custom_pool_id(7000, 0);
client.Create(HSHM_MCTX, chi::PoolQuery::Broadcast(), "my_pool", custom_pool_id);

This pattern is mandatory for all ChiMod clients and ensures correct pool ID management throughout the client lifecycle.

Memory Segments​

Three shared memory segments are used:

  1. Main Segment: Tasks and control structures
  2. Client Data Segment: User data buffers
  3. Runtime Data Segment: Runtime-only data

IPC Queue​

Tasks are submitted via a lock-free multi-producer single-consumer queue:

// Client side
auto task = ipc_manager->NewTask<CustomTask>(...);
ipc_manager->Enqueue(task, chi::kLowLatency);

// Server side
hipc::Pointer task_ptr = ipc_manager->Dequeue(chi::kLowLatency);

Memory Management​

Allocator Usage​

// Get context allocator for current segment
hipc::CtxAllocator<CHI_MAIN_ALLOC_T> ctx_alloc(HSHM_MCTX, allocator);

// Allocate serializable string
chi::string my_string(ctx_alloc, "initial value");

// Allocate vector
chi::vector<u32> my_vector(ctx_alloc);
my_vector.resize(100);

Best Practices​

  1. Always use HSHM types for shared data
  2. Pass CtxAllocator to constructors
  3. Use FullPtr for cross-process references
  4. Let framework handle task cleanup via ipc_manager->DelTask()

Task Allocation and Deallocation Pattern​

// Client side - allocation (NewTask uses main allocator automatically)
auto task = ipc_manager->NewTask<CustomTask>(
    chi::CreateTaskId(),
    pool_id_,
    pool_query,
    input_data,
    operation_id);

// Client side - cleanup (after task completion)
ipc_manager->DelTask(task);

// Runtime side - automatic cleanup (no code needed)
// Framework Del dispatcher calls ipc_manager->DelTask() automatically

CHI_IPC Buffer Allocation​

The CHI_IPC singleton provides centralized buffer allocation for shared memory operations in client code. Use this for allocating temporary buffers that need to be shared between client and runtime processes.

Important: AllocateBuffer returns hipc::FullPtr<char>, not hipc::Pointer. It is NOT a template function.

Basic Usage​

#include <chimaera/chimaera.h>

// Get the IPC manager singleton
auto* ipc_manager = CHI_IPC;

// Allocate a buffer in shared memory (returns FullPtr<char>)
size_t buffer_size = 1024;
hipc::FullPtr<char> buffer_ptr = ipc_manager->AllocateBuffer(buffer_size);

// Use the buffer (example: copy data into it)
char* buffer_data = buffer_ptr.ptr_;
memcpy(buffer_data, source_data, data_size);

// Alternative: Use directly
strncpy(buffer_ptr.ptr_, "example data", buffer_size);

// The buffer will be automatically freed when buffer_ptr goes out of scope
// or when explicitly deallocated by the framework

Use Cases for CHI_IPC Buffers​

  • Temporary data transfer: When passing large data to tasks
  • Intermediate storage: For computations that need shared memory
  • I/O operations: Reading/writing data that needs to be accessible by runtime

Best Practices​

// ✅ Good: Use CHI_IPC for temporary shared buffers
auto* ipc_manager = CHI_IPC;
hipc::FullPtr<char> temp_buffer = ipc_manager->AllocateBuffer(data_size);

// ✅ Good: Use chi::ipc types for persistent task data
chi::ipc::string task_string(ctx_alloc, "persistent data");

// ❌ Avoid: Don't use CHI_IPC for small, simple task parameters
// Use chi::ipc types directly in task definitions instead

Shared-Memory Compatible Data Structures​

For task definitions and any data that needs to be shared between client and runtime processes, always use shared-memory compatible types instead of standard C++ containers.

chi::ipc::string​

Use chi::ipc::string or hipc::string instead of std::string in task definitions:

#include <[namespace]/types.h>

// Task definition using shared-memory string
struct CustomTask : public chi::Task {
  INOUT hipc::string input_data_;     // Shared-memory compatible string
  INOUT hipc::string output_data_;    // Results stored in shared memory
  
  CustomTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T>& alloc,
             const std::string& input) 
    : input_data_(alloc, input),      // Initialize from std::string
      output_data_(alloc) {}          // Empty initialization
      
  // Conversion to std::string when needed
  std::string getResult() const {
    return std::string(output_data_.data(), output_data_.size());
  }
};

chi::ipc::vector​

Use chi::ipc::vector instead of std::vector for arrays in task definitions:

// Task definition using shared-memory vector
struct ProcessArrayTask : public chi::Task {
  INOUT chi::ipc::vector<chi::u32> data_array_;
  INOUT chi::ipc::vector<chi::f32> result_array_;
  
  ProcessArrayTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T>& alloc,
                   const std::vector<chi::u32>& input_data)
    : data_array_(alloc),
      result_array_(alloc) {
    // Copy from std::vector to shared-memory vector
    data_array_.resize(input_data.size());
    std::copy(input_data.begin(), input_data.end(), data_array_.begin());
  }
};

When to Use Each Type​

Use shared-memory types (chi::ipc::string, hipc::string, chi::ipc::vector, etc.) for:

  • Task input/output parameters
  • Data that persists across task execution
  • Any data structure that needs serialization
  • Data shared between client and runtime

Use std::string/vector for:

  • Local variables in client code
  • Temporary computations
  • Converting to/from shared-memory types

Type Conversion Examples​

// Converting between std::string and shared-memory string types
std::string std_str = "example data";
hipc::string shm_str(ctx_alloc, std_str);          // std -> shared memory
std::string result = std::string(shm_str);         // shared memory -> std

// Converting between std::vector and shared-memory vector types
std::vector<int> std_vec = {1, 2, 3, 4, 5};
chi::ipc::vector<int> shm_vec(ctx_alloc);
shm_vec.assign(std_vec.begin(), std_vec.end());    // std -> shared memory

std::vector<int> result_vec(shm_vec.begin(), shm_vec.end());  // shared memory -> std

Serialization Support​

Both chi::ipc::string and chi::ipc::vector automatically support serialization for task communication:

// Task definition - no manual serialization needed
struct SerializableTask : public chi::Task {
  INOUT hipc::string message_;
  INOUT chi::ipc::vector<chi::u64> timestamps_;

  // Cereal automatically handles chi::ipc types
  template<class Archive>
  void serialize(Archive& ar) {
    ar(message_, timestamps_);  // Works automatically
  }
};

Bulk Transfer Support with ar.bulk​

For tasks that involve large data transfers (such as I/O operations), Chimaera provides ar.bulk() for efficient bulk data serialization. This feature integrates with the Lightbeam networking layer to enable zero-copy data transfer and RDMA optimization.

Overview​

The ar.bulk() method marks data pointers for bulk transfer during task serialization. This is essential for:

  • Large I/O Operations: Read/write tasks with multi-megabyte payloads
  • Zero-Copy Transfer: Avoiding unnecessary data copies during serialization
  • RDMA Optimization: Preparing data for remote direct memory access
  • Distributed Execution: Sending tasks with large data buffers across nodes

Bulk Transfer Flags​

Two flags control bulk transfer behavior:

// Defined in hermes_shm/lightbeam/lightbeam.h
#define BULK_EXPOSE  // Metadata only - no data transfer (receiver allocates)
#define BULK_XFER    // Marks bulk for actual data transmission

Flag Usage:

  • BULK_EXPOSE: Sender exposes metadata (size, pointer info) but doesn't transfer data
    • Receiver sees the bulk size and allocates local buffer
    • Useful when receiver will write data (e.g., Read operations)
  • BULK_XFER: Marks bulk for actual data transmission
    • Data is transferred over network
    • Used when sender has data to send (e.g., Write operations)

Basic Usage Pattern​

Write Operation (Sender Has Data)​

For write operations, the sender has data to transfer:

struct WriteTask : public chi::Task {
  IN Block block_;              // Block to write to
  IN hipc::Pointer data_;       // Data buffer pointer
  IN size_t length_;            // Data length
  OUT chi::u64 bytes_written_;  // Result

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // Use BULK_XFER to transfer data from sender to receiver
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(bytes_written_);
    // No bulk transfer needed for output (just metadata)
  }
};

Workflow:

  1. Client Side (SerializeIn):
    • Serializes block_ and length_ metadata
    • Marks data_ buffer with BULK_XFER flag
    • Lightbeam transmits the data buffer to receiver
  2. Runtime Side (Execute):
    • Receives metadata and data buffer
    • Executes write operation using transferred data
    • Sets bytes_written_ result
  3. Client Side (SerializeOut):
    • Receives bytes_written_ result
    • No bulk transfer needed for small output values
Read Operation (Receiver Needs Data)​

For read operations, the receiver allocates buffer for incoming data:

struct ReadTask : public chi::Task {
  IN Block block_;              // Block to read from
  OUT hipc::Pointer data_;      // Data buffer pointer (allocated by receiver)
  INOUT size_t length_;         // Requested/actual length
  OUT chi::u64 bytes_read_;     // Result

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // Use BULK_EXPOSE - metadata only, receiver will allocate buffer
    ar.bulk(data_, length_, BULK_EXPOSE);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_, bytes_read_);
    // Use BULK_XFER to transfer read data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }
};

Workflow:

  1. Client Side (SerializeIn):
    • Serializes block_ and length_ metadata
    • Marks data_ with BULK_EXPOSE (no data sent yet)
    • Receiver sees buffer size needed
  2. Runtime Side (Execute):
    • Receives metadata including buffer size
    • Allocates local buffer for data_
    • Executes read operation filling the buffer
    • Sets length_ and bytes_read_ results
  3. Client Side (SerializeOut):
    • Marks data_ with BULK_XFER flag
    • Lightbeam transfers read data back to client
    • Client receives length_, bytes_read_, and data buffer

API Reference​

template <typename Archive>
void ar.bulk(hipc::Pointer ptr, size_t size, uint32_t flags);

Parameters:

  • ptr: Pointer to data buffer (hipc::Pointer, hipc::FullPtr, or raw pointer)
  • size: Size of data in bytes
  • flags: Transfer flags (BULK_EXPOSE or BULK_XFER)

Behavior:

  • Records bulk transfer metadata in the archive
  • For BULK_XFER: Prepares data for network transmission
  • For BULK_EXPOSE: Records metadata only (size and pointer info)
  • Integrates with Lightbeam networking for actual data transfer

Advanced Pattern: Bidirectional Transfer​

Some operations require data transfer in both directions:

struct ProcessTask : public chi::Task {
  INOUT hipc::Pointer data_;    // Data buffer (modified in-place)
  INOUT size_t length_;         // Buffer length

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(length_);
    // Send input data to runtime
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_);
    // Send modified data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }
};

Integration with Lightbeam​

The ar.bulk() calls integrate seamlessly with the Lightbeam networking layer:

  1. Archive Records Bulks:

    • TaskOutputArchive stores bulk metadata in bulk_transfers_ vector
    • Each bulk includes pointer, size, and flags

  2. Lightbeam Transmission:

    • Bulks marked BULK_XFER are transmitted via Send() and RecvBulks()
    • Bulks marked BULK_EXPOSE provide metadata only
    • Receiver inspects all bulks to determine buffer sizes

  3. Zero-Copy Optimization:

    • Data stays in original buffers during serialization
    • Only pointers and metadata are serialized
    • Actual data transfer handled separately by Lightbeam

Complete Example: BDev Read Task​

struct ReadTask : public chi::Task {
  IN Block block_;              // Block descriptor
  OUT hipc::Pointer data_;      // Data buffer
  INOUT size_t length_;         // Buffer length
  OUT chi::u64 bytes_read_;     // Bytes actually read

  /** SHM constructor */
  explicit ReadTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc)
      : chi::Task(alloc), length_(0), bytes_read_(0) {}

  /** Emplace constructor */
  explicit ReadTask(const hipc::CtxAllocator<CHI_MAIN_ALLOC_T> &alloc,
                    const chi::TaskId &task_node,
                    const chi::PoolId &pool_id,
                    const chi::PoolQuery &pool_query,
                    const Block &block,
                    hipc::Pointer data,
                    size_t length)
      : chi::Task(alloc, task_node, pool_id, pool_query, 10),
        block_(block), data_(data), length_(length), bytes_read_(0) {
    task_id_ = task_node;
    pool_id_ = pool_id;
    method_ = Method::kRead;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }

  /** Serialize IN and INOUT parameters */
  template <typename Archive>
  void SerializeIn(Archive &ar) {
    ar(block_, length_);
    // BULK_EXPOSE: Tell receiver the buffer size, but don't send data yet
    // Receiver will allocate local buffer
    ar.bulk(data_, length_, BULK_EXPOSE);
  }

  /** Serialize OUT and INOUT parameters */
  template <typename Archive>
  void SerializeOut(Archive &ar) {
    ar(length_, bytes_read_);
    // BULK_XFER: Transfer the read data back to client
    ar.bulk(data_, length_, BULK_XFER);
  }

  /** Copy from another ReadTask */
  void Copy(const hipc::FullPtr<ReadTask> &other) {
    // Copy task-specific fields only
    // Base Task fields are copied automatically by NewCopy
    block_ = other->block_;
    data_ = other->data_;
    length_ = other->length_;
    bytes_read_ = other->bytes_read_;
  }

  /** Aggregate results from replica */
  void Aggregate(const hipc::FullPtr<ReadTask> &other) {
    // For reads, just copy the result from the replica
    Copy(other);
  }
};

Best Practices​

DO:

  • ✅ Use BULK_XFER when sender has data to transmit (Write operations)
  • ✅ Use BULK_EXPOSE when receiver needs to allocate buffer (Read operations)
  • ✅ Always specify both SerializeIn() and SerializeOut() for consistency
  • ✅ Use ar.bulk() for data buffers larger than a few KB
  • ✅ Ensure data buffer lifetime extends until serialization completes

DON'T:

  • ❌ Don't use ar.bulk() for small data (< 4KB) - serialize directly instead
  • ❌ Don't forget to specify bulk size - it determines receiver buffer allocation
  • ❌ Don't mix ar() and ar.bulk() for the same data - choose one approach
  • ❌ Don't use BULK_EXPOSE for write operations (sender has data to send)
  • ❌ Don't use BULK_XFER in SerializeIn for read operations (no data to send yet)

Performance Considerations​

  1. Buffer Alignment: Ensure buffers are properly aligned (typically 4KB for I/O operations)
  2. Size Thresholds: Use bulk transfer for data > 4KB; use regular serialization for smaller data
  3. Zero-Copy: Lightbeam can use zero-copy techniques when data is in shared memory
  4. RDMA Ready: The bulk transfer API is designed for future RDMA transport integration

Troubleshooting​

Common Issues:

  1. Missing Data Transfer:

    • Ensure BULK_XFER flag is used when data should be transmitted
    • Check that SerializeOut uses BULK_XFER for read operations

  2. Buffer Size Mismatch:

    • Verify length_ parameter matches actual buffer size
    • Ensure receiver allocates buffer matching the exposed size

  3. Serialization Order:

    • Serialize metadata (block, length) before ar.bulk() call
    • This ensures receiver knows buffer size before allocating

chi::unordered_map_ll - Lock-Free Unordered Map​

The chi::unordered_map_ll is a hash map implementation using a vector of lists design that provides efficient concurrent access when combined with external locking. This container is specifically designed for runtime module data structures that require external synchronization control.

Overview​

Key Characteristics:

  • Vector of Lists Design: Uses a vector of buckets, each containing a list of key-value pairs
  • External Locking Required: No internal mutexes - users must provide synchronization
  • Bucket Partitioning: Hash space is partitioned across multiple buckets for better cache locality
  • Standard API: Compatible with std::unordered_map interface
  • NOT Shared-Memory Compatible: For runtime-only data structures, not task parameters

Basic Usage​

#include <chimaera/unordered_map_ll.h>

class Runtime : public chi::Container {
private:
  // Runtime data structure with external locking
  chi::unordered_map_ll<chi::u32, ModuleData> data_map_;

  // External synchronization using CoRwLock
  static chi::CoRwLock data_lock_;

public:
  Runtime() : data_map_(32) {}  // 32 buckets for hash partitioning

  void ReadData(hipc::FullPtr<ReadTask> task, chi::RunContext& ctx) {
    chi::ScopedCoRwReadLock lock(data_lock_);

    // Safe to access data_map_ with external lock held
    auto* value = data_map_.find(task->key_);
    if (value) {
      task->result_ = *value;
    }
  }

  void WriteData(hipc::FullPtr<WriteTask> task, chi::RunContext& ctx) {
    chi::ScopedCoRwWriteLock lock(data_lock_);

    // Safe to modify data_map_ with exclusive lock
    data_map_.insert_or_assign(task->key_, task->data_);
  }
};

// Static member definition
chi::CoRwLock Runtime::data_lock_;

Constructor​

// Create map with specified bucket count (determines max useful concurrency)
chi::unordered_map_ll<Key, T> map(max_concurrency);

// Example: 32 buckets provides good distribution for most workloads
chi::unordered_map_ll<int, std::string> map(32);

Parameters:

  • max_concurrency: Number of buckets (default: 16)
    • Higher values = better distribution, more memory overhead
    • Typical values: 16-64 for most use cases
    • Should be power of 2 for optimal hash distribution

API Reference​

The container provides a std::unordered_map-compatible interface:

// Insertion operations
auto [inserted, value_ptr] = map.insert(key, value);          // Insert if not exists
auto [inserted, value_ptr] = map.insert_or_assign(key, value); // Insert or update
T& ref = map[key];                                            // Insert default if missing

// Lookup operations
T* ptr = map.find(key);                    // Returns nullptr if not found
const T* ptr = map.find(key) const;        // Const version
T& ref = map.at(key);                      // Throws if not found
bool exists = map.contains(key);           // Check existence
size_t count = map.count(key);             // Returns 0 or 1

// Removal operations
size_t erased = map.erase(key);            // Returns number of elements erased
void map.clear();                          // Remove all elements

// Size operations
size_t size = map.size();                  // Total element count
bool empty = map.empty();                  // Check if empty
size_t buckets = map.bucket_count();       // Number of buckets

// Iteration
map.for_each([](const Key& key, T& value) {
  // Process each element
  // Note: External lock must be held during iteration
});

Return Value Semantics​

Insert operations return std::pair<bool, T*>:

  • first: true if insertion occurred, false if key already exists
  • second: Pointer to the value (existing or newly inserted)
auto [inserted, value_ptr] = map.insert(42, "hello");
if (inserted) {
  // New element was inserted
  std::cout << "Inserted: " << *value_ptr << std::endl;
} else {
  // Key already existed
  std::cout << "Existing: " << *value_ptr << std::endl;
}

External Locking Patterns​

Pattern 1: CoRwLock for Read-Heavy Workloads

class Runtime : public chi::Container {
private:
  chi::unordered_map_ll<chi::u64, CachedData> cache_;
  static chi::CoRwLock cache_lock_;

public:
  void LookupCache(hipc::FullPtr<LookupTask> task, chi::RunContext& ctx) {
    chi::ScopedCoRwReadLock lock(cache_lock_);  // Multiple readers allowed

    auto* data = cache_.find(task->cache_key_);
    if (data) {
      task->result_ = *data;
      task->found_ = true;
    } else {
      task->found_ = false;
    }
  }

  void UpdateCache(hipc::FullPtr<UpdateTask> task, chi::RunContext& ctx) {
    chi::ScopedCoRwWriteLock lock(cache_lock_);  // Exclusive writer

    cache_.insert_or_assign(task->cache_key_, task->new_data_);
  }
};

chi::CoRwLock Runtime::cache_lock_;

Pattern 2: CoMutex for Write-Heavy Workloads

class Runtime : public chi::Container {
private:
  chi::unordered_map_ll<std::string, RequestCounter> counters_;
  static chi::CoMutex counters_mutex_;

public:
  void IncrementCounter(hipc::FullPtr<IncrementTask> task, chi::RunContext& ctx) {
    chi::ScopedCoMutex lock(counters_mutex_);

    auto [inserted, counter_ptr] = counters_.insert(task->counter_name_, RequestCounter{});
    counter_ptr->count++;
    task->new_count_ = counter_ptr->count;
  }
};

chi::CoMutex Runtime::counters_mutex_;

Pattern 3: Instance-Level Locking

class Runtime : public chi::Container {
private:
  // Per-container instance data
  chi::unordered_map_ll<chi::u32, TaskState> active_tasks_;
  chi::CoMutex instance_lock_;  // Instance member, not static

public:
  void RegisterTask(hipc::FullPtr<RegisterTask> task, chi::RunContext& ctx) {
    chi::ScopedCoMutex lock(instance_lock_);  // Lock this container instance only

    active_tasks_.insert(task->task_id_, TaskState{task->start_time_});
  }
};

When to Use chi::unordered_map_ll​

✅ Use chi::unordered_map_ll for:

  • Runtime container data structures (caches, registries, counters)
  • Module-internal state management
  • Lookup tables for fast key-value access
  • Data structures protected by CoMutex/CoRwLock
  • Non-shared memory data (runtime process only)

❌ Do NOT use chi::unordered_map_ll for:

  • Task input/output parameters (use chi::ipc:: types instead)
  • Shared-memory data structures (not compatible with HSHM allocators)
  • Client-side code (use std::unordered_map instead)
  • Data that needs to be serialized (use std::unordered_map with cereal)

Performance Considerations​

Bucket Count Selection:

// Small datasets (< 100 elements): 16 buckets
chi::unordered_map_ll<Key, Value> small_map(16);

// Medium datasets (100-10000 elements): 32-64 buckets
chi::unordered_map_ll<Key, Value> medium_map(32);

// Large datasets (> 10000 elements): 64-128 buckets
chi::unordered_map_ll<Key, Value> large_map(64);

// Very large datasets or high concurrency: 128+ buckets
chi::unordered_map_ll<Key, Value> huge_map(128);

Iteration Performance:

// Iteration requires external lock for entire duration
void ProcessAllEntries(hipc::FullPtr<Task> task, chi::RunContext& ctx) {
  chi::ScopedCoRwReadLock lock(data_lock_);  // Hold lock during entire iteration

  size_t count = 0;
  data_map_.for_each([&count](const Key& key, Value& value) {
    // Process entry
    count++;
  });

  task->processed_count_ = count;
  // Lock released when scope exits
}

Complete Example: Request Tracking Module​

// In MOD_NAME_runtime.h
#include <chimaera/unordered_map_ll.h>
#include <chimaera/corwlock.h>

class Runtime : public chi::Container {
private:
  // Request tracking data structure
  struct RequestInfo {
    chi::u64 start_time_us_;
    chi::u64 bytes_processed_;
    chi::u32 status_code_;
  };

  // Map of active requests (external locking required)
  chi::unordered_map_ll<chi::u64, RequestInfo> active_requests_;

  // Completed request statistics
  chi::unordered_map_ll<chi::u32, chi::u64> status_counts_;

  // Synchronization primitives
  static chi::CoRwLock requests_lock_;
  static chi::CoMutex stats_mutex_;

public:
  Runtime()
    : active_requests_(64),   // 64 buckets for active requests
      status_counts_(16) {}   // 16 buckets for status codes

  void StartRequest(hipc::FullPtr<StartRequestTask> task, chi::RunContext& ctx) {
    chi::ScopedCoRwWriteLock lock(requests_lock_);

    RequestInfo info{
      .start_time_us_ = task->timestamp_,
      .bytes_processed_ = 0,
      .status_code_ = 0
    };

    active_requests_.insert(task->request_id_, info);
  }

  void CompleteRequest(hipc::FullPtr<CompleteRequestTask> task, chi::RunContext& ctx) {
    {
      // Update active requests
      chi::ScopedCoRwWriteLock lock(requests_lock_);

      auto* info = active_requests_.find(task->request_id_);
      if (info) {
        task->duration_us_ = task->end_time_ - info->start_time_us_;
        task->bytes_processed_ = info->bytes_processed_;

        // Update statistics
        {
          chi::ScopedCoMutex stats_lock(stats_mutex_);
          auto [inserted, count_ptr] = status_counts_.insert_or_assign(
            info->status_code_, 0);
          (*count_ptr)++;
        }

        active_requests_.erase(task->request_id_);
      }
    }
  }

  void GetStatistics(hipc::FullPtr<GetStatsTask> task, chi::RunContext& ctx) {
    // Read statistics with read lock
    chi::ScopedCoRwReadLock lock(requests_lock_);

    task->active_count_ = active_requests_.size();

    // Get status code distribution
    chi::ScopedCoMutex stats_lock(stats_mutex_);
    status_counts_.for_each([&task](const chi::u32& status, const chi::u64& count) {
      task->status_distribution_.push_back({status, count});
    });
  }
};

// Static member definitions
chi::CoRwLock Runtime::requests_lock_;
chi::CoMutex Runtime::stats_mutex_;

Key Differences from std::unordered_map​

Featurestd::unordered_mapchi::unordered_map_ll
Thread SafetyNone (external locking required)None (external locking required)
Internal StructureImplementation-definedVector of lists (explicit)
Bucket CountDynamic rehashingFixed at construction
Iterator StabilityUnstable across insertionsStable (list-based)
Shared MemoryNot compatibleNot compatible
Return ValuesIteratorsPointers to values
Use CaseGeneral purposeRuntime data structures

Summary​

chi::unordered_map_ll provides a specialized hash map implementation optimized for Chimaera runtime modules:

  1. External Locking: Must be protected by CoMutex or CoRwLock
  2. Fixed Buckets: Bucket count set at construction (no rehashing)
  3. Pointer Interface: Operations return pointers instead of iterators
  4. Runtime Only: Not for shared-memory or task parameters
  5. Efficient Lookup: O(1) average case for find/insert/erase operations

For runtime container data structures requiring fast key-value access with external synchronization, chi::unordered_map_ll provides an efficient and predictable solution.

Build System Integration​

CMakeLists.txt Template​

ChiMod CMakeLists.txt files should use the standardized ChimaeraCommon.cmake functions for consistency and proper configuration:

cmake_minimum_required(VERSION 3.10)

# Create both client and runtime libraries for your module
# This creates targets: ${NAMESPACE}_${CHIMOD_NAME}_runtime and ${NAMESPACE}_${CHIMOD_NAME}_client
# CMake aliases: ${NAMESPACE}::${CHIMOD_NAME}_runtime and ${NAMESPACE}::${CHIMOD_NAME}_client
add_chimod_client(
  CHIMOD_NAME YOUR_MODULE_NAME
  SOURCES src/YOUR_MODULE_NAME_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME YOUR_MODULE_NAME
  SOURCES src/YOUR_MODULE_NAME_runtime.cc src/autogen/YOUR_MODULE_NAME_lib_exec.cc
)

# Installation is automatic - no separate install_chimod() call required

CMakeLists.txt Guidelines​

DO:

  • Use add_chimod_client() and add_chimod_runtime() utility functions (installation is automatic)
  • Set CHIMOD_NAME to your module's name
  • List source files explicitly in SOURCES parameters
  • Include autogen source files in runtime SOURCES
  • Keep the CMakeLists.txt minimal and consistent

DON'T:

  • Use manual add_library() calls - use the utilities instead
  • Call install_chimod() separately - it's handled automatically
  • Include relative paths like ../include/* - use proper include directories
  • Set custom compile definitions - the utilities handle this
  • Manually configure target properties - the utilities provide standard settings

ChiMod Build Functions Reference​

add_chimod_client() Function​

Creates a ChiMod client library target with automatic dependency management.

add_chimod_client(
  SOURCES source_file1.cc source_file2.cc ...
  [COMPILE_DEFINITIONS definition1 definition2 ...]
  [LINK_LIBRARIES library1 library2 ...]
  [LINK_DIRECTORIES directory1 directory2 ...]
  [INCLUDE_LIBRARIES target1 target2 ...]
  [INCLUDE_DIRECTORIES directory1 directory2 ...]
)

Parameters:

  • SOURCES (required): List of source files for the client library
  • COMPILE_DEFINITIONS (optional): Additional preprocessor definitions beyond automatic ones
  • LINK_LIBRARIES (optional): Additional libraries to link beyond automatic dependencies
  • LINK_DIRECTORIES (optional): Additional library search directories
  • INCLUDE_LIBRARIES (optional): Target libraries whose include directories should be inherited
  • INCLUDE_DIRECTORIES (optional): Additional include directories beyond automatic ones

Automatic Behavior:

  • Creates target: ${NAMESPACE}_${CHIMOD_NAME}_client
  • Creates alias: ${NAMESPACE}::${CHIMOD_NAME}_client
  • Automatically links core Chimaera library (chimaera::cxx or hermes_shm::cxx)
  • For non-admin ChiMods: automatically links chimaera_admin_client
  • Automatically includes module headers from include/ directory
  • Installs library and headers with proper CMake export configuration

Example:

add_chimod_client(
  SOURCES src/my_module_client.cc
  COMPILE_DEFINITIONS MY_MODULE_DEBUG=1
  LINK_LIBRARIES additional_lib
  INCLUDE_DIRECTORIES ${EXTERNAL_INCLUDE_DIR}
)

add_chimod_runtime() Function​

Creates a ChiMod runtime library target with automatic dependency management.

add_chimod_runtime(
  SOURCES source_file1.cc source_file2.cc ...
  [COMPILE_DEFINITIONS definition1 definition2 ...]
  [LINK_LIBRARIES library1 library2 ...]
  [LINK_DIRECTORIES directory1 directory2 ...]
  [INCLUDE_LIBRARIES target1 target2 ...]
  [INCLUDE_DIRECTORIES directory1 directory2 ...]
)

Parameters:

  • SOURCES (required): List of source files for the runtime library (include autogen files)
  • COMPILE_DEFINITIONS (optional): Additional preprocessor definitions beyond automatic ones
  • LINK_LIBRARIES (optional): Additional libraries to link beyond automatic dependencies
  • LINK_DIRECTORIES (optional): Additional library search directories
  • INCLUDE_LIBRARIES (optional): Target libraries whose include directories should be inherited
  • INCLUDE_DIRECTORIES (optional): Additional include directories beyond automatic ones

Automatic Behavior:

  • Creates target: ${NAMESPACE}_${CHIMOD_NAME}_runtime
  • Creates alias: ${NAMESPACE}::${CHIMOD_NAME}_runtime
  • Automatically defines CHIMAERA_RUNTIME=1 for runtime code
  • Automatically links core Chimaera library (chimaera::cxx or hermes_shm::cxx)
  • Automatically links rt library for POSIX real-time support
  • For non-admin ChiMods: automatically links both chimaera_admin_runtime and chimaera_admin_client
  • Automatically includes module headers from include/ directory
  • Links to client library if it exists
  • Installs library and headers with proper CMake export configuration

Example:

add_chimod_runtime(
  SOURCES 
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
  COMPILE_DEFINITIONS MY_MODULE_RUNTIME_DEBUG=1
  LINK_LIBRARIES libaio
  INCLUDE_DIRECTORIES ${LIBAIO_INCLUDE_DIR}
)

Configuration Requirements​

Before using these functions, ensure your ChiMod directory contains:

  1. chimaera_mod.yaml: Module configuration file defining the module name

    module_name: my_module

  2. Include structure: Headers organized as include/[namespace]/[module_name]/

  3. Source files: Client and runtime implementations with autogen files for runtime

Typical Usage Pattern​

Most ChiMods use both functions together:

# Create client library
add_chimod_client(
  SOURCES src/my_module_client.cc
)

# Create runtime library
add_chimod_runtime(
  SOURCES 
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
)

Function Dependencies: Both functions automatically handle common dependencies:

  • Core Library: Automatically links appropriate Chimaera core library
  • Runtime Libraries: add_chimod_runtime() automatically links rt library for async I/O operations
  • Admin ChiMod Integration: For non-admin chimods, both functions automatically link admin libraries and include admin headers
  • Client-Runtime Linking: Runtime automatically links to client library when both exist

This eliminates the need for manual dependency configuration in individual ChiMod CMakeLists.txt files.

Target Naming and Linking​

Target Format​

The system uses underscore-based target names for consistency with CMake conventions:

Target Names:

  • Runtime: ${NAMESPACE}_${CHIMOD_NAME}_runtime (e.g., chimaera_admin_runtime)
  • Client: ${NAMESPACE}_${CHIMOD_NAME}_client (e.g., chimaera_admin_client)

CMake Aliases (Recommended):

  • Runtime: ${NAMESPACE}::${CHIMOD_NAME}_runtime (e.g., chimaera::admin_runtime)
  • Client: ${NAMESPACE}::${CHIMOD_NAME}_client (e.g., chimaera::admin_client)

Package Names:

  • Format: ${NAMESPACE}_${CHIMOD_NAME} (e.g., chimaera_admin)
  • Used with: find_package(chimaera_admin REQUIRED)
  • Core package: chimaera (automatically included by find_package(chimaera))

External Application Linking (Based on test/unit/external/CMakeLists.txt):

# External applications typically only need ChiMod client libraries
# Core library dependencies are automatically included
find_package(chimaera REQUIRED)
find_package(chimaera_admin REQUIRED)

target_link_libraries(my_external_app
  chimaera::admin_client            # Admin client (includes all dependencies)
  ${CMAKE_THREAD_LIBS_INIT}         # Threading support
)
# Note: chimaera::cxx is automatically included by ChiMod client libraries

Internal Development Linking:

# For internal development within the Chimaera project
target_link_libraries(internal_app
  chimaera::admin_client            # ChiMod client
  chimaera::bdev_client             # BDev client 
  # Core dependencies are automatically linked by ChiMod libraries
)

Automatic Dependencies​

The ChiMod build functions automatically handle common dependencies:

For Runtime Code:

  • rt library: Automatically linked for POSIX real-time library support (async I/O operations)
  • Admin ChiMod: Automatically linked for all non-admin ChiMods (both runtime and client)
  • Admin includes: Automatically added to include directories for non-admin ChiMods

For All ChiMods:

  • Creates both client and runtime shared libraries
  • Sets proper include directories (include/, ${CMAKE_SOURCE_DIR}/include)
  • Automatically links core Chimaera dependencies
  • Sets required compile definitions (CHI_CHIMOD_NAME, CHI_NAMESPACE)
  • Configures proper build flags and settings

Simplified Development: ChiMod developers no longer need to manually specify:

  • rt library dependencies
  • Admin ChiMod dependencies (chimaera_admin_runtime, chimaera_admin_client)
  • Admin include directories
  • Core Chimaera library dependencies
  • Common linking patterns

Important Note for External Applications: External applications linking against ChiMod libraries receive all necessary dependencies automatically. The ChiMod client libraries include the core Chimaera library as a transitive dependency.

Automatic Installation: The ChiMod build functions automatically handle installation:

  • Installs libraries to the correct destination
  • Sets up proper runtime paths
  • Configures installation properties
  • Includes automatic dependencies in exported CMake packages
  • No separate install_chimod() call required

Targets Created by ChiMod Functions​

When you call add_chimod_client() and add_chimod_runtime() with CHIMOD_NAME YOUR_MODULE_NAME, they create the following CMake targets using the underscore-based naming format:

Target Naming System​

  • Actual Target Names: ${NAMESPACE}_${CHIMOD_NAME}_runtime and ${NAMESPACE}_${CHIMOD_NAME}_client
  • CMake Aliases: ${NAMESPACE}::${CHIMOD_NAME}_runtime and ${NAMESPACE}::${CHIMOD_NAME}_client (recommended)
  • Package Names: ${NAMESPACE}_${CHIMOD_NAME} (for find_package())

Runtime Target: ${NAMESPACE}_${CHIMOD_NAME}_runtime​

  • Target Name: chimaera_YOUR_MODULE_NAME_runtime (e.g., chimaera_admin_runtime, chimaera_MOD_NAME_runtime)
  • CMake Alias: chimaera::YOUR_MODULE_NAME_runtime (e.g., chimaera::admin_runtime) - recommended for linking
  • Type: Shared library (.so file)
  • Purpose: Contains server-side execution logic, runs in the Chimaera runtime process
  • Compile Definitions:
    • CHI_CHIMOD_NAME="${CHIMOD_NAME}" - Module name for runtime identification
    • CHI_NAMESPACE="${NAMESPACE}" - Project namespace
  • Include Directories:
    • include/ - Local module headers
    • ${CMAKE_SOURCE_DIR}/include - Chimaera framework headers
  • Dependencies: Links against chimaera library, rt library (automatic), admin dependencies (automatic)

Client Target: ${NAMESPACE}_${CHIMOD_NAME}_client​

  • Target Name: chimaera_YOUR_MODULE_NAME_client (e.g., chimaera_admin_client, chimaera_MOD_NAME_client)
  • CMake Alias: chimaera::YOUR_MODULE_NAME_client (e.g., chimaera::admin_client) - recommended for linking
  • Type: Shared library (.so file)
  • Purpose: Contains client-side API, runs in user processes
  • Compile Definitions:
    • CHI_CHIMOD_NAME="${CHIMOD_NAME}" - Module name for client identification
    • CHI_NAMESPACE="${NAMESPACE}" - Project namespace
  • Include Directories:
    • include/ - Local module headers
    • ${CMAKE_SOURCE_DIR}/include - Chimaera framework headers
  • Dependencies: Links against chimaera library, admin dependencies (automatic)

Namespace Configuration​

The namespace is automatically read from chimaera_repo.yaml files. The system searches up the directory tree from the CMakeLists.txt location to find the first chimaera_repo.yaml file:

Main project chimaera_repo.yaml:

namespace: chimaera  # Main project namespace

Module repository chimods/chimaera_repo.yaml:

namespace: chimods   # Modules get this namespace

This means modules in the chimods/ directory will use the "chimods" namespace, creating targets like chimods_admin_runtime, while other components use the main project namespace.

Example Output Files​

For a module named "admin" with namespace "chimods" (from chimods/chimaera_repo.yaml), the build produces:

build/bin/libchimods_admin_runtime.so    # Runtime library  
build/bin/libchimods_admin_client.so     # Client library

Using the Targets​

You can reference these targets in your CMakeLists.txt using the full target name:

# Add custom properties to the runtime target
set_target_properties(chimaera_${CHIMOD_NAME}_runtime PROPERTIES
  VERSION 1.0.0
  SOVERSION 1
)

# Add additional dependencies if needed
target_link_libraries(chimaera_${CHIMOD_NAME}_runtime PRIVATE some_external_lib)

# Or use the global property to get the actual target name
get_property(RUNTIME_TARGET GLOBAL PROPERTY ${CHIMOD_NAME}_RUNTIME_TARGET)
target_link_libraries(${RUNTIME_TARGET} PRIVATE some_external_lib)

Module Configuration (chimaera_mod.yaml)​

name: MOD_NAME
version: 1.0.0
description: "Module description"
author: "Author Name"
methods:
  - kCreate
  - kCustom
dependencies: []

Auto-Generated Method Files​

Each module requires an auto-generated methods file at include/[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h. This file must:

  1. Include chimaera.h: Required for GLOBAL_CONST macro
  2. Use namespace constants: Define methods as GLOBAL_CONST chi::u32 values
  3. Follow naming convention: Method names should start with k (e.g., kCreate, kCustom)

Required Template:

#ifndef MOD_NAME_AUTOGEN_METHODS_H_
#define MOD_NAME_AUTOGEN_METHODS_H_

#include <chimaera/chimaera.h>

namespace chimaera::MOD_NAME {

namespace Method {
  // Standard inherited methods (always include these)
  GLOBAL_CONST chi::u32 kCreate = 0;
  GLOBAL_CONST chi::u32 kDestroy = 1;
  GLOBAL_CONST chi::u32 kNodeFailure = 2;
  GLOBAL_CONST chi::u32 kRecover = 3;
  GLOBAL_CONST chi::u32 kMigrate = 4;
  GLOBAL_CONST chi::u32 kUpgrade = 5;
  
  // Module-specific methods (customize these)
  GLOBAL_CONST chi::u32 kCustom = 10;
  // Add more module-specific methods starting from 10+
}

} // namespace chimaera::MOD_NAME

#endif // MOD_NAME_AUTOGEN_METHODS_H_

Important Notes:

  • GLOBAL_CONST is required: Do not use const or constexpr - use GLOBAL_CONST
  • Include chimaera.h: This header defines the GLOBAL_CONST macro
  • Standard methods 0-5: Always include the inherited methods (kCreate through kUpgrade)
  • Custom methods 10+: Start custom methods from ID 10 to avoid conflicts
  • No static casting needed: Use method values directly (e.g., method_ = Method::kCreate;)

Runtime Entry Points​

Use the CHI_TASK_CC macro to define module entry points:

// At the end of your runtime source file (_runtime.cc)
CHI_TASK_CC(your_namespace::YourContainerClass)

This macro automatically generates all required extern "C" functions and gets the module name from YourContainerClass::CreateParams::chimod_lib_name:

  • alloc_chimod() - Creates container instance
  • new_chimod() - Creates and initializes container
  • get_chimod_name() - Returns module name
  • destroy_chimod() - Destroys container instance

Requirements for CHI_TASK_CC to work:

  1. Your runtime class must define a public typedef: using CreateParams = your_namespace::CreateParams;
  2. Your CreateParams struct must have: static constexpr const char* chimod_lib_name = "your_module_name";

IMPORTANT: The chimod_lib_name should NOT include the _runtime suffix. The module manager automatically appends _runtime when loading the library. For example, use "chimaera_mymodule" not "chimaera_mymodule_runtime".

Example:

namespace chimaera::your_module {

struct CreateParams {
  static constexpr const char* chimod_lib_name = "chimaera_your_module";
  // ... other parameters
};

class Runtime : public chi::Container {
public:
  using CreateParams = chimaera::your_module::CreateParams;  // Required for CHI_TASK_CC
  // ... rest of class
};

}  // namespace chimaera::your_module
  • is_chimaera_chimod_ - Module identification flag

Auto-Generated Code Pattern​

Overview​

ChiMods use auto-generated source files to implement the Container virtual APIs (Run, Monitor, Del, SaveIn, LoadIn, SaveOut, LoadOut, NewCopy). This approach provides consistent dispatch logic and reduces boilerplate code.

New Pattern: Auto-Generated Source Files​

Instead of using inline functions in headers, ChiMods now use auto-generated .cc source files that implement the virtual methods directly. This pattern:

  • Eliminates inline dispatchers: Virtual methods are implemented directly in auto-generated source
  • Reduces header dependencies: No need to include autogen headers in runtime files
  • Improves compilation: Source files compile once, not in every including file
  • Maintains consistency: All ChiMods use the same dispatch pattern

File Structure​

src/
└── autogen/
    └── MOD_NAME_lib_exec.cc    # Auto-generated virtual method implementations

The auto-generated source file contains:

  • Container virtual method implementations (Run, Del, etc.)
  • Switch-case dispatch based on method IDs
  • Proper task type casting and method invocation
  • IPC manager integration for task lifecycle management

Auto-Generated Source Template​

/**
 * Auto-generated execution implementation for MOD_NAME ChiMod
 * Implements Container virtual APIs directly using switch-case dispatch
 * 
 * This file is autogenerated - do not edit manually.
 * Changes should be made to the autogen tool or the YAML configuration.
 */

#include <[namespace]/MOD_NAME/MOD_NAME_runtime.h>
#include <[namespace]/MOD_NAME/autogen/MOD_NAME_methods.h>
#include <chimaera/chimaera.h>

namespace chimaera::MOD_NAME {

//==============================================================================
// Runtime Virtual API Implementations
//==============================================================================

void Runtime::Run(chi::u32 method, hipc::FullPtr<chi::Task> task_ptr, chi::RunContext& rctx) {
  switch (method) {
    case Method::kCreate: {
      Create(task_ptr.Cast<CreateTask>(), rctx);
      break;
    }
    case Method::kDestroy: {
      Destroy(task_ptr.Cast<DestroyTask>(), rctx);
      break;
    }
    case Method::kCustom: {
      Custom(task_ptr.Cast<CustomTask>(), rctx);
      break;
    }
    default: {
      // Unknown method - do nothing
      break;
    }
  }
}

void Runtime::Del(chi::u32 method, hipc::FullPtr<chi::Task> task_ptr) {
  // Use IPC manager to deallocate task from shared memory
  auto* ipc_manager = CHI_IPC;
  
  switch (method) {
    case Method::kCreate: {
      ipc_manager->DelTask(task_ptr.Cast<CreateTask>());
      break;
    }
    case Method::kDestroy: {
      ipc_manager->DelTask(task_ptr.Cast<DestroyTask>());
      break;
    }
    case Method::kCustom: {
      ipc_manager->DelTask(task_ptr.Cast<CustomTask>());
      break;
    }
    default: {
      // For unknown methods, still try to delete from main segment
      ipc_manager->DelTask(task_ptr);
      break;
    }
  }
}

// SaveIn, LoadIn, SaveOut, LoadOut, and NewCopy follow similar patterns...

} // namespace chimaera::MOD_NAME

Runtime Implementation Changes​

With the new autogen pattern, runtime source files (MOD_NAME_runtime.cc) no longer include autogen headers or implement dispatcher methods:

Before (Old Pattern):​

// No autogen header includes needed with new pattern

void Runtime::Run(chi::u32 method, hipc::FullPtr<chi::Task> task_ptr, chi::RunContext& rctx) {
  // Dispatch to the appropriate method handler
  chimaera::MOD_NAME::Run(this, method, task_ptr, rctx);
}

// Similar dispatcher implementations for Del, SaveIn, LoadIn, SaveOut, LoadOut, NewCopy...

After (New Pattern):​

// No autogen header includes needed
// No dispatcher method implementations needed

// Virtual method implementations are now in src/autogen/MOD_NAME_lib_exec.cc
// Runtime source focuses only on business logic methods like Create(), Custom(), etc.

CMake Integration​

The auto-generated source file must be included in the RUNTIME_SOURCES:

add_chimod_client(
  CHIMOD_NAME MOD_NAME
  SOURCES src/MOD_NAME_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME MOD_NAME
  SOURCES src/MOD_NAME_runtime.cc src/autogen/MOD_NAME_lib_exec.cc
)

Benefits of the New Pattern​

  1. Cleaner Runtime Code: Runtime implementations focus on business logic, not dispatching
  2. Better Compilation: Source files compile once instead of being inlined in every header include
  3. Consistent Pattern: All ChiMods use identical dispatch logic
  4. Header Simplification: No need to include complex autogen headers
  5. Better IDE Support: Proper source files work better with IDEs than inline templates

Migration Guide​

To migrate from the old inline header pattern to the new source pattern:

  1. Create autogen source directory: mkdir -p src/autogen/
  2. Generate new autogen source: Create src/autogen/MOD_NAME_lib_exec.cc with virtual method implementations
  3. Remove autogen header includes: Delete #include <[namespace]/MOD_NAME/autogen/MOD_NAME_lib_exec.h> from runtime source (replaced by .cc files)
  4. Remove dispatcher methods: Delete all virtual method implementations from runtime source (Run, Del, etc.)
  5. Update CMakeLists.txt: Add autogen source to RUNTIME_SOURCES
  6. Keep business logic: Retain the actual task processing methods (Create, Custom, etc.)

Important Notes​

  • Auto-generated files: These files should be generated by tools, not hand-written
  • Do not edit: Manual changes to autogen files will be lost when regenerated
  • Template consistency: All ChiMods should follow the same autogen template pattern
  • Build integration: Autogen source files must be included in CMake build

External ChiMod Development​

When developing ChiMods in external repositories (outside the main Chimaera project), you need to link against the installed Chimaera libraries and use the CMake package discovery system.

Prerequisites​

Before developing external ChiMods, ensure Chimaera is properly installed:

# Configure and build Chimaera
cmake --preset debug
cmake --build build

# Install Chimaera libraries and CMake configs  
cmake --install build --prefix /usr/local

This installs:

  • Core Chimaera library (libcxx.so)
  • ChiMod libraries (libchimaera_admin_runtime.so, libchimaera_admin_client.so, etc.)
  • CMake package configuration files for external discovery
  • Header files for development

External ChiMod Repository Structure​

Your external ChiMod repository should follow this structure:

my_external_chimod/
├── chimaera_repo.yaml          # Repository namespace configuration
├── CMakeLists.txt              # Root CMake configuration
├── modules/                    # ChiMod modules directory (name is flexible)
│   └── my_module/
│       ├── chimaera_mod.yaml   # Module configuration
│       ├── CMakeLists.txt      # Module build configuration  
│       ├── include/
│       │   └── [namespace]/
│       │       └── my_module/
│       │           ├── my_module_client.h
│       │           ├── my_module_runtime.h
│       │           ├── my_module_tasks.h
│       │           └── autogen/
│       │               └── my_module_methods.h
│       └── src/
│           ├── my_module_client.cc
│           ├── my_module_runtime.cc
│           └── autogen/
│               └── my_module_lib_exec.cc

Note: The directory name for modules (shown here as modules/) is flexible. You can use chimods/, components/, plugins/, or any other name that fits your project structure. The directory name doesn't need to match the namespace.

Repository Configuration (chimaera_repo.yaml)​

Create a chimaera_repo.yaml file in your repository root to define the namespace:

# Repository-level configuration
namespace: myproject      # Your custom namespace (replaces "chimaera")

This namespace will be used for:

  • CMake target names: myproject_my_module_runtime, myproject_my_module_client
  • Library file names: libmyproject_my_module_runtime.so, libmyproject_my_module_client.so
  • C++ namespaces: myproject::my_module

Root CMakeLists.txt​

Your repository's root CMakeLists.txt must find and link to the installed Chimaera packages:

cmake_minimum_required(VERSION 3.20)
project(my_external_chimod)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Find required Chimaera packages
# These packages are installed by 'cmake --install build --prefix /usr/local'
find_package(chimaera REQUIRED)              # Core Chimaera (automatically includes ChimaeraCommon.cmake)
find_package(chimaera_admin REQUIRED)        # Admin ChiMod (often required)

# Set CMAKE_PREFIX_PATH if Chimaera is installed in a custom location
# set(CMAKE_PREFIX_PATH "/path/to/[namespace]/install" ${CMAKE_PREFIX_PATH})

# ChimaeraCommon.cmake utilities are automatically included by find_package(chimaera)
# This provides add_chimod_client(), add_chimod_runtime(), and other build functions

# Add subdirectories containing your ChiMods
add_subdirectory(modules/my_module)  # Use your actual directory name

ChiMod CMakeLists.txt​

Each ChiMod's CMakeLists.txt uses the standard Chimaera build utilities:

cmake_minimum_required(VERSION 3.20)

# Create both client and runtime libraries using standard Chimaera utilities
# These functions are provided by ChimaeraCommon.cmake (automatically included via find_package(chimaera))
# Creates targets: my_namespace_my_module_client, my_namespace_my_module_runtime
# Creates aliases: my_namespace::my_module_client, my_namespace::my_module_runtime
add_chimod_client(
  CHIMOD_NAME my_module
  SOURCES src/my_module_client.cc
)
add_chimod_runtime(
  CHIMOD_NAME my_module
  SOURCES
    src/my_module_runtime.cc 
    src/autogen/my_module_lib_exec.cc
)

# Installation is automatic - no separate install_chimod() call required
# Package name: my_namespace_my_module (for find_package)

# Optional: Add additional dependencies if your ChiMod needs external libraries
# get_property(RUNTIME_TARGET GLOBAL PROPERTY my_module_RUNTIME_TARGET)
# get_property(CLIENT_TARGET GLOBAL PROPERTY my_module_CLIENT_TARGET)
# target_link_libraries(${RUNTIME_TARGET} PRIVATE some_external_lib)
# target_link_libraries(${CLIENT_TARGET} PRIVATE some_external_lib)

External Applications Using Your ChiMod​

Once installed, external applications can find and link to your ChiMod. Based on our external unit test patterns (see test/unit/external-chimod/CMakeLists.txt):

# External application CMakeLists.txt
find_package(my_namespace_my_module REQUIRED)  # Your ChiMod package
find_package(chimaera REQUIRED)                # Core Chimaera (automatically includes utilities)
find_package(chimaera_admin REQUIRED)          # Admin ChiMod (often required)

# Simple linking pattern - ChiMod libraries include all dependencies
target_link_libraries(my_external_app
  my_namespace::my_module_client    # Your ChiMod client
  chimaera::admin_client            # Admin client (if needed)
  ${CMAKE_THREAD_LIBS_INIT}         # Threading support
)
# Core Chimaera library is automatically included by ChiMod dependencies

External ChiMod Implementation​

Your external ChiMod implementation follows the same patterns as internal ChiMods:

CreateParams Configuration​

In your my_module_tasks.h, the CreateParams must reference your custom namespace:

struct CreateParams {
  // Your module-specific parameters
  std::string config_data_;
  chi::u32 worker_count_;
  
  // CRITICAL: Library name must match your namespace
  static constexpr const char* chimod_lib_name = "myproject_my_module";
  
  // Constructors and serialization...
};

C++ Namespace​

Use your custom namespace throughout your implementation:

// In all header and source files
namespace myproject::my_module {

// Your ChiMod implementation...
class Runtime : public chi::Container {
  // Implementation...
};

class Client : public chi::ContainerClient {  
  // Implementation...
};

} // namespace myproject::my_module

Building External ChiMods​

# Configure your external ChiMod project
mkdir build && cd build
cmake ..

# Build your ChiMods
make

# Optional: Install your ChiMods
make install

The build system will automatically:

  • Link all necessary core Chimaera dependencies
  • Link against chimaera::admin_client and chimaera::admin_runtime (for non-admin modules)
  • Generate libraries with your custom namespace: libmyproject_my_module_runtime.so
  • Configure proper include paths and dependencies

Usage in Applications​

Applications using your external ChiMod would reference it as:

#include <chimaera/chimaera.h>
#include <[namespace]/my_module/my_module_client.h>
#include <[namespace]/admin/admin_client.h>

int main() {
  // Initialize Chimaera client
  chi::CHIMAERA_CLIENT_INIT();
  
  // Create your ChiMod client
  const chi::PoolId pool_id = chi::PoolId(7000, 0);
  myproject::my_module::Client client(pool_id);
  
  // Use your ChiMod
  auto pool_query = chi::PoolQuery::Local();
  client.Create(HSHM_MCTX, pool_query);
}

CHIMAERA_RUNTIME_INIT for Testing and Benchmarks​

For simple unit tests and benchmarks, Chimaera provides CHIMAERA_RUNTIME_INIT() as a convenience function that initializes both the client and runtime in a single process. This is an alternative to using CHIMAERA_CLIENT_INIT() when you need both components initialized together.

Important Notes:

  • Primary Use Case: Unit tests and benchmarks only
  • Not for Production: Should NOT be used in main production applications
  • Single Process: Initializes both client and runtime in the same process
  • Simplified Testing: Eliminates need for separate runtime and client processes during testing

Usage Example (Unit Tests/Benchmarks):

#include <chimaera/chimaera.h>
#include <[namespace]/my_module/my_module_client.h>

TEST(MyModuleTest, BasicOperation) {
  // Initialize both client and runtime in single process
  chi::CHIMAERA_RUNTIME_INIT();

  // Create your ChiMod client
  const chi::PoolId pool_id = chi::PoolId(7000, 0);
  myproject::my_module::Client client(pool_id);

  // Test your ChiMod functionality
  auto pool_query = chi::PoolQuery::Local();
  client.Create(HSHM_MCTX, pool_query, "test_pool");

  // Assertions and test logic...
}

When to Use Each:

  • CHIMAERA_CLIENT_INIT(): Production applications connecting to existing runtime
  • CHIMAERA_RUNTIME_INIT(): Unit tests, benchmarks, and simple testing scenarios

Dependencies and Installation Paths​

External ChiMod development requires these components to be installed:

  1. Core Package: chimaera (includes main library and ChimaeraCommon.cmake utilities)
  2. Admin ChiMod: chimaera::admin_client and chimaera::admin_runtime (required for most modules)
  3. CMake Configs: Package discovery files (automatically installed with packages)
  4. Headers: All Chimaera framework headers (installed with packages)

All build utilities (add_chimod_client(), add_chimod_runtime()) are automatically available via find_package(chimaera).

If Chimaera is installed in a custom location, set CMAKE_PREFIX_PATH:

export CMAKE_PREFIX_PATH="/path/to/[namespace]/install:$CMAKE_PREFIX_PATH"

Common External Development Issues​

ChimaeraCommon.cmake Not Found:

  • Ensure Chimaera was installed with cmake --install build --prefix <path>
  • Verify CMAKE_PREFIX_PATH includes the Chimaera installation directory
  • Check that find_package(chimaera REQUIRED) succeeded (ChimaeraCommon.cmake is included automatically)

Library Name Mismatch:

  • Ensure CreateParams::chimod_lib_name exactly matches your namespace and module name
  • For namespace "myproject" and module "my_module": chimod_lib_name = "myproject_my_module"
  • The system automatically appends "_runtime" to find the runtime library
  • Target names use format: myproject_my_module_runtime and myproject_my_module_client

Missing Dependencies:

  • The ChiMod build functions automatically link admin and rt library dependencies
  • Ensure all external dependencies (Boost, MPI, etc.) are available in your build environment
  • Use the same dependency versions that Chimaera was built with
  • For runtime code, rt library is automatically included for async I/O support

External ChiMod Checklist​

  • Repository Configuration: chimaera_repo.yaml with custom namespace
  • CMake Setup: Root CMakeLists.txt finds chimaera package
  • ChiMod Configuration: chimaera_mod.yaml with method definitions
  • Library Name: CreateParams::chimod_lib_name matches namespace pattern
  • C++ Namespace: All code uses custom namespace consistently
  • Build Integration: ChiMod CMakeLists.txt uses add_chimod_client() and add_chimod_runtime() (installation is automatic)
  • Dependencies: All required external libraries available at build time
  • Automatic Linking: Rely on ChiMod build functions for rt and admin dependencies

Example Module​

See the chimods/MOD_NAME directory for a complete working example that demonstrates:

  • Task definition with proper constructors
  • Client API with sync/async methods
  • Runtime container with execution logic
  • Build system integration
  • YAML configuration

Creating a New Module​

  1. Copy the MOD_NAME template directory
  2. Rename all MOD_NAME occurrences to your module name
  3. Update the chimaera_mod.yaml configuration
  4. Define your tasks in the _tasks.h file
  5. Implement client API in _client.h/cc
  6. Implement runtime logic in _runtime.h/cc
  7. Add CHI_TASK_CC(YourContainerClass) at the end of runtime source
  8. Add to the build system
  9. Test with client and runtime

Recent Changes and Best Practices​

Container Initialization Pattern​

Starting with the latest version, container initialization has been simplified:

  1. No Separate Init Method: The Init method has been merged with Create
  2. Create Does Everything: The Create method now handles both container creation and initialization
  3. Access to Task Data: Since Create receives the CreateTask, you have access to pool_id and pool_query from the task

Framework-Managed Task Cleanup​

Task cleanup is handled by the framework using the IPC manager:

  1. No Custom Del Methods Required: Individual DelTaskType methods are no longer needed
  2. IPC Manager Handles Cleanup: The framework automatically calls ipc_manager->DelTask() to deallocate tasks from shared memory
  3. Memory Segment Deallocation: Tasks are properly removed from their respective memory segments (typically kMainSegment)

Simplified ChiMod Entry Points​

ChiMod entry points are now hidden behind the CHI_TASK_CC macro:

  1. Single Macro Call: Replace complex extern "C" blocks with one macro
  2. Automatic Container Integration: Works seamlessly with chi::Container base class
  3. Cleaner Module Code: Eliminates boilerplate entry point code
// Old approach (complex extern "C" block)
extern "C" {
  chi::ChiContainer* alloc_chimod() { /* ... */ }
  chi::ChiContainer* new_chimod(/*...*/) { /* ... */ }
  const char* get_chimod_name() { /* ... */ }
  void destroy_chimod(/*...*/) { /* ... */ }
  bool is_chimaera_chimod_ = true;
}

// New approach (simple macro)
CHI_TASK_CC(chimaera::MOD_NAME::Runtime)
void Create(hipc::FullPtr<CreateTask> task, chi::RunContext& ctx) {
  // Container is already initialized via Init() before Create is called
  // Do NOT call Init() here

  // Container-specific initialization logic
  // All tasks will be routed through the external queue lanes
  // which are automatically mapped to workers at runtime startup

  // Container is now ready for operation
}

FullPtr Parameter Pattern​

All runtime methods now use hipc::FullPtr<TaskType> instead of raw pointers:

// Old pattern (deprecated)
void Custom(CustomTask* task, chi::RunContext& ctx) { ... }

// New pattern (current)
void Custom(hipc::FullPtr<CustomTask> task, chi::RunContext& ctx) { ... }

Benefits of FullPtr:

  • Shared Memory Safety: Provides safe access across process boundaries
  • Automatic Dereferencing: Use task->field just like raw pointers
  • Memory Management: Framework handles allocation/deallocation
  • Null Checking: Use task.IsNull() to check validity

Migration Guide​

When updating existing modules:

  1. Remove Init Override: Delete custom Init method implementations
  2. Update Create Method: Move initialization logic from Init to Create
  3. Change Method Signatures: Replace TaskType* with hipc::FullPtr<TaskType>
  4. Update Monitor Methods: Ensure all monitoring methods use FullPtr
  5. Implement kLocalSchedule: Every Monitor method MUST implement kLocalSchedule mode
  6. Remove Del Methods: Delete all DelTaskType methods - framework calls ipc_manager->DelTask() automatically
  7. Update Autogen Files: Ensure Del dispatcher calls ipc_manager->DelTask() instead of custom Del methods
  8. Replace Entry Points: Replace extern "C" blocks with CHI_TASK_CC(ClassName) macro
  9. Remove Completion Calls: Framework handles task completion automatically

Custom Namespace Configuration​

Overview​

While the default namespace is chimaera, you can customize the namespace for your ChiMod modules. This is useful for:

  • Project Branding: Use your own project or company namespace
  • Avoiding Conflicts: Prevent naming conflicts with other ChiMod collections
  • Module Organization: Group related modules under a custom namespace

Configuring Custom Namespace​

The namespace is controlled by the chimaera_repo.yaml file in your project root:

namespace: your_custom_namespace

For example:

namespace: mycompany

Required Changes for Custom Namespace​

When using a custom namespace, you must update several components:

1. CreateParams chimod_lib_name​

The most critical change is updating the chimod_lib_name in your CreateParams:

// Default chimaera namespace
struct CreateParams {
  static constexpr const char* chimod_lib_name = "chimaera_your_module";
};

// Custom namespace example
struct CreateParams {
  static constexpr const char* chimod_lib_name = "mycompany_your_module";
};

2. Module Namespace Declaration​

Update your module's C++ namespace:

// Default
namespace chimaera::your_module {
  // module code
}

// Custom
namespace mycompany::your_module {
  // module code
}

3. CMake Library Names​

The CMake system automatically uses your custom namespace. Libraries will be named:

  • Default: libchimaera_module_runtime.so, libchimaera_module_client.so
  • Custom: libmycompany_module_runtime.so, libmycompany_module_client.so

4. Runtime Integration​

If your runtime code references the admin module or other system modules, update the references:

// Default admin module reference
auto* admin_chimod = module_manager->GetChiMod("chimaera_admin");

// Custom namespace admin module
auto* admin_chimod = module_manager->GetChiMod("mycompany_admin");

Checklist for Custom Namespace​

  • Update chimaera_repo.yaml with your custom namespace
  • Update CreateParams::chimod_lib_name to use custom namespace prefix
  • Update C++ namespace declarations in all module files
  • Update runtime references to admin module and other system modules
  • Update any hardcoded module names in configuration or startup code
  • Rebuild all modules after namespace changes
  • Update library search paths if needed for deployment

Example: Complete Custom Namespace Module​

# chimaera_repo.yaml
namespace: mycompany
// mymodule_tasks.h
namespace mycompany::mymodule {

struct CreateParams {
  static constexpr const char* chimod_lib_name = "mycompany_mymodule";
  // ... other parameters
};

using CreateTask = chimaera::admin::BaseCreateTask<CreateParams, Method::kCreate>;

}  // namespace mycompany::mymodule
// mymodule_runtime.h
namespace mycompany::mymodule {

class Runtime : public chi::Container {
public:
  using CreateParams = mycompany::mymodule::CreateParams;  // Required for CHI_TASK_CC
  // ... rest of class
};

}  // namespace mycompany::mymodule
// mymodule_runtime.cc
CHI_TASK_CC(mycompany::mymodule::Runtime)

Important Notes​

  • Library Name Consistency: The chimod_lib_name must exactly match what the CMake system generates
  • Admin Module: If you customize the namespace, you may also want to rebuild the admin module with your custom namespace
  • Backward Compatibility: Changing namespace breaks compatibility with existing deployments using default namespace
  • Documentation: Update any module-specific documentation to reflect the new namespace

Advanced Topics​

Task Scheduling​

Tasks can be scheduled with different priorities:

  • kLowLatency: For time-critical operations
  • kHighLatency: For batch processing

Automatic Routing Architecture​

The framework handles all task routing automatically:

  1. Client-Side Enqueuing:

    • Tasks are enqueued via IpcManager::Enqueue() from client code
    • Lane selection uses PID+TID hash for automatic distribution across lanes
    • Formula: lane_id = hash(PID, TID) % num_lanes

  2. Worker-Lane Mapping (1:1 Direct Mapping):

    • Number of lanes automatically equals number of sched workers (default: 8)
    • Each worker assigned exactly one lane: worker i → lane i
    • No round-robin needed - perfect 1:1 correspondence
    • Lane headers track assigned worker ID

  3. No Configuration Required:

    • Lane count automatically matches sched worker count from config
    • No separate task_queue_lanes configuration needed
    • Change worker count → lane count adjusts automatically

Example: With 8 sched workers (default):

  • 8 lanes created automatically in external queue
  • Worker 0 → Lane 0, Worker 1 → Lane 1, ..., Worker 7 → Lane 7
  • Client tasks distributed via hash to lanes 0-7
  • Each worker processes tasks from its dedicated lane

Error Handling​

void Custom(hipc::FullPtr<CustomTask> task, chi::RunContext& ctx) {
  try {
    // Operation logic
    task->result_code_ = 0;
  } catch (const std::exception& e) {
    task->result_code_ = 1;
    task->data_ = chi::string(main_allocator_, e.what());
  }
  // Framework handles task completion automatically
}

Debugging Tips​

  1. Check Shared Memory: Use ipcs -m to view segments
  2. Verify Task State: Check task completion status
  3. Monitor Queue Depth: Use GetProcessQueue() to inspect queues
  4. Enable Debug Logging: Set CHI_DEBUG environment variable
  5. Use GDB: Attach to runtime process for debugging

Common Issues and Solutions​

Tasks Not Being Executed:

  • Cause: Tasks not being routed to worker queues
  • Solution: Verify task pool_query is set correctly and pool exists
  • Debug: Add logging in task execution methods to verify they're being called

Queue Overflow or Deadlocks:

  • Cause: Tasks being enqueued but not dequeued from lanes
  • Solution: Verify lane creation in Create() method and proper task routing
  • Debug: Check lane sizes with lane->Size() and lane->IsEmpty()

Memory Leaks in Shared Memory:

  • Cause: Tasks not being properly cleaned up
  • Solution: Ensure framework Del dispatcher is working correctly
  • Debug: Monitor shared memory usage with ipcs -m

Performance Considerations​

  1. Minimize Allocations: Reuse buffers when possible
  2. Batch Operations: Submit multiple tasks together
  3. Use Appropriate Segments: Put large data in client_data_segment
  4. Avoid Blocking: Use async operations when possible
  5. Profile First: Measure before optimizing

Unit Testing​

Unit testing for ChiMods can be implemented following these key principles:

  • Set up test environment with proper configuration
  • Configure environment variables for module discovery
  • Integrate with test framework patterns
  • Create test examples with fixtures
  • Configure CMake for test builds
  • Follow best practices for ChiMod testing

Test both runtime and client components in the same process for comprehensive integration testing without complex multi-process coordination.

Quick Reference Checklist​

When creating a new Chimaera module, ensure you have:

Task Definition Checklist (_tasks.h)​

  • Tasks inherit from chi::Task or use GetOrCreatePoolTask template (recommended for non-admin modules)
  • Use GetOrCreatePoolTask: For non-admin modules instead of BaseCreateTask directly
  • Use BaseCreateTask with IS_ADMIN=true: Only for admin module
  • SHM constructor with CtxAllocator parameter (if custom task)
  • Emplace constructor with all required parameters (if custom task)
  • Uses HSHM serializable types (chi::string, chi::vector, etc.)
  • Method constant assigned in constructor (e.g., method_ = Method::kCreate;)
  • No static casting: Use Method namespace constants directly
  • Include auto-generated methods file for Method constants

Runtime Container Checklist (_runtime.h/cc)​

  • Inherits from chi::Container
  • Init() method overridden - calls base class Init() then initializes client for this ChiMod
  • Create() method does NOT call chi::Container::Init() (container is already initialized before Create is called)
  • All task methods use hipc::FullPtr<TaskType> parameters
  • NO custom Del methods needed - framework calls ipc_manager->DelTask() automatically
  • Uses CHI_TASK_CC(ClassName) macro for entry points
  • Routing is automatic - tasks are routed through external queue lanes mapped to workers (1:1 worker-to-lane mapping)

Client API Checklist (_client.h/cc)​

  • Inherits from chi::ContainerClient
  • Uses CHI_IPC->NewTask<TaskType>() for allocation
  • Uses CHI_IPC->Enqueue() for task submission
  • Uses CHI_IPC->DelTask() for cleanup
  • Provides both sync and async methods
  • CRITICAL: Create methods update pool_id_ = task->new_pool_id_ after task completion

Build System Checklist​

  • CMakeLists.txt creates both client and runtime libraries
  • chimaera_mod.yaml defines module metadata
  • Auto-generated methods file: autogen/MOD_NAME_methods.h with Method namespace
  • Include chimaera.h: In methods file for GLOBAL_CONST macro
  • GLOBAL_CONST constants: Use namespace constants, not enum class
  • Proper install targets configured
  • Links against chimaera library

Common Pitfalls to Avoid​

  • ❌ CRITICAL: Not updating pool_id_ in Create methods (leads to incorrect pool ID for subsequent operations)
  • ❌ Using raw pointers instead of FullPtr in runtime methods
  • ❌ Calling chi::Container::Init() in Create method (container is already initialized by framework before Create is called)
  • ❌ Not overriding Init() method (required to initialize the client member)
  • ❌ Using non-HSHM types in task data members
  • ❌ Implementing custom Del methods (framework calls ipc_manager->DelTask() automatically)
  • ❌ Writing complex extern "C" blocks (use CHI_TASK_CC macro instead)
  • ❌ Using static_cast with Method values (use Method::kName directly)
  • ❌ Attempting to manually manage task routing (framework handles automatically)
  • ❌ Missing chimaera.h include in methods file (GLOBAL_CONST won't work)
  • ❌ Using enum class for methods (use namespace with GLOBAL_CONST instead)
  • ❌ Using BaseCreateTask directly for non-admin modules (use GetOrCreatePoolTask instead)
  • ❌ Forgetting GetOrCreatePoolTask template for container creation (reduces boilerplate)

Pool Name Requirements​

CRITICAL: All ChiMod Create functions MUST require a user-provided pool_name parameter. Never auto-generate pool names using pool_id_ during Create operations.

Why Pool Names Are Required​

  1. pool_id_ Not Available: pool_id_ is not set until after Create completes
  2. User Intent: Users should explicitly name their pools for better organization
  3. Uniqueness: Users can ensure uniqueness better than auto-generation
  4. Debugging: Named pools are easier to identify during debugging

Pool Naming Guidelines​

  • Descriptive Names: Use names that identify purpose or content
  • File-based Devices: For BDev file devices, pool_name serves as the file path
  • RAM-based Devices: For BDev RAM devices, pool_name should be unique identifier
  • Unique Identifiers: Consider timestamp + PID combinations when needed

Correct Pool Naming Usage​

// BDev file-based device - pool_name is the file path
std::string file_path = "/path/to/device.dat";
const chi::PoolId bdev_pool_id(7000, 0);
bdev_client.Create(mctx, pool_query, file_path, bdev_pool_id,
                   chimaera::bdev::BdevType::kFile);

// BDev RAM-based device - pool_name is unique identifier
std::string pool_name = "my_ram_device_" + std::to_string(timestamp);
const chi::PoolId ram_pool_id(7001, 0);
bdev_client.Create(mctx, pool_query, pool_name, ram_pool_id,
                   chimaera::bdev::BdevType::kRam, ram_size);

// Other ChiMods - pool_name is descriptive identifier
std::string pool_name = "my_container_" + user_identifier;
const chi::PoolId mod_pool_id(7002, 0);
mod_client.Create(mctx, pool_query, pool_name, mod_pool_id);

Incorrect Pool Naming Usage​

// WRONG: Using pool_id_ before it's set (will be 0 or garbage)
std::string bad_name = "pool_" + std::to_string(pool_id_.ToU64());

// WRONG: Using empty strings
client.Create(mctx, pool_query, "");

// WRONG: Auto-generating inside Create function
// Create functions should not auto-generate names
void Create(mctx, pool_query) {
    std::string auto_name = "pool_" + generate_id();  // Wrong approach
}

Client Interface Pattern​

All ChiMod clients should follow this interface pattern:

class Client : public chi::ContainerClient {
 public:
  // Synchronous Create with required pool_name
  void Create(const hipc::MemContext& mctx, 
              const chi::PoolQuery& pool_query,
              const std::string& pool_name /* user-provided name */) {
    auto task = AsyncCreate(mctx, pool_query, pool_name);
    task->Wait();
    pool_id_ = task->new_pool_id_;  // Set AFTER Create completes
    // ... cleanup
  }
  
  // Asynchronous Create with required pool_name
  hipc::FullPtr<CreateTask> AsyncCreate(
      const hipc::MemContext& mctx,
      const chi::PoolQuery& pool_query,
      const std::string& pool_name /* user-provided name */) {
    // Use pool_name directly, never generate internally
    auto task = ipc_manager->NewTask<CreateTask>(
        chi::CreateTaskId(),
        chi::kAdminPoolId,  // Always use admin pool
        pool_query,
        CreateParams::chimod_lib_name,  // Never hardcode
        pool_name,  // User-provided name
        pool_id_    // Target pool ID (unset during Create)
    );
    return task;
  }
};

BDev-Specific Requirements​

  • Single Interface: Use only one Create() and AsyncCreate() method (no multiple overloads)
  • File Devices: pool_name parameter serves as the file path
  • RAM Devices: pool_name parameter serves as unique identifier
  • Method Signature: Create(mctx, pool_query, pool_name, bdev_type, total_size=0, io_depth=32, alignment=4096)

Compose Configuration Feature​

The compose feature allows automatic pool creation from YAML configuration files. This enables declarative infrastructure setup where all required pools can be defined in configuration and created during runtime initialization or via utility script.

CreateParams LoadConfig Requirement​

CRITICAL: All ChiMod CreateParams structures MUST implement a LoadConfig() method to support compose feature.

/**
 * Load configuration from PoolConfig (for compose mode)
 * Required for compose feature support
 * @param pool_config Pool configuration from compose section
 */
void LoadConfig(const chi::PoolConfig& pool_config) {
  // Parse YAML config string
  YAML::Node config = YAML::Load(pool_config.config_);

  // Load module-specific parameters from YAML
  if (config["parameter_name"]) {
    parameter_name_ = config["parameter_name"].as<Type>();
  }

  // Parse size strings (e.g., "2GB", "512MB")
  if (config["capacity"]) {
    std::string capacity_str = config["capacity"].as<std::string>();
    total_size_ = hshm::ConfigParse::ParseSize(capacity_str);
  }
}

Compose Configuration Format​

compose:
- mod_name: chimaera_bdev        # ChiMod library name
  pool_name: ram://test          # Pool name (or file path for BDev)
  pool_query: dynamic            # Either "dynamic" or "local"
  pool_id: 200.0                 # Pool ID in "major.minor" format
  capacity: 2GB                  # Module-specific parameters
  bdev_type: ram                 # Additional parameters as needed
  io_depth: 32
  alignment: 4096

- mod_name: chimaera_another_mod
  pool_name: my_pool
  pool_query: local
  pool_id: 201.0
  custom_param: value

Usage Modes​

1. Automatic During Runtime Init: Pools are automatically created when runtime initializes if compose section is present in configuration:

export CHI_SERVER_CONF=/path/to/config_with_compose.yaml
chimaera_start_runtime

2. Manual via chimaera_compose Utility: Create pools using compose configuration against running runtime:

chimaera_compose /path/to/compose_config.yaml

Implementation Checklist​

When adding compose support to a ChiMod:

  • Add LoadConfig(const chi::PoolConfig& pool_config) method to CreateParams
  • Parse all module-specific parameters from YAML config
  • Handle optional parameters with defaults
  • Use hshm::ConfigParse::ParseSize() for size strings
  • Include <yaml-cpp/yaml.h> and <chimaera/config_manager.h> in tasks header
  • Test with compose configuration before release

Example Admin ChiMod LoadConfig​

void LoadConfig(const chi::PoolConfig& pool_config) {
  // Admin doesn't have additional configuration fields
  // YAML config parsing would go here for modules with config fields
  (void)pool_config;  // Suppress unused parameter warning
}

Example BDev ChiMod LoadConfig​

void LoadConfig(const chi::PoolConfig& pool_config) {
  YAML::Node config = YAML::Load(pool_config.config_);

  // Load BDev type
  if (config["bdev_type"]) {
    std::string type_str = config["bdev_type"].as<std::string>();
    if (type_str == "file") {
      bdev_type_ = BdevType::kFile;
    } else if (type_str == "ram") {
      bdev_type_ = BdevType::kRam;
    }
  }

  // Load capacity (parse size strings)
  if (config["capacity"]) {
    std::string capacity_str = config["capacity"].as<std::string>();
    total_size_ = hshm::ConfigParse::ParseSize(capacity_str);
  }

  // Load optional parameters
  if (config["io_depth"]) {
    io_depth_ = config["io_depth"].as<chi::u32>();
  }
  if (config["alignment"]) {
    alignment_ = config["alignment"].as<chi::u32>();
  }
}