esengine/docs/en/guide/worker-system.md

Worker System

The Worker System (WorkerEntitySystem) is a multi-threaded processing system based on Web Workers in the ECS framework. It's designed for compute-intensive tasks, fully utilizing multi-core CPU performance for true parallel computing.

Core Features

• True Parallel Computing: Execute compute-intensive tasks in background threads using Web Workers
• Automatic Load Balancing: Automatically distribute workload based on CPU core count
• SharedArrayBuffer Optimization: Zero-copy data sharing for improved large-scale computation performance
• Graceful Degradation: Automatic fallback to main thread processing when Workers are not supported
• Type Safety: Full TypeScript support and type checking

Basic Usage

Simple Physics System Example

interface PhysicsData {
  id: number;
  x: number;
  y: number;
  vx: number;
  vy: number;
  mass: number;
  radius: number;
}

@ECSSystem('Physics')
class PhysicsWorkerSystem extends WorkerEntitySystem<PhysicsData> {
  constructor() {
    super(Matcher.all(Position, Velocity, Physics), {
      enableWorker: true,              // Enable Worker parallel processing
      workerCount: 8,                  // Worker count, auto-limited to hardware capacity
      entitiesPerWorker: 100,          // Entities per Worker
      useSharedArrayBuffer: true,      // Enable SharedArrayBuffer optimization
      entityDataSize: 7,               // Data size per entity
      maxEntities: 10000,              // Maximum entity count
      systemConfig: {                  // Configuration passed to Worker
        gravity: 100,
        friction: 0.95
      }
    });
  }

  // Data extraction: Convert Entity to serializable data
  protected extractEntityData(entity: Entity): PhysicsData {
    const position = entity.getComponent(Position);
    const velocity = entity.getComponent(Velocity);
    const physics = entity.getComponent(Physics);

    return {
      id: entity.id,
      x: position.x,
      y: position.y,
      vx: velocity.x,
      vy: velocity.y,
      mass: physics.mass,
      radius: physics.radius
    };
  }

  // Worker processing function: Pure function executed in Worker
  protected workerProcess(
    entities: PhysicsData[],
    deltaTime: number,
    config: any
  ): PhysicsData[] {
    return entities.map(entity => {
      // Apply gravity
      entity.vy += config.gravity * deltaTime;

      // Update position
      entity.x += entity.vx * deltaTime;
      entity.y += entity.vy * deltaTime;

      // Apply friction
      entity.vx *= config.friction;
      entity.vy *= config.friction;

      return entity;
    });
  }

  // Apply results: Apply Worker processing results back to Entity
  protected applyResult(entity: Entity, result: PhysicsData): void {
    const position = entity.getComponent(Position);
    const velocity = entity.getComponent(Velocity);

    position.x = result.x;
    position.y = result.y;
    velocity.x = result.vx;
    velocity.y = result.vy;
  }

  // SharedArrayBuffer optimization support
  protected getDefaultEntityDataSize(): number {
    return 7; // id, x, y, vx, vy, mass, radius
  }

  protected writeEntityToBuffer(entityData: PhysicsData, offset: number): void {
    if (!this.sharedFloatArray) return;

    this.sharedFloatArray[offset + 0] = entityData.id;
    this.sharedFloatArray[offset + 1] = entityData.x;
    this.sharedFloatArray[offset + 2] = entityData.y;
    this.sharedFloatArray[offset + 3] = entityData.vx;
    this.sharedFloatArray[offset + 4] = entityData.vy;
    this.sharedFloatArray[offset + 5] = entityData.mass;
    this.sharedFloatArray[offset + 6] = entityData.radius;
  }

  protected readEntityFromBuffer(offset: number): PhysicsData | null {
    if (!this.sharedFloatArray) return null;

    return {
      id: this.sharedFloatArray[offset + 0],
      x: this.sharedFloatArray[offset + 1],
      y: this.sharedFloatArray[offset + 2],
      vx: this.sharedFloatArray[offset + 3],
      vy: this.sharedFloatArray[offset + 4],
      mass: this.sharedFloatArray[offset + 5],
      radius: this.sharedFloatArray[offset + 6]
    };
  }
}

Configuration Options

The Worker system supports rich configuration options:

interface WorkerSystemConfig {
  /** Enable Worker parallel processing */
  enableWorker?: boolean;
  /** Worker count, defaults to CPU core count, auto-limited to system maximum */
  workerCount?: number;
  /** Entities per Worker for load distribution control */
  entitiesPerWorker?: number;
  /** System configuration data passed to Worker */
  systemConfig?: any;
  /** Enable SharedArrayBuffer optimization */
  useSharedArrayBuffer?: boolean;
  /** Float32 count per entity in SharedArrayBuffer */
  entityDataSize?: number;
  /** Maximum entity count (for SharedArrayBuffer pre-allocation) */
  maxEntities?: number;
  /** Pre-compiled Worker script path (for platforms like WeChat Mini Game that don't support dynamic scripts) */
  workerScriptPath?: string;
}

Configuration Recommendations

constructor() {
  super(matcher, {
    // Decide based on task complexity
    enableWorker: this.shouldUseWorker(),

    // Worker count: System auto-limits to hardware capacity
    workerCount: 8, // Request 8 Workers, actual count limited by CPU cores

    // Entities per Worker (optional)
    entitiesPerWorker: 200, // Precise load distribution control

    // Enable SharedArrayBuffer for many simple calculations
    useSharedArrayBuffer: this.entityCount > 1000,

    // Set according to actual data structure
    entityDataSize: 8, // Ensure it matches data structure

    // Estimated maximum entity count
    maxEntities: 10000,

    // Global configuration passed to Worker
    systemConfig: {
      gravity: 9.8,
      friction: 0.95,
      worldBounds: { width: 1920, height: 1080 }
    }
  });
}

private shouldUseWorker(): boolean {
  // Decide based on entity count and complexity
  return this.expectedEntityCount > 100;
}

// Get system info
getSystemInfo() {
  const info = this.getWorkerInfo();
  console.log(`Worker count: ${info.workerCount}/${info.maxSystemWorkerCount}`);
  console.log(`Entities per Worker: ${info.entitiesPerWorker || 'auto'}`);
  console.log(`Current mode: ${info.currentMode}`);
}

Processing Modes

The Worker system supports two processing modes:

1. Traditional Worker Mode

Data is serialized and passed between main thread and Workers:

// Suitable for: Complex computation logic, moderate entity count
constructor() {
  super(matcher, {
    enableWorker: true,
    useSharedArrayBuffer: false, // Use traditional mode
    workerCount: 2
  });
}

protected workerProcess(entities: EntityData[], deltaTime: number): EntityData[] {
  // Complex algorithm logic
  return entities.map(entity => {
    // AI decisions, pathfinding, etc.
    return this.complexAILogic(entity, deltaTime);
  });
}

2. SharedArrayBuffer Mode

Zero-copy data sharing, suitable for many simple calculations:

// Suitable for: Many entities with simple calculations
constructor() {
  super(matcher, {
    enableWorker: true,
    useSharedArrayBuffer: true, // Enable shared memory
    entityDataSize: 6,
    maxEntities: 10000
  });
}

protected getSharedArrayBufferProcessFunction(): SharedArrayBufferProcessFunction {
  return function(sharedFloatArray: Float32Array, startIndex: number, endIndex: number, deltaTime: number, config: any) {
    const entitySize = 6;
    for (let i = startIndex; i < endIndex; i++) {
      const offset = i * entitySize;

      // Read data
      let x = sharedFloatArray[offset];
      let y = sharedFloatArray[offset + 1];
      let vx = sharedFloatArray[offset + 2];
      let vy = sharedFloatArray[offset + 3];

      // Physics calculations
      vy += config.gravity * deltaTime;
      x += vx * deltaTime;
      y += vy * deltaTime;

      // Write back data
      sharedFloatArray[offset] = x;
      sharedFloatArray[offset + 1] = y;
      sharedFloatArray[offset + 2] = vx;
      sharedFloatArray[offset + 3] = vy;
    }
  };
}

Use Cases

The Worker system is particularly suitable for:

1. Physics Simulation

• Gravity systems: Gravity calculations for many entities
• Collision detection: Complex collision algorithms
• Fluid simulation: Particle fluid systems
• Cloth simulation: Vertex physics calculations

2. AI Computation

• Pathfinding: A*, Dijkstra algorithms
• Behavior trees: Complex AI decision logic
• Swarm intelligence: Boid, fish school algorithms
• Neural networks: Simple AI inference

3. Data Processing

• Bulk entity updates: State machines, lifecycle management
• Statistical calculations: Game data analysis
• Image processing: Texture generation, effect calculations
• Audio processing: Sound synthesis, spectrum analysis

Best Practices

1. Worker Function Requirements

// Recommended: Worker processing function is a pure function
protected workerProcess(entities: PhysicsData[], deltaTime: number, config: any): PhysicsData[] {
  // Only use parameters and standard JavaScript APIs
  return entities.map(entity => {
    // Pure computation logic, no external state dependencies
    entity.y += entity.velocity * deltaTime;
    return entity;
  });
}

// Avoid: Using external references in Worker function
protected workerProcess(entities: PhysicsData[], deltaTime: number): PhysicsData[] {
  // this and external variables are not available in Worker
  return entities.map(entity => {
    entity.y += this.someProperty; // Error
    return entity;
  });
}

2. Data Design

// Recommended: Reasonable data design
interface SimplePhysicsData {
  x: number;
  y: number;
  vx: number;
  vy: number;
  // Keep data structure simple for easy serialization
}

// Avoid: Complex nested objects
interface ComplexData {
  transform: {
    position: { x: number; y: number };
    rotation: { angle: number };
  };
  // Complex nested structures increase serialization overhead
}

3. Worker Count Control

// Recommended: Flexible Worker configuration
constructor() {
  super(matcher, {
    // Specify needed Worker count, system auto-limits to hardware capacity
    workerCount: 8,                     // Request 8 Workers
    entitiesPerWorker: 100,            // 100 entities per Worker
    enableWorker: this.shouldUseWorker(), // Conditional enable
  });
}

private shouldUseWorker(): boolean {
  // Decide based on entity count and complexity
  return this.expectedEntityCount > 100;
}

// Get actual Worker info
checkWorkerConfiguration() {
  const info = this.getWorkerInfo();
  console.log(`Requested Workers: 8`);
  console.log(`Actual Workers: ${info.workerCount}`);
  console.log(`System maximum: ${info.maxSystemWorkerCount}`);
  console.log(`Entities per Worker: ${info.entitiesPerWorker || 'auto'}`);
}

4. Performance Monitoring

// Recommended: Performance monitoring
public getPerformanceMetrics(): WorkerPerformanceMetrics {
  return {
    ...this.getWorkerInfo(),
    entityCount: this.entities.length,
    averageProcessTime: this.getAverageProcessTime(),
    workerUtilization: this.getWorkerUtilization()
  };
}

Performance Optimization Tips

1. Compute Intensity Assessment

Only use Workers for compute-intensive tasks to avoid thread overhead for simple calculations.

2. Data Transfer Optimization

• Use SharedArrayBuffer to reduce serialization overhead
• Keep data structures simple and flat
• Avoid frequent large data transfers

3. Degradation Strategy

Always provide main thread fallback to ensure normal operation in environments without Worker support.

4. Memory Management

Clean up Worker pools and shared buffers promptly to avoid memory leaks.

5. Load Balancing

Use entitiesPerWorker parameter to precisely control load distribution, avoiding idle Workers while others are overloaded.

WeChat Mini Game Support

WeChat Mini Game has special Worker limitations and doesn't support dynamic Worker script creation. ESEngine provides the @esengine/worker-generator CLI tool to solve this problem.

WeChat Mini Game Worker Limitations

Feature	Browser	WeChat Mini Game
Dynamic scripts (Blob URL)	Supported	Not supported
Worker count	Multiple	Maximum 1
Script source	Any	Must be in code package
SharedArrayBuffer	Requires COOP/COEP	Limited support

Using Worker Generator CLI

1. Install the Tool

pnpm add -D @esengine/worker-generator

2. Configure workerScriptPath

Configure workerScriptPath in your WorkerEntitySystem subclass:

@ECSSystem('Physics')
class PhysicsWorkerSystem extends WorkerEntitySystem<PhysicsData> {
  constructor() {
    super(Matcher.all(Position, Velocity, Physics), {
      enableWorker: true,
      workerScriptPath: 'workers/physics-worker.js', // Specify Worker file path
      systemConfig: {
        gravity: 100,
        friction: 0.95
      }
    });
  }

  protected workerProcess(
    entities: PhysicsData[],
    deltaTime: number,
    config: any
  ): PhysicsData[] {
    // Physics calculation logic
    return entities.map(entity => {
      entity.vy += config.gravity * deltaTime;
      entity.x += entity.vx * deltaTime;
      entity.y += entity.vy * deltaTime;
      return entity;
    });
  }

  // ... other methods
}

3. Generate Worker Files

Run the CLI tool to automatically extract workerProcess functions and generate WeChat Mini Game compatible Worker files:

# Basic usage
npx esengine-worker-gen --src ./src --wechat

# Full options
npx esengine-worker-gen \
  --src ./src \           # Source directory
  --wechat \              # Generate WeChat Mini Game compatible code
  --mapping \             # Generate worker-mapping.json
  --verbose               # Verbose output

The CLI tool will:

1. Scan source directory for all WorkerEntitySystem subclasses
2. Read each class's workerScriptPath configuration
3. Extract workerProcess method body
4. Convert to ES5 syntax (WeChat Mini Game compatible)
5. Generate to configured path

4. Configure game.json

Configure workers directory in WeChat Mini Game's game.json:

{
  "deviceOrientation": "portrait",
  "workers": "workers"
}

5. Project Structure

your-game/
├── game.js
├── game.json           # Configure "workers": "workers"
├── src/
│   └── systems/
│       └── PhysicsSystem.ts  # workerScriptPath: 'workers/physics-worker.js'
└── workers/
    ├── physics-worker.js     # Auto-generated
    └── worker-mapping.json   # Auto-generated

Temporarily Disabling Workers

If you need to temporarily disable Workers (e.g., for debugging), there are two ways:

Method 1: Configuration Disable

constructor() {
  super(matcher, {
    enableWorker: false, // Disable Worker, use main thread processing
    // ...
  });
}

Method 2: Platform Adapter Disable

Return Worker not supported in custom platform adapter:

class MyPlatformAdapter implements IPlatformAdapter {
  isWorkerSupported(): boolean {
    return false; // Return false to disable Worker
  }
  // ...
}

Important Notes

1. Re-run CLI tool after each workerProcess modification to generate new Worker files

2. Worker functions must be pure functions, cannot depend on this or external variables:

   // Correct: Only use parameters
   protected workerProcess(entities, deltaTime, config) {
     return entities.map(e => {
       e.y += config.gravity * deltaTime;
       return e;
     });
   }
   
   // Wrong: Using this
   protected workerProcess(entities, deltaTime, config) {
     return entities.map(e => {
       e.y += this.gravity * deltaTime; // Cannot access this in Worker
       return e;
     });
   }
   

3. Pass configuration data via systemConfig, not class properties

4. Developer tool warnings can be ignored:

   • getNetworkType:fail not support - WeChat DevTools internal behavior
   • SharedArrayBuffer will require cross-origin isolation - Development environment warning, won't appear on real devices

Online Demo

See the complete Worker system demo: Worker System Demo

The demo showcases:

• Multi-threaded physics computation
• Real-time performance comparison
• SharedArrayBuffer optimization
• Parallel processing of many entities

The Worker system provides powerful parallel computing capabilities for the ECS framework, allowing you to fully utilize modern multi-core processor performance, offering efficient solutions for complex game logic and compute-intensive tasks.