feat(worker): 添加微信小游戏 Worker 支持和 Worker Generator CLI (#297)
* feat(worker): 添加微信小游戏 Worker 支持和 Worker Generator CLI - 新增 @esengine/worker-generator 包，用于从 WorkerEntitySystem 生成 Worker 文件 - WorkerEntitySystem 添加 workerScriptPath 配置项，支持预编译 Worker 脚本 - CLI 工具支持 --wechat 模式，自动转换 ES6+ 为 ES5 语法 - 修复微信小游戏 Worker 消息格式差异（res 直接是数据，无需 .data） - 更新中英文文档，添加微信小游戏支持章节 * docs: 更新 changelog，添加 v2.3.1 说明并标注 v2.3.0 为废弃 * fix: 修复 CI 检查问题 - 移除 cli.ts 中未使用的 toKebabCase 函数 - 修复 generator.ts 中正则表达式的 ReDoS 风险（使用 [ \t] 替代 \s*） - 更新 changelog 版本号（2.3.1 -> 2.3.2） * docs: 移除未发布版本的 changelog 条目 * fix(worker-generator): 使用 TypeScript 编译器替代手写正则进行 ES5 转换 - 修复 CodeQL 检测的 ReDoS 安全问题 - 使用 ts.transpileModule 进行安全可靠的代码转换 - 移除所有可能导致回溯的正则表达式
This commit is contained in:
YHH
GitHub
570
docs/en/guide/worker-system.md
Normal file
570
docs/en/guide/worker-system.md
Normal file
@@ -0,0 +1,570 @@
|
||||
# Worker System
|
||||
|
||||
The Worker System (WorkerEntitySystem) is a multi-threaded processing system based on Web Workers in the ECS framework. It's designed for compute-intensive tasks, fully utilizing multi-core CPU performance for true parallel computing.
|
||||
|
||||
## Core Features
|
||||
|
||||
- **True Parallel Computing**: Execute compute-intensive tasks in background threads using Web Workers
|
||||
- **Automatic Load Balancing**: Automatically distribute workload based on CPU core count
|
||||
- **SharedArrayBuffer Optimization**: Zero-copy data sharing for improved large-scale computation performance
|
||||
- **Graceful Degradation**: Automatic fallback to main thread processing when Workers are not supported
|
||||
- **Type Safety**: Full TypeScript support and type checking
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Simple Physics System Example
|
||||
|
||||
```typescript
|
||||
interface PhysicsData {
|
||||
id: number;
|
||||
x: number;
|
||||
y: number;
|
||||
vx: number;
|
||||
vy: number;
|
||||
mass: number;
|
||||
radius: number;
|
||||
}
|
||||
|
||||
@ECSSystem('Physics')
|
||||
class PhysicsWorkerSystem extends WorkerEntitySystem<PhysicsData> {
|
||||
constructor() {
|
||||
super(Matcher.all(Position, Velocity, Physics), {
|
||||
enableWorker: true, // Enable Worker parallel processing
|
||||
workerCount: 8, // Worker count, auto-limited to hardware capacity
|
||||
entitiesPerWorker: 100, // Entities per Worker
|
||||
useSharedArrayBuffer: true, // Enable SharedArrayBuffer optimization
|
||||
entityDataSize: 7, // Data size per entity
|
||||
maxEntities: 10000, // Maximum entity count
|
||||
systemConfig: { // Configuration passed to Worker
|
||||
gravity: 100,
|
||||
friction: 0.95
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Data extraction: Convert Entity to serializable data
|
||||
protected extractEntityData(entity: Entity): PhysicsData {
|
||||
const position = entity.getComponent(Position);
|
||||
const velocity = entity.getComponent(Velocity);
|
||||
const physics = entity.getComponent(Physics);
|
||||
|
||||
return {
|
||||
id: entity.id,
|
||||
x: position.x,
|
||||
y: position.y,
|
||||
vx: velocity.x,
|
||||
vy: velocity.y,
|
||||
mass: physics.mass,
|
||||
radius: physics.radius
|
||||
};
|
||||
}
|
||||
|
||||
// Worker processing function: Pure function executed in Worker
|
||||
protected workerProcess(
|
||||
entities: PhysicsData[],
|
||||
deltaTime: number,
|
||||
config: any
|
||||
): PhysicsData[] {
|
||||
return entities.map(entity => {
|
||||
// Apply gravity
|
||||
entity.vy += config.gravity * deltaTime;
|
||||
|
||||
// Update position
|
||||
entity.x += entity.vx * deltaTime;
|
||||
entity.y += entity.vy * deltaTime;
|
||||
|
||||
// Apply friction
|
||||
entity.vx *= config.friction;
|
||||
entity.vy *= config.friction;
|
||||
|
||||
return entity;
|
||||
});
|
||||
}
|
||||
|
||||
// Apply results: Apply Worker processing results back to Entity
|
||||
protected applyResult(entity: Entity, result: PhysicsData): void {
|
||||
const position = entity.getComponent(Position);
|
||||
const velocity = entity.getComponent(Velocity);
|
||||
|
||||
position.x = result.x;
|
||||
position.y = result.y;
|
||||
velocity.x = result.vx;
|
||||
velocity.y = result.vy;
|
||||
}
|
||||
|
||||
// SharedArrayBuffer optimization support
|
||||
protected getDefaultEntityDataSize(): number {
|
||||
return 7; // id, x, y, vx, vy, mass, radius
|
||||
}
|
||||
|
||||
protected writeEntityToBuffer(entityData: PhysicsData, offset: number): void {
|
||||
if (!this.sharedFloatArray) return;
|
||||
|
||||
this.sharedFloatArray[offset + 0] = entityData.id;
|
||||
this.sharedFloatArray[offset + 1] = entityData.x;
|
||||
this.sharedFloatArray[offset + 2] = entityData.y;
|
||||
this.sharedFloatArray[offset + 3] = entityData.vx;
|
||||
this.sharedFloatArray[offset + 4] = entityData.vy;
|
||||
this.sharedFloatArray[offset + 5] = entityData.mass;
|
||||
this.sharedFloatArray[offset + 6] = entityData.radius;
|
||||
}
|
||||
|
||||
protected readEntityFromBuffer(offset: number): PhysicsData | null {
|
||||
if (!this.sharedFloatArray) return null;
|
||||
|
||||
return {
|
||||
id: this.sharedFloatArray[offset + 0],
|
||||
x: this.sharedFloatArray[offset + 1],
|
||||
y: this.sharedFloatArray[offset + 2],
|
||||
vx: this.sharedFloatArray[offset + 3],
|
||||
vy: this.sharedFloatArray[offset + 4],
|
||||
mass: this.sharedFloatArray[offset + 5],
|
||||
radius: this.sharedFloatArray[offset + 6]
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
The Worker system supports rich configuration options:
|
||||
|
||||
```typescript
|
||||
interface WorkerSystemConfig {
|
||||
/** Enable Worker parallel processing */
|
||||
enableWorker?: boolean;
|
||||
/** Worker count, defaults to CPU core count, auto-limited to system maximum */
|
||||
workerCount?: number;
|
||||
/** Entities per Worker for load distribution control */
|
||||
entitiesPerWorker?: number;
|
||||
/** System configuration data passed to Worker */
|
||||
systemConfig?: any;
|
||||
/** Enable SharedArrayBuffer optimization */
|
||||
useSharedArrayBuffer?: boolean;
|
||||
/** Float32 count per entity in SharedArrayBuffer */
|
||||
entityDataSize?: number;
|
||||
/** Maximum entity count (for SharedArrayBuffer pre-allocation) */
|
||||
maxEntities?: number;
|
||||
/** Pre-compiled Worker script path (for platforms like WeChat Mini Game that don't support dynamic scripts) */
|
||||
workerScriptPath?: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Recommendations
|
||||
|
||||
```typescript
|
||||
constructor() {
|
||||
super(matcher, {
|
||||
// Decide based on task complexity
|
||||
enableWorker: this.shouldUseWorker(),
|
||||
|
||||
// Worker count: System auto-limits to hardware capacity
|
||||
workerCount: 8, // Request 8 Workers, actual count limited by CPU cores
|
||||
|
||||
// Entities per Worker (optional)
|
||||
entitiesPerWorker: 200, // Precise load distribution control
|
||||
|
||||
// Enable SharedArrayBuffer for many simple calculations
|
||||
useSharedArrayBuffer: this.entityCount > 1000,
|
||||
|
||||
// Set according to actual data structure
|
||||
entityDataSize: 8, // Ensure it matches data structure
|
||||
|
||||
// Estimated maximum entity count
|
||||
maxEntities: 10000,
|
||||
|
||||
// Global configuration passed to Worker
|
||||
systemConfig: {
|
||||
gravity: 9.8,
|
||||
friction: 0.95,
|
||||
worldBounds: { width: 1920, height: 1080 }
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private shouldUseWorker(): boolean {
|
||||
// Decide based on entity count and complexity
|
||||
return this.expectedEntityCount > 100;
|
||||
}
|
||||
|
||||
// Get system info
|
||||
getSystemInfo() {
|
||||
const info = this.getWorkerInfo();
|
||||
console.log(`Worker count: ${info.workerCount}/${info.maxSystemWorkerCount}`);
|
||||
console.log(`Entities per Worker: ${info.entitiesPerWorker || 'auto'}`);
|
||||
console.log(`Current mode: ${info.currentMode}`);
|
||||
}
|
||||
```
|
||||
|
||||
## Processing Modes
|
||||
|
||||
The Worker system supports two processing modes:
|
||||
|
||||
### 1. Traditional Worker Mode
|
||||
|
||||
Data is serialized and passed between main thread and Workers:
|
||||
|
||||
```typescript
|
||||
// Suitable for: Complex computation logic, moderate entity count
|
||||
constructor() {
|
||||
super(matcher, {
|
||||
enableWorker: true,
|
||||
useSharedArrayBuffer: false, // Use traditional mode
|
||||
workerCount: 2
|
||||
});
|
||||
}
|
||||
|
||||
protected workerProcess(entities: EntityData[], deltaTime: number): EntityData[] {
|
||||
// Complex algorithm logic
|
||||
return entities.map(entity => {
|
||||
// AI decisions, pathfinding, etc.
|
||||
return this.complexAILogic(entity, deltaTime);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### 2. SharedArrayBuffer Mode
|
||||
|
||||
Zero-copy data sharing, suitable for many simple calculations:
|
||||
|
||||
```typescript
|
||||
// Suitable for: Many entities with simple calculations
|
||||
constructor() {
|
||||
super(matcher, {
|
||||
enableWorker: true,
|
||||
useSharedArrayBuffer: true, // Enable shared memory
|
||||
entityDataSize: 6,
|
||||
maxEntities: 10000
|
||||
});
|
||||
}
|
||||
|
||||
protected getSharedArrayBufferProcessFunction(): SharedArrayBufferProcessFunction {
|
||||
return function(sharedFloatArray: Float32Array, startIndex: number, endIndex: number, deltaTime: number, config: any) {
|
||||
const entitySize = 6;
|
||||
for (let i = startIndex; i < endIndex; i++) {
|
||||
const offset = i * entitySize;
|
||||
|
||||
// Read data
|
||||
let x = sharedFloatArray[offset];
|
||||
let y = sharedFloatArray[offset + 1];
|
||||
let vx = sharedFloatArray[offset + 2];
|
||||
let vy = sharedFloatArray[offset + 3];
|
||||
|
||||
// Physics calculations
|
||||
vy += config.gravity * deltaTime;
|
||||
x += vx * deltaTime;
|
||||
y += vy * deltaTime;
|
||||
|
||||
// Write back data
|
||||
sharedFloatArray[offset] = x;
|
||||
sharedFloatArray[offset + 1] = y;
|
||||
sharedFloatArray[offset + 2] = vx;
|
||||
sharedFloatArray[offset + 3] = vy;
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
The Worker system is particularly suitable for:
|
||||
|
||||
### 1. Physics Simulation
|
||||
- **Gravity systems**: Gravity calculations for many entities
|
||||
- **Collision detection**: Complex collision algorithms
|
||||
- **Fluid simulation**: Particle fluid systems
|
||||
- **Cloth simulation**: Vertex physics calculations
|
||||
|
||||
### 2. AI Computation
|
||||
- **Pathfinding**: A*, Dijkstra algorithms
|
||||
- **Behavior trees**: Complex AI decision logic
|
||||
- **Swarm intelligence**: Boid, fish school algorithms
|
||||
- **Neural networks**: Simple AI inference
|
||||
|
||||
### 3. Data Processing
|
||||
- **Bulk entity updates**: State machines, lifecycle management
|
||||
- **Statistical calculations**: Game data analysis
|
||||
- **Image processing**: Texture generation, effect calculations
|
||||
- **Audio processing**: Sound synthesis, spectrum analysis
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Worker Function Requirements
|
||||
|
||||
```typescript
|
||||
// Recommended: Worker processing function is a pure function
|
||||
protected workerProcess(entities: PhysicsData[], deltaTime: number, config: any): PhysicsData[] {
|
||||
// Only use parameters and standard JavaScript APIs
|
||||
return entities.map(entity => {
|
||||
// Pure computation logic, no external state dependencies
|
||||
entity.y += entity.velocity * deltaTime;
|
||||
return entity;
|
||||
});
|
||||
}
|
||||
|
||||
// Avoid: Using external references in Worker function
|
||||
protected workerProcess(entities: PhysicsData[], deltaTime: number): PhysicsData[] {
|
||||
// this and external variables are not available in Worker
|
||||
return entities.map(entity => {
|
||||
entity.y += this.someProperty; // Error
|
||||
return entity;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Data Design
|
||||
|
||||
```typescript
|
||||
// Recommended: Reasonable data design
|
||||
interface SimplePhysicsData {
|
||||
x: number;
|
||||
y: number;
|
||||
vx: number;
|
||||
vy: number;
|
||||
// Keep data structure simple for easy serialization
|
||||
}
|
||||
|
||||
// Avoid: Complex nested objects
|
||||
interface ComplexData {
|
||||
transform: {
|
||||
position: { x: number; y: number };
|
||||
rotation: { angle: number };
|
||||
};
|
||||
// Complex nested structures increase serialization overhead
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Worker Count Control
|
||||
|
||||
```typescript
|
||||
// Recommended: Flexible Worker configuration
|
||||
constructor() {
|
||||
super(matcher, {
|
||||
// Specify needed Worker count, system auto-limits to hardware capacity
|
||||
workerCount: 8, // Request 8 Workers
|
||||
entitiesPerWorker: 100, // 100 entities per Worker
|
||||
enableWorker: this.shouldUseWorker(), // Conditional enable
|
||||
});
|
||||
}
|
||||
|
||||
private shouldUseWorker(): boolean {
|
||||
// Decide based on entity count and complexity
|
||||
return this.expectedEntityCount > 100;
|
||||
}
|
||||
|
||||
// Get actual Worker info
|
||||
checkWorkerConfiguration() {
|
||||
const info = this.getWorkerInfo();
|
||||
console.log(`Requested Workers: 8`);
|
||||
console.log(`Actual Workers: ${info.workerCount}`);
|
||||
console.log(`System maximum: ${info.maxSystemWorkerCount}`);
|
||||
console.log(`Entities per Worker: ${info.entitiesPerWorker || 'auto'}`);
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Performance Monitoring
|
||||
|
||||
```typescript
|
||||
// Recommended: Performance monitoring
|
||||
public getPerformanceMetrics(): WorkerPerformanceMetrics {
|
||||
return {
|
||||
...this.getWorkerInfo(),
|
||||
entityCount: this.entities.length,
|
||||
averageProcessTime: this.getAverageProcessTime(),
|
||||
workerUtilization: this.getWorkerUtilization()
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Optimization Tips
|
||||
|
||||
### 1. Compute Intensity Assessment
|
||||
Only use Workers for compute-intensive tasks to avoid thread overhead for simple calculations.
|
||||
|
||||
### 2. Data Transfer Optimization
|
||||
- Use SharedArrayBuffer to reduce serialization overhead
|
||||
- Keep data structures simple and flat
|
||||
- Avoid frequent large data transfers
|
||||
|
||||
### 3. Degradation Strategy
|
||||
Always provide main thread fallback to ensure normal operation in environments without Worker support.
|
||||
|
||||
### 4. Memory Management
|
||||
Clean up Worker pools and shared buffers promptly to avoid memory leaks.
|
||||
|
||||
### 5. Load Balancing
|
||||
Use `entitiesPerWorker` parameter to precisely control load distribution, avoiding idle Workers while others are overloaded.
|
||||
|
||||
## WeChat Mini Game Support
|
||||
|
||||
WeChat Mini Game has special Worker limitations and doesn't support dynamic Worker script creation. ESEngine provides the `@esengine/worker-generator` CLI tool to solve this problem.
|
||||
|
||||
### WeChat Mini Game Worker Limitations
|
||||
|
||||
| Feature | Browser | WeChat Mini Game |
|
||||
|---------|---------|------------------|
|
||||
| Dynamic scripts (Blob URL) | Supported | Not supported |
|
||||
| Worker count | Multiple | Maximum 1 |
|
||||
| Script source | Any | Must be in code package |
|
||||
| SharedArrayBuffer | Requires COOP/COEP | Limited support |
|
||||
|
||||
### Using Worker Generator CLI
|
||||
|
||||
#### 1. Install the Tool
|
||||
|
||||
```bash
|
||||
pnpm add -D @esengine/worker-generator
|
||||
```
|
||||
|
||||
#### 2. Configure workerScriptPath
|
||||
|
||||
Configure `workerScriptPath` in your WorkerEntitySystem subclass:
|
||||
|
||||
```typescript
|
||||
@ECSSystem('Physics')
|
||||
class PhysicsWorkerSystem extends WorkerEntitySystem<PhysicsData> {
|
||||
constructor() {
|
||||
super(Matcher.all(Position, Velocity, Physics), {
|
||||
enableWorker: true,
|
||||
workerScriptPath: 'workers/physics-worker.js', // Specify Worker file path
|
||||
systemConfig: {
|
||||
gravity: 100,
|
||||
friction: 0.95
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
protected workerProcess(
|
||||
entities: PhysicsData[],
|
||||
deltaTime: number,
|
||||
config: any
|
||||
): PhysicsData[] {
|
||||
// Physics calculation logic
|
||||
return entities.map(entity => {
|
||||
entity.vy += config.gravity * deltaTime;
|
||||
entity.x += entity.vx * deltaTime;
|
||||
entity.y += entity.vy * deltaTime;
|
||||
return entity;
|
||||
});
|
||||
}
|
||||
|
||||
// ... other methods
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Generate Worker Files
|
||||
|
||||
Run the CLI tool to automatically extract `workerProcess` functions and generate WeChat Mini Game compatible Worker files:
|
||||
|
||||
```bash
|
||||
# Basic usage
|
||||
npx esengine-worker-gen --src ./src --wechat
|
||||
|
||||
# Full options
|
||||
npx esengine-worker-gen \
|
||||
--src ./src \ # Source directory
|
||||
--wechat \ # Generate WeChat Mini Game compatible code
|
||||
--mapping \ # Generate worker-mapping.json
|
||||
--verbose # Verbose output
|
||||
```
|
||||
|
||||
The CLI tool will:
|
||||
1. Scan source directory for all `WorkerEntitySystem` subclasses
|
||||
2. Read each class's `workerScriptPath` configuration
|
||||
3. Extract `workerProcess` method body
|
||||
4. Convert to ES5 syntax (WeChat Mini Game compatible)
|
||||
5. Generate to configured path
|
||||
|
||||
#### 4. Configure game.json
|
||||
|
||||
Configure workers directory in WeChat Mini Game's `game.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"deviceOrientation": "portrait",
|
||||
"workers": "workers"
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. Project Structure
|
||||
|
||||
```
|
||||
your-game/
|
||||
├── game.js
|
||||
├── game.json # Configure "workers": "workers"
|
||||
├── src/
|
||||
│ └── systems/
|
||||
│ └── PhysicsSystem.ts # workerScriptPath: 'workers/physics-worker.js'
|
||||
└── workers/
|
||||
├── physics-worker.js # Auto-generated
|
||||
└── worker-mapping.json # Auto-generated
|
||||
```
|
||||
|
||||
### Temporarily Disabling Workers
|
||||
|
||||
If you need to temporarily disable Workers (e.g., for debugging), there are two ways:
|
||||
|
||||
#### Method 1: Configuration Disable
|
||||
|
||||
```typescript
|
||||
constructor() {
|
||||
super(matcher, {
|
||||
enableWorker: false, // Disable Worker, use main thread processing
|
||||
// ...
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
#### Method 2: Platform Adapter Disable
|
||||
|
||||
Return Worker not supported in custom platform adapter:
|
||||
|
||||
```typescript
|
||||
class MyPlatformAdapter implements IPlatformAdapter {
|
||||
isWorkerSupported(): boolean {
|
||||
return false; // Return false to disable Worker
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Important Notes
|
||||
|
||||
1. **Re-run CLI tool after each `workerProcess` modification** to generate new Worker files
|
||||
|
||||
2. **Worker functions must be pure functions**, cannot depend on `this` or external variables:
|
||||
```typescript
|
||||
// Correct: Only use parameters
|
||||
protected workerProcess(entities, deltaTime, config) {
|
||||
return entities.map(e => {
|
||||
e.y += config.gravity * deltaTime;
|
||||
return e;
|
||||
});
|
||||
}
|
||||
|
||||
// Wrong: Using this
|
||||
protected workerProcess(entities, deltaTime, config) {
|
||||
return entities.map(e => {
|
||||
e.y += this.gravity * deltaTime; // Cannot access this in Worker
|
||||
return e;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
3. **Pass configuration data via `systemConfig`**, not class properties
|
||||
|
||||
4. **Developer tool warnings can be ignored**:
|
||||
- `getNetworkType:fail not support` - WeChat DevTools internal behavior
|
||||
- `SharedArrayBuffer will require cross-origin isolation` - Development environment warning, won't appear on real devices
|
||||
|
||||
## Online Demo
|
||||
|
||||
See the complete Worker system demo: [Worker System Demo](https://esengine.github.io/ecs-framework/demos/worker-system/)
|
||||
|
||||
The demo showcases:
|
||||
- Multi-threaded physics computation
|
||||
- Real-time performance comparison
|
||||
- SharedArrayBuffer optimization
|
||||
- Parallel processing of many entities
|
||||
|
||||
The Worker system provides powerful parallel computing capabilities for the ECS framework, allowing you to fully utilize modern multi-core processor performance, offering efficient solutions for complex game logic and compute-intensive tasks.
|
||||
Reference in New Issue
Block a user