[question] Is it advisable to have nested worker pools in Node.js for handling heavy data operations?

[wojtekKrol]

Description

I am developing a Node.js application that involves heavy data processing with three distinct services, each managed by a separate worker pool. Here's the breakdown of the setup:

• ServiceX and ServiceY: Each runs with a single worker (serviceX.js and serviceY.js).
• ServiceZ: This is the core of my application, handling extensive data structures (like an array of 10k objects) and data processing, running with N workers (serviceZ.js).

Current Architecture

In ServiceZ, a task involves checking for duplicates in a database where a specific column is not null. Additionally, I need to handle duplicates where the column is null in returned data to optimize processing and database updates. Given the intensity of these operations, I am considering using a nested worker pool within ServiceZ.

Question

Is it practical to initiate a new worker pool within an existing worker's task in ServiceZ? This new pool would potentially launch single worker instances dynamically to handle specific heavy operations like duplicate checking.

Concerns

• Performance: Could nesting worker pools lead to significant performance hits due to overhead or complex resource management?
• Best Practices: Is this approach advisable in terms of architectural best practices, or are there more efficient methods to manage such heavy operations without risking call stack blocks?