Limit the Memory Consumption

Limit the Memory Consumption

I'm running my program in a HPC environment (no hard drives, no swap). The system kills any program that is using too much memory. I used the garbage tuners on my collection, but I'm afraid that at some point where I reach my maximum memory capacity, your scheduler launches a step that produces more data thus increasing the memory footprint instead of launching a step that consumes data thus decreasing the memory footprint.

Is there anyway to limit the memory consumption?

2 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

The current implementation has no built-in support for controlling the lower level memory consumption, other than the get-count mechanism. If you don't see any effect of the get-count issues, I suggest enabling the tracing feature on a small example and verify that the get-counts actually reach 0 at some point. If you are using pointers, you should use smart-pointers to actually free the memory the pointer points to.

There are a few other things you could try

  1. if your data types are bigger than a few bytes, use pointers.
  2. try a different scheduler (CNC_SCHEDULER=FIFO_STEAL)
  3. use step_tuner::priority and a scheduler that evaluates priorities to guide the scheduler
  4. don't use hyperthreading, use only as many threads as available HW cores
  5. introduce "artificial tuning constraints", e.g. something like a sliding window for certain steps. How this can work depends on the application.

If you provide a few more details on your application (like how the CnC graph looks like, the data types your using, characteristics of the tag-spaces etc) we might be able to give you more specific advice.



Leave a Comment

Please sign in to add a comment. Not a member? Join today