How to Limit Memory Usage During Compression on Ubuntu

This article details methods to restrict RAM consumption when performing high-ratio compression tasks on Ubuntu. High compression settings often exhaust system memory, causing crashes or slowdowns. You will learn how to use built-in command flags, adjust shell resource limits, and implement control groups to manage memory allocation effectively during compression processes.

Use Compression Tool Flags

Most modern compression utilities offer specific flags to cap memory usage directly. This is the most efficient method as it prevents the tool from allocating more RAM than necessary without affecting other system processes.

For xz, which is known for high memory consumption at extreme compression levels, use the -M flag followed by the limit:

xz -M 500MiB file.tar

For zstd, a modern alternative, use the --memory flag to define the maximum memory allowed:

zstd --memory=500MB file.tar

For gzip, memory usage is typically low, but using lower compression flags (e.g., -1 instead of -9) inherently reduces memory load.

Set System Resource Limits

You can limit the virtual memory available to the shell session using the ulimit command. This prevents any child process launched from that shell, including compression tools, from exceeding the set value.

Run the following command in your terminal before starting the compression task:

ulimit -v 500000

This sets the limit to 500000 KB (approximately 500MB). If the compression process attempts to exceed this limit, the system will terminate it. Note that this setting applies only to the current shell session.

Restrict Memory with Cgroups

For precise and isolated control, use Linux Control Groups (cgroups). This method creates a specific environment where the memory limit is enforced strictly by the kernel. You may need to install the cgroup-tools package first.

Create a new cgroup subgroup:

sudo cgcreate -g memory:/compression_limit

Set the memory limit (e.g., 512MB):

echo 536870912 | sudo tee /sys/fs/cgroup/memory/compression_limit/memory.limit_in_bytes

Run the compression command within the restricted group:

sudo cgexec -g memory:compression_limit xz -9 file.tar

This ensures that even if the compression tool requests more memory, the kernel will restrict it to the defined boundary, protecting the rest of your Ubuntu system from instability.