The Java ProcessBuilder class provides an interface for starting and communicating with a new process that runs outside the Java process that started it. Also see Runtime.exec method.

This new process is managed by the operating system, as part of process management. It has a self-contained execution environment, generally with a complete, private set of basic run-time resources: in particular, with its own memory space.

This post describes a problem with pdftoppm, one of the Poppler utilities also used by SavaPage:

“I'm running pdftoppm to convert a user-provided PDF into a 300DPI image. This works great, except if the user provides an PDF with a very large page size. pdftoppm will allocate enough memory to hold a 300DPI image of that size in memory, which for a 100 inch square page is 100*300 * 100*300 * 4 bytes per pixel = 3.5GB. A malicious user could just give me a silly-large PDF and cause all kinds of problems”

Indeed, what will happen when an A0 sized PDF is offered for pdftoppm -png conversion, or perhaps an invalid PDF that will trigger a program bug that leads to excessive memory consumption?

We have to protect the SavaPage server process from excessive memory consumption of forked processes.

With ulimit we can check and limit various process resources:

# Check resources of user 'savapage'
# -S (soft limits) of -a (all) resources
sudo su - savapage -c "ulimit -Sa"

Soft limits are the current setting for a particular limit. They can be increased only to the current hard limit setting.
Display all current soft settings with: ulimit -Sa

Hard limits are the maximum limit that can be configured. Any changes to these require root access.
Display all current hard settings with: ulimit -Ha

We focus on the virtual memory (kbytes, -v) resource, since the virtual address space is a good approximation for the memory used. Things get more complicated when a process spawns other processes: see Limiting time and memory consumption of a program in Linux. But we assume this is not the case for the simple programs used by SavaPage.

Let's do some tests to get a feel of what ulimit does. Open a terminal session …

# Check
ulimit -Sv
# this will probably show "unlimited"
 
# Set soft limit of virtual memory to 500MB 
# (the setting lives in this session only)
ulimit -Sv 500000
 
# Check again
ulimit -Sv

A simple tactic is to use ulimit at the invocation of the new child process, as shown below:

# The 'sh' command simulates the Java exec.
sh -c "ulimit -Sv 500000;pdftoppm -png -f 1 -l 1 test.pdf > test.png"
# You can also throw cpu time (seconds, -t) in the mix
sh -c "ulimit -Sv 500000;ulimit -St 2;pdftoppm -png -f 1 -l 1 test.pdf > test.png"

/etc/security/limits.conf holds the system wide limits for various users. The following lines could be added at the end of this file to set global virtual memory limits (the as keyword):

savapage  hard as 500000
savapage  soft as 500000 

Configure ulimit properties for various exec commands in server.properties file. For example:

# This property will resolve as: 
#   ulimit -Sv 500000;ulimit -St 4;pdftoppm
java.runtime.exec.ulimit.pdftoppm = -Sv 500000;-St 4

Use /etc/security/limits.conf to limit virtual memory for user savapage.

This will also limit the java process of the SavaPage server. Besides, java has its own command line options to allocate JVM memory. Therefore, set ulimit -Sv unlimited; when starting the java server process.

systemd has its own way to limit memory to the java process:

systemctl show savapage.service | grep MemoryLimit

However, when java exec forks a child process, that process gets its ulimit values from /etc/security/limits.conf

Relying on one “System Config” limits.conf value for all types of forked child processes is too crude a mechanism. Therefore, a more tailored approach with “SavaPage Config” is recommended.