# High Throughput Computing Administration Guide

This document contains Slurm administrator information specifically for high throughput computing, namely the execution of many short jobs. Getting optimal performance for high throughput computing does require some tuning and this document should help you off to a good start. A working knowledge of Slurm should be considered a prerequisite for this material.

## Performance Results

Slurm has also been validated to execute 500 simple batch jobs per second on a sustained basis with short bursts of activity at a much higher level. Actual performance depends upon the jobs to be executed plus the hardware and configuration used.

## System configuration

Several system configuration parameters may require modification to support a large number of open files and TCP connections with large bursts of messages. Changes can be made using the /etc/rc.d/rc.local or /etc/sysctl.conf script to preserve changes after reboot. In either case, you can write values directly into these files (e.g. "echo 32832 > /proc/sys/fs/file-max").

- /proc/sys/fs/file-max: The maximum number of concurrently open files. We recommend a limit of at least 32,832.
- /proc/sys/net/ipv4/tcp_max_syn_backlog: The maximum number of SYN requests to keep in memory that we have yet to get the third packet in a 3-way handshake from. The default value is 1024 for systems with more than 128MiB of memory, and 128 for low memory machines. If server suffers of overload, try to increase this number.
- /proc/sys/net/ipv4/tcp_syncookies: Used to send out syncookies to hosts when the kernels syn backlog queue for a specific socket is overflowed. The default value is 0, which disables this functionality. Set the value to 1.
- /proc/sys/net/ipv4/tcp_synack_retries: How many times to retransmit the SYN,ACK reply to an SYN request. In other words, this tells the system how many times to try to establish a passive TCP connection that was started by another host. This variable takes an integer value, but should under no circumstances be larger than 255. Each retransmission will take approximately 30 to 40 seconds. The default value of 5, which results in a timeout of passive TCP connections of approximately 180 seconds and is generally satisfactory.
- /proc/sys/net/core/somaxconn: Limit of socket listen() backlog, known in userspace as SOMAXCONN. Defaults to 128 or more depending on OS version. Sites should consider a minimum value of 4096 or 8192, or a higher value that is greater the sum of all nodes plus all active clients in the cluster.
- /proc/sys/net/ipv4/ip_local_port_range: Identify the ephemeral ports available, which are used for many Slurm communications. The value may be raised to support a high volume of communications. For example, write the value "32768 65535" into the ip_local_port_range file in order to make that range of ports available.

The transmit queue length (txqueuelen) may also need to be modified using the ifconfig command. A value of 4096 has been found to work well for one site with a very large cluster (e.g. "ifconfig txqueuelen 4096").

## Munge configuration

By default the Munge daemon runs with two threads, but a higher thread count can improve its throughput. We suggest starting the Munge daemon with ten threads for high throughput support (e.g. "munged --num-threads 10").

## User limits

The ulimit values in effect for the slurmctld daemon should be set quite high for memory size, open file count and stack size.

## Slurm Configuration

Several Slurm configuration parameters should be adjusted to reflect the needs of high throughput computing. The changes described below will not be possible in all environments, but these are the configuration options that you may want to consider for higher throughput.

- AccountingStorageType: Disabling storing accounting records by not setting this plugin. Turning accounting off provides minimal improvement in performance. If using the SlurmDBD increased speedup can be achieved by setting the CommitDelay option in the slurmdbd.conf
- JobAcctGatherType: Disabling the collection of job accounting information will improve job throughput. Disable collection of accounting by using the jobacct_gather/none plugin.
- JobCompType: Disabling recording of job completion information will improve job throughput. Disable recording of job completion information by using the jobcomp/none plugin.
- JobSubmitPlugins: Use of a lua job submit plugin is not recommended. slurmctld runs this script while holding internal locks, and only a single copy of this script can run at a time. This blocks most concurrency in slurmctld. Therefore, we do not recommend using it in a high throughput environment.
- MaxJobCount: Controls how many jobs may be in the slurmctld daemon records at any point in time (pending, running, suspended or completed[temporarily]). The default value is 10,000.
- MessageTimeout: Controls how long to wait for a response to messages. The default value is 10 seconds. While the slurmctld daemon is highly threaded, its responsiveness is load dependent. This value might need to be increased somewhat.
- MinJobAge: Controls how soon the record of a completed job can be purged from the slurmctld memory and thus not visible using the squeue command. The record of jobs run will be preserved in accounting records and logs. The default value is 300 seconds. The value should be reduced to a few seconds if possible. Use of accounting records for older jobs can increase the job throughput rate compared with retaining old jobs in the memory of the slurmctld daemon.
- PriorityType: The priority/basic is considerably faster than other options, but schedules jobs only on a First In First Out (FIFO) basis.
- PrologSlurmctld/EpilogSlurmctld: Neither of these is recommended for a high throughput environment. When they are enabled a separate slurmctld thread has to be created for every job start (or task for a job array). Current architecture requires acquisition of a job write lock in every thread, which is a costly operation that severely limits scheduler throughput.
- SchedulerParameters: Many scheduling parameters are available.
  - Setting option batch_sched_delay will control how long the scheduling of batch jobs can be delayed. This effects only batch jobs. For example, if many jobs are submitted each second, the overhead of trying to schedule each one will adversely impact the rate at which jobs can be submitted. The default value is 3 seconds.
  - Setting option defer will avoid attempting to schedule each job individually at job submit time, but defer it until a later time when scheduling multiple jobs simultaneously may be possible. This option may improve system responsiveness when large numbers of jobs (many hundreds) are submitted at the same time, but it will delay the initiation time of individual jobs.
  - Setting the defer_batch option is similar to the defer option, as explained above. The difference is that defer_batch will allow interactive jobs to be started immediately, but jobs submitted with sbatch will be deferred to allow multiple jobs to accumulate and be scheduled at once.
  - sched_min_interval is yet another configuration parameter to control how frequently the scheduling logic runs. It can still be triggered on each job submit, job termination, or other state change which could permit a new job to be started. However that triggering does not cause the scheduling logic to be started immediately, but only within the configured sched_interval. For example, if sched_min_interval=2000000 (microseconds) and 100 jobs are submitted within a 2 second time window, then the scheduling logic will be executed one time rather than 100 times if sched_min_interval was set to 0 (no delay).
  - Besides controlling how frequently the scheduling logic is executed, the default_queue_depth configuration parameter controls how many jobs are considered to be started in each scheduler iteration. The default value of default_queue_depth is 100 (jobs), which should be fine in most cases.
  - The sched/backfill plugin has relatively high overhead if used with large numbers of job. Configuring bf_max_job_test to a modest size (say 100 jobs or less) and bf_interval to 30 seconds or more will limit the overhead of backfill scheduling (NOTE: the default values are fine for both of these parameters). Other backfill options available for tuning backfill scheduling include bf_max_job_user, bf_resolution and bf_window. See the slurm.conf man page for details.
  - A set of scheduling parameters currently used for running hundreds of jobs per second on a sustained basis on one cluster follows. Note that every environment is different and this set of parameters will not work well in every case, but it may serve as a good starting point.
  - batch_sched_delay=20
  - bf_continue
  - bf_interval=300
  - bf_min_age_reserve=10800
  - bf_resolution=600
  - bf_yield_interval=1000000
  - partition_job_depth=500
  - sched_max_job_start=200
  - sched_min_interval=2000000
- SlurmctldParameters: Many slurmctld daemon parameters are available.
  - Increasing conmgr_max_connections will allow slurmctld to accept more connections at once to avoid connect() timeouts during times of high load but not necessarily read or write timeouts. The trade off is that slurmctld will use more memory as each connection reserves memory to buffer inbound and outbound data along with the connection state. conmgr_max_connections should at least be the number of hardware CPU threads available but less than `sysctl net.nf_conntrack_max` and `sysctl net.core.somaxconn`. Enabling `sysctl net.ipv4.tcp_syncookies=1` is also suggested to allow the kernel to better manage larger bursts of incoming sockets. When modifying this parameter, you should monitor for relative changes in `sdiag`'s output. The ave_time field section under Remote Procedure Call statistics should be given special attention as changes to that can have a dramatic impact on overall response times. Increasing conmgr_max_connections too much could cause an Out of Memory event which will cause slurmctld to crash, potentially losing jobs and accounting. Sites are advised to try changing MessageTimeout and TCPTimeout before changing the conmgr_max_connections parameter.
  - The conmgr_threads option controls the size of the thread pool that is used to process communications. Threads are used as needed to handle I/O or to process incoming RPCs and generate replies. The trade off is that slurmctld will use more memory for each additional thread. Increasing thread counts will also cause increased kernel scheduler contention when there are more threads than available hardware CPUs, increasing the potential for thread starvation. While processing incoming RPC requests, slurmctld usually has to obtain one or more of the global slurmctld locks. Each thread attempting to obtain a lock can cause increased contention with the scheduler threads. Lock contention will result in the job scheduler running slower or with noticeable delays, and increase the average time per RPC. Sites wishing for more RPC throughput can increase conmgr_threads from the defaults, while sites wishing to prioritize scheduler threads can decrease the thread count. We recommend trying to modify conmgr_max_connections and the SchedulerParameters to get the performance you want, before attempting to change conmgr_threads. You should monitor for changes in `sdiag`'s job start statistics when changing this parameter. In most cases, sites should consider reducing conmgr_threads instead of adding more threads, as having too many threads can cause locking overhead and cause `slurmctld` to hang. Increasing conmgr_threads also uses more memory, potentially causing an Out of Memory event for `slurmctld`, which could result in lost jobs and accounting.
- SchedulerType: If most jobs are short lived then use of the sched/builtin plugin is recommended. This manages a queue of jobs on a First-In-First-Out (FIFO) basis and eliminates logic used to sort the queue by priority.
- SlurmctldDebug: More detailed logging will decrease system throughput. Set to error or info for regular operations with high throughput workload.
- SlurmdDebug: More detailed logging will decrease system throughput. Set to error or info for regular operations with high throughput workload.
- SlurmdParameters:
  - Increasing conmgr_max_connections will allow slurmd to accept more simultaneous connections from slurmctld. On nodes running many concurrent jobs, each job may generate multiple inbound connections from slurmctld (e.g., job launch, signals, termination). If the number of concurrent jobs approaches or exceeds the default connection limit, slurmctld will time out trying to reach slurmd, causing jobs to become stuck in the COMPLETING state until slurmctld retries successfully. Sites should set this to at least the number of jobs expected to run concurrently on a single node. The same memory trade-off described under SlurmctldParameters=conmgr_max_connections applies here.
    NOTE: When SlurmctldParameters=enable_stepmgr is set, slurmctld manages step lifecycles directly, which increases the number of connections to each slurmd and makes tuning this parameter more important.
- SlurmdLogFile: Writing to local storage is recommended.
- The ability to do RPC rate limiting on a per-user basis is a new feature with 23.02. It acts as a virtual bucket of tokens that users consume with Remote Procedure Calls. This allows users to submit a large number of requests in a short period of time, but not a sustained high rate of requests that would add stress to the scheduler. You can define the maximum number of tokens with rl_bucket_size, the rate at which new tokens are added with rl_refill_rate, the frequency with which tokens are refilled with rl_refill_period and the number of entities to track with rl_table_size. It is enabled with rl_enable.
- Other: Configure logging, accounting and other overhead to a minimum appropriate for your environment.

## SlurmDBD Configuration

Turning accounting off provides a minimal improvement in performance. If using SlurmDBD increased speedup can be achieved by setting the CommitDelay option in the slurmdbd.conf to introduce a delay between the time slurmdbd receives a connection from slurmctld and when it commits the information to the database. This allows multiple requests to be accumulated and reduces the number of commit requests to the database.

High job throughput leads to the database growing much faster than in a typical environment, potentially to a size that becomes difficult to manage depending on the capabilities of the database server. Sites are strongly encouraged to make a plan for data retention early on and to configure the relevant Purge options to keep the database files at a manageable size. Also consider the frequency of these purges as explained in the linked documentation. Alternately, sites may prefer to skip the storage of steps or jobs entirely through the relevant AccountingStorageEnforce options (nosteps and nojobs).