NUMA Calculation in SAP HANA and Setting NUMA Parameters

HANA SIZINGHANA INSTALLATIONHANA SIZING ON SUSESAP HANA ADMINISTRATIONSAP HANAHANA TROUBLESHOOTING

Biswa Ranjan

5/23/20244 min read

Non-Uniform Memory Access (NUMA) is a memory design used in multiprocessor systems where the memory access time depends on the memory location relative to the processor. NUMA is commonly implemented in large-scale SAP HANA deployments to optimize memory access and performance. Here's a detailed explanation of how NUMA works in the context of an SAP HANA database and how to calculate NUMA-related parameters:

Understanding NUMA in SAP HANA

Architecture: In a NUMA system, each processor is connected to its local memory, and the entire system is divided into multiple nodes. Processors can access both local and remote memory, but accessing local memory is faster due to lower latency compared to remote memory.
Performance Impact: SAP HANA is an in-memory database, meaning it heavily relies on fast memory access. Proper NUMA configuration ensures that memory allocation and processor usage are optimized to minimize latency and maximize throughput.
Configuration: SAP HANA needs to be configured to be NUMA-aware, ensuring that workloads are distributed across nodes in a balanced manner. This involves setting up appropriate memory and CPU affinity.

Calculating NUMA Parameters in SAP HANA

To calculate NUMA-related parameters and ensure optimal configuration, follow these steps:

1. Identify NUMA Nodes

Identify the number of NUMA nodes and the CPU and memory allocation for each node. This can be done using system commands.

On Linux, use the numactl --hardware command to list NUMA nodes and their respective CPUs and memory.

2. SAP HANA Configuration Parameters

Adjust the following SAP HANA parameters to align with your NUMA architecture:
Global Allocation Limit: Ensure that the global allocation limit is set considering the total memory available across all NUMA nodes.
CPU Allocation: Use the cpu_affinity parameter to bind SAP HANA processes to specific CPUs on each NUMA node.
Memory Allocation: Use the memory_affinity parameter to bind SAP HANA memory allocation to specific NUMA nodes.

3. Memory Distribution

Ensure that the memory is distributed evenly across all NUMA nodes. This can be checked and adjusted using HANA studio or command line tools.

4. NUMA Balancing

Enable NUMA balancing to allow the operating system to automatically optimize memory allocation and reallocation. This can be enabled through the Linux kernel settings.

Example Configuration

Suppose you have a system with 2 NUMA nodes, each with 16 CPUs and 256 GB of memory.

Check NUMA Configuration: numactl --hardware
1. Configure SAP HANA Parameters:
  - Set the global allocation limit to 512 GB (sum of both nodes' memory).
  - Set CPU affinity:
  - ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('cpu', 'cpu_affinity') = '0-15,16-31' WITH RECONFIGURE;
  - Set memory affinity:
  - ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('memory', 'memory_affinity') = '0,1' WITH RECONFIGURE;
2. Enable NUMA Balancing:
  echo 1 > /proc/sys/kernel/numa_balancing

CPU Affinity:

In SAP HANA, the cpu_affinity parameter is used to bind specific SAP HANA processes to particular CPU cores, ensuring that the workloads are efficiently distributed across the available CPUs in a NUMA system. Proper configuration of CPU affinity can significantly enhance performance by reducing context switching and ensuring that processes run on the most appropriate cores, taking advantage of local memory.

Understanding CPU Affinity in SAP HANA

CPU Affinity Basics: CPU affinity binds processes to specific CPU cores. This ensures that a process runs on the same core(s) every time it is scheduled to run, which can improve cache utilization and reduce latency.
NUMA Considerations: In a NUMA system, each CPU has its local memory. Configuring CPU affinity helps in utilizing local memory effectively by keeping the processes on the CPUs closer to their allocated memory.

Configuring CPU Affinity in SAP HANA

Step-by-Step Configuration

Identify CPU and NUMA Node Layout:
- Use the numactl --hardware command to identify the NUMA nodes and associated CPUs.
  numactl --hardware
  Example output:
  available: 2 nodes (0-1)
- node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
- node 0 size: 256 GB
- node 0 free: 128 GB
- node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
- node 1 size: 256 GB
- node 1 free: 128 GB
Modify CPU Affinity Configuration:
Use the SQL command to set the CPU affinity for SAP HANA processes. This can be done via the SAP HANA SQL Console or any SQL client connected to the HANA database.
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('cpu', 'cpu_affinity') = '0-15' WITH RECONFIGURE;
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('cpu', 'cpu_affinity') = '16-31' WITH RECONFIGURE;
In this example, the first command binds HANA processes to CPUs 0-15 (node 0), and the second command binds other HANA processes to CPUs 16-31 (node 1).
Restart SAP HANA:
- For the changes to take effect, you might need to restart the SAP HANA instance.
  HDB stop
- HDB start

Monitoring and Adjustments

After configuring CPU affinity, it’s essential to monitor the performance and adjust settings as needed. Use SAP HANA monitoring tools and Linux performance monitoring tools to ensure that the load is balanced and performance is optimized.

SAP HANA Studio: Use the Performance tab to monitor CPU and memory usage.
Linux Tools: Use tools like htop, top, and numastat to monitor CPU usage and memory distribution across NUMA nodes.

Summary

Configuring cpu_affinity in SAP HANA involves binding HANA processes to specific CPU cores, ensuring efficient use of CPUs and local memory in a NUMA architecture. This configuration can significantly improve the performance of SAP HANA by optimizing cache usage and reducing memory latency. Regular monitoring and adjustments are essential to maintain optimal performance.