Kernel KnowHow: Per CPU Variables

I just read the section on per CPU variables from the Linux Device Drivers book from O'Reilly (TODO link).

The use of per CPU variable helps improving performance by limiting the cache sharing between CPUs. The idea is that each CPU has his own private instance of a given variable.

A variable can be defined using the following macro: DEFINE_PER_CPU(type,name);

It is important to note that a kind of locking mechanism is still needed for the CPU variables in case:

• the processor would move the process to another cpu
• the processor would be preemted in the middle of a modification of the variable

Therefore, each instance should be updated using a simple mechanism to lock the variable during the changes. This can be done by using the get_cpu_var() and put_cpu_var() functions, for example:

get_cpu_var(variabletoupdate)++;
put_cpu_var(variabletoupdate);

It is also possible to access the variable values from other processors using: per_cpu(variable, int cpu_id); but a locking mechanism must be implemented for these cases.

Memory allocation for dynamically allocated per-CPU variables is performed using:

void *alloc_percpu(type);
void *__alloc_percpu(size_t size , size_t align);

Deallocation is achieved using: free_percpu().

Accessing a dynamically allocated per-CPU variable is performed by using

per_cpu_ptr(void * per_cpu_var, int cpu_id).

It returns a pointer to the variable content. For example,

int cpu;
cpu= get_cpu();
ptr = per_cpu_ptr(per_cpu_var, cpu);
put_cpu();

Note: the use of the method to free the cpu from the lock.

SMP and NUMA architectures

I have had a look at the SMP and NUMA architectures which are for multiprocessor systems.

I found a number of interesting resources:

• OpenMP.org

SMP means symmetric multiprocessor while NUMA stands for Non-Uniform Memory Access. Actually SMP is a type of UMA, i.e Uniform Memory Access. Basically, SMP processor share all the same memory, this imposes some overhead on the cache of the processor to speed up the processing. NUMA architectures (of which different flavor exist, in particular cache-coherent models, aka ccNUMA) have a non uniform access to memory, some processor have access to local caches. In that way, the processes do not need to synchronize to access the data.