Reading RAPL energy measurements from Linux

RAPL support

I also have a list of processors and when Linux added support for their power interfaces.

RAPL validation

I also have a page describing my RAPL validation work.

RAPL on Linux

There was at one point *four* ways to read RAPL results on Linux:

Reading the files under /sys/class/powercap/intel-rapl/intel-rapl:0 using the powercap interface. This required no special permissions, and was introduced in Linux 3.13. Despite my pointing out lack of permissions was a problem at the time of introduction, nothing was done about this until Linux 5.10 where the security issue was noticed (949dd0104c496fa7c14991a23c03c62e44637e71) and limitations were added.
Using the perf_event interface with Linux 3.14 or newer. This requires root or a paranoid less than 1 (as do all system wide measurements with -a) sudo perf stat -a -e "power/energy-cores/" /bin/ls Available events can be found via perf list or under /sys/bus/event_source/devices/power/events/
Using raw-access to the underlying MSRs under /dev/msr. This requires root. Some HPC tools will attempt to use this interface to bypass the kernel, but the kernel developers want to make raw MSR access go away as there are a lot of questionable things you can do to a system with raw MSR access. There have been attempts to make alternate drivers that whitelist the MSRs that can be used but they are probably going away in the future too.
An AMD RAPL driver without premissions checks was available under the kernel hwmon interface, but it was removed in Linux 5.13 (9049572fb) also due to security concerns.

Note that you cannot get readings for individual processes, the results are for the entire CPU socket.

Userspace Access without perf

Here is some quick code to read the RAPL values without the perf tool. It can read all three interfaces (syfs powercap, perf_event, or msr) depending on the command line options.

You need to either run as root or have the /dev/cpu/*/msr permissions set for msr access.

The sample program produces results like the following.

Power units = 0.125W
Energy units = 0.00001526J
Time units = 0.00097656s

Package thermal spec: 130.000W
Package minimum power: 51.000W
Package maximum power: 200.000W
Package maximum time window: 0.046s

Package energy before: 48460.887909J
PowerPlane0 (core) for core 0 energy before: 36127.280838J
DRAM energy before: 0.000000J

Sleeping 1 second

Package energy after: 48468.194504  (7.306595J consumed)
PowerPlane0 (core) for core 0 energy after: 36128.297287  (1.016449J consumed)
DRAM energy after: 0.000000  (0.000000J consumed)

Note: the energy measurements can overflow in 60s or so
      so try to sample the counters more often than that.

Source code: rapl-read.c

Also available from: https://github.com/deater/uarch-configure

PAPI

PAPI can read the RAPL values in various ways. We are working on consolidating them into one component rather than having multiple confusing ways of gathering the info.

Trying all RAPL events
Found rapl component at cid 2

Starting measurements...

Doing a naive 1024x1024 MMM...
Matrix multiply sum: s=1016404871450364.375000

Stopping measurements, took 4.110s, gathering results...

Energy measurements:
PACKAGE_ENERGY:PACKAGE0   176.450363J   (Average Power 42.9W)
PACKAGE_ENERGY:PACKAGE1    75.812454J   (Average Power 18.4W)
DRAM_ENERGY:PACKAGE0       11.899246J   (Average Power 2.9W)
DRAM_ENERGY:PACKAGE1        8.341141J   (Average Power 2.0W)
PP0_ENERGY:PACKAGE0       118.029236J   (Average Power 28.7W)
PP0_ENERGY:PACKAGE1        16.759064J   (Average Power 4.1W)

Fixed values:
THERMAL_SPEC:PACKAGE0        135.000W
THERMAL_SPEC:PACKAGE1        135.000W
MINIMUM_POWER:PACKAGE0        51.000W
MINIMUM_POWER:PACKAGE1        51.000W
MAXIMUM_POWER:PACKAGE0       215.000W
MAXIMUM_POWER:PACKAGE1       215.000W
MAXIMUM_TIME_WINDOW:PACKAGE0   0.046s
MAXIMUM_TIME_WINDOW:PACKAGE1   0.046s
rapl_basic.c                           PASSED

RAPL Tool

This tool uses PAPI to poll a machine's RAPL energy usage and dump the results to various files which can then be plotted.

Here is some sample results generated using the tool:

And here is the tool. To use it you'll need to modify the Makefile to build against a recent version of PAPI compiled with --with-components="rapl".

rapl_plot.tar.gz

Documentation

More details on the RAPL interface can be found in the Intel Dev manuals, Volume 3.

For notes on possibly measuring these values on Windows see here

Back to my projects page