perf_event ABI breakage

perf_event has a unique design philosophy. Part of that involves having the source code for the flagship perf tool included in the Linux kernel source tree. The claim is that this will enhance the ABI. In practice the ABI is frequently broken, and as long as the userspace perf tool still works none of the kernel developers seem to notice or care.

The sysfs layout is particularly troublesome. Some of the files are documented in Documentation/ABI/testing/ but since it is "testing" the developers don't feel the need to keep things stable.

What follows is a list of recent ABI breakage. I haven't included things from the early pre-2.6.34 days (perf support started with 2.6.31) as any new interface takes a while to shake out. Those early kernels did have problems though, and PAPI still carries the workaround for them.

4.0

Before, a value of "0" in /sys/devices/cpu/rdpmc meant rdpmc support was disabled, and "1" meant enabled for everyone. Starting with 4.0 "1" means only processes with active events can read it, and "2" means enabled for everyone.

Introduced-by: a66734297f78707ce39d756b656bfae861d53f62 7911d3f7af14a614617e38245fedf98a724e46a9

Discussion thread: 16 April 2015: [patch 10/10] perf_event_open.2: 4.0 update rdpmc documentation

3.18

Before, when polling, code would get a result=0/WIFEXITED() return when a child with events being polled exited. Now instead I get a result=1/POLLHUP return.

Introduced-by: 179033b3e064d2cd3f5f9945e76b0a0f0fbf4883

Programs broken: Some perf_event_test testcases

Discussion thread: https://lkml.org/lkml/2015/1/20/716

3.14

PERF_EVENT_IOC_PERIOD changes take effect immediately rather than after next stop/start. ARM had different behavior than the rest of the architectures from 3.7 until 3.14 (i.e. 3581fe0ef37ce12ac7a4f74831168352ae848edc to 9450d14fb959336803e5209119eb422b667b96aa

Introduced by: bad7192b842c83e580747ca57104dd51fe08c223

Programs broken: ??

Discussion thread: https://lkml.org/lkml/2013/10/27/215

3.10

In /sys/bus/event_source/bus/devices/*/events/ files started appearing having decimal (not hex) values, contrary to the Documentation/ABI/testing/ documentation.

Introduced-by: f9134f36aed59ab55c0ab1a4618dd455f15aef5f

Programs-broken: trinity

Discussion thread: lkml
Move of AMD Fam15h Northbridge events from being part of the generic CPU events to being a separate PMU.

Introduced-by: c43ca5091a374c1f6778bd7e4a39a5a10735a917, feature originally added in e259514eef764a5286873618e34c560ecb6cff13.

Programs-broken: PAPI, libpfm4

Discussion-thread: lkml

Notes: Support for NB was only added in 3.9 so the hope was no one would notice. But now PAPI has to carry support for both possibilities, forever. Peter Zijlstra apologizes, blames pooe judgement due to illness.

3.7

The PERF_EVENT_IOC_PERIOD ioctl as originally designed updates the period on the *next* refresh. In 3.7 in commit 3581fe0ef37ce12ac7a4f74831168352ae848edc this was changed on ARM to update on the *current* refresh.

See the 3.14 changes for when this was "fixed" on the rest of the architectures.

3.3

Raw access to Offcore Response events added. This support was previously disabled (offcore response was merged in 2.6.39 but raw access was disabled) but indicated success even if support not there. Ther interface would even return plausible garbage values as results too. So once support was finally added in 3.3 it was impossible to tell if support was working or not.

Commit-that-disabled-support:b52c55c6a25e4515b5e075a989ff346fc251ed09

Commit-that-re-enabled-support:To find

Discussion-threads: LWN article, lkml, lkml

Programs-broken: PAPI

3.2

If there was not enough room for the event, ENOSPC was returned. Linus did not like this, and this was changed to EINVAL. ENOSPC is still returned if you try to read results into too small of a buffer.

Introduced-by: aa2bc1ade59003a379ffc485d6da2d92ea3370a6

Discussion-threads: To find? The link in the commit is self-referential.

3.0

Prior to this, an exec'd process with active perf_event overflows did not get signals. After, it did.

Introduced-by: f506b3dc0ec454a16d40cab9ee5d75435b39dc50

Mailing list discussions: PAPI, lkml

Programs-broken: PAPI
PERF_IOC_REFRESH behavior What events count as POLL_IN vs POLL_HUP changed What happens when you PERF_IOC_REFRESH other than 1 changed

Discussion-threads: lkml

Programs broken: PAPI

2.6.38

The file /sys/devices/system/cpu/perf_events was used by some to detect the existence of perf_event support. It was removed.

Introduced-by: 15ac9a395a753cb28c674e7ea80386ffdff21785

Discussion-thread: lkml Programs-broken: Various internal tools

Notes: This is when /proc/sys/kernel/perf_event_paranoid was made the official ABI-sanctioned way of detecting perf_event.

2.6.37

Starting in 2.6.37 the nmi watchdog can steal an event counter. If you try to use the full number of counters, perf_event won't notice that NMI is stealing one until start time, not at perf_event_open() time. Eventsets that used to work now do not and an extra test has to be done before telling the user an EventSet will work.

Programs-broken: PAPI

Notes:Unlikely to ever be fixed.

2.6.34

Context switch events were reported as happening in userspace before, now reported as kernelspace.

Introduced-by: e49a5bd38159dfb1928fd25b173bc9de4bbadb21