Linux Kernel perf_event bugs

Status as of last time I extensively Ran Things

• ARM32 -- chugs along for days on a pi2 (4.8-rc2)
• x86/intel -- multiple warnings. Many current issues are with BTS support
  • pentium4 -- 5.0 crashes more or less immediately
  • core2 -- 4.9-rc1 crashes paranoid 1 and lower
  • haswell -- 5.0 - doesn't crash? (still some warnings)
  • skylake -- 5.2 - doesn't crash? (still some warnings)
• x86/amd -- 4.9-rc7 -- chugs along for over a week?
• sparc -- newest kernel I can run is 3.2, but locks up more or less immediately

Current Crashes (reproducible):

• 5.11-rc5 PEBS crash
• 4.8-rc2 amd_nb BUG
• 4.4-rc4 AMD IBS crash
• 4.4-rc4 ARM/ARM64 RCU stall
• 3.19-rc5 haswell pmu_del() warning/lockup
• 3.17-rc7 put_ctx lockup
• 3.17-rc6 do_IRQ overflown stack

Current Crashes (not reproducible):

• (crash) 4.20-rc5 General Protection fault, BTS related
• (crash) 4.1.0-rc2 NULL pointer in constraint table
• (crash) 4.1.0-rc1-next ctx_lock deadlock
• (crash) 4.0-rc1+ x86_pmu_event_init() hang
• (crash) 4.1-rc0 x86_schedule_events

Current Crashes (unknown if reproducible):

• (crash) 4.9-rc0 pentium 4 NMI
• (bug) 4.8-rc1 spinlock bug
• (crash) 3.19-next perf_event_mmap() gpf
• (crash) 3.2-sparc.bad_page

Current Warnings (reproducible):

• (warning) 4.9-rc5 KASAN stack-out-of-bounds in unwind_next_frame
• (warning) 4.9-rc1 overlap scheduling
• (warning) 4.9-rc1 uncore_get_constraint lockdep
• (warning) 4.8-rc1 event function local warning
• (warning) 4.8-rc1 irq loop Warning
• (warning) 4.8-rc1 x86_pmu_start_warning
• (warning) 4.5-rc0 breakpoint warning
• (warning) 4.2-rc1 kprobe warnings
• (warning) 4.2-rc0 perf_aux_output_begin()
• (warning) 4.2-rc0 nmi_flood messages
• (warning) 3.15-rc4 p4 x86_pmu_start/stop warnings

Current Warnings (unknown if reproducible):

• (warning) 4.9-rc5 rb_free_aux
• (warning) 4.9-rc5 perf_mmap_close.html
• (warning) 4.9-rc0 x86_pmu_stop_warning
• (warning) 4.8-rc3 haswell perf_read
• (slab corruption) 4.8-rc1 haswell slab corruption
• (GPF) 4.8-rc1 haswell disable gpf
• (warning) 4.4-rc4 intel_pmu_drain_pebs_nhm_warning.html
• (warning) 4.1-rc2 kill_fasync triggers watchdog

Current correctness Issues:

Older Issues. May have been fixed; have not encountered recently.

Older Crashes (not easily reproducible):

• 3.15-rc7 get_cpu_context GPF
• 3.15-rc-next perf_swevent_del bug
• 3.14-rc5.perf_event_overflow bug
• 3.15-rc-next KVM perf_mmap corruption
• 3.15-rc-next perf_remove_from_context() gpf
• 3.14-rc8 perf_event_exit_task lockdep
• 3.14-rc1 x32 oops reboot in filp
• 3.13 task_output lockup
• 3.12 perf ext4 vma error

Older Warnings (not easily reproducible):

• 3.15-rc5 mutex_lock_nested
• 3.15-rc4 perf_event.c:1076 Warning x86_pmu_start()
• 3.15-rc2 kernel/events/core.c:2384 Warning
• 3.14 perf_event.c:1158 Warning
• 3.14-rc kernel/event/core:5659 warning

Un Fixed bugs:

Fixed bugs:

Fixed in 5.6?

• (warning) x86/entry/64: Fix unwind hints in kernel exit path WARNING: can't dereference registers at 000000003aeb0cdd for ip swapgs_restore_regs_and_return_to_usermode+0x93/0xa0
• (warning) objtool: Fix stack offset tracking for indirect CFAs WARNING: can't dereference registers at 00000000f0a6bdba for ip interrupt_entry+0x9f/0xa0

Fixed in 5.5

• (correctness) Introduced in 5.4. If aux_sample_size set improperly, an event will fail to create but perf_event_open() will return 0 Reported 2 January 2020 Fixed by: da9ec3d3dd0f1240a48920be063448a2242dbd90
• (oops/null-pointer dereference) i915 driver get_timeline_name+0x13/0x20 [i915] Reported 12 December 2019 Fixed by: 242bff7fc515d8e5275e5b8cd8c9c85a8d037dbf

Fixed in 5.2

• (warning) perf/x86/regs: Check reserved bits With the addition of XMM bits, some bits that were reserved weren't checked. Fixed by 90d424915ab6550826d297fd62df8ee255345b95
• (crash) perf/x86: Disable extended registers for non-supported PMUs Fixed by e321d02db87af7840da29ef833a2a71fc0eab198
• (crash) perf/ioctl: Add check for the sample_period value causes crash on PowerPC Fixed by 913a90bc5a3a06b1f04c337320e9aeee2328dd77

Fixed in 5.0

• (crash) perf/x86: Add check_period PMU callback
  Fixed by 81ec3f3c4c4d78f2d3b6689c9816bfbdf7417dbb
  Last easy to trigger x86/Intel bug?
• (warning) perf/core: Don't WARN() for impossible ring-buffer sizes Found with perf_fuzzer by Mark Rutland
  Fixed by 9dff0aa95a324e262ffb03f425d00e4751f3294e

Fixed in 4.20

• (crash) 4.20-rc5 BUG: unable to handle kernel NULL pointer BTS
  Fixed by 472de49fdc53365c880ab81ae2b5cfdd83db0b06

Fixed in 4.19

• (warning) powerpc/perf: Remove sched_task function defined for thread-imc
  Found with perf_fuzzer by Anju T Sudhakar
  Fixed by 7ccc4fe5ff9e3a134e863beed0dba18a5e511659
• (warning) perf/x86/intel: Fix unwind errors from PEBS entries (mk-II)
  Found with perf_fuzzer by Vince Weaver
  Fixed by 6cbc304f2f360f25cc8607817239d6f4a2fd3dc5

Fixed in 4.17

• Fix sample_max_stack maximum check
  Found with syzkaller syzbot
  Fixed by: 5af44ca53d019de47efe6dbc4003dd518e5197ed
• (warning?) armv8, perf/core: Fix perf_output_read_group()
  Found by Mark Rutland with perf_fuzzer
  Fixed by: 9e5b127d6f33468143d90c8a45ca12410e4c3fa7

Fixed in 4.16

• Subject [PATCH 0/1] x86/kprobes: Prohibit probing of .entry_trampoline code
  Found by: Francis Deslauriers
  Mentions found with fuzzer, did not say which one.

Fixed in 4.15

• Change to kernel left Haswell-EX using wrong msr registers for RAPL.
  Found with perf_fuzzer by Kan Liang.
  Fixed by 1289e0e29857e606a70a0200bf7849ae38d3493a
• The KPTI (meltdown) fixes led to various issues. Intel BTS in particular led to instant crashes. Temporarily fixed by 99a9dc98ba52267ce5e062b52de88ea1f1b2a7d8

Fixed in 4.13

• pmu::read() called erroneously in v4.13-rc{3,4} Found with perf_fuzzer by Mark Rutland on ARM64. https://lkml.org/lkml/2017/8/10/742 Fix never committed for this? Double check.

Fixed in 4.11

• perf/core: Fix use-after-free in perf_release() e552a8389aa409e257b7dcba74f67f128f979ccc Found with syzkaller by Dmitry Vyukov
• perf/core: Fix the perf_cpu_time_max_percent check 1572e45a924f254d9570093abde46430c3172e3d Found with perf_fuzzer by Tan Xiaojun

Fixed in 4.10

• perf: Fix concurrent sys_perf_event_open() vs. 'move_group' race 321027c1fe77f892f4ea07846aeae08cefbbb290 CVE-2017-6001 While not explicitly said to be found with a fuzzer, it's definitely the kind of bug a fuzzer makes easy to find. Didn't ask for confirmation as this is another one of those mysterious hackers who doesn't believe in e-mail but is OK having a twitter account.
• perf/x86/intel: Account interrupts for PEBS errors Found by Jiri Olsa (475113d937adfd150eb82b5e2c5507125a68e7af)
• perf/x86: Fix overlap counter scheduling bug Found by Jiri Olsa, presumaby with perf_fuzzer (1134c2b5cb840409ffd966d8c2a9468f64e6a494)

Fixed in 4.9

• (warning) 4.9-rc5 KASAN global-out-of-bounds in match_token (e96271f3ed7e702fa36dd0605c0c5b5f065af816)
• powerpc: Don't call perf_event_disable() from atomic context (trinity) (5aab90ce1ec449912a2ebc4d45e0c85dac29e9dd)
• (kasan) 4.9-rc5 BUG: KASAN: slab-out-of-bounds in snb_uncore_imc_event_del (c499336cea8bbe15554c6fcea2138658c5395bfe)
• 4.9-rc1 vmalloc stack unwinder crash (7fbe6ac02485504b964b283aca62b36b4313ca79)

Fixed in 4.8

• (crash) 4.8-rc0 perf_cgroup_attach (0b8f1e2e26bfc6b9abe3f0f3faba2cb0eecb9fb9)

Fixed in 4.6

• (minor) perf/core: Fix dynamic interrupt throttle (91a612eea9a316c464cc170ff8492ec09e7d1c69)

Fixed in 4.5

• (panic) hw_breakpoint: Fix Oops at destroying hw_breakpoint event on powerpc (fb822e6076d972691c5dd33431)

Fixed in 4.2

• (warning) intel_pmu_lbr_disable_warning.html
• (crash) 4.2-rc0 AUX related spinlock recursion
• (warning) 4.1-rc8 trace replace_preds()

Fixed in 4.1

• (warning) 4.1-rc7 trace replace_preds()
• (crash) 3.19.0+ snb_uncore_imc_event_start crash
• (crash) 4.0-rc2 arm/arm64 validate group spanning PMUs CVE-2015-8955

Fixed in 4.0

• (crash) 4.0-rc1 perf_callchain() hang
• (mem leak) 4.0-rc1 mem leak in put_event() ctx leak fix
• (warning) 3.19-next add_event_to_context() warning

Fixed in 3.19

• (crash) 3.19-rc6 perf_remove_from_context() hang
• (crash) 3.19-rc5 find_get_context() hang
• (crash) 3.19-rc5 x86_schedule_events
• (crash) 3.19-rc5 rapl_scale_crash
• (warning) 3.15-rc4 breakpoint warning
• (bug) 3.17-rc IVB-EP uncore assign events BUG

Fixed in 3.18

• (bug) 3.17-next issue caught by trinity

Fixed in 3.17

• (reboot) 3.17-rc4 insta-reboot

Fixed in 3.16

• (correctness) No error on invalid high bits in flags

Fixed in 3.15

• (crash) perf/fork memory leak issue
• (crash) 3.15-rc4 perf_event_init_context() oops
• (correctness) sample_period 64-bit signed/unsigned overflow

Fixed in 3.14

• (segfaults) double-fault/corrupted cr2 causing weird segfaults
• (warning) long-standing perf_event.c:1076 WARNING

Fixed in 3.13

• perf ftrace function tracer permission lockup
  
  perf/ftrace: Fix paranoid level for enabling function tracer
  
  This one took months to isolate and track down. ftrace function tracing can spend so much time in the kernel that the kernel gets wedged and for all intents and purposes locks up. The ftrace people dragged their feet on this one for months.
  
  Reported-by: Vince Weaver, Dave Jones
  
  Fixed-by-Commit: 12ae030d54ef250706da5642fc7697cc60ad0df7 (in 3.13)
  
  CVE-2013-2930
  
  Found by trinity and perf_fuzzer, it is trivially easy for a plain user to lock the kernel due to an improper check for root permissions in the ftrace code.
  
  
• Tracepoint aliasing issue
  
  Linux-kernel: 15 November 2013 -- [patch] perf/trace properly use u64 to hold event_id
  
  Reported-by: Vince Weaver (noticed in fuzzer trace)
  
  Fixed-by-Commit: 0022cedd4a7d8a87841351e2b018bb6794cf2e67 (in 3.13-rc3)
  
  
• Alpha bug found by perf_fuzzer
  
  alpha: perf: fix out-of-bounds array access triggered from raw event
  
  Reported-by: Will Deacon
  
  Fixed-by-Commit: 6e22f8f2e8d81dcab4c40bc229d53388fda63dbc (in 3.13)
  
  

Fixed in 3.11

• ARM: 7810/1: perf: Fix array out of bounds access in armpmu_map_hw_event()
  
  Found by perf_fuzzer.
  
  Reported-by: Vince Weaver
  
  Fixed-by-Commit: d9f966357b14e356dbd83b8f4a197a287ab4ff83
  
  Introduced with Linux 3.2 with 8a16b34e2119
  
  
• CVE 2013-4254: ARM and ARM64 oops in validate event
  
  ARM: 7809/1: perf: fix event validation for software group leaders
  
  Reported-by: Vince Weaver
  
  Fixed-by-Commit: c95eb3184ea1a3a2551df57190c81da695e2144b
  
  Found by perf_fuzzer.
  
  This bug is potentially a local root exploit, but as far as I can tell there was an extremely narrow window of time where it was easy to exploit (from 3.11-rc1 through 3.11-rc6)
  
  Introduced with Linux 3.2 with 8a16b34e2119
  
  
• arm64: perf: fix array out of bounds access in armpmu_map_hw_event()
  
  Reported-by: Will Deacon
  
  Fixed-by-Commit: 868f6fea8fa63f09acbfa93256d0d2abdcabff79
  
  
• arm64: perf: fix event validation for software group leaders
  
  Reported-by: Will Deacon Fixed-by-Commit: ee7538a008a45050c8f706d38b600f55953169f9
  
  
• WARNING: at kernel/events/core.c:2122

Fixed in 3.10

• perf: Fix mmap() accounting hole
  
  Reported-and-tested-by: Vince Weaver
  
  Fixed-by-Commit: 9bb5d40cd93c9dd4be74834b1dcb1ba03629716b
  
  Denial of service.
  
  Found by perf_fuzzer, it happens if you do some complicated event setup involving mmap buffers and then exiting.
  
  Introduced sometime between 3.0 and 3.2?
  
  
• perf: Fix perf mmap bugs
  
  Reported-by: Vince Weaver
  
  Fixed-by-Commit: 26cb63ad11e04047a64309362674bcbbd6a6f246
  
  

Fixed in 3.9

• CVE-2013-2094 : Software Event u64/u32 Problem
  
  Local root exploit.
  
  Found by trinity, this panic is caused by setting attr.config too high for software events, and it wasn't caught due to a 32/64-bit check?
  
  Introduced in commit b0a873ebb ("perf: Register PMU implementations").
  
  Patch that fixes is: [PATCH] perf: treat attr.config as u64 in perf_swevent_init() vulnerable through 3.9
  
  Fixed in 8176cced706b5e5d15887584150764894e94e02f
  
  
• offcore_response_mask
  
  Denial of service
  
  On Sandybrige and Ivybridge the offcore events were not properly masked to avoid setting reserved bits. This could potentially cause a GPF but I have not been able to cause one with a 3.8 kernel.
  
  Fixed in f1923820c447e986a9da0fc6bf60c1dccdf0408e
  
  
• perf,x86: fix kernel crash with PEBS/BTS after suspend/resume
  
  Denial of service
  
  Fixed in 1d9d8639c063caf6efc2447f5f26aa637f844ff6
  
  The test involves suspending/resuming while perf is being used, and there's not a good way to automate such a test.
  
  

Fixed in 3.5 (pre-dates perf_fuzzer)

• CGROUP reference counting problem
  
  Fixed by changeset: 9c5da09d266ca9b3

Fixed in 3.2 (pre-dates perf_fuzzer)

• SLUB problem
  
  Linux 3.1.4 and 3.1.5 hard lock or panic when under high perf_event load if SLUB is enabled rather than SLAB. See full details in this thread: perf_event hard locks in 3.1.5. Still no resolution; git-bisect was inconclusive.
  
  

Fixed in 3.1 (pre-dates perf_fuzzer)

• CVE 2011-2918: Software Event Overflow Bug
  
  Denial of service.
  
  An error with overflows and perf::perf_count_sw_cpu_clock (and probably any software event) will cause your test to either become unkillable, or else hard lock your system. Sometimes generates a WARN_ON first. Affects Linux 2.6.32 - 3.0.0.
  
  Linux-kernel thread where I report this issue.
  
  Fixed by changeset a8b0ca17b80e92faab46ee7179ba9e99ccb61233
  
  Also backported to some stable releases, 462fee3af72df0de7b60b96c525ffe8baf4db0f0.
  
  

Fixed in 2.6.39 (pre-dates perf_fuzzer)

• CVE-2011-4611 : perf, powerpc
  
  Denial of service.
  
  Fixed in 0837e3242c7
  
  
• CVE-2011-2521 : perf, x86: fix Intel fixed counters base initialization
  
  Denial of service
  
  Introduced by 41bf498, fixed by fc66c5210ec2
  
  
• Inherit Bug
  
  Denial of service
  
  With this bug you could quickly out-of-memory your system if your program enabled the "inherit" option, counted at least two events, and spawned threads.
  
  Linux kernel thread where I report the issue.
  
  Fixed by changeset 38b435b16c36b0d863efcf3f07b34a6fac9873fd
• Task Context Scheduling Bug
  
  Denial of service
  
  This bug is most noticable when using multiple threads. The perfsuite "make -s check" test failure is what made me notice it. Symptoms were hard system lockup.
  
  Fixed by changeset ab711fe08297de1485fff0a366e6db8828cafd6a

Fixed in 2.6.37 (pre-dates perf_fuzzer)

• CVE-2010-4169
  
  Denial of service.
  
  Introduced by dab5855, fixed by 63bfd7384b1
  
  

Fixed in 2.6.32 (pre-dates perf_fuzzer)

• CVE-2009-3234 Buffer overflow in the perf_copy_attr
  
  Local root exploit.
  
  Fixed in b3e62e350
  
  

• CVE-2017-6001 (???)
• CVE-2017-0403 Android (???)
• CVE-2016-6786 Android (???)
• CVE-2016-6787 Android (???)
• CVE-2016-3843 Qualcomm (???)
• CVE-2016-3768 Qualcomm (???)
• CVE-2016-0843 Qualcomm (???)
• CVE-2016-0819 Qualcomm (likely perf_fuzzer)
• CVE-2016-0805 Qualcomm (likely perf_fuzzer)
• CVE-2015-8955 (perf_fuzzer)
• CVE-2015-8963 (???)
• CVE-2013-2930 (perf_fuzzer)
• CVE 2013-4254 (perf_fuzzer)
• CVE-2013-2094 (trinity)
• CVE 2011-2918
• CVE-2011-4611
• CVE-2011-2521
• CVE-2010-4169
• CVE-2009-3234