name : stacksnoop_example.txt Demonstrations of stacksnoop, the Linux eBPF/bcc version. This program traces the given kernel function and prints the kernel stack trace for every call. This tool is useful for studying low frequency kernel functions, to see how they were invoked. For example, tracing the submit_bio() call: # ./stacksnoop submit_bio TIME(s) SYSCALL 3592.838736000 submit_bio submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread ret_from_fork This shows that submit_bio() was called by submit_bh(), which was called by jbd2_journal_commit_transaction(), and so on. For high frequency functions, see stackcount, which summarizes in-kernel for efficiency. If you don't know if your function is low or high frequency, try funccount. The -v option includes more fields, including the on-CPU process (COMM and PID): # ./stacksnoop -v submit_bio TIME(s) COMM PID CPU SYSCALL 3734.855027000 jbd2/dm-0-8 313 0 submit_bio submit_bio submit_bh jbd2_journal_commit_transaction kjournald2 kthread ret_from_fork This identifies the application issuing the sync syscall: the jbd2 process (COMM column). Here's another example, showing the path to second_overflow() and on-CPU process: # ./stacksnoop -v second_overflow TIME(s) COMM PID CPU SYSCALL 3837.526433000 <idle> 0 1 second_overflow second_overflow tick_do_update_jiffies64 tick_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt default_idle arch_cpu_idle default_idle_call cpu_startup_entry start_secondary 3838.526953000 <idle> 0 1 second_overflow second_overflow tick_do_update_jiffies64 tick_irq_enter irq_enter smp_apic_timer_interrupt apic_timer_interrupt default_idle arch_cpu_idle default_idle_call cpu_startup_entry start_secondary This fires every second (see TIME(s)), and is from tick_do_update_jiffies64(). USAGE message: # ./stacksnoop -h usage: stacksnoop [-h] [-p PID] [-s] [-v] function Trace and print kernel stack traces for a kernel function positional arguments: function kernel function name optional arguments: -h, --help show this help message and exit -p PID, --pid PID trace this PID only -s, --offset show address offsets -v, --verbose print more fields examples: ./stacksnoop ext4_sync_fs # print kernel stack traces for ext4_sync_fs ./stacksnoop -s ext4_sync_fs # ... also show symbol offsets ./stacksnoop -v ext4_sync_fs # ... show extra columns ./stacksnoop -p 185 ext4_sync_fs # ... only when PID 185 is on-CPU