[Libunwind-devel] Remote unwinding support for perf on MIPS

[Sapir Natan]

Hi,

We are trying to obtain a callgraph using 'perf' utility, while running on MIPS architecture with linux 3.10 (64Bit processes)
We wish to obtain callgraph on:

· Terminated processes

· Running processes (using -p <pid>)

Following the resolution that perf is unable to create callgraph on MIPS using frame pointer, we tried using libunwind
Since 'dwarf' option for perf isn't part of linux 3.10 (or any other version), we added a patch (composed of the following 3 patches):

· https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html

· https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html

· https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html

· As part of the patch we also linked 'perf' with 'libunwind' (open source version 1.2.1).

This patch added the 'dwarf' capability to 'perf' on MIPS, however we still couldn't obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching 'maps__findŽ function (linux/tools/perf/util/map.c), the binary search for the required entry failed (could not be located)
From our investigation, we can support local unwinding but not remote (which is the type of unwinding required by 'perf')

Checking online we saw on 'libunwind' NEWS: '** Mips remote unwind support' for version 1.3.
We replaced our '1.2.1' version with '1.3-rc1' and re-checked, however the results remained the same (no callgraph).

Could you please assist?

· What was the issue you encountered that the fix in 1.3 version resolved?

· Can the new addition enable 'perf' to support callgraph on MIPS?

· Did we miss anything in our steps (as described above)?

· Can you advice on the next steps?

Thanks,
Sapir

[Sergey Korolev]

In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as

+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,

but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.

[1]
http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a629634b3225fccb1ed8a0a765bccc77f2e

Hi,
We are trying to obtain a callgraph using ‘perf’ utility, while running on
MIPS architecture with linux 3.10 (64Bit processes)
· Terminated processes
· Running processes (using –p <pid>)
Following the resolution that perf is unable to create callgraph on MIPS
using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
· https://www.linux-mips.org/archives/linux-mips/2013-05/
msg00123.html
· https://www.linux-mips.org/archives/linux-mips/2013-05/
msg00115.html
· https://www.linux-mips.org/archives/linux-mips/2016-04/
msg00003.html
· As part of the patch we also linked ‘perf’ with ‘libunwind’
(open source version *1.2.1*).
This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we
still couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching *‘maps__find®* function
(*linux/tools/perf/util/map.c*), the binary search for the required entry
failed (could not be located)
From our investigation, we can support local unwinding but not remote
(which is the type of unwinding required by ‘perf’)
Checking online we saw on ‘libunwind’ NEWS: *‘** Mips remote unwind
support’* for version *1.3*.
We replaced our ‘*1.2.1*’ version with ‘*1.3-rc1*’ and re-checked,
however the results remained the same (no callgraph).
Could you please assist?
· What was the issue you encountered that the fix in 1.3 version
resolved?
· Can the new addition enable ‘perf’ to support callgraph on MIPS?
· Did we miss anything in our steps (as described above)?
· Can you advice on the next steps?
Thanks,
Sapir
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel

[Luke Diamand]

I had a similar problem on mipsel32. At least on that the problem was that
the accessor function was calling read_s32 which seems to do the wrong
thing on that platform. Fixing that seems to have made it work. I'm in the
process of trying to organise a patch.

Post by Sergey Korolev
In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as
+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,
but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.
[1]
http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a629634b3225fccb1ed8a0a765bccc77f2e

Hi,
We are trying to obtain a callgraph using ‘perf’ utility, while running
on MIPS architecture with linux 3.10 (64Bit processes)
· Terminated processes
· Running processes (using –p <pid>)
Following the resolution that perf is unable to create callgraph on MIPS
using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
·
https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
· As part of the patch we also linked ‘perf’ with ‘libunwind’
(open source version *1.2.1*).
This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we
still couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching *‘maps__find®*
function (*linux/tools/perf/util/map.c*), the binary search for the
required entry failed (could not be located)
From our investigation, we can support local unwinding but not remote
(which is the type of unwinding required by ‘perf’)
Checking online we saw on ‘libunwind’ NEWS: *‘** Mips remote unwind
support’* for version *1.3*.
We replaced our ‘*1.2.1*’ version with ‘*1.3-rc1*’ and re-checked,
however the results remained the same (no callgraph).
Could you please assist?
· What was the issue you encountered that the fix in 1.3 version
resolved?
· Can the new addition enable ‘perf’ to support callgraph on MIPS?
· Did we miss anything in our steps (as described above)?
· Can you advice on the next steps?
Thanks,
Sapir
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel

_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel

[Sapir Natan]

Hi Luke,

Thank you very much for your reply.
I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’ linked to perf or called from it at all (or anything from libunwind-ptrace).
It is also already part of version 1.3-rc1, which means my perf still fails to unwind even with this latest fix.

Did this patch fix the perf unwinding on MIPS issue for you?
If so, can you tell me which patch you used to add this capability to perf on MIPS? (patch to the linux code)
Is there anything I’m missing here?

Thanks,
Sapir

From: Luke Diamand [mailto:***@diamand.org]
Sent: Sunday, July 29, 2018 8:59 PM
To: Sergey Korolev
Cc: Sapir Natan; Chen Muchtar; Roi Chen; libunwind-***@nongnu.org
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

I had a similar problem on mipsel32. At least on that the problem was that the accessor function was calling read_s32 which seems to do the wrong thing on that platform. Fixing that seems to have made it work. I'm in the process of trying to organise a patch.

On Sun, 29 Jul 2018, 18:25 Sergey Korolev, <***@ndmsystems.com<mailto:***@ndmsystems.com>> wrote:
In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as

+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,

but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.

[1] http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a629634b3225fccb1ed8a0a765bccc77f2e

On Sun, Jul 29, 2018 at 6:18 PM, Sapir Natan <***@checkpoint.com<mailto:***@checkpoint.com>> wrote:
Hi,

We are trying to obtain a callgraph using ‘perf’ utility, while running on MIPS architecture with linux 3.10 (64Bit processes)
We wish to obtain callgraph on:

• Terminated processes

• Running processes (using –p <pid>)

Following the resolution that perf is unable to create callgraph on MIPS using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other version), we added a patch (composed of the following 3 patches):

• https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html

• https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html

• https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html

• As part of the patch we also linked ‘perf’ with ‘libunwind’ (open source version 1.2.1).

This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we still couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching ‘maps__find® function (linux/tools/perf/util/map.c), the binary search for the required entry failed (could not be located)
From our investigation, we can support local unwinding but not remote (which is the type of unwinding required by ‘perf’)

Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’ for version 1.3.
We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however the results remained the same (no callgraph).

Could you please assist?

• What was the issue you encountered that the fix in 1.3 version resolved?

• Can the new addition enable ‘perf’ to support callgraph on MIPS?

• Did we miss anything in our steps (as described above)?

• Can you advice on the next steps?

Thanks,
Sapir


_______________________________________________
Libunwind-devel mailing list
Libunwind-***@nongnu.org<mailto:Libunwind-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/libunwind-devel

_______________________________________________
Libunwind-devel mailing list
Libunwind-***@nongnu.org<mailto:Libunwind-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/libunwind-devel


Email secured by Check Point.

[Luke Diamand]

Post by Sapir Natan
Hi Luke,
Thank you very much for your reply.
I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’ linked
to perf or called from it at all (or anything from libunwind-ptrace).
It is also already part of version 1.3-rc1, which means my perf still fails
to unwind even with this latest fix.

There are a ton of things that can go wrong along the way! I'm on
vacation at the moment and I don't have easy access to the relevant
source code, so this is all a bit from memory.

I used the 3 patches I think you mentioned to add perf MIPS support.

I've been getting to the point of capturing userspace callstacks
(DWARF, not fp) using perf+libunwind on MIPSEL32, and it appears to
work reasonably well, although I do see some callstacks possibly
failing (most do, just a few mysteriously missing).

You need to make sure the code you are testing actually has any unwind
tables in it - check the .eh_frame section is present.

I am using the current version of libunwind (1.3rc1 or thereabouts)
and I don't think it will work with older versions. You need to make
sure that you have the fix "dwarf: make dwarf_find_debug_frame public"
as otherwise perf just won't use libunwind at all.

I also found that in tdep-mips/libunwind_i.h, around line 150,
read_s32() seems to do some weird stuff with the address, which I
think is for handling big-endian 64 bit addresses, but it doesn't seem
to be the right thing to do on MIPSEL32. For now, I've just hacked out
the calls to read_s32() and that seems to make it work - but you're
actually on a 64 bit target so it might be different for you. I need
to create a proper patch (sorry, not had time recently).

I also had no end of fun with split-debug (as generated by
OpenEmbedded) because the debug files end up in a place that libunwind
does not expect to find them in. Watch out for that.

Another thing you will find is that with your version of perf, if you
are running the unwinding on an x86-64 host, then most/all versions of
perf will try to unwind the callstacks as though they are x86-64
callstacks rather than MIPS, which obviously doesn't work. From about
linux 4.7 onwards there is cross-platform perf unwinding available,
but the hooks for MIPS support have not been added (I don't yet have
working hooks either). There are a couple of ways around this: you can
run "perf report" on the target (but that is very slow) or you can run
"perf report" on your x86-64 host under qemu-mips (you want the
userland emulator not the full system version). This is what I've
mostly been doing.

Make sure that libunwind has been built with --enable-debug-frame
otherwise it won't work if your callstack goes through any code that
was not compiled with -fasynchronous-unwind-tables or -fexceptions.

If you run "perf report -D" then it dumps out the file contents. Look
through to see if you have any register dumps and check they are
convincing looking MIPS register values.

Also look for the stack captures in the "perf report -D" report.
That's the giant 8kByte samples. If you haven't got any, then it's not
doing the DWARF sampling at all.

Also check for the MMAP sections in the perf capture - sometimes perf
can't read the /proc/PID/maps file of the thing it is trying to
capture.

It sounds to me a bit as though you don't have a .eh_frame section in
your image (there's a linker option as well that you need), and if
perf/libunwind doesn't have access to the debug image, it won't work.
But it could also be quite a few other things!

Good luck!

Luke

Post by Sapir Natan
Did this patch fix the perf unwinding on MIPS issue for you?
If so, can you tell me which patch you used to add this capability to perf
on MIPS? (patch to the linux code)
Is there anything I’m missing here?
Thanks,
Sapir
Sent: Sunday, July 29, 2018 8:59 PM
To: Sergey Korolev
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
I had a similar problem on mipsel32. At least on that the problem was that
the accessor function was calling read_s32 which seems to do the wrong thing
on that platform. Fixing that seems to have made it work. I'm in the process
of trying to organise a patch.
In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as
+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,
but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.
[1]
http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a629634b3225fccb1ed8a0a765bccc77f2e
Hi,
We are trying to obtain a callgraph using ‘perf’ utility, while running on
MIPS architecture with linux 3.10 (64Bit processes)
· Terminated processes
· Running processes (using –p <pid>)
Following the resolution that perf is unable to create callgraph on MIPS
using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
·
https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
· As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
source version 1.2.1).
This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we still
couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching ‘maps__find´ function
(linux/tools/perf/util/map.c), the binary search for the required entry
failed (could not be located)
From our investigation, we can support local unwinding but not remote (which
is the type of unwinding required by ‘perf’)
Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
for version 1.3.
We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however the
results remained the same (no callgraph).
Could you please assist?
· What was the issue you encountered that the fix in 1.3 version resolved?
· Can the new addition enable ‘perf’ to support callgraph on MIPS?
· Did we miss anything in our steps (as described above)?
· Can you advice on the next steps?
Thanks,
Sapir
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
Email secured by Check Point.

[Sapir Natan]

Hi Luke,

Thank you so much for your detailed reply.

We have the '.eh_frame' in our code.
We also located a problem when reaching the function 'perf_evsel__parse_sample' in file /tools/perf/util/evsel.c (linux 3.10)

We noticed that the field 'data->user_stack.size' is assign the value saved at 'array += size / sizeof(*array)' which is 0 We changed the line to use the value stored in 'size' (which we checked to be the valid stack size), and now 'perf' seems to create a valid callgraph This might be an issue in MIPS architecture, while all other supported architectures contain a valid argument at that offset (although I should note this is only a speculation at this point)

The fix on line 1176:
data->user_stack.size = *array; // Old line
data->user_stack.size = size; // New line

Have you encountered any problems with this code section?
Do you think it might be a register issue on MIPS?

Thanks,
Sapir

-----Original Message-----
From: Luke Diamand [mailto:***@diamand.org]
Sent: Monday, July 30, 2018 11:57 AM
To: Sapir Natan
Cc: Sergey Korolev; Chen Muchtar; Roi Chen; libunwind-***@nongnu.org
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

Post by Sapir Natan
Hi Luke,
Thank you very much for your reply.
I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’
linked to perf or called from it at all (or anything from libunwind-ptrace).
It is also already part of version 1.3-rc1, which means my perf still
fails to unwind even with this latest fix.

There are a ton of things that can go wrong along the way! I'm on vacation at the moment and I don't have easy access to the relevant source code, so this is all a bit from memory.

I used the 3 patches I think you mentioned to add perf MIPS support.

I've been getting to the point of capturing userspace callstacks (DWARF, not fp) using perf+libunwind on MIPSEL32, and it appears to work reasonably well, although I do see some callstacks possibly failing (most do, just a few mysteriously missing).

You need to make sure the code you are testing actually has any unwind tables in it - check the .eh_frame section is present.

I am using the current version of libunwind (1.3rc1 or thereabouts) and I don't think it will work with older versions. You need to make sure that you have the fix "dwarf: make dwarf_find_debug_frame public"
as otherwise perf just won't use libunwind at all.

I also found that in tdep-mips/libunwind_i.h, around line 150,
read_s32() seems to do some weird stuff with the address, which I think is for handling big-endian 64 bit addresses, but it doesn't seem to be the right thing to do on MIPSEL32. For now, I've just hacked out the calls to read_s32() and that seems to make it work - but you're actually on a 64 bit target so it might be different for you. I need to create a proper patch (sorry, not had time recently).

I also had no end of fun with split-debug (as generated by
OpenEmbedded) because the debug files end up in a place that libunwind does not expect to find them in. Watch out for that.

Another thing you will find is that with your version of perf, if you are running the unwinding on an x86-64 host, then most/all versions of perf will try to unwind the callstacks as though they are x86-64 callstacks rather than MIPS, which obviously doesn't work. From about linux 4.7 onwards there is cross-platform perf unwinding available, but the hooks for MIPS support have not been added (I don't yet have working hooks either). There are a couple of ways around this: you can run "perf report" on the target (but that is very slow) or you can run "perf report" on your x86-64 host under qemu-mips (you want the userland emulator not the full system version). This is what I've mostly been doing.

Make sure that libunwind has been built with --enable-debug-frame otherwise it won't work if your callstack goes through any code that was not compiled with -fasynchronous-unwind-tables or -fexceptions.

If you run "perf report -D" then it dumps out the file contents. Look through to see if you have any register dumps and check they are convincing looking MIPS register values.

Also look for the stack captures in the "perf report -D" report.
That's the giant 8kByte samples. If you haven't got any, then it's not doing the DWARF sampling at all.

Also check for the MMAP sections in the perf capture - sometimes perf can't read the /proc/PID/maps file of the thing it is trying to capture.

It sounds to me a bit as though you don't have a .eh_frame section in your image (there's a linker option as well that you need), and if perf/libunwind doesn't have access to the debug image, it won't work.
But it could also be quite a few other things!

Good luck!

Luke

Post by Sapir Natan
Did this patch fix the perf unwinding on MIPS issue for you?
If so, can you tell me which patch you used to add this capability to
perf on MIPS? (patch to the linux code)
Is there anything I’m missing here?
Thanks,
Sapir
Sent: Sunday, July 29, 2018 8:59 PM
To: Sergey Korolev
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
I had a similar problem on mipsel32. At least on that the problem was
that the accessor function was calling read_s32 which seems to do the
wrong thing on that platform. Fixing that seems to have made it work.
I'm in the process of trying to organise a patch.
In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as
+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,
but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.
[1]
http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a6
29634b3225fccb1ed8a0a765bccc77f2e
Hi,
We are trying to obtain a callgraph using ‘perf’ utility, while
running on MIPS architecture with linux 3.10 (64Bit processes)
· Terminated processes
· Running processes (using –p <pid>)
Following the resolution that perf is unable to create callgraph on
MIPS using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
·
https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
· As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
source version 1.2.1).
This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we
still couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching ‘maps__find´
function (linux/tools/perf/util/map.c), the binary search for the
required entry failed (could not be located)
From our investigation, we can support local unwinding but not remote
(which is the type of unwinding required by ‘perf’)
Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
for version 1.3.
We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however
the results remained the same (no callgraph).
Could you please assist?
· What was the issue you encountered that the fix in 1.3 version resolved?
· Can the new addition enable ‘perf’ to support callgraph on MIPS?
· Did we miss anything in our steps (as described above)?
· Can you advice on the next steps?
Thanks,
Sapir
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
Email secured by Check Point.

Email secured by Check Point.

[Luke Diamand]

Post by Sapir Natan
Hi Luke,
Thank you so much for your detailed reply.
We have the '.eh_frame' in our code.
We also located a problem when reaching the function 'perf_evsel__parse_sample' in file /tools/perf/util/evsel.c (linux 3.10)
We noticed that the field 'data->user_stack.size' is assign the value saved at 'array += size / sizeof(*array)' which is 0 We changed the line to use the value stored in 'size' (which we checked to be the valid stack size), and now 'perf' seems to create a valid callgraph This might be an issue in MIPS architecture, while all other supported architectures contain a valid argument at that offset (although I should note this is only a speculation at this point)
data->user_stack.size = *array; // Old line
data->user_stack.size = size; // New line

I think I may have a somewhat newer version of the code - I think you
said you were using 3.10; I've got that kernel version but I also took
a bunch of patches to bring the perf support up-to-date.

The code I have says:

data->user_stack.size = *array++;

That change came in with a change from Adrian Hunter in 54bd269 (Linux
kernel) so that may explain the difference?

But I'm using MIPSEL32, so perhaps that makes a difference? Or perhaps
this is the explanation for my other missing data!

Luke

Post by Sapir Natan
Have you encountered any problems with this code section?
Do you think it might be a register issue on MIPS?
Thanks,
Sapir
-----Original Message-----
Sent: Monday, July 30, 2018 11:57 AM
To: Sapir Natan
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

Post by Sapir Natan
Hi Luke,
Thank you very much for your reply.
I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’
linked to perf or called from it at all (or anything from libunwind-ptrace).
It is also already part of version 1.3-rc1, which means my perf still
fails to unwind even with this latest fix.

There are a ton of things that can go wrong along the way! I'm on vacation at the moment and I don't have easy access to the relevant source code, so this is all a bit from memory.
I used the 3 patches I think you mentioned to add perf MIPS support.
I've been getting to the point of capturing userspace callstacks (DWARF, not fp) using perf+libunwind on MIPSEL32, and it appears to work reasonably well, although I do see some callstacks possibly failing (most do, just a few mysteriously missing).
You need to make sure the code you are testing actually has any unwind tables in it - check the .eh_frame section is present.
I am using the current version of libunwind (1.3rc1 or thereabouts) and I don't think it will work with older versions. You need to make sure that you have the fix "dwarf: make dwarf_find_debug_frame public"
as otherwise perf just won't use libunwind at all.
I also found that in tdep-mips/libunwind_i.h, around line 150,
read_s32() seems to do some weird stuff with the address, which I think is for handling big-endian 64 bit addresses, but it doesn't seem to be the right thing to do on MIPSEL32. For now, I've just hacked out the calls to read_s32() and that seems to make it work - but you're actually on a 64 bit target so it might be different for you. I need to create a proper patch (sorry, not had time recently).
I also had no end of fun with split-debug (as generated by
OpenEmbedded) because the debug files end up in a place that libunwind does not expect to find them in. Watch out for that.
Another thing you will find is that with your version of perf, if you are running the unwinding on an x86-64 host, then most/all versions of perf will try to unwind the callstacks as though they are x86-64 callstacks rather than MIPS, which obviously doesn't work. From about linux 4.7 onwards there is cross-platform perf unwinding available, but the hooks for MIPS support have not been added (I don't yet have working hooks either). There are a couple of ways around this: you can run "perf report" on the target (but that is very slow) or you can run "perf report" on your x86-64 host under qemu-mips (you want the userland emulator not the full system version). This is what I've mostly been doing.
Make sure that libunwind has been built with --enable-debug-frame otherwise it won't work if your callstack goes through any code that was not compiled with -fasynchronous-unwind-tables or -fexceptions.
If you run "perf report -D" then it dumps out the file contents. Look through to see if you have any register dumps and check they are convincing looking MIPS register values.
Also look for the stack captures in the "perf report -D" report.
That's the giant 8kByte samples. If you haven't got any, then it's not doing the DWARF sampling at all.
Also check for the MMAP sections in the perf capture - sometimes perf can't read the /proc/PID/maps file of the thing it is trying to capture.
It sounds to me a bit as though you don't have a .eh_frame section in your image (there's a linker option as well that you need), and if perf/libunwind doesn't have access to the debug image, it won't work.
But it could also be quite a few other things!
Good luck!
Luke

Post by Sapir Natan
Did this patch fix the perf unwinding on MIPS issue for you?
If so, can you tell me which patch you used to add this capability to
perf on MIPS? (patch to the linux code)
Is there anything I’m missing here?
Thanks,
Sapir
Sent: Sunday, July 29, 2018 8:59 PM
To: Sergey Korolev
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
I had a similar problem on mipsel32. At least on that the problem was
that the accessor function was calling read_s32 which seems to do the
wrong thing on that platform. Fixing that seems to have made it work.
I'm in the process of trying to organise a patch.
In my opinion a mentioned patch [1] does not work properly
since it defines _UPT_reg_offset elements as
+ [UNW_MIPS_R0] = 0,
+ [UNW_MIPS_R1] = 1,
+ [UNW_MIPS_R2] = 2,
+ [UNW_MIPS_R3] = 3,
+ [UNW_MIPS_R4] = 4,
+ [UNW_MIPS_R5] = 5,
+ [UNW_MIPS_R6] = 6,
+ [UNW_MIPS_R7] = 7,
+ [UNW_MIPS_R8] = 8,
+ [UNW_MIPS_R9] = 9,
+ [UNW_MIPS_R10] = 10,
+ [UNW_MIPS_R11] = 11,
+ [UNW_MIPS_R12] = 12,
+ [UNW_MIPS_R13] = 13,
+ [UNW_MIPS_R14] = 14,
+ [UNW_MIPS_R15] = 15,
+ [UNW_MIPS_R16] = 16,
+ [UNW_MIPS_R17] = 17,
+ [UNW_MIPS_R18] = 18,
+ [UNW_MIPS_R19] = 19,
+ [UNW_MIPS_R20] = 20,
+ [UNW_MIPS_R21] = 21,
+ [UNW_MIPS_R22] = 22,
+ [UNW_MIPS_R23] = 23,
+ [UNW_MIPS_R24] = 24,
+ [UNW_MIPS_R25] = 25,
+ [UNW_MIPS_R26] = 26,
+ [UNW_MIPS_R27] = 27,
+ [UNW_MIPS_R28] = 28,
+ [UNW_MIPS_R29] = 29,
+ [UNW_MIPS_R30] = 30,
+ [UNW_MIPS_R31] = 31,
+ [UNW_MIPS_PC] = 64,
but a code in _UPT_access_mem.c expects that _UPT_reg_offset
is a byte offset, and not a word offset or a register index.
[1]
http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a6
29634b3225fccb1ed8a0a765bccc77f2e
Hi,
We are trying to obtain a callgraph using ‘perf’ utility, while
running on MIPS architecture with linux 3.10 (64Bit processes)
· Terminated processes
· Running processes (using –p <pid>)
Following the resolution that perf is unable to create callgraph on
MIPS using frame pointer, we tried using libunwind
Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
·
https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
·
https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
· As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
source version 1.2.1).
This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we
still couldn’t obtain a usable callgraph (most functions had no trace)
The problem we encountered was that when reaching ‘maps__find´
function (linux/tools/perf/util/map.c), the binary search for the
required entry failed (could not be located)
From our investigation, we can support local unwinding but not remote
(which is the type of unwinding required by ‘perf’)
Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
for version 1.3.
We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however
the results remained the same (no callgraph).
Could you please assist?
· What was the issue you encountered that the fix in 1.3 version resolved?
· Can the new addition enable ‘perf’ to support callgraph on MIPS?
· Did we miss anything in our steps (as described above)?
· Can you advice on the next steps?
Thanks,
Sapir
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
_______________________________________________
Libunwind-devel mailing list
https://lists.nongnu.org/mailman/listinfo/libunwind-devel
Email secured by Check Point.

Email secured by Check Point.