From dblaikie@gmail.com  Tue May  2 19:09:40 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Tue, 02 May 2017 19:09:40 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
Message-ID: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>

I've recently been trying to resolve the use of Fission in LLVM's ThinLTO
mode (though this would apply to plain LTO too).

One of the things that happens here is that cross-CU DIE references
(DW_FORM_ref_addr) are used to describe inlining a function in one CU into
another CU.

This format has been implemented in LLVM and GCC for ~years and seems to
work well outside of Fission.

So the question is: what to do with Fission?

It seemed to me that a good representation would be to produce multiple CUs
into a single DWO file, which GDB can't yet consume, but I'm working on
patches to help there. DW_FORM_ref_addr would not use any ELF relocation,
but be assumed to be "relative to the chunk of debug_info it was in"
(within the .dwo file)

But what about DWP files? Currently binutils dwp produces records like this:

(this dwp contains 3 CUs, two from one LTO compile, and one from a
standalone compile linked in for comparison):

Index Signature          INFO     ABBR     LINE     STR_OFF
----- ------------------ -------- -------- -------- --------
    2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c)
    8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14)
   11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)

So the ABBR/LINE/STR_OFF sections are kept as-is (no analysis is done to
find which portions of the dwo file are used by which CUs, etc), but the
INFO section is fragmented on the CU boundaries. Fragmenting the TYPES
section on the TU boundaries is necessary/useful for deduplication of
types, but this fragmenting of the CU makes it impossible (I think) to use
ref_addr in a dwp file.

If this fragmenting were not done - consumers (GDB, etc) would need to
change to account for this - searching through the INFO range to find the
CU matching the signature, rather than knowing it starts at the start of
the INFO range. This could have a noticeable performance impact especially
in a full LTO build (where /all/ the CUs were in the same .dwo - so the
index would be entirely unhelpful, I think).

Does all this sound right/sane - anyone have ideas/perspectives/thoughts on
how this should work?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170502/155f11be/attachment.htm>


From aaronrobinson@nokomis.k12.il.us  Wed May  3 16:53:24 2017
From: aaronrobinson@nokomis.k12.il.us (Aaron Robinson)
Date: Wed, 3 May 2017 11:53:24 -0500
Subject: [Dwarf-Discuss] Dwarf-Discuss Digest, Vol 99, Issue 1
In-Reply-To: <mailman.1.1493827521.11547.dwarf-discuss-dwarfstd.org@lists.dwarfstd.org>
References: <mailman.1.1493827521.11547.dwarf-discuss-dwarfstd.org@lists.dwarfstd.org>
Message-ID: <CAOgs1vq3xjBkV7sj-fO9GJOqU42B=aHWKXULwD=0C9AYpqBcjg@mail.gmail.com>

So are we going to discuss dwarves? Or what?

On Wed, May 3, 2017 at 11:05 AM, <dwarf-discuss-request at lists.dwarfstd.org>
wrote:

> Send Dwarf-Discuss mailing list submissions to
>         dwarf-discuss at lists.dwarfstd.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
> or, via email, send a message with subject or body 'help' to
>         dwarf-discuss-request at lists.dwarfstd.org
>
> You can reach the person managing the list at
>         dwarf-discuss-owner at lists.dwarfstd.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Dwarf-Discuss digest..."
>
>
> Today's Topics:
>
>    1. Fission + cross-CU references (ref_addr) (David Blaikie)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 02 May 2017 19:09:40 +0000
> From: David Blaikie <dblaikie at gmail.com>
> To: "dwarf-discuss at lists.dwarfstd.org"
>         <dwarf-discuss at lists.dwarfstd.org>
> Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
> Message-ID:
>         <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=p
> B_XLuSw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I've recently been trying to resolve the use of Fission in LLVM's ThinLTO
> mode (though this would apply to plain LTO too).
>
> One of the things that happens here is that cross-CU DIE references
> (DW_FORM_ref_addr) are used to describe inlining a function in one CU into
> another CU.
>
> This format has been implemented in LLVM and GCC for ~years and seems to
> work well outside of Fission.
>
> So the question is: what to do with Fission?
>
> It seemed to me that a good representation would be to produce multiple CUs
> into a single DWO file, which GDB can't yet consume, but I'm working on
> patches to help there. DW_FORM_ref_addr would not use any ELF relocation,
> but be assumed to be "relative to the chunk of debug_info it was in"
> (within the .dwo file)
>
> But what about DWP files? Currently binutils dwp produces records like
> this:
>
> (this dwp contains 3 CUs, two from one LTO compile, and one from a
> standalone compile linked in for comparison):
>
> Index Signature          INFO     ABBR     LINE     STR_OFF
> ----- ------------------ -------- -------- -------- --------
>     2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c)
>     8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14)
>    11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)
>
> So the ABBR/LINE/STR_OFF sections are kept as-is (no analysis is done to
> find which portions of the dwo file are used by which CUs, etc), but the
> INFO section is fragmented on the CU boundaries. Fragmenting the TYPES
> section on the TU boundaries is necessary/useful for deduplication of
> types, but this fragmenting of the CU makes it impossible (I think) to use
> ref_addr in a dwp file.
>
> If this fragmenting were not done - consumers (GDB, etc) would need to
> change to account for this - searching through the INFO range to find the
> CU matching the signature, rather than knowing it starts at the start of
> the INFO range. This could have a noticeable performance impact especially
> in a full LTO build (where /all/ the CUs were in the same .dwo - so the
> index would be entirely unhelpful, I think).
>
> Does all this sound right/sane - anyone have ideas/perspectives/thoughts on
> how this should work?
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-
> dwarfstd.org/attachments/20170502/155f11be/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Dwarf-Discuss mailing list
> Dwarf-Discuss at lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
>
>
> ------------------------------
>
> End of Dwarf-Discuss Digest, Vol 99, Issue 1
> ********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170503/828901a1/attachment.htm>


From paul.robinson@sony.com  Thu May  4 19:46:12 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Thu, 4 May 2017 19:46:12 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>

I think David is correct, that we did not consider LTO and assumed a .dwo file would have a single compilation unit in the .debug_info section.  It seems to me not hard to fix, but my idea would require an extension to the package-file index and I don't see provision in the package-file index for vendor extensions (another oversight?).

In a split-DWARF scenario producing multiple CUs, it's clear that each split-full unit in the .dwo file would need a corresponding skeleton unit in the .o file, with matching unique DWO IDs.  The v5 spec basically already says that.  With multiple split-full units in the same .debug_info section, then DW_FORM_ref_addr can support cross-CU references within the section; the producer can supply the correct offset within the section without needing any relocations.

How to describe this in the package file?  I'd leave DW_SECT_INFO meaning what it does now?describing the base and size of the individual unit.  I'd add a new "section identifier" DW_SECT_INFO_FILE or whatever, which describes the base and size of the entire .debug_info section contributed by the .dwo *file* that the unit came from.  This allows a consumer to find each individual unit by DWO ID, as today, and the extra _FILE column describes the base-and-size to use when interpreting a DW_FORM_ref_addr from that unit.  For any .dwo file that contains only one unit, DW_SECT_INFO and DW_SECT_INFO_FILE would have the same values.  The tool that creates the package file can omit DW_SECT_INFO_FILE from the index if every input .dwo file has only one unit.

This solution avoids the problem of the *consumer* having to scan the .debug_info contribution to find the units; that work can be done once up front by the packaging tool.

Section identifiers are 32 bits wide, and the defined values are just 1-8; surely we can allocate some for vendor extensions!  And then it's no problem to have tools produce the new column for the index.  Consumers will just ignore section identifiers that they don't recognize, same as any other part of DWARF.

Would that address the problem?
--paulr

From: Dwarf-Discuss [mailto:dwarf-discuss-bounces@lists.dwarfstd.org] On Behalf Of David Blaikie
Sent: Tuesday, May 02, 2017 12:10 PM
To: dwarf-discuss at lists.dwarfstd.org
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)

I've recently been trying to resolve the use of Fission in LLVM's ThinLTO mode (though this would apply to plain LTO too).

One of the things that happens here is that cross-CU DIE references (DW_FORM_ref_addr) are used to describe inlining a function in one CU into another CU.

This format has been implemented in LLVM and GCC for ~years and seems to work well outside of Fission.

So the question is: what to do with Fission?

It seemed to me that a good representation would be to produce multiple CUs into a single DWO file, which GDB can't yet consume, but I'm working on patches to help there. DW_FORM_ref_addr would not use any ELF relocation, but be assumed to be "relative to the chunk of debug_info it was in" (within the .dwo file)

But what about DWP files? Currently binutils dwp produces records like this:

(this dwp contains 3 CUs, two from one LTO compile, and one from a standalone compile linked in for comparison):

Index Signature          INFO     ABBR     LINE     STR_OFF
----- ------------------ -------- -------- -------- --------
    2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c)
    8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14)
   11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)

So the ABBR/LINE/STR_OFF sections are kept as-is (no analysis is done to find which portions of the dwo file are used by which CUs, etc), but the INFO section is fragmented on the CU boundaries. Fragmenting the TYPES section on the TU boundaries is necessary/useful for deduplication of types, but this fragmenting of the CU makes it impossible (I think) to use ref_addr in a dwp file.

If this fragmenting were not done - consumers (GDB, etc) would need to change to account for this - searching through the INFO range to find the CU matching the signature, rather than knowing it starts at the start of the INFO range. This could have a noticeable performance impact especially in a full LTO build (where /all/ the CUs were in the same .dwo - so the index would be entirely unhelpful, I think).

Does all this sound right/sane - anyone have ideas/perspectives/thoughts on how this should work?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170504/93ce0082/attachment-0001.htm>


From dblaikie@gmail.com  Thu May  4 20:27:15 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Thu, 04 May 2017 20:27:15 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>

On Thu, May 4, 2017 at 12:46 PM Robinson, Paul <paul.robinson at sony.com>
wrote:

> I think David is correct, that we did not consider LTO and assumed a .dwo
> file would have a single compilation unit in the .debug_info section.  It
> seems to me not hard to fix, but my idea would require an extension to the
> package-file index and I don't see provision in the package-file index for
> vendor extensions (another oversight?).
>

(jumping the gun a bit) - that said, an extension to the set of valid
columns is backwards compatible-ish (yeah, an existing consumer might error
on it, I suppose). But, yeah, might be a good opportunity to revisit & add
versioning and vendor extension point, maybe?


>  In a split-DWARF scenario producing multiple CUs, it's clear that each
> split-full unit in the .dwo file would need a corresponding skeleton unit
> in the .o file,
>

Eric was tossing around an idea that would diverge from that - having a
single skeleton for the whole DWO file (where the DWO file would contain
multiple CUs - though the specifics there we hadn't hashed out - maybe
every CU having the same DWO ID or the like), which could reduce the size
of the debug info in object files in these situations. (LLVM produces a
single debug_addr/debug_ranges/etc in Fission anyway - so every CU in a
fission object file would include the same addr_base, ranges_base, the same
abbrev offset, etc anyway)

But that'd probably be rather invasive a change to Fission? Not sure.


> with matching unique DWO IDs.  The v5 spec basically already says that.
> With multiple split-full units in the same .debug_info section, then
> DW_FORM_ref_addr can support cross-CU references within the section; the
> producer can supply the correct offset within the section without needing
> any relocations.
>

Yep


> How to describe this in the package file?  I'd leave DW_SECT_INFO meaning
> what it does now?describing the base and size of the individual unit.  I'd
> add a new "section identifier" DW_SECT_INFO_FILE or whatever, which
> describes the base and size of the entire .debug_info section contributed
> by the .dwo *file* that the unit came from.  This allows a consumer to find
> each individual unit by DWO ID, as today, and the extra _FILE column
> describes the base-and-size to use when interpreting a DW_FORM_ref_addr
> from that unit.  For any .dwo file that contains only one unit,
> DW_SECT_INFO and DW_SECT_INFO_FILE would have the same values.  The tool
> that creates the package file can omit DW_SECT_INFO_FILE from the index if
> every input .dwo file has only one unit.
>

Ah - that's a smart idea I hadn't considered & makes a lot of sense. This
sounds like it could address my concern/idea around type units too... (will
detail that a bit more later)


> This solution avoids the problem of the *consumer* having to scan the
> .debug_info contribution to find the units; that work can be done once up
> front by the packaging tool.
>

Yep - I don't have a good sense of how expensive such a scan is, but the
index is relatively small, I think (would be good to measure what the index
looks like for LLVM's ThinLTO which will cause many more CUs to exist
(because every primary compilation that imports a few functions from other
CUs will get a separate CU for each CU it imports from))


> Section identifiers are 32 bits wide, and the defined values are just 1-8;
> surely we can allocate some for vendor extensions!
>

Seems legit.


> And then it's no problem to have tools produce the new column for the
> index.  Consumers will just ignore section identifiers that they don't
> recognize, same as any other part of DWARF.
>

For sure.


> Would that address the problem?
>

Sounds like it.

So here's some extra wrinkles/ideas:

Wrinkle 1: I think binutils DWP currently drops duplicate units (units with
the same DWO ID). With this change, that wouldn't be possible - or at least
all the /bytes/ of the DWO would have to be imported regardless, and in a
contiguous chunk, so that cross-CU references would resolve correctly (if
you had a DWO with 3 CUs in it, the middle of which turned out to be
duplicate & was dropped, then the offset from a DIE in CU 1 to a DIE in CU
3 (now 2) would be broken). I think that's probably OK - such a DWP can
only have one entry in the cu_index for that signature - but it can't drop
the bytes anymore... *shrug*

Wrinkle 2: Type units go in the debug_info section, but you really do want
to be able to drop duplicate type units when creating a DWP (that being the
point of type units). So maybe require that DWO files have all the CUs
first, then the TUs? and the INFO_FILE range only applies to the range over
the CUs? This hurts/walks back the unification of CUs and TUs by special
casing, unfortunately... - other ideas?

Idea: I've been wondering about the idea of not putting types in type units
if the producer knows there won't be duplicates (for example Clang's (&
GCC's) vtable-based optimization - if a type has a vtable, only put the
type definition where the vtable is emitted - well, if the key function is
strong, then you know it's going in exactly one place... so why add the
overhead of a type unit?). But that makes it awkward for types in type
units that want to refer to these ununited types. A simple implementation
could produce a declaration of the ununited type in the united type, but
that's some overhead - an alternative would be to use ref_addr to refer to
the ununited type in the CU - /assuming/ that ref_addr always refers to the
debug_info section, not the debug_types section - or in the case of DWP,
assuming that ref_addr is resolved relative to the new INFO_FILE range
you're proposing (with my ammendment above that it only apply to the
(required to be) contiguous range of CUs).

- Dave


>
> --paulr
>
>
>
> *From:* Dwarf-Discuss [mailto:dwarf-discuss-bounces at lists.dwarfstd.org] *On
> Behalf Of *David Blaikie
> *Sent:* Tuesday, May 02, 2017 12:10 PM
> *To:* dwarf-discuss at lists.dwarfstd.org
> *Subject:* [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
>
>
>
> I've recently been trying to resolve the use of Fission in LLVM's ThinLTO
> mode (though this would apply to plain LTO too).
>
>
>
> One of the things that happens here is that cross-CU DIE references
> (DW_FORM_ref_addr) are used to describe inlining a function in one CU into
> another CU.
>
>
>
> This format has been implemented in LLVM and GCC for ~years and seems to
> work well outside of Fission.
>
>
>
> So the question is: what to do with Fission?
>
>
>
> It seemed to me that a good representation would be to produce multiple
> CUs into a single DWO file, which GDB can't yet consume, but I'm working on
> patches to help there. DW_FORM_ref_addr would not use any ELF relocation,
> but be assumed to be "relative to the chunk of debug_info it was in"
> (within the .dwo file)
>
>
>
> But what about DWP files? Currently binutils dwp produces records like
> this:
>
>
>
> (this dwp contains 3 CUs, two from one LTO compile, and one from a
> standalone compile linked in for comparison):
>
>
>
> Index Signature          INFO     ABBR     LINE     STR_OFF
>
> ----- ------------------ -------- -------- -------- --------
>
>     2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c)
>
>     8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14)
>
>    11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)
>
>
>
> So the ABBR/LINE/STR_OFF sections are kept as-is (no analysis is done to
> find which portions of the dwo file are used by which CUs, etc), but the
> INFO section is fragmented on the CU boundaries. Fragmenting the TYPES
> section on the TU boundaries is necessary/useful for deduplication of
> types, but this fragmenting of the CU makes it impossible (I think) to use
> ref_addr in a dwp file.
>
>
>
> If this fragmenting were not done - consumers (GDB, etc) would need to
> change to account for this - searching through the INFO range to find the
> CU matching the signature, rather than knowing it starts at the start of
> the INFO range. This could have a noticeable performance impact especially
> in a full LTO build (where /all/ the CUs were in the same .dwo - so the
> index would be entirely unhelpful, I think).
>
>
>
> Does all this sound right/sane - anyone have ideas/perspectives/thoughts
> on how this should work?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170504/a0ccc33f/attachment.htm>


From paul.robinson@sony.com  Thu May  4 21:26:11 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Thu, 4 May 2017 21:26:11 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>

 In a split-DWARF scenario producing multiple CUs, it's clear that each split-full unit in the .dwo file would need a corresponding skeleton unit in the .o file,
Eric was tossing around an idea that would diverge from that - having a single skeleton for the whole DWO file (where the DWO file would contain multiple CUs - though the specifics there we hadn't hashed out - maybe every CU having the same DWO ID or the like), which could reduce the size of the debug info in object files in these situations. (LLVM produces a single debug_addr/debug_ranges/etc in Fission anyway - so every CU in a fission object file would include the same addr_base, ranges_base, the same abbrev offset, etc anyway)

Skeleton units are pretty small; it's a 20-byte header, plus values for the compile_unit DIE, which is spec'd to have no children.  I would not be concerned about space there.  And having unique DWO IDs per unit seems pretty useful.

Wrinkle 1: I think binutils DWP currently drops duplicate units (units with the same DWO ID). With this change, that wouldn't be possible - or at least all the /bytes/ of the DWO would have to be imported regardless, and in a contiguous chunk, so that cross-CU references would resolve correctly (if you had a DWO with 3 CUs in it, the middle of which turned out to be duplicate & was dropped, then the offset from a DIE in CU 1 to a DIE in CU 3 (now 2) would be broken). I think that's probably OK - such a DWP can only have one entry in the cu_index for that signature - but it can't drop the bytes anymore... *shrug*

That's true, each .debug_info section that had at least one retained split-full unit would have to be brought in completely.

Wrinkle 2: Type units go in the debug_info section, but you really do want to be able to drop duplicate type units when creating a DWP (that being the point of type units). So maybe require that DWO files have all the CUs first, then the TUs? and the INFO_FILE range only applies to the range over the CUs? This hurts/walks back the unification of CUs and TUs by special casing, unfortunately... - other ideas?

In non-split DWARF, type units are spec'd to have their own object-file section contributions, separate from the compile unit(s); that's what lets type units have a COMDAT key and be uniqued by the linker, even though all those separate contributions have the same section name.  (In ELF, you have multiple section headers with the same section name.)  Surely the DWO file could be (is?) done the same way, with each type unit in its own contribution to the .debug_info section?  Then the packager can incorporate or drop each of those contributions independently.  Being separate .debug_info lumps, the CUs are already on their own, and this Wrinkle goes away.

--paulr

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170504/0f1d075d/attachment-0001.htm>


From dblaikie@gmail.com  Thu May  4 21:41:27 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Thu, 04 May 2017 21:41:27 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>

On Thu, May 4, 2017 at 2:26 PM Robinson, Paul <paul.robinson at sony.com>
wrote:

>  In a split-DWARF scenario producing multiple CUs, it's clear that each
> split-full unit in the .dwo file would need a corresponding skeleton unit
> in the .o file,
>
> Eric was tossing around an idea that would diverge from that - having a
> single skeleton for the whole DWO file (where the DWO file would contain
> multiple CUs - though the specifics there we hadn't hashed out - maybe
> every CU having the same DWO ID or the like), which could reduce the size
> of the debug info in object files in these situations. (LLVM produces a
> single debug_addr/debug_ranges/etc in Fission anyway - so every CU in a
> fission object file would include the same addr_base, ranges_base, the same
> abbrev offset, etc anyway)
>
> Skeleton units are pretty small; it's a 20-byte header, plus values for
> the compile_unit DIE, which is spec'd to have no children.  I would not be
> concerned about space there.  And having unique DWO IDs per unit seems
> pretty useful.
>

I tend to agree, though - what sort of uses do you have in mind?

I don't see any particular benefit to having separate skeletons/DWO IDs for
ecah CU in a single DWO file. The (GNU) pubnames & pubtypes that are used
to produce a gdb_index might have to be modified a bit as they're probably
CU-relative currently, but would have to be "DWO info chunk" relative
instead, which might require a fair bit of work. & having them per CU does
reduce the workload for the consumer a little - consumer doesn't have to
build a table of CU ranges for a DWO info chunk and then find which CU
range the offset falls in to figure out which CU to deserialize.

Other ideas/uses/issues to consider?


>  Wrinkle 1: I think binutils DWP currently drops duplicate units (units
> with the same DWO ID). With this change, that wouldn't be possible - or at
> least all the /bytes/ of the DWO would have to be imported regardless, and
> in a contiguous chunk, so that cross-CU references would resolve correctly
> (if you had a DWO with 3 CUs in it, the middle of which turned out to be
> duplicate & was dropped, then the offset from a DIE in CU 1 to a DIE in CU
> 3 (now 2) would be broken). I think that's probably OK - such a DWP can
> only have one entry in the cu_index for that signature - but it can't drop
> the bytes anymore... *shrug*
>
>
> That's true, each .debug_info section that had at least one retained
> split-full unit would have to be brought in completely.
>

*nod* (good clarification - as you say, a whole DWO could be dropped, but
if any CU in it is retained, all the CUs must be)


>
>
> Wrinkle 2: Type units go in the debug_info section, but you really do want
> to be able to drop duplicate type units when creating a DWP (that being the
> point of type units). So maybe require that DWO files have all the CUs
> first, then the TUs? and the INFO_FILE range only applies to the range over
> the CUs? This hurts/walks back the unification of CUs and TUs by special
> casing, unfortunately... - other ideas?
>
> In non-split DWARF, type units are spec'd to have their own object-file
> section contributions, separate from the compile unit(s);
>

That's sort of an implementation detail though, isn't it? DWARF just talks
about bytes in sections (type units go in the debug_types section (or, now,
the debug_info section)) and, yeah, you can use comdat groups and separate
chunks of debug_types sections to deduplicate them, but I don't think DWARF
requires/speaks about that, does it?


> that's what lets type units have a COMDAT key and be uniqued by the
> linker, even though all those separate contributions have the same section
> name.  (In ELF, you have multiple section headers with the same section
> name.)  Surely the DWO file could be (is?) done the same way, with each
> type unit in its own contribution to the .debug_info section?
>

Funny story about that...

Back when I implemented Clang/LLVM's type unit support, then implemented
the Fission+type unit support - when I read the Fission spec/proposal and
it said "type units don't need to be in comdat groups" I did the natural
thing, removed the comdating, and LLVM naturally produced a single
debug_types section containing all the types.

This broke the binutils dwp tool at the time - since it was still expecting
separate debug_types sections, one for each type. That was fixed and we
carried on...

At some point I noticed that LLVM's .dwo files not only were smaller than
GCC's for other reasons, but they got /much/ smaller when
compress-debug-sections was enabled. LLVM's debug info was somehow more
compressible than GCC's.

It wasn't until 2-3 years later and I was implementing the llvm-dwp
prototype that it struck me: By placing each type in a separate debug_types
section, each type was compressed separately - not nearly as efficient as
compressing them all together.

So one of the reasons GCC's debug info was bigger than Clang's was because
it was less compressible due to these boundaries between types. (it could,
arguably, make the DWP tool more efficient by allowing it to uncompress
only a single type at a time - rather than having to uncompress teh whole
debug_types section in one go, though)


>   Then the packager can incorporate or drop each of those contributions
> independently.  Being separate .debug_info lumps, the CUs are already on
> their own, and this Wrinkle goes away.
>

Not sure I quite follow this - now in a .dwo file there will be some number
of debug_info chunks - arguably one for all the CUs and one for each TU
(notwithstanding the suboptimality of the compressibility of such a
representation) - still there would need to be a special case where the
TU's debug_info chunk would have an INFO_FILE contribution that represente
the CU chunk. So a DWP tool would have to special case the info chunk that
contained the CUs (& would have to require that there be only one if there
are to be TU->CU references. I suppose if TU->CU references aren't
supported then you could actually emit chunks of CUs at whatever
granularity you like - that could then allow some multi-CU dwos to still
haev CUs linked in independently, rather than all having to be pulled in
together, if any are used (LLVM probably wouldn't use this, except at some
weird -O0 LTO mode (for that whole program devirt/analysis mode, maybe) I
guess))

- Dave


>
>
> --paulr
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170504/3165168b/attachment.htm>


From paul.robinson@sony.com  Fri May  5 00:05:27 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Fri, 5 May 2017 00:05:27 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>

Skeleton units are pretty small; it's a 20-byte header, plus values for the compile_unit DIE, which is spec'd to have no children.  I would not be concerned about space there.  And having unique DWO IDs per unit seems pretty useful.
I tend to agree, though - what sort of uses do you have in mind?

A unique DWO ID per unit lets each DWO unit have a distinct entry in the index? saves the consumer the trouble of having to read the .debug_info section to find the units.  If you want to require consumers to do more work, you can make DWO IDs be per-file instead of per-unit, and then there's no need for an INFO_FILE column because the INFO column would necessarily have to cover the entire .debug_info section from that file.

In non-split DWARF, type units are spec'd to have their own object-file section contributions, separate from the compile unit(s);
That's sort of an implementation detail though, isn't it? DWARF just talks about bytes in sections (type units go in the debug_types section (or, now, the debug_info section)) and, yeah, you can use comdat groups and separate chunks of debug_types sections to deduplicate them, but I don't think DWARF requires/speaks about that, does it?

Actually DWARF 5 Appendix E does describe this; not as a required tactic, but a way to achieve the useful effect of deduplicating type units etc.  So yes it was overstating the case to say they are _spec'd_ to have their own contributions, but that would be what a producer would normally do.

that's what lets type units have a COMDAT key and be uniqued by the linker, even though all those separate contributions have the same section name.  (In ELF, you have multiple section headers with the same section name.)  Surely the DWO file could be (is?) done the same way, with each type unit in its own contribution to the .debug_info section?
Funny story about that...

Heh.  Which way works better (types all together, or each in their own contribution) depends on whether your packager wants to deduplicate in a linker-like way, based on COMDATs, or in a purpose-built way, by looking at the type-unit signatures directly.  DWARF doesn't say you have to do one or the other, which provides implementation flexibility to the toolchain.  If your packager is willing to look through TUs for signatures and deduplicate that way, then you can stuff all the TUs into one section contribution and get better compression.  Quality of Implementation, as we like to say.

still there would need to be a special case where the TU's debug_info chunk would have an INFO_FILE contribution that represente the CU chunk. So a DWP tool would have to special case the info chunk that contained the CUs (& would have to require that there be only one if there are to be TU->CU references. I suppose if TU->CU references aren't supported

I don't think TU->CU references are permitted.  Certainly you could not have a v4 split TU referencing a CU, that would be impossible.  (Without relocations, you can't use DW_FORM_ref_addr to point from .debug_types to .debug_info; and DW_FORM_ref_sig8 is only for references to other type units.)  While you could engineer the possibility in v5, because type units have moved back into .debug_info and in principle you could arrange for DW_FORM_ref_addr to do that, I am morally certain there was no intent to allow that.
--paulr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170505/51d5d267/attachment-0001.htm>


From dblaikie@gmail.com  Fri May  5 00:30:11 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Fri, 05 May 2017 00:30:11 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>

On Thu, May 4, 2017 at 5:05 PM Robinson, Paul <paul.robinson at sony.com>
wrote:

> Skeleton units are pretty small; it's a 20-byte header, plus values for
> the compile_unit DIE, which is spec'd to have no children.  I would not be
> concerned about space there.  And having unique DWO IDs per unit seems
> pretty useful.
>
> I tend to agree, though - what sort of uses do you have in mind?
>
>
>
> A unique DWO ID per unit lets each DWO unit have a distinct entry in the
> index? saves the consumer the trouble of having to read the .debug_info
> section to find the units.
>

Yep


>   If you want to require consumers to do more work, you can make DWO IDs
> be per-file instead of per-unit, and then there's no need for an INFO_FILE
> column because the INFO column would necessarily have to cover the entire
> .debug_info section from that file.
>

Yep - time/space tradeoff, and I'd probably err on the side of time myself
(by having separate skeletons and cu_index entries for each CU) as you've
suggested. Just floating the other as an alternative since it did come up.


>
>
> In non-split DWARF, type units are spec'd to have their own object-file
> section contributions, separate from the compile unit(s);
>
> That's sort of an implementation detail though, isn't it? DWARF just talks
> about bytes in sections (type units go in the debug_types section (or, now,
> the debug_info section)) and, yeah, you can use comdat groups and separate
> chunks of debug_types sections to deduplicate them, but I don't think DWARF
> requires/speaks about that, does it?
>
> Actually DWARF 5 Appendix E does describe this; not as a required tactic,
> but a way to achieve the useful effect of deduplicating type units etc.  So
> yes it was overstating the case to say they are _spec'd_ to have their own
> contributions, but that would be what a producer would normally do.
>

Ah, cool - thanks for the pointer about where the wording is.


>
>
> that's what lets type units have a COMDAT key and be uniqued by the
> linker, even though all those separate contributions have the same section
> name.  (In ELF, you have multiple section headers with the same section
> name.)  Surely the DWO file could be (is?) done the same way, with each
> type unit in its own contribution to the .debug_info section?
>
> Funny story about that...
>
> Heh.  Which way works better (types all together, or each in their own
> contribution) depends on whether your packager wants to deduplicate in a
> linker-like way, based on COMDATs,
>

I think neither GCC nor Clang used COMDATs in DWO files - but GCC still put
them in separate sections (sections with the same name... ) - a weird beast
to me, but apparently it's a thing that works.

But yeah, for non-Fission, COMDATs seem solid, though do represent a
limitation on compressibility, etc.


> or in a purpose-built way, by looking at the type-unit signatures
> directly.  DWARF doesn't say you have to do one or the other, which
> provides implementation flexibility to the toolchain.  If your packager is
> willing to look through TUs for signatures and deduplicate that way, then
> you can stuff all the TUs into one section contribution and get better
> compression.  Quality of Implementation, as we like to say.
>

To be sure!


>
>
> still there would need to be a special case where the TU's debug_info
> chunk would have an INFO_FILE contribution that represente the CU chunk. So
> a DWP tool would have to special case the info chunk that contained the CUs
> (& would have to require that there be only one if there are to be TU->CU
> references. I suppose if TU->CU references aren't supported
>
>
>
> I don't think TU->CU references are permitted.
>

For now, with Fission, I agree.

I think without Fission you could certainly use ref_addr to refer to
something in a CU from a TU - /maybe/ even from a TU to another TU but I
don't think so (not sure if the linker would do the right thing about
reachability, etc - and if your TUs differed in layout, which they can even
in Clang, that wouldn't work out well if it picked a different TU and
either null'd out the ref_addr, or made it refer to the same offset in a
different copy of the type (I don't think any linker/reloc construct would
really result in this latter situation))


> Certainly you could not have a v4 split TU referencing a CU, that would be
> impossible.
>

v4 didn't have split things (Fission being a v5 feature, I think), did it?
What's the distinction you're drawing there?

  (Without relocations, you can't use DW_FORM_ref_addr to point from
> .debug_types to .debug_info; and DW_FORM_ref_sig8 is only for references to
> other type units.)  While you could engineer the possibility in v5, because
> type units have moved back into .debug_info and in principle you could
> arrange for DW_FORM_ref_addr to do that, I am morally certain there was no
> intent to allow that.
>

Right, I doubt there was any intent - but as we're choosing some new
representations, etc, I'm wondering if it's something to think about.

Even without the TU->XU reference question, the TU/CU unification still
means that a DWP creation tool would have to special case the CUs in some,
or require the TUs to be placed in separate sections as GCC does it. (then
it could treat each unit section as an indivisible blob)

Maybe this fits into quality of implementation - but I Think the presence
of cross-unit references makes this a bit more of a matter for the standard
as to how these groups are defined, where cross-CU references are resolved
relative to, how can type units be dropped (or not), etc.

I'm sort of leaning towards "ref_addr offsets are resolved relative to the
widest range of CUs in a single section that contains the referring DIE" -
though that is a bit of a mouthful/awkward thing to implement.

- Dave


> --paulr
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170505/25820836/attachment.htm>


From dblaikie@gmail.com  Fri May  5 02:43:04 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Fri, 05 May 2017 02:43:04 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>

On Thu, May 4, 2017 at 6:22 PM Robinson, Paul <paul.robinson at sony.com>
wrote:

> I think it's pretty safe to say:
>
> -        A reference into a TU from a CU or a different TU is invariably
> by ref_sig8, never by section offset.
>
> -        A reference into a CU from another CU has to be by ref_addr; in
> a .o file this can use a relocation, in a .dwo file it has to be from
> inside the same .debug_info contribution.
>
> -        A reference into a CU from a TU is not allowed, even if the TU
> lives in the same .debug_info contribution.
>
> I don't have my hands on words in the document that say these things, but
> I am quite sure that's the intent.  It's not important whether object-file
> mechanics would allow you to do the things that aren't allowed above.
>

Fair points, all - I'd be curious to see the wording, for sure. From what I
could read it certainly seemed implied/assumed, but not explicit.

I was mentioning this mostly by way of opportunistically suggesting "it
might be a thing that would be good to think explicitly about, possibly
allow, and design this new stuff around the possibility, perhaps".


>
>
> The rest of what I'm suggesting all follows (reiterating for clarity):
>
> -        If a .dwo file has multiple split-full CUs, they each have a
> unique DWO ID (so the index can describe them individually).
>
> -        Therefore, the corresponding .o has a distinct corresponding
> skeleton CU for each split-full CU.
>
> -        Cross-CU references within a .dwo file are by DW_FORM_ref_addr
> and the related CUs must be in the same .debug_info contribution.
>
> -        Split-full CUs without cross-CU references can be in separate
> .debug_info contributions within the .dwo file.
>
Adding semantics to "contributions" in the .dwo file seems like a big step
that wasn't present before I think.

It would mandate the use of (at least 2) sections when using Fission+type
units, reducing some compression opportunity. (yeah, in theory,
implementation detail - a platform could agree to some other format where,
say, .debug_info.dwo starts with an int specifying how long the CU prefix
is, after which there are type units)

> -        A packager should look for multiple CUs in a .debug_info
> contribution, be willing to create an index entry for each one, and not
> split up the contribution even if one or more of the CUs has already been
> included from elsewhere.
>
But only for CUs, not TUs, right. So as long as the producer used a
separate section chunk/"contribution" for the TUs (even in the v5 TU/CU
unification into the debug_info section) then the packager could continue
to fragment a chunk containing only TUs, but if it contained any CUs that
chunk would be indivisible.

> -        A packager can drop an entire .debug_info contribution if *all*
> of the CUs in that contribution have been included from elsewhere.  (This
> trivially covers the one-CU-per-contribution case.)
>
> -        The package index should get a new column to describe the entire
> .debug_info contribution containing the CU, so that consumers can know how
> to resolve DW_FORM_ref_addr.
>
>
>
> You're probably still thinking of wrinkles I haven't addressed; let me
> know.
>

Not much really - it about covers it, I think it just gets a little hairy
in spots mentioned above.

- Dave


> --paulr
>
>
>
> *From:* David Blaikie [mailto:dblaikie at gmail.com]
> *Sent:* Thursday, May 04, 2017 5:30 PM
> *To:* Robinson, Paul; dwarf-discuss at lists.dwarfstd.org; Eric Christopher
> *Subject:* Re: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
>
>
>
>
>
> On Thu, May 4, 2017 at 5:05 PM Robinson, Paul <paul.robinson at sony.com>
> wrote:
>
> Skeleton units are pretty small; it's a 20-byte header, plus values for
> the compile_unit DIE, which is spec'd to have no children.  I would not be
> concerned about space there.  And having unique DWO IDs per unit seems
> pretty useful.
>
> I tend to agree, though - what sort of uses do you have in mind?
>
>
>
> A unique DWO ID per unit lets each DWO unit have a distinct entry in the
> index? saves the consumer the trouble of having to read the .debug_info
> section to find the units.
>
>
> Yep
>
>
>   If you want to require consumers to do more work, you can make DWO IDs
> be per-file instead of per-unit, and then there's no need for an INFO_FILE
> column because the INFO column would necessarily have to cover the entire
> .debug_info section from that file.
>
>
> Yep - time/space tradeoff, and I'd probably err on the side of time myself
> (by having separate skeletons and cu_index entries for each CU) as you've
> suggested. Just floating the other as an alternative since it did come up.
>
>
>
>
> In non-split DWARF, type units are spec'd to have their own object-file
> section contributions, separate from the compile unit(s);
>
> That's sort of an implementation detail though, isn't it? DWARF just talks
> about bytes in sections (type units go in the debug_types section (or, now,
> the debug_info section)) and, yeah, you can use comdat groups and separate
> chunks of debug_types sections to deduplicate them, but I don't think DWARF
> requires/speaks about that, does it?
>
> Actually DWARF 5 Appendix E does describe this; not as a required tactic,
> but a way to achieve the useful effect of deduplicating type units etc.  So
> yes it was overstating the case to say they are _spec'd_ to have their own
> contributions, but that would be what a producer would normally do.
>
>
> Ah, cool - thanks for the pointer about where the wording is.
>
>
>
>
> that's what lets type units have a COMDAT key and be uniqued by the
> linker, even though all those separate contributions have the same section
> name.  (In ELF, you have multiple section headers with the same section
> name.)  Surely the DWO file could be (is?) done the same way, with each
> type unit in its own contribution to the .debug_info section?
>
> Funny story about that...
>
> Heh.  Which way works better (types all together, or each in their own
> contribution) depends on whether your packager wants to deduplicate in a
> linker-like way, based on COMDATs,
>
>
> I think neither GCC nor Clang used COMDATs in DWO files - but GCC still
> put them in separate sections (sections with the same name... ) - a weird
> beast to me, but apparently it's a thing that works.
>
> But yeah, for non-Fission, COMDATs seem solid, though do represent a
> limitation on compressibility, etc.
>
>
> or in a purpose-built way, by looking at the type-unit signatures
> directly.  DWARF doesn't say you have to do one or the other, which
> provides implementation flexibility to the toolchain.  If your packager is
> willing to look through TUs for signatures and deduplicate that way, then
> you can stuff all the TUs into one section contribution and get better
> compression.  Quality of Implementation, as we like to say.
>
>
> To be sure!
>
>
>
>
> still there would need to be a special case where the TU's debug_info
> chunk would have an INFO_FILE contribution that represente the CU chunk. So
> a DWP tool would have to special case the info chunk that contained the CUs
> (& would have to require that there be only one if there are to be TU->CU
> references. I suppose if TU->CU references aren't supported
>
>
>
> I don't think TU->CU references are permitted.
>
>
> For now, with Fission, I agree.
>
> I think without Fission you could certainly use ref_addr to refer to
> something in a CU from a TU - /maybe/ even from a TU to another TU but I
> don't think so (not sure if the linker would do the right thing about
> reachability, etc - and if your TUs differed in layout, which they can even
> in Clang, that wouldn't work out well if it picked a different TU and
> either null'd out the ref_addr, or made it refer to the same offset in a
> different copy of the type (I don't think any linker/reloc construct would
> really result in this latter situation))
>
>
> Certainly you could not have a v4 split TU referencing a CU, that would be
> impossible.
>
>
>
> v4 didn't have split things (Fission being a v5 feature, I think), did it?
> What's the distinction you're drawing there?
>
>
>
>   (Without relocations, you can't use DW_FORM_ref_addr to point from
> .debug_types to .debug_info; and DW_FORM_ref_sig8 is only for references to
> other type units.)  While you could engineer the possibility in v5, because
> type units have moved back into .debug_info and in principle you could
> arrange for DW_FORM_ref_addr to do that, I am morally certain there was no
> intent to allow that.
>
>
> Right, I doubt there was any intent - but as we're choosing some new
> representations, etc, I'm wondering if it's something to think about.
>
> Even without the TU->XU reference question, the TU/CU unification still
> means that a DWP creation tool would have to special case the CUs in some,
> or require the TUs to be placed in separate sections as GCC does it. (then
> it could treat each unit section as an indivisible blob)
>
> Maybe this fits into quality of implementation - but I Think the presence
> of cross-unit references makes this a bit more of a matter for the standard
> as to how these groups are defined, where cross-CU references are resolved
> relative to, how can type units be dropped (or not), etc.
>
> I'm sort of leaning towards "ref_addr offsets are resolved relative to the
> widest range of CUs in a single section that contains the referring DIE" -
> though that is a bit of a mouthful/awkward thing to implement.
>
> - Dave
>
>
> --paulr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170505/79bc250d/attachment-0001.htm>


From paul.robinson@sony.com  Fri May  5 01:22:08 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Fri, 5 May 2017 01:22:08 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>

I think it's pretty safe to say:

-        A reference into a TU from a CU or a different TU is invariably by ref_sig8, never by section offset.

-        A reference into a CU from another CU has to be by ref_addr; in a .o file this can use a relocation, in a .dwo file it has to be from inside the same .debug_info contribution.

-        A reference into a CU from a TU is not allowed, even if the TU lives in the same .debug_info contribution.
I don't have my hands on words in the document that say these things, but I am quite sure that's the intent.  It's not important whether object-file mechanics would allow you to do the things that aren't allowed above.

The rest of what I'm suggesting all follows (reiterating for clarity):

-        If a .dwo file has multiple split-full CUs, they each have a unique DWO ID (so the index can describe them individually).

-        Therefore, the corresponding .o has a distinct corresponding skeleton CU for each split-full CU.

-        Cross-CU references within a .dwo file are by DW_FORM_ref_addr and the related CUs must be in the same .debug_info contribution.

-        Split-full CUs without cross-CU references can be in separate .debug_info contributions within the .dwo file.

-        A packager should look for multiple CUs in a .debug_info contribution, be willing to create an index entry for each one, and not split up the contribution even if one or more of the CUs has already been included from elsewhere.

-        A packager can drop an entire .debug_info contribution if *all* of the CUs in that contribution have been included from elsewhere.  (This trivially covers the one-CU-per-contribution case.)

-        The package index should get a new column to describe the entire .debug_info contribution containing the CU, so that consumers can know how to resolve DW_FORM_ref_addr.

You're probably still thinking of wrinkles I haven't addressed; let me know.
--paulr

From: David Blaikie [mailto:dblaikie@gmail.com]
Sent: Thursday, May 04, 2017 5:30 PM
To: Robinson, Paul; dwarf-discuss at lists.dwarfstd.org; Eric Christopher
Subject: Re: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)


On Thu, May 4, 2017 at 5:05 PM Robinson, Paul <paul.robinson at sony.com<mailto:paul.robinson at sony.com>> wrote:
Skeleton units are pretty small; it's a 20-byte header, plus values for the compile_unit DIE, which is spec'd to have no children.  I would not be concerned about space there.  And having unique DWO IDs per unit seems pretty useful.
I tend to agree, though - what sort of uses do you have in mind?

A unique DWO ID per unit lets each DWO unit have a distinct entry in the index? saves the consumer the trouble of having to read the .debug_info section to find the units.

Yep

  If you want to require consumers to do more work, you can make DWO IDs be per-file instead of per-unit, and then there's no need for an INFO_FILE column because the INFO column would necessarily have to cover the entire .debug_info section from that file.

Yep - time/space tradeoff, and I'd probably err on the side of time myself (by having separate skeletons and cu_index entries for each CU) as you've suggested. Just floating the other as an alternative since it did come up.


In non-split DWARF, type units are spec'd to have their own object-file section contributions, separate from the compile unit(s);
That's sort of an implementation detail though, isn't it? DWARF just talks about bytes in sections (type units go in the debug_types section (or, now, the debug_info section)) and, yeah, you can use comdat groups and separate chunks of debug_types sections to deduplicate them, but I don't think DWARF requires/speaks about that, does it?
Actually DWARF 5 Appendix E does describe this; not as a required tactic, but a way to achieve the useful effect of deduplicating type units etc.  So yes it was overstating the case to say they are _spec'd_ to have their own contributions, but that would be what a producer would normally do.

Ah, cool - thanks for the pointer about where the wording is.


that's what lets type units have a COMDAT key and be uniqued by the linker, even though all those separate contributions have the same section name.  (In ELF, you have multiple section headers with the same section name.)  Surely the DWO file could be (is?) done the same way, with each type unit in its own contribution to the .debug_info section?
Funny story about that...
Heh.  Which way works better (types all together, or each in their own contribution) depends on whether your packager wants to deduplicate in a linker-like way, based on COMDATs,

I think neither GCC nor Clang used COMDATs in DWO files - but GCC still put them in separate sections (sections with the same name... ) - a weird beast to me, but apparently it's a thing that works.

But yeah, for non-Fission, COMDATs seem solid, though do represent a limitation on compressibility, etc.

or in a purpose-built way, by looking at the type-unit signatures directly.  DWARF doesn't say you have to do one or the other, which provides implementation flexibility to the toolchain.  If your packager is willing to look through TUs for signatures and deduplicate that way, then you can stuff all the TUs into one section contribution and get better compression.  Quality of Implementation, as we like to say.

To be sure!


still there would need to be a special case where the TU's debug_info chunk would have an INFO_FILE contribution that represente the CU chunk. So a DWP tool would have to special case the info chunk that contained the CUs (& would have to require that there be only one if there are to be TU->CU references. I suppose if TU->CU references aren't supported

I don't think TU->CU references are permitted.

For now, with Fission, I agree.

I think without Fission you could certainly use ref_addr to refer to something in a CU from a TU - /maybe/ even from a TU to another TU but I don't think so (not sure if the linker would do the right thing about reachability, etc - and if your TUs differed in layout, which they can even in Clang, that wouldn't work out well if it picked a different TU and either null'd out the ref_addr, or made it refer to the same offset in a different copy of the type (I don't think any linker/reloc construct would really result in this latter situation))

Certainly you could not have a v4 split TU referencing a CU, that would be impossible.

v4 didn't have split things (Fission being a v5 feature, I think), did it? What's the distinction you're drawing there?

  (Without relocations, you can't use DW_FORM_ref_addr to point from .debug_types to .debug_info; and DW_FORM_ref_sig8 is only for references to other type units.)  While you could engineer the possibility in v5, because type units have moved back into .debug_info and in principle you could arrange for DW_FORM_ref_addr to do that, I am morally certain there was no intent to allow that.

Right, I doubt there was any intent - but as we're choosing some new representations, etc, I'm wondering if it's something to think about.

Even without the TU->XU reference question, the TU/CU unification still means that a DWP creation tool would have to special case the CUs in some, or require the TUs to be placed in separate sections as GCC does it. (then it could treat each unit section as an indivisible blob)

Maybe this fits into quality of implementation - but I Think the presence of cross-unit references makes this a bit more of a matter for the standard as to how these groups are defined, where cross-CU references are resolved relative to, how can type units be dropped (or not), etc.

I'm sort of leaning towards "ref_addr offsets are resolved relative to the widest range of CUs in a single section that contains the referring DIE" - though that is a bit of a mouthful/awkward thing to implement.

- Dave

--paulr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20170505/b50f5da4/attachment-0001.htm>


From paul.robinson@sony.com  Fri May  5 18:14:41 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Fri, 5 May 2017 18:14:41 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
	<CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21B701@USCULXMSG08.am.sony.com>

I was mentioning this mostly by way of opportunistically suggesting "it might be a thing that would be good to think explicitly about, possibly allow, and design this new stuff around the possibility, perhaps".
The entire point of a type unit is to allow size reduction by deduplication.  Any kind of section-offset reference into a TU depends on that exact instance of the TU to remain; therefore such references defeat the purpose of type units.  Do we need to have explicit words that say so?  I hope not.
Similarly, a reference from a TU to a CU implies that the TU is not self-contained and therefore the exactness of any duplicates is suspect.  What guarantees that the information referred to is the same across all "duplicate" TUs or otherwise does not affect the semantic content of the TU?  There is none.  So, this kind of reference defeats the purpose of a TU.  Do we need to have explicit words that say so?  I hope not.

Adding semantics to "contributions" in the .dwo file seems like a big step that wasn't present before I think.

It would mandate the use of (at least 2) sections when using Fission+type units, reducing some compression opportunity

Deduplication by COMDAT in v4 was depending on semantics of object-file characteristics; additional semantics don't seem like that big a leap to me.  LTO implications (e.g. cross-CU references in a DWO file, which can't have relocations) are a new thing that we hadn't considered before.  I don't have a problem with new situations with new implications needing new paragraphs/sections/requirements/whatever.  We even have a place to put stuff like that now (Appendix E).
--paulr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170505/5ee7194e/attachment.htm>


From dblaikie@gmail.com  Fri May  5 18:37:32 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Fri, 05 May 2017 18:37:32 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21B701@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
	<CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21B701@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6EtWg=knpMe-WCjx5zJVVfg10ViFzX1bTMB=xRr6o28GCg@mail.gmail.com>

On Fri, May 5, 2017 at 11:14 AM Robinson, Paul <paul.robinson at sony.com>
wrote:

> I was mentioning this mostly by way of opportunistically suggesting "it
> might be a thing that would be good to think explicitly about, possibly
> allow, and design this new stuff around the possibility, perhaps".
>
> The entire point of a type unit is to allow size reduction by
> deduplication.  Any kind of section-offset reference into a TU depends on
> that exact instance of the TU to remain; therefore such references defeat
> the purpose of type units.  Do we need to have explicit words that say so?
> I hope not.
>

Sure, agreed for a TU to TU reference.


> Similarly, a reference from a TU to a CU implies that the TU is not
> self-contained and therefore the exactness of any duplicates is suspect.
>

I disagree here. TU can be semantically equivalent while not being
bit-for-bit identical, and the presence of a TU->CU reference doesn't make
it less reliably able to maintain that property.


>   What guarantees that the information referred to is the same across all
> "duplicate" TUs or otherwise does not affect the semantic content of the TU?
>

The same guarantees that exist without TU->CU references, I would think?
It's up to the implementation to ensure that any two TUs are semantically
equivalent. (so in one TU you might have a TU->CU reference to refer to a
type, in another you might have a declaration of that type in the TU
itself, because no definition is available)


>   There is none.  So, this kind of reference defeats the purpose of a TU.
> Do we need to have explicit words that say so?  I hope not.
>
>
>
> Adding semantics to "contributions" in the .dwo file seems like a big step
> that wasn't present before I think.
>
> It would mandate the use of (at least 2) sections when using Fission+type
> units, reducing some compression opportunity
>
>
>
> Deduplication by COMDAT in v4 was depending on semantics of object-file
> characteristics;
>

But that's an implementation detail - not a requirement of the format to
convey the necessary information. The necessary grouping of chunks of DWARF
seems to be a semantic feature that hasn't been necessary to interpret
DWARF correctly until this point/proposal. An inefficient but correct
implementation could concatenate all the DWARF together, including the
duplicates - or could do the DWARF aware thing that DWP requires and look
for type unit boundaries and deduplicate on those, etc.

eg: if a file format didn't support separate chunks of sections by the same
name (which, to me, seems like a somewhat odd feature, TBH - caught me by
surprise that "you don't need to use COMDATs for type units in DWO files"
meant anything other than "all the type units go in the (singular)
debug_types section") it wouldn't support this format without extra
bookkeeping which would seem weird to me. In all prior cases the DWARF
bytes have described the bounds of relevant chunks that a consumer would
need to care about, right? (pretty much everything has a header that
specifies its length, and/or has terminators to specify where it ends and
the next thing begins, etc)

Type units enabled the chunks to be separated so existing linker features
could handle them - but didn't /require/ them to be separated to convey
specific DWARF semantics. That's the distinction I think I'm trying to
draw/be surprised by/etc.

DWARF requires a format that supports different named sections - but until
this proposal/discussion hasn't required it to support the notion of
different chunks in the same named section, it seems?

Sorry, just trying different phrasing to see if I can get something that
seems to best convey what's in my head - not sure if it's making sense.


> additional semantics don't seem like that big a leap to me.  LTO
> implications (e.g. cross-CU references in a DWO file, which can't have
> relocations) are a new thing that we hadn't considered before.  I don't
> have a problem with new situations with new implications needing new
> paragraphs/sections/requirements/whatever.  We even have a place to put
> stuff like that now (Appendix E).
>
> --paulr
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170505/96077e09/attachment-0001.htm>


From paul.robinson@sony.com  Fri May  5 18:50:48 2017
From: paul.robinson@sony.com (Robinson, Paul)
Date: Fri, 5 May 2017 18:50:48 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EtWg=knpMe-WCjx5zJVVfg10ViFzX1bTMB=xRr6o28GCg@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
	<CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21B701@USCULXMSG08.am.sony.com>
	<CAENS6EtWg=knpMe-WCjx5zJVVfg10ViFzX1bTMB=xRr6o28GCg@mail.gmail.com>
Message-ID: <E3B07FDB86BFF041819DC057DEED8FEA012F21B7CC@USCULXMSG08.am.sony.com>

  What guarantees that the information referred to is the same across all "duplicate" TUs or otherwise does not affect the semantic content of the TU?
The same guarantees that exist without TU->CU references, I would think? It's up to the implementation to ensure that any two TUs are semantically equivalent.

No it's not.  DWARF specifies what makes TU's semantically equivalent, by specifying what feeds into the signature computation, and matching signatures provide the guarantee.  References use forms that are not included in the signature and therefore the referenced information is not included in the guarantee.
--paulr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170505/0c356fd9/attachment.htm>


From dblaikie@gmail.com  Fri May  5 19:06:20 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Fri, 05 May 2017 19:06:20 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <E3B07FDB86BFF041819DC057DEED8FEA012F21B7CC@USCULXMSG08.am.sony.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A4C4@USCULXMSG08.am.sony.com>
	<CAENS6EsbdgjiZicpnE9MK0YaPhhbjx755Q5fEKHXOUE2fwpPnA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21A621@USCULXMSG08.am.sony.com>
	<CAENS6EuYY6tSZitS0X3CWsNdF6wEcN-nnyA2X-eieMPQWvgPXA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AE50@USCULXMSG08.am.sony.com>
	<CAENS6EvDbfEv=MhVp+fZWfCyOmZDb-ctoV4YTVOd4xXGBvPy7Q@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21AFD4@USCULXMSG08.am.sony.com>
	<CAENS6Es+ke3h0iuP8T_DjMW0tYBCA=xp34og14SWxZ=bCgEwbA@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21B701@USCULXMSG08.am.sony.com>
	<CAENS6EtWg=knpMe-WCjx5zJVVfg10ViFzX1bTMB=xRr6o28GCg@mail.gmail.com>
	<E3B07FDB86BFF041819DC057DEED8FEA012F21B7CC@USCULXMSG08.am.sony.com>
Message-ID: <CAENS6EuLrXYUtySp5G6TmOsQBW1oYGEkjvgaV3hoofirBfUjDA@mail.gmail.com>

On Fri, May 5, 2017 at 11:50 AM Robinson, Paul <paul.robinson at sony.com>
wrote:

>   What guarantees that the information referred to is the same across all
> "duplicate" TUs or otherwise does not affect the semantic content of the TU?
>
> The same guarantees that exist without TU->CU references, I would think?
> It's up to the implementation to ensure that any two TUs are semantically
> equivalent.
>
>
>
> No it's not.  DWARF specifies what makes TU's semantically equivalent, by
> specifying what feeds into the signature computation, and matching
> signatures provide the guarantee.  References use forms that are not
> included in the signature and therefore the referenced information is not
> included in the guarantee.
>

Fair enough, I though maybe that'd changed/become more open-ended - LLVM
goes off-spec here and uses a hash of the mangled name of the type. This
ensures more (semantically equivalent) definitions can be deduplicated and
reduces the overhead/complexity of the implementation. (it does risk LLVM's
hashes colliding with another implementation's hashes in
surprising/incorrect ways, though).

In any case that's somewhat of a tangent to my other/broader
point/curiosity about the structural representation/implications of having
a concept of "contributions" that's not "the whole section" or "the unit"
in the DWO file.

- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170505/69eb6c61/attachment.htm>


From aoliva@redhat.com  Fri May 12 09:47:18 2017
From: aoliva@redhat.com (Alexandre Oliva)
Date: Fri, 12 May 2017 06:47:18 -0300
Subject: [Dwarf-Discuss] implementing SFN,
	support for multiple views per PC
In-Reply-To: <59029BE6.8080207@eagercon.com> (Michael Eager's message of "Thu, 
	27 Apr 2017 18:33:26 -0700")
References: <orfup8zwow.fsf@livre.home> <orpoi865kg.fsf@lxoliva.fsfla.org>
	<58AF499A.3060405@eagercon.com> <or1ssd1hve.fsf@lxoliva.fsfla.org>
	<59029BE6.8080207@eagercon.com>
Message-ID: <orfugawgnt.fsf@lxoliva.fsfla.org>

On Apr 27, 2017, Michael Eager <eager at eagercon.com> wrote:

> I don't know in what way the CAPTCHA "takes control of your computer",

The CAPTCHA is software; running software means the computer is
following the instructions encoded in the software, i.e., the software
is controlling (the behavior of) the computer.

> but I'll add your proposals manually.

Thank you!

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer


From dblaikie@gmail.com  Wed May 17 23:17:34 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Wed, 17 May 2017 23:17:34 +0000
Subject: [Dwarf-Discuss] address pool + offset representation
Message-ID: <CAENS6EvgnkgVgZ11JeagDtXE1i6s14nEnT_jNvZpOiQov1=ERA@mail.gmail.com>

A big part of Fission debug info in object files an optimized build, are
unique address relocation in debug_addr and debug_ranges. I have an example
binary where for N bytes of .text, there are ~2N bytes of .rela.debug_addr
and >2N bytes of .rela.debug_ranges.

Given that .rela.debug_line is about ~1% of the size of .text - that gives
a sense of the lower bound - there should be only a few more relocations
needed in .debug_addr than in .debug_line (relocs for global variables).

This arises because things like low_pc (for each subprogram and then each
lexical block inside it) use distinct addresses in the address pool, when
they could use an offset relative to some other known address in the pool
(basically one address for each .text section - and everything would use
that + offset).

The new *x (startx*, base_addressx) forms in the debug_rnglists format
address the relocations for the debug_ranges section - allowing it to reuse
addresses in debug_addr.

But to address debug_addr's redundancy, it'd would be nice to have an
abbreviation form to represent "address in the pool, plus a constant
offset".

What form should this take?

Currently there's addrx{,1-4} - a LEB128, or 1-4 byte fixed-length
representation.
Similarly the offset between addresses could be a variety of lengths.
high_pc for example allows for any form of the constant class.

Should this support the combination of all of these forms
(addrx{,1-4}*data{,1-4}). Or is there some better option?

I hope to try prototyping this as a GNU extension form if/when people have
some idea of what might be best for the representation.


Bonus points: It'd be great to move the address pool into the line table -
then it'd be really one relocation per section. That'd need a new assembly
directive, though. Anyone got ideas on that too? (this would be orthogonal
to the above improvement - both things are good to do)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170517/eb0a40fe/attachment.htm>


From dblaikie@gmail.com  Fri May 19 22:51:18 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Fri, 19 May 2017 22:51:18 +0000
Subject: [Dwarf-Discuss] Including the DWO name in the CU hash
Message-ID: <CAENS6EvGfeOr32N5Yzkkj7pYS+LM-yytO5ROEWdsvOLOC2-4gw@mail.gmail.com>

some context:

1) A little while ago, I added the dwo_name to the dwo CU to improve
diagnostic quality on CU hash collisions in during a dwp action (previously
it could only report which input files (possibly other DWP files) contained
the duplicates/collision which could be very manual to track back to the
original input DWO files - having the original DWO names in the diagnostic
made it relatively easy to track down).

2) LLVM's new ThinLTO presents a high chance of duplicate DWO CUs - it does
this by creating effectively "new" CUs containing a stripped down version
of an existing CU - containing only a handful of functions that may be
relevant to optimizing some other CU. (imagine two CUs both using a single
inline function from a 3rd CU - the 3rd CU's inline function and the basic
CU itself is imported into the compilation steps of the other two CUs - so
in the end you get two DWO files, each with two CUs, where one CU contains
only an abstract definition of the inline function).

My initial thinking here was that I could cross-pollinate the CU hash from
each CU within a single compilation, since the primary CU would have enough
uniqueness (hash all the CUs, then cross-hash them).

But then I realized the CUs should already be unique because they include
the dwo_name which will be different between the two stripped down CU
clones. But the dwo_name isn't included in the hash - so I prototyped
including it & it does what you'd expect.

Extra wrinkle: Once the dwo_name is in the hash, then it defeats my
original motivation for including it in the DWO CU in the first place: such
CUs will never collide, so the name would never be useful for diagnostic
quality.

Should I drop the dwo_name from the DWO CU and manually/explicitly include
it in the hash? Does cross pollination sound better? Should I only do
either of these when dealing with more than one CU in a DWO? (in which case
the diagnostic improvement would still be valid - it catches some
interesting cases, but they're not /very/ interesting like major bugs (&
does DWO ID collisions have some false positives too, which hashing the
dwo_id would fix), etc... and the mechanism wasn't built for bug catching
in any case)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170519/109c014e/attachment.htm>


From dblaikie@gmail.com  Wed May 31 20:25:33 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Wed, 31 May 2017 20:25:33 +0000
Subject: [Dwarf-Discuss] address pool + offset representation
In-Reply-To: <CAENS6EvgnkgVgZ11JeagDtXE1i6s14nEnT_jNvZpOiQov1=ERA@mail.gmail.com>
References: <CAENS6EvgnkgVgZ11JeagDtXE1i6s14nEnT_jNvZpOiQov1=ERA@mail.gmail.com>
Message-ID: <CAENS6Ev7ZPLSvtGZpcP81q=jhW18yR97A1LFq36aehQaiUuC5A@mail.gmail.com>

Ping - any ideas?

On Wed, May 17, 2017 at 4:17 PM David Blaikie <dblaikie at gmail.com> wrote:

> A big part of Fission debug info in object files an optimized build, are
> unique address relocation in debug_addr and debug_ranges. I have an example
> binary where for N bytes of .text, there are ~2N bytes of .rela.debug_addr
> and >2N bytes of .rela.debug_ranges.
>
> Given that .rela.debug_line is about ~1% of the size of .text - that gives
> a sense of the lower bound - there should be only a few more relocations
> needed in .debug_addr than in .debug_line (relocs for global variables).
>
> This arises because things like low_pc (for each subprogram and then each
> lexical block inside it) use distinct addresses in the address pool, when
> they could use an offset relative to some other known address in the pool
> (basically one address for each .text section - and everything would use
> that + offset).
>
> The new *x (startx*, base_addressx) forms in the debug_rnglists format
> address the relocations for the debug_ranges section - allowing it to reuse
> addresses in debug_addr.
>
> But to address debug_addr's redundancy, it'd would be nice to have an
> abbreviation form to represent "address in the pool, plus a constant
> offset".
>
> What form should this take?
>
> Currently there's addrx{,1-4} - a LEB128, or 1-4 byte fixed-length
> representation.
> Similarly the offset between addresses could be a variety of lengths.
> high_pc for example allows for any form of the constant class.
>
> Should this support the combination of all of these forms
> (addrx{,1-4}*data{,1-4}). Or is there some better option?
>
> I hope to try prototyping this as a GNU extension form if/when people have
> some idea of what might be best for the representation.
>
>
> Bonus points: It'd be great to move the address pool into the line table -
> then it'd be really one relocation per section. That'd need a new assembly
> directive, though. Anyone got ideas on that too? (this would be orthogonal
> to the above improvement - both things are good to do)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170531/4b25490a/attachment.htm>


From dblaikie@gmail.com  Wed May 31 20:28:51 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Wed, 31 May 2017 20:28:51 +0000
Subject: [Dwarf-Discuss] Including the DWO name in the CU hash
In-Reply-To: <CAENS6EvGfeOr32N5Yzkkj7pYS+LM-yytO5ROEWdsvOLOC2-4gw@mail.gmail.com>
References: <CAENS6EvGfeOr32N5Yzkkj7pYS+LM-yytO5ROEWdsvOLOC2-4gw@mail.gmail.com>
Message-ID: <CAENS6EuyxBh4csERgvsDReo3P1HQCRoe9iWv72mrATXSkmnF5Q@mail.gmail.com>

Ping?

I did end up hacking around this by hashing in the DWO name into the CU
hash if LLVM's producing more than one CU. It's not perfect (really it's
more about the ThinLTO importing stage - normal LTO doesn't need this sort
of mangling) but suffices for now:
http://llvm.org/viewvc/llvm-project?rev=304119&view=rev

But it seems to suffice for now.

On Fri, May 19, 2017 at 3:51 PM David Blaikie <dblaikie at gmail.com> wrote:

> some context:
>
> 1) A little while ago, I added the dwo_name to the dwo CU to improve
> diagnostic quality on CU hash collisions in during a dwp action (previously
> it could only report which input files (possibly other DWP files) contained
> the duplicates/collision which could be very manual to track back to the
> original input DWO files - having the original DWO names in the diagnostic
> made it relatively easy to track down).
>
> 2) LLVM's new ThinLTO presents a high chance of duplicate DWO CUs - it
> does this by creating effectively "new" CUs containing a stripped down
> version of an existing CU - containing only a handful of functions that may
> be relevant to optimizing some other CU. (imagine two CUs both using a
> single inline function from a 3rd CU - the 3rd CU's inline function and the
> basic CU itself is imported into the compilation steps of the other two CUs
> - so in the end you get two DWO files, each with two CUs, where one CU
> contains only an abstract definition of the inline function).
>
> My initial thinking here was that I could cross-pollinate the CU hash from
> each CU within a single compilation, since the primary CU would have enough
> uniqueness (hash all the CUs, then cross-hash them).
>
> But then I realized the CUs should already be unique because they include
> the dwo_name which will be different between the two stripped down CU
> clones. But the dwo_name isn't included in the hash - so I prototyped
> including it & it does what you'd expect.
>
> Extra wrinkle: Once the dwo_name is in the hash, then it defeats my
> original motivation for including it in the DWO CU in the first place: such
> CUs will never collide, so the name would never be useful for diagnostic
> quality.
>
> Should I drop the dwo_name from the DWO CU and manually/explicitly include
> it in the hash? Does cross pollination sound better? Should I only do
> either of these when dealing with more than one CU in a DWO? (in which case
> the diagnostic improvement would still be valid - it catches some
> interesting cases, but they're not /very/ interesting like major bugs (&
> does DWO ID collisions have some false positives too, which hashing the
> dwo_id would fix), etc... and the mechanism wasn't built for bug catching
> in any case)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170531/52e12b2b/attachment.htm>


From dblaikie@gmail.com  Wed May 31 20:36:48 2017
From: dblaikie@gmail.com (David Blaikie)
Date: Wed, 31 May 2017 20:36:48 +0000
Subject: [Dwarf-Discuss] Fission + cross-CU references (ref_addr)
In-Reply-To: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
References: <CAENS6EuSZ+8__YTXdPgbTfKgs819sVPW972nRFcMF=pB_XLuSw@mail.gmail.com>
Message-ID: <CAENS6EvCObQKPDR=8ZBA3fa4wu5kNyRNj5taO18YTu+-J8wWdg@mail.gmail.com>

Pinging this thread & bringing it to the attention of Cary & Doug in case
they've got some thoughts.

I'd love to have some kind of ideas about this to work on an implementation
in GDB (as an extension/prototype).

(don't mean to dismiss Paul's suggestion - and I generally agree with it,
except where it gets a bit difficult when type units are folded into
debug_info (Paul's suggestion requiring some kind of grouping within a
section ("contributions") which seems like a new constraint/detail I
wouldn't think to have here))

On another thread Cary talked about the idea of avoiding ref_addr and
expanding the functionality of DW_FORM_sig8/signature and perhaps
generalizing the type unit header to allow multiple hash+offset pairs. This
is a feature Adrian & I had talked about previously (when looking at the
overhead of type units - the need to duplicate member function declarations
in any unit that has a definition of the member function, since the
specific DIE in the type unit couldn't be referred to from the definition
in the CU) & I think would be great (& would further generalize the unit
header - no need for a difference between TU and CU header, instead have a
"count" & then that many {hash, offset} pairs for referring to any DIEs
from elsewhere). Might not be the perfect solution to the CU->TU
references, since signatures are large-ish. So might require a bit more
care/data/etc.

I'm also not sure/don't think I'd have the time to push that ^ feature, as
much as it is a nice one. Paul's cu_index column extension wouldn't be too
hard to implement, I think.

On Tue, May 2, 2017 at 12:09 PM David Blaikie <dblaikie at gmail.com> wrote:

> I've recently been trying to resolve the use of Fission in LLVM's ThinLTO
> mode (though this would apply to plain LTO too).
>
> One of the things that happens here is that cross-CU DIE references
> (DW_FORM_ref_addr) are used to describe inlining a function in one CU into
> another CU.
>
> This format has been implemented in LLVM and GCC for ~years and seems to
> work well outside of Fission.
>
> So the question is: what to do with Fission?
>
> It seemed to me that a good representation would be to produce multiple
> CUs into a single DWO file, which GDB can't yet consume, but I'm working on
> patches to help there. DW_FORM_ref_addr would not use any ELF relocation,
> but be assumed to be "relative to the chunk of debug_info it was in"
> (within the .dwo file)
>
> But what about DWP files? Currently binutils dwp produces records like
> this:
>
> (this dwp contains 3 CUs, two from one LTO compile, and one from a
> standalone compile linked in for comparison):
>
> Index Signature          INFO     ABBR     LINE     STR_OFF
> ----- ------------------ -------- -------- -------- --------
>     2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c)
>     8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14)
>    11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)
>
> So the ABBR/LINE/STR_OFF sections are kept as-is (no analysis is done to
> find which portions of the dwo file are used by which CUs, etc), but the
> INFO section is fragmented on the CU boundaries. Fragmenting the TYPES
> section on the TU boundaries is necessary/useful for deduplication of
> types, but this fragmenting of the CU makes it impossible (I think) to use
> ref_addr in a dwp file.
>
> If this fragmenting were not done - consumers (GDB, etc) would need to
> change to account for this - searching through the INFO range to find the
> CU matching the signature, rather than knowing it starts at the start of
> the INFO range. This could have a noticeable performance impact especially
> in a full LTO build (where /all/ the CUs were in the same .dwo - so the
> index would be entirely unhelpful, I think).
>
> Does all this sound right/sane - anyone have ideas/perspectives/thoughts
> on how this should work?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20170531/5669bc0e/attachment.htm>


