From Anitha.Boyapati@atmel.com  Mon Sep  5 09:15:04 2011
From: Anitha.Boyapati@atmel.com (Boyapati, Anitha)
Date: Mon, 5 Sep 2011 17:15:04 +0800
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for DW_AT_bit_offset
	fields
Message-ID: <C92FFA3D5F1BC641990154AC4B1C50BA0250590A@penmb01.corp.atmel.com>



Hello All,

In DWARF2, I have some trouble understanding why GCC is generating negative offsets for DW_AT_bit_offset field of structure using bitfields (C language). The target is little-endian. 

Incidentally I happen to come across the endianity issues discussed in http://dwarfstd.org/ShowIssue.php?issue=081130.1&type=closed3


It mentions the original problem described in three emails -

"
Brender email of 2008-05-17
Robinson email of 2008-05-19
Allen email of 2008-07-07
"


I tried searching them in dwarf-discuss archives but could not find. Can someone give me the links to the above conversations?

Thanks


Anitha



From ron.brender@gmail.com  Tue Sep  6 10:34:55 2011
From: ron.brender@gmail.com (Ron Brender)
Date: Tue, 06 Sep 2011 06:34:55 -0400
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for DW_AT_bit_offset
 fields
Message-ID: <4E65F74F.80406@charter.net>

The three emails referenced are attached.

Ron Brender

---------------
Hello All,

In DWARF2, I have some trouble understanding why GCC is generating 
negative offsets for DW_AT_bit_offset field of structure using bitfields 
(C language). The target is little-endian.

Incidentally I happen to come across the endianity issues discussed in 
http://dwarfstd.org/ShowIssue.php?issue=081130.1&type=closed3


It mentions the original problem described in three emails -

"
Brender email of 2008-05-17
Robinson email of 2008-05-19
Allen email of 2008-07-07
"


I tried searching them in dwarf-discuss archives but could not find. Can 
someone give me the links to the above conversations?

Thanks


Anitha
_______________________________________________
Dwarf-Discuss mailing list
Dwarf-Discuss at lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Allen email of 2008-07-07.txt
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20110906/552c8570/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Brender email of 2008-05-17.txt
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20110906/552c8570/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Robinson email of 2008-05-19.txt
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20110906/552c8570/attachment-0002.txt>


From Anitha.Boyapati@atmel.com  Wed Sep  7 13:52:57 2011
From: Anitha.Boyapati@atmel.com (Boyapati, Anitha)
Date: Wed, 7 Sep 2011 21:52:57 +0800
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for DW_AT_bit_offset
	fields
In-Reply-To: <4E65F74F.80406@charter.net>
References: <4E65F74F.80406@charter.net>
Message-ID: <C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>


Hello Brender,

Thanks for the attachments. The more I go through the issue, the more confused I am!

To start with I'll stick to simple example. Please help me out.

struct test {
        unsigned char a:1;
        unsigned char b:1;
        unsigned int  k:5;
        unsigned char l:2;
};

On a little-endian machine like x86, what is (are) the expected values for DW_AT_byte_size, DW_AT_bit_offset, DW_AT_data_member_location for member l of test?

That said, with gcc, I see the following output:

Abbrev Number: 3 (DW_TAG_member)
     DW_AT_name        : l
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 6
     DW_AT_type        : <af>
     DW_AT_byte_size   : 1
     DW_AT_bit_size    : 2
     DW_AT_bit_offset  : 6
     DW_AT_data_member_location: 2 byte block: 23 1     (DW_OP_plus_uconst: 1)


I don't understand how DW_AT_bit_offset turns out to be 6. Following the convention used by Allen:

39    32 31    24 23    16 15     8 7      0
+--------+--------+--------+--------+--------+
| addr+4 | addr+3 | addr+2 | addr+1 | addr+0 |
+--------+--------+--------+--------+--------+
 ..................................L lKkkkkBA

Should not DW_AT_byte_size for 'l' be 2, DW_AT_bit_offset be 7 and DW_AT_data_member_location be 0?

Please correct me if I am going wrong.


Anitha

>-----Original Message-----
>From: Ron Brender [mailto:ron.brender at gmail.com]
>Sent: Tuesday, September 06, 2011 4:05 PM
>To: Boyapati, Anitha; DWARF discussion mailing list
>Subject: RE: [Dwarf-Discuss] DWARF2 - Endianity issues for DW_AT_bit_offset
>fields
>
>The three emails referenced are attached.
>
>Ron Brender
>
>---------------
>Hello All,
>
>In DWARF2, I have some trouble understanding why GCC is generating
>negative offsets for DW_AT_bit_offset field of structure using bitfields
>(C language). The target is little-endian.
>
>Incidentally I happen to come across the endianity issues discussed in
>http://dwarfstd.org/ShowIssue.php?issue=081130.1&type=closed3
>
>
>It mentions the original problem described in three emails -
>
>"
>Brender email of 2008-05-17
>Robinson email of 2008-05-19
>Allen email of 2008-07-07
>"
>
>
>I tried searching them in dwarf-discuss archives but could not find. Can
>someone give me the links to the above conversations?
>
>Thanks
>
>
>Anitha
>_______________________________________________
>Dwarf-Discuss mailing list
>Dwarf-Discuss at lists.dwarfstd.org
>http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org



From David.Earlam@csr.com  Wed Sep  7 15:36:26 2011
From: David.Earlam@csr.com (David Earlam)
Date: Wed, 7 Sep 2011 15:36:26 +0000
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for
	DW_AT_bit_offset	fields
In-Reply-To: <C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>
References: <4E65F74F.80406@charter.net>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>
Message-ID: <FC0D83C633273A488E45F10E6CBCB46130EBBAEB@CAMEUREXM01.EUROPE.ROOT.PRI>

Anitha, 

I checked two other little-endian 32-bit, octet-addressing C compilers, one producing Dwarf2, one producing Dwarf3.

They both agree with the gcc you used on little-endian x86.

<2><b3>: Abbrev Number: 37 (DW_TAG_member)
   <b4>   DW_AT_name        : l
   <b6>   DW_AT_type        : DW_FORM_ref1 <0x61> "unsigned char"
   <b8>   DW_AT_data_member_location: 2 byte block: 23 1                         (DW_OP_plus_uconst: 1)
   <bb>   DW_AT_byte_size   : 1
   <bc>   DW_AT_bit_size    : 2
   <bd>   DW_AT_bit_offset  : 6


When the bitfield 'l' would have straddled two storage units, the compiler chose to start a new storage unit (C structure packing is implementation- defined). Updates to 'l' then only need access one byte at addr+1 with a mask 0x03.

If the bitfield straddled two bytes, generated code would usually need to access two bytes, shift the first left 1, shift the second right 7, bitwise-or together and maybe mask with 0x03 to form the field into a unsigned char value (or a similar sequence leastwise on a processor without deposit/extract instructions). 

So with that debug_info, the struct test surely looks like this:

 39    32 31    24 23    16 15     8 7      0
 +--------+--------+--------+--------+--------+
 | addr+4 | addr+3 | addr+2 | addr+1 | addr+0 |
 +--------+--------+--------+--------+--------+
  .................................Ll pKkkkkBA

where p is padding.

That might not be the case for __packed structures, if the compiler has this non-standard extension, which avoid such padding and pack as closely as possible.

According to Dwarf 2.0.0 section 5.5.4 Structure Data Member Entries,
the DW_AT_data_member_location is the address relative to the composite object that most closely encloses the bitfield; so if 'l' straddled that relative address could be DW_OP_plus_uconst 0 or DW_OP_plus_uconst 1, but the DW_AT_bit_offset is always from the MSB of the anonymous addressed object containing the field, so it can be disambiguated, but it is awkward, hence Dwarf v4's new DW_AT_data_bit_offset.


Also, don't rely on DW_AT_byte_size being there. It is optional and you then have to use the size of the container type.

David




Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog



From Anitha.Boyapati@atmel.com  Thu Sep  8 06:56:48 2011
From: Anitha.Boyapati@atmel.com (Boyapati, Anitha)
Date: Thu, 8 Sep 2011 14:56:48 +0800
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for
	DW_AT_bit_offsetfields
In-Reply-To: <FC0D83C633273A488E45F10E6CBCB46130EBBAEB@CAMEUREXM01.EUROPE.ROOT.PRI>
References: <4E65F74F.80406@charter.net>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>
	<FC0D83C633273A488E45F10E6CBCB46130EBBAEB@CAMEUREXM01.EUROPE.ROOT.PRI>
Message-ID: <C92FFA3D5F1BC641990154AC4B1C50BA0256F9FB@penmb01.corp.atmel.com>


Hello David,

Thanks for the explanation.

There is one more little-endian architecture - AVR on which I work. I get this output for 'l' of test. (gcc 4.5.1)

<2><5e>: Abbrev Number: 4 (DW_TAG_member)
    <5f>   DW_AT_name        : l
    <61>   DW_AT_decl_file   : 1
    <62>   DW_AT_decl_line   : 6
    <63>   DW_AT_type        : <0x75>
    <67>   DW_AT_byte_size   : 1
    <68>   DW_AT_bit_size    : 2
    <69>   DW_AT_bit_offset  : 0xffffffff 0xffffffff
    <71>   DW_AT_data_member_location: 2 byte block: 23 0       (DW_OP_plus_uconst: 0)


>From the output I think the layout is as follows:

39    32 31    24 23    16 15     8 7      0
+--------+--------+--------+--------+--------+
| addr+4 | addr+3 | addr+2 | addr+1 | addr+0 |
+--------+--------+--------+--------+--------+
 ..................................L lKkkkkBA

However it assumes a container type of char which 1 byte aligned on 0 byte (same as that of the structure) and uses -ve offset to indicate that MSB of l extends beyond the container type. Hence -1 for DW_AT_bit_offset.

As DWARF2/3 spec, does not say anything about DW_AT_bit_offset being unsigned, I am presuming that the above output is equally correct. Is it not?

Anitha

>-----Original Message-----
>From: David Earlam [mailto:David.Earlam at csr.com]
>Sent: Wednesday, September 07, 2011 9:06 PM
>To: Boyapati, Anitha; DWARF discussion mailing list
>Subject: RE: [Dwarf-Discuss] DWARF2 - Endianity issues for
>DW_AT_bit_offsetfields
>
>Anitha,
>
>I checked two other little-endian 32-bit, octet-addressing C compilers, one
>producing Dwarf2, one producing Dwarf3.
>
>They both agree with the gcc you used on little-endian x86.
>
><2><b3>: Abbrev Number: 37 (DW_TAG_member)
>   <b4>   DW_AT_name        : l
>   <b6>   DW_AT_type        : DW_FORM_ref1 <0x61> "unsigned char"
>   <b8>   DW_AT_data_member_location: 2 byte block: 23 1
>(DW_OP_plus_uconst: 1)
>   <bb>   DW_AT_byte_size   : 1
>   <bc>   DW_AT_bit_size    : 2
>   <bd>   DW_AT_bit_offset  : 6
>
>
>When the bitfield 'l' would have straddled two storage units, the compiler
>chose to start a new storage unit (C structure packing is implementation-
>defined). Updates to 'l' then only need access one byte at addr+1 with a
>mask 0x03.
>
>If the bitfield straddled two bytes, generated code would usually need to
>access two bytes, shift the first left 1, shift the second right 7,
>bitwise-or together and maybe mask with 0x03 to form the field into a
>unsigned char value (or a similar sequence leastwise on a processor without
>deposit/extract instructions).
>
>So with that debug_info, the struct test surely looks like this:
>
> 39    32 31    24 23    16 15     8 7      0
> +--------+--------+--------+--------+--------+
> | addr+4 | addr+3 | addr+2 | addr+1 | addr+0 |
> +--------+--------+--------+--------+--------+
>  .................................Ll pKkkkkBA
>
>where p is padding.
>
>That might not be the case for __packed structures, if the compiler has
>this non-standard extension, which avoid such padding and pack as closely
>as possible.
>
>According to Dwarf 2.0.0 section 5.5.4 Structure Data Member Entries,
>the DW_AT_data_member_location is the address relative to the composite
>object that most closely encloses the bitfield; so if 'l' straddled that
>relative address could be DW_OP_plus_uconst 0 or DW_OP_plus_uconst 1, but
>the DW_AT_bit_offset is always from the MSB of the anonymous addressed
>object containing the field, so it can be disambiguated, but it is awkward,
>hence Dwarf v4's new DW_AT_data_bit_offset.
>
>
>Also, don't rely on DW_AT_byte_size being there. It is optional and you
>then have to use the size of the container type.
>
>David
>
>
>
>
>Member of the CSR plc group of companies. CSR plc registered in England and
>Wales, registered number 4187346, registered office Churchill House,
>Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
>More information can be found at www.csr.com. Follow CSR on Twitter at
>http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog



From David.Earlam@csr.com  Thu Sep  8 10:27:35 2011
From: David.Earlam@csr.com (David Earlam)
Date: Thu, 8 Sep 2011 10:27:35 +0000
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for
 DW_AT_bit_offsetfields
In-Reply-To: <C92FFA3D5F1BC641990154AC4B1C50BA0256F9FB@penmb01.corp.atmel.com>
References: <4E65F74F.80406@charter.net>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>
	<FC0D83C633273A488E45F10E6CBCB46130EBBAEB@CAMEUREXM01.EUROPE.ROOT.PRI>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F9FB@penmb01.corp.atmel.com>
Message-ID: <FC0D83C633273A488E45F10E6CBCB46130EBBB34@CAMEUREXM01.EUROPE.ROOT.PRI>

> Hello David,
> 
> Thanks for the explanation.
> 
> There is one more little-endian architecture - AVR on which I work. I get
> this output for 'l' of test. (gcc 4.5.1)
> 
> <2><5e>: Abbrev Number: 4 (DW_TAG_member)
>     <5f>   DW_AT_name        : l
>     <61>   DW_AT_decl_file   : 1
>     <62>   DW_AT_decl_line   : 6
>     <63>   DW_AT_type        : <0x75>
>     <67>   DW_AT_byte_size   : 1
>     <68>   DW_AT_bit_size    : 2
>     <69>   DW_AT_bit_offset  : 0xffffffff 0xffffffff
>     <71>   DW_AT_data_member_location: 2 byte block: 23 0
> (DW_OP_plus_uconst: 0)
> 
Hi Anitha,

Yes, it's valid.

If I use gcc 4.4.4 (x86 mingw tdm) with '-g -fpack-struct' arguments

firstly, the compiler reports
  dwarfbits.c:6: note: Offset of packed bit-field 'l' has changed in GCC 4.4
  - since an earlier gcc version I guess -

then secondly, 'objdump -W' of the produced Dwarf2 also says 

<2><80>: Abbrev Number: 4 (DW_TAG_member)
    <81>   DW_AT_name        : l	
    <83>   DW_AT_decl_file   : 1	
    <84>   DW_AT_decl_line   : 5	
    <85>   DW_AT_type        : <0x93>	
    <89>   DW_AT_byte_size   : 1	
    <8a>   DW_AT_bit_size    : 2	
    <8b>   DW_AT_bit_offset  : 0xffffffff	
    <8f>   DW_AT_data_member_location: 2 byte block: 23 0 	(DW_OP_plus_uconst: 0)

The debug_abbrev for DW_AT_bit_offset is DW_FORM_data4, I assume it is DW_FORM_data8 for you. (I didn't actually know AVR's could address that much memory, or just maybe there's a problem around here in GCC4.5.3 or the AVR port.)

It'd be easier to understand if the compiler were to use DW_FORM_sdata for 'signed constant' class of DW_AT_bit_offset then it is clearly signed for the DWARF consumer.

However if you interpret that DW_AT_bit_offset value as signed the 2 bit-field 'l' straddles addr+1 and addr+0. It all makes sense.

In Dwarf v4 the new DW_AT_data_bit_offset (alternative to Dwarf2 or Dwarf3 style DW_AT_data_member_location +or- DW_AT_bit_offset)
"must be greater than or equal to zero" so it is unsigned.

David 






Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog



From gerald.rigdon@bsci.com  Wed Sep 14 21:19:14 2011
From: gerald.rigdon@bsci.com (Rigdon, Gerald STP)
Date: Wed, 14 Sep 2011 16:19:14 -0500
Subject: [Dwarf-Discuss] DWARF2 Make Error
Message-ID: <3CD72B6D0BC0A14FAB6F6C3D81E42EE8257D44C0B9@STPEXCMSP08.bsci.bossci.com>



________________________________
From: Rigdon, Gerald (STP)
Sent: Wednesday, September 14, 2011 4:09 PM
To: 'dwarf-discuss at lists.dwarfstd.org'
Cc: Sharma, Shailesh
Subject: DWARF2 Make Error


I followed the instructions from the Readme file:

"To build dwarfdump2, first build libdwarf in the neighboring

directory then type

./configure

make"





Results Below:

./configure appeared to work but when I ran "make", I got the following errors:


g++ -g -O2 -I. -I. -I./../libdwarf -DCONFPREFIX=/usr/local/lib -c ./tag_common.cc
g++ -g -O2 -I. -I. -I./../libdwarf -DCONFPREFIX=/usr/local/lib   -c -o naming.onaming.cc
g++ -g -O2 -I. -I. -I./../libdwarf -DCONFPREFIX=/usr/local/lib -c ./common.cc
g++ -g -O2 -I. -I. -I./../libdwarf -DCONFPREFIX=/usr/local/lib ./tag_tree.cc naming.o tag_common.o common.o -L../libdwarf -ldwarf -lelf -o tag_tree_build
/usr/bin/ld: cannot find -ldwarf
collect2: ld returned 1 exit status
make: *** [tag_tree_build] Error 1

Can you help?

Thx

Gerry





Gerald T. Rigdon
Fellow, Firmware Design Engineering
Boston Scientific Corporation
4100 Hamline Avenue North
St. Paul, MN 55112-5798 USA
Tel: 651.582.7106
gerald.rigdon at bsci.com
www.bostonscientific.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/private.cgi/dwarf-discuss-dwarfstd.org/attachments/20110914/f09ca930/attachment.htm>


From davea42@earthlink.net  Thu Sep 15 00:07:33 2011
From: davea42@earthlink.net (David Anderson)
Date: Wed, 14 Sep 2011 17:07:33 -0700
Subject: [Dwarf-Discuss] DWARF2 Make Error
In-Reply-To: <3CD72B6D0BC0A14FAB6F6C3D81E42EE8257D44C0B9@STPEXCMSP08.bsci.bossci.com>
References: <3CD72B6D0BC0A14FAB6F6C3D81E42EE8257D44C0B9@STPEXCMSP08.bsci.bossci.com>
Message-ID: <4E7141C5.3020409@earthlink.net>

On 09/14/2011 02:19 PM, Rigdon, Gerald (STP) wrote:
>
> *From:* Rigdon, Gerald (STP)
> *Sent:* Wednesday, September 14, 2011 4:09 PM
> *To:* 'dwarf-discuss at lists.dwarfstd.org'
> *Cc:* Sharma, Shailesh
> *Subject:* DWARF2 Make Error
>
> I followed the instructions from the Readme file:
>
> "To build dwarfdump2, first build libdwarf in the neighboring
>
> directory then type
>
> ./configure
>
> make"
>
It appears you neglected to do the same two steps (ie 'build libdwarf'
with ./configure and make) in the libdwarf directory first.
Simple as that.  Then redo the 'make' in the dwarfdump2 directory. It 
should suffice.

DavidA.

-- 
/* Haley */ (Haley's comment.)




From Anitha.Boyapati@atmel.com  Fri Sep 16 12:30:28 2011
From: Anitha.Boyapati@atmel.com (Boyapati, Anitha)
Date: Fri, 16 Sep 2011 20:30:28 +0800
Subject: [Dwarf-Discuss] DWARF2 - Endianity issues for
	DW_AT_bit_offsetfields
In-Reply-To: <FC0D83C633273A488E45F10E6CBCB46130EBBB34@CAMEUREXM01.EUROPE.ROOT.PRI>
References: <4E65F74F.80406@charter.net>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F540@penmb01.corp.atmel.com>
	<FC0D83C633273A488E45F10E6CBCB46130EBBAEB@CAMEUREXM01.EUROPE.ROOT.PRI>
	<C92FFA3D5F1BC641990154AC4B1C50BA0256F9FB@penmb01.corp.atmel.com>
	<FC0D83C633273A488E45F10E6CBCB46130EBBB34@CAMEUREXM01.EUROPE.ROOT.PRI>
Message-ID: <C92FFA3D5F1BC641990154AC4B1C50BA026B31A5@penmb01.corp.atmel.com>


(Sorry for the delay. I was traveling)



>>
>Hi Anitha,
>
>Yes, it's valid.
>
>If I use gcc 4.4.4 (x86 mingw tdm) with '-g -fpack-struct' arguments
>
>firstly, the compiler reports
>  dwarfbits.c:6: note: Offset of packed bit-field 'l' has changed in GCC
>4.4
>  - since an earlier gcc version I guess -
>
>then secondly, 'objdump -W' of the produced Dwarf2 also says
>
><2><80>: Abbrev Number: 4 (DW_TAG_member)
>    <81>   DW_AT_name        : l
>    <83>   DW_AT_decl_file   : 1
>    <84>   DW_AT_decl_line   : 5
>    <85>   DW_AT_type        : <0x93>
>    <89>   DW_AT_byte_size   : 1
>    <8a>   DW_AT_bit_size    : 2
>    <8b>   DW_AT_bit_offset  : 0xffffffff
>    <8f>   DW_AT_data_member_location: 2 byte block: 23 0
>	(DW_OP_plus_uconst: 0)
>
>The debug_abbrev for DW_AT_bit_offset is DW_FORM_data4, I assume it is
>DW_FORM_data8 for you. (I didn't actually know AVR's could address that
>much memory, or just maybe there's a problem around here in GCC4.5.3 or the
>AVR port.)
>

This is a bug with gcc. (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47946)

The information is not configured in AVR. So I guess whatever is the default behavior of GCC, it should be reflected.


>It'd be easier to understand if the compiler were to use DW_FORM_sdata for
>'signed constant' class of DW_AT_bit_offset then it is clearly signed for
>the DWARF consumer.
>
>However if you interpret that DW_AT_bit_offset value as signed the 2 bit-
>field 'l' straddles addr+1 and addr+0. It all makes sense.

Thanks!


Anitha



From ccoutant@google.com  Fri Sep 23 00:36:21 2011
From: ccoutant@google.com (Cary Coutant)
Date: Thu, 22 Sep 2011 17:36:21 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info Files
	("Fission")
In-Reply-To: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
Message-ID: <CAHACq4rg9cH8sdb_AV2VytPEZBu==6HO3Cak3=o7ODvNu_9GSg@mail.gmail.com>

[Reposting -- My google.com address wasn't subscribed to this list, so
my first attempt bounced.]


---------- Forwarded message ----------
From: Cary Coutant <ccoutant@google.com>
Date: Thu, Sep 22, 2011 at 5:21 PM
Subject: RFC: DWARF Extensions for Separate Debug Info Files ("Fission")
To: dwarf-discuss at lists.dwarfstd.org, gcc at gcc.gnu.org, gdb at sourceware.org
Cc: Doug Evans <dje at google.com>, Paul Pluzhnikov
<ppluzhnikov at google.com>, Sterling Augustine <saugustine at google.com>


At Google, we've found that the cost of linking applications with
debug info is much too high. A large C++ application that might be,
say, 200MB without debug info, is somewhere around 1GB with debug
info, and the total size of the object files that we send to the
linker is around 5GB (and that's with compressed debug sections).
We've come to the conclusion that the most promising solution is to
eliminate the debug info from the link step. I've had direct
experience with HP's approach to this, and I've looked into Sun's and
Apple's approaches, but none of those three approaches actually
separates the debug info from the non-debug info at the object file
(.o) level. I know we're not alone in having concerns about the size
of debug info, so we've developed the following proposal to extend the
DWARF format and produce separate .o and ".dwo" (DWARF object) files
at the compilation step. Our plan is to develop the gcc and gdb
changes on new upstream branches.

After we get the basics working and have some results to show
(assuming it all works out and proves worthwhile), I'll develop this
into a formal proposal to the DWARF committee.

I've also posted this proposal on the GCC wiki:

? http://gcc.gnu.org/wiki/DebugFission

We've named the project "Fission."

I'd appreciate any comments.

-cary


DWARF Extensions for Separate Debug Information Files

September 22, 2011


Problems with Size of the Debug Information
===========================================

Large applications compiled with debug information experience
slow link times, possible out-of-memory conditions at link time,
and slow gdb startup times. In addition, they can contribute to
significant increases in storage requirements, and additional
network latency when transferring files in a distributed build
environment.

* Out-of-memory conditions: When the total size of the input
?files is large, the linker may exceed its total memory
?allocation during the link and may get killed by the operating
?system. As a rule of thumb, the link job total memory
?requirements can be estimated at about 200% of the total size
?of its input files.

* Slow link times: Link times can be frustrating when recompiling
?only a small source file or two. Link times may be aggravated
?when linking on a machine that has insufficient RAM, resulting
?in excessive page thrashing.

* Slow gdb startup times: The debugger today performs a partial
?scan of the debug information in order to build its internal
?tables that allow it to map names and addresses to the debug
?information. This partial scan was designed to improve startup
?performance, and avoids a full scan of the debug information,
?but for large applications, it can still take a minute or more
?before the debugger is ready for the first command. The
?debugger now has the ability to save a ".gdb_index" section in
?the executable and the gold linker now supports a --gdb-index
?option to build this index at link time, but both of these
?options still require the initial partial scan of the debug
?information.

These conditions are largely a direct result of the amount of
debug information generated by the compiler. In a large C++
application compiled with -O2 and -g, the debug information
accounts for 87% of the total size of the object files sent as
inputs to the link step, and 84% of the total size of the output
binary.

Recently, the -Wa,--compress-debug-sections option has been made
available. This option reduces the total size of the object files
sent to the linker by more than a third, so that the debug
information now accounts for 70-80% of the total size of the
object files. The output file is unaffected: the linker
decompresses the debug information in order to link it, and
outputs the uncompressed result (there is an option to recompress
the debug information at link time, but this step would only
reduce the size of the output file without improving link time or
memory usage).


What's All That Space Being Used For?
=====================================

The debugging information in the relocatable object files sent to
the linker consists of a number of separate tables (percentages
are for uncompressed debug information relative to the total
object file size):

* Debug Information Entries - .debug_info (11%): This table
?contains the debug info for subprograms and variables defined
?in the program, and many of the trivial types used.

* Type Units - .debug_types (12%): This table contains the debug
?info for most of the non-trivial types (e.g., structs and
?classes, enums, typedefs), keyed by a hashed type signature so
?that duplicate type definitions can be eliminated by the
?linker. During the link, about 85% of this data is discarded as
?duplicate. These sections have the same structure as the
?.debug_info sections.

* Strings - .debug_str (25%): This table contains strings that
?are not placed inline in the .debug_info and .debug_types
?sections. The linker merges the string tables to eliminate
?duplicates, discarding about 93% of the data as duplicate.

* Range tables - .debug_ranges (2%) and .debug_aranges (0.1%):
?These tables contain range lists to define what pieces of a
?program's text belong to which subprograms and compilation
?units.

* Location lists - .debug_loc (2%): These tables contain lists of
?expressions that describe to the debugger the location of a
?variable based on the PC value.

* Line number tables - .debug_line (1%): These tables contain a
?description of the mapping from PC values to source locations.

* Debug abbreviation codes - .debug_abbrev (<1%): These tables
?provide the definitions for abbreviation codes used in
?describing the debug info in the .debug_info and .debug_types
?sections.

* Public names - .debug_pubnames (<1%): These tables provide a
?list of public names defined in the compilation unit, intended
?to allow the debugger to find the appropriate compilation units
?quickly for a given name. (In practice, these tables are unused
?by gdb.)

* Relocations for debug information (46%): The relocations
?identify to the linker where all the relocatable references are
?in the debug information. Of the 46%, about 20 percentage
?points are for the .debug_info section and about 17 are for the
?.debug_types section. Nine of ten of these relocations are for
?references to the .debug_str section; the remaining tenth are
?mostly references to locations in the program. Another 9
?percentage points are for the .debug_ranges and .debug_loc
?sections; these are entirely references to locations in the
?program. These relocations are used by the linker and are not
?copied to the output file.

Using compressed debug sections, the percentages are adjusted as
follows:

* Debug Information Entries (4%)
* Type Units (7%)
* Strings (10%)
* Range tables (<<1%) (These compress to almost nothing, because
?most of the information is in the relocations.)
* Location lists (1%)
* Line number tables (<1%)
* Debug abbreviation codes (<1%)
* Public names (<1%)
* Relocations (60%) (Relocations are not compressed.)


Towards Reducing the Amount of Debug Information Sent to the Linker
===================================================================

The numbers above suggest several possible approaches to
improving build performance by reducing the amount of debug
information sent to the linker. Of the approaches listed below,
the first has already been implemented, and we are planning to
proceed with options #3 and #5.

1. Compress debug sections

This option is already in the binutils assembler.

Total estimated benefit: 36% reduction

All estimated benefits below assume the use of compressed debug
sections.

2. Intermediate links

We can use intermediate (ld -r) link steps to discard a good
fraction of the duplicate type information and strings in the
debug information. COMDAT elimination on the .debug_types section
would ultimately reduce the total size by 75%, and string merge
processing would ultimately reduce the total size of the
.debug_str section by 80%.

By using ld -r on the input files, there would be some risk of
including object files that would have been passed over during an
archive library search, but this could be mitigated by removing
duplicate definitions of the same symbol in different libraries.

Total estimated benefit: 6% reduction

3. Eliminate relocations to strings

Relocations in the .debug_info and .debug_types sections that
refer to the .debug_str section can be replaced by an extra
indirection and a new dedicated table of string offsets. In the
debug info, each string reference takes 8 bytes in the debug info
plus 24 bytes for the relocation (the 8 bytes compress to 1 byte
on average, but the relocations are not compressed). These would
be replaced by an average 1-2 byte string index, and an 8-byte
string offset in a separate table. The separate table can be
implicitly relocated, so no relocations are necessary, and would
probably compress by about 80% (estimated). Furthermore, the
number of unique strings (within each compilation unit) is only
about 55% of the total number of string references, so the
separate table would be reduced by another 45%.

Total estimated benefit: 53% reduction

4. Move type units to a separate repository

The information in .debug_types is largely independent of the
rest of the compilation unit; each entry describes a type and is
the same in each compilation unit that contains a definition of
that type. The compiler computes a unique signature for each type
definition, and could store the debug info for that type in an
external repository, keyed by the type's signature. This would
reduce the object file size by the 7% used directly for the
.debug_types sections, by another estimated 5% for the strings
referenced by the type entries, and by 22% for the relocations
associated with the .debug_types sections.

Total estimated benefit: 34% reduction (less if option 3 is also
implemented)

5. Move debug info and type units to a ".dwo" file

Alternatively, we can move the .debug_info, .debug_types, and
.debug_str sections from the object (.o) files to a separate
DWARF object (.dwo) file (or ".dsym", in Apple's nomenclature).
Assuming that we could ignore these .dwo files during the link
step, this would remove the bulk of the data that would be sent
to the linker, and debug builds would be be much closer to
non-debug builds in terms of object file size and link speed.

There are two options for how to deal with the separate .dwo
files when it's time to debug a program.

The first option is for the debugger to look for the debug info
directly in the .dwo files, requiring both that the output binary
contains enough information to find the .dwo files, and that the
.dwo files must remain available for use for as long as the
binary might need debugging. This option requires no additional
link step for the debug information.

The second option is to invoke a separate link step to combine
the .dwo files into a single .dwo file that can be easily stored
and located by the debugger (this is the approach that Apple
takes with its dsymutil tool). While this option dramatically
reduces the size of the main link, the separate link step for the
debug information is still close to the original order of
magnitude (debug info being more than 80% of the total size of
the object files). A dsymutil-like linker, however, could be made
to operate much more efficiently than a full-featured ELF linker.

Total estimated benefit: 70% reduction


Previous Implementations of Separate Debug Information
======================================================

Sun, HP, and Apple have all implemented similar mechanisms where
the debug information is not linked into the final binary. All
three implementations simply leave the debug information in the
relocatable object (.o) files, with summary information in the
final binary to enable the debugger to locate the object files
and to apply the relocations to the debug information. Sun's
implementation is for stabs only; for DWARF, the linker always
copies the full debug information to the output file. HP's
implementation is for DWARF, and includes summary information
with the names of the original object files, and a link map that
allows the debugger to locate each object file's contribution to
each section in the output file. Apple's solution is similar, but
the summary information is synthesized by the linker in stabs
format, and the DWARF information is linked together in a
separate link step by the dsymutil utility.

In the Sun and HP implementations, the debug information in the
relocatable objects still requires relocation at debug time, and
the debugger must read the summary information from the
executable file in order to map symbols and sections to the
output file when processing and applying the relocations. The
Apple implementation avoids this cost at debug time, but at the
cost of having a separate link step for the debug information.

As we have seen above, a significant factor in the space used by
debug information is the number of relocations, so a solution
that minimizes the number of relocations not only reduces the
total size of the binary plus its debug information, but also
reduces the complexity and cost of reading the debug information
at debug time.


Design for Moving Debug Information to ".dwo" Files
===================================================

In order for the debugger to be able to locate and process the
information in raw (unrelocated) .dwo files, some information
must still be left behind in the .o files for the linker to
combine and relocate. The bulk of the reduction in .o file size
will come from moving .debug_info, .debug_types, and .debug_str
sections into the .dwo file (and eliminating most of the
associated relocations). Therefore, to simplify the initial
implementation, the .debug_ranges, .debug_loc, .debug_line,
.debug_pubnames, .debug_aranges, and .debug_gdb_scripts sections
(and their relocation sections) will remain in the .o file. The
.debug_abbrev section, although tiny, will naturally move with
the .debug_info and .debug_types sections.

The .dwo file will follow the ELF format, but will have only a
file header, section table, and the debug sections. The debug
sections in the .dwo file will all be named with ".dwo" instead
of ".debug".

For the sections that will move into a .dwo file, the existing
relocations can be handled as follows:

* Each compilation unit header in .debug_info and .debug_types
?normally has a relocation to the corresponding abbrev table.
?This relocation is no longer necessary, and the field in the
?compilation unit header should be interpreted as a direct
?offset relative to the .dwo_abbrev section in the .dwo file.

* The DW_TAG_compile_unit DIE at the top level of the .debug_info
?section contains a relocated reference to a line table in the
?.debug_line section (for the DW_AT_stmt_list attribute). To
?handle this case, the .o file will contain a skeleton
?.debug_info section (and a small associated .debug_abbrev
?section), with a single DW_TAG_compile_unit DIE. The
?DW_TAG_compile_unit DIE will contain the DW_AT_stmt_list
?attribute, and will have no children. The bulk of the debug
?information will instead be placed in the .dwo file in the
?.dwo_info section.

* Likewise, the DW_TAG_type_unit DIE at the top level of the
?.debug_types section normally contains a DW_AT_stmt_list
?attribute with a relocated reference to a line table. In this
?case, the line table serves only to provide a list of directory
?and file names that are indexed by DW_AT_file attributes. For
?this purpose, the .dwo file will contain a skeleton .dwo_line
?section whose only purpose is to list the directories and file
?names needed by DW_AT_file attributes in the type definition.
?Its format will be the same as the .debug_line section, but
?without the line tables. The value of the DW_AT_stmt_list
?attribute in a .dwo_types section will be the unrelocated
?offset to the skeleton line table.

* References to strings in the .dwo_str section will be replaced
?with an indirect string index (using a new FORM code,
?DW_FORM_str_index), expressed as an unsigned LEB128 number. A
?new section in the .dwo file, .dwo_str_offsets, will contain a
?list of offsets to the strings in the string table, and the
?indirect string index will select one of the offsets in that
?list. The entries in .dwo_str_offsets will not need
?relocations.

* References to ranges in the .debug_ranges section, and to
?location lists in the .debug_loc section, from DWARF attributes
?using the form DW_FORM_sec_offset, will be replaced with an
?indirect symbol index (using a new FORM code,
?DW_FORM_ref_index), expressed as an unsigned LEB128 number. A
?new section in the .o file, .debug_ref, will contain a list of
?relocated addresses, one for each reference needed. Each entry
?in the .debug_ref section will be a 4- or 8-byte location
?(depending on the format of the DWARF information), with the
?relocation that applied to the original reference. The linker
?will combine the .debug_ref sections from all the input .o
?files, and will apply the relocations normally. In order for
?the debugger to find the contribution to the .debug_ref section
?corresponding to a particular compilation unit, the skeleton
?DW_TAG_compile_unit entry in the .o file's .debug_info section
?will contain a DW_AT_dwo_name attribute to identify the .dwo
?file, and a DW_AT_ref_base attribute whose value points to the
?base of that compilation unit's .debug_ref contribution.

* Similarly, references to addresses in loadable sections (e.g.,
?.text or .data), from DWARF attributes using the form
?DW_FORM_addr, will be replaced with an indirect symbol index
?(using a new FORM code, DW_FORM_addr_index), expressed as an
?unsigned LEB128 number. A new section in the .o file,
?.debug_addr, will contain a list of relocated addresses, one
?for each reference needed. Each entry in the .debug_addr
?section will be an address-sized value, with the relocation
?that applied to the original reference. The linker will combine
?the .debug_addr sections from all the input .o files, and will
?apply the relocations normally. In order for the debugger to
?find the contribution to the .debug_ref section corresponding
?to a particular compilation unit, the skeleton
?DW_TAG_compile_unit entry in the .o file's .debug_info section
?will also contain a DW_AT_addr_base attribute whose value
?points to the base of that compilation unit's .debug_addr
?contribution.

In the initial implementation, we will modify gcc to emit all the
debug information into the single .o file, and we will use
post-compile processing to move the appropriate sections into the
separate .dwo file.

The debug sections to remain in the .o file are:

* .debug_abbrev - Defines the abbreviation codes used by the
?skeleton .debug_info section.

* .debug_info - Contains the skeleton DW_TAG_compile_unit DIE.
?This DIE has the following attributes: DW_AT_comp_dir,
?DW_AT_dwo_name, DW_AT_dwo_id, DW_AT_ref_base, and
?DW_AT_addr_base. (All string attributes will use
?DW_FORM_string, so no .debug_str section is necessary in the .o
?file.)

* .debug_ref - New section to hold references to other debug
?sections, indexed by attributes of DW_FORM_ref_index.

* .debug_addr - New section to hold references to loadable
?sections, indexed by attributes of form DW_FORM_addr_index.

* .debug_line - Line tables, unaffected by this design. (These
?could be moved to the .dwo file, but in order to do so, each
?DW_LNE_set_address opcode would need to be replaced by a new
?opcode that referenced an entry in the .debug_addr section.)

* .debug_loc - Location lists, unaffected by this design.

* .debug_ranges - Range lists, unaffected by this design.

* .debug_macinfo - Macro information, unaffected by this design.

* .debug_pubnames - Public names for use in building the
?.gdb_index section at link time. This section will have the
?same format and use as always, but we will fix gcc to emit all
?names that need to appear in the index.

* .debug_pubtypes - Public types for use in building the
?.gdb_index section at link time. This section will have an
?extended format to allow it to represent both types in the
?.debug_dwo_info section and type units in .debug_types.

* .debug_aranges - Range table for the compilation unit, for use
?in building the .gdb_index section at link time. It's likely,
?but unverified, that the contents of this section as produced
?by gcc today are sufficient for this use.

The following debug sections will be generated by gcc and the
assembler in the .o file, but will be moved to the separate .dwo
file in post-compile processing:

* .dwo_abbrev - Defines the abbreviation codes used by the
?.debug_dwo_info section.

* .dwo_info - Contains the complete debug information for the
?compilation unit. The top-level DW_TAG_compile_unit DIE will
?contain the standard attributes (with the possible addition of
?DW_AT_id to match the .dwo file with the corresponding .o
?file).

* .dwo_types - Contains the complete debug information for each
?type unit. (There may be multiple instances of this section.
?Unlike in a .o file, the .dwo_types sections will not be placed
?in COMDAT groups.)

* .dwo_str - Contains the strings referenced indirectly via
?DW_FORM_strp from the .dwo_info and .dwo_types sections.

* .dwo_str_offsets - Contains an array of offsets of strings in
?the .dwo_str section. Attributes using DW_FORM_str_index will
?reference an entry in this array.


Building a GDB Index
====================

In order to build the .gdb_index section, the linker today needs
to do three things:

1. ?Scan the .debug_info sections to extract all the public names
? ?mentioned in those sections and build a hash table mapping
? ?those names to a list of compilation units that provide
? ?definitions for those names.

2. ?Scan the compilation unit headers of the .debug_info sections
? ?to build a list of compilation units, and to build a range
? ?table that can be used to map and address to a specific
? ?compilation unit.

3. ?Scan the .debug_types sections to build a list of type units.

In this new design, the .debug_info and .debug_types sections
will not be available for the linker to scan. Instead, the linker
will extract the public names from the .debug_pubnames and
.debug_pubtypes sections, the range table from the .debug_aranges
sections, and the the list of type units from the .debug_pubtypes
sections.

The format of the .gdb_index section will remain unchanged (with
the possible exception of removing the range tables in favor of
having gdb use .debug_aranges directly).

The .debug_pubnames section was always intended as a
comprehensive list of symbols that gdb could use for quick
lookup, but bugs in gcc have so far prevented gdb from using this
section for its intended purpose, and the .gdb_index section has
instead been built from a scan over the .debug_info section. We
will need to fix gcc so that the .debug_pubnames section contains
all of the names that gdb requires in its index.

The .debug_pubtypes section, likewise, was intended as a list of
public types, but has not yet been extended to include type
defined in type units in the .debug_types sections. We will have
to extend the format to allow this, and modify gcc to produce the
required information.

The .debug_aranges section currently contains all the required
information for producing the .gdb_index section (as far as I
know), and could in fact be used directly by gdb instead of
having the linker reformat its contents into .gdb_index. We can
either modify the linker to extract the necessary range
information from .debug_aranges to produce .gdb_index, or modify
gdb to use the existing information from .debug_aranges instead
of expecting it in .gdb_index.


Proposed Extensions to the DWARF Specification
==============================================

We propose adding several new sections, attribute codes, and form
codes to the DWARF specification, and extending the
.debug_pubtypes format to support type units.

New DWARF sections

* .debug_ref -- This section contains relocated references to the
?.debug_ranges and .debug_loc sections. No header is required;
?the base of each compilation unit's contribution to the section
?is given by the DW_AT_ref_base attribute of the
?DW_TAG_compile_unit DIE in the .debug_info section. Each entry
?is 4 bytes for DWARF-32 compilation units, or 8 bytes for
?DWARF-64 compilation units, as determined by the unit_length
?field of the compilation unit header in the .debug_info
?section.

* .debug_addr -- This section contains relocated references to
?loadable sections in the program. No header is required; the
?base of each compilation unit's contribution to the section is
?given by the DW_AT_addr_base attribute of the
?DW_TAG_compile_unit DIE in the .debug_info section. Each entry
?is 4 or 8 bytes, depending on the address_size field of the
?compilation unit header in the .debug_info section.

* .dwo_abbrev -- This section contains the abbreviation table for
?the .dwo_info section. Its format is identical to the
?.debug_abbrev section.

* .dwo_info -- This section contains the bulk of the debug
?information for a compilation unit. Its format is identical to
?the .debug_info section. The DW_AT_dwo_name attribute of the
?DW_TAG_compile_unit DIE in the .debug_info section is used to
?locate the .dwo file, and the DW_AT_dwo_id attribute of the
?DW_TAG_compile_unit DIE in each section use used to verify a
?match.

* .dwo_types -- This section contains a type unit. Its format is
?identical to the .debug_types section. Each type unit is placed
?in a separate .dwo_types section; unlike .debug_types sections,
?however, .dwo_types sections are not placed in COMDAT groups.

* .dwo_str -- This section contains the DWARF string table. Its
?format is identical to the .debug_str section.

* .dwo_str_offsets -- This section contains a list of offsets to
?strings in the .dwo_str section. References to strings using
?the DW_FORM_str_index form code refer to an entry in this
?section. No header is required. The size of each entry is 4
?bytes for DWARF-32 compilation units, or 8 bytes for DWARF-64
?compilation units, as determined by the unit_length field of
?the compilation unit header in the .dwo_info section. Entries
?in this section are not relocated--each entry is an offset
?relative to the beginning of the .dwo_str section in the same
?.dwo file.

New DWARF attributes

* DW_AT_dwo_name -- A string-valued attribute that identifies the
?name of the corresponding .dwo file. This attribute is found in
?the DW_TAG_compile_unit DIE of the .debug_info section.

* DW_AT_dwo_id -- A block-valued attribute that provides a unique
?ID for the compilation unit. This attribute is found in the
?DW_TAG_compile_unit DIE in both .debug_info and .dwo_info
?sections, and can be used to verify a match between .o file (or
?linked binary) and .dwo file.

* DW_AT_ref_base -- A reference-valued attribute that refers to
?the beginning of the compilation unit's contribution to the
?.debug_ref section. This attribute is found in the
?DW_TAG_compile_unit DIE of the .debug_info section.

* DW_AT_addr_base -- A reference-valued attribute that refers to
?the beginning of the compilation unit's contribution to the
?.debug_addr section. This attribute is found in the
?DW_TAG_compile_unit DIE of the .debug_info section.

New DWARF form codes

* DW_FORM_ref_index -- An unsigned LEB128 value that refers to an
?entry in the .debug_ref section. This form belongs to the
?reference class, and may be used in a .dwo_info section for any
?attribute that would have otherwise used another
?reference-class form to refer to an entry in the .debug_ranges
?or .debug_loc section. The first entry in the compilation
?unit's contribution to the debug_ref section has an index of 0.

* DW_FORM_addr_index -- An unsigned LEB128 value that refers to
?an entry in the .debug_addr section. This form belongs to the
?address class, and may be used in a .dwo_info section for any
?attribute that would have otherwise used DW_FORM_addr to refer
?to an address in a loadable section of the linked binary. The
?first entry in the compilation unit's contribution to the
?debug_addr section has an index of 0.

* DW_FORM_str_index -- An unsigned LEB128 value that refers to an
?entry in the .dwo_str_offsets section. This form belongs to the
?string class, and may be used in a .dwo_info or .dwo_types
?section for any attribute that would have otherwise used
?DW_FORM_strp to refer to a string in the .debug_str section.
?The first entry in the .dwo_str_offsets section has an index of
?0.

Extension to .debug_pubtypes

The .debug_pubtypes table will be extended with a second table of
type units following the existing table of name entries. Each
entry in the table of type units will consist of two fields: an
eight-byte type signature, followed by the null-terminated name
of the type. The final entry of the table will be followed by an
entry where the signature is all zeroes, and the name is a single
null byte. The version number in the .debug_pubtypes header will
change to 5 to mark the presence of this second table.



From jmolenda@apple.com  Fri Sep 23 01:35:44 2011
From: jmolenda@apple.com (Jason Molenda)
Date: Thu, 22 Sep 2011 18:35:44 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
Message-ID: <B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>

Hi Cary, just one quick clarification -

On Sep 22, 2011, at 5:21 PM, Cary Coutant wrote:

> Previous Implementations of Separate Debug Information
> ======================================================
> 
> In the Sun and HP implementations, the debug information in the
> relocatable objects still requires relocation at debug time, and
> the debugger must read the summary information from the
> executable file in order to map symbols and sections to the
> output file when processing and applying the relocations. The
> Apple implementation avoids this cost at debug time, but at the
> cost of having a separate link step for the debug information.


The Apple approach has both the features of the Sun/HP implementation as well as the ability to create a standalone debug info file. 

The compiler puts DWARF in the .o file, the linker adds some records in the executable which help us to understand where files/function/symbols landed in the final executable[1].  If the user runs our gdb or lldb on one of these binaries, the debugger will read the DWARF directly out of the .o files on the fly.  Because the linker doesn't need to copy around/update/modify the DWARF, link times are very fast.  If the developer decides to debug the program, no extra steps are required - the debugger can be started up & used with the debug info still in the .o files.

Clearly this is only viable if you have the .o files on your computer so we added a command, "dsymutil", which links the DWARF from the .o files into a single standalone ".dSYM" file.  The executable file and the dSYM file have a shared 128-bit number to ensure that the debug info and the executable match; the debugger will ignore a dSYM with a non-matching UUID for a given executable.  A developer will typically create a dSYM when they sending a copy of the binary to someone and want to provide debug information, or they are archiving a released binary, or they want to debug it on another machine (where the .o files will not be in place.)

In practice people create dSYMs rarely -- when they are doing iterative development on their computer, all of the DWARF sits in the .o files unprocessed unless they launch a debugger, link times are fast.


As a minor detail, the dSYM is just another executable binary image on our system (Mach-O file format), sans any of the text or data of the real binary file, with only the debug_info, etc. sections.  The name "dSYM" was a little joke based on the CodeWarrior "xSYM" debug info format.

J



From ppluzhnikov@google.com  Fri Sep 23 02:20:24 2011
From: ppluzhnikov@google.com (Paul Pluzhnikov)
Date: Thu, 22 Sep 2011 19:20:24 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
	<B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>
Message-ID: <CALoOobN1fniThrVctRFzA_oLPPpO7DGygX-OrPPAeH2K30ushA@mail.gmail.com>

On Thu, Sep 22, 2011 at 6:35 PM, Jason Molenda <jmolenda at apple.com> wrote:

> Because the linker doesn't need to copy around/update/modify the DWARF,
> link times are very fast.

AFAIU, the link times are fast only if all the files are local to the
developers' machine.

They will not be fast (and the .o files *will* need to be copied) if a
distributed compilation system (a build farm) is used (as is the case here).

Thanks,
-- 
Paul Pluzhnikov



From jan.kratochvil@redhat.com  Fri Sep 23 13:51:09 2011
From: jan.kratochvil@redhat.com (Jan Kratochvil)
Date: Fri, 23 Sep 2011 15:51:09 +0200
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
Message-ID: <20110923135109.GA22315@host1.jankratochvil.net>

On Fri, 23 Sep 2011 02:21:44 +0200, Cary Coutant wrote:
> * .debug_pubtypes - Public types for use in building the
>   .gdb_index section at link time. This section will have an
>   extended format to allow it to represent both types in the
>   .debug_dwo_info section and type units in .debug_types.
    ^^^^^^^^^^^^^^^
    = .dwo_info , maybe both .debug_info and .dwo_info


> * .dwo_abbrev - Defines the abbreviation codes used by the
>   .debug_dwo_info section.
    ^^^^^^^^^^^^^^^
    = .dwo_info


I find this .dwo_* setup is great for rapid development rebuilds but it should
remain optional as the currently used DWARF final separate .debug info file is
smaller than all the .dwo files together.  In the case of the final linked
.debug builds (rpm/deb/...) one does not consider the build speed as important.
It probably does not make sense to merge + convert .dwo files back to a single
.debug file for the rpm/deb/... build performance reasons.


Thanks,
Jan



From ccoutant@google.com  Fri Sep 23 17:50:05 2011
From: ccoutant@google.com (Cary Coutant)
Date: Fri, 23 Sep 2011 10:50:05 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <20110923135109.GA22315@host1.jankratochvil.net>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
	<20110923135109.GA22315@host1.jankratochvil.net>
Message-ID: <CAHACq4q=KWQdh2ErgDMSMf-PKezdLD3kFcAkNdgYCMEADqncug@mail.gmail.com>

>> * .debug_pubtypes - Public types for use in building the
>> ? .gdb_index section at link time. This section will have an
>> ? extended format to allow it to represent both types in the
>> ? .debug_dwo_info section and type units in .debug_types.
> ? ?^^^^^^^^^^^^^^^
> ? ?= .dwo_info , maybe both .debug_info and .dwo_info
>
>
>> * .dwo_abbrev - Defines the abbreviation codes used by the
>> ? .debug_dwo_info section.
> ? ?^^^^^^^^^^^^^^^
> ? ?= .dwo_info

Thanks, I've fixed the wiki page.

> I find this .dwo_* setup is great for rapid development rebuilds but it should
> remain optional as the currently used DWARF final separate .debug info file is
> smaller than all the .dwo files together. ?In the case of the final linked
> .debug builds (rpm/deb/...) one does not consider the build speed as important.
> It probably does not make sense to merge + convert .dwo files back to a single
> .debug file for the rpm/deb/... build performance reasons.

Yes, we'll definitely make this a compile-time option.

While I haven't finished designing the package format for collecting
all the .dwo files, I do plan on having the packaging tool do at least
duplicate type elimination to reduce the size of the package file.

-cary



From ccoutant@google.com  Fri Sep 23 17:58:34 2011
From: ccoutant@google.com (Cary Coutant)
Date: Fri, 23 Sep 2011 10:58:34 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
	<B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>
Message-ID: <CAHACq4p5-EscmeTZ0giD+F2M0T7=S77FTpq8=a-4aoOVc9FX_A@mail.gmail.com>

> The Apple approach has both the features of the Sun/HP implementation as well as the ability to create a standalone debug info file.

Thanks for the clarifications. I based my comments on a description
you sent me a couple of years ago, and I apologize for any
oversimplifications I introduced.

> The compiler puts DWARF in the .o file, the linker adds some records in the executable which help us to understand where files/function/symbols landed in the final executable[1].

Did you intend to add a footnote?

> ?If the user runs our gdb or lldb on one of these binaries, the debugger will read the DWARF directly out of the .o files on the fly. ?Because the linker doesn't need to copy around/update/modify the DWARF, link times are very fast. ?If the developer decides to debug the program, no extra steps are required - the debugger can be started up & used with the debug info still in the .o files.

We're trying to achieve something very similar, but we have the
additional goal of separating the info from the .o files because of
our distributed build environment. I also wanted to attempt to
standardize the approach, instead of having each vendor go in separate
directions.

Thanks,

-cary



From jmolenda@apple.com  Fri Sep 23 21:01:44 2011
From: jmolenda@apple.com (Jason Molenda)
Date: Fri, 23 Sep 2011 14:01:44 -0700
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
	Files ("Fission")
In-Reply-To: <CAHACq4p5-EscmeTZ0giD+F2M0T7=S77FTpq8=a-4aoOVc9FX_A@mail.gmail.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>
	<B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>
	<CAHACq4p5-EscmeTZ0giD+F2M0T7=S77FTpq8=a-4aoOVc9FX_A@mail.gmail.com>
Message-ID: <E4D200C3-534F-4562-9813-E765975E8692@apple.com>


On Sep 23, 2011, at 10:58 AM, Cary Coutant wrote:

>> The compiler puts DWARF in the .o file, the linker adds some records in the executable which help us to understand where files/function/symbols landed in the final executable[1].
> 
> Did you intend to add a footnote?

Yeah, I realized after I sent the email - it didn't seem interesting enough to warrant a separate followup.

The records that our linker puts in the executable are in the form of stabs entries.  There are a handful of stabs records created - file start, file end, function start, function end, symbol, pointer to a .o file, maybe one or two others.  We chose that format because it was trivial to support and we already had tools for stripping these records out of the executable once the dSYM had been created.

Once a dSYM has been created with all of the DWARF collected in a single file, our DWARF is parseable by any debug info consumer with minimal changes -- they need to know to look in a separate file for the DWARF from the main executable, but the format itself is unchanged.  Supporting the debug-information-in-.o-files is more involved, I don't know if any of the third-party debuggers on our platform work with it.


> We're trying to achieve something very similar, but we have the
> additional goal of separating the info from the .o files because of
> our distributed build environment. I also wanted to attempt to
> standardize the approach, instead of having each vendor go in separate
> directions.


Yeah, if your regular build environment involves distributed compilation, and the .o files need to be copied to a central system for the linker, then I can see why you're pursuing this approach.  For us, the most common usage is single-computer compilation & linking -- where the linker never pages in the debug info sections from the .o files so their size is not particular important.

J



From John.DelSignore@roguewave.com  Fri Sep 23 21:35:01 2011
From: John.DelSignore@roguewave.com (John DelSignore)
Date: Fri, 23 Sep 2011 17:35:01 -0400
Subject: [Dwarf-Discuss] RFC: DWARF Extensions for Separate Debug Info
 Files ("Fission")
In-Reply-To: <E4D200C3-534F-4562-9813-E765975E8692@apple.com>
References: <CAHACq4o43uGDEEKQRiYgnWWXSnDzRD1Z_-4z2HQ=bSaTpgyrjQ@mail.gmail.com>	<B6D4E49C-77EC-4BE7-8EAA-8897A4682F9F@apple.com>	<CAHACq4p5-EscmeTZ0giD+F2M0T7=S77FTpq8=a-4aoOVc9FX_A@mail.gmail.com>
	<E4D200C3-534F-4562-9813-E765975E8692@apple.com>
Message-ID: <4E7CFB85.7000800@roguewave.com>

Hi Jason,

Jason Molenda wrote:
> On Sep 23, 2011, at 10:58 AM, Cary Coutant wrote:
> 
>>> The compiler puts DWARF in the .o file, the linker adds some records in the executable which help us to understand where files/function/symbols landed in the final executable[1].
>> Did you intend to add a footnote?
> 
> Yeah, I realized after I sent the email - it didn't seem interesting enough to warrant a separate followup.
> 
> The records that our linker puts in the executable are in the form of stabs entries.  There are a handful of stabs records created - file start, file end, function start, function end, symbol, pointer to a .o file, maybe one or two others.  We chose that format because it was trivial to support and we already had tools for stripping these records out of the executable once the dSYM had been created.

I don't remember the exact details, but the problem I recall with the Darwin scheme is that it builds an incomplete index in the Mach-O symbol table. IIRC, it was missing things that a user might want to lookup by-name in the debugger, like static functions or variables, and type names with external linkage. Without a reasonably complete index, the debugger can't know where to find the definitions of certain things, and that forces the user to navigate using other information, like source file name or global function definitions to force the debug information in the object to be read.

Of course, the current DWARF indexes (like pubnames/pubtypes) have the same problem, and some compilers do a really bad job at generating those sections. But at least when there's a single .debug_info section, the debugger can decide to ignore the indexes and "skim" the full debug information. The compilers on IRIX did a better job at generating indexes, so the debugger could find by-name static functions/objects.

> Once a dSYM has been created with all of the DWARF collected in a single file, our DWARF is parseable by any debug info consumer with minimal changes -- they need to know to look in a separate file for the DWARF from the main executable, but the format itself is unchanged.  Supporting the debug-information-in-.o-files is more involved, I don't know if any of the third-party debuggers on our platform work with it.

TotalView supports debug information in .o files on Darwin, and has since day one. Perhaps you recall all those email exchanges you and I had several years back. It was a modest amount of work, given that we already supported debug information in .o files on the Sun and HP platforms.

I seem to recall one of the sore spots for us on Dawrin was getting good address information for certain DWARF location operations, like DW_OP_addr. Fortran was a particularly messy because some compilers didn't supply a linkage name attribute, so the debugger had to make several guesses at the name, and look things up by trial and error.

Cheers, John D.

>> We're trying to achieve something very similar, but we have the
>> additional goal of separating the info from the .o files because of
>> our distributed build environment. I also wanted to attempt to
>> standardize the approach, instead of having each vendor go in separate
>> directions.
> 
> 
> Yeah, if your regular build environment involves distributed compilation, and the .o files need to be copied to a central system for the linker, then I can see why you're pursuing this approach.  For us, the most common usage is single-computer compilation & linking -- where the linker never pages in the debug info sections from the .o files so their size is not particular important.
> 
> J
> _______________________________________________
> Dwarf-Discuss mailing list
> Dwarf-Discuss at lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
> 



