From louzonr@yahoo.com  Fri Jul  8 18:59:06 2011
From: louzonr@yahoo.com (Ron Louzon)
Date: Fri, 8 Jul 2011 11:59:06 -0700 (PDT)
Subject: [Dwarf-Discuss] libdwarf - Working with 64 bit images
Message-ID: <1310151546.78254.YahooMailRC@web114705.mail.gq1.yahoo.com>

I have been using libelf/libdwarf to parse 32-bit debug images.? I now need to 
accomodate 64-bit image files.? I enabled some libelf defines to allow the 
reading of those 64-bit image files.? After examining libdwarf, it appears to me 
that the 2 halves of 64-bit address data should be available in the Dwarf_loc 
structures as number1 and number 2.? Is there anything else I need to know as I 
attempt to parse these files?? Thanks.



From davea42@earthlink.net  Fri Jul  8 20:31:08 2011
From: davea42@earthlink.net (David Anderson)
Date: Fri, 08 Jul 2011 13:31:08 -0700
Subject: [Dwarf-Discuss] libdwarf - Working with 64 bit images
In-Reply-To: <1310151546.78254.YahooMailRC@web114705.mail.gq1.yahoo.com>
References: <1310151546.78254.YahooMailRC@web114705.mail.gq1.yahoo.com>
Message-ID: <4E17690C.5010201@earthlink.net>

On 07/08/2011 11:59 AM, Ron Louzon wrote:
> I have been using libelf/libdwarf to parse 32-bit debug images.  I now need to
> accomodate 64-bit image files.  I enabled some libelf defines to allow the
> reading of those 64-bit image files.  After examining libdwarf, it appears to me
> that the 2 halves of 64-bit address data should be available in the Dwarf_loc
> structures as number1 and number 2.  Is there anything else I need to know as I
> attempt to parse these files?  Thanks.

Unless you have done something special (why would you?) the types
for all  libdwarf fields are such that they all hold 64bit values.
So the address data (regardless if 32 or 64 bits) will be in individual 
fields.

libdwarf has always done everything in 64bit signed/unsigned at all times
(except the fields like the DWARF version and a few others that are defined
to be smaller than 64 bits).

So reading 64 bits requires no effort on your part other than getting the
libelf that has the 64bit interfaces (and indeed I would expect 64bit 
libelf functions to be
on any libelf one would be likely to find!).

DavidAnderson

-- 
The thunder was ominous sounding, much like the sound of a thin sheet of metal being shaken backstage during the storm scene in a play. -- Barbara Fetherolf




From louzonr@yahoo.com  Wed Jul 13 17:55:16 2011
From: louzonr@yahoo.com (Ron Louzon)
Date: Wed, 13 Jul 2011 10:55:16 -0700 (PDT)
Subject: [Dwarf-Discuss] libdwarf - Working with 64 bit images
In-Reply-To: <4E17690C.5010201@earthlink.net>
References: <1310151546.78254.YahooMailRC@web114705.mail.gq1.yahoo.com>
	<4E17690C.5010201@earthlink.net>
Message-ID: <1310579716.2412.YahooMailRC@web114711.mail.gq1.yahoo.com>

I have not modified the libDwarf code as it works great "as is".? I had looked 
at the dwarf_loc structure and I saw that the "number" fields were of type 
Dwarf_Unsigned.? I made the mistake of assuming they were equivalent to unsigned 
32 bit types but in actuality, they are 64 bit types.? It is great that you had 
the foresight to build this package to accomodate 64 bit data.? Thanks for the 
help.



----- Original Message ----
From: David Anderson <davea42@earthlink.net>
To: dwarf-discuss at lists.dwarfstd.org
Sent: Fri, July 8, 2011 4:31:08 PM
Subject: Re: [Dwarf-Discuss] libdwarf - Working with 64 bit images

On 07/08/2011 11:59 AM, Ron Louzon wrote:
> I have been using libelf/libdwarf to parse 32-bit debug images.? I now need to
> accomodate 64-bit image files.? I enabled some libelf defines to allow the
> reading of those 64-bit image files.? After examining libdwarf, it appears to 
>me
> that the 2 halves of 64-bit address data should be available in the Dwarf_loc
> structures as number1 and number 2.? Is there anything else I need to know as 
I
> attempt to parse these files?? Thanks.

Unless you have done something special (why would you?) the types
for all? libdwarf fields are such that they all hold 64bit values.
So the address data (regardless if 32 or 64 bits) will be in individual fields.

libdwarf has always done everything in 64bit signed/unsigned at all times
(except the fields like the DWARF version and a few others that are defined
to be smaller than 64 bits).

So reading 64 bits requires no effort on your part other than getting the
libelf that has the 64bit interfaces (and indeed I would expect 64bit libelf 
functions to be
on any libelf one would be likely to find!).

DavidAnderson

-- The thunder was ominous sounding, much like the sound of a thin sheet of 
metal being shaken backstage during the storm scene in a play. -- Barbara 
Fetherolf

_______________________________________________
Dwarf-Discuss mailing list
Dwarf-Discuss at lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org




From jakub@redhat.com  Wed Jul 13 18:20:12 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Wed, 13 Jul 2011 20:20:12 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
Message-ID: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>

Hi!

The current .debug_macinfo format isn't very much extensible
and is not really designed to allow reducing of duplicate debug
info, so when compilers want to produce maco information, size
of .debug_macinfo is prohibitive.

We are discussing some extensions that would allow us to reduce
the size of the macinfo hundredfolds while preserving all information
it contains, see
http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01070.html
but as .debug_macinfo in DWARF{2,3,4} currently defines just
0 opcode as terminator, 4 standard opcodes and an unusable vendor
extension opcode (the arguments are DW_FORM_udata (or sdata) and
DW_FORM_string, both need to be specified, but the string
is zero terminated and thus can't accomodate arbitrarily encoded operands
and the uleb128 constant operand can't accomodate e.g. section offsets,
because they need to be fixed size (4byte or 8 byte, so that relocations
can be applied there), I'd like to ask if it would be possible for the
upcoming DWARF to reserve a range of DW_MACINFO_* opcodes for vendor
purposes, say DW_MACINFO_lo_user 0xe0 through DW_MACINFO_hi_user 0xfe.
Debug info consumers which would encounter opcodes they don't support
(be it in this range or in the standard reserved range) would just need to
give up on parsing that .debug_macinfo sequence, which is unfortunate of
course, but e.g. the above proposal has an opcode to allow for future
extensions that would allow consumers to skip the unknown opcodes if they
are documented through the special new opcode.
If this works out, we would of course like to propose the extensions later
on for DWARF5 in a standard opcode range.

So, is this acceptable or do we need to just use different section
names (.debug_gnu_macinfo) and use different, vendor DW_AT_GNU_macro_info
attributes to point into that section, and stop producing .debug_macinfo)?

	Jakub



From eager@eagercon.com  Wed Jul 13 19:25:51 2011
From: eager@eagercon.com (Michael Eager)
Date: Wed, 13 Jul 2011 12:25:51 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E1DF13F.60704@eagercon.com>

On 07/13/2011 11:20 AM, Jakub Jelinek wrote:

> http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01070.html

Thanks for mentioning this.  I have a filter on the gcc mailing
list for discussions about DWARF, but not on the gcc-patches list.

I'll take a look and offer some comments later.

> So, is this acceptable or do we need to just use different section
> names (.debug_gnu_macinfo) and use different, vendor DW_AT_GNU_macro_info
> attributes to point into that section, and stop producing .debug_macinfo)?

My general bias is to fix and/or extend existing data formats, rather than
replace them with something completely different.   Worse, in my mind,
would be to replace an existing format with something which is almost the same.

-- 
Michael Eager	 eager at eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077



From davea42@earthlink.net  Thu Jul 14 15:48:31 2011
From: davea42@earthlink.net (David Anderson)
Date: Thu, 14 Jul 2011 08:48:31 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E1F0FCF.3070902@earthlink.net>

On 07/13/2011 11:20 AM, Jakub Jelinek wrote:
> The current .debug_macinfo format isn't very much extensible
> and is not really designed to allow reducing of duplicate debug
> info, so when compilers want to produce maco information, size
> of .debug_macinfo is prohibitive.
The existing macinfo format is the result of committee exhaustion
in 1992 (we had been having ~monthly meetings around the
country for many months and wanted to get done with Version 1 and
macro information was the last and  least-critical stumbling-block).
Back in the day when companies would readily send people to
far-off meetings!

There were two competing proposals, both allowing some space savings,
and neither I nor the other proposer explained our respective concepts 
very well.
So the committee adopted something simple that would work.   Yes, it takes
far too much space.

M. Eager's comments are right on, but in addition, I think it is clear we
would expect some real numbers (not guessed numbers)
showing the space advantage for real applications of any
proposed revision.  I guess this is obvious :-)

DavidA.

-- 
 From the attic came an unearthly howl. The whole scene had an eerie, surreal quality, like when you're on vacation in another city and "Jeopardy" comes on at 7:00 p.m. instead of 7:30. -- Roy Ashley




From jakub@redhat.com  Thu Jul 14 16:10:21 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Thu, 14 Jul 2011 18:10:21 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E1F0FCF.3070902@earthlink.net>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E1F0FCF.3070902@earthlink.net>
Message-ID: <20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>

On Thu, Jul 14, 2011 at 08:48:31AM -0700, David Anderson wrote:
> On 07/13/2011 11:20 AM, Jakub Jelinek wrote:
> There were two competing proposals, both allowing some space savings,
> and neither I nor the other proposer explained our respective
> concepts very well.
> So the committee adopted something simple that would work.   Yes, it takes
> far too much space.
> 
> M. Eager's comments are right on, but in addition, I think it is clear we
> would expect some real numbers (not guessed numbers)
> showing the space advantage for real applications of any
> proposed revision.  I guess this is obvious :-)

The real numbers were given in the mail I've posted URL to,
in particular e.g. on x86_64-linux cc1plus:
old:
-rwxrwxr-x. 1 jakub jakub 484483231 Jul 13 12:43 cc1plus
  [12] .text             PROGBITS        00000000004886e0 0886e0 df7938 00  AX  0   0 16
  [26] .debug_aranges    PROGBITS        0000000000000000 1239215 00be90 00      0   0  1
  [27] .debug_info       PROGBITS        0000000000000000 12450a5 276ae23 00      0   0  1
  [28] .debug_abbrev     PROGBITS        0000000000000000 39afec8 08bb08 00      0   0  1
  [29] .debug_line       PROGBITS        0000000000000000 3a3b9d0 47be2d 00      0   0  1
  [30] .debug_str        PROGBITS        0000000000000000 3eb77fd 0faa87 01  MS  0   0  1
  [31] .debug_loc        PROGBITS        0000000000000000 3fb2284 2e56f19 00      0   0  1
  [32] .debug_macinfo    PROGBITS        0000000000000000 6e0919d 154a7e83 00      0   0  1
  [33] .debug_ranges     PROGBITS        0000000000000000 1c2b1020 9fed80 00      0   0  1
new:
-rwxrwxr-x. 1 jakub jakub 130126820 Jul 13 18:05 cc1plus
  [12] .text             PROGBITS        00000000004886e0 0886e0 df8538 00  AX  0   0 16
  [26] .debug_aranges    PROGBITS        0000000000000000 1239fb5 00be90 00      0   0  1
  [27] .debug_info       PROGBITS        0000000000000000 1245e45 276c42e 00      0   0  1
  [28] .debug_abbrev     PROGBITS        0000000000000000 39b2273 08bb0e 00      0   0  1
  [29] .debug_line       PROGBITS        0000000000000000 3a3dd81 491054 00      0   0  1
  [30] .debug_str        PROGBITS        0000000000000000 3ecedd5 2865fe 01  MS  0   0  1
  [31] .debug_loc        PROGBITS        0000000000000000 41553d3 2e5a185 00      0   0  1
  [32] .debug_macinfo    PROGBITS        0000000000000000 6faf558 110100 00      0   0  1
  [33] .debug_ranges     PROGBITS        0000000000000000 70bf658 9ffaf0 00      0   0  1
The binaries aren't 100% the same as can be seen on the .text difference,
the difference in between them is the patch that implements this change,
noise in the overall numbers though.

	Jakub



From davea42@earthlink.net  Thu Jul 14 16:33:29 2011
From: davea42@earthlink.net (David Anderson)
Date: Thu, 14 Jul 2011 09:33:29 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E1F0FCF.3070902@earthlink.net>
	<20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E1F1A59.2060606@earthlink.net>

On 07/14/2011 09:10 AM, Jakub Jelinek wrote:

The real numbers were given in the mail I've posted URL to,
> in particular e.g. on x86_64-linux cc1plus:
> old:
> -rwxrwxr-x. 1 jakub jakub 484483231 Jul 13 12:43 cc1plus
>    [12] .text             PROGBITS        00000000004886e0 0886e0 df7938 00  AX  0   0 16
>    [26] .debug_aranges    PROGBITS        0000000000000000 1239215 00be90 00      0   0  1
>    [27] .debug_info       PROGBITS        0000000000000000 12450a5 276ae23 00      0   0  1
>    [28] .debug_abbrev     PROGBITS        0000000000000000 39afec8 08bb08 00      0   0  1
>    [29] .debug_line       PROGBITS        0000000000000000 3a3b9d0 47be2d 00      0   0  1
>    [30] .debug_str        PROGBITS        0000000000000000 3eb77fd 0faa87 01  MS  0   0  1
>    [31] .debug_loc        PROGBITS        0000000000000000 3fb2284 2e56f19 00      0   0  1
>    [32] .debug_macinfo    PROGBITS        0000000000000000 6e0919d 154a7e83 00      0   0  1
>    [33] .debug_ranges     PROGBITS        0000000000000000 1c2b1020 9fed80 00      0   0  1
> new:
> -rwxrwxr-x. 1 jakub jakub 130126820 Jul 13 18:05 cc1plus
>    [12] .text             PROGBITS        00000000004886e0 0886e0 df8538 00  AX  0   0 16
>    [26] .debug_aranges    PROGBITS        0000000000000000 1239fb5 00be90 00      0   0  1
>    [27] .debug_info       PROGBITS        0000000000000000 1245e45 276c42e 00      0   0  1
>    [28] .debug_abbrev     PROGBITS        0000000000000000 39b2273 08bb0e 00      0   0  1
>    [29] .debug_line       PROGBITS        0000000000000000 3a3dd81 491054 00      0   0  1
>    [30] .debug_str        PROGBITS        0000000000000000 3ecedd5 2865fe 01  MS  0   0  1
>    [31] .debug_loc        PROGBITS        0000000000000000 41553d3 2e5a185 00      0   0  1
>    [32] .debug_macinfo    PROGBITS        0000000000000000 6faf558 110100 00      0   0  1
>    [33] .debug_ranges     PROGBITS        0000000000000000 70bf658 9ffaf0 00      0   0  1

So the .debug_macinfo sizes, in decimal are 115,000,000 and 117,000,000 
bytes.
The new macinfo is a little larger than the old.

Now the new .debug_ranges is much smaller than the old (but I thought we 
were speaking
about .debug_macinfo).

So, what am I missing here, where is the savings?

DavidA.

-- 
 From the attic came an unearthly howl. The whole scene had an eerie, surreal quality, like when you're on vacation in another city and "Jeopardy" comes on at 7:00 p.m. instead of 7:30. -- Roy Ashley




From davea42@earthlink.net  Thu Jul 14 16:38:48 2011
From: davea42@earthlink.net (David Anderson)
Date: Thu, 14 Jul 2011 09:38:48 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E1F1A59.2060606@earthlink.net>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>	<4E1F0FCF.3070902@earthlink.net>	<20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E1F1A59.2060606@earthlink.net>
Message-ID: <4E1F1B98.4090503@earthlink.net>

On 07/14/2011 09:33 AM, David Anderson wrote:
> On 07/14/2011 09:10 AM, Jakub Jelinek wrote:
>
> The real numbers were given in the mail I've posted URL to,
>> in particular e.g. on x86_64-linux cc1plus:
>> old:
>> -rwxrwxr-x. 1 jakub jakub 484483231 Jul 13 12:43 cc1plus
>>    [12] .text             PROGBITS        00000000004886e0 0886e0 
>> df7938 00  AX  0   0 16
>>    [26] .debug_aranges    PROGBITS        0000000000000000 1239215 
>> 00be90 00      0   0  1
>>    [27] .debug_info       PROGBITS        0000000000000000 12450a5 
>> 276ae23 00      0   0  1
>>    [28] .debug_abbrev     PROGBITS        0000000000000000 39afec8 
>> 08bb08 00      0   0  1
>>    [29] .debug_line       PROGBITS        0000000000000000 3a3b9d0 
>> 47be2d 00      0   0  1
>>    [30] .debug_str        PROGBITS        0000000000000000 3eb77fd 
>> 0faa87 01  MS  0   0  1
>>    [31] .debug_loc        PROGBITS        0000000000000000 3fb2284 
>> 2e56f19 00      0   0  1
>>    [32] .debug_macinfo    PROGBITS        0000000000000000 6e0919d 
>> 154a7e83 00      0   0  1
>>    [33] .debug_ranges     PROGBITS        0000000000000000 1c2b1020 
>> 9fed80 00      0   0  1
>> new:
>> -rwxrwxr-x. 1 jakub jakub 130126820 Jul 13 18:05 cc1plus
>>    [12] .text             PROGBITS        00000000004886e0 0886e0 
>> df8538 00  AX  0   0 16
>>    [26] .debug_aranges    PROGBITS        0000000000000000 1239fb5 
>> 00be90 00      0   0  1
>>    [27] .debug_info       PROGBITS        0000000000000000 1245e45 
>> 276c42e 00      0   0  1
>>    [28] .debug_abbrev     PROGBITS        0000000000000000 39b2273 
>> 08bb0e 00      0   0  1
>>    [29] .debug_line       PROGBITS        0000000000000000 3a3dd81 
>> 491054 00      0   0  1
>>    [30] .debug_str        PROGBITS        0000000000000000 3ecedd5 
>> 2865fe 01  MS  0   0  1
>>    [31] .debug_loc        PROGBITS        0000000000000000 41553d3 
>> 2e5a185 00      0   0  1
>>    [32] .debug_macinfo    PROGBITS        0000000000000000 6faf558 
>> 110100 00      0   0  1
>>    [33] .debug_ranges     PROGBITS        0000000000000000 70bf658 
>> 9ffaf0 00      0   0  1
>

My bad, I was looking at the offset, not size.

Old size 0x154a7e83   357MB new 0x110100  1.1MB?

Yes, that is dramatic savings.

DavidA.


-- 
 From the attic came an unearthly howl. The whole scene had an eerie, surreal quality, like when you're on vacation in another city and "Jeopardy" comes on at 7:00 p.m. instead of 7:30. -- Roy Ashley




From jakub@redhat.com  Thu Jul 14 16:47:01 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Thu, 14 Jul 2011 18:47:01 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E1F1B98.4090503@earthlink.net>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E1F0FCF.3070902@earthlink.net>
	<20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E1F1A59.2060606@earthlink.net> <4E1F1B98.4090503@earthlink.net>
Message-ID: <20110714164701.GG2687@tyan-ft48-01.lab.bos.redhat.com>

On Thu, Jul 14, 2011 at 09:38:48AM -0700, David Anderson wrote:
> My bad, I was looking at the offset, not size.
> 
> Old size 0x154a7e83   357MB new 0x110100  1.1MB?

Yeah, .debug_macinfo shrunk from 340MiB to 1MiB, on the
other side .debug_str grew up from 1MiB to 2.5MiB
(as the longer define/undef strings now live in .debug_str
instead of .debug_macinfo).

	Jakub



From eager@eagercon.com  Thu Jul 14 17:48:43 2011
From: eager@eagercon.com (Michael Eager)
Date: Thu, 14 Jul 2011 10:48:43 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110714164701.GG2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>	<4E1F0FCF.3070902@earthlink.net>	<20110714161021.GD2687@tyan-ft48-01.lab.bos.redhat.com>	<4E1F1A59.2060606@earthlink.net>
	<4E1F1B98.4090503@earthlink.net>
	<20110714164701.GG2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E1F2BFB.2080104@eagercon.com>

On 07/14/2011 09:47 AM, Jakub Jelinek wrote:
> On Thu, Jul 14, 2011 at 09:38:48AM -0700, David Anderson wrote:
>> My bad, I was looking at the offset, not size.
>>
>> Old size 0x154a7e83   357MB new 0x110100  1.1MB?
>
> Yeah, .debug_macinfo shrunk from 340MiB to 1MiB, on the
> other side .debug_str grew up from 1MiB to 2.5MiB
> (as the longer define/undef strings now live in .debug_str
> instead of .debug_macinfo).

Not a bad trade-off.


-- 
Michael Eager	 eager at eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077



From eager@eagercon.com  Wed Jul 20 17:16:10 2011
From: eager@eagercon.com (Michael Eager)
Date: Wed, 20 Jul 2011 10:16:10 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E270D5A.9060806@eagercon.com>

On 07/13/2011 11:20 AM, Jakub Jelinek wrote:
> Hi!
>
> The current .debug_macinfo format isn't very much extensible
> and is not really designed to allow reducing of duplicate debug
> info, so when compilers want to produce maco information, size
> of .debug_macinfo is prohibitive.
>
> We are discussing some extensions that would allow us to reduce
> the size of the macinfo hundredfolds while preserving all information
> it contains, see
> http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01070.html

Hi Jakub --

It took me a few days to review the current DWARF macinfo specification
and review this proposal.

The existing macro data format is clearly in need of revision.   I don't
think that there is any way to modify this format which would be transparent
and usable by a consumer which expected the current format.

I recommend that you create a new section to contain this data, perhaps
.debug_macinfo2 or .debug_macro.  Create a file header which includes at a
minimum a version number, similar to the other DWARF data sections.
This will give more flexibility in defining the contents of the section.

Define new macinfo opcodes as in your proposal (you can number them from 0x01).
Rather than using either the existing DW_MACINFO_vendor_ext opcode or your
DW_MACINFO_GNU_define_opcode for vendor extensions, I would define a vendor
extension range using the same *_lo_user and *_hi_user scheme used in other
parts of DWARF.  Actually, I would suggest that GCC not implement a vendor
extension scheme at this time, but wait until the DWARF committee has had an
opportunity to review the proposal and incorporate it into a future version
of DWARF.  Should you discover a need for an additional opcode, simply add
it to the implementation.

The DW_AT_macro_info attribute would be a section offset into either
.debug_macinfo or .debug_macro, depending on which was present.

I think that your modifications would be a significant improvement to DWARF.
Please submit a proposal at http://dwarfstd.org/Comment.php.


-- 
Michael Eager	 eager at eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077



From jakub@redhat.com  Wed Jul 20 18:00:47 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Wed, 20 Jul 2011 20:00:47 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E270D5A.9060806@eagercon.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E270D5A.9060806@eagercon.com>
Message-ID: <20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>

On Wed, Jul 20, 2011 at 10:16:10AM -0700, Michael Eager wrote:
> It took me a few days to review the current DWARF macinfo specification
> and review this proposal.
> 
> The existing macro data format is clearly in need of revision.   I don't
> think that there is any way to modify this format which would be transparent
> and usable by a consumer which expected the current format.
> 
> I recommend that you create a new section to contain this data, perhaps
> .debug_macinfo2 or .debug_macro.  Create a file header which includes at a
> minimum a version number, similar to the other DWARF data sections.
> This will give more flexibility in defining the contents of the section.

Yeah, this is the other option I was giving, one was modifying the existing
.debug_macinfo and the other option is coming up with a new section format.

> Define new macinfo opcodes as in your proposal (you can number them from 0x01).
> Rather than using either the existing DW_MACINFO_vendor_ext opcode or your
> DW_MACINFO_GNU_define_opcode for vendor extensions, I would define a vendor
> extension range using the same *_lo_user and *_hi_user scheme used in other
> parts of DWARF.

Well, one thing is extension range for vendor extensions and the other is
how to teach consumers about what arguments the new opcodes have.  The
define_opcode op was meant like an embedded abbrev which would specify the
argument encodings, which is still useful.  Of course vendor_ext opcode
would be useless in that case.

>  Actually, I would suggest that GCC not implement a vendor
> extension scheme at this time,

If we go the new section way, then surely the initial ops would be all
non-vendor extensions.

> but wait until the DWARF committee has had an
> opportunity to review the proposal and incorporate it into a future version
> of DWARF.  Should you discover a need for an additional opcode, simply add
> it to the implementation.

I'm afraid we don't want to wait a few years for DWARF5 though, we need
this right now, which is why if DWARF committee preference is a new section,
I think we should introduce .debug_gnu_macro section and keep it as whole
as a GNU extension and propose .debug_macro section with the same content
for DWARF5.  Then, if DWARF5 uses the same format, perfect, if it uses a
different one, the consumers could just choose to support both formats or
something and GCC as a producer would phase the GNU extension format out.

> The DW_AT_macro_info attribute would be a section offset into either
> .debug_macinfo or .debug_macro, depending on which was present.

That won't work, you need different attributes, because DW_FORM_sec_offset
is relative to the start of the section, but doesn't say which (in ELF
.debug* sections have all 0 as starting address).  If DW_AT_macro_info
value was .debug_macinfo, .debug_macro or .debug_gnu_macro section index
depending on which of the 3 sections is present, any time you link together
objects compiled with different compilers and one e.g. produces
.debug_macinfo, one .debug_gnu_macro, then both sections will be present and
it is unclear in each of the CUs what section is DW_AT_macro_info refering
to.

> I think that your modifications would be a significant improvement to DWARF.
> Please submit a proposal at http://dwarfstd.org/Comment.php.

Sure.

	Jakub



From eager@eagercon.com  Wed Jul 20 18:07:31 2011
From: eager@eagercon.com (Michael Eager)
Date: Wed, 20 Jul 2011 11:07:31 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E270D5A.9060806@eagercon.com>
	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E271963.7040301@eagercon.com>

On 07/20/2011 11:00 AM, Jakub Jelinek wrote:
> On Wed, Jul 20, 2011 at 10:16:10AM -0700, Michael Eager wrote:
>> It took me a few days to review the current DWARF macinfo specification
>> and review this proposal.
>>
>> The existing macro data format is clearly in need of revision.   I don't
>> think that there is any way to modify this format which would be transparent
>> and usable by a consumer which expected the current format.
>>
>> I recommend that you create a new section to contain this data, perhaps
>> .debug_macinfo2 or .debug_macro.  Create a file header which includes at a
>> minimum a version number, similar to the other DWARF data sections.
>> This will give more flexibility in defining the contents of the section.
>
> Yeah, this is the other option I was giving, one was modifying the existing
> .debug_macinfo and the other option is coming up with a new section format.
>
>> Define new macinfo opcodes as in your proposal (you can number them from 0x01).
>> Rather than using either the existing DW_MACINFO_vendor_ext opcode or your
>> DW_MACINFO_GNU_define_opcode for vendor extensions, I would define a vendor
>> extension range using the same *_lo_user and *_hi_user scheme used in other
>> parts of DWARF.
>
> Well, one thing is extension range for vendor extensions and the other is
> how to teach consumers about what arguments the new opcodes have.  The
> define_opcode op was meant like an embedded abbrev which would specify the
> argument encodings, which is still useful.  Of course vendor_ext opcode
> would be useless in that case.
>
>>   Actually, I would suggest that GCC not implement a vendor
>> extension scheme at this time,
>
> If we go the new section way, then surely the initial ops would be all
> non-vendor extensions.
>
>> but wait until the DWARF committee has had an
>> opportunity to review the proposal and incorporate it into a future version
>> of DWARF.  Should you discover a need for an additional opcode, simply add
>> it to the implementation.
>
> I'm afraid we don't want to wait a few years for DWARF5 though, we need
> this right now, which is why if DWARF committee preference is a new section,
> I think we should introduce .debug_gnu_macro section and keep it as whole
> as a GNU extension and propose .debug_macro section with the same content
> for DWARF5.  Then, if DWARF5 uses the same format, perfect, if it uses a
> different one, the consumers could just choose to support both formats or
> something and GCC as a producer would phase the GNU extension format out.

Sorry if I was not clear.  I was only referring to deferring implementation
of the vendor extension opcode, not the new data section.

>
>> The DW_AT_macro_info attribute would be a section offset into either
>> .debug_macinfo or .debug_macro, depending on which was present.
>
> That won't work, you need different attributes, because DW_FORM_sec_offset
> is relative to the start of the section, but doesn't say which (in ELF
> .debug* sections have all 0 as starting address).  If DW_AT_macro_info
> value was .debug_macinfo, .debug_macro or .debug_gnu_macro section index
> depending on which of the 3 sections is present, any time you link together
> objects compiled with different compilers and one e.g. produces
> .debug_macinfo, one .debug_gnu_macro, then both sections will be present and
> it is unclear in each of the CUs what section is DW_AT_macro_info refering
> to.

You are right.  A new attribute is required.

-- 
Michael Eager	 eager at eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077



From davea42@earthlink.net  Wed Jul 20 20:07:17 2011
From: davea42@earthlink.net (David Anderson)
Date: Wed, 20 Jul 2011 13:07:17 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>	<4E270D5A.9060806@eagercon.com>
	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E273575.7010702@earthlink.net>


> If we go the new section way, then surely the initial ops would be all
> non-vendor extensions.
>

DW_*  identifiers not defined by the standard are extensions, so I would 
expect
them to be named and numbered as such.

On the other hand,  I have not seen the detailed proposal, so perhaps 
that proposal
will address the namespace/namespace-numbering issue effectively somehow.

DavidA.
The libdwarf web site (sgiweb.org) has gone off the air (presumably 
temporarily).
libdwarf is still available in git on sourceforge.  If sgiweb.org does 
not come back
soon I will establish a new website for libdwarf.

-- 
The young fighter had a hungry look, the kind you get from not eating for a while. -- Malcolm Fleschner




From davea42@earthlink.net  Wed Jul 20 20:17:57 2011
From: davea42@earthlink.net (David Anderson)
Date: Wed, 20 Jul 2011 13:17:57 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E273575.7010702@earthlink.net>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>	<4E270D5A.9060806@eagercon.com>	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E273575.7010702@earthlink.net>
Message-ID: <4E2737F5.1050304@earthlink.net>

On 07/20/2011 01:07 PM, David Anderson wrote:
>
> On the other hand,  I have not seen the detailed proposal, so perhaps 
> that proposal
> will address the namespace/namespace-numbering issue effectively somehow.
Duh. Of course the proposal will propose non-extension names and numbers 
as required.
Silly me.

A preliminary vendor implementation  on the other hand... Hmm.
DavidA.



From jakub@redhat.com  Thu Jul 21 11:36:13 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Thu, 21 Jul 2011 13:36:13 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E2737F5.1050304@earthlink.net>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E270D5A.9060806@eagercon.com>
	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E273575.7010702@earthlink.net> <4E2737F5.1050304@earthlink.net>
Message-ID: <20110721113613.GN2687@tyan-ft48-01.lab.bos.redhat.com>

On Wed, Jul 20, 2011 at 01:17:57PM -0700, David Anderson wrote:
> On 07/20/2011 01:07 PM, David Anderson wrote:
> >
> >On the other hand,  I have not seen the detailed proposal, so
> >perhaps that proposal

The previous proposal (the one adding opcodes to the current
.debug_macinfo instead of creating a new section) was worded roughly in
http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01284.html

In http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01736.html
is a rough implementation of .debug_gnu_macro section and
DW_AT_GNU_macros referencing it.

Here are rough DWARF4 edits for this.
I'm not sure in what way should the old .debug_macinfo be
deprecated in DWARF5, the following edits describe both the
.debug_macro section and .debug_macinfo too, and refer to the
latter as legacy.  Another option I guess would be to document
in DWARF5 only .debug_macro section and just in DW_AT_macro_info
description say that it doesn't refer to .debug_macro section,
but instead to legacy .debug_macinfo as defined in DWARF2 through DWARF4
and other than that don't document .debug_macinfo at all, in appendix
F it would have a rows like:
.debug_macinfo	-	-	-	x
.debug_macro	x	x	x	5
then.  What do you prefer?

Also, regarding the section headers, I'm still not convinced they are
particularly useful, as .debug_macro is only useful with the corresponding
referencing .debug_info which has its own version number.  But if you think
it is useful, should it be in front of all .debug_macro chunks (even the
transparent include chains)?  And, should it include the offset size
or is that unnecessary?  With the section headers everywhere and offset size
everywhere some tools could parse/display .debug_macro content even without
parsing referencing .debug_info, just it couldn't give meaning to start_file
filenames (translate the constant into filename string).  Without them it
can't and would need to handle it e.g. similarly how .debug_loc needs to be
handled in such tools.  As the msg01736.html URL above refers to,
in the test binary from the compiler itself, there are 395 DW_AT_GNU_macros
attributes and 511 unique DW_GNU_MACRO_transparent_include chains, so
the size of 3 byte headers for 395+511 chunks is < 3KB for the ~ 1MB size
of .debug_gnu_macro, i.e. .3%.

2.2
Change "Macro information (#define, #undef)" to "Legacy macro information (#define, #undef)".
Add
"DW_AT_macros		Macro information (#define, #undef)"
row to the figure 2.  In figure 3 replace:
add "legacy " before "macro definition" in "macptr" row.  Add
"macroptr	Refers to a location in the DWARF section that holds macro definition information."
row.

3.1.1
5. Add "legacy " before "macro information".

Add:
12. A DW_AT_macros attribute whose value is a section offset to the
macro information for this compilation unit.

This information is placed in a separate object file section from the
debugging information entries themselves. The value of the macro information
attribute is the offset in the .debug_macro section of the section header
of the macro information for this compilation unit (see Section 6.3).

6.3
Replace ".debug_macinfo" with ".debug_macro".
Replace:
"The macro information for each compilation unit is represented as a series
of ?macinfo? entries. Each macinfo entry consists of a ?type code?
and up to two additional operands."
with:
"The macro information for each compilation unit starts with a section
header followed by a series of ?macinfo? entries. Each macinfo entry
consists of a ?type code? and zero or more operands."
Add at the end:
"The section header starts with a 2-byte version number, followed by
1-byte offset size, which is 4 in 32-bit DWARF and 8 in 64-bit DWARF."

6.3.1
Replace:
"DW_MACINFO_define	A macro definition.
DW_MACINFO_undef	A macro undefinition.
DW_MACINFO_start_file	The start of a new source file inclusion.
DW_MACINFO_end_file	The end of the current source file inclusion.
DW_MACINFO_vendor_ext	Vendor specific macro information directives."
with
"DW_MACRO_define		A macro definition.
DW_MACRO_undef			A macro undefinition.
DW_MACRO_start_file		The start of a new source file inclusion.
DW_MACRO_end_file		The end of the current source file inclusion.
DW_MACRO_define_indirect	A macro definition.
DW_MACRO_undef_indirect		A macro undefinition.
DW_MACRO_transparent_include	Include a sequence of entries from given offset.
DW_MACRO_define_opcode		Define extension opcode and its arguments."

6.3.1.1
Replace all "DW_MACINFO_define" occurrences with "DW_MACRO_define" and
all "DW_MACINFO_undef" with "DW_MACRO_undef".  Append:
a"All DW_MACRO_define_indirect and DW_MACRO_undef_indirect entries have
two operands.  The first operand encodes the line number of the source line
on which the relevant defining or undefining macro directives appeared.
The second operand consists of an offset into a string table contained in
the .debug_str section of the object file.  The size of the operand is
given in the section header offset size.  Apart from the
encoding of the operands these entries are equivalent to DW_MACRO_define
resp. DW_MACRO_undef.

6.3.1.2
Replace all "DW_MACINFO_start_file" occurrences with "DW_MACRO_start_file".

6.3.1.3
Replace all "DW_MACINFO_end_file" occurrences with "DW_MACRO_end_file".

6.3.1.4
Replace whole section with:
"6.3.1.4  Transparent inclusion of a sequence of entries

A DW_MACRO_transparent_include entry has one operand, offset into
another part of the .debug_macro section.  The size of the operand
is given in the section header offset size.  This entry instructs
the consumer to replace this entry with a sequence of macinfo entries found
at the given .debug_macro offset, up to, but excluding, the terminating
entry with type code 0.  This entry type is aimed at sharing duplicate
sequences of macinfo entries between macinfo from different compilation
units.  The producer should ensure that all referers to the sequence have
the same section header and that either DW_MACRO_start_file entries aren't
in those sequences, or only macinfo entries referencing the same
.debug_line section part include the sequence."

Add:
"6.3.1.5  Defining new opcodes and operands

A DW_MACRO_define_opcode entry has 2 operands.  The first operand
is a one byte constant with the type code it defines operand types for,
the second operand is a DW_FORM_block encoded array of operand forms.
The second operand starts with an unsigned LEB128 encoded number of operands
and for each of the operands there is one byte, containing a form encoding
how the corresponding operand is encoded.  This entry allows to define
new vendor extension entry types which consumers will be able to skip over
and ignore.  Each so defined opcode is valid for subsequent entries
until the terminating entry with type code 0, including any sequences
included from those entries using DW_MACRO_transparent_include.
Vendor extension macinfo entry codes should be added in the range
DW_MACRO_lo_user to DW_MACRO_hi_user, all other unassigned opcodes
are reserved for future DWARF standards."

6.3.2
Replace "DW_MACINFO_start_file" with "DW_MACRO_start_file" and
"DW_MACINFO_end_file" with "DW_MACRO_end_file".

6.3.3
Replace "DW_MACINFO_define and DW_MACINFO_undef" with
"DW_MACRO_define, DW_MACRO_define_indirect, DW_MACRO_undef and
DW_MACRO_undef_indirect" and replace
"DW_MACINFO_define or DW_MACINFO_undef" with
"DW_MACRO_define, DW_MACRO_define_indirect, DW_MACRO_undef or
DW_MACRO_undef_indirect".
Replace "DW_MACINFO_start_file" with "DW_MACRO_start_file".

6.3.4
Replace ".debug_macinfo" with ".debug_macro".

Add new section:
"6.3.5  Legacy macro information

The legacy macro information is present in section .debug_macinfo
instead of .debug_macro and doesn't start with any section header, it is
just a series of macinfo entries.  There are 5 legacy macinfo opcode types:

DW_MACINFO_define	A macro definition.
DW_MACINFO_undef	A macro undefinition.
DW_MACINFO_start_file	The start of a new source file inclusion.
DW_MACINFO_end_file	The end of the current source file inclusion.
DW_MACINFO_vendor_ext	Vendor specific macro information directives.

The first 4 are, appart from different name, counterparts of
DW_MACRO_define, DW_MACRO_undef, DW_MACRO_start_file and DW_MACRO_end_file,
with the same encoding of arguments and same semantics.

6.3.5.1  Vendor Extension Entries
A DW_MACINFO_vendor_ext entry has two operands. The first is a constant. The
second is a null-terminated character string. The meaning and/or significance
of these operands is intentionally left undefined by this specification.
A consumer must be able to totally ignore all DW_MACINFO_vendor_ext entries
that it does not understand (see Section 7.1)."

7.5.4
Replace
"loclistptr, macptr or rangelistptr;" with "loclistptr, macptr, macroptr or
rangelistptr;".
Add:
"macroptr
This is an offset into the .debug_macro section (DW_FORM_sec_offset). It
consists of an offset from the beginning of the .debug_macinfo section to the
first byte of the data making up the macro information list for the compilation
unit. It is relocatable in a relocatable object file, and relocated in an
executable or shared object. In the 32-bit DWARF format, this
offset is a 4-byte unsigned value; in the 64-bit DWARF format, it is an
8-byte unsigned value (see Section 7.4)."

Add:
"DW_AT_macros	0xXX	macroptr"
to figure 20.

In figure 21 on "DW_FORM_sec_offset" row change "macptr," to "macptr, macroptr,".

7.22
After:
"The macinfo type is encoded as a single byte. The encodings are given in Figure 39."
add a new table (new figure 39):

"Macinfo type name		Value
DW_MACRO_define			0x01
DW_MACRO_undef			0x02
DW_MACRO_start_file		0x03
DW_MACRO_end_file		0x04
DW_MACRO_define_indirect	0x05
DW_MACRO_undef_indirect		0x06
DW_MACRO_transparent_include	0x07
DW_MACRO_define_opcode		0x08
DW_MACRO_lo_user		0xe0
DW_MACRO_hi_user		0xff

Macinfo type encodings"

followed by:

"The legacy macinfo type is encoded as a single byte. The encodings are
given in Figure 40." and replacing "Macinfo type name" with
"Legacy macinfo type name" in the table header and "Macinfo type encodings"
with "Legacy macinfo type encodings" in the description.

Appendix A
Add "DW_AT_macros" to allowable attributes for "DW_TAG_compile_unit"
and "DW_TAG_partial_unit".

Appendix B
Fix a typo, replace "DW_AT_macinfo" with "DW_AT_macro_info".  Add
( .debug_info/.debug_types ) -> [ DW_AT_macros (j) ] -> ( .debug_macro )
and from ( .debug_macro ) -> [ DW_MACRO_transparent_include (k) ] -> ( .debug_macro ) and
( .debug_macro ) -> [ DW_MACRO_define_indirect, DW_MACRO_undef_indirect (l) ] -> (.debug_str )
to the picture.
In (h) add "legacy " before ".debug_macinfo".
Add:
"(j) .debug_info An attribute value of class macroptr (specifically form
		 DW_FORM_sec_offset) is an offset within the .debug_macro
		 section of the section header of the macro information
		 for the referencing unit.
(k) .debug_macro A macinfo operand of the form DW_FORM_sec_offset is an
		 offset into another part of the .debug_macro section,
		 to the first macinfo entry in the sequence instead of
		 a section header.
(l) .debug_macro A macinfo operand of the form DW_FORM_strp is an offset
		 into the .debug_str section of the corresponding string."

Appendix F
Add
".debug_macro	x	x	x	5"
row to the table.

	Jakub



From mjw@redhat.com  Thu Jul 21 11:37:54 2011
From: mjw@redhat.com (Mark Wielaard)
Date: Thu, 21 Jul 2011 13:37:54 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E270D5A.9060806@eagercon.com>
	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <1311248274.4259.22.camel@springer.wildebeest.org>

On Wed, 2011-07-20 at 20:00 +0200, Jakub Jelinek wrote:
> On Wed, Jul 20, 2011 at 10:16:10AM -0700, Michael Eager wrote:
> > but wait until the DWARF committee has had an
> > opportunity to review the proposal and incorporate it into a future version
> > of DWARF.  Should you discover a need for an additional opcode, simply add
> > it to the implementation.
> 
> I'm afraid we don't want to wait a few years for DWARF5 though, we need
> this right now, which is why if DWARF committee preference is a new section,
> I think we should introduce .debug_gnu_macro section and keep it as whole
> as a GNU extension and propose .debug_macro section with the same content
> for DWARF5.  Then, if DWARF5 uses the same format, perfect, if it uses a
> different one, the consumers could just choose to support both formats or
> something and GCC as a producer would phase the GNU extension format out.

Since there is a version field, maybe we could make use of that. So the
preliminary GNU vendor implementation of the .debug_macro section would
use version 1. When DWARF5 is officially published we can check whether
it still matches the preliminary implementation. If it does, the
standard just blesses version 1 as the official version, if not it
starts at 2 (or maybe 5 to match up with the spec number) so consumers
know whether they are dealing with the preliminary implementation or the
"officially blessed" spec compliant version.

Cheers,

Mark



From jakub@redhat.com  Thu Jul 21 17:55:18 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Thu, 21 Jul 2011 19:55:18 +0200
Subject: [Dwarf-Discuss] [RFC] More compact (100x) -g3 .debug_gnu_macro
	(take 4)
In-Reply-To: <4E285D8F.1000602@redhat.com>
References: <20110713170053.GX2687@tyan-ft48-01.lab.bos.redhat.com>
	<20110715154223.GM2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E206952.8000601@redhat.com>
	<20110715205817.GP2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E261290.6080004@redhat.com>
	<20110721112234.GM2687@tyan-ft48-01.lab.bos.redhat.com>
	<4E285D8F.1000602@redhat.com>
Message-ID: <20110721175518.GU2687@tyan-ft48-01.lab.bos.redhat.com>

On Thu, Jul 21, 2011 at 10:10:39AM -0700, Richard Henderson wrote:
> On 07/21/2011 04:22 AM, Jakub Jelinek wrote:
> > Currently, the patch emits 3 byte section headers at the start of
> > the .debug_gnu_macro chunks referenced from .debug_info (through
> > DW_AT_GNU_macros), containing version number (2 byte, 4 ATM) and
> > 1 byte section offset, but the DW_GNU_MACRO_transparent_include
> > referenced sequences don't have it.
> > The .debug_gnu_macro section isn't completely usable without the referencing
> > CUs anyway, so IMHO we could still get away completely without
> > any section header, but if we need it, the question is if
> > the offset size there is useful and if the section header shouldn't
> > go before the transparent_include chains as well (only with that
> > e.g. readelf -wm would be able to dump .debug_gnu_macro without
> > reading .debug_info and tracking offsets to it).
> 
> I've been wondering if the header shouldn't contain the opcode
> definitions, similar to .debug_line, and drop your _define_opcode.
> It does mean that you couldn't re-define opcodes within any one
> sequence, but does that actually seem useful?

I've talked to Tom about it last night.  The advantage of
not having it in the header is saving 1 byte for the case when
no extension opcodes need to be defined, and perhaps if we changed
the wording that the defined opcodes end at 0 termination to allow
it to last, then we could with many opcodes share the opcode arguments
descriptions.
So we could have
DW_GNU_MACRO_transparent_include .Ldebug_macro17
after the header of many sections and then
.Ldebug_macro17:
DW_GNU_MACRO_define_opcode DW_GNU_MACRO_lo_user+0 1 DW_FORM_udata
DW_GNU_MACRO_define_opcode DW_GNU_MACRO_lo_user+1 2 DW_FORM_udata DW_FORM_sdata
DW_GNU_MACRO_define_opcode DW_GNU_MACRO_lo_user+2 1 DW_FORM_strp
0

If the opcode definitions were in the header, then they could be
either after a uleb128 that would say how many of the definitions there
are, followed by what I've been proposing as DW_GNU_MACRO_define_opcode
arguments alone (i.e. opcode number and DW_FORM_block array of forms
for arguments).  Or it could be instead without the uleb128, but zero
terminated.

> Defining the opcodes in the header makes it clear that there 
> should be a header for the include sequences, and that makes it
> clear that the defined opcodes are local to a given sequence,
> without having to have awkward wording as for _define_opcode.
> 
> I do like mjw's idea of using the version number to distinguish
> our implementation and one with the dwarf5 stamp of approval.
> This suggests going ahead with .debug_macro as the section name.

If we knew that DWARF5 would either start the .debug_macro sections
with a header starting with the 2 byte version and the version there would be
5 (I think if it does start with a 2 byte version number, it would use 5),
then perhaps it would be safe to use .debug_macro section with version 4 (or
1?).  Shall we use DW_GNU_MACRO_* names, or DW_MACRO_GNU_* names?

> > In x86_64 cc1plus for which I've been posting figures, I see
> > 395 CUs referencing .debug_gnu_macro and at most 511 different
> > .debug_gnu_macro chains with unique md5sums.  So, the cost of the
> > 3 byte headers is for cc1plus just in CU referenced chunks
> > 1185 bytes, 3 byte headers in all .debug_gnu_macro chunks
> > 2718 bytes.
> 
> Putting the opcode definitions into the header would increase
> the overhead more, somewhere between 12 and 20 bytes per chain.
> Which is, I think still manageable.

The question is, do we want to always describe all the opcodes we use,
or can we assume the ops described in the corresponding standard as
given?  Say if DWARF 5 (and our version 4) defines 8 standard opcodes,
and DWARF 6 adds another 3, and we want to use the new opcodes, with
-gdwarf-5 -gno-strict-dwarf we'd define the opcode arguments for the
3 DWARF 6 ops (or a subset of them that we actually use), while for
-gdwarf-6 we wouldn't define any and just put version 6 into the section.

> > Also, should the decision whether to emit .debug_gnu_macro or .debug_macinfo
> > depend on -gdwarf-strict, or should we have a separate switch for that?
> 
> I'm fine with strict.  Anyone else have an opinion?

Ok.

	Jakub



From eager@eagercon.com  Thu Jul 21 18:37:26 2011
From: eager@eagercon.com (Michael Eager)
Date: Thu, 21 Jul 2011 11:37:26 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <20110721113613.GN2687@tyan-ft48-01.lab.bos.redhat.com>
References: <20110713182012.GY2687@tyan-ft48-01.lab.bos.redhat.com>	<4E270D5A.9060806@eagercon.com>	<20110720180047.GH2687@tyan-ft48-01.lab.bos.redhat.com>	<4E273575.7010702@earthlink.net>
	<4E2737F5.1050304@earthlink.net>
	<20110721113613.GN2687@tyan-ft48-01.lab.bos.redhat.com>
Message-ID: <4E2871E6.7060207@eagercon.com>

On 07/21/2011 04:36 AM, Jakub Jelinek wrote:
> On Wed, Jul 20, 2011 at 01:17:57PM -0700, David Anderson wrote:
>> On 07/20/2011 01:07 PM, David Anderson wrote:
>>>
>>> On the other hand,  I have not seen the detailed proposal, so
>>> perhaps that proposal
>
> The previous proposal (the one adding opcodes to the current
> .debug_macinfo instead of creating a new section) was worded roughly in
> http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01284.html
>
> In http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01736.html
> is a rough implementation of .debug_gnu_macro section and
> DW_AT_GNU_macros referencing it.
>
> Here are rough DWARF4 edits for this.
> I'm not sure in what way should the old .debug_macinfo be
> deprecated in DWARF5, the following edits describe both the
> .debug_macro section and .debug_macinfo too, and refer to the
> latter as legacy.  Another option I guess would be to document
> in DWARF5 only .debug_macro section and just in DW_AT_macro_info
> description say that it doesn't refer to .debug_macro section,
> but instead to legacy .debug_macinfo as defined in DWARF2 through DWARF4
> and other than that don't document .debug_macinfo at all, in appendix
> F it would have a rows like:
> .debug_macinfo	-	-	-	x
> .debug_macro	x	x	x	5
> then.  What do you prefer?

There are pros and cons to deprecation vs removal of the .debug_macinfo
section.  I think that this is something the DWARF committee will need to
discuss.

There's no need to address this in your proposal.  Just describe a replacement
for .debug_macinfo.  If we decide to include both formats in a future version
of DWARF, we will address this as an editorial issue.

> Also, regarding the section headers, I'm still not convinced they are
> particularly useful, as .debug_macro is only useful with the corresponding
> referencing .debug_info which has its own version number.  But if you think
> it is useful, should it be in front of all .debug_macro chunks (even the
> transparent include chains)?  And, should it include the offset size
> or is that unnecessary?  With the section headers everywhere and offset size
> everywhere some tools could parse/display .debug_macro content even without
> parsing referencing .debug_info, just it couldn't give meaning to start_file
> filenames (translate the constant into filename string).  Without them it
> can't and would need to handle it e.g. similarly how .debug_loc needs to be
> handled in such tools.  As the msg01736.html URL above refers to,
> in the test binary from the compiler itself, there are 395 DW_AT_GNU_macros
> attributes and 511 unique DW_GNU_MACRO_transparent_include chains, so
> the size of 3 byte headers for 395+511 chunks is<  3KB for the ~ 1MB size
> of .debug_gnu_macro, i.e. .3%.

We have a header for almost every data section.  Unless there is a compelling
argument why this particular section should not have one, I believe that we
will include one in any new macro data description.  I do not see that the
macro description is specifically tied in a one-to-one relationship with the
.debug_info section.  In a hypothetical future revision (say DWARF version 6)
we could make changes to the .debug_info section without making changes to
the macro description.  Conversely, at some future date the macro description
could be extended in some (currently unknown) incompatible manner which did not
require changes to the .debug_info format.

We generally have a header before the data from each compilation unit.  Fields
like a reference to the offset of the compilation unit or the offset to the next
set of data in the section are designed to make it easier for a consumer to read
and process a data section incrementally, and, ideally, with little or no reference
to other sections.  Any particular consumer may ignore these fields, preferring to
read all of the data at one time.

I can't comment on your questions about transparent include chains or other details.
Do what you think works best.

Thank you for your detailed proposal.   Please submit the proposed changes at
http://dwarfstd.org/Comment.php.

-- 
Michael Eager	 eager at eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077



From davea42@earthlink.net  Mon Jul 25 17:18:06 2011
From: davea42@earthlink.net (David Anderson)
Date: Mon, 25 Jul 2011 10:18:06 -0700
Subject: [Dwarf-Discuss] Kindle and the  DWARF4 pdf  document.
Message-ID: <4E2DA54E.3030307@earthlink.net>

An open-source application  "Calibre"  can convert  a PDF to mobi format 
(and to other
formats as well).
It's a GUI app and works on Linux (and, I suspect, MacOS  X and Windows).
On Ubuntu 10.x or 11.x:   "sudo apt-get install calibre"

Do not just copy the result of Calibre conversion
to Kindle by hand like a normal
file-copy-to-usb-disk, the result won't be recognized by Kindle.

Calibre will find a Kindle if you plug Kindle into your USB device.
A sort of picture of the Kindle appears in the Calibre menus
at the top of the panel.
Copy mobi format to Kindle using Calibre menus (I found the
Calibre menus a little difficult to use but after a bit of guessing
I got things to work).
Use Calibre menus to dismount/eject Kindle and the document will show
up in the Kindle marked as 'new' in the usual kindle menu.

As of Calibre 0.7.44 the result is usable on Kindle but not wonderful.
The Kindle search mechanism is awkward and limited
and not  configurable.  You can usefully search for whole terms
like dw_at_high_pc, for example.

The text reflows so text shows up ok on the Kindle screen obeying your
Kindle appearance choices, but there are some odd line breaks   -- it would
be a bit much to expect a perfect conversion of the original formatting 
intent.

All the ..... dots in the pdf table of contents of course show up as, 
well, useless screen-consuming dots  :-)
All the big tables wind up pretty difficult to make any sense of, though 
the content seems to be there.

The Calibre-created mobi version text is much more readable on Kindle 
than the pdf version, however.

Just FYI.

David Anderson

-- 
It hurt the way your tongue hurts after you accidentally staple it to the wall. -- Brian Broadus




From davea42@earthlink.net  Fri Jul 29 21:37:33 2011
From: davea42@earthlink.net (David Anderson)
Date: Fri, 29 Jul 2011 14:37:33 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
Message-ID: <4E33281D.2040600@earthlink.net>

ISSUE 1

In the new 6.3.1.5 text:

"Each so defined opcode is valid for subsequent entries
until the terminating entry with type code 0, including any sequences
included from those entries using DW_MACRO_transparent_include."

seems awkward,  it is really unclear whether any transparent_include
terminating entry terminates the DW_MACRO_define_opcode or not.


My attempts at better wording have not been fruitful so far.
I think you mean 'nested include terminating entries don't terminate
DW_MACRO_define_opcode'  but since a nested include could define its own
opcodes (right?) it gets messy to say properly.


-----------------------------------------
ISSUE 2

6.3.5 Legacy macro information




"just a series of macinfo entries.  There are 5 legacy macinfo opcode types:"

Remove the noise word "just", it adds nothing.

---------------------------------------------
ISSUE 3

Final  comment: IMO we should be  careful to say
that any unused bytes in the section are all zero.

That still leaves a problem in that detecting
the next section header after  an arbitrary
series of zero bytes  is endian-dependent (version
number being 2 bytes, of course).

Or we could disallow unused bytes entirely.
Or we could say all bytes must be part of valid .debug_macro
sequences (not necessarily saying that all are referenced from
debug_info).
-----------------

Summary:
Other than the above three issues  .debug_macro seems fine to me.

DavidA.

-- 
On two occasions I have been asked [by members of Parliament!], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.  -- Charles Babbage




From davea42@earthlink.net  Sat Jul 30 00:40:02 2011
From: davea42@earthlink.net (David Anderson)
Date: Fri, 29 Jul 2011 17:40:02 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E33281D.2040600@earthlink.net>
References: <4E33281D.2040600@earthlink.net>
Message-ID: <4E3352E2.6030601@earthlink.net>

I wrote, earlier today, a longer discussion of why I want the 
.debug_macro section
to be readable on its own.  I guess I never actually sent it, operator 
error on my part.

We have several sections that are not readable on their own.  
Implementors (writing
objects and executables) are allowed to insert arbitrary unused bytes , 
making reading
the sections on their own problematic.

       .debug_abbrev,  .debug_loc,  .debug_ranges,  .debug_macinfo
are examples.

It's annoying that while early compiler implementations let me print some
sections independently
(in spite of a lack of any guarantee) current compiler implementations 
sometimes
do not, though they violate no DWARF constraint.

I think there is value in assuring that a section, on its own, is 
complete and consistent
with DWARF rules.

Implementors do (now and always) have to make sure the 'garbage bytes' 
are not
referenced from elsewhere, but that is a fairly weak guarantee.

I am requesting that in any new sections we not allow garbage to sneak in.
There is little reason to allow it.      Well-formed but unused data
in a given section would, in my view, be no problem at all, just a minor
Quality Of Implementation issue.

I see no reason to rework existing sections, I am merely requesting
that new sections be slightly more constrained.  When that is
reasonable to do.

As in .debug_macro.    My earlier comments addressed this issue
in ISSUE 3 of the previous email, but that did not offer any
of my reasoning as to 'why', hence this email.

DavidA.

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Don Knuth




From ccoutant@gmail.com  Sat Jul 30 00:48:30 2011
From: ccoutant@gmail.com (Cary Coutant)
Date: Fri, 29 Jul 2011 17:48:30 -0700
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E3352E2.6030601@earthlink.net>
References: <4E33281D.2040600@earthlink.net> <4E3352E2.6030601@earthlink.net>
Message-ID: <CAJimCsH+gC_JoaM1bTn_mC06BLs6m9RuUR_vmNyUh_n-P2gkqQ@mail.gmail.com>

> I think there is value in assuring that a section, on its own, is complete
> and consistent
> with DWARF rules.
>
> Implementors do (now and always) have to make sure the 'garbage bytes' are
> not
> referenced from elsewhere, but that is a fairly weak guarantee.

On the other hand, I have reasons for wanting garbage bytes to sneak
in. I'm working on incremental linking support in the gold linker, and
it's very useful to be able to leave holes when doing an incremental
link: both for patch space for future use; and where an
incrementally-linked object contributes a slightly larger section than
it did originally, forcing the linker to go find space elsewhere and
try to reuse the now-unused space for something else that might be a
bit smaller. If everything else derives from .debug_info, then all I
have to worry about is .debug_info, and I can leave holes in the other
sections without concern for consumers trying to read those sections
independently.

-cary



From jakub@redhat.com  Sat Jul 30 08:01:35 2011
From: jakub@redhat.com (Jakub Jelinek)
Date: Sat, 30 Jul 2011 10:01:35 +0200
Subject: [Dwarf-Discuss] Vendor extensions in .debug_macinfo
In-Reply-To: <4E33281D.2040600@earthlink.net>
References: <4E33281D.2040600@earthlink.net>
Message-ID: <20110730080135.GP2687@tyan-ft48-01.lab.bos.redhat.com>

On Fri, Jul 29, 2011 at 02:37:33PM -0700, David Anderson wrote:
> ISSUE 1
> 
> In the new 6.3.1.5 text:
> 
> "Each so defined opcode is valid for subsequent entries
> until the terminating entry with type code 0, including any sequences
> included from those entries using DW_MACRO_transparent_include."

You are not looking here at the final text, which is at
http://gcc.gnu.org/ml/gcc-patches/2011-07/msg01949.html
and has been submitted as DWARF comment through the web interface.
DW_MACRO_define_opcode is gone.

> -----------------------------------------
> ISSUE 2
> 
> 6.3.5 Legacy macro information
> 
> 
> 
> 
> "just a series of macinfo entries.  There are 5 legacy macinfo opcode types:"
> 
> Remove the noise word "just", it adds nothing.

This section is gone too.

> ---------------------------------------------
> ISSUE 3
> 
> Final  comment: IMO we should be  careful to say
> that any unused bytes in the section are all zero.

At least one of our current readers (readelf) depends on it
and the producers produce it that way.  So I'm fine with standardizing
that.

	Jakub



