From scott  Wed Jan 31 22:05:03 2018
From: scott (scott@scottlinder.com)
Date: Wed, 31 Jan 2018 14:05:03 -0800
Subject: [Dwarf-Discuss] DWARF and source text embedding
Message-ID: <a4b65934f05e313da71d222de02cd6ae@scottlinder.com>

Hello all,

I am a compiler engineer at AMD, working on tools for debugging 
online-compiled
programs. The problem I am attempting to solve was brought up previously 
in the
DWARF Standard issue 161018.1 titled "DWARF-embedded source for 
online-compiled
programs", and is the result of runtimes like OpenCL doing online 
compilation
in an environment where it is not desireable (or even feasible) to write
sources to disk. In these cases, it would be useful to support embedding 
the
source directly in the resulting DWARF. I would like to propose a 
similar
solution to the one outlined in the above issue, but without structural 
changes
to the specification.

====

Add two new optional fields to the file_names prologue of the line 
table.

Section 6.2.4.1:
Add two bullets after "5. DW_LNCT_MD5"
6. DW_LNCT_has_source
     DW_LNCT_has_source indicates that the value is a boolean which 
affects the
     interpretation of an accompanying DW_LNCT_source value. When present 
there
     must be an accompanying DW_LNCT_source value. When true, consumers 
may use
     the embedded source instead of attempting to discover the source on 
disk.
     When false, consumers will ignore the DW_LNCT_source value. This 
code point
     is always paired with a flag form (e.g. DW_FORM_flag or
     DW_FORM_flag_present).
7. DW_LNCT_source
     DW_LNCT_source indicates that the value is a null-terminated string 
which
     is the original source text of the file. When present there must be 
an
     accompanying DW_LNCT_has_source value. The string will contain the 
UTF-8
     encoded source text with '\n' line endings. When the accompanying
     DW_LNCT_has_source value is false, the value of DW_LNCT_source will 
be the
     empty string. This code point is always paired with a string form 
(e.g.
     DW_FORM_string, DW_FORM_line_strp, DW_FORM_strp).

New type codes can be allocated for them in a backwards-compatible way, 
or
codes for these new content types can be added in the range of
[DW_LNCT_lo_user, DW_LNCT_hi_user] to avoid changing the spec itself.

Table 7.27:
Add DW_LNCT_has_source  0x6
Add DW_LNCT_source      0x7

Any DWARFv5 consumer which is unaware of this extension would continue 
to
operate as before, ignoring the new fields. Any consumer which is aware 
of the
extension would know to check DW_LNCT_has_source for each file_name 
entry in
order to determine whether the embedded source field (DW_LNCT_source) 
contains
the source text of the corresponding file.

====

My team and I believe this simplifies the design by removing the need 
for
changes to the compile unit sections, and by avoiding the addition of 
multiple
file_name_entry_formats in a single program, all without sacrificing any
information. We have a preliminary implementation in LLVM/Clang, which 
supports
embedding source (clang -gdwarf-5 -gembed-source) and inspecting it via
llvm-dwarfdump and llvm-objdump (with the -source flag). The patches are
available at https://reviews.llvm.org/D42765 (LLVM) and
https://reviews.llvm.org/D42766 (Clang).

I would like any and all feedback on the design, and want to see about 
the
possibility of adding the new content type codes outside of the "user" 
range
(i.e. adding new entries for them in Table 7.27) in the next version of 
the
specification.

Regards,
Scott Linder



From eager@eagerm.com  Wed Jan 31 22:33:37 2018
From: eager@eagerm.com (Michael Eager)
Date: Wed, 31 Jan 2018 14:33:37 -0800
Subject: [Dwarf-Discuss] DWARF and source text embedding
In-Reply-To: <a4b65934f05e313da71d222de02cd6ae@scottlinder.com>
References: <a4b65934f05e313da71d222de02cd6ae@scottlinder.com>
Message-ID: <a597bbee-1e22-cec9-dd29-ddbb97c88351@eagerm.com>

Hi Scott --

Please submit your proposal at http://dwarfstd.org/Comment.php.



On 01/31/2018 02:05 PM, scott at scottlinder.com wrote:
> Hello all,
> 
> I am a compiler engineer at AMD, working on tools for debugging 
> online-compiled
> programs. The problem I am attempting to solve was brought up previously 
> in the
> DWARF Standard issue 161018.1 titled "DWARF-embedded source for 
> online-compiled
> programs", and is the result of runtimes like OpenCL doing online 
> compilation
> in an environment where it is not desireable (or even feasible) to write
> sources to disk. In these cases, it would be useful to support embedding 
> the
> source directly in the resulting DWARF. I would like to propose a similar
> solution to the one outlined in the above issue, but without structural 
> changes
> to the specification.
> 
> ====
> 
> Add two new optional fields to the file_names prologue of the line table.
> 
> Section 6.2.4.1:
> Add two bullets after "5. DW_LNCT_MD5"
> 6. DW_LNCT_has_source
>  ??? DW_LNCT_has_source indicates that the value is a boolean which 
> affects the
>  ??? interpretation of an accompanying DW_LNCT_source value. When 
> present there
>  ??? must be an accompanying DW_LNCT_source value. When true, consumers 
> may use
>  ??? the embedded source instead of attempting to discover the source on 
> disk.
>  ??? When false, consumers will ignore the DW_LNCT_source value. This 
> code point
>  ??? is always paired with a flag form (e.g. DW_FORM_flag or
>  ??? DW_FORM_flag_present).
> 7. DW_LNCT_source
>  ??? DW_LNCT_source indicates that the value is a null-terminated string 
> which
>  ??? is the original source text of the file. When present there must be an
>  ??? accompanying DW_LNCT_has_source value. The string will contain the 
> UTF-8
>  ??? encoded source text with '\n' line endings. When the accompanying
>  ??? DW_LNCT_has_source value is false, the value of DW_LNCT_source will 
> be the
>  ??? empty string. This code point is always paired with a string form 
> (e.g.
>  ??? DW_FORM_string, DW_FORM_line_strp, DW_FORM_strp).
> 
> New type codes can be allocated for them in a backwards-compatible way, or
> codes for these new content types can be added in the range of
> [DW_LNCT_lo_user, DW_LNCT_hi_user] to avoid changing the spec itself.
> 
> Table 7.27:
> Add DW_LNCT_has_source? 0x6
> Add DW_LNCT_source????? 0x7
> 
> Any DWARFv5 consumer which is unaware of this extension would continue to
> operate as before, ignoring the new fields. Any consumer which is aware 
> of the
> extension would know to check DW_LNCT_has_source for each file_name 
> entry in
> order to determine whether the embedded source field (DW_LNCT_source) 
> contains
> the source text of the corresponding file.
> 
> ====
> 
> My team and I believe this simplifies the design by removing the need for
> changes to the compile unit sections, and by avoiding the addition of 
> multiple
> file_name_entry_formats in a single program, all without sacrificing any
> information. We have a preliminary implementation in LLVM/Clang, which 
> supports
> embedding source (clang -gdwarf-5 -gembed-source) and inspecting it via
> llvm-dwarfdump and llvm-objdump (with the -source flag). The patches are
> available at https://reviews.llvm.org/D42765 (LLVM) and
> https://reviews.llvm.org/D42766 (Clang).
> 
> I would like any and all feedback on the design, and want to see about the
> possibility of adding the new content type codes outside of the "user" 
> range
> (i.e. adding new entries for them in Table 7.27) in the next version of the
> specification.
> 
> Regards,
> Scott Linder
> 
> _______________________________________________
> Dwarf-Discuss mailing list
> Dwarf-Discuss at lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
> 

-- 
Michael Eager    eager at eagerm.com
1960 Park Blvd., Palo Alto, CA 94306


From John.DelSignore@roguewave.com  Wed Jan 31 22:40:53 2018
From: John.DelSignore@roguewave.com (John DelSignore)
Date: Wed, 31 Jan 2018 17:40:53 -0500
Subject: [Dwarf-Discuss] DWARF and source text embedding
In-Reply-To: <a4b65934f05e313da71d222de02cd6ae@scottlinder.com>
References: <a4b65934f05e313da71d222de02cd6ae@scottlinder.com>
Message-ID: <ed39d89f-6e7c-6bb8-e11a-3ccfbd5919d7@roguewave.com>

Hi Scott,

Question: What does the DW_TAG_compile_unit look like for an embedded source file? For example, what does the DW_AT_name and DW_AT_comp_dir look like?

Cheers, John D.


On 01/31/18 17:05, scott at scottlinder.com wrote:
> Hello all,
>
> I am a compiler engineer at AMD, working on tools for debugging online-compiled
> programs. The problem I am attempting to solve was brought up previously in the
> DWARF Standard issue 161018.1 titled "DWARF-embedded source for online-compiled
> programs", and is the result of runtimes like OpenCL doing online compilation
> in an environment where it is not desireable (or even feasible) to write
> sources to disk. In these cases, it would be useful to support embedding the
> source directly in the resulting DWARF. I would like to propose a similar
> solution to the one outlined in the above issue, but without structural changes
> to the specification.
>
> ====
>
> Add two new optional fields to the file_names prologue of the line table.
>
> Section 6.2.4.1:
> Add two bullets after "5. DW_LNCT_MD5"
> 6. DW_LNCT_has_source
>     DW_LNCT_has_source indicates that the value is a boolean which affects the
>     interpretation of an accompanying DW_LNCT_source value. When present there
>     must be an accompanying DW_LNCT_source value. When true, consumers may use
>     the embedded source instead of attempting to discover the source on disk.
>     When false, consumers will ignore the DW_LNCT_source value. This code point
>     is always paired with a flag form (e.g. DW_FORM_flag or
>     DW_FORM_flag_present).
> 7. DW_LNCT_source
>     DW_LNCT_source indicates that the value is a null-terminated string which
>     is the original source text of the file. When present there must be an
>     accompanying DW_LNCT_has_source value. The string will contain the UTF-8
>     encoded source text with '\n' line endings. When the accompanying
>     DW_LNCT_has_source value is false, the value of DW_LNCT_source will be the
>     empty string. This code point is always paired with a string form (e.g.
>     DW_FORM_string, DW_FORM_line_strp, DW_FORM_strp).
>
> New type codes can be allocated for them in a backwards-compatible way, or
> codes for these new content types can be added in the range of
> [DW_LNCT_lo_user, DW_LNCT_hi_user] to avoid changing the spec itself.
>
> Table 7.27:
> Add DW_LNCT_has_source  0x6
> Add DW_LNCT_source      0x7
>
> Any DWARFv5 consumer which is unaware of this extension would continue to
> operate as before, ignoring the new fields. Any consumer which is aware of the
> extension would know to check DW_LNCT_has_source for each file_name entry in
> order to determine whether the embedded source field (DW_LNCT_source) contains
> the source text of the corresponding file.
>
> ====
>
> My team and I believe this simplifies the design by removing the need for
> changes to the compile unit sections, and by avoiding the addition of multiple
> file_name_entry_formats in a single program, all without sacrificing any
> information. We have a preliminary implementation in LLVM/Clang, which supports
> embedding source (clang -gdwarf-5 -gembed-source) and inspecting it via
> llvm-dwarfdump and llvm-objdump (with the -source flag). The patches are
> available at https://reviews.llvm.org/D42765 (LLVM) and
> https://reviews.llvm.org/D42766 (Clang).
>
> I would like any and all feedback on the design, and want to see about the
> possibility of adding the new content type codes outside of the "user" range
> (i.e. adding new entries for them in Table 7.27) in the next version of the
> specification.
>
> Regards,
> Scott Linder
>
> _______________________________________________
> Dwarf-Discuss mailing list
> Dwarf-Discuss at lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
>



