Issue 231110.1: DWARF Extension Registry

Author:	Ben Woodard
Champion:	Mark Wielaard
Date submitted:	2023-11-10
Date revised:	2024-01-08
Date closed:	2024-03-25
Type:	Enhancement
Status:	Replaced
DWARF Version:	6

Background

The DWARF standard has always had the wisdom to acknowledge the need for Vendor Extensibility. Section 1.3.13 describes this policy. For the producers it explicitly reserves some of the valid values for encoding various constructs and promises not to use those values in future versions of the standard. Consumers are given the freedom and expected to skip over data that they do not recognize.

The original intent of these vendor extensions was that they would be a private agreement between a particular producer and a cooperating consumer. However, the range of tools expanded beyond a toolchain provided by a single vendor and so the concept of a private agreement between a specific producer and a cooperating consumer came to be regarded as a registry of vendor extensions that every tool must be aware of for the sake of compatibility.

Because compatibility between tools was prioritized, the extensions in these de facto registries were effectively considered permanent unofficial DWARF encodings. This presumptive permanence is in spite of the fact that in some cases:

Producers that made use of these extensions were never implemented or publicly released.
Producers that made use of these extensions are so obsolete that not only does the tool chain no longer exist but the hardware that this toolchain was designed for no longer exists except as historical artifacts.
Official versions of DWARF encodings that accomplish the same thing have been added to the standard making the vendor extension obsolete.

The accumulation of these vendor extensions over the years has led to problems with some DWARF constructs where they are running out of available encodings that do not overlap with the ones used by or reserved by the standard. In particular the range of vendor
extension encoding is running particularly short in the range of DW_OP encoding. This has led to several informal proposals within the vendor community along the lines of an official proposal 230324.1 Expression Operation Vendor Extensibility Opcode. The basic idea in most of these proposals is to extend the range of available vendor extension encodings. While DWARF expression operations, Section 2.5, has the most extreme shortage of vendor encodings, other DWARF constructs are also running out of usable encodings.

Overview

This proposal is an alternative to 230324.1 and similar informal approaches to increase the available encoding space. It does this in two ways.

First, it makes it explicit that Vendor Extensions can only be interpreted in the context of a particular producer and a DWARF standard version. This way obsolete vendor extensions can be retired when a new version of the standard is released. It also begins the process of replacing the expectation for consumers that they can simply ignore encodings that they do not understand with the expectation they will need to have producer and version awareness when interpreting encodings. This is because tools chains are no longer developed by a close knit group of developers who all work for a single vendor. Tools are now developed by diverse distributed teams and users need and rightly expect interoperability.

Because the bulk of tool development is no longer done by vendors who build a vertically integrated tool chain but rather is done by looser confederation of projects that need to cooperate to achieve compatibility, the term “vendor” is replaced with more generic terms when referring to these extensions.

The second big change is it seeks to foster compatibility and collaboration by taking the collection of informal registries such as elfutils and llvm and collects them and brings them under the aegis of the DWARF standards body. To facilitate innovation and quickly meet producer’s and consumer’s needs the expectation is that registering a new producer specific construct and encoding would be a simple, light weight or even partially automated task. It would not be at all like amending the standard. The registry would be published on the DWARF standard’s web site so that consumers can easily find it. Each extension would be listed along with the producers that emit it and the range of DWARF standard versions that it applies to. When new DWARF standard versions are released, each extension will be reconsidered to see if it continues to add utility in the context of the new version of the DWARF standard. Extensions that persist over several versions of the DWARF standard probably should be considered for inclusion into the official standard.

Proposed Changes

Section 1.3.13

Section 1.3.13 currently named “Vendor Extensibility” shall be renamed to “Extensibility” to reflect the fact that cooperating tools using these extensions are no longer being developed by vendors who control the entire tool chain. Likewise the following paragraphs should change to:

This document does not attempt to cover all interesting languages or even to cover all of the possible debugging information needs for its primary target languages. Therefore, the document provides tool developers a way to define their own debugging information tags, attributes, base type encodings, location operations, language names, calling conventions and call frame instructions by reserving a subset of the valid values for these constructs for tool developer specific additions and defining related naming conventions. Producers may also use debugging information entries and attributes defined here in new situations. Future versions of this document will not use names or values reserved for these specific additions but the meanings of these encoding may change from DWARF version to version. Therefore the values of these extensions must be interpreted in the context of the DWARF standard versions for which they apply. All names and values not explicitly reserved for producer specific additions, however, are reserved for future versions of this document.

Where this specification provides a means for describing the source language, implementers are expected to adhere to that specification. For language features that are not supported, implementers may use existing attributes in novel ways or add producer-defined attributes.

Implementers who make extensions are strongly encouraged to design them to be compatible with this specification in the absence of those extensions. Implementers should also register them at <url> so that they can be included in the database of known extensions which can be found at <url>. This will allow consumers that they are not directly cooperating with to be aware of their extensions and know how to interpret them.

While the DWARF format is organized so that a consumer can skip over data which it does not recognize and this may allow a consumer to read and process files generated according to a later version of this standard or which contain producer specific extensions, albeit possibly in a degraded manner, this has been found to reduce the tool compatibility which users have come to expect. Consumers should therefore refer to <url> when encountering a producer specific construct that they are not familiar with.

Section 6.1.1.2 Structure of the Name Index

The paragraph:

A producer may define additional vendor-specific attributes, and a consumer will be able to ignore and skip over any attributes it is not prepared to handle.

is replaced by:

A producer may define additional producer-specific attributes, and a consumer should be able to ignore and skip over any attributes it is not prepared to handle. However, to achieve full compatibility a consumer should refer to <url> where all known producer specific attributes that apply to the Name Index are enumerated.

Section 6.2.4 The Line Number Program Header

In the definition of “opcode base”, the phrase “vendor-specific extensions” should be replaced with “producer-specific extensions” in all three places where it is used.

Section 6.2.4.2 Vendor-defined Content Descriptions

The name of the section should be changed to “Producer-defined Content Descriptions” and the first word of the first sentence should also be changed to match.

The non-normative text following the section should be changed to:

If a consumer encounters a producer-defined content type that it does not understand, it should skip the content data as though it were not present. However, to achieve full compatibility the consumer’s developer should refer to <url> where all known producer-defined line number content descriptors are enumerated.

Section 6.3.1 Macro Information Header

In the second paragraph of item 4 the phrase “Vendor extension entry opcodes” should be replaced with “Producer specific entry opcodes”.

Section 7.1 Vendor Extensibility

The name of the section should be changed from “Vendor Extensibility” to “Extensibility”.

Throughout section 7.1 the phrase “vendor specific” should be replaced with “producer specific”.

In the second paragraph, the sentence beginning with “Vendors may use values” should be replaced with “Producers may use values”.

The paragraph and list of points at the end of the section should be replaced with:

Producer defined tags, attributes, base type encodings, location atoms, language names, line number actions, calling conventions and call frame instructions, historically have used the form prefix_vendor_id_name, where vendor_id is some identifying character sequence chosen so as to avoid conflicts with other vendors. This established convention can continue in the cases where there is historical precedent or where there is a vendor but it can also be the name of the producer's project when there is no specific vendor.

To ensure that extensions added by one producer may be safely ignored by consumers that do not understand those extensions, the following rules must be followed:

Producer defined extensions must be evaluated in the context of a DWARF standard version.

New attributes are added in such a way that a debugger may recognize the format of a new attribute value without knowing the content of that attribute value.

The semantics of any new attributes do not alter the semantics of previously existing attributes.

The semantics of any new tags do not conflict with the semantics of previously existing tags.

New forms of attribute value are not added.

To ensure full tool intercompatibility, producer defined DWARF extensions should be registered with the DWARF standards body at <url>.

Section 7.32 Type Signature Computation

In the 3rd paragraph of point 4 replace “vendor-specific attributes” with “producer-specific attributes”.

Appendix A

In the first paragraph, replace “and new, vendor-defined ones” with “and producer defined ones”

Appendix D.7.3

In the text between the examples replace “vendor-specific” with “producer-specific” in both cases.

2024-01-08: Recommended splitting into 3 proposals: (1) Change "vendor" to "producer"; (2) Add the registry; (3) Make the namespace of extensions dependent on DWARF version.

2024-03-25: Replaced by 3 separate proposals: 231110.2, 231110.3, and 231110.4.