Proposal: Support for memory access and disassembly

Gregg Miskelly (@gregg-miskelly)

This proposal covers both disassembly and memory access, since they both depend on the same underlying primitive - a "memory reference".

Memory references

A memory reference is an opaque identifier that can be used (along with an optional byte offset) to identify a single location in memory. An adapter has the option of providing a memory reference on several protocol objects that represent things a frontend might want to display in a memory view or use as a starting point for disassembly:

export interface EvaluateResponse {
    // ...

    /**  Memory reference to an adapter-determined location appropriate for this result.  For pointer types, this is generally a reference to the memory address contained in the pointer. */
    memoryReference?: string;
}

export interface Variable {
    // ...

    /** Memory reference for the variable if the variable represents executable code, such as a function pointer. */
    memoryReference?: string;
}

export interface StackFrame {
    // ...

    /** Memory reference for the current instruction pointer in this frame. */
    instructionPointerReference?: string;
}

export interface GotoTarget {
    // ...

    /** Memory reference for the instruction pointer value represented by this target. */
    instructionPointerReference?: string;
}

Memory references are expressed as strings rather than numbers because the most common implementation will likely be to use the address directly, and a 64-bit memory address will not fit in a JSON number.

Reading memory

Reading memory from a location represented by a memory reference and offset is accomplished via a new readMemory request:

export interface Capabilities {
    // ...

    /** The debug adapter supports the 'readMemory' request. */
    supportsReadMemoryRequest?: boolean;
}

/** Reads bytes from memory at the provided location. */
export interface ReadMemoryRequest extends Request {
    // command: 'readMemory'
    arguments: ReadMemoryArguments;
}

/** Arguments for 'readMemory' request. */
export interface ReadMemoryArguments {
    /** Memory reference to the base location from which data should be read. */
    memoryReference: string;

    /** Optional offset (in bytes) to be applied to the reference location before reading data.  Can be negative. */
    offset?: number;

    /** Number of bytes to read at the specified location and offset. */
    count: number;
}

/** Response to 'readMemory' request. */
export interface ReadMemoryResponse extends Response {
    body: {
        /** The address of the first byte of data returned.  Treated as a hex value if prefixed with '0x', or as a decimal value otherwise. */
        address: string;

        /** The number of unreadable bytes encountered after the last successfully read byte.  This can be used to determine the number of bytes that must be skipped before a subsequent 'readMemory' request will succeed. */
        unreadableBytes?: number;

        /** The bytes read from memory, encoded using base64. **/
        data: string;
    }
}

Note that the read memory request should be considered to have succeeded even if none of the requested bytes are available. For instance, a read might extend into a page that is not mapped - in this case, the adapter would be expected to return the bytes it could read (if any), along with a value for unreadableBytes that can be used to skip the unmapped page.

Additionally, it is intended that a zero-byte read (that is, a readMemory request with a count of zero) could be used to determine the actual address identified by the memory reference and offset.

Disassembly

Disassembling code at a location represented by a memory reference is accomplished via a new disassemble request:

export interface Capabilities {
    // ...

    /** The debug adapters supports the 'disassemble' request. */
    supportsDisassembleRequest?: boolean;
}

/** Disassembles code stored at the provided location. */
export interface DisassembleRequest extends Request {
    // command: 'disassemble'
    arguments: DisassembleArguments;
}

/** Arguments for 'disassemble' request. */
export interface DisassembleArguments {
    /** Memory reference to the base location containing the instructions to disassemble. */
    memoryReference: string;

    /** Optional offset (in bytes) to be applied to the reference location before disassembling.  Can be negative. */
    offset?: number;

    /** Optional offset (in instructions) to be applied after the byte offset (if any) before disassembling.  Can be negative. */
    instructionOffset?: number;

    /** Number of instructions to disassemble starting at the specified location and offset.  An adapter must return exactly this number of instructions - any unavailable instructions should be replaced with an implementation-defined 'invalid instruction' value. */
    instructionCount: number;

    /** If true, the adapter should attempt to resolve memory addresses and other values to symbolic names. */
    resolveSymbols?: boolean;
}

/** Response to 'disassemble' request */
export interface DisassembleResponse extends Response {
    body: {
        /** The list of disassembled instructions. */
        instructions: DisassembledInstruction[];
    }
}

/** Represents a single disassembled instruction. */
export interface DisassembledInstruction {
    /** The address of the instruction.  Treated as a hex value if prefixed with '0x', or as a decimal value otherwise. */
    address: string;

    /** Raw bytes representing the instruction and its operands, in an implementation-defined format. */
    instructionBytes?: string;

    /** Text representing the instruction and its operands, in an implementation-defined format. */
    instruction: string;

    /** Name of the symbol that correponds with the location of this instruction, if any. */
    symbol?: string;

    /** Source location that coresponds to this instruction, if any.  Should always be set (if available) on the first instruction returned, but can be omitted afterwards if this instruction maps to the same source file as the previous instruction. */
    location?: Source;

    /** The line within the source location that corresponds to this instruction, if any. */
    line?: number;

    /** The column within the line that corresponds to this instruction, if any. */
    column?: number;

    /** The end line of the range that corresponds to this instruction, if any. */
    endLine?: number;

    /** The end column of the range that corresponds to this instruction, if any. */
    endColumn?: number;
}

The location, line, column, endLine and endColumn members on DisassembledInstruction are intended to provide a mapping between a disassembled instruction and the source code it was generated from, if this information is available to the adapter.

Formatting

A debugger UI that implements this proposal will likely want to display addresses differently for 32-bit vs 64-bit processes. This information would be conveyed via an additional property on the process event:

export interface ProcessEvent extends Event {
    // ...

    /** The size of a pointer or address for this process, in bits. */
    pointerLength?: number;
}

CC: Gregg Miskelly (@gregg-miskelly)

I made a slight tweak to the proposal based on my experience implementing this in vsdbg:

Added an offset parameter to the disassemble request
Updated some of the comments to better describe the expected behavior of an adapter

Andrew Crawley (@andrewcrawley)

Andrew Crawley (@andrewcrawley) Yes, memoryReference has to be a string. But wondering if the proposal should say what the string contents should be for parsing/formatting purposes. For instance a decimal or with an 0x prefixed hex number or either?

Nit. Should ReadMemoryResponse.body.address be named ReadMemoryResponse.body.memoryReference just to indicate it follows the same rules as other memoryReference? Same thing with DisassembledInstruction.address Perhaps your intent was different?

Andrew Crawley (@andrewcrawley)

Andrew Crawley (@andrewcrawley) I hope I am not offending you with my opinions. Your knowledge probably dwarfs mine

I was wondering if the DisassembleResponse would become too heavy for transport. I am imagining stupidly large functions (generally auto-generated) or entire objects (elf-sections/obj/exe/dll/so). One reason to show more than a function is to see instructions as well as const data packed around them.

Please visit -data-disassemble section in this link

I am not saying gdb is the gold standard. But, it kinda makes sense. Especially with -a option. The debugger was paused, I want to get the whole context around that instruction with the DisassembleRequest API.

If we just follow the gdb MI interface, it is a lot less burdensome for the MIEngine (mostly pass it on), less burdensome on the client to figure out what kind of request to make. Let the clients deal with the rest -- presentation, debug-interface, etc.

What you are proposing is an interpreted and decoded super nice interface. What I was thinking was a mostly raw interface taking advantage of what the debugger already provides.

Once we feel comfortable with the proposals, I can prototype some stuff.

Affordable only if you have either in-line interpreter or incremental compiler. There are only few exist
This is a kind of rare server capability.

As a client implementer, I would say this API proposal is very sensible.

The only bit that would be fiddly is this:

Should always be set (if available) on the first instruction returned, but can be omitted afterwards

This makes handling stateful. I think we should either:

require it is always sent, or
push it up a message level and require all instructions to be supplied file-wise

My thinking was that all the location info should be in its own bag and it is either there or not there, ie not required

The purpose of such protocols is decoupling UI from debugger service implementation. This is achieved by LSP. This proposal gives some hint how memory reference can be expressed by UI. Disassembly is generally impossible for languages with bytecode runtime and jit compilers. like C#, java and javascript. They are 90% of world today. Imagine basic Cloud use case: Debugger UI running in browser as a part of IDE uses DAP to communicate with Debugger service running as Cloud container based on DotNet runtime. The source code is loaded by demand from the user's workspace via workspace Rest API.

Pavel Sosin (@PavelSosin)

Pavel Sosin (@PavelSosin) I don't even understand your objections. This proposal does not affect those languages or their debuggers or even add any overhead to you or your UI.

They are 90% of world today.

Even if that were true, those in the 10% build the platforms to run the 90% on. And they require good debuggers.

Transfer of few kilobytes of disassembly result to the client side is not a serious boundary condition. The time can be estimated as download of text file of the same size from good site deployed to good cloud infrastructure. Literally, every developer today can watch Youtube and never use modem connection. Since DAP protocol is built on top JsonRPC over WebSocket thereof it is non-blocking protocol the 99% of time will be spent by dissasembler and not more than 1% of UI latency will be spent by communication and rendering. We never had issues related to communication speed in our LSP adoption projects.

To avoid UI freeze the response on memory access and disassembly request should not contain data but only primary result of memory access or disassembly request. Upon receiving positive response UI displays control to expose data which will be received in the next step. Next step is getXXXData request. The data got as response of this request fills pre-opened control with result. The notifications sent from server to client informs Debugger UI about progress and any issues detected during disassembly. If disassembly request sent from UI to server contains data which prevents disassembly server responses with error response and UI displays error message instead of opening control.
Another option is displaying disassembly result in-line but it will require integration of debbuger server with editor back-end framework. LSP Code completion cycle is a good example of such 2-step conversation.

It looks like I share my conclusions with https://kichwacoders.com/2017/08/02/gdbs-mi-is-not-a-debug-protocol/

Andrew Crawley (@andrewcrawley)

Andrew Crawley (@andrewcrawley) I've added more to the "first batch of feedback" above. Sorry for making you jump back an forth through the comments...

Andre Weinand (@weinand)

Andre Weinand (@weinand): Don't worry about it : )

Disassembly is actually surprisingly complicated, as I've been learning over the past few weeks while implementing this proposal in vsdbg. It has to be handled by the DA because only the actual debugger knows the CPU architecture of the debugged process (which may be different from the architecture of the host in remote debugging cases), has access to the necessary info from debug symbols, ability to map source lines to a memory location in jitted code, etc.

I don't think a dynamic document model would work for disassembly. For one thing, the usual model is to start disassembling at some arbitrary address (current IP, function entry point, arbitrary source line via gotoTargets request, etc), and then as the user scrolls around, the UI requests additional instructions as necessary. The current dynamic document model in the protocol assumes that documents are either static or only change by replacing the entire document, so it would require a lot of modification to work with "lazy" content. In addition, the VS disassembly window is also configurable in terms of formatting, similar to the memory view window. A basic UI would just have to concatenate together the address and instruction fields from DisassembledInstruction, which doesn't seem onerous.

Setting BPs in disassembly is not in scope at the moment, but we'll probably want to do it eventually. I think the best option here would be to add an AddressBreakpoint (or something similar) that you set via an address - this is how VS handles it. Supporting BPs via dynamic documents (assuming the protocol related to dynamic documents was extended to support virtualized scrolling) would require the DA to maintain an internal mapping between "dynamic document line number" -> DisassembledInstruction, since the line numbers would be changing as more content was added, which seems like a lot of complexity and overhead compared to just using on the address field already present in a DiassembledInstruction.

Yup, lazy reading/rendering for disassembly is how it needs to be done. I would do that even for memory windows. In fact, I would do that everywhere, if it were up to me. Super large stack. super large array for a variable. In the case of stack/variables, I had a Show Next 100 and/or Show All UI element with a hint of how much more and let the user decide. For memory/disassembly, I did it based on the scrolling. Those concepts were baked into my frontend and backend.

My past has no relevance here...

Even in the current DA MIEngine, don't we just request a limited # (like 1000) of frames of a stack?

Andrew Crawley (@andrewcrawley)

Andrew Crawley (@andrewcrawley) thanks for helping me understand disassembly and addressing my concerns.
Overall the proposal looks good to me.
I will create a PR with the schema changes (including the client capability in the Initialize request).

Andre Weinand (@weinand)

Andre Weinand (@weinand) Andrew has a json schema version of all these proposals. So he can comment more when he gets to work, but it might be easier for him to submit a PR if you would like.

Recently I learned that disassembly feature is heavily assembler implementation related. In the real world we have 2 assembler languages for Intel and yet another one for ARM. They are really different programming languages. I propose to keep reflection of this feature the in protocol at highest level. For LSP we separated universal LSP from other services used in editor, like coloring and templating, which have separate APIs due to heavy language dependencies. These services share only the triggering command. Maybe it is more convenient for DA implementor to use more generic protocol and for developer to have language specific disassembler UI.

We should not confuse assembly languages from disassembly. There are two main assemblers for ARM (ARM and GNU) and Intel (Microsoft MASM, GNU), but there are others. But that is input. Disassembly has two main flavors people are used to. AT&T and Intel. Some debuggers allow you to choose the flavor. Not sure if the DA should allow that or stick to just one flavor or just do a pass through.

I don't see disassembly output as a language, Mnemonics, macros, labels are all normalized, but it may still look like original assembler input.

Also helpful to read this link

The memory and register access is not simpler case because Intel architecture uses 4 different memory models and ARM architecture provides yet another one, very different from Intel. The only common thing is that all low-level debugger APIs implement evaluate expression command which accept expression as a string.The duty of DA is to bridge between UI and low-level debugger.
interface EvaluateArguments used in EvaluateExpression request already has scope and format attributes.
I propose to extend EvaluateEArguments with single attribute qualifier defined in the following way:
qualifier: string

// qualifier is how the expression is interpreted by server and
// helps debugger server to interpret expression and translate it into low-level protocol call
// according context of specific assembly language and target architecture.
Up to server to translate this request into low-level debugging protocol or API.
If translation is not possible server responses with negative response
Values:
nill : If not specified or null the regular context language expression
register: //used to read register value. it is assumed
expression attribute value is a register name
"memory": //used to read memory content.
expression attribute value is symbolic memory reference which is translated by server into memory referenced used by low-level protocol according in target architecture and target assembler language,e.g.
//Intel debugger can pass different forms of expressions: MASM expressions or C++ for corresponding assembler languages
// Arm debugger uses memory reference expression according Memory model of ARM processors
Grammar validity f memory expression is checked by regular on-line validation mechanism provided by Hosted editor UI control for corresponding language.
Remark1: register and memory can refer to either physical registers and memory or simulator registers and memory
Remark2 : Up to low-level API and architecture to prevent access to memory outside application virtual memory.
Remark3 : If context is hover the qualifier is derived from source token type. Hosted by UI source editor control provides this information out of-the-box because displayed source is already colored.

Add memory access and disassembly support; fixes #42

Andrew Crawley (@andrewcrawley)

Andrew Crawley (@andrewcrawley) Gregg Miskelly (@gregg-miskelly) I've created PR #50 for the complete DAP change.

I strongly oppose any proposal to project some low-level debuggers features to Core Debugging Protocol.
Low-level debuggers are functionally incomplete and inconsistent. Windows 10 already killed last "local" scenario via introduction of virtualization feature. Everybody can enable hardware virtualization feature, Open Hyper-V manager, create few Virtual Machines, install few Ubuntu iso images and connect to these VM using built-in OpenSSH client or Chrome extension.
I suppose nobody tested dbg in this scenario but I'm not sure that it can work.
Also dbg semantics is not clear: memory of which VM ? maybe Host OS memory? Registers of which core?
Unlike clear expressions context in all high-level debuggers - VM and Thread, Memory and register semantic of dbg is not clear for me. Evaluate expression has universal meaning. Registers and memory access operations are applicable only in extreme scenario: Low-level programming of single core machine without hardware virtualization. The dbg definition of register and memory operations lack context. Manufacturing of such architecture has been terminated.
I also strongly oppose usage the laptop resource in 25% of its capability after 100% of price has been payed.

Navigation Menu

Repository navigation

Proposal: Support for memory access and disassembly #42

Memory references

Reading memory

Disassembly

Formatting

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: Support for memory access and disassembly #42

Description

Memory references

Reading memory

Disassembly

Formatting

Activity

andrewcrawley commented on May 17, 2019

haneefdm commented on May 18, 2019

haneefdm commented on May 19, 2019

PavelSosin commented on May 19, 2019

puremourning commented on May 19, 2019

haneefdm commented on May 19, 2019

PavelSosin commented on May 19, 2019

haneefdm commented on May 19, 2019

puremourning commented on May 19, 2019

PavelSosin commented on May 20, 2019

PavelSosin commented on May 20, 2019

PavelSosin commented on May 20, 2019

23 remaining items

weinand commented on May 23, 2019

andrewcrawley commented on May 23, 2019

haneefdm commented on May 23, 2019

weinand commented on May 24, 2019

gregg-miskelly commented on May 24, 2019

PavelSosin commented on May 26, 2019

haneefdm commented on May 26, 2019

PavelSosin commented on May 26, 2019

weinand commented on May 27, 2019

PavelSosin commented on May 29, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions