Andb

Noslate Debugger is a v8 offline debugging tool for gdb/lldb, generally used to debug v8 core files.

The following will use andb to refer to this project, which is the abbreviation of Alibaba Noslate Debugger.

The mainly ability

1) C++ debugging ability

Official node/alinode/aworker are all released as Release versions, which cannot be debugged without debugging information. andb completes non-position-related debugging information through the "typ" file, and provides C++ debugging capabilities, such as structure display.

2) Check v8 heap objects (d8)

d8 or node_g can provide %DebugPrint to inspect the details of the object, andb provides the ability to %DebugPrint on Release builds without requiring the app to recompile the Debug build.

3) Analysis Time Capability (Heap Snapshot)

Heap Snapshot is the main means of locating the OOM problem. The acquisition of traditional Heap Snapshot depends on the high runtime overhead. With andb, you can export Heap Snapshot offline from the core file.

4) Automatic matching of debugging information

Generate a core file through Arthur or kernel, which contains the BuildId information of the node binary, andb can automatically match the version of node and alinode official Release and the supporting debugging information file from the database. For the core file generated by gcore, the BuildId information is not saved, and it needs to be manually matched and debugged.

Supported debug environments

Linux + gdb 10.0+
Linux + lldb 8.0+
MacOS + lldb 11+ (catelina, bigsur, monterey)
MacOS + gdb 10.0+ (andb-gdb)

Currently only x86_64 core file analysis is supported.

Usage

1) Clone the andb project 2) Use env.sh to set environment variables 3) Use the "andb" command to start debugging

# Set environment variables, alias andb
git clone git@github.com:noslate-project/debugger.git
cd andb
source env.sh

For core files generated by Arthur or kernel, you can select gdb/lldb to start offline debugging.

# Use lldb to start debug
andb -l -c core

# or use gdb to start debug
andb -g -c core

For the core file generated by gcore, you need to prepare binary, supporting node.typ file, Use lldb to start debugging, andb will load the node binary and the node.typ file in the current directory

andb -l node -c core
# or
andb -g node -c core

Isolate location

andb currently provides two built-in methods to quickly find Isolate, First, through the control field of the page Second, search through the isolate pointer retained on the current stack

# find isolate by page
(gdb) isolate guess page
(class v8::internal::Isolate *) $isolate = 0x416ec60

# find isolate by stack
(gdb) isolate guess stack
(class v8::internal::Isolate *) $isolate = 0x416ec60

C++ debug

andb associates important objects through Convenience variables, which are prefixed with "\$" for such debugger variables. For example, after v8::internal::Isolate is located in isolate, you can use "\$isolate" to access the structure content, similar to "\$node" and so on.

(gdb) p $isolate->heap_.old_space_
$1 = (class v8::internal::OldSpace *) 0x421de80

(gdb) p $isolate->heap_.gc_state_
$2 = v8::internal::Heap::NOT_IN_GC

Command line

andb provides a unified command line through the debugger's command line interface.

The command line is composed of "prefix + subcommand + command parameter", different prefixes,

isolate : isolate the core structure command
heap : v8 heap related commands
v8 : v8 related debug commands
node : node related debugging commands

Command line completion can be provided under gdb, but due to the implementation of lldb, automatic completion cannot be provided. Subcommands can be obtained using "?".

(gdb) heap ?
snapshot page dump space
(gdb) iso ?
guess

In the absence of conflict, command abbreviations can be used, such as

# gdb
(gdb) iso g p
(class v8::internal::Isolate *) $isolate = 0x416ec60
(gdb) iso g s
(class v8::internal::Isolate *) $isolate = 0x416ec60

# lldb
(lldb) iso g p
(v8::internal::Isolate *) $isolate = 0x416ec60L
(lldb) iso g s
(v8::internal::Isolate *) $isolate = 0x416ec60L

heap space [new|old|code|lo|...]

Display a summary of each Space in the v8 heap,

(gdb) heap space
SPACE NAME          COMMIT        MAX
RO_SPACE      :     151552     262144
MAP_SPACE     :   22548480   38014976
CODE_SPACE    :    8749056   12681216
CODE_LO_SPACE :      49152      49152
OLD_SPACE     :  328626176  883847168
LO_SPACE      :  707457024 1462448128
NEW_LO_SPACE  :     299008  418394112
NEW_SPACE     :   33554432
 - from_space :   16777216
 - to_space   :   16777216
Total Committed 1101434880

Show all pages in the specified spacepages

(gdb) heap spa map
 0x30b45bd80000 : size(4096), sweep(1), start(0x30b45bd80120)
 0x2da021080000 : size(262144), sweep(1), start(0x2da021080120)
 ...
 0x1b609b100000 : size(262144), sweep(0), start(0x1b609b100120)
 0x3edd93fc0000 : size(262144), sweep(0), start(0x3edd93fc0120)
Total 32 pages.
(gdb)

heap page \<page_address>

Print in-page objects,

(gdb) heap page 0x3edd93fc0000
...
0x3edd93ffff30 : size(72), mapsize(72), MAP_TYPE
0x3edd93ffff78 : size(72), mapsize(72), MAP_TYPE
0x3edd93ffffc0 : size(64), mapsize(0), FREE_SPACE_TYPE

heap summary [old|code|lo|...]

The digest specifies the number of objects on the Space and the number of bytes occupied,

(gdb) heap sum old
...
0x3490b88c58c1:    42991      1375712 LOAD_HANDLER_TYPE
0x3490b88c0891:    43315      5587496 SCOPE_INFO_TYPE
0x3490b88c11e1:    45508     19311584 NAME_DICTIONARY_TYPE
0x14295ec40439:    47647      2668232 JS_OBJECT_TYPE
0x1f9e87f00439:    48833      3515976 JS_OBJECT_TYPE
0x3490b88c1bb9:    48926      1402440 INTERNALIZED_STRING_TYPE
0x3490b88c1781:    49847      1196328 UNCOMPILED_DATA_WITHOUT_PREPARSE_DATA_TYPE
0x3490b88c0241:    54119      7636728 DESCRIPTOR_ARRAY_TYPE
0x3490b88c02e1:    57338       458704 FILLER_TYPE
0x3490b88c12b9:    58272      1398528 FEEDBACK_CELL_TYPE
0x3490b88c1931:    58568     69492736 STRING_TYPE
0x14295ec403f1:    60238      3373328 JS_FUNCTION_TYPE
0x3490b88c08d9:    63642      3563952 SHARED_FUNCTION_INFO_TYPE
0x3490b88c04f9:    88135      1410160 HEAP_NUMBER_TYPE
0x3490b88c1979:    91588      2930816 CONS_ONE_BYTE_STRING_TYPE
0x3490b88c1421:    94980      4937800 PROPERTY_ARRAY_TYPE
0x14295ec40ca9:   131817      4218144 JS_ARRAY_TYPE
0x14295ec40751:   132339      8469696 JS_FUNCTION_TYPE
0x14295ec403a9:   171768     10993152 JS_FUNCTION_TYPE
0x14295ec40c61:   175716     10745528 FUNCTION_CONTEXT_TYPE
0x3490b88c0729:   176399     20811432 FIXED_ARRAY_TYPE
0x3490b88c0409:   184370     14378824 ONE_BYTE_INTERNALIZED_STRING_TYPE
0x3490b88c0849:   643297     52945368 ONE_BYTE_STRING_TYPE
ShowMapSummary() takes 166.896 second(s).
(gdb)

heap find [old|lo|...] \<tag>

Finds objects referencing tag on the specified heap.

(gdb) heap find old 0xda2d6b4f4e9
<FixedArray 0x1449d494b869>
<FixedArray 0x1449d494c9c9>
find 2
(gdb) heap find old 0x1449d494b869
<JsArray 0x1449d494f1f9>
find 1

heap snapshot

Export core.heapsnapshot file from core,

(gdb) heap snap
Synchronize: (Strong roots)
Synchronize: (Bootstrapper)
Synchronize: (Relocatable)
Synchronize: (Debugger)
Synchronize: (Compilation cache)
Synchronize: (Builtins)
Synchronize: (Thread manager)
Synchronize: (Global handles)
Synchronize: (Stack roots)
Synchronize: (Handle scope)
Synchronize: (Eternal handles)
Synchronize: (Startup object cache)
Synchronize: (Internalized strings)
Synchronize: (External strings)
Iterated 0 RO Heap Objects
failed RO Heap Object: 0
(v8::internal::AllocationSpace) v8::internal::RO_SPACE
(v8::internal::AllocationSpace) v8::internal::MAP_SPACE
(v8::internal::AllocationSpace) v8::internal::CODE_SPACE
(v8::internal::AllocationSpace) v8::internal::CODE_LO_SPACE
(v8::internal::AllocationSpace) v8::internal::OLD_SPACE
18.6%: 999.9/sec, Object(10000), Entry(42830), Edge(119869)
(v8::internal::AllocationSpace) v8::internal::LO_SPACE
(v8::internal::AllocationSpace) v8::internal::NEW_LO_SPACE
(v8::internal::AllocationSpace) v8::internal::NEW_SPACE
Iterated 46355 Objects
failed HeapObject: 0
heap snapshot written to 'core.heapsnapshot'
Generate() takes 14.012 second(s).
(gdb)

Then use devtools to open the core.heapsnapshot file.

v8 inspect \<tag>

The v8 inspect command is similar to %DebugPrint for printing detailed information about an object. The input of v8 inspect is a \<tag>, if it is a non-HeapObject tag, it will be displayed as Smi.

(gdb) v8 i 0xda2d6b4f4e9
[JSObject 0xda2d6b4f4e9]
- Properties:
 - originalColumn: <Smi 6 0x600000000>
 - name: <Oddball 0x3490b88c01b1>
 - generatedColumn: <Smi 161 0xa100000000>
 - source: <Smi 0 0x0>
 - originalLine: <Smi 81 0x5100000000>
 - generatedLine: <Smi 43 0x2b00000000>
- Elements: []
[Map 0x1f9e87f00439]
- InstanceSizeInWords: 9
- InobjectPropertiesStartOrConstructorFunctionIndex: 3
- UsedOrUnusedInstanceSizeInWords: 9
- VisitorId: 25
- InstanceType: JS_OBJECT_TYPE (1057)
- BitField: 0x0
[MapBitFields1 0x0]
- HasNonInstancePrototype: 0
- IsCallable: 0
- HasNamedInterceptor: 0
- HasIndexedInterceptor: 0
- IsUndetectable: 0
- IsAccessCheckNeeded: 0
- IsConstructor: 0
- HasPrototypeSlot: 0
- BitField2: 0x19
[MapBitFields2 0x19]
- NewTargetIsBase: 1
- IsImmutablePrototype: 0
- Unused: 0
- ElementsKind: HOLEY_ELEMENTS (3)
- BitField3: 0x8c01bff
[MapBitFields3 0x8c01bff]
- EnumLength: 1023
- NumberOfOwnDescriptors: 6
- IsPrototypeMap: 0
- IsDictionaryMap: 0
- OwnsDescriptors: 1
- IsInRetainedMapList: 1
- IsDeprecated: 0
- IsUnstable: 0
- IsMigrationTarget: 0
- IsExtensible: 1
- MayHaveIntrestingSymbols: 0
- ConstructionCounter: 0
- Prototype: <JsObject 0x2beeb7f52579>
- ConstructorOrBackPointerOrNativeContext: <Map 0x1f9e87f003f1>
- InstanceDescriptors: <DescriptorArray 0x2beeb7f525b1>
- DependentCode: <WeakFixedArray 0x21f08217e919>
- PrototypeValidityCell: <Cell 0x21f08217e411>
- TransitionsOrPrototypeInfo: <Smi 0 0x0>
[HeapObject]
 - MapWord: 0x1f9e87f00439
 - Size: 72
 - Page: 0xda2d6b40000
   - NextObject: 0xda2d6b4f531

v8 bt

Display v8 call frame information, JS functions, this, parameters, etc.

(gdb) v8 bt
#0  0x00000e5ce6f04010 home(this=<JsObject 0x620f5fe24e9>) at /disks/banana/zlei/alibaba/andb/test/backtrace/deadloop/dist/controller/home.controller.js:0
#1  0x0000000001462119 <arguments_adaptor>()
#2  0x00000000014681e2 (anonymous)(this=<Oddball 0x5d12f880471>, arg0=<JsObject 0x620f5fc8421>, arg1=<JsBoundFunction 0x620f5fe1c39>) at /disks/banana/zlei/alibaba/andb/test/backtrace/deadloop/node_modules/@midwayjs/core/dist/common/webGenerator.js:0
#3  0x0000000001494db0 (anonymous)(this=<JsGlobalProxy 0x12b3e36823e1>, arg0=<JsObject 0x620f5fe24e9>) at undefined:1489
#4  0x00000000015110ee <stub>()
#5  0x0000000001487eda <stub>()
#6  0x0000000001465e58 <entry>()
#7  0x0000000000d2863b v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&)()
#8  0x0000000000d29553 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*)()
#9  0x0000000000d52be1 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*)()
#10 0x0000000000d52fc1 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*)()
#11 0x0000000000c38261 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments)()
#12 0x0000000000c39eff v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*)()
#13 0x00000000014cf459 <builtin_exit>()
#14 0x00000000014681e2 processTicksAndRejections(this=<JsApiObject 0x12b3e3681d81>) at internal/process/task_queues.js:0
#15 0x0000000001465efa <internal>()
#16 0x0000000001465cd8 <entry>()
#17 0x0000000000d281e1 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&)()
#18 0x0000000000d2900f v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*)()
#19 0x0000000000bd4879 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*)()
#20 0x000000000098af6e node::InternalCallbackScope::Close()()
#21 0x000000000098c172 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context)()
#22 0x000000000099ae06 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*)()
#23 0x0000000000a473cd non-virtual thunk to node::(anonymous namespace)::Parser::OnStreamRead(long, uv_buf_t const&)()
#24 0x0000000000b0fcbc node::LibuvStreamWrap::ReadStart()::{lambda(uv_stream_s*, long, uv_buf_t const*)#2}::_FUN(uv_stream_s*, long, uv_buf_t const*)()
#25 0x00000000014551d2 uv__read(stream=<optimized out>) at ../deps/uv/src/unix/stream.c:1259
#26 0x00000000014558d8 uv__stream_io(loop=<optimized out>, w=<optimized out>, events=<optimized out>) at ../deps/uv/src/unix/stream.c:1326
#27 0x000000000145c2e5 uv__io_poll(loop=<optimized out>, timeout=<optimized out>) at ../deps/uv/src/unix/linux-core.c:472
#28 0x0000000001449595 uv_run(loop=<optimized out>, mode=<optimized out>) at ../deps/uv/src/unix/core.c:394
#29 0x0000000000a6fef5 node::NodeMainInstance::Run()()
#30 0x00000000009f7c41 node::Start(int, char**)()
#31 0x00007f71ae0fe445 __libc_start_main(main=0x9885b0 <main>, argc=2, argv=0x7fff155e5378, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff155e5368) at ../csu/libc-start.c:266
#32 0x0000000000989c5d _start()

Andb

The mainly ability​

Supported debug environments​

Usage​

Isolate location​

C++ debug​

Command line​

heap space [new|old|code|lo|...]​

heap page \<page_address>​

heap summary [old|code|lo|...]​

heap find [old|lo|...] \<tag>​

heap snapshot​

v8 inspect \<tag>​

v8 bt​