gdb

ruizhou2023年4月25日大约 4 分钟

一次线上gdb分析

debian 换源

sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list

Ubuntu 22.04 换源，vim /etc/apt/sources.list

deb http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse

or 执行下面命令

sudo bash -c "cat << EOF > /etc/apt/sources.list
deb http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
EOF"

分析环境

M1 mackoopro 是 arm 平台（aarch 是 arm v8 的一种状态）

我们的服务环境是 debian，x86_64，所以分析的环境最好也是同样的环境

强制拉 amd64 的 image

docker pull --platform=linux/amd64 debian:stable

运行，需要指定 amd64 平台

docker run --name debians-dev -v /Users/zhou/dockersyncfile:/root/sync -itd --platform linux/amd64 debian:stable

安装 gdb

apt update
apt install gdb

gdb 开始分析 core 文件

gdb php

core-file

core-file coredumpfile
// output
(gdb) core-file core.998
Core was generated by `/data/app/room-service/frontend/boot/index.php [ServerProcess] [app:room][worke'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  swHashMap_find_int (hmap=0x0, key=1) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/core/hashmap.c:237
237	/root/repo/php-bin/php-7.0.33/ext/swoole/src/core/hashmap.c: No such file or directory.

(gdb) bt
#0  swHashMap_find_int (hmap=0x0, key=1) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/core/hashmap.c:237
#1  0x000000000086976e in swoole_http2_onFrame (client=0x2662f50, req=0x1, req@entry=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_v2_server.c:594
#2  0x0000000000867fcf in http_onReceive (serv=0x25a75f0, req=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_server.c:1066
#3  0x000000000089e941 in swWorker_onTask (factory=0xb0, task=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Worker.c:307
#4  0x000000000089ecd1 in swWorker_onPipeReceive (reactor=<optimized out>, event=0x7ffc9b760ef0) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Worker.c:814
#5  0x000000000088ca60 in swReactorEpoll_wait (reactor=0x25aca70, timeo=0x1) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/reactor/ReactorEpoll.c:255
#6  0x000000000089f1b4 in swWorker_loop (factory=factory@entry=0x25a7a68, worker_id=worker_id@entry=1) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Worker.c:772
#7  0x000000000089c14a in swManager_spawn_worker (factory=0x25a7a68, worker_id=1) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Manager.c:507
#8  0x000000000089cc69 in swManager_loop (factory=<optimized out>) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Manager.c:386
#9  swManager_start (factory=0x25a7a68) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Manager.c:178
#10 0x0000000000888fba in swFactoryProcess_start (factory=0x25a7a68) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/factory/FactoryProcess.c:86
#11 0x000000000089138b in swServer_start (serv=0x25a75f0) at /root/repo/php-bin/php-7.0.33/ext/swoole/src/network/Server.c:747
#12 0x00000000008637bc in zim_swoole_http_server_start (execute_data=0x7f4f1d014e60, return_value=0x7f4f1d014e50) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_server.c:1633
#13 0x00000000009ca692 in ZEND_DO_FCALL_SPEC_HANDLER () at /root/repo/php-bin/php-7.0.33/Zend/zend_vm_execute.h:842
#14 0x0000000000985f5b in execute_ex (ex=<optimized out>) at /root/repo/php-bin/php-7.0.33/Zend/zend_vm_execute.h:414
#15 0x00000000009d9d47 in zend_execute (op_array=0x7f4f1d08a000, op_array@entry=0x7f4f1d1b8e40, return_value=return_value@entry=0x7f4f1d014af0)
    at /root/repo/php-bin/php-7.0.33/Zend/zend_vm_execute.h:458
#16 0x00000000009482f4 in zend_execute_scripts (type=type@entry=8, retval=0x7f4f1d014af0, retval@entry=0x0, file_count=file_count@entry=3) at /root/repo/php-bin/php-7.0.33/Zend/zend.c:1445
#17 0x00000000008eaa30 in php_execute_script (primary_file=0x7ffc9b7636f0) at /root/repo/php-bin/php-7.0.33/main/main.c:2516
#18 0x00000000009db95a in do_cli (argc=0, argv=0x1) at /root/repo/php-bin/php-7.0.33/sapi/cli/php_cli.c:977
#19 0x0000000000499cb7 in main (argc=0, argv=0x1) at /root/repo/php-bin/php-7.0.33/sapi/cli/php_cli.c:1347

frame 1

// output
(gdb) frame 1
#1  0x000000000086976e in swoole_http2_onFrame (client=0x2662f50, req=0x1, req@entry=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_v2_server.c:594
594	/root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_v2_server.c: No such file or directory.

没有源代码，需要 set substitute-path

set substitute-path /root/repo/php-bin/php-7.0.33/ext/swoole /root/sync/swoole-bili-1.10.5

重新查看

(gdb) frame 1
#1  0x000000000086976e in swoole_http2_onFrame (client=0x2662f50, req=0x1, req@entry=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_v2_server.c:594
594	    {
(gdb) print client->http2
$1 = 1

http2 为 1，不符合预期

查看 req 的实际信息

(gdb) frame 2
#2  0x0000000000867fcf in http_onReceive (serv=0x25a75f0, req=0x7ffc9b75ee90) at /root/repo/php-bin/php-7.0.33/ext/swoole/swoole_http_server.c:1066
1066	        return swoole_http2_onFrame(client, req);
(gdb) print *req
$2 = {info = {fd = 2068, len = 88, from_id = 6, type = 10 '\n', flags = 0 '\000', from_fd = 3},
  data = "\000\000O\000\000\000\000\000\001\000\000\000\000J\n\026live.live.room-service\022\005sh001\032\023\067\066\064\067\070\062\064\064\060\065\060\064\066\064\067\061\071\062\071\"\005\062.3.0*\r10.150.113.62cd61cd1ed475571ba5d3f94cf49cf4 HTTP/1.1\r\nhost: api.live.bilibili.co\r\nx-forwarded-for: 10.230.17.157,172.22.33.26"...}

把变量输出的 size 调大

(gdb) set print elements 4096
(gdb) print *req
$3 = {info = {fd = 2068, len = 88, from_id = 6, type = 10 '\n', flags = 0 '\000', from_fd = 3},
  data = "\000\000O\000\000\000\000\000\001\000\000\000\000J\n\026live.live.room-service\022\005sh001\032\023\067\066\064\067\070\062\064\064\060\065\060\064\066\064\067\061\071\062\071\"\005\062.3.0*\r10.150.113.62cd61cd1ed475571ba5d3f94cf49cf4 HTTP/1.1\r\nhost: api.live.bilibili.co\r\nx-forwarded-for: 10.230.17.157,172.22.33.26\r\nuser-agent: haoguanwei@bilibili.com go1.17.2 app-feed-list-api-test-77645bb6cd-lcz7d main.app-svr.app-feed\r\nx-backend-bili-criticality: \r\nx1-bilispy-color: \r\nx1-bilispy-id: 238066525567990090\r\nx1-bilispy-mirror: \r\nx1-bilispy-parentid: 0\r\nx1-bilispy-sampled: true\r\nx1-bilispy-spanid: 238066525567990090\r\naccept-encoding: gzip\r\nx-forwarded-proto: http\r\nx-envoy-external-address: 172.22.33.26\r\nx-request-id: ab160a44-cca8-9e55-abf7-3cc0d7dd15df\r\nx-b3-traceid: 2f094b2c5ca7a552034dc83e6164394a\r\nx-b3-spanid: 0152b03b1fdfb655\r\nx-b3-parentspanid: 034dc83e6164394a\r\nx-b3-sampled: 1\r\nbili-trace-id: 034dc83e6164394a:0152b03b1fdfb655:034dc83e6164394a:1\r\nx-bili-trace-id: 2f094b2c5ca7a552034dc83e6164394a:0152b03b1fdfb655:034dc83e6164394a:1\r\nx1-bilispy-timeout: 10000\r\nx1-bilispy-user: ekango\r\ncontent-length: 0\r\nx-envoy-expected-rq-timeout-ms: 15000\r\n\r\n\r\n\374\177\000\000\060\000\000\000\060\000\000\000>\363u\233\374\177\000\000\320\362u\233\374\177\000\000@\000\000\000\000\000\000\000\020\000\000\000\377\377\377\377\000\000\000\000\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "\220\363u\233\374\177\000\000\340\060\v\035O\177\000\000\220\344\244\016O\177\000\000\300G\212\001\000\000\000\000\330\344\244\016O\177\000\000\020\364u\233\374\177\000\000\000\340\b\035O\177\000\000\300G\212\001\000\000\000\000\372\070\223\000\000\000\000\000\370&\273\017O\177\000\000\337<O<\000\340\377\377\240\370u\233\374\177\000\000k\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\250\340\244\016O\177\000\000\330\344\244\016O\177\000\000\313A\223\000\000\000\000\000\300G\212\001", '\000' <repeats 12 times>

Reference

https://sourceware.org/gdb/onlinedocs/gdb/Source-Path.html#set-substitute_002dpath

https://sourceware.org/gdb/onlinedocs/gdb/Command-Settings.html#Command-Settings