by ·0评论

# 在nginx和php-fpm下一访问nginx就瞬间502的问题 php-fpmsignal 7 (SIGBUS)

故障现象

使用TinkPHP3.2.x框架，页面偶尔会出现一访问nginx就报502 bad gateway，并不是等一段时间后nginx才报502，打开页面的一瞬间就502了。

php-fpm日志

 

[28-Sep-2015 23:25:07] WARNING: [pool www] child 2965 exited on signal 7 (SIGBUS) after 319.686109 seconds from start

[28-Sep-2015 23:25:07] NOTICE: [pool www] child 3223 started

[28-Sep-2015 23:25:49] WARNING: [pool www] child 1251 exited on signal 7 (SIGBUS) after 1005.893950 seconds from start

[28-Sep-2015 23:25:49] NOTICE: [pool www] child 3242 started

[28-Sep-2015 23:26:46] WARNING: [pool www] child 1467 exited on signal 7 (SIGBUS) after 1057.791921 seconds from start

[28-Sep-2015 23:26:46] NOTICE: [pool www] child 3284 started

[28-Sep-2015 23:30:33] WARNING: [pool www] child 1178 exited on signal 7 (SIGBUS) after 1289.872260 seconds from start

[28-Sep-2015 23:30:33] NOTICE: [pool www] child 3450 started

[28-Sep-2015 23:30:35] WARNING: [pool www] child 1115 exited on signal 7 (SIGBUS) after 1292.014337 seconds from start

[28-Sep-2015 23:30:35] NOTICE: [pool www] child 3451 started

[28-Sep-2015 23:30:35] WARNING: [pool www] child 1487 exited on signal 7 (SIGBUS) after 1285.395573 seconds from start

[28-Sep-2015 23:30:35] NOTICE: [pool www] child 3452 started

[28-Sep-2015 23:30:36] WARNING: [pool www] child 1958 exited on signal 7 (SIGBUS) after 1181.206336 seconds from start

[28-Sep-2015 23:30:36] NOTICE: [pool www] child 3453 started

可以看到有大量的 php-fpm进程收到 signal 7 退出了， 由于fastcgi php-fpm 瞬间退出了，nginx就瞬间502了

对php-fpm进行调试

既然看不出来头绪就需要进行调试了，这里需要把php-fpm退出的时候导出coredump

 php-fpm coredump方法

核心文件，也称核心转储，是操作系统在进程收到某些信号而终止运行时，将此时进程地址空间的内容以及有关进程状态的其他信息写出的一个磁盘文件。这种信息往往用于调试。 核心文件一词来源于磁芯内存。

php需要打开debug参数，如果编译的时候没有打开，需要重新编译，以下是我这里的

--enable-debug
--prefix=/usr/local/php --with-config-file-path=/usr/local/php/etc --enable-fpm --with-fpm-user=www --with-fpm-group=www --with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-iconv-dir --with-freetype-dir=/usr/local/freetype --with-jpeg-dir --with-png-dir --with-zlib --with-libxml-dir=/usr --enable-xml --disable-rpath --enable-bcmath --enable-shmop --enable-sysvsem --enable-inline-optimization --with-curl --enable-mbregex --enable-mbstring --with-mcrypt --enable-ftp --with-gd --enable-gd-native-ttf --with-openssl --with-mhash --enable-pcntl --enable-sockets --with-xmlrpc --enable-zip --enable-soap --with-gettext --disable-fileinfo --enable-debug

修改dump出来的路径

echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

允许程序崩溃的时候dump

ulimit -c unlimited

关闭dump功能

ulimit -c 0

用gdb进行调试

gdb /usr/local/php/sbin/php-fpm core.php-fpm.4555.web1.iamle.com.1443458079

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/local/php/sbin/php-fpm...done.

[New Thread 4555]

Missing separate debuginfo for /usr/lib/libmcrypt.so.4

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c5/699ec0c783a6055a7f983f9eb64146371eb2e6

Missing separate debuginfo for /usr/lib/libpng12.so.0

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063

Missing separate debuginfo for /usr/lib/libjpeg.so.62

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd

Missing separate debuginfo for /usr/local/lib/libiconv.so.2

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/df/7400febafa74aeef81e48b638c9fde24cb2beb

Missing separate debuginfo for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/64/1aa8440943921a4d553e380282d15828f18377

Missing separate debuginfo for

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fa/36da47e774d9bc5a739905887a5d25790090f7

Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypt.so.1

Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libz.so.1

Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/librt.so.1

Reading symbols from /usr/lib/libmcrypt.so.4...done.

Loaded symbols for /usr/lib/libmcrypt.so.4

Reading symbols from /usr/lib/libpng12.so.0...Missing separate debuginfo for /usr/lib/libpng12.so.0

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063.debug

(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libpng12.so.0

Reading symbols from /usr/lib/libjpeg.so.62...Missing separate debuginfo for /usr/lib/libjpeg.so.62

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd.debug

(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libjpeg.so.62

Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libcurl.so.4

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnsl.so.1

Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libxml2.so.2

Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssl.so.10

Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libcrypto.so.10

Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libfreetype.so.6

Reading symbols from /usr/local/lib/libiconv.so.2...done.

Loaded symbols for /usr/local/lib/libiconv.so.2

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libresolv.so.2

Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libfreebl3.so

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done.

Loaded symbols for /lib64/libidn.so.11

Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libldap-2.4.so.2

Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgssapi_krb5.so.2

Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5.so.3

Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libk5crypto.so.3

Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcom_err.so.2

Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssl3.so

Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libsmime3.so

Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libnss3.so

Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libnssutil3.so

Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libplds4.so

Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libplc4.so

Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnspr4.so

Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssh2.so.1

Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/liblber-2.4.so.2

Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libsasl2.so.2

Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5support.so.0

Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkeyutils.so.1

Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libselinux.so.1

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Core was generated by `php-fpm: pool www                                                             '.

Program terminated with signal 7, Bus error.

#0  lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:2265

2265 switch (yych) {

Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6.x86_64 freetype-2.3.11-14.el6_3.1.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libcurl-7.19.7-40.el6_6.1.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6.x86_64 libxml2-2.7.6-17.el6_6.1.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-14.el6.x86_64 nss-softokn-freebl-3.14.3-17.el6.x86_64 nss-util-3.16.1-3.el6.x86_64 openldap-2.4.39-8.el6.x86_64 openssl-1.0.1e-30.el6_6.4.x86_64 zlib-1.2.3-29.el6.x86_64

(gdb) bt

#0  lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:1088

#1  0x00000000007ad730 in zendlex (zendlval=0x7fff8083fae0) at /root/oneshell-master/php-5.4.45/Zend/zend_compile.c:6545

#2  0x0000000000795aca in zendparse () at /root/oneshell-master/php-5.4.45/Zend/zend_language_parser.c:3471

#3  0x00000000007a2990 in compile_file (file_handle=0x7fff8083fe30, type=2) at Zend/zend_language_scanner.l:585

#4  0x000000000063255a in phar_compile_file (file_handle=0x7fff8083fe30, type=2) at /root/oneshell-master/php-5.4.45/ext/phar/phar.c:3414

#5  0x00000000007a20bb in compile_filename (type=2, filename=0x299d018) at Zend/zend_language_scanner.l:628

#6  0x00000000008156de in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0x7f46209c6158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:27443

#7  0x0000000000832c80 in execute (op_array=0x27b4158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#8  0x00007f4614b27db6 in nb_execute (op_array=0x27b4158) at /tingyun/nbprof/nbprof.c:318

#9  0x00000000007c0e14 in zend_call_function (fci=0x7fff808401b0, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#10 0x00000000006d8ae5 in zif_call_user_func_array (ht=<value optimized out>, return_value=0x29733b0, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/standard/basic_functions.c:4754

#11 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5f78, return_value_used=1) at /tingyun/nbprof/nbprof.c:403

#12 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#13 0x0000000000832c80 in execute (op_array=0x7f46209f8a70) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#14 0x00007f4614b27db6 in nb_execute (op_array=0x7f46209f8a70) at /tingyun/nbprof/nbprof.c:318

#15 0x00000000007c0e14 in zend_call_function (fci=0x7fff80840540, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#16 0x00000000007e41bf in zend_call_method (object_pp=0x0, obj_ce=<value optimized out>, fn_proxy=0x7f46209f6618, function_name=0xb02423 "__callstatic",

    function_name_len=<value optimized out>, retval_ptr_ptr=0x7fff80840668, param_count=2, arg1=0x29730a8, arg2=0x29735c0)

    at /root/oneshell-master/php-5.4.45/Zend/zend_interfaces.c:97

#17 0x00000000007efb39 in zend_std_callstatic_user_call (ht=<value optimized out>, return_value=0x2972f58, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_object_handlers.c:1110

#18 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5d88, return_value_used=0) at /tingyun/nbprof/nbprof.c:403

#19 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#20 0x0000000000832c80 in execute (op_array=0x304d188) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#21 0x00007f4614b27db6 in nb_execute (op_array=0x304d188) at /tingyun/nbprof/nbprof.c:318

#22 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#23 0x0000000000832c80 in execute (op_array=0x2831020) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#24 0x00007f4614b27db6 in nb_execute (op_array=0x2831020) at /tingyun/nbprof/nbprof.c:318

#25 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#26 0x0000000000832c80 in execute (op_array=0x27a48a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#27 0x00007f4614b27db6 in nb_execute (op_array=0x27a48a0) at /tingyun/nbprof/nbprof.c:318

#28 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#29 0x0000000000832c80 in execute (op_array=0x27a25b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#30 0x00007f4614b27db6 in nb_execute (op_array=0x27a25b8) at /tingyun/nbprof/nbprof.c:318

#31 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#32 0x0000000000832c80 in execute (op_array=0x2832410) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#33 0x00007f4614b27db6 in nb_execute (op_array=0x2832410) at /tingyun/nbprof/nbprof.c:318

---Type <return> to continue, or q <return> to quit---

#34 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#35 0x0000000000832c80 in execute (op_array=0x282c1e0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#36 0x00007f4614b27db6 in nb_execute (op_array=0x282c1e0) at /tingyun/nbprof/nbprof.c:318

#37 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#38 0x0000000000832c80 in execute (op_array=0x2896bb8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#39 0x00007f4614b27db6 in nb_execute (op_array=0x2896bb8) at /tingyun/nbprof/nbprof.c:318

#40 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#41 0x0000000000832c80 in execute (op_array=0x286ced8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#42 0x00007f4614b27db6 in nb_execute (op_array=0x286ced8) at /tingyun/nbprof/nbprof.c:318

#43 0x00000000007c0e14 in zend_call_function (fci=0x7fff80841170, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#44 0x000000000064ad3f in zim_reflection_method_invoke (ht=<value optimized out>, return_value=0x288e4e8, return_value_ptr=<value optimized out>, this_ptr=0x2859330,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/reflection/php_reflection.c:2898

#45 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c0530, return_value_used=0) at /tingyun/nbprof/nbprof.c:403

#46 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#47 0x0000000000832c80 in execute (op_array=0x27a5a58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#48 0x00007f4614b27db6 in nb_execute (op_array=0x27a5a58) at /tingyun/nbprof/nbprof.c:318

#49 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#50 0x0000000000832c80 in execute (op_array=0x27a5b58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#51 0x00007f4614b27db6 in nb_execute (op_array=0x27a5b58) at /tingyun/nbprof/nbprof.c:318

#52 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#53 0x0000000000832c80 in execute (op_array=0x27b66a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#54 0x00007f4614b27db6 in nb_execute (op_array=0x27b66a0) at /tingyun/nbprof/nbprof.c:318

#55 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#56 0x0000000000832c80 in execute (op_array=0x7f46209f4108) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#57 0x00007f4614b27e84 in nb_execute (op_array=0x7f46209f4108) at /tingyun/nbprof/nbprof.c:318

#58 0x0000000000816a9c in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER (execute_data=0x7f46209bd318) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:2650

#59 0x0000000000832c80 in execute (op_array=0x7f46209f35b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#60 0x00007f4614b27dff in nb_execute (op_array=0x7f46209f35b8) at /tingyun/nbprof/nbprof.c:318

#61 0x00000000007cc5ae in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /root/oneshell-master/php-5.4.45/Zend/zend.c:1329

#62 0x000000000077221e in php_execute_script (primary_file=0x7fff80845fc0) at /root/oneshell-master/php-5.4.45/main/main.c:2502

#63 0x000000000087b34e in main (argc=<value optimized out>, argv=<value optimized out>) at /root/oneshell-master/php-5.4.45/sapi/fpm/fpm/fpm_main.c:1938

(gdb)

 

发现 php的lex_scan执行的时候崩溃了，通过搜索明白是某php正在被词法分析的时候又被修改了。

最后确定是TinkPHP开启调试模式后，模板是动态编译的，这样当有并发访问的时候就发生了php文件又在被解析，又在被写入的情况导致lex_scan的时候php-fpm就挂掉了。

当有并发的时候关闭调试模式即可解决

也算是见识了php程序的问题可以让php-fpm进程都挂掉

参考

提供SIGABRT的 backtrace ，如何提供backtrace, 请参看:

http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32

Generating core-dump for php5-fpm

https://rtcamp.com/tutorials/php/core-dump-php5-fpm/

对nginx报502bad gateway的一次原因定位

http://blog.chunshiban.com/2013/07/05/%E5%AF%B9nginx%E6%8A%A5502bad-gateway%E7%9A%84%E4%B8%80%E6%AC%A1%E5%8E%9F%E5%9B%A0%E5%AE%9A%E4%BD%8D/

一例php进程的SIGBUS故障

http://blog.druggo.org/post/2013/05/02/%E4%B8%80%E4%BE%8Bphp%E8%BF%9B%E7%A8%8B%E7%9A%84SIGBUS%E6%95%85%E9%9A%9C

低并发502错误signal 7 (SIGBUS)

http://kokahkhk.blog.163.com/blog/static/209428040201411595622729/

压力测试下httpd进程由于lex_scan和Runtime缓存文件而崩溃

http://www.thinkphp.cn/topic/27464.html

php进程崩溃跟踪方法

http://blog.chinaunix.net/uid-23504396-id-4819011.html

by ·0评论

openwrt 已经正式发布 openwrt chaos_calmer 15.05

ZTE-h618b cpu是mips构架

cpu型号为brm5354

h618b刷openwrt chaos_calmer 15.05版本为

http://downloads.openwrt.org/chaos_calmer/15.05/brcm47xx/generic/openwrt-15.05-brcm47xx-generic-squashfs.trx

中文语言包

opkg install http://downloads.openwrt.org/chaos_calmer/15.05/brcm47xx/generic/packages/luci/luci-i18n-base-zh-cn_git-15.248.30277-3836b45-1_all.ipk

 

Tp-link-wr703n cpu是arm架构

wr703n刷openwrt chaos_calmer 15.05版本为

其他系统刷openwrt用这个 http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/openwrt-15.05-ar71xx-generic-tl-wr703n-v1-squashfs-factory.bin

已经是openwrt的老版本升级用这个

http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/openwrt-15.05-ar71xx-generic-tl-wr703n-v1-squashfs-sysupgrade.bin

中文语言包

opkg install http://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/packages/luci/luci-i18n-base-zh-cn_git-15.248.30277-3836b45-1_all.ipk

by ·3条评论

Centos使用MySQL工具Percona Toolkit

安装Percona Toolkit 的Repo 得以支持直接用yum 安装二进制包

yum install -y https://www.percona.com/redir/downloads/percona-release/redhat/latest/percona-release-0.1-4.noarch.rpm
yum install -y percona-toolkit

改MySQL表结构DDL pt-online-schema-change

pt-online-schema-change -u root -h 10.8.8.8 -p password --alter='add column vid int ' --execute A=utf8,D=database,t=table

D=database 库名
t=table 表名
–alter=’add column vid int ‘ 要执行的语句

注意：
mysql版本5.5 数据库和表都是utf8编码,
表的字段注释维中文
在执行 pt-online-schema-change的时候一定的加参数 -A=utf-8

Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `wwek`.`wwek_users`...
Creating new table...
Created new table wwek._wwek_users_new OK.
Altering new table...
Altered `wwek`.`_wwek_users_new` OK.
2015-09-18T10:56:16 Creating triggers...
2015-09-18T10:56:16 Created triggers OK.
2015-09-18T10:56:16 Copying approximately 183066 rows...
Copying `wwek`.`wwek_users`: 30% 01:10 remain
Copying `wwek`.`wwek_users`: 49% 01:01 remain
Copying `wwek`.`wwek_users`: 63% 00:51 remain
Copying `wwek`.`wwek_users`: 76% 00:37 remain
Copying `wwek`.`wwek_users`: 88% 00:20 remain
2015-09-18T10:59:03 Copied rows OK.
2015-09-18T10:59:03 Swapping tables...
2015-09-18T10:59:03 Swapped original and new tables OK.
2015-09-18T10:59:03 Dropping old table...
2015-09-18T10:59:04 Dropped old table `wwek`.`_wwek_users_old` OK.
2015-09-18T10:59:04 Dropping triggers...
2015-09-18T10:59:04 Dropped triggers OK.
Successfully altered `wwek`.`wwek_users`.

 

文档&&参考

官方文档Percona Toolkit Documentation

by ·0评论

Nginx+Logstash+Elasticsearch+Kibana搭建网站日志分析系统笔记

前言

流程,nignx格式化日志成json,通过logstash直接采集到elasticsearch,然后通过kibana gui界面展示分析

要点nignx日志成json格式,避免nignx默认日志是空格,需要正则匹配,导致logstash占过多cpu
elasticsearch机配置防火墙,只让指定的logstash机访问
kibana只监听本地127.0.0.1使用nignx方向代理,nginx中配置Http Basic Auth账号密码登陆

比较粗略的笔记,备忘
安装java
yum install java-1.8.0-openjdk*

nginx配置

为了让nignx机跑logstash采集日志负载最低，建议直接生成json的方式，直接就可以用logstash读取写入到Elasticsearch

http{} 中定义 格式化日志成json

log_format logstash_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"request":"$request",'
'"url":"$uri",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';

server内输出日志 access_log可以配置多个同时输出，可以保留你以前的

access_log /data/wwwlogs/www.iamle.log iamle.com;
access_log /data/wwwlogs/www.iamle.com.logstash_json.log logstash_json;

nginx机安装Logstash1.5.x

rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install logstash

 

在目录 /etc/logstash/conf.d/
建立配置文件 nginx_json.conf

input {
file {
path => "/data/wwwlogs/www.iamle.com.logstash_json.log"
codec => json
}
}
filter {
mutate {
split => [ "upstreamtime", "," ]
}
mutate {
convert => [ "upstreamtime", "float" ]
}
}
output {
elasticsearch {
host => "elk.server.iamle.com"
protocol => "http"
index => "logstash-%{type}-%{+YYYY.MM.dd}"
index_type => "%{type}"
workers => 5
template_overwrite => true
}
}

service logstash start

日志存储机安装Elasticsearch1.7.x提供数据底层支持

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install elasticsearch

配置文件
配置数据保存位置

vim /etc/elasticsearch/elasticsearch.yml
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
path.data: /data
目录会自动生成,只需要指定一个空目录就可以了

service elasticsearch start

centos7
systemctl start elasticsearch
systemctl status elasticsearch
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled)
Active: active (running) since Fri 2015-09-04 15:37:08 CST; 1s ago
Docs: http://www.elastic.co
Main PID: 19376 (java)
CGroup: /system.slice/elasticsearch.service
└─19376 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X...
Sep 04 15:37:08 elk systemd[1]: Starting Elasticsearch...
Sep 04 15:37:08 elk systemd[1]: Started Elasticsearch.
检查是否已经成功开启
ss -ltnp |grep 9200

centos7配置firewalld固定ip可访问elasticsearch
systemctl start firewalld.service
systemctl status firewalld.service

 

只允许nignx机访问elasticsearch机9200 9300端口

firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="10.8.8.2" \
port protocol="tcp" port="9200" accept"

firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="10.8.8.2" \
port protocol="tcp" port="9300" accept"
firewall-cmd --reload

iptables -L -n |grep 9200
ACCEPT tcp -- 10.8.8.2 0.0.0.0/0 tcp dpt:9200 ctstate NEW

 

安装Kibana4展示Elasticsearch中的数据

 

wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
tar zxvf kibana-4.1.1-linux-x64.tar.gz
cd kibana-4.1.1-linux-x64
修改配置文件
vim /usr/local/kibana-4.1.1-linux-x64/config/kibana.yml
# Kibana is served by a back end server. This controls which port to use.
port: 5601

# The host to bind the server to.
#监听本地地址 用nignx反向代理
host: "127.0.0.1"

nohup ./bin/kibana &

检查是否已经成功开启
ss -ltnp |grep 5601

 

使用nignx反向代理kibana
nginx配置Http Basic Auth账号密码登陆
http://trac.edgewall.org/export/10770/trunk/contrib/htpasswd.py （nginx wiki里推荐的）
运行示例
chmod 777 htpasswd.py
./htpasswd.py -c -b htpasswd username password
#-c为生成文件 htpasswd为文件名

server
{
listen 80;
#listen [::]:80;
server_name elk.server.iamle.com;

location / {
auth_basic "Password please";
auth_basic_user_file /usr/local/nginx/conf/htpasswd;
proxy_pass http://127.0.0.1:5601/;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

老版本
https://download.elastic.co/kibana/kibana/kibana-3.1.3.tar.gz
https://www.elastic.co/downloads/past-releases/kibana-3-1-3

参考

http://kibana.logstash.es/
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

扩展centos7 firewall的使用

 

检查防火墙状态
firewall-cmd --stat

临时开放ftp服务
firewall-cmd --add-service=ftp
永久开放ftp服务
firewall-cmd --add-service=ftp --permanent
关闭ftp服务
firewall-cmd --remove-service=ftp --permanent
配置防火墙在public区域永久开放http服务
firewall-cmd --permanent --zone=public --add-service=http
加入指定开放端口
firewall-cmd --add-port=1324/tcp

为了让之前的设定生效当然要重启服务咯
systemctl restart firewalld
或者使用下面的命令免去重启服务(防火墙策略配置后重新载入)
firewall-cmd --complete-reload
firewall-cmd --reload (这两句功能相同)

检查ftp服务的21端口是否开放
iptables -L -n | grep 21
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:21 ctstate NEW

查询ftp服务启用状态
firewall-cmd --query-service ftp

查看当前规则
firewall-cmd --list-all

仅允许部分IP访问本机服务配置
firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="192.168.0.4/24" service name="http" accept"

仅允许部分IP访问本机端口配置
firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \
source address="192.168.0.4/24" \
port protocol="tcp" port="8080" accept"

 

by ·0评论

故障现象

微信服务号发送模版消息失败,生成事件二维码图片也失败

代码没有任何改动,莫名其妙就这样了

# 调试过程
检查后发现如下返回错误

{
“errcode”: 40001,
“errmsg”: “invalid credential, access_token is invalid or not latest”
}
{
“errcode”: 40001,
“errmsg”: “invalid credential, access_token is invalid or not latest”
}
检查微信所有api相关参数是否正确,检查都是对的
重新获取新的access_token测试能通过
一度怀疑是不是缓存有效期是不是出问题了
过期了应该是下图的样子42001,而不是40001呀

{
“errcode”: 42001,
“errmsg”: “access_token expired”
}
 
 
{
“errcode”: 42001,
“errmsg”: “access_token expired”
}
https://api.weixin.qq.com/cgi-bin/qrcode/create?access_token=CM2Dt8rXdcMB5k_In4mGRF6M2jT_NYAakU4RyrHyj1uRXNxdmupGDRhWYA_DSgG_LRgcwXRbsJCVrR5IVBl5rx3jNRyo-T_xPTJ4ZhrjCxc
 

找到问题

http://segmentfault.com/q/1010000002998425
最后一句话点醒了我
yi5344 说道 access_token的有效期目前为2个小时，需定时刷新，重复获取将导致上次获取的access_token失效。
 
突然想起来,在我们的系统当中有两处独立的微信公众号SDK调用,缓存access_token是独立工作的。
对于同一个微信公众号来说 access_token 是一把钥匙。
如果本地access_token多处单独存储和缓存,这样会导致多处都去更新微信api中access_token值,当更新发生的时候这样只有一处是正常的,其他地方本地的access_token就失效了。
 

解决方法

对于同一个微信公众账号的  access_token 只能存储在一个地方。

by ·0评论

故障现象

苏州移动网络宽带在使用企业QQ的时候企业QQ群里发送图片失败，即便再次重发也不行，所有人在企业QQ群里面发图都不行

使用电信线路正常，普通QQ也没问题，企业QQ使用代理登陆正常

故障分析

使用网络抓包分析看到底是什么情况

由于使用wireshark无法指定进程抓包，也没法看到是哪个进程的包，虽然可以用没运行其他网络软件的系统来进行抓包分析。

我使用更方便的工具Microsoft Message Analyzer来抓包分析。（

Add Columns增加字段 PID，

找到你要抓包的网络软件相应的PID号，我这里企业QQ当前运行的PID是4100

View Filter 写

*Pid == 4100

）

还推荐一款可以指定进程抓包的软件SRSniffer

 

通过抓包得知，普通QQ私信和群发送图片和接收都是走的http协议

企业QQ私信发图片和接收图片走的http协议，企业QQ群发送图片和接收图片是走的UDP协议。

当在企业QQ群里发送图片的时候，同时观察抓包情况。

仔细观察分析发现，企业QQ发送图片会开始发送UDP数据包，腾讯的服务器ip无返回数据包。

把无返回数据包的ip进行ping和tracert操作，发现无法ping，也无法tracert，证明到这个ip网络是不通的。

我这里是routeros路由器，多线接入，所以把发现移动线路不通的ip策略路由到电信线路上去，这下企业QQ群图片就发送正常了。

我这里抓到的企业QQ群图片发送腾讯服务器IP地址为（这里肯定是不全的，只采集到部分发现的）

183.60.16.207
183.60.56.92
183.60.48.171
183.60.18.38
183.60.62.185

查了下ip归属，居然都是电信的ip，看来企业QQ这里有问题啊，为什么没解析到对应的线路上去。

还不知道有多少ip不通，我先治标把183.60.0.0/16全路由到电信线路上去了。

解决方案

1、如果自己可以做路由，把不通的ip走其他线路，如果只有一个线路可以用vpn的方式解决，走vpn里面去。治标。

2、立马联系移动的客户经理，联系到他们的网络软调，反馈这些ip不通的问题，做路由调通。治本。

3、在企业QQ上反馈，为什么企业QQ群发送图片，移动带宽线路，为什么也是电信ip的服务器，应该智能分配到移动线路的ip上去，或者说有BGP也行

 

 

后记2015年12月企业QQ群出现群里面接收图片有的能接收有的不能接收

更新最新版企业QQ无效,换了一个网络还是无效
通过抓包分析知道企业QQ群接收图片会先从183.60.62.185 udp 8000 去尝试取图片数据,多次尝试失败后会走http协议到另外的地址下载图片
除了慢以外,而且有时候图片也显示不了
办公网络 ping 183.60.62.185 不通, 用17ce多节点ping也不通
思路,通过dst-nat目标nat修改nat包实现重定向
需要重定向183.60.62.185 tcp 443 udp 80 到 另外一个企业QQ群图片服务器 183.60.18.38 tcp 443 udp 80
路由器使用的是routeros,所以直接改nat包实现重定向即可,命令为

/ip firewall nat
add action=dst-nat chain=dstnat comment=\
    "\C6\F3\D2\B5qq\C8\BA\CD\BC\C6\AC\B7\FE\CE\F1\C6\F7\D6\D8\B6\A8\CF\F2udp" \
    disabled=no dst-address=183.60.62.185 protocol=udp to-addresses=\
    183.60.18.38 to-ports=0-65535
add action=dst-nat chain=dstnat comment=\
    "\C6\F3\D2\B5qq\C8\BA\CD\BC\C6\AC\B7\FE\CE\F1\C6\F7\D6\D8\B6\A8\CF\F2tcp" \
    disabled=no dst-address=183.60.62.185 protocol=tcp to-addresses=\
    183.60.18.38 to-ports=0-65535