Beats(PHP-FPM Module@Metricbeat) + Elasticsearch + Kibana 采集分析phpfpm status状态数据

by ·0评论

需求

当前php-fpm进程是不是不够了,一通命令一打,也只能看当前情况。
熟话说无监控无度量就无诊断。
市面上有成熟的php-fpm status状态采集，展示工具那就是elasti stack全家桶
目标采集分析php-fpm status 状态数据

选型

Elastic公司全家桶之Beats(PHP-FPM Module@Metricbeat) + Elasticsearch + Kibana

Metricbeat 是 Beats 的一个采集组件
PHP-FPM Module 是 Metricbeat 的一个采集模块

整体的数据流方式

方式1 : Beats (Metricbeat )>Elasticsearch>Kibana
方式2: Beats (Metricbeat )>Logstash>[直连,redis队列,Kafka队列]>Elasticsearch>Kibana

先讲方式1

配置php-fpm

php-fpm.conf

[www]
...
pm.status_path = /phpfpm-status-www

[u]
...
pm.status_path = /phpfpm-status-u

[www] [u] 是独立的php-fpm pool 进程池,每个池子的pm.status_path是需要单独设置的

测试配置文件是否正确

/usr/local/php/sbin/php-fpm -t
[29-Mar-2017 16:15:08] NOTICE: configuration file /usr/local/php/etc/php-fpm.conf test is successful

重启php-fpm

/etc/init.d/php-fpm reload
Reload service php-fpm done

配置nginx

phpfpm.conf

server
{
  listen 80;
  server_name localhost;
  location ~ ^/(phpfpm-status-www|phpstatuswww)$
  {
    fastcgi_pass unix:/tmp/php-cgi.sock;
    include fastcgi.conf;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
  }
  location ~ ^/(phpfpm-status-u|phpstatusu)$
  {
    fastcgi_pass unix:/tmp/u-php-cgi.sock;
    include fastcgi.conf;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
  }
}

只给本地访问,这里监听的是localhost,也可用nginx allow deny的方式

测试配置文件是否正确

/usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

重启nginx

/etc/init.d/nginx reload
Reloading nginx daemon configuration....

用curl测试php-fpm status状态是否可以获取

curl http://localhost/phpfpm-status-www
pool: www
process manager: static
start time: 29/Mar/2017:16:19:09 +0800
start since: 75
accepted conn: 900
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 696
active processes: 4
total processes: 700
max active processes: 60
max children reached: 0
slow requests: 1

配置Metricbeat

Metricbeat Reference [5.3] » Modules » PHP-FPM Module

/etc/metricbeat/metricbeat.yml

#========================== Modules configuration ============================
metricbeat.modules:

#------------------------------- PHP-FPM Module -------------------------------
- module: php_fpm
  metricsets: ["pool"]
  enabled: true
  period: 10s
  status_path: "/phpfpm-status-www"
  hosts: ["localhost:80"]

- module: php_fpm
  metricsets: ["pool"]
  enabled: true
  period: 10s
  status_path: "/phpfpm-status-u"
  hosts: ["localhost:80"]

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
  hosts: ["localhost:9200"]

测试配置文件是否正确

/usr/share/metricbeat/bin/metricbeat -configtest
Config OK

给metricbeat创建Elasticsearch的Index Template
Loading the Index Template in Elasticsearch

curl -XPUT 'http://localhost:9200/_template/metricbeat' -d@/etc/metricbeat/metricbeat.template.json

给metricbeat创建Kibana Dashboards
Loading Sample Kibana Dashboards

./scripts/import_dashboards -es http://localhost:9200

metricbeat中phpfpm模块，没有elastic官方的 Sample Kibana Dashboard
所以需要分析什么数据的自己在Kibana中自己创建visualize然后做成Dashboard

启动metricbeat

/etc/init.d/metricbeat start

配置Kibana

Management / Kibana / Indices > Add New
Index name or pattern 填写 “metricbeat-*”
Time-field name 选 “@timestamp”
Alt text

分享我做的 metricbeat phpfpm kibana dashboard （不完整）
export.json

[
  {
    "_id": "PHP-FPM导航",
    "_type": "visualization",
    "_source": {
      "title": "PHPFPM导航",
      "visState": "{\n  \"title\": \"PHP-FPM导航\",\n  \"type\": \"markdown\",\n  \"params\": {\n    \"markdown\": \"- [Overview](#/dashboard/Metricbeat-phpfpm-overview)\\n\\n\\n```\\npool：php-fpm池的名称，一般都是应该是www\\nprocess manage：进程的管理方法，php-fpm支持三种管理方法，分别是static,dynamic和ondemand，一般情况下都是dynamic\\nstart time：php-fpm启动时候的时间，不管是restart或者reload都会更新这里的时间\\nstart since：php-fpm自启动起来经过的时间，默认为秒\\naccepted conn：当前接收的连接数\\nlisten queue：在队列中等待连接的请求个数，如果这个数字为非0，那么最好增加进程的fpm个数\\nmax listen queue：从fpm启动以来，在队列中等待连接请求的最大值\\nlisten queue len：等待连接的套接字队列大小\\nidle processes：空闲的进程个数\\nactive processes：活动的进程个数\\ntotal processes：总共的进程个数\\nmax active processes：从fpm启动以来，活动进程的最大个数，如果这个值小于当前的max_children，可以调小此值\\nmax children reached：当pm尝试启动更多的进程，却因为max_children的限制，没有启动更多进程的次数。如果这个值非0，那么可以适当增加fpm的进程数\\nslow requests：慢请求的次数，一般如果这个值未非0，那么可能会有慢的php进程，一般一个不好的mysql查询是最大的祸首。\\n```\"\n  },\n  \"aggs\": [],\n  \"listeners\": {}\n}",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\n  \"query\": {\n    \"query_string\": {\n      \"query\": \"*\",\n      \"analyze_wildcard\": true\n    }\n  },\n  \"filter\": []\n}"
      }
    }
  },
  {
    "_id": "PHPFPM-processes-active",
    "_type": "visualization",
    "_source": {
      "title": "PHPFPM processes active",
      "visState": "{\"title\":\"PHPFPM processes active\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"smoothLines\":false,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"php_fpm.pool.processes.active\",\"customLabel\":\"活动\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"时间\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"php_fpm.pool.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"pool池\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"beat.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"主机\",\"row\":true}}],\"listeners\":{}}",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\"index\":\"metricbeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
      }
    }
  },
  {
    "_id": "PHPFPM-connections-queued",
    "_type": "visualization",
    "_source": {
      "title": "PHPFPM connections queued",
      "visState": "{\"title\":\"PHPFPM connections queued\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"smoothLines\":false,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"php_fpm.pool.connections.queued\",\"customLabel\":\"队列\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"时间\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"php_fpm.pool.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"pool池\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"beat.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"主机\",\"row\":true}}],\"listeners\":{}}",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\"index\":\"metricbeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
      }
    }
  },
  {
    "_id": "PHPFPM-processes-idle",
    "_type": "visualization",
    "_source": {
      "title": "PHPFPM processes idle",
      "visState": "{\"title\":\"PHPFPM processes idle\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"smoothLines\":false,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"php_fpm.pool.processes.idle\",\"customLabel\":\"空闲\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"时间\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"php_fpm.pool.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"pool池\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"beat.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"主机\",\"row\":true}}],\"listeners\":{}}",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\"index\":\"metricbeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
      }
    }
  },
  {
    "_id": "PHPFPM-slow_requests",
    "_type": "visualization",
    "_source": {
      "title": "PHPFPM slow_requests",
      "visState": "{\"title\":\"PHPFPM slow_requests\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"smoothLines\":false,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"php_fpm.pool.slow_requests\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"时间\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"php_fpm.pool.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"pool池\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"beat.name\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"主机\",\"row\":true}}],\"listeners\":{}}",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\"index\":\"metricbeat-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
      }
    }
  }
]

补充logstash采集数据（方式2）

配置Metricbeat

/etc/metricbeat/metricbeat.yml

#================================ Outputs =====================================
# The Logstash hosts
output.logstash:

  hosts: ["localhost:5044"]

output输出到logstash的input tcp 5044上
input输入参考上面方式1 Metricbeat 的配置方法

配置logstash2.x

Logstash Reference [2.4] » Input plugins » beats

/etc/logstash/conf.d/beats.conf

input {
    beats {
        host => "127.0.0.1"
        port => 5044
    }
}
filter {
}
output {
    ... 输出到kafka 或者reids 或者 elasticsearch等,具体配置方法看官方文档
}

input输入tcp 5044 监听本机
output输出，根据自己环境来

测试配置文件是否正确

/opt/logstash/bin/logstash -t
Configuration OK

启动logstash

/etc/init.d/logstash start

提示

  • 使用测试配置文件的功能测试配置文件是否书写正确
  • 使用tail -f 日志的方式查错
  • 注意tcp监听的安全问题，别暴露到公网IP上
📌本文《Beats(PHP-FPM Module@Metricbeat) + Elasticsearch + Kibana 采集分析phpfpm status状态数据》由流水理鱼-wwek原创， 采用 CC BY 4.0 CN协议 许可。可自由转载、引用，但需署名作者且注明文章出处。如转载至微信公众号，请在文末增加作者微信公众号二维码。
微信公众号: 流水理鱼(liushuiliyu)

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注