监控docker

环境准备

主机名 WanIP 角色
elk01 10.0.0.211 Prometheus服务端
elk02 10.0.0.212 作为docker的被监控端，安装docker，cadvisor
elk03 10.0.0.213 作为docker的被监控端，安装docker，cadvisor

Prometheus监控docker容器环境
1.cadVisor
cadVisor是Google公司开源的一款容器监控工具，支持docker，podman容器管理工具的监控。

GitHub地址:
https://github.com/google/cadvisor

两台需要机器能够翻墙，因为拉取的是谷歌容器镜像仓库

我有cadvisor镜像，所有直接导入就行了

1
2
root@elk02:~# docker load  < cadvisor-amd64-0.49.1.tar.gz 
root@elk03:~# docker load  < cadvisor-amd64-0.49.1.tar.gz 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
后续考虑提供cadvisor镜像

docker run \
  -v /:/rootfs:ro \
  -v /var/run:/var/run:ro \
  -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  -v /dev/disk/:/dev/disk:ro \
  -p 18080:8080 \
  -d  \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor-amd64:v0.49.1
  
  
可以尝使用dockerhub里的cadvisor，镜像换为google/cadvisor:latest

访问测试

1
2
3
4
5
6
7
8
[root@elk03:1 ~]# curl -s 10.0.0.213:18080/metrics | head -3
# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="6f3f25ba",cadvisorVersion="v0.49.1",dockerVersion="",kernelVersion="5.15.0-124-generic",osVersion="Alpine Linux v3.18"} 1


或者浏览器访问
http://10.0.0.212:18080/containers/

prometheus定义要监控的docker主机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
1.修改Prometheus的配置文件
[root@elk02:2 ~]# vim /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 
···
   - job_name: "docker"
      static_configs:
      - targets:
        - 10.0.0.212:18080
        - 10.0.0.213:18080
        
2.检查Prometheus语法是否有误
[root@elk01:0 ~]# /softwares/prometheus-2.53.2.linux-amd64/promtool check config /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 

3.重新发起请求（热加载Prometheus）
[root@elk02 ~]# curl -X POST http://10.0.0.211:9090/-/reload

4.grafana导入对应的模板Id
315
10619

监控数据库

监控MySQL

mysql本身并不支持metrics接口，因此需要独立部署mysql-exporter来提供metrics接口。

这里使用docker部署MySQL，使用Prometheus监控MySQL(不再赘述docker安装部署了)

GitHub地址:

https://github.com/prometheus/mysqld_exporter/releases/download/v0.16.0/mysqld_exporter-0.16.0.linux-amd64.tar.gz

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
1.下载mysql_exporter
[root@elk02:1 ~]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.16.0/mysqld_exporter-0.16.0.linux-amd64.tar.gz

2.拉取镜像
[root@elk02:1 ~]# docker pull mysql:8.3.0-oracle

3.部署MySQL数据库 
[root@elk02:1 ~]# docker run --name mysql-server -t \
             -e MYSQL_DATABASE="linux" \
             -e MYSQL_USER="luay" \
             -e MYSQL_PASSWORD="123" \
             -e MYSQL_ALLOW_EMPTY_PASSWORD="yes" \
             --restart unless-stopped \
			 --network host \
             -d mysql:8.3.0-oracle \
             --character-set-server=utf8mb4 --collation-server=utf8mb4_bin \
             --default-authentication-plugin=mysql_native_password

4.查看运行状态
[root@elk02:1 ~]# docker ps -l
CONTAINER ID   IMAGE                COMMAND                  CREATED         STATUS      
a069ae1367ff   mysql:8.3.0-oracle   "docker-entrypoint.s…"   5 seconds ago   Up 4 seconds

[root@elk02:1 ~]# netstat -lntup|grep 3306
tcp6       0      0 :::3306      :::*      LISTEN      38663/mysqld       
tcp6       0      0 :::33060     :::*      LISTEN      38663/mysqld 

5.解压软件包(后面步骤意思是将mysqld_exporter直接解压到/usr/local/bin/下，跳过一层目录)
tar xf mysqld_exporter-0.15.1.linux-amd64.tar.gz -C /usr/local/bin/ mysqld_exporter-0.15.1.linux-amd64/mysqld_exporter  --strip-components=1

6.创建MySQL的配置，指定默认的用户名和密码
[root@elk02:1 ~]#  cat > /root/.my.cnf <<EOF
[client]
user=luay
password=123
EOF

7.运行mysqld-exporter
[root@elk02:1 ~]# mysqld_exporter --mysqld.address="10.0.0.212:3306" --web.listen-address=:9104 --config.my-cnf="/root/.my.cnf"

8.访问mysqld-exporter的webui
http://10.0.0.212:9104/metrics

9.修改Prometheus的配置文件
[root@elk02:2 ~]# vim /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 
···
   - job_name: "mysql"
      static_configs:
      - targets:
        - 10.0.0.212:9104
        
10.检查Prometheus语法是否有误
[root@elk01:0 ~]# /softwares/prometheus-2.53.2.linux-amd64/promtool check config /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 

11.重新发起请求（热加载Prometheus）
[root@elk02 ~]# curl -X POST http://10.0.0.211:9090/-/reload

12.检查Prometheus的WebUI
http://10.0.0.211:9090/targets

13.grafana导入对应的模板ID
17320
18949
14057

监控redis

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1.拉取镜像
[root@elk02:1 ~]# docker pull redis:7.2.5

2.下载redis_exporter
[root@elk02:1 ~]# wget https://github.com/oliver006/redis_exporter/releases/download/v1.62.0/redis_exporter-v1.62.0.linux-amd64.tar.gz

3.启动redis
[root@elk02:1 ~]# docker run -d --name redis-sever --network host redis:7.2.5 

[root@elk02:1 ~]# netstat -lntup|grep 6379
tcp     0   0 0.0.0.0:6379     0.0.0.0:*   LISTEN      39212/redis-server
tcp6    0   0 :::6379          :::*        LISTEN      39212/redis-server

4.解压软件包 
[root@elk02:1 ~]# tar xf redis_exporter-v1.62.0.linux-amd64.tar.gz -C /usr/local/bin/ redis_exporter-v1.62.0.linux-amd64/redis_exporter --strip-components=1

5.启动redis-exporter
[root@elk92 ~]# redis_exporter -redis.addr redis://10.0.0.212:6379 -web.telemetry-path /metrics -web.listen-address :9121

#查看redis_exporter帮助
[root@elk02:1 ~]# redis_exporter -h

5.浏览器访问
10.0.0.212:9121/metrics

image-20241123135827708

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
1.进入redis数据库
[root@elk02:2 ~]# docker exec -it redis-sever redis-cli --raw -n 1
127.0.0.1:6379[1]> 

2.写入数据
127.0.0.1:6379[1]> set name zhangsan 
OK
127.0.0.1:6379[1]> get name
zhangsan
127.0.0.1:6379[1]> set age 18
OK
127.0.0.1:6379[1]> get age
18
127.0.0.1:6379[1]> keys *
name
age
127.0.0.1:6379[1]>

image-20241123140117078

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
1.修改Prometheus的配置文件
[root@elk02:2 ~]# vim /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 
···
   - job_name: "redis"
      static_configs:
      - targets:
        - 10.0.0.212:9121
        
10.检查Prometheus语法是否有误
[root@elk01:0 ~]# /softwares/prometheus-2.53.2.linux-amd64/promtool check config /softwares/prometheus-2.53.2.linux-amd64/prometheus.yml 

11.重新发起请求（热加载Prometheus）
[root@elk02 ~]# curl -X POST http://10.0.0.211:9090/-/reload

12.检查Prometheus的WebUI
http://10.0.0.211:9021/targets

13.grafana导入对应的模板ID
763
14091
14615