ElasticStack从入门到精通

什么是ElasticStack

ElasticStack早期名称为elk

elk代表了三个组件

• ElasticSearch
  负责数据存储和检索。
• Logstash
  负责数据的采集，将源数据采集到ElasticSearch进行存储。
• Kibana
  负责数据的展示。类似Granfa

由于Logstash是一个重量级产品，安装包超过300MB+，很多同学只是用于采集日志，于是使用其他采集工具代替，比如flume，fluentd等产品替代。

后来elastic公司也发现了这个问题，于是开发了一堆beats产品，其中典型代表就是Filebeat，metricbeat，heartbeat等。

而后，对于安全而言，又推出了xpack等相关组件，以及云环境的组件。

后期名称命名为elk stack(elk 技术栈)，后来公司为了宣传ElasticStack

ElasticStack 架构

ElasticStack版本

https://www.elastic.co/ elastic官网

最新版本8+，8版本默认启用了https协议，我们先安装7.17版本，然后手动启动https协议。

后面再练习安装8版本

选择elastic安装方式，我们再Ubuntu上部署elastic

二进制包部署单机es环境

部署

1.下载elk安装包
root@elk:~# cat install_elk.sh 
#!/bin/bash
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-linux-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-7.17.28-linux-x86_64.tar.gz.sha512
tar -xzf elasticsearch-7.17.28-linux-x86_64.tar.gz -C /usr/local
cd elasticsearch-7.17.28/
2.修改配置文件
root@elk:~# vim /usr/local/elasticsearch-7.17.28/config/elasticsearch.yml 
root@elk:~# egrep -v "^#|^$" /usr/local/elasticsearch-7.17.28/config/elasticsearch.yml
cluster.name: xu-elasticstack
path.data: /var/lib/es7
path.logs: /var/log/es7
network.host: 0.0.0.0
discovery.type: single-node
相关参数说明:
	port
	默认端口是9200
	# By default Elasticsearch listens for HTTP traffic on the first free port it
	# finds starting at 9200. Set a specific HTTP port here:
	#
	#http.port: 9200
	
	cluster.name
	集群的名称
	
	path.data
	ES的数据存储路径。
	
	path.logs
	ES的日志存储路径。
	
	network.host
	# elasticStack默认只允许本机访问
	# By default Elasticsearch is only accessible on localhost. Set a different
	# address here to expose this node on the network:
	#
	#network.host: 192.168.0.1
	ES服务监听的地址。
	
	discovery.type
	# 如果部署的es集群就需要配置discovery.seed_hosts和cluster.initial_master_nodes参数
	# Pass an initial list of hosts to perform discovery when this node is started:
	# The default list of hosts is ["127.0.0.1", "[::1]"]
	#
	#discovery.seed_hosts: ["host1", "host2"]
	#
	# Bootstrap the cluster using an initial set of master-eligible nodes:
	#
	#cluster.initial_master_nodes: ["node-1", "node-2"]	
	指的ES集群的部署类型，此处的"single-node"，表示的是一个单点环境。
	
3.如果此时直接启动elastic会报错
3.1测试报错，官方给出的启动命令
Elasticsearch can be started from the command line as follows:
./bin/elasticsearch
root@elk:~# /usr/local/elasticsearch-7.17.28/bin/elasticsearch
# 这些是java类型报错
Mar 17, 2025 7:44:51 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
[2025-03-17T07:44:53,125][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elk] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.28.jar:7.17.28]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.28.jar:7.17.28]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.28.jar:7.17.28]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.28.jar:7.17.28]
	... 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run elasticsearch as root # 不允许root直接启动
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /var/log/es7/xu-elasticstack.log
2025-03-17 07:44:53,713764 UTC [1860] INFO Main.cc@111 Parent process died - ML controller exiting
3.2创建启动用户
root@elk:~# useradd -m elastic
root@elk:~# id elastic 
uid=1001(elastic) gid=1001(elastic) groups=1001(elastic)
# 通过elastic用户启动，此时还有一个报错
root@elk:~# su - elastic -c "/usr/local/elasticsearch-7.17.28/bin/elasticsearch"
could not find java in bundled JDK at /usr/local/elasticsearch-7.17.28/jdk/bin/java
# 系统中是存在java包的，但是elastic用户找不到，切换到elastic查看下
root@elk:~# ll /usr/local/elasticsearch-7.17.28/jdk/bin/java
-rwxr-xr-x 1 root root 12328 Feb 20 09:09 /usr/local/elasticsearch-7.17.28/jdk/bin/java*
root@elk:~# su - elastic 
$ pwd 
/home/elastic
$ ls /usr/local/elasticsearch-7.17.28/jdk/bin/java
# 报错的原因就是权限被拒绝，也就是elastic没有权限访问java包
ls: cannot access '/usr/local/elasticsearch-7.17.28/jdk/bin/java': Permission denied
# 一层一层地向外找，最后能找到/usr/local/elasticsearch-7.17.28/jdk/bin目录没有权限，导致报错
root@elk:~# chown elastic:elastic -R /usr/local/elasticsearch-7.17.28/
root@elk:~# ll -d /usr/local/elasticsearch-7.17.28/jdk/bin/
drwxr-x--- 2 elastic elastic 4096 Feb 20 09:09 /usr/local/elasticsearch-7.17.28/jdk/bin//
# 此时再进行启动测试，发现其他错误
# 我们指定地path.data和path.log不存在，我们需要手动创建
java.lang.IllegalStateException: Unable to access 'path.data' (/var/lib/es7)
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/var/lib/es7)
root@elk:~# install -d /var/{log,lib}/es7 -o elastic -g elastic
root@elk:~# ll -d /var/{log,lib}/es7
drwxr-xr-x 2 elastic elastic 4096 Mar 17 08:01 /var/lib/es7/
drwxr-xr-x 2 elastic elastic 4096 Mar 17 07:44 /var/log/es7/
# 现在重新启动服务，可以启动成功，检测端口
root@elk:~# su - elastic -c "/usr/local/elasticsearch-7.17.28/bin/elasticsearch"
root@elk:~# netstat -tunlp | egrep "9[2|3]00"
tcp6 0 0 :::9200 :::* LISTEN 2544/java 
tcp6 0 0 :::9300 :::* LISTEN 2544/java 

通过浏览器访问9200

同时elastic提供了一个api，可以查看当前的主机数

[root@zabbix ~]# curl 192.168.121.21:9200/_cat/nodes
172.16.1.21 40 97 0 0.11 0.29 0.20 cdfhilmrstw * elk
# 在命令行访问，由于目前的单节点部署es，所以node只有一个
# 前面我们启动es是前台启动前台启动会存在两个问题
1.占用终端
2.如果想结束es比较困难，所以这里一般我们采用后台运行的方式启动
官方给我们的后台运行方式
elasticsearch 的-d参数
To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:
./bin/elasticsearch -d -p pid
root@elk:~# su - elastic -c '/usr/local/elasticsearch-7.17.28/bin/elasticsearch -d'
# 常见报错问题
Q1：最大虚拟内存映射太小
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /var/log/es7/AAA.log
root@elk:~# sysctl -q vm.max_map_count
vm.max_map_count = 65530
root@elk:~# echo "vm.max_map_count = 262144" >> /etc/sysctl.d/es.conf
root@elk:~# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
root@elk:~# sysctl -q vm.max_map_count
vm.max_map_count = 262144
Q2：es配置文件写错
java.net.UnknownHostException: single-node
Q3：出现lock字样说明已经有ES实例启动。杀死现有进程后再重新执行启动命令
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
Q5：ES集群部署的有问题，缺少master角色。
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

卸载环境

1.停止elasticsearch
root@elk:~# kill `ps -ef | grep java | grep -v grep |awk '{print $2}'`
root@elk:~# ps -ef | grep java
root 4437 1435 0 09:21 pts/2 00:00:00 grep --color=auto java
2.删除数据目录、日志目录、安装包、用户
root@elk:~# rm -rf /usr/local/elasticsearch-7.17.28/ /var/{lib,log}/es7/
root@elk:~# userdel -r elastic 

基于deb包安装ES单点

1.安装deb包
root@elk:~# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
2.安装es
root@elk:~# dpkg -i elasticsearch-7.17.28-amd64.deb 
# 通过二进制包安装es可以使用systemctl管理
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
3.修改es配置文件
root@elk:~# vim /etc/elasticsearch/elasticsearch.yml 
root@elk:~# egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: xu-es
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: single-node
4.启动es
systemctl enable elasticsearch --now
# 查看es的service文件，下面的参数都是在二进制安装的时候我们自己做的
User=elasticsearch
Group=elasticsearch
ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quie
cat /usr/share/elasticsearch/bin/systemd-entrypoint 
#!/bin/sh
# This wrapper script allows SystemD to feed a file containing a passphrase into
# the main Elasticsearch startup script
if [ -n "$ES_KEYSTORE_PASSPHRASE_FILE" ] ; then
 exec /usr/share/elasticsearch/bin/elasticsearch "$@" < "$ES_KEYSTORE_PASSPHRASE_FILE"
else
 exec /usr/share/elasticsearch/bin/elasticsearch "$@"
fi

es常见术语

1.索引 Index
	用户进行数据读写的单元
2.分片 Shared
	一个索引至少要有一个分片，如果一个索引仅有一个分片，意味着该索引的数据只能全量存储在某个节点上，且分片是不可拆分的，隶属于某个节点。
	换句话说，分片是ES集群最小的调度单元。
	一个索引数据也可以被分散的存储在不同的分片上，且这些分片可以放在不同的节点，从而实现数据的分布式存储。
3.副本 replica
	副本是针对分片来说的，一个分片可以有0个或多个副本。
	当副本数量为0时，意味着只有主分片(priamry shard)，当主分片所在的节点宕机时，数据就无法访问了。
	当副本数量大于0时，意味着同时存在主分片和副本分片(replica shard):
	主分片负责数据的读写(read write,rw)
	副本分片负责数据的读的负载均衡(read only,ro)
4.文档 document
	指的是用户存储的数据。其中包含元数据和源数据。
	元数据：
	用于描述源数据的数据。
	源数据: 
	用户实际存储的数据。
5.分配： allocation
	指的是将索引的不同分片(包含主分片和副本分片)分配到整个集群的过程。

查看集群状态

# es提供了api /_cat/health
root@elk:~# curl 127.1:9200/_cat/health
1742210504 11:21:44 xu-es green 1 1 3 3 0 0 0 0 - 100.0%
root@elk:~# curl 127.1:9200/_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1742210512 11:21:52 xu-es green 1 1 3 3 0 0 0 0 - 100.0%

es集群环境部署

1.安装es集群服务
root@elk1:~# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
root@elk1:~# dpkg -i elasticsearch-7.17.28-amd64.deb
root@elk2:~# dpkg -i elasticsearch-7.17.28-amd64.deb
root@elk3:~# dpkg -i elasticsearch-7.17.28-amd64.deb
2.配置es，三台机器一样的配置
# 不需要配置discovery.type了
[root@elk1 ~]# grep -E "^(cluster|path|network|discovery|http)" /etc/elasticsearch/elasticsearch.yml 
cluster.name: es-cluster
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.121.91", "192.168.121.92", "192.168.121.93"]
3.启动服务
systemctl enable elasticsearch --now 
4.测试，带有*的是master节点
root@elk:~# curl 127.1:9200/_cat/nodes
172.16.1.23 6 97 25 0.63 0.57 0.25 cdfhilmrstw - elk3
172.16.1.22 5 96 23 0.91 0.76 0.33 cdfhilmrstw - elk2
172.16.1.21 19 90 39 1.22 0.87 0.35 cdfhilmrstw * elk
root@elk:~# curl 127.1:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.16.1.23 9 83 2 0.12 0.21 0.18 cdfhilmrstw - elk3
172.16.1.22 8 96 3 0.16 0.28 0.24 cdfhilmrstw - elk2
172.16.1.21 22 97 3 0.09 0.30 0.25 cdfhilmrstw * elk
# 集群部署故障 没有uuid 集群缺少master
[root@elk3 ~]# curl http://192.168.121.92:9200/_cat/nodes?v
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
[root@elk3 ~]# curl 192.168.121.91:9200
{
 "name" : "elk91",
 "cluster_name" : "es-cluster",
 "cluster_uuid" : "_na_",
 ...
}
[root@elk3 ~]# 
[root@elk3 ~]# curl 10.0.0.92:9200
{
 "name" : "elk92",
 "cluster_name" : "es-cluster",
 "cluster_uuid" : "_na_",
 ...
}
[root@elk3 ~]# 
[root@elk3 ~]# 
[root@elk3 ~]# curl 10.0.0.93:9200
{
 "name" : "elk93",
 "cluster_name" : "es-cluster",
 "cluster_uuid" : "_na_",
 ...
}
[root@elk3 ~]# 
[root@elk3 ~]# curl http://192.168.121.91:9200/_cat/nodes
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
# 解决方式
1.停止集群的ES服务
[root@elk91 ~]# systemctl stop elasticsearch.service 
[root@elk92 ~]# systemctl stop elasticsearch.service 
[root@elk93 ~]# systemctl stop elasticsearch.service 
2.删除数据，日志，和临时数据 
[root@elk91 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
[root@elk92 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
[root@elk93 ~]# rm -rf /var/{lib,log}/elasticsearch/* /tmp/*
3.添加配置项
[root@elk1 ~]# grep -E "^(cluster|path|network|discovery|http)" /etc/elasticsearch/elasticsearch.yml 
cluster.name: es-cluster
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.121.91", "192.168.121.92", "192.168.121.93"]
cluster.initial_master_nodes: ["192.168.121.91", "192.168.121.92", "192.168.121.93"] ######
4.重启服务
5.测试
es集群master选举流程
1.启动时会检查集群是否有master，如果有则不发起选举master;
1.刚开始启动，所有节点均为人自己是master，并向集群的其他节点发送信息（包含ClusterStateVersion，ID等）
2.基于类似gossip协议获取所有可以参与master选举的节点列表;
3.先比较"ClusterStateVersion"，谁最大，谁优先级高，会被选举出master;
4.如果比不出来，则比较ID，谁的ID小，就优先成为master;
5.当集群半数以上节点参与选举完成后，则完成master选举，比如有N个节点，仅需要"(N/2)+1"节点就可以确认master;
6.master选举完成后，会向集群列表通报最新的master节点，此时才意味着选举完成;

DSL

• ES相比与MySQL
  • MySQL属于关系型数据库:
    增删改查： 基于SQL
  • ES属于文档型数据库，和MangoDB很相似
    增删改查： DSL语句，属于ES独有的有种查询语言。
    针对模糊查询，mysql无法充分利用索引，性能较低，而是用ES查询模糊数据，是非常高效的。

往es中添加单条请求

使用postman进行测试

# 本质上是使用了curl
curl --location 'http://192.168.121.21:9200/test_linux/doc' \
--header 'Content-Type: application/json' \
--data '{
 "name": "孙悟空",
 "hobby": [
 "蟠桃",
 "紫霞仙子"
 ]
}
curl --location '192.168.121.21:9200/_bulk' \
--header 'Content-Type: application/json' \
--data '{ "create" : { "_index" : "test_linux_ss", "_id" : "1001" } }
{ "name" : "猪八戒","hobby": ["猴哥","高老庄"] }
{"create": {"_index":"test_linux_ss","_id":"1002"}}
{"name":"白龙马","hobby":["驮唐僧","吃草"]}
'

查询数据

curl --location '192.168.121.22:9200/test_linux_ss/_doc/1001' \
--data ''
curl --location --request GET '192.168.121.22:9200/test_linux_ss/_search' \
--header 'Content-Type: application/json' \
--data '{
 "query":{
 "match":{
 "name":"猪八戒"
 }
 }
}'

删除数据

curl --location --request DELETE '192.168.121.22:9200/test_linux_ss/_doc/1001'

kibana

部署kibana

kibana是针对ES做的一款可视化工具。将来的操作都可以在ES中完成。

1.下载kibana
root@elk:~# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.28-amd64.deb
2.安装kibana
root@elk:~# dpkg -i kibana-7.17.28-amd64.deb
3.修改配置文件
root@elk:~# vim /etc/kibana/kibana.yml 
root@elk:~# grep -E "^(elasticsearch.host|i18n|server)" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.121.21:9200","http://192.168.121.22:9200","http://192.168.121.23:9200"]
i18n.locale: "zh-CN"
4.启动kibana
root@elk:~# systemctl enable kibana.service --now
Synchronizing state of kibana.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable kibana
Created symlink /etc/systemd/system/multi-user.target.wants/kibana.service → /etc/systemd/system/kibana.service.
root@elk:~# netstat -tunlp | grep 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 19392/node 

web端访问测试

基于KQL基本使用

过滤数据

Filebeat

部署Filebeat

1.下载Filebeat
root@elk2:~# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.28-amd64.deb
2.安装Filebeat
root@elk2:~# dpkg -i filebeat-7.17.28-amd64.deb 
3.编写Filebeat配置文件
# Filebeat需要我们自己创建config目录，然后编写配置文件
mkdir /etc/filebeat/config
vim /etc/filebeat/config/01-log-to-console.yaml
# Filebeat配置文件包含两个部分，Input和Output，Input指明要从哪里采集数据；Output指明数据存储到哪里，根据官方文档进行配置
# 目前还没有服务的运行日志，所以Input来源指定为特殊文件，Output去向输出的终端console
root@elk2:~# cat /etc/filebeat/config/01-log-to-console.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /tmp/student.log
# 定义数据到终端
output.console:
 pretty: true
4.运行filebeat实例
filebeat -e -c /etc/filebeat/config/01-log-to-console.yaml
5.创建student.log文件，并写入数据
root@elk2:~# echo ABC > /tmp/student.log
// 输出提示
{
 "@timestamp": "2025-03-18T14:48:42.432Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "7.17.28"
 },
 "message": "ABC", // 检测到的内容变化
 "input": {
 "type": "log"
 },
 "ecs": {
 "version": "1.12.0"
 },
 "host": {
 "name": "elk2"
 },
 "agent": {
 "type": "filebeat",
 "version": "7.17.28",
 "hostname": "elk2",
 "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe",
 "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
 "name": "elk2"
 },
 "log": {
 "offset": 0, // 偏移量offset=0 表示从0开始 
 "file": {
 "path": "/tmp/student.log"
 }
 }
}
# 向文件student.log中追加写入数据
root@elk2:~# echo 123 >> /tmp/student.log
// 查看filebeat输出提示
{
 "@timestamp": "2025-03-18T14:51:17.449Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "7.17.28"
 },
 "log": {
 "offset": 4, // 偏移量从4开始的
 "file": {
 "path": "/tmp/student.log"
 }
 },
 "message": "123", // 统计到了数据 123
 "input": {
 "type": "log"
 },
 "ecs": {
 "version": "1.12.0"
 },
 "host": {
 "name": "elk2"
 },
 "agent": {
 "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
 "name": "elk2",
 "type": "filebeat",
 "version": "7.17.28",
 "hostname": "elk2",
 "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe"
 }
}

Filebeat特性

# 使用echo的静默输出进行追加 # 此时filebeat采集不到数据
root@elk2:~# echo -n 456 >> /tmp/student.log
root@elk2:~# cat /tmp/student.log
ABC
123
456root@elk2:~# 
root@elk2:~# echo -n abc >> /tmp/student.log
root@elk2:~# cat /tmp/student.log
ABC
123
456789abcroot@elk2:~# 
# 使用非静默写入数据 此时filebeat就可以采集到数据了
root@elk2:~# echo haha >> /tmp/student.log
root@elk2:~# cat /tmp/student.log
ABC
123
456789abchaha
// 查看filebeat输出信息
{
 "@timestamp": "2025-03-18T14:55:37.476Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "7.17.28"
 },
 "host": {
 "name": "elk2"
 },
 "agent": {
 "name": "elk2",
 "type": "filebeat",
 "version": "7.17.28",
 "hostname": "elk2",
 "ephemeral_id": "7f116862-382c-48f4-8797-c4b689e6e6fe",
 "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf"
 },
 "log": {
 "offset": 8, // 偏移量=8
 "file": {
 "path": "/tmp/student.log"
 }
 },
 "message": "456789abchaha", // 采集到的数据是静默输出+非静默输出 的所有数据
 "input": {
 "type": "log"
 },
 "ecs": {
 "version": "1.12.0"
 }
}

由此可以得出Filebeat的第一条性质：

filebeat默认是按行采集数据;

# 现在将Filebeat停止，修改student.log，如果重新开始进行数据采集，那么Filebeat是采集文件中所有的内容，还是只采集Filebeat停止后新增的内容
root@elk2:~# echo xixi >> /tmp/student.log
# 重启Filebeat
// 查看Filebeat的输出信息
{
 "@timestamp": "2025-03-18T15:00:51.759Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "7.17.28"
 },
 "ecs": {
 "version": "1.12.0"
 },
 "host": {
 "name": "elk2"
 },
 "agent": {
 "type": "filebeat",
 "version": "7.17.28",
 "hostname": "elk2",
 "ephemeral_id": "81db6575-7f98-4ca4-a86f-4d0127c1e2a4",
 "id": "ba0b7fa3-59b2-4988-bfa1-d9ac8728bcaf",
 "name": "elk2"
 },
 "log": {
 "offset": 22, // 偏移量也不是从0开始
 "file": {
 "path": "/tmp/student.log"
 }
 }, 
 "message": "xixi", // 只采集Filebeat停止后新增的内容
 "input": {
 "type": "log"
 }
}
// 在上面有一条提示，当再次启动filebeat时回导入/var/lib/filebeat/registry/filebeat目录下的json文件
2025-03-18T15:00:51.756Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=5
// 在我们第一次启动时也会从/var/lib/filebeat/registry/filebeat目录下导入，但是第一次启动时不会存在该目录，自然也没有信息
// 查看/var/lib/filebeat/registry/filebeat下的json文件，在这个json文件中记录了offset值，这就是filebeat能够在停止后，重启不会从头开始记录文件内容
{"op":"set","id":1}
{"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","prev_id":"","timestamp":[431172511,1742309322],"ttl":-1,"identifier_name":"native","source":"/tmp/student.log","offset":0,"type":"log","FileStateOS":{"inode":1441831,"device":64768}}}
{"op":"set","id":2}
{"k":"filebeat::logs::native::1441831-64768","v":{"prev_id":"","source":"/tmp/student.log","type":"log","FileStateOS":{"inode":1441831,"device":64768},"id":"native::1441831-64768","offset":4,"timestamp":[434614328,1742309323],"ttl":-1,"identifier_name":"native"}}
{"op":"set","id":3}
{"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","identifier_name":"native","ttl":-1,"type":"log","FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","source":"/tmp/student.log","offset":8,"timestamp":[450912955,1742309478]}}
{"op":"set","id":4}
{"k":"filebeat::logs::native::1441831-64768","v":{"type":"log","identifier_name":"native","offset":22,"timestamp":[478003874,1742309738],"source":"/tmp/student.log","ttl":-1,"FileStateOS":{"inode":1441831,"device":64768},"id":"native::1441831-64768","prev_id":""}}
{"op":"set","id":5}
{"k":"filebeat::logs::native::1441831-64768","v":{"id":"native::1441831-64768","ttl":-1,"FileStateOS":{"device":64768,"inode":1441831},"identifier_name":"native","prev_id":"","source":"/tmp/student.log","offset":22,"timestamp":[478003874,1742309738],"type":"log"}}
{"op":"set","id":6}
{"k":"filebeat::logs::native::1441831-64768","v":{"offset":22,"timestamp":[759162512,1742310051],"type":"log","FileStateOS":{"device":64768,"inode":1441831},"id":"native::1441831-64768","prev_id":"","identifier_name":"native","source":"/tmp/student.log","ttl":-1}}
{"op":"set","id":7}
{"k":"filebeat::logs::native::1441831-64768","v":{"offset":22,"timestamp":[759368397,1742310051],"type":"log","FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","source":"/tmp/student.log","ttl":-1,"identifier_name":"native","id":"native::1441831-64768"}}
{"op":"set","id":8}
{"k":"filebeat::logs::native::1441831-64768","v":{"ttl":-1,"identifier_name":"native","id":"native::1441831-64768","source":"/tmp/student.log","timestamp":[761513338,1742310052],"FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","offset":27,"type":"log"}}
{"op":"set","id":9}
{"k":"filebeat::logs::native::1441831-64768","v":{"source":"/tmp/student.log","timestamp":[795028411,1742310356],"FileStateOS":{"inode":1441831,"device":64768},"prev_id":"","offset":27,"ttl":-1,"type":"log","identifier_name":"native","id":"native::1441831-64768"}}

这是Filebeat的第二个特性

filebeat默认会在"/var/lib/filebeat"目录下记录已经采集的文件offset信息，以便于下一次采集接着该位置继续采集数据;

Filebeat写入es

将Filebeat的Output写入到es中
根据官方文档查看配置
The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API.
Example configuration:
output.elasticsearch: 
 hosts: ["https://myEShost:9200"] 
root@elk2:~# cat /etc/filebeat/config/02-log-to-es.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /tmp/student.log
# 定义数据到终端
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
# 删除filebeat的json文件
root@elk2:~# rm -rf /var/lib/filebeat
# 启动filebeat实例
root@elk2:~# filebeat -e -c /etc/filebeat/config/02-log-to-es.yaml 

在kibana中收集到数据了

查看收集到的数据

设置刷新频率

自定义索引

# 索引名称我们可以自己定义，官方给出了定义方式 index参数设置
output.elasticsearch:
 hosts: ["http://localhost:9200"]
 index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}" 
 
root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /tmp/student.log
# 定义数据到终端
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
 # 自定义索引名称
 index: "test_filebeat-%{+yyyy.MM.dd}"
# 启动filebeat，此时会报错
root@elk2:~# filebeat -e -c /etc/filebeat/config/03-log-to-constom.yaml 
2025-03-19T02:55:18.951Z	INFO	instance/beat.go:698	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2025-03-19T02:55:18.958Z	INFO	instance/beat.go:706	Beat ID: a109c2d1-fbb6-4b82-9416-29f9488ccabc
# 你必须要设置setup.template.name and setup.template.patter这两个参数，也就是说如果我们想自定义index名称，必须设置setup.template.name and setup.template.patter
2025-03-19T02:55:18.958Z	ERROR	instance/beat.go:1027	Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified
Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified
# setup.template.name and setup.template.patter在官网中给出了提示、
 If you change this setting, you also need to configure the setup.template.name and setup.template.pattern options (see Elasticsearch index template).
# 官方给出的实例
setup.template.name
	The name of the template. The default is filebeat. The Filebeat version is always appended to the given name, so the final name is filebeat-%{[agent.version]}.
	
setup.template.pattern
	The template pattern to apply to the default index settings. The default pattern is filebeat. The Filebeat version is always included in the pattern, so the final pattern is filebeat-%{[agent.version]}.
Example:
setup.template.name: "filebeat"
setup.template.pattern: "filebeat"
# 还需要设置shared和replicas 官方给出的默认设置
setup.template.settings:
 index.number_of_shards: 1
 index.number_of_replicas: 1
# 配置我们自己的索引模板(就是创建索引的规则)
root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /tmp/student.log
# 定义数据到终端
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
 # 自定义索引名称
 index: "test_filebeat-%{+yyyy.MM.dd}"
# 定义索引模板(就是创建索引的规则)的名称
setup.template.name: "test_filebeat"
# 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
setup.template.pattern: "test_filebeat-*"
# # 定义索引模板的规则信息
setup.template.settings:
 # 分片数
 index.number_of_shards: 3
 # 每个分片有多少个副本
 index.number_of_replicas: 0
# 启动filebeat 此时可以正常启动filebeat，但是在kibana中发现没有建立我们设置index，查看启动信息
root@elk2:~# filebeat -e -c /etc/filebeat/config/03-log-to-constom.yaml 
# 这里的大概意思就是ILM设置为了auto 如果启用了此配置，则忽略自定义索引的所有信息 所以我们要将ILM设置为false
2025-03-19T03:10:02.548Z	INFO	[index-management]	idxmgmt/std.go:260	Auto ILM enable success.
2025-03-19T03:10:02.558Z	INFO	[index-management.ilm]	ilm/std.go:170	ILM policy filebeat exists already.
2025-03-19T03:10:02.559Z	INFO	[index-management]	idxmgmt/std.go:396	Set setup.template.name to '{filebeat-7.17.28 {now/d}-000001}' as ILM is enabled.
# 查看官网对于index lifecycle management ILM配置
When index lifecycle management (ILM) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}", for example, "filebeat-8.17.3-2025-03-17-000001". Custom index settings are ignored when ILM is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name.
# ilm默认是auto模式，支持true、false、auto
Enables or disables index lifecycle management on any new indices created by Filebeat. Valid values are true, false, and auto. When auto (the default) is specified on version 7.0 and later
setup.ilm.enabled: auto
# 添加ilm配置到我们自己的配置文件中
# 启动filebeat
root@elk2:~# filebeat -e -c /etc/filebeat/config/03-log-to-constom.yaml 

索引模板已被建立

# 此时我想修改我的shared和replicas
# 直接修改配置文件 改为5shared 0repicas

此时还是3shared 0replicas

# 这是由于 setup.template.overwrite 参数 默认是false也就是不覆盖
setup.template.overwrite
A boolean that specifies whether to overwrite the existing template. The default is false. Do not enable this option if you start more than one instance of Filebeat at the same time. It can overload Elasticsearch by sending too many template update requests.
# 设置setup.template.overwrite 为 true
root@elk2:~# cat /etc/filebeat/config/03-log-to-constom.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /tmp/student.log
# 定义数据到终端
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
 # 自定义索引名称
 index: "test_filebeat-%{+yyyy.MM.dd}"
# 禁用索引生命周期管理(index lifecycle management，ILM)
# 如果启用了此配置，则忽略自定义索引的所有信息
setup.ilm.enabled: false
# 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
# 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
setup.template.overwrite: true
# 定义索引模板(就是创建索引的规则)的名称
setup.template.name: "test_filebeat"
# 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
setup.template.pattern: "test_filebeat-*"
# # 定义索引模板的规则信息
setup.template.settings:
 # 分片数
 index.number_of_shards: 5
 # 每个分片有多少个副本
 index.number_of_replicas: 0
# 启动filebeat
# 此时就修改完成了shared和replicas

Filebeat采集nginx实战

1.安装nginx
root@elk2:~# apt install -y nginx
2.启动nginx
root@elk2:~# systemctl start nginx
root@elk2:~# netstat -tunlp | grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 17956/nginx: master 
tcp6 0 0 :::80 :::* LISTEN 17956/nginx: master
3.测试访问
root@elk2:~# curl 127.1
# 日志位置
root@elk2:~# ll /var/log/nginx/access.log 
-rw-r----- 1 www-data adm 86 Mar 19 06:58 /var/log/nginx/access.log
root@elk2:~# cat /var/log/nginx/access.log
127.0.0.1 - - [19/Mar/2025:06:58:31 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
4.编写Filebeat实例
root@elk2:~# cat /etc/filebeat/config/04-log-to-nginx.yaml
# 定义数据从哪里来
filebeat.inputs:
 # 指定数据源的类型是log，表示从文件读取数据
- type: log
 # 指定文件的路径
 paths:
 - /var/log/nginx/access.log*
# 定义数据到终端
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
 # 自定义索引名称
 index: "test_filebeat-%{+yyyy.MM.dd}"
# 禁用索引生命周期管理(index lifecycle management，ILM)
# 如果启用了此配置，则忽略自定义索引的所有信息
setup.ilm.enabled: false
# 如果索引模板存在，是否覆盖，默认值为false，如果明确需要，则可以将其设置为ture。
# 但是官方建议将其设置为false，原因是每次写数据时，都会建立tcp链接，消耗资源。
setup.template.overwrite: true
# 定义索引模板(就是创建索引的规则)的名称
setup.template.name: "test_filebeat"
# 定义索引模板的匹配模式，表示当前索引模板针对哪些索引生效。
setup.template.pattern: "test_filebeat-*"
# # 定义索引模板的规则信息
setup.template.settings:
 # 分片数
 index.number_of_shards: 5
 # 每个分片有多少个副本
 index.number_of_replicas: 0
5.启动filebeat
root@elk2:~# filebeat -e -c /etc/filebeat/config/04-log-to-nginx.yaml 

Filebeat分析nginx日志

filebeat modules

# filebeat support any modules
# 关于filebeat模块官方的解释：他能简化filebeat分析的日志格式
# Filebeat modules simplify the collection, parsing, and visualization of common log formats.
# 默认情况下这些模块都是disable 在需要时我们需要自己设置为enabled
root@elk2:~# ls -l /etc/filebeat/modules.d/
total 300
-rw-r--r-- 1 root root 484 Feb 13 16:58 activemq.yml.disabled
-rw-r--r-- 1 root root 476 Feb 13 16:58 apache.yml.disabled
-rw-r--r-- 1 root root 281 Feb 13 16:58 auditd.yml.disabled
-rw-r--r-- 1 root root 2112 Feb 13 16:58 awsfargate.yml.disabled
。。。
root@elk2:~# ls -l /etc/filebeat/modules.d/ | wc -l
72
# 查看哪些模块处于enabled和disabled
root@elk2:~# filebeat modules list
# 启动模块
root@elk2:~# filebeat modules enable apache nginx mysql redis
Enabled apache
Enabled nginx
Enabled mysql
Enabled redis
# 停止模块
root@elk2:~# filebeat modules disable apache mysql redis
Disabled apache
Disabled mysql
Disabled redis

配置filebeat监控nginx

# 需要在filebeat的配置文件中配置模块功能 在/etc/filebeat/filebeat.yml文件中规定了配置方式
filebeat.config.modules:
 # Glob pattern for configuration loading
 path: ${path.config}/modules.d/*.yml
 # Set to true to enable config reloading
 reload.enabled: false
 # Period on which files under path should be checked for changes
 #reload.period: 10s
module_nginx# 写入filebeat实例
root@elk2:~# cat /etc/filebeat/config/07-module-nginx-to-es.yaml
# Config modules
filebeat.config.modules:
 # Glob pattern for configuration loading 指定在哪个路径下加载
 path: ${path.config}/modules.d/*.yml
 # Set to true to enable config reloading 自动加载/etc/filebeat/modules.d/下面的yml文件。
 reload.enabled: true
 # Period on which files under path should be checked for changes
 #reload.period: 10s
output.elasticsearch: 
 hosts:
 - 192.168.121.21:9200
 - 192.168.121.22:9200
 - 192.168.121.23:9200
 index: "module_nginx-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "module_nginx"
setup.template.pattern: "module_nginx-*"
setup.template.settings:
 index.number_of_shards: 5
 index.number_of_replicas: 0
# 准备nginx日志测试用例
root@elk2:~# cat /var/log/nginx/access.log
192.168.121.1 - - [19/Mar/2025:16:42:23 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 CrKey/1.54.248666"
1.168.121.1 - - [19/Mar/2025:16:42:26 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 CrKey/1.54.248666"
92.168.121.1 - - [19/Mar/2025:16:42:29 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
192.168.11.1 - - [19/Mar/2025:16:42:31 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 13; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Mobile Safari/537.36"
192.168.121.1 - - [19/Mar/2025:16:42:40 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
# 启动filebeat实例
root@elk2:~# filebeat -e -c /etc/filebeat/config/07-module-nginx-to-es.yaml 
# 在收集到的结果中包含正确的日志和错误的日志，我们如果想只收集正确的，需要设置nginx模板的配置文件 /etc/filebeat/module.d/nginx.yml
- module: nginx
 # Access logs
 access:
 enabled: true
 # Set custom paths for the log files. If left empty,
 # Filebeat will choose the paths depending on your OS.
 var.paths: ["/var/log/nginx/access.log"]
 # Error logs
 error:
 enabled: false # 从true改为false
 # Set custom paths for the log files. If left empty,
 # Filebeat will choose the paths depending on your OS.
 #var.paths:
 # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
 ingress_controller:
 enabled: false
 # Set custom paths for the log files. If left empty,
 # Filebeat will choose the paths depending on your OS.
 #var.paths:

kibana分析PV

pv： page view 页面访问量
1个request 就是一个 pv

kibana分析IP

kibana分析带宽

kibana制作Dashboard

kibana分析设备

kibana分析操作系统占比

kibana分析全球用户占比

filebeat采集tomcat日志

部署tomcat

[root@elk2 ~]# wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.5/bin/apache-tomcat-11.0.5.tar.gz
[root@elk2 ~]# tar xf apache-tomcat-11.0.5.tar.gz -C /usr/local
# 配置环境变量
# es本身有jdk环境，我们将es的jdk环境配置到环境变量中，让tomcat调用es的jdk环境
# es的jdk环境目录
[root@elk2 ~]# ll /usr/share/elasticsearch/jdk/
# 添加环境变量
[root@elk2 ~]# vim /etc/profile.d/tomcat.sh
[root@elk2 ~]# source /etc/profile.d/tomcat.sh
[root@elk2 ~]# cat /etc/profile.d/tomcat.sh 
#!/bin/bash
export JAVA_HOME=/usr/share/elasticsearch/jdk
export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.5
export PATH=$PATH:$JAVA_HOME/bin:$TOMCAT_HOME/bin
[root@elk3 ~]# java -version
openjdk version "22.0.2" 2024-07-16
OpenJDK Runtime Environment (build 22.0.2+9-70)
OpenJDK 64-Bit Server VM (build 22.0.2+9-70, mixed mode, sharing)
# 因为tomcat默认的日志格式显示的信息很少，所以这里需要修改tomcat配置文件，修改日志格式
[root@elk3 ~]# vim /usr/local/apache-tomcat-11.0.5/conf/server.xml 
...
 <Host name="tomcat.test.com" appBase="webapps"
 unpackWARs="true" autoDeploy="true">
	<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
 prefix="tomcat.test.com_access_log" suffix=".json"
pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;request&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;http_user_agent&quot;:&quot;%{User-Agent}i&quot;}"/>
 </Host>
# 启动tomcat
[root@elk2 ~]# catalina.sh start 
[root@elk2 ~]# netstat -tunlp | grep 8080
tcp6 0 0 :::8080 :::* LISTEN 98628/java 
# 访问测试
[root@elk2 ~]# cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.121.92 tomcat.test.com
[root@elk2 ~]# cat /usr/local/apache-tomcat-11.0.5/logs/tomcat.test.com_access_log.2025-03-23.json
{"clientip":"192.168.121.92","ClientUser":"-","authenticated":"-","AccessTime":"[23/Mar/2025:20:55:41 +0800]","request":"GET / HTTP/1.1","status":"200","SendBytes":"11235","Query?string":"","partner":"-","http_user_agent":"curl/7.81.0"}

配置filebeat监控tomcat

# 启动tomcat模块
[root@elk3 ~]# fi