搜索到
5
篇与
的结果
-
centos7系统直接安装filebeat并传输登录和history到elk 前言{callout color="#f0ad4e"}上一篇文章通过docker启动filebeat，有时候目标服务器无法安装docker，将filebeat直接安装在系统内。 利用elk系统记录分析所有服务器ssh登录信息 利用elk日志分析系统收集history历史命令 {/callout}下载filebeatwget http://dl.itbunan.xyz/filebeat-8.2.0-linux-x86_64.tar.gz解压tar xvf filebeat-8.2.0-linux-x86_64.tar.gz -C /opt/ mv filebeat-8.2.0-linux-x86_64/ filebeatfilebeat.ymlfilebeat.inputs: - type: log enabled: true paths: - /var/log/secure fields: log_type: secure - type: log enabled: true paths: - /var/log/command.log fields: log_type: command output.logstash: hosts: ["IP:5044"] #logstash 服务器 enabled: true worker: 1 compression_level: 3 loadbalance: true close_older: 10m force_close_files: true修改主机名hostnamectl set-hostname wlmq-xxx-xxx-server-125193生成command.log# vim /etc/bashrc HISTDIR='/var/log/command.log' # 定义Command日志的格式 export HISTTIMEFORMAT="{\"TIME\":\"%F %T\",\"HOSTNAME\":\"$HOSTNAME\",\"LI\":\"$(who -u am i 2>/dev/null| awk '{print $NF}'|sed -e 's/[()]//g')\",\"LU\":\"$(who am i|awk '{print $1}')\",\"NU\":\"${USER}\",\"CMD\":\"" # 输出日志到指定的log文件 export PROMPT_COMMAND='history 1|tail -1|sed "s/^[ ]\+[0-9]\+ //"|sed "s/$/\"}/">> ${HISTDIR}' # 创建command.log touch /var/log/command.log && chmod 666 /var/log/command.log启动filebeat# 调试 cd /opt/filebeat ./filebeat -e -c filebeat.yml # 后台启动 screen ./filebeat -e -c filebeat.yml验证是否生效{message type="success" content="访问kibana确认收到数据"/}{card-default label="kibana面板" width="80%"}{/card-default} -
利用elk日志分析系统收集history历史命令 前言{callout color="#f0ad4e"}上一篇文章， 利用elk系统记录分析所有服务器ssh登录信息 。本篇继续收集所有服务器的history命令历史，同时对集群做出优化。{/callout}{card-default label="仪表盘" width="90%"}{/card-default}收集准备{callout color="#f0ad4e"}通过编辑 /etc/bashrc文件，将命令记录保存到文件。然后通过filebeat传输到集群logstash模块，由logstash模块输入到es集群，最后通过kibana展示。{/callout}{message type="success" content="/etc/bashrc 内容"/}HISTDIR='/var/log/command.log' # 定义Command日志的格式 export HISTTIMEFORMAT="{\"TIME\":\"%F %T\",\"HOSTNAME\":\"$HOSTNAME\",\"LI\":\"$(who -u am i 2>/dev/null| awk '{print $NF}'|sed -e 's/[()]//g')\",\"LU\":\"$(who am i|awk '{print $1}')\",\"NU\":\"${USER}\",\"CMD\":\"" # 输出日志到指定的log文件 export PROMPT_COMMAND='history 1|tail -1|sed "s/^[ ]\+[0-9]\+ //"|sed "s/$/\"}/">> ${HISTDIR}' {message type="success" content="创建日志文件"/}touch /var/log/command.log && chmod 666 /var/log/command.logfilebeat{callout color="#f0ad4e"}filebeat部署在需要收集history命令的服务器上。{/callout}{message type="success" content="docker-compose.yml"/}version: "3" services: filebeat: # 容器名称 container_name: filebeat # 主机名称 hostname: wlmq-xxx-server-1 # 镜像 image: elastic/filebeat:8.2.0 # 重启机制 restart: always # 启动用户 user: root # 持久化挂载 volumes: # 映射到容器中[作为数据源] - /var/log/:/log/ # 方便查看数据及日志(可不映射) - /data/ly-elk/filebeat/logs:/usr/share/filebeat/logs - /data/ly-elk/filebeat/data:/usr/share/filebeat/data # 映射配置文件到容器中 - ./conf/filebeat.yml:/usr/share/filebeat/filebeat.yml # 使用主机网络模式 network_mode: host{message type="success" content="filebeat.yml"/}filebeat.inputs: - type: log enabled: true paths: - /log/secure fields: log_type: secure - type: log enabled: true paths: - /log/command.log fields: log_type: command output.logstash: hosts: ["IP:5044"] enabled: true worker: 1 compression_level: 3 loadbalance: true close_older: 10m force_close_files: truelogstash{callout color="#f0ad4e"}logstash由单通道配置改成多通道配置{/callout}{message type="success" content="docker-compose.yml"/}logstash: image: logstash:7.16.2 container_name: logstash restart: always depends_on: - es-master #logstash在elasticsearch启动之后再启动 - es-slave1 environment: - "elasticsearch.hosts=http://es-master:9200" - "xpack.monitoring.elasticsearch.hosts=http://es-master:9200" - "xpack.monitoring.enabled=true" - "TZ=Asia/Shanghai" volumes: - ./conf/pipelines.yml:/usr/share/logstash/config/pipelines.yml - ./conf/logstash/:/usr/share/logstash/pipeline ports: - "5044:5044" #设置端口 networks: - elk{message type="success" content="pipelines.yml"/}- pipeline.id: beats-server config.string: | input { beats { port => 5044 } } output { if [fields][log_type] == 'secure' { pipeline { send_to => secure } } else if [fields][log_type] == 'command' { pipeline { send_to => command } } } - pipeline.id: logstash-filebeat-secure path.config: "/usr/share/logstash/pipeline/logstash-filebeat-secure.conf" - pipeline.id: logstash-filebeat-command path.config: "/usr/share/logstash/pipeline/logstash-filebeat-command.conf"{message type="success" content="logstash-filebeat-secure.conf"/}input { pipeline { address => secure } } filter { grok { match => { "message" => ".*sshd\[\d+\]: %{WORD:status} .* %{USER:username} from.*%{IP:clientip}.*" } } } output { if ([status] == "Accepted" or [status] == "Failed") { elasticsearch { hosts => [ "es-master:9200" ] index => "secure-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } }{message type="success" content="logstash-filebeat-command.conf"/}input { pipeline { address => command } } filter { json { source => "message" } } output { elasticsearch { hosts => [ "es-master:9200" ] index => "command-%{+YYYY.MM.dd}" codec => "json" } stdout { codec => rubydebug } }kibana{callout color="#f0ad4e"}kibana配置仪表板,如文章头部所示{/callout}{message type="success" content="Analytics->Dashboard"/}
-
利用elk系统记录分析所有服务器ssh登录信息 前言{callout color="#f0ad4e"}前几天部署了elk日志分析系统，就想將所有服务器的登录日志统一分析。边使用边熟悉各个模块的配置{/callout}{card-default label="结果展示" width="85%"}{/card-default}filebeat{message type="success" content="修改docker-compose.yml"/}{callout color="#f0ad4e"}hostname:区分不同的主机volumes:需要传输的日志文件映射到容器内{/callout}{message type="success" content="filebeat.yml"/}filebeat.inputs: - type: log enabled: true paths: - /log/secure fields: log_type: secure output.logstash: hosts: ["49.12.10.10:5044"] enabled: true worker: 1 compression_level: 3 loadbalance: true{message type="success" content="启动服务"/}docker-compose up -dlogstash{message type="success" content="logstash-filebeat-secure.conf 配置"/}input { beats { port => 5044 } } filter { grok { match => { "message" => ".*sshd\[\d+\]: %{WORD:status} .* %{USER:username} from.*%{IP:clientip}.*" } } } output { if ([status] == "Accepted" or [status] == "Failed") { elasticsearch { hosts => [ "es-master:9200" ] index => "secure-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } }kibana{message type="success" content="管理索引和生命周期->索引模式->创建索引模式"/}{card-default label="索引模式" width="80%"}{/card-default}{message type="success" content="通过Analytics->Discover查看"/}{card-default label="将字段添加为列" width="75%"}{/card-default}{message type="info" content="最终图如开头所示"/}{message type="success" content="还可以设置仪表盘，更直观的显示信息"/}{card-default label="仪表盘" width="80%"}{/card-default}
-
ElasticSearch常用命令 前言{callout color="#f0ad4e"}Elasticsearch部署完成后,通过接口查看集群状态{/callout}_catcurl 127.0.0.1:9200/_cat{card-default label="结果" width="75%"}=^.^=/_cat/allocation/_cat/shards/_cat/shards/{index}/_cat/master/_cat/nodes/_cat/tasks/_cat/indices/_cat/indices/{index}/_cat/segments/_cat/segments/{index}/_cat/count/_cat/count/{index}/_cat/recovery/_cat/recovery/{index}/_cat/health/_cat/pending_tasks/_cat/aliases/_cat/aliases/{alias}/_cat/thread_pool/_cat/thread_pool/{thread_pools}/_cat/plugins/_cat/fielddata/_cat/fielddata/{fields}/_cat/nodeattrs/_cat/repositories/_cat/snapshots/{repository}/_cat/templates/_cat/ml/anomaly_detectors/_cat/ml/anomaly_detectors/{job_id}/_cat/ml/trained_models/_cat/ml/trained_models/{model_id}/_cat/ml/datafeeds/_cat/ml/datafeeds/{datafeed_id}/_cat/ml/data_frame/analytics/_cat/ml/data_frame/analytics/{id}/_cat/transforms/_cat/transforms/{transform_id}{/card-default}verbose{message type="success" content="?v 参数，显示详细信息"/}# 主节点信息 curl 127.0.0.1:9200/_cat/master/?v # 节点信息 curl 127.0.0.1:9200/_cat/nodes/?vindicescurl 127.0.0.1:9200/_cat/indices/?v{card-default label="结果" width="75%"}{/card-default}
-
基于docker部署elk进行日志管理和分析 前言{callout color="#f0ad4e"}ELK 是 Elasticsearch、Logstrash 和 Kibana 的缩写，它们代表的是一套成熟的日志管理系统，ELK Stack 已经成为目前最流行的集中式日志解决管理方案。{/callout}{message type="success" content="Elasticsearch"/}{callout color="#f0ad4e"}分布式搜索和分析引擎，具有高可伸缩、高可靠和易管理等特点。基于 Apache Lucene 构建，能对大容量的数据进行接近实时的存储、搜索和分析操作{/callout}{message type="success" content="Logstrash"/}{callout color="#f0ad4e"}数据收集引擎。它支持动态的从各种数据源搜集数据，并对数据进行过滤、分析、丰富、统一格式等操作，然后存储到用户指定的位置；{/callout}{message type="success" content="Kibana"/}{callout color="#f0ad4e"}数据分析和可视化平台。通常与 Elasticsearch 配合使用，对其中数据进行搜索、分析和以统计图表的方式展示{/callout}{message type="success" content="Filebeat"/}{callout color="#f0ad4e"}ELK 协议栈的新成员，一个轻量级开源日志文件数据搜集器，基于 Logstash-Forwarder 源代码开发是对它的替代。在需要采集日志数据的服务上安装 Filebeat，并指定日志目录或日志文件后，Filebeat 就能读取日志文件数据，迅速发送到 Logstash 进行解析，或直接发送到 Elasticsearch 进行集中式存储和分析。{/callout}{card-default label="架构图" width="75%"}{/card-default}es集群部署安装dockerdocker和docker-compose一键安装脚本docker-compose.ymlversion: '3' services: es-master: image: elasticsearch:7.16.2 container_name: es-master restart: always user: root environment: # 开启内存锁定 - "bootstrap.memory_lock=true" - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - "TAKE_FILE_OWNERSHIP=true" - "TZ=Asia/Shanghai" ulimits: # 取消内存相关限制 用于开启内存锁定 memlock: soft: -1 hard: -1 volumes: - ./conf/es-master.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /data/ly-elk/master/data:/usr/share/elasticsearch/data - /data/ly-elk/master/logs:/usr/share/elasticsearch/logs - /data/ly-elk/master/plugins:/usr/share/elasticsearch/plugins ports: - "9200:9200" - "9300:9300" mem_limit: 3g networks: - elk es-slave1: image: elasticsearch:7.16.2 container_name: es-slave1 restart: always user: root environment: # 开启内存锁定 - "bootstrap.memory_lock=true" - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - "TAKE_FILE_OWNERSHIP=true" - "TZ=Asia/Shanghai" ulimits: # 取消内存相关限制 用于开启内存锁定 memlock: soft: -1 hard: -1 volumes: - ./conf/es-slave1.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /data/ly-elk/slave1/data:/usr/share/elasticsearch/data - /data/ly-elk/slave1/logs:/usr/share/elasticsearch/logs - /data/ly-elk/slave1/plugins:/usr/share/elasticsearch/plugins ports: - "9201:9201" - "9301:9301" mem_limit: 3g networks: - elk kibana: image: kibana:7.16.2 container_name: kibana restart: always depends_on: - es-master #kibana在elasticsearch启动之后再启动 - es-slave1 environment: - "ELASTICSEARCH_HOSTS=http://es-master:9200" - "I18N_LOCALE=zh-CN" - "TZ=Asia/Shanghai" ports: - "5601:5601" volumes: - ./conf/kibana.yml:/usr/share/kibana/config/kibana.yml networks: - elk logstash: image: logstash:7.16.2 container_name: logstash restart: always depends_on: - es-master #logstash在elasticsearch启动之后再启动 - es-slave1 environment: - "elasticsearch.hosts=http://es-master:9200" - "xpack.monitoring.elasticsearch.hosts=http://es-master:9200" - "xpack.monitoring.enabled=true" - "TZ=Asia/Shanghai" volumes: - ./conf/logstash/:/usr/share/logstash/pipeline ports: - "5044:5044" #设置端口 networks: - elk networks: elk:es-master.yml# 集群名称 cluster.name: es-cluster # 节点名称 node.name: es-master # 是否可以成为master节点 node.master: true # 是否允许该节点存储数据,默认开启 node.data: true # 网络绑定 network.host: 0.0.0.0 # 设置对外服务的http端口 http.port: 9200 # 设置节点间交互的tcp端口 transport.port: 9300 # 集群发现 discovery.seed_hosts: - es-master - es-slave1 # 手动指定可以成为 mater 的所有节点的 name 或者 ip，这些配置将会在第一次选举中进行计算 cluster.initial_master_nodes: - es-master # 支持跨域访问 http.cors.enabled: true http.cors.allow-origin: "*" # 安全认证 xpack.security.enabled: false #http.cors.allow-headers: "Authorization"es-slave1.yml# 集群名称 cluster.name: es-cluster # 节点名称 node.name: es-slave1 # 是否可以成为master节点 node.master: true # 是否允许该节点存储数据,默认开启 node.data: true # 网络绑定 network.host: 0.0.0.0 # 设置对外服务的http端口 http.port: 9200 # 设置节点间交互的tcp端口 transport.port: 9300 # 集群发现 discovery.seed_hosts: - es-master - es-slave1 # 手动指定可以成为 mater 的所有节点的 name 或者 ip，这些配置将会在第一次选举中进行计算 cluster.initial_master_nodes: - es-master # 支持跨域访问 http.cors.enabled: true http.cors.allow-origin: "*" # 安全认证 xpack.security.enabled: false #http.cors.allow-headers: "Authorization"启动docker-compose up -d健康检查{message type="success" content="es 集群状态"/}{card-default label="通过浏览器访问9200" width="75%"}{/card-default}通过filebeat传输日志服务器部署{message type="success" content="基于docker的filebeat在需要收集日志的服务器上单独启动"/}{card-describe title="docker-compose.yml"}version: "3" services: filebeat: # 容器名称 container_name: filebeat # 主机名称 hostname: filebeat # 镜像 image: elastic/filebeat:8.2.0 # 重启机制 restart: always # 启动用户 user: root # 持久化挂载 volumes: # 映射到容器中[作为数据源] - /var/log/messages:/log/messages # 方便查看数据及日志(可不映射) - /data/ly-elk/filebeat/logs:/usr/share/filebeat/logs - /data/ly-elk/filebeat/data:/usr/share/filebeat/data # 映射配置文件到容器中 - ./conf/filebeat.yml:/usr/share/filebeat/filebeat.yml # 使用主机网络模式 network_mode: host{/card-describe}{card-describe title="filebeat.yml"}filebeat.inputs: - type: log enabled: true paths: - /log/messages fields: log_type: syslog output.logstash: hosts: ["127.0.0.1:5044"] enabled: true worker: 1 compression_level: 3 loadbalance: true{/card-describe}{card-describe title="启动"}docker-compose up -d{/card-describe}es集群确认{message type="success" content="确认es是否收到日志"/}# 检测集群是否健康 # green表示正常 curl '127.0.0.1:9200/_cat/health?v' # 列出所有索引 # 是否收到日志 curl '127.0.0.1:9200/_cat/indices?v'{card-default label="结果" width="75%"}{/card-default}kibana 使用访问http://ip:5601 {card-default label="首页" width="75%"}{/card-default}message日志{message type="success" content="可观测性-->日志-->流式传输"/}{card-default label="messages" width="75%"}{/card-default}
