背景#
在日常使用
efk日志系统的过程中，每天造成的日志索引量是巨大的，需要进行对相关的索引进行定期清理，来缓解后端储存的占用。如何实现在 kubernetes 中进行优雅的定期清理？本文将介绍使用 kubernetes 中的Cronjob资源对象进行其需求的实现。
常规情况下清理 n day 以前的数据
curl -XDELETE http://elasticsearch-logging.kube-system:9200/logstash-`date -d"n days ago" +"%Y.%m.%d"`
这种方法，是网上比较常规的方式，如某一天忘记执行了，就会出现数据未清理干净的情况，有点不优雅，如何解决？
进行优化后的 清理策略
cleanPrefixPath='k8s-'
agoCleanTime=`date -d "n days ago" +%s`
for i in `curl http://user:pass@es_url/_cat/indices?v 2>&1 |awk -F '[ ]+' '{print $3}'|sort -n |grep -v '^\.'|egrep "${cleanPrefixPath}[0-9]+.[0-9]+.[0-9]+"`;do
secTime=`echo ${i#*-}|tr '.' '-'|xargs -I {} date -d "{}" +%s`
if [ "${secTime}" -le "${agoCleanTime}" ];then
curl -XDELETE http://user:pass@es_url/"${i}"
fi
done
转换为Yaml资源清单后呢？#
# es-clean-cronjob.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: es-auth-up
data:
user: "elastic"
password: "xxxxx"
es_url: "elasticsearch-data.logging:9200"
---
apiVersion: batch/v1beta1 # or batch/v1
kind: CronJob
metadata:
name: es-clean-job
spec:
schedule: "0 12 * * *" # 每天中午 12:00 执行
jobTemplate:
spec:
template:
spec:
containers:
- name: es-clean-job
image: idocker.io/centos:7
imagePullPolicy: IfNotPresent
env:
- name: ES_USER
valueFrom:
configMapKeyRef:
name: es-auth-up
key: user
- name: ES_PASS
valueFrom:
configMapKeyRef:
name: es-auth-up
key: password
- name: ES_URL
valueFrom:
configMapKeyRef:
name: es-auth-up
key: es_url
command: ["/bin/bash"]
args:
- -c
- |
set -eu
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
cleanPrefixPath='k8s-' # 删除前缀, 设置为空时将对所有索引生效
agoCleanTime=`date -d "30 days ago" +%s` # 删除 30 天以前的日志
for i in `curl http://"${ES_USER}":"${ES_PASS}"@"${ES_URL}"/_cat/indices?v 2>&1 |awk -F '[ ]+' '{print $3}'|sort -n |grep -v '^\.'|egrep "${cleanPrefixPath}[0-9]+.[0-9]+.[0-9]+"`;do
secTime=`echo ${i#*-}|tr '.' '-'|xargs -I {} date -d "{}" +%s`
if [ "${secTime}" -le "${agoCleanTime}" ];then
curl -XDELETE http://"${ES_USER}":"${ES_PASS}"@"${ES_URL}"/"${i}"
fi
done
restartPolicy: Never
backoffLimit: 2
执行创建
kubectl apply -f ./es-clean-cronjob.yaml -n logging
示例的索引结构
执行时的日志输出
