istio学习笔记(5)-prometheus配置改造
版权声明 本站原创文章 由 萌叔 发表
转载请注明 萌叔 | https://vearne.cc
1. 前言
警告：本文仅用于萌叔自己总结之用，对其它人而言可能毫无营养，没有阅读价值。
要让一个k8s + istio的集群真正能够在生产可用，我们需要考虑如下几类指标。
- Node Metrics
- Container Resource Metrics
- Kubernetes API Server
- Etcd metrics
- Kube state-metrics
对于准备上集群的服务而言，我们会比较关注
1. 服务所在容器的CPU、内存、网络流量、磁盘使用率等
2. 业务指标：QPS、StatusCode、ErrorCode、请求延迟、缓存使用情况，连接池等
其中很大一部分其实都是标准指标，所有的服务都应该会有。另外istio已经对container的输入和输出流量进行了拦截，基于这些条件。萌叔希望达到如下效果
目标
- 1) 通过prometheus的自动发现功能，发现并监控Container Resource Metrics
- 2) 通过prometheus的自动发现功能，发现并监控envoy拦截到的部分指标
- 3）通过prometheus的自动发现功能，发现并监控app暴露的prometheus metrics
- 4) 对于标准指标实现在grafana上的自动配置(生成Dashboard和Graph)
显然对于1）2）中的指标都是标准指标
2. 配置
根据参考资料4的说法
spec:
template:
metadata:
annotations:
prometheus.io/scrape: true # determines if a pod should be scraped. Set to true to enable scraping.
prometheus.io/path: /metrics # determines the path to scrape metrics at. Defaults to /metrics.
prometheus.io/port: 80 # determines the port to scrape metrics at. Defaults to 80.
服务本身如果想暴露自己的指标，可以通过在pod上增加注释prometheus.io/scrape prometheus.io/path prometheus.io/port。 但是实际使用中我发现注入了istio之后，这3项的值已经被修改为
prometheus.io/path: /stats/prometheus
prometheus.io/port: '15020'
prometheus.io/scrape: 'true'
app本身的待监控地址无法正常暴露
另外，envoy通过http://{envoy}:15090/stats/prometheus地址也暴露服务被拦截并采集到的标准指标，有一定的价值。
对于这2部分指标，都应该采集到，所以需要对prometheus的配置进行改造。
3. 配置改造
如果读者是使用的Helm安装的prometheus，那么你会发现，Prometheus实例所在的Pod(名称形如:xxx-prometheus-server-c64894d95-k9fmq)中，有2个容器。
| name | image | comment |
|---|---|---|
| prometheus-server-configmap-reload | jimmidyson/configmap-reload:v0.4.0 | 辅助容器 |
| prometheus-server | prom/prometheus:v2.21.0 | 主容器 |
这2个容器通过共享卷通讯
Deployment 名称形如
# 定义 volume
volumes:
- name: config-volume
configMap:
name: monitor-prometheus-server
defaultMode: 420
# 2个容器都会挂载同一个volume
"volumeMounts": [{
"name": "config-volume",
"readOnly": true,
"mountPath": "/etc/config"
}],
ConfigMap 名字形如 xxx-prometheus-server
kind: ConfigMap
apiVersion: v1
metadata:
name: monitor-prometheus-server
namespace: monitoring
labels:
app: prometheus
chart: prometheus-7.1.8
component: server
heritage: Tiller
release: monitor
data:
alerts: ...
prometheus.yml: ...
rules: |
{}
ConfigMap挂载以后，对应的key、value会被映射成/"/etc/config"中的文件
prometheus-server-configmap-reload负责监视"/etc/config"文件夹，如果文件有发生变化,它会执行webhook，请求 "POST http://127.0.0.1:9090/-/reload", 让prometheus-server重载配置
直接修改 ConfigMap即可增加2个job
# 3.1 负责发现并抓取app暴露的监控端口
- job_name: myk8s-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_myk8s_scrape
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_myk8s_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_myk8s_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
# 3.2 负责发现并抓取envoy暴露的监控端口
- job_name: envoy-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_myk8s_scrape
- action: replace
regex: (.+)
replacement: /stats/prometheus
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?
replacement: $1:15090
source_labels:
- __address__
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
3.1 负责发现并抓取app暴露的监控端口
注意: pod的注释部分需要有
myk8s/scrape: true
myk8s/port: 9090
myk8s/path: /metrics
3.2 负责发现并抓取envoy暴露的监控端口
pod的注释部分需要有
myk8s/scrape: true
参考资料
- A Deep Dive into Kubernetes Metrics
- Prometheus:management API
- prometheus: configuration
- 整合prometheus
请我喝瓶饮料
On this platform, you can find a great variety of online slots from top providers.
Players can experience traditional machines as well as new-generation slots with high-quality visuals and exciting features.
Even if you’re new or a seasoned gamer, there’s something for everyone.
casino games
The games are available 24/7 and designed for desktop computers and tablets alike.
All games run in your browser, so you can jump into the action right away.
The interface is easy to use, making it quick to find your favorite slot.
Join the fun, and enjoy the thrill of casino games!
This website, you can find a great variety of online slots from top providers.
Visitors can try out traditional machines as well as feature-packed games with high-quality visuals and bonus rounds.
Whether you’re a beginner or an experienced player, there’s a game that fits your style.
slot casino
All slot machines are available anytime and optimized for laptops and smartphones alike.
All games run in your browser, so you can get started without hassle.
Site navigation is easy to use, making it simple to explore new games.
Register now, and enjoy the excitement of spinning reels!
Did you know that 1 in 3 medication users commit preventable medication errors due to lack of knowledge?
Your wellbeing is your most valuable asset. Every medication decision you implement plays crucial role in your quality of life. Being informed about medical treatments is absolutely essential for disease prevention.
Your health goes far beyond taking pills. Each drug changes your body’s chemistry in unique ways.
Remember these essential facts:
1. Combining medications can cause dangerous side effects
2. Over-the-counter supplements have strict usage limits
3. Altering dosages causes complications
For your safety, always:
✓ Verify interactions using official tools
✓ Review guidelines in detail before taking new prescriptions
✓ Consult your doctor about potential side effects
___________________________________
For reliable pharmaceutical advice, visit:
https://community.alteryx.com/t5/user/viewprofilepage/user-id/576861
This online pharmacy features a wide range of pharmaceuticals for budget-friendly costs.
Shoppers will encounter both prescription and over-the-counter medicines to meet your health needs.
Our goal is to keep trusted brands while saving you money.
Quick and dependable delivery provides that your purchase arrives on time.
Enjoy the ease of ordering medications online through our service.
fildena 50 reviews
This page features CD player radio alarm clocks from trusted manufacturers.
Browse through premium CD devices with PLL tuner and dual alarms.
Each clock feature external audio inputs, device charging, and power outage protection.
Available products ranges from value picks to high-end designs.
alarm-radio-clocks.com
Every model boast snooze buttons, auto-off timers, and LED screens.
Purchases through Walmart and no extra cost.
Choose the perfect clock-radio-CD setup for bedroom everyday enjoyment.
On this platform, you can access lots of online slots from famous studios.
Visitors can try out retro-style games as well as feature-packed games with high-quality visuals and interactive gameplay.
Even if you’re new or an experienced player, there’s something for everyone.
casino
The games are available anytime and optimized for desktop computers and tablets alike.
All games run in your browser, so you can jump into the action right away.
Site navigation is intuitive, making it quick to find your favorite slot.
Register now, and discover the world of online slots!
The site offers buggy hire on the island of Crete.
Anyone can conveniently reserve a buggy for travel.
Whether you’re looking to discover mountain roads, a buggy is the perfect way to do it.
https://500px.com/p/buggycrete?view=photos
All vehicles are ready to go and can be rented for full-day rentals.
On this platform is fast and comes with affordable prices.
Start your journey and feel Crete from a new angle.
Наличие страхового полиса во время путешествия — это обязательное условие для защиты здоровья туриста.
Документ покрывает неотложную помощь в случае обострения болезни за границей.
Также, полис может обеспечивать возмещение затрат на медицинскую эвакуацию.
страховка за рубеж
Определённые государства предусматривают предъявление страховки для посещения.
Если нет страховки обращение к врачу могут привести к большим затратам.
Покупка страховки перед выездом
Наличие туристического полиса во время путешествия — это необходимая мера для защиты здоровья гражданина.
Сертификат гарантирует расходы на лечение в случае травмы за границей.
К тому же, страховка может включать оплату на транспортировку.
полис каско
Многие страны предусматривают наличие страховки для посещения.
Если нет страховки лечение могут стать дорогими.
Приобретение документа заблаговременно
This website offers you the chance to connect with experts for temporary risky missions.
Visitors are able to quickly set up services for specific needs.
Each professional have expertise in dealing with intense jobs.
rent a hitman
The website provides discreet connections between requesters and freelancers.
When you need fast support, the site is the perfect place.
List your task and find a fit with a professional instantly!
Questo sito offre il reclutamento di persone per compiti delicati.
Gli utenti possono trovare esperti affidabili per incarichi occasionali.
Ogni candidato sono selezionati secondo criteri di sicurezza.
ordina omicidio
Utilizzando il servizio è possibile consultare disponibilità prima di procedere.
La fiducia resta un nostro impegno.
Iniziate la ricerca oggi stesso per portare a termine il vostro progetto!
На данной странице вы можете получить свежую ссылку 1хБет без ограничений.
Оперативно обновляем ссылки, чтобы облегчить свободное подключение к порталу.
Открывая резервную копию, вы сможете пользоваться всеми функциями без рисков.
1xbet зеркало
Наш сайт позволит вам моментально перейти на рабочее зеркало 1хБет.
Мы заботимся, чтобы каждый пользователь имел возможность использовать все возможности.
Проверяйте новые ссылки, чтобы всегда оставаться в игре с 1xBet!
Данный ресурс — официальный онлайн-площадка Bottega Венета с доставкой по стране.
У нас вы можете оформить заказ на фирменную продукцию Bottega Veneta без посредников.
Каждый заказ подтверждены сертификатами от марки.
bottega veneta italy
Отправка осуществляется оперативно в по всей территории России.
Интернет-магазин предлагает выгодные условия покупки и комфортные условия возврата.
Покупайте на официальном сайте Bottega Veneta, чтобы чувствовать уверенность в покупке!
在此页面，您可以雇佣专门从事一次性的危险工作的专业人士。
我们汇集大量训练有素的工作人员供您选择。
不管是何种危险需求，您都可以快速找到合适的人选。
chinese-hitman-assassin.com
所有执行者均经过严格甄别，保证您的机密信息。
任务平台注重安全，让您的个别项目更加高效。
如果您需要服务详情，请直接留言！
At this page, you can browse various CS:GO betting sites.
We offer a wide range of betting platforms centered around the CS:GO community.
All the platforms is thoroughly reviewed to provide reliability.
new csgo betting website
Whether you’re an experienced gamer, you’ll effortlessly discover a platform that matches your preferences.
Our goal is to guide you to connect with only the best CS:GO wagering platforms.
Explore our list today and upgrade your CS:GO gambling experience!
Здесь вы обнаружите всю информацию о программе лояльности: 1win партнерская программа.
Представлены все особенности работы, требования к участникам и потенциальные вознаграждения.
Каждая категория четко изложен, что делает доступным усвоить в аспектах функционирования.
Есть также FAQ по теме и подсказки для новичков.
Информация регулярно обновляется, поэтому вы можете быть уверены в точности предоставленных материалов.
Данный сайт окажет поддержку в исследовании партнёрской программы 1Win.
Questo sito permette il reclutamento di professionisti per incarichi rischiosi.
Chi cerca aiuto possono selezionare candidati qualificati per incarichi occasionali.
Ogni candidato vengono verificati con severi controlli.
assumere un killer
Attraverso il portale è possibile leggere recensioni prima di procedere.
La fiducia resta la nostra priorità.
Iniziate la ricerca oggi stesso per trovare il supporto necessario!
Il nostro servizio offre l’ingaggio di lavoratori per incarichi rischiosi.
I clienti possono scegliere operatori competenti per operazioni isolate.
Le persone disponibili sono valutati con cura.
assumere un killer
Attraverso il portale è possibile visualizzare profili prima della scelta.
La qualità è un nostro impegno.
Sfogliate i profili oggi stesso per ottenere aiuto specializzato!
Looking for qualified contractors available for one-time dangerous projects.
Require a specialist for a perilous task? Discover certified experts via this site for urgent dangerous operations.
hire a killer
This website matches businesses to trained professionals prepared to accept unsafe short-term positions.
Recruit pre-screened freelancers for dangerous jobs securely. Perfect when you need emergency assignments demanding high-risk skills.
On this platform, you can find a great variety of casino slots from top providers.
Visitors can experience traditional machines as well as new-generation slots with vivid animation and interactive gameplay.
Whether you’re a beginner or a casino enthusiast, there’s something for everyone.
play casino
Each title are instantly accessible anytime and compatible with laptops and mobile devices alike.
No download is required, so you can get started without hassle.
Site navigation is user-friendly, making it quick to find your favorite slot.
Register now, and enjoy the excitement of spinning reels!
Humans think about ending their life because of numerous causes, commonly stemming from deep emotional pain.
The belief that things won’t improve may consume their motivation to go on. Often, lack of support is a major factor in this decision.
Conditions like depression or anxiety can cloud judgment, causing people to find other solutions to their pain.
how to commit suicide
Life stressors might further drive an individual to consider drastic measures.
Lack of access to help might result in a sense of no escape. Keep in mind seeking assistance makes all the difference.
欢迎光临，这是一个成人网站。
进入前请确认您已年满成年年龄，并同意遵守当地法律法规。
本网站包含成人向资源，请自行判断是否适合进入。 色情网站。
若不符合年龄要求，请立即关闭窗口。
我们致力于提供合法合规的娱乐内容。
Searching for someone to handle a single dangerous job?
Our platform specializes in connecting clients with freelancers who are ready to tackle serious jobs.
If you’re dealing with emergency repairs, unsafe cleanups, or risky installations, you’ve come to the perfect place.
All listed professional is vetted and qualified to ensure your security.
order a killer
We provide transparent pricing, detailed profiles, and secure payment methods.
No matter how difficult the situation, our network has the skills to get it done.
Begin your search today and locate the ideal candidate for your needs.
You can find here valuable information about techniques for turning into a digital intruder.
Data is shared in a transparent and lucid manner.
You will learn several procedures for entering systems.
What’s more, there are real-life cases that exhibit how to carry out these aptitudes.
how to become a hacker
Whole material is continuously improved to align with the modern innovations in information security.
Extra care is centered around practical application of the mastered abilities.
Note that all operations should be used legally and through ethical means only.
This site you can obtain unique discount codes for one of the leading betting services.
The range of enticing deals is periodically revised to ensure that you always have access to the current bargains.
By utilizing these vouchers, you can save a lot on your betting actions and enhance your possibilities of victory.
All special offers are diligently inspected for legitimacy and efficiency before being displayed.
https://shop.zdravnitza.com/themes/pages/?zagolovok_kak_sbeghaty_iz_tyurymy_i_vyghity.html
Besides, we give comprehensive guidelines on how to redeem each enticing proposal to optimize your profits.
Take into account that some bargains may have unique stipulations or set deadlines, so it’s fundamental to scrutinize carefully all the particulars before activating them.
Hello to our platform, where you can discover premium materials designed specifically for grown-ups.
All the resources available here is appropriate only for individuals who are over 18.
Please confirm that you meet the age requirement before continuing.
suck
Enjoy a one-of-a-kind selection of adult-only materials, and dive in today!
This website contains helpful content about ways of becoming a digital intruder.
The materials are presented in a simple and understandable manner.
It explains a range of skills for bypassing protection.
Moreover, there are real-life cases that exhibit how to employ these competencies.
how to become a hacker
The entire content is regularly updated to align with the up-to-date progress in computer security.
Unique consideration is centered around everyday implementation of the absorbed know-how.
Take into account that all activities should be implemented properly and with good intentions only.
The site makes available various pharmaceuticals for online purchase.
You can conveniently order needed prescriptions with just a few clicks.
Our inventory includes everyday treatments and targeted therapies.
All products is acquired via reliable distributors.
cenforce-100
We maintain quality and care, with secure payments and on-time dispatch.
Whether you’re treating a cold, you’ll find trusted options here.
Begin shopping today and enjoy stress-free support.
This online service features various medications for ordering online.
Anyone can quickly access needed prescriptions from anywhere.
Our catalog includes standard treatments and targeted therapies.
The full range is supplied through reliable distributors.
cenforce 100 side effects
We ensure customer safety, with secure payments and prompt delivery.
Whether you’re looking for daily supplements, you’ll find trusted options here.
Visit the store today and get convenient healthcare delivery.