Add Prometheus HTTP service discovery endpoint for easy discovery of all workers in Docker image #19336
+307
−34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change is now being done in #19325
…rate-config` This is necessary as the Docker image actually uses `--generate-config` to generate the main homeserver config. It's only in worker mode that it uses the other route.
Let `ServerConfig.generate_config_section(...)` figure it out
3 tasks
MadLittleMods
commented
Jan 1, 2026
Comment on lines
+346
to
+361
| NGINX_LOCATION_REGEX_CONFIG_BLOCK = """ | ||
| location ~* {endpoint} {{ | ||
| proxy_pass {upstream}; | ||
| proxy_set_header X-Forwarded-For $remote_addr; | ||
| proxy_set_header X-Forwarded-Proto $scheme; | ||
| proxy_set_header Host $host; | ||
| }} | ||
| """ | ||
| NGINX_LOCATION_EXACT_CONFIG_BLOCK = """ | ||
| location = {endpoint} {{ | ||
| proxy_pass {upstream}; | ||
| proxy_set_header X-Forwarded-For $remote_addr; | ||
| proxy_set_header X-Forwarded-Proto $scheme; | ||
| proxy_set_header Host $host; | ||
| }} | ||
| """ |
Contributor
Author
There was a problem hiding this comment.
Separating regex match vs exact match is necessary because we can't use a URI path in proxy_pass http://localhost:19090/_synapse/metrics with the regex version.
nginx | 2025/12/31 22:58:34 [emerg] 21#21: "proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /etc/nginx/conf.d/matrix-synapse.conf:732
MadLittleMods
commented
Jan 1, 2026
Comment on lines
685
to
+687
| def parse_worker_types( | ||
| requested_worker_types: list[str], | ||
| ) -> dict[str, set[str]]: | ||
| ) -> list[Worker]: |
Contributor
Author
There was a problem hiding this comment.
Previously, this was using a dictionary mapping worker_name -> worker_types
I refactored these functions to use the parsed Worker type as I now need to access the worker_base_name in the new code in order to set the Prometheus job label correctly.
MadLittleMods
commented
Jan 1, 2026
Comment on lines
+1076
to
+1084
| # This allows us to change the `metrics_path` on a per-target basis. | ||
| # We want to grab the metrics from our nginx proxied location (setup | ||
| # below). | ||
| # | ||
| # While there doesn't seem to be official docs on these special | ||
| # labels (`__metrics_path__`, `__scheme__`, `__scrape_interval__`, | ||
| # `__scrape_timeout__`), this discussion best summarizes how this | ||
| # works: https://github.com/prometheus/prometheus/discussions/13217 | ||
| "__metrics_path__": f"/metrics/worker/{worker.worker_name}", |
Contributor
Author
There was a problem hiding this comment.
I wish there was some better docs around this intricacy but I've tested and it does work ✅
For example, we end up with these scrape targets in Prometheus:
Here are some other references (not all good):
- Supporting multiple metrics_path in configuration prometheus/prometheus#1852
- file_sd_config: Support custom metrics_path prometheus/prometheus#4121
- Target ordering inconsistent when using custom metrics path prometheus/prometheus#4001
- Setting
__metrics_path__in thelabelsdoes seem to work. After setting this up, I can see all of the targets as expected in Prometheus (http://localhost:9090/targets)
- Setting
- Discovery for scrape intervals, scheme & paths prometheus/prometheus#910
- Expose more scrape_config parameters via relabeling prometheus/prometheus#1176
- Prometheus special labels prometheus/prometheus#13217
- This is the best consolidated explanation how how
__metrics_path__works
- This is the best consolidated explanation how how
- Proposal: metrics_port in scrape config prometheus/prometheus#9466
- Static config targets don't allow '/' characters prometheus/prometheus#2697 (comment)
- Feature request: metrics_path per target prometheus/prometheus#4037
Contributor
Author
There was a problem hiding this comment.
Made a PR to update the Prometheus docs to make this more clear: prometheus/prometheus#17765
MadLittleMods
commented
Jan 1, 2026
Comment on lines
1117
to
1139
| # Proxy all of the Synapse metrics endpoints through a central place so that | ||
| # people only need to expose the single 9469 port and service discovery can take | ||
| # care of the rest: `/metrics/worker/<worker_name>` -> | ||
| # http://localhost:19090/_synapse/metrics | ||
| # | ||
| # Build the nginx location config blocks | ||
| metrics_proxy_locations = "" | ||
| for worker in requested_workers: | ||
| metrics_proxy_locations += NGINX_LOCATION_EXACT_CONFIG_BLOCK.format( | ||
| endpoint=f"/metrics/worker/{worker.worker_name}", | ||
| upstream=f"http://localhost:{worker_name_to_metrics_port_map[worker.worker_name]}/_synapse/metrics", | ||
| ) | ||
| # Add the main Synapse process as well | ||
| metrics_proxy_locations += NGINX_LOCATION_EXACT_CONFIG_BLOCK.format( | ||
| endpoint="/metrics/worker/main", | ||
| upstream="http://localhost:19090/_synapse/metrics", | ||
| ) | ||
|
|
||
| # Add a nginx server/location to serve the JSON file | ||
| nginx_prometheus_metrics_service_discovery = NGINX_PROMETHEUS_METRICS_SERVICE_DISCOVERY.format( | ||
| service_discovery_file_path=PROMETHEUS_METRICS_SERVICE_DISCOVERY_FILE_PATH, | ||
| metrics_proxy_locations=metrics_proxy_locations, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
If you want to inspect the result of all of this,
- Start your Synapse container per the Testing strategy instructions in the readme
- Jump into the container:
docker exec -it synapse /bin/bash cat /etc/nginx/conf.d/matrix-synapse.conf
/etc/nginx/conf.d/matrix-synapse.conf
docker exec -it synapse /bin/bash
root@ab1f580e63ac:/# cat /etc/nginx/conf.d/matrix-synapse.conf
# This file contains the base config for the reverse proxy, as part of ../Dockerfile-workers.
# configure_workers_and_start.py uses and amends to this file depending on the workers
# that have been selected.
upstream events {
server localhost:18009;
server localhost:18010;
}
upstream event_persister {
server localhost:18009;
server localhost:18010;
}
upstream background_worker {
server localhost:18011;
}
upstream event_creator {
server localhost:18012;
}
upstream user_dir {
server localhost:18013;
}
upstream media_repository {
server localhost:18014;
}
upstream federation_inbound {
server localhost:18015;
}
upstream federation_reader {
server localhost:18016;
}
upstream federation_sender {
server localhost:18017;
}
upstream synchrotron {
server localhost:18018;
}
upstream client_reader {
server localhost:18019;
}
upstream appservice {
server localhost:18020;
}
upstream pusher {
server localhost:18021;
}
upstream device_lists {
server localhost:18022;
server localhost:18023;
}
upstream typing {
server localhost:18024;
}
upstream to_device {
server localhost:18024;
}
upstream presence {
server localhost:18024;
}
upstream account_data {
server localhost:18024;
}
upstream receipts {
server localhost:18024;
}
server {
# Listen on an unoccupied port number
listen 8008;
listen [::]:8008;
server_name localhost;
# Nginx by default only allows file uploads up to 1M in size
# Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
client_max_body_size 100M;
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing {
proxy_pass http://typing;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/sendToDevice/ {
proxy_pass http://to_device;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/ {
proxy_pass http://presence;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/media/ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_synapse/admin/v1/purge_media_cache$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_synapse/admin/v1/room/.*/media.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_synapse/admin/v1/user/.*/media.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_synapse/admin/v1/media/.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_synapse/admin/v1/quarantine_media/.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/v1/media/.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/v1/media/.*$ {
proxy_pass http://media_repository;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* /_matrix/federation/(v1|v2)/send/ {
proxy_pass http://federation_inbound;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/v1/version$ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/event/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/state/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/state_ids/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/backfill/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/get_missing_events/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/publicRooms {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/query/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/make_join/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/make_leave/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/send_join/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/send_leave/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/v1/make_knock/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/v1/send_knock/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/invite/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/query_auth/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/event_auth/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/v1/timestamp_to_event/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/exchange_third_party_invite/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/user/devices/ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/federation/(v1|v2)/get_groups_publicised$ {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/key/v2/query {
proxy_pass http://federation_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$ {
proxy_pass http://user_dir;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/v1/rooms/.*/hierarchy$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(v1|unstable)/rooms/.*/relations/ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/v1/rooms/.*/threads$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/login$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/account/3pid$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/account/whoami$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/account/deactivate$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/devices(/|$) {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3)/delete_devices$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/versions$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/register$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/register/available$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/auth/.*/fallback/web$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/v1/rooms/.*/timestamp_to_event$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/search {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/user/.*/filter(/|$) {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/password_policy$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/directory/room/.*$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/capabilities$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/notifications$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/device_signing/upload$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/signatures/upload$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/unstable/org.matrix.msc4140/delayed_events(/.*/restart)?$ {
proxy_pass http://client_reader;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/.*/tags {
proxy_pass http://account_data;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/.*/account_data {
proxy_pass http://account_data;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(v2_alpha|r0|v3)/sync$ {
proxy_pass http://synchrotron;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$ {
proxy_pass http://synchrotron;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3)/initialSync$ {
proxy_pass http://synchrotron;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ {
proxy_pass http://synchrotron;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt {
proxy_pass http://receipts;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers {
proxy_pass http://receipts;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/knock/ {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location ~* ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ {
proxy_pass http://event_creator;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
# Send all other traffic to the main process
location ~* ^(\\/_matrix|\\/_synapse) {
# note: do not add a path (even a single /) after the port in `proxy_pass`,
# otherwise nginx will canonicalise the URI and cause signature verification
# errors.
proxy_pass http://localhost:8080;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host:$server_port;
}
}
server {
listen 9469;
location = /metrics/service_discovery {
alias /data/prometheus_service_discovery.json;
default_type application/json;
}
# Make the service discovery endpoint easy to find; redirect to the correct spot.
location = / {
return 302 /metrics/service_discovery;
}
location = /metrics/worker/event_persister1 {
proxy_pass http://localhost:19091/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/event_persister2 {
proxy_pass http://localhost:19092/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/background_worker1 {
proxy_pass http://localhost:19093/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/event_creator1 {
proxy_pass http://localhost:19094/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/user_dir1 {
proxy_pass http://localhost:19095/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/media_repository1 {
proxy_pass http://localhost:19096/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/federation_inbound1 {
proxy_pass http://localhost:19097/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/federation_reader1 {
proxy_pass http://localhost:19098/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/federation_sender1 {
proxy_pass http://localhost:19099/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/synchrotron1 {
proxy_pass http://localhost:19100/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/client_reader1 {
proxy_pass http://localhost:19101/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/appservice1 {
proxy_pass http://localhost:19102/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/pusher1 {
proxy_pass http://localhost:19103/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/device_lists1 {
proxy_pass http://localhost:19104/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/device_lists2 {
proxy_pass http://localhost:19105/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/stream_writers1 {
proxy_pass http://localhost:19106/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
location = /metrics/worker/main {
proxy_pass http://localhost:19090/_synapse/metrics;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
}
MadLittleMods
commented
Jan 1, 2026
Comment on lines
+1117
to
+1120
| # Proxy all of the Synapse metrics endpoints through a central place so that | ||
| # people only need to expose the single 9469 port and service discovery can take | ||
| # care of the rest: `/metrics/worker/<worker_name>` -> | ||
| # http://localhost:19090/_synapse/metrics |
Contributor
Author
There was a problem hiding this comment.
(Something for a follow-up)
Instead of serving this endpoint from the random nginx inside the Docker container, it would be interesting to have this as an built-in endpoint in Synapse itself (/_synapse/metrics/service_discovery).
I think the main problem with this is that the workers don't really know about each other at the moment so we couldn't populate the response properly. We don't require the instance_map to be exhaustive.
Showing off how instance_map isn't exhaustive
$ docker exec -it synapse /bin/bash
root@ab1f580e63ac:/# cat /conf/workers/shared.yaml
# This file contains the base for the shared homeserver config file between Synapse workers,
# as part of ./Dockerfile-workers.
# configure_workers_and_start.py uses and amends to this file depending on the workers
# that have been selected.
redis:
enabled: true
enable_media_repo: false
enable_metrics: true
federation_sender_instances:
- federation_sender1
instance_map:
device_lists1:
host: localhost
port: 18022
device_lists2:
host: localhost
port: 18023
event_persister1:
host: localhost
port: 18009
event_persister2:
host: localhost
port: 18010
main:
host: 127.0.0.1
port: 9093
stream_writers1:
host: localhost
port: 18024
listeners:
- bind_address: 127.0.0.1
port: 9093
resources:
- names:
- replication
type: http
- bind_addresses:
- '::'
port: 8080
resources:
- compress: true
names:
- client
- compress: false
names:
- federation
tls: false
type: http
x_forwarded: false
- port: 19090
type: metrics
log_config: /conf/workers/master.log.config
media_instance_running_background_jobs: media_repository1
notify_appservices_from_worker: appservice1
pusher_instances:
- pusher1
run_background_tasks_on: background_worker1
stream_writers:
account_data:
- stream_writers1
device_lists:
- device_lists1
- device_lists2
events:
- event_persister1
- event_persister2
presence:
- stream_writers1
receipts:
- stream_writers1
to_device:
- stream_writers1
typing:
- stream_writers1
update_user_directory_from_worker: user_dir1
And we'd also have to make some assumptions on worker naming in order to set the job correctly (strip off number, etc). It works great here because we are generating the worker naming and the config in the same place here.
…ervice-discovery Conflicts: docker/README-testing.md docker/configure_workers_and_start.py
Discovered while playing with element-hq/synapse-rust-apps#397. Homerunner will map a random port to 9469 (`0.0.0.0:32939->9469/tcp`) which means Prometheus won't be able find it on port 9469 as our service discovery response says it can. Instead, we just reflect back whatever host was used to access to the service discovery endpoint; as if they can access the service discovery endpoint with that host, they can also access all of the metrics on the same host. This is also better in terms of not having to hard-code `host.docker.internal`
MadLittleMods
commented
Jan 2, 2026
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Prometheus HTTP service discovery endpoint for easy discovery of all workers in Docker image.
Follow-up to #19324
Spawning from wanting to run a load test against the Complement Docker image of Synapse and see metrics from the homeserver.
GET http://<synapse_container>:9469/metrics/service_discoveryThe metrics from each worker can also be accessed via
http://<synapse_container>:9469/metrics/worker/<worker_name>which is what the service discovery response points to behind the scenes. This way, you only need to expose a single port (9469) to access all metrics.Real HTTP service discovery response
And how it ends up as targets in Prometheus (http://localhost:9090/targets):
Testing strategy
host.docker.internal) so they can access exposed ports of other Docker containers. We want to allow Synapse to access the Prometheus container and Grafana to access to the Prometheus container.sudo ufw allow in on docker0 comment "Allow traffic from the default Docker network to the host machine (host.docker.internal)"sudo ufw allow in on br-+ comment "(from Matrix Complement testing) Allow traffic from custom Docker networks to the host machine (host.docker.internal)"docker build -t matrixdotorg/synapse -f docker/Dockerfile . && docker build -t matrixdotorg/synapse-workers -f docker/Dockerfile-workers .(docs)prometheus.yml)synapse_build_infoadmin/admin)http://host.docker.internal:9090Dev notes
HTTP service discovery references:
Talking about
jobandinstancelabels:Being able to specify
metrics_pathin service discoverytargets:__metrics_path__in thelabelsdoes seem to work. After setting this up, I can see all of the targets as expected in Prometheus (http://localhost:9090/targets)__metrics_path__works__address__,__scheme__,__metrics_path__,__scrape_interval__,__scrape_timeout__) prometheus/prometheus#17765synapse_build_infosynapse_http_server_requests_received_totalPull Request Checklist
EventStoretoEventWorkerStore.".code blocks.