I also get an error when using the “NTP Check”.
5 posts - 3 participants
]]>7 posts - 2 participants
]]>I’m currently managing a small distributed setup where multiple nodes (mix of Linux VMs and a few edge devices) are synchronized using the NTP Pool Project via regional pool servers.
While most systems stay within acceptable offset ranges, I’ve noticed intermittent time drift on a subset of nodes — sometimes jumping beyond 100–200 ms before re-syncing. This becomes noticeable in log correlation and time-sensitive processes.
A few details about the setup:
-
Using default
pool.ntp.orgentries (no dedicated servers) -
Standard NTP client (chrony on Linux, systemd-timesyncd on some nodes)
-
Nodes distributed across different regions with varying network latency
-
No strict firewall blocking, but NAT is involved in some cases
One interesting side effect I ran into: during testing, even minor time offsets caused inconsistencies when syncing timestamps with externally generated assets (e.g., preview files from a video editor; visit this site, used in a separate workflow), which made debugging a bit more confusing than expected.
I’m trying to understand whether this behavior is expected when relying purely on the public pool, or if there are recommended best practices to improve consistency.
4 posts - 3 participants
]]>Seting-up a new test monitor, sudo journalctl -u ntppool-agent@\* -f gives:
Apr 24 08:44:40 tumbleweed.home ntppool-agent[51010]: level=WARN msg="no API key, please run ntppool-agent setup" env=test cmd="ntppool-agent setup --env test --state-dir '/var/lib/ntppool-agent'" wait_time=4m59s
Apr 24 08:44:43 tumbleweed.home ntppool-agent[51010]: level=WARN msg="failed to refresh JWT token" env=test appconfig-manager.err="no API key available for JWT token request"
and that is normal. However, the next step
sudo -u ntpmon ntppool-agent setup -e test -a 355n9ds
gives a 404 error:
tumbleweed:~ # sudo -u ntpmon ntppool-agent setup -e test -a 355n9ds
time=2026-04-24T08:01:30.440Z level=INFO msg="using hostname for registration" env=test hostname=tumbleweed.home
default backend - 404
tumbleweed:~ #
11 posts - 4 participants
]]>Is this related to the recent issues noted on the Status page?
2 posts - 2 participants
]]>Evidence: My server 95.111.202.5 is included in the cn DNS even though the last time its score was above 10 was at 2026-04-22 07:10:23 UTC (around 8 hours ago as of now).
$ dig 0.cn.pool.ntp.org
…
0.cn.pool.ntp.org. 45 IN A 95.111.202.5
$ TZ=UTC date
Wed Apr 22 15:14:36 UTC 2026
Monitoring seems to work, but changes to server’s score status (transition to over 10 or transition to less than 10) do not seem to have an effect.
I guess I’ll need to ping @ask about this.
20 posts - 4 participants
]]>
I’m asking, because I have some issues with https://manage.beta.grundclock.com/ - I can’t add servers and the NTP Check doesn’t work either.
4 posts - 2 participants
]]>After looking through the forum a bit I understand the maintainers are busy, which I completely understand and I thank them for their work providing this extremely helpful resource! But I would appreciate clarification from them or knowledgable members on what the mentioned clause means, before going through the process of getting finance to pay yearly for something they won’t see the immediate need for 
.
7 posts - 3 participants
]]>Before:
ntppool-agent[2643477]: level=INFO msg="batch processing" env=test ip_version=v4 monitor_ip=xxx count=1
ntppool-agent[2643477]: level=INFO msg="batch processing" env=test ip_version=v6 monitor_ip=xxx count=4
ntppool-agent[2643488]: level=INFO msg="ntp error" env=prod ip_version=v6 batchID=01KPD3YJ6BX5VC59F0R686QRA9 server=2a05:dfc1:3ccc:6fe6::123 err="network: i/o timeout"
ntppool-agent[2643488]: level=INFO msg="batch processing" env=prod ip_version=v6 monitor_ip=xxx count=8
ntppool-agent[2643488]: level=INFO msg="batch processing" env=prod ip_version=v4 monitor_ip=xxx count=30
ntppool-agent[2643477]: level=INFO msg="batch processing" env=test ip_version=v6 monitor_ip=xxx count=1
ntppool-agent[2643488]: level=INFO msg="batch processing" env=prod ip_version=v6 monitor_ip=xxx count=10
ntppool-agent[2643488]: level=WARN msg="dns lookup failed" env=prod ip_version=v6 host=uklon5-ntp-002.aaplimg.com err="lookup uklon5-ntp-002.aaplimg.com on 127.0.0.1:53: no such host" trace_id=41014188b08e6c847e9e57311bfd5ed2 span_id=a9368404cbd49289
ntppool-agent[2643488]: level=INFO msg="batch processing" env=prod ip_version=v4 monitor_ip=xxx count=28
ntppool-agent[2643477]: level=INFO msg="batch processing" env=test ip_version=v4 monitor_ip=xxx count=2
ntppool-agent[2643488]: level=INFO msg=local-check env=prod ip_version=v6 failures=0 threshold=3 hosts=7 trace_id=41014188b08e6c847e9e57311bfd5ed2 span_id=a9368404cbd49289
ntppool-agent[2643477]: level=INFO msg="batch processing" env=test ip_version=v4 monitor_ip=xxx count=2
ntppool-agent[2643488]: level=INFO msg="batch processing" env=prod ip_version=v4 monitor_ip=xxx count=12
Now:
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: ntp error
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: dns lookup failed
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: local-check
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: batch processing
ntppool-agent[2936471]: ntp error
So it looks like it’ll log only the “msg” part of the previous log message entries.
4 posts - 3 participants
]]>2001:470:b80f:28::15) which was auto-assigned to the europe and fr zones, but the server is physically located in the US (Texas). The incorrect geolocation is caused by Hurricane Electric’s tunnel broker — the routed /48 prefix (2001:470:b80f::/48) geolocates to France/Europe in IP databases, but the actual server is in North Texas (Vexus Fiber, AS40676 area).
The correct zones should be north-america and us, matching my IPv4 entry (38.28.93.135) which is correctly assigned.
Is there a way to manually correct the zone assignment for this server?
3 posts - 2 participants
]]>4 posts - 3 participants
]]>
21 posts - 4 participants
]]>Ubuntu will use ntpd-rs in future and they want to improve it. Additionally they want to benchmark it against chrony. Sounds interesting.
- Benchmarking & Testing: Comprehensive benchmarking of long-term memory, CPU usage, and synchronization performance against
chronyto give our cloud partners and enterprise users complete confidence in the transition.
22 posts - 6 participants
]]>https://boat.horse/clock/index.html
8 posts - 3 participants
]]>We found that a volunteer who provided hosting for one of our GeoDNS servers used their access to manipulate DNS zone weights for the NTP Pool service domain. The server has been secured and removed from the DNS NS-set.
What happened
One of our geodns servers (ntpmnl1, in Manila) was hosted on a VM provided by a volunteer. When we set up the server, we followed our standard process: full administrative control, firewall rules, locked down access. The volunteer’s SSH key remained on the system from the initial VM provisioning. Later, they asked us to open a firewall exception so they could retrieve personal files from the machine. We made an exception. That was a mistake, and it’s not something we’ll do again.
The volunteer used that access to install a tool that modified the geodns zone data every two minutes, boosting the AAAA record weights of 42 specific IPv6 addresses, all NTP servers the same person had registered in the pool. Many of these servers were already active in the pool in Asian countries, the US, and Mexico.
They also installed a reverse proxy tunnel for persistent remote access and ran a packet capture tool to log IPv6 source addresses of DNS queries hitting the server.
Our configuration system refreshes zone data on geodns servers regularly, so the modifications were overwritten each time. The tool ran every two minutes to re-apply them between refreshes.
What was the actual impact?
The impact was limited. ntpmnl1 was one of many geodns servers and handled only 2-10% of AAAA traffic for any given country. The zone refresh cycle also kept overwriting the modifications, though the two-minute cron was often fast enough to re-inject before the next refresh. Some countries saw clean stretches of several hours (DE had a 5-hour gap on March 19), but it wasn’t a regular pattern.
For the first ~41 hours the tool only boosted weights of IPs that were already present in zone entries, similar to what an operator could do by adjusting netspeed on the pool management website. For countries where these servers weren’t registered (France, Poland, Sweden, Argentina, Nigeria, etc.) the net effect was effectively 0% of total AAAA queries, at most 0.5%. For countries where the servers were already registered (US, SG, AU, BR, etc.) the net effect was higher, around 0.5-7% of total queries, since ntpmnl1 handled 2-10% of each country’s traffic and the volunteer’s servers were heavily favored in its responses.
Later versions of the tool tried to inject IPs into zone entries for countries where the servers weren’t registered. One test ran for 68 minutes before being reverted; the most aggressive version ran for about 20 minutes before being discovered. Neither had much time to take effect.
Even in affected responses, clients usually got a mix of the volunteer’s servers and regular pool servers. In the US, about half of affected queries had all 4 answer IPs from the volunteer; the other half included at least one normal server. In Canada, most affected responses had 3 regular servers and 1 from the volunteer. NTP clients using multiple servers (ntpd, chrony, systemd-timesyncd) would still have had unaffected time sources available.
All 42 servers are registered pool members. Most have the maximum monitoring score of 20, and we have no data indicating NTP queries to these servers weren’t answered accurately. The servers that were most prominent in affected responses matched their labeled geography: US-labeled IPs dominated US queries, SG-labeled IPs dominated Singapore queries, and so on.
None of that changes the fact that what this volunteer did was wildly inappropriate. Tampering with DNS infrastructure that hundreds of millions to billions of devices depend on, regardless of whether the NTP responses were accurate, is a serious breach of trust.
How we run our DNS infrastructure
Our policy is to maintain complete administrative control of our DNS servers. We don’t give outside access to anyone. The servers run either on infrastructure we acquire commercially or on machines hosted by long-standing, trusted community members.
The NTP Pool runs on limited resources. We depend on the community to help us operate, and that means trusting the people we work with. This volunteer had been a pool contributor for over a year, running NTP servers in parts of the world that are poorly served. That track record is why we worked with them and made the exception on the firewall.
Going forward, we’ll be more careful about who we work with on hosting, and we won’t be making exceptions to our access policies. If you’re a long-time pool participant and want to help with DNS server hosting, have a look at pool.ntp.org: NTP Pool DNS servers .
What we’re doing about it
- The server has been secured and removed from the DNS NS-set
- We’re reviewing access controls across all geodns nodes
- We’re looking at the pool account that registered these servers
- We’re adding integrity checks for zone data on geodns servers
- We won’t be granting firewall exceptions for host access going forward
Timeline (UTC, March 2026)
- March 14: Volunteer installs a reverse proxy tunnel on the server for persistent remote access
- March 17-18: DNS packet capture begins
- March 18 08:16: First zone file modification. For the first ~41 hours, the tool only boosted weights of servers already present in zone entries, similar to changing netspeed on the manage website. Impact on countries where the servers weren’t registered was under 0.1%.
- March 19 10:54: A 68-minute test of a more aggressive version that injected IPs into previously empty zone entries. Reverted afterward.
- March 20 07:42: Most aggressive version starts, but only runs for about 20 minutes before discovery.
- March 20 09:00: Data clean across all countries. Server secured and removed from the DNS NS-set.
50 posts - 13 participants
]]>I have been operating NTP servers for many years. Currently, I run a Stratum 1 node powered by a Raspberry Pi 5 with a GPS/PPS reference clock (u-blox NEO-M9N). Real-time operational metrics are available on my public statistics page: https://time.orlinum.fr. I appreciate any feedback regarding this NTP server.
Have a good day.
Orlinum
11 posts - 4 participants
]]>$ curl --connect-timeout 5 --max-time 10 -s ‘https://www.ntppool.org/scores/212.69.48.211/json?limit=1&monitor=24’
{“message”:“Not Found”}
Is there some known issue? Has the API call changed? It has been working fine for many months.
3 posts - 3 participants
]]>
7 posts - 6 participants
]]>7 posts - 4 participants
]]>some of the authoritative DNS servers for pool.ntp.org are unresponsive at least since about two weeks, thus are slowing down name resolution.
Currently (2026-03-12, 22:50Z) the following servers do not respond to DNS queries, tested from multiple locations in Europe and North America:
185.134.197.79(part ofa.ntpns.org), no response77.90.25.251(part ofb.ntpns.org), connection refused160.119.216.201(part ofb.ntpns.org), connection refused2001:43f8:d60:300::201(part ofb.ntpns.org), no response, network not in BGP DFZ2a0e:b107:27f9:123::53(part ofb.ntpns.org), no response, ICMP network-unreachable89.40.214.141(part ofc.ntpns.org), connection refused2600:3c02::f03c:92ff:fe5f:baf1(part ofc.ntpns.org), connection refused2a00:14b0:4200:32e0::1e5(part ofc.ntpns.org), connection refused2a05:91c0:1506:145::(part ofc.ntpns.org), connection refused2a0b:4341:1500:142:5054:ff:fef5:ba1c(part ofc.ntpns.org), connection refused2407:b9c0:f001:3a2:5054:ff:fe83:a2ff(part ofd.ntpns.org), no response51.89.70.90(part ofd.ntpns.org), no response2001:41d0:700:335d::12(part ofd.ntpns.org), connection refused
I have a feeling, some of these servers were shut off by their operators (those that are unresponsive to DNS and ping) – which could pose a risk, if the corresponding IP gets reassigned to some other customer in the future. In that scenario, the new owner of that IP could send malicious replies to questions for the NTP pool.
Those servers, that respond with some sort of ICMP “port-unreachable” or TCP-RST might just have trouble with the DNS service and might just need a restart.
Is this a known problem?
3 posts - 3 participants
]]>I wanted to report a firmware bug in the Marantz SR5015 AV receiver that’s hammering NTP infrastructure.
After a recent firmware update, the device sends NTP-related DNS queries every 20 seconds – 10 requests per cycle:
-
time1.google.com/time2.google.com/time3.google.com/time4.google.com(A + AAAA) -
0.linksys.pool.ntp.org(A + AAAA)
That’s approximately 4,300 DNS lookups per hour or 43,000 per day – per device. After each DNS resolution, it sends the actual NTP request.
I have logs showing 65,000+ queries accumulated in roughly 36 hours.
My network mitigates the impact:
-
AdGuard Home caches the DNS responses (response times ~0.1ms confirm cache hits)
-
dst-nat rules hijack all outbound NTP traffic to my local MikroTik Router
So in my case, neither the pool nor Google’s NTP servers actually see the traffic. But I suspect the vast majority of SR5015 owners don’t have this setup – meaning this bug is likely generating millions of unnecessary requests against pool.ntp.org and time.google.com globally.
The device queries 0.linksys.pool.ntp.org – that’s Linksys’ vendor zone, not Marantz’s. Suggests they’re using inherited code without a proper vendor zone.
I’ve submitted a bug report with 1000 lines of DNS logs to Marantz DE support. No response yet.
Wanted to give you a heads-up in case this is visible in pool traffic patterns or if there’s a way to escalate with vendors.
Cheers
9 posts - 5 participants
]]>
5 posts - 4 participants
]]>503 Service Unavailable
No server is available to handle this request.
13 posts - 8 participants
]]>Monitoring and DNS services appear to be experiencing issues today, but the reported status remains fully green.
49 posts - 21 participants
]]>Any suggestions?
23 posts - 10 participants
]]>Most of the move is already done: for example monitoring, storage, and build servers are running at NetActuate (plus lots of auxiliary systems).
Still need to cut over the web app, databases, and ClickHouse. DNS and NTP won’t be
affected. The website and management interface will have a short maintenance window, and graphs will go stale for most of a day while ClickHouse migrates. I’ll post updates here as things move. Full write-up:
6 posts - 4 participants
]]>14 posts - 5 participants
]]>21 posts - 10 participants
]]>as far as I am aware, the current default server distribution works like this: Each active server is included in the zone for the country it is in, the zone for the continent it is on, and (at sufficient netspeed) the global zone.
When a client asks for servers, the response is:
- if there is at least one server in the country zone of the country the client is asking from, up to four servers from the same country,
- if there are no servers in the country zone but in the encompassing continent zone, up to four servers from the continental zone,
- if there are no servers in neither the country nor the continental zone, up to four servers from the global zone
The main issue for this thread is that the different countries do not balance each other out. In countries with few servers, these are often overwhelmed and sometimes an entire zone collapses when the servers can’t keep up with the request rate and get demoted by the monitoring. Meanwhile in countries with lots of servers, the servers still have capacity left unused. As an interesting side effect, coverage in countries with no servers is sometimes more stable than a country with a small amount of servers since the load gets distributed across the entire continent. All of this has been discussed in different threads here before.
The goal of this thread is to collect and discuss different approaches on how to distribute the available servers across the country zones, so we achieve a better coverage and better utilization of the resources made available in the pool.
Parameters to think about:
- How do we select servers to support underserved zones? Simply by continental grouping? Geographic adjacency of countries?
- What is the impact on the servers used as support? How do we prevent a scenario where we degrade a not yet underserved zone by adding too much load from nearby underserved zones?
- How do we handle the difference between zones? Does an optimization approach fit for large and small zones alike?
- How do we balance higher availability from adding more servers to a zone against higher latency from adding servers that are farther away?
- How do we define which zones are actually “underserved”? A certain amount of DNS requests per server? A certain amount of DNS request in proportion to netspeed? The amount of servers?
- How much complexity do we want to introduce?
I want to explicitly exclude the topic of IPv4 and IPv6 from the scope of this thread. Any server distribution algorithm should work well for both protocol versions, as I assume there will be both IPv4-only and IPv6-only clients for the foreseeable future, where the distribution of servers in one protocol version is irrelevant to the quality of distribution in the other.
I want to present two options that I have thought about. Both are designed purely on the NTP Pool management level without having to introduce new features to the GeoDNS servers.
The first approach was suggested here Minor new features on the website - #9 by ask and is to define a limit on at least how many servers should be in a zone. If there are not enough servers in a given zone, then servers from the surrounding zones could be added.
For the specific implementation: The Pool could calculate the average netspeed of the servers directly mapped to the zone, then add all servers (or at least all that are not themselves in an underserved zone) from the surrounding continent to the zone and scale their netspeeds down so that in the end, they add a total netspeed equal to the total netspeed of the “missing” servers if they all had the average netspeed. Pros: Low complexity, works with information already present in the core Pool database, no change for zones that are already empty or have a huge amount of servers. Cons: Static, might overload other zones in the same continent. Variation: Define a list with the minimum server count per zone based on the DNS statistics to account for differences in Pool usage by country.
Another approach could be to calculate the “load” on a zone based on the amount of DNS requests the zone gets in proportion to the “amount” of netspeed it has. Then the servers from zones with below average load are added to nearby (=same continent?) zones with above average load, again with a scaled down netspeed to support the above-average-loaded zones without needlessly dominating them. This could be mapped again on a continental level, and might even implement that countries that have a lower relative load are weighted to take more support load. Pros: Dynamic, scales with zone usage and server counts. Cons: Need to implement DNS metrics in the Pool management algorithms, zone generation now depends on an “external” factor, might degrade service quality for “below average” zones that were already well served.
For any approach, instead of just using all available servers of the continental zone, we could define a distance matrix, indicating the distance between countries as a simple number abstracting geographical distance and internet interconnections. If a zone needs support, the support is provided by servers from all countries but scaled to weigh servers from nearby countries with a higher netspeed.
What are your thoughts on this topic? Do you have further input or ideas on how to improve the server distribution?
24 posts - 10 participants
]]>7 posts - 4 participants
]]>