To newer names better reflecting their types and yet sorting together
with folder.go. Doing it now without asking because there are no open
PRs that will get killed by it, and to avoid bikeshedding the names.
The actual pull method (which is really the only thing that differs
between them) is now an interface member which gets overridden by the
subclass.
"Subclass?!" Well, this is dynamic dispatch with overriding, I guess.
Bumping the limit to 2 * the max block size (16 MiB) is a slight
increase compared to previously. Nonetheless I think it's good to allow
us to queue one request and have one on the way in, or conversely have
one large block on the way in and be able to ask for smaller blocks from
others at the same time.
We did this to minimize source size and make it compile faster. Nowadays
byte slice literals compile as fast as anything, and the source isn't
checked in so it doesn't matter how big it is.
Not using base64 avoids a decode step at startup and makes the binary
about 400k smaller.
Two small behavior changes: don't "charge" the data to the global rate
limit until it's been accepted by the device specific limiter, and fix
the send/recv direction in the log print on per device rate limits.
Given that we've taken on the resposibility of maintaining this forked
package I've added it to the Syncthing organization. We still vendor it
like an external package, because it's convenient to keep it as a fork
of upstream to easier merge and file pull requests towards them.
When dropping delta index IDs due to upgrade, only drop our local one.
Previously, when dropping all of them, we would trigger a full send in
both directions on first connect after upgrade. Then the other side
would upgrade, doing the same thing. Net effect is full index data gets
sent twice in both directions.
With this change we just drop our local ID, meaning we will send our
full index on first connect after upgrade. When the other side upgrades,
they will do the same. This is a bit less cruel.
This doesn't happen today, but it might in the future if the block size
were increased or made variable and we were talking to a client from the
future.
The current 500 "test failed" looks and sounds like a problem in the
relay pool server, while it actually indicates a problem on the
announcing side. Instead use 400 "connection test failed" to indicate
that the request was bad and what was the test.
* lib/db: Don't panic on negative counts (fixes#4659)
So, negative counts should never happen and hence the original idea to
panic. However, this sucks as the panic will happen in a folder runner,
be automatically swallowed by suture, and the runner gets restarted but
now we are in a bad state. (Related: #4758)
At the time of writing the global list is somewhat in flux (we've
changed how ignored files are handled, invalid bits, etc.) and I think
that can cause unusual conditions here. Hence just fixing up the numbers
instead until the next full recount.
* release:
lib/osutil: Fix priority lowering on Windows
lib/osutil: Don't attempt to reduce our niceness level (fixes#4681)
lib/osutil: Check PGID before trying to set it (fixes#4679)
This removes a number of timing related things, leaving just the total
test timeout now bumped to one minute. Normally we get the filesystem
events within a second or so, so this doesn't affect the test time in
the successfull case. If we don't actually get the events we expect
within a minute I think we are legitimately in "failed" territory.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4715
LGTM: imsodin, AudriusButkevicius
This is a new revision of the discovery server. Relevant changes and
non-changes:
- Protocol towards clients is unchanged.
- Recommended large scale design is still to be deployed nehind nginx (I
tested, and it's still a lot faster at terminating TLS).
- Database backend is leveldb again, only. It scales enough, is easy to
setup, and we don't need any backend to take care of.
- Server supports replication. This is a simple TCP channel - protect it
with a firewall when deploying over the internet. (We deploy this within
the same datacenter, and with firewall.) Any incoming client announces
are sent over the replication channel(s) to other peer discosrvs.
Incoming replication changes are applied to the database as if they came
from clients, but without the TLS/certificate overhead.
- Metrics are exposed using the prometheus library, when enabled.
- The database values and replication protocol is protobuf, because JSON
was quite CPU intensive when I tried that and benchmarked it.
- The "Retry-After" value for failed lookups gets slowly increased from
a default of 120 seconds, by 5 seconds for each failed lookup,
independently by each discosrv. This lowers the query load over time for
clients that are never seen. The Retry-After maxes out at 3600 after a
couple of weeks of this increase. The number of failed lookups is
stored in the database, now and then (avoiding making each lookup a
database put).
All in all this means clients can be pointed towards a cluster using
just multiple A / AAAA records to gain both load sharing and redundancy
(if one is down, clients will talk to the remaining ones).
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4648
This adds one new feature, that discovery servers can have ?nolookup to
be used only for announces. The default set of discovery servers is
changed to:
- discovery.s.n used for lookups. This is dual stack load balanced over
all discovery servers, and returns both IPv4 and IPV6 results when they
exist.
- discovery-v4.s.n used for announces. This has IPv4 addresses only and
the discovery servers will update the unspecified address with the IPv4
source address, as usual.
- discovery-v6.s.n which is exactly the same for IPv6.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4647
These files always have the symlink bit set, because they are reparse
points. Nonetheless they are not symlinks, and Lstat reports a size for
them. We use this fact to disambiguate, and hope fervently that nothing
else matches this description so it comes back to bite us...
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4622
Also attempt to handle this nicer by ignoring the truncate failure when
it doesn't matter, and recover by deleting the temp file when it does.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4594
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).
The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.
A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
