| Commit message (Collapse) | Author | Age |
|
|
|
| |
Change-Id: I126075a330f305c85f8fe1b8c9d408f368be95d1
|
|
|
|
|
|
|
| |
Function 'udp_tunnel6_xmit_skb' had been redefined in commit 4d5805d.
Compat hack must be removed to fix compilation issue.
Change-Id: I155b1c45ef57ca2be4fb3f005a5df174fc9041b9
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit cad80597c7947f0def83caf8cb56aff0149c83a8.
Because this commit has not been backported so far, due to the implications
of building Ubuntu's backport of wireguard in a timely manner.
For now, reverting this fix would allow wireguard-linux-compat CI to work
on Ubuntu 18.04.
A different fix or the same one can be applied again when the time is
right.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I5eadc2e7ca7495a012c7f3fb1a9dcd8eabdbe139
|
|
|
|
|
|
|
|
|
|
|
| |
CentOS Stream 8 by now (4.18.0-301.1.el8) reports RHEL_MINOR=5. The
current RHEL 8 minor release is still 3. RHEL 8.4 is in beta. Replace
equal comparison by greater equal to (hopefully) be a little bit more
future proof.
Signed-off-by: Peter Georg <peter.georg@physik.uni-regensburg.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I94a227b5bfcfe7bb7cbadd9caac5d7b5b3f0fd7d
|
|
|
|
|
| |
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: Ic8e394cf71535f494d500b3955d3c0a9df1c655b
|
|
|
|
|
|
|
|
| |
This corresponds to the fancier upstream commit that's still on lkml,
which passes a zeroed ip_options struct to __icmp_send.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I8853014649d6b4595b9f407045991361299d9560
|
|
|
|
|
|
|
|
|
|
| |
linux commit 22f6bbb7bcfcef0b373b0502a7ff390275c575dd ("net: use
skb_list_del_init() to remove from RX sublists") will be backported to Ubuntu
18.04 default kernel, which is based on linux 4.15.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: Ib10df4441e52ce85a4285f41253f2d08dd955a07
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Having two ring buffers per-peer means that every peer results in two
massive ring allocations. On an 8-core x86_64 machine, this commit
reduces the per-peer allocation from 18,688 bytes to 1,856 bytes, which
is an 90% reduction. Ninety percent! With some single-machine
deployments approaching 500,000 peers, we're talking about a reduction
from 7 gigs of memory down to 700 megs of memory.
In order to get rid of these per-peer allocations, this commit switches
to using a list-based queueing approach. Currently GSO fragments are
chained together using the skb->next pointer (the skb_list_* singly
linked list approach), so we form the per-peer queue around the unused
skb->prev pointer (which sort of makes sense because the links are
pointing backwards). Use of skb_queue_* is not possible here, because
that is based on doubly linked lists and spinlocks. Multiple cores can
write into the queue at any given time, because its writes occur in the
start_xmit path or in the udp_recv path. But reads happen in a single
workqueue item per-peer, amounting to a multi-producer, single-consumer
paradigm.
The MPSC queue is implemented locklessly and never blocks. However, it
is not linearizable (though it is serializable), with a very tight and
unlikely race on writes, which, when hit (some tiny fraction of the
0.15% of partial adds on a fully loaded 16-core x86_64 system), causes
the queue reader to terminate early. However, because every packet sent
queues up the same workqueue item after it is fully added, the worker
resumes again, and stopping early isn't actually a problem, since at
that point the packet wouldn't have yet been added to the encryption
queue. These properties allow us to avoid disabling interrupts or
spinning. The design is based on Dmitry Vyukov's algorithm [1].
Performance-wise, ordinarily list-based queues aren't preferable to
ringbuffers, because of cache misses when following pointers around.
However, we *already* have to follow the adjacent pointers when working
through fragments, so there shouldn't actually be any change there. A
potential downside is that dequeueing is a bit more complicated, but the
ptr_ring structure used prior had a spinlock when dequeueing, so all and
all the difference appears to be a wash.
Actually, from profiling, the biggest performance hit, by far, of this
commit winds up being atomic_add_unless(count, 1, max) and atomic_
dec(count), which account for the majority of CPU time, according to
perf. In that sense, the previous ring buffer was superior in that it
could check if it was full by head==tail, which the list-based approach
cannot do.
But all and all, this enables us to get massive memory savings, allowing
WireGuard to scale for real world deployments, without taking much of a
performance hit.
[1] http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I0ff1f036b5cff6d7be89042459e40a86e7e7840c
|
|
|
|
|
|
|
|
|
| |
With the 4.4.256 and 4.9.256 kernels, the previous calculation for
integer comparison overflowed. This commit redefines the broken
constants to have more space for the sublevel.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I38b2b592ce769adbdca56955cac18dbd394d3258
|
|
|
|
|
|
|
| |
We don't need this in all files, and it just complicates things.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: Ie99fab32d50bb636ad7b99e38e05b9698e57e9d8
|
|
|
|
|
| |
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I507bc2de13645d206a771b9e381e06bb453359d1
|
|
|
|
|
| |
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I3bfa3b8eacc6d880e87749e705f3f96096ff9f37
|
|
|
|
|
|
|
|
|
|
| |
The 5.4 series of -rt kernels moved from PREEMPT_RT_BASE/PREEMPT_RT_FULL
to PREEMPT_RT, so we have to account for it here. Otherwise users get
scheduling-while-atomic splats.
Reported-by: Erik Schuitema <erik@essd.nl>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I307514d09a82b40d165eaf92c64f70e76c1bfae2
|
|
|
|
|
| |
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: Ia04067608c91f6c39c71e2df1995ebb1ca73fb3c
|
|
Change-Id: I1c8e130a514a7b0329f8df8099cc84f4cc8d5822
|