release54

DragonFly BSD 5.4

Version 5.4.0 released 03 December 2018
Version 5.4.1 released 24 December 2018
Version 5.4.2 released 22 April 2019
Version 5.4.3 released 20 May 2019

DragonFly version 5.4 brings a new system compiler in GCC 8, improved NUMA support, a large number of network and virtual machine driver updates, and updates to video support. This release is 64-bit only, as with previous releases.

DragonFly version 5.4.1 has significant HAMMER2 improvements (meta-consistency protection for snapshots and crashes, better on-media topology), better WLAN interface handling, and improved WLAN connection with dhcpcd(8).

DragonFly 5.4.2 has a fix for a rare potential HAMMER2 corruption problem. Other minor 5.4.2 changes are marked in the full notes below.

DragonFly 5.4.3 has a fix for an Intel hardware floating-point bug.

The details of all commits between the 5.2 and 5.4 branches are available in the associated commit messages for 5.4.0rc, 5.4.0, 5.4.1, 5.4.2 and 5.4.3.

Big-ticket items

Much better support for asymmetric NUMA (Non-Uniform Memory Access) configurations. In particular, both the memory subsystem and the scheduler now understand the Threadripper 2990WX's architecture. The scheduler will prioritize CPU nodes with direct-attached memory and the memory subsystem will normalize memory queues for CPU nodes without direct-attached memory (which improves cache locality on those CPUs).
Incremental performance work. DragonFly as a whole is very SMP friendly. The type of performance work we are doing now mostly revolves around improving fairness for shared-vs-exclusive lock clashes, reducing cache ping-ponging due to non-contending SMP locks (i.e. massive use of shared locks on shared resources), and so forth.
Major updates to dports brings us to within a week or two of FreeBSD's ports as of this writing, in particular major updates to chromium, and making the whole mess work with gcc-8.
Major rewriting of the tty clist code and the tty locking code, significantly improving concurrency across multiple ttys and ptys.

GCC 8

DragonFly now ships with GCC 8.0, and runs as the default compiler. It is also now used for building dports.
GCC 4.7.4 and GCC 5.4.1 are still installed. 4.7.4 is our backup compiler, and 5.4.1 is still there to ensure a smooth transition, but should generally not be used. buildworld builds all three by default to ensure maximum compatibility.
Many passes through world sources were made to address various warnings and errors the new GCC brought with it.

HAMMER2

HAMMER2 is recommended as the default root filesystem in non-clustered mode.
Clustered support is not yet available.
Increased bulkfree cache to reduce the number of iterations required.
Fixed numerous bugs.
Improved support on low-memory machines.
Significant pre-work on the XOP API to help support future networked operations.
(in 5.4.1) HAMMER2 filesystem meta-consistency protection for snapshots and crashes has been improved, as has speed of snapshot generation. See full notes at the end of the document

Details

Checksums

MD5 (dfly-x86_64-5.4.3_REL.img) = 8ba26d359444aa785ed01fc0d08cc281
MD5 (dfly-x86_64-5.4.3_REL.iso) = 3e5e5416aee739091e87c6805e50f821
MD5 (dfly-x86_64-5.4.3_REL.img.bz2) = 4303314a0cbcaec7f82e5bd0ad635122
MD5 (dfly-x86_64-5.4.3_REL.iso.bz2) = b78d943e916eeed735771476e4aad18f

Upgrading

If you have an existing 5.2.x or 5.4.x system and are running a generic kernel, the normal upgrade process, described below, will work.

Change your local /usr/src to 5.4:

cd /usr/src
git fetch origin
git branch DragonFly_RELEASE_5_4 origin/DragonFly_RELEASE_5_4
git checkout DragonFly_RELEASE_5_4
git pull

And then rebuild: (in /usr/src)

make buildworld
make buildkernel
make installkernel
make installworld
make upgrade

(reboot your system)

make initrd

make initrd is optional for systems with unencrypted disks, and has been possible for several releases; it's mentioned here for completeness. Look at the initrd(7) man page for details on what it does.

Don't forget to upgrade your existing packages. 5.4 packages have already been built and are immediately available.

pkg update
pkg upgrade

All changes since DragonFly 5.2

Security Issues

Fix CVE-2018-8897, debug register issue.
Complete the full Spectre mitigation support. Default disabled due to extreme performance loss. The machdep.spectre_support sysctl can be used to probe the spectre support, and machdep.spectre_mitigation sysctl can be used to enable/disable support (can be done at runtime).
Change the default /root perms from 755 to 700 in the build template.
OpenBSD collab - completely remove delayed FP state to avoid the known side-channel attack.
OpenBSD collab - proactively clean FP state on switch to avoid known side-channel attack.
OpenBSD collab - proactively zero user registers on entry into kernel (syscall, interrupt, or exception) to avoid certain speculative side-channel attacks.

Kernel

drm has been updated to match Linux kernel 4.7.10 in a number of locations.
The radeon driver has been updated; currently matches Linux 3.18.
CVE-2018-8897 mitigated.
x2apic timer support added.
Add a private_data field to struct file to improve application support.
Fix pageout/sbusy race.
Refactor AcpiOs*Lock() operation to use lockmgr locks to reduce freezes and improve debugability.
Improve SPINLOCK and acpi_timer performance when under emulation.
nvme - Improve the likelihood of dump success by removing certain blocking locks from the dump path.
Numerous spinlock performance and edge-case improvements. Deal with shared starvation in the face of a long series of exclusive locks, and vise-versa. Add TSC-based windowing as a fall-back when excessive contention is present.
Add a dirty vnode management facility.
Remove bottlenecks from the rlimit handling code by reflecting p_limit into td_limit.
Increase the size of the vm_object hash table by 4x to reduce collisions.
Fix duplicate da* reporting at boot time and fix the related lockup.
Implement a per-thread (fd,fp) cache to reduce cache ping-ponging in heavily-threaded processes.
Refactor the lwkt_token pool hash algorithm to improve concurrency.
Refactor uidinfo
Refactor lockf operation to remove SMP bottlenecks.
Increase copyin item batching from 8 to 32 for select() and poll().
Improve concurrent tmpfs use by removing SMP bottlenecks.
Improve allocvnode() by removing SMP bottlenecks.
Improve common file descriptor internals for better SMP operation.
Improve namecache performance.
Improve scheduler concurrency by addressing some of the more blatent cache ping-ponging.
Remove numerous kernel global debugging and (unused) statistics counters that were causing cache ping-ponging.
Carefully refactored contended tokens and spinlocks for better SMP operation on many-cores systems.
Carefully refactored the mutex contention code.
Fix a kfree() statistics accounting bug.
Improves kfree() SMP operation.
Fix legacy IRQ bug by ensuring that legacy IRQs only appear on one CPU.
Fix a multitude of usched issues that could cause unrelated processes to be misscheduled.
Refactor the nice priority calculations to improve expected relative behavior when multiple cpu-bound processes are NICEd to different values.
Refactor machdep.cpu_idle_hlt for better AMD handling.
Rearrange pmap PV caching a bit to improve performance.
Optimize the syscall path to improve performance.
Rearrange a few structures to localize fields for cache line optimization.
Remove vm_zeroidle.c. Remove the asynchronous page zeroing code. Instead, we now zero pages on demand, just before they are used, to reduce unnecessary cache interactions and to reduce a long-time debuggability issue with pre-zero'd pages.
X2APIC support added and tested.
ACPI updated from Intel base.
Fix a swapcache cleaning issue that could cause swapcache to behave in a non-useful manner.
Change closefrom() EINTR behavior. closefrom() no longer breaks out of its loop when EINTR is received from close() (which could occur with NFS descriptors). EINTR will still be rolled up and returned (callers usually ignore this).
Fix a rare pmap_scan_callback() panic.
Fix a umtx race in the kernel.
Fix a NULL pointer indirection on lost socket bug.
Enable NX for PROT_READ-only mappings by default. NX cannot be enabled for PROT_WRITE (without PROT_EXEC) due to assumptions made by applications. Suggested by OpenBSD.
Refactor low-memory handling in the objcache subsystem.
Bring in some jail work from FreeBSD.
Rewrite the TSC probe code several times to improve boot times and reduce SMP issues on many-cores cpus and under emulation.
Refactor the if_clone code and improve stability when used with TAP.
Increase MAXCPUFIFO from 32 to 256 to fix deadlocks which can occur on ncpus > 32 systems.
AMD topology detection rewritten.
Add a heuristic to the scheduler to prioritize the 'better' cores in assymetric NUMA configurations (aka threadripper 2990WX).
Add memory-on-node weighting to the schedule which detects assymetric NUMA configurations and assigns memory to nodes with too little actual memory. This gives us better memory isolation on nodes without direct-attached memory, significantly improving their performance.
Fix missing wakeup()s in kern_lock.c that could (rarely) lead to unkillable blocked processes.
Fix a startup race in usched_dfly which sometimes caused early boot panics.
Fix a missing brelse() in nvtruncbuf()'s error path which could deadlock the system on certain filesystem errors.
Fix an early-boot interrupt race that sometimes caused a null-pointer deref panic.
Optimize bcopy, bzero, memset, part 1. part 2 didn't quite make it into this release.
Improve kernel performance by macroing trivial mem*() and bcopy/bzero*() operations with known, small, static sizes.
Improve kernel performance
Add wait6(), waitid(), and si_pid/si_uid siginfo support. This fixes the 'lightdm' X display manager.
Move nearly all global tty_token uses to per-tty or per-pty tokens, significantly improving concurrency when writing to multiple ttys or ptys. Remove unnecessary critical sections.
Remove use of tty_token in the session management code, removing an annoying bottleneck from the fork/exec/wait/exit path.
The keyboard subsystem now uses its own kbd_token instead of using tty_token.
Rewrite the tty clist code. Get rid of clists. Instead, use a single linear buffer and a FIFO mechanism. The original clist code was obsolete 20 years ago.
Fix a kernel exit race in killalllwps() that could sometimes prevent a heavily threaded process from completing its exit().
(5.4.3) Intel floating-point bug worked around.

Filesystems

autofs - Various updates to autofs
ufs - Fix an old UFS bug that would panic on certain filesystem full conditions.
ufs - Rename UFS constants throughout to improve separation.
ufs - Remove duplicate TRIMs and significantly improve 'rm -rf' performance.
tmpfs - Fix NFS exports of tmpfs filesystems.
tmpfs - Significantly improve tmpfs concurrency.
tmpfs - Fix a rare deadlock.
Fix minor bugs in HAMMER1 that could result in an assertion under heavy loads.
HAMMER2 is now the recommended default filesystem for root.
Fix ENOTDIR/EISDIR logic in certain situations.
uid, gid, and (fake) inum added to the fstat() information for pipe()s and socket()s.

Networking

The network tunnel driver, tun(4), has been cleaned up and updated. It's now clonable for anyone building VPN links.
The tap(4) and interface cloning have also received related updates and fixes.
Fix an arp issue in the bridge code, plus add a few bug fixes.
Interface groups are now supported in the kernel and pf(4). They can be configured with ifconfig(8)
DragonFly now can use dhcpcd(8) for DHCP, for both IPv4 and IPv6. See the announcement for details. The existing utilities dhclient(8) and rtsold(8) are still supported, so no configuration change is required yet.
The ena (Elastic Network Adapter) network driver has been added.
if_iwm has had multiple updates.
if_vtnet, the virtual I/O Ethernet driver, has been updated.
Significant work to ipfw3.
Add BPF_MOD and BPF_XOR support to bpf, and update libpcap.
Fix a user-after-free mbuf issue - obtained from NetBSD via FreeBSD.
IPSEC has been removed from the system, for many reasons. Basically, VPNs are far superior and more deterministic in terms of maintainability.
ifconfig(8) - synchronized from FreeBSD.
Reduce confusion by giving SEQPACKET sockets their own sysctls for limit setting.
Increase all default limits for unix domain sockets to 65536 (fixes issues with wpa_supplicant and provides better defaults against application expectations).
Add ip6addrctl(8) from FreeBSD.
Fix a SACK related NULL pointer dereference.
dhcpcd has been added to the base system (for IPV6), and will replace dhclient and rtsold - though not in this release.
A number of network device drivers have been added.
A couple of edge-case panics have been fixed.
wpa_supplicant is installed via dports, by default, to override the default version in base. The base version remains so that a mass pkg deletion doesn't accidentally remove the ability to get online.
Fix getsockname() on unnamed AF_LOCAL sockets to properly initialize the sockaddr.
Set net.inet6.icmp6.nd6_onlink_ns_rfc4861 to 1 by default, contemporary implementations pretty much require it.
Fix a misunderstanding in how igb(4) and ix(4) calculate the packet hash for encapsulated packets.

Driver updates

A number of drivers of use when running DragonFly as a guest VM have been added or updated, including ena (for AWS support), if_vtnet, virtio_balloon, and virtio_pci.
Improvements for DRM and radeon.
Serial-output-only installs are now possible.
The virtio_balloon memory driver has been added.
/dev/sndstat can now be opened multiple times by the same device.
mpr(4) driver for LSI Fusion-MPT 3/3.5 SAS controllers added.
MosChip PCIe serial communications now supported.
Add missing descriptions for usb4bsd C610/X99 controllers
igb(4) bring in WOL (wake-on-lan) support from FreeBSD.
Add support for PCIe serial com, and console support.
Remove numerous old PCI and ISA serial drivers.
nvme(4) fixes for AWS. Handle interfaces without attached volumes.
corepower(4) - Add support for platform energy counter.
ig4(4) Numerous changes to improve low-power support and stability. Handle situations where ig4(4) is listed but is non-operational (probably because the BIOS has stolen it).
dsp(4) fixes for mmap() handling.
sound(4) now allows /dev/sndstat to be open multiple times from one process.
(5.4.2) AC256 (sound) quirk added.

Userland

Refactor /usr/Makefile (for src-create and other targets).
Refactor buildworld, in particular moving the initrd build from installworld to buildworld
Add rc support for ipfw3
Major refactor of the initrd infrastructure.
Update pwcache(3) from NetBSD
Getty - minor sync from FreeBSD
Update vis(3) and unvis(3) from NetBSD
Update pciconf database.
Fix -Walloc-size-large-than bug in gcc-4.7 and 5.0.
Remove rhosts from the default template (the rcmds, like rcp, rlogin, etc. were removed from the tree in the previous release).
Fix a signed overflow in mktime() that blew up GCC-8's -O2 constant folding code.
Add strsuftoll,x to libc in preparation for makefs(8). This helps various dports as well. Taken from NetBSD.
Add tcsetsid() to libc.
Adjust symlink() prototype to match standards.
Improve buildworld concurrency when building the compilers.
Significant crunchgen updates. Add 70 more tools to the crunch.
Add -N option to install(1).
The resolver now allows underscores in the name, fixing numerous domains and domain forwarding issues.
mtree - Replace with NetBSD's version.
dumpon(8) - Improve error message clarity.
Cache 4K, 8K, and other larger allocations in posix_memalign(), significantly improving its performance.

Boot

The initrd has had significant improvement and now forms a much more complete rescue system, with over 70 tools added via crunchgen(1). See the initrd(7) man page for details.
Crypto bootstrap, fix vfs.root.real_root to be vfs.root.realroot.
Rescue boot mode does not try to mount the real root.
Handle tftp block overflows properly, allowing the kernel image to exceed 65536 blocks when tftp pxe-booting is used instead of NFS pxe-booting.
Slow down the NFS twiddle a bit.

Various tools have been upgraded in the base system:

dhcpcd 7.0.8 imported.
openresolv 3.9.0 imported, installed as resolvconf(8).
ip6addrctl(8) together with a rc script imported from FreeBSD.
DHCP server for network installs updated to net/isc-dhcp44-server.
OpenSSH updated to 7.6p1.
ACPICA updated to 20181031.
Time zone data updated to tzdata2018g. (tzdata2019a in 5.4.2)
Compiler - see next section
diff utils updated.

Compiler status

gcc 8.0 is the default compiler, for kernel, world, and dports. GCC versions 4 and 5 are still installed and built if needed.

Package updates

A full set of new binary packages has been built for 5.4, available through the pkg tool.
(5.4.3) Another binary package update and rebuild completed, and is available for download with pkg.
There's a number of options now for running a web browser on DragonFly; check the browser documentation page for a full list.
wpa_supplicant is installed as a package by default; delete if your system does not use wireless. The version in base remains as a safety measure in case the dports version is deleted.

HAMMER2 changes in 5.4.1

The HAMMER2 filesystem now has meta-consistency protection for snapshots and crashes. This is on top of the lower-level radix tree protections it already had. The coherency protection handles directory-entry-vs-inode dependencies which prior to this patch could sometimes get broken by a crash, causing the nlinks count on the inode to not match the number of directory entries referencing it, sometimes putting directory entry flushes and the related inode flushes in different sync groups, and often causing snapshot operations to 'cut' the filesystem while it is in an inconsistent state, when made in the midst of modifying activity. Some people have noticed these issues recently, and while they can be cleaned up with manual hammer2 directives after the fact, it was really annoying to have to deal with.
Files should now be completely consistent at write() boundaries on crash or snapshot.
As an added bonus, concurrency between frontend filesystem calls and backend flushes is now much, much better than it was before. Instead of having to wait for most of the flush to complete, modifying system calls (create/delete/write/truncate/etc) are now able to run concurrently in most situations. In situations where concurrency is not possible, frontend operations reorder the flush sequence that is underway in the background to retire the stalled inodes as quickly as possible. The kernel's buffer cache operations also run more smoothly, improving read-to-write concurrency.
The new patch significantly improves snapshot operation, and adds a new directive called 'snapshot-debug' (which I will probably rename) which foregoes the standard sync-before-snapshot that the primary 'snapshot' directive implements. Both directives will snapshot a fully consistent filesystem, the only difference is that snapshot-debug might not get changes made just prior to issuing the command (it uses a recorded blockmap from the most recent sync instead of forcing a new sync).

HAMMER2 changes in 5.4.2

A HAMMER2 corruption issue has been found: it can ONLY occur during heavy write activity during a bulkfree operation, typically when heavy manipulation of directory entries or inodes occurs. This problem is fixed in 5.4.2.

HAMMER2 now has immediate (async) flush follow-through when the operating system requests it, with write-clustering support at the device level if enabled. This, plus other changes have improved concurrent filesystem ops and I/O and sequential filesystem write I/O.