release46

DragonFly BSD 4.6

Version 4.6.0 released 02 August 2016
Version 4.6.1 released 17 October 2016
Version 4.6.2 released 4 April 2017

DragonFly version 4.6 brings more updates to accelerated video for both i915 and radeon users, home-grown support for NVMe controllers, preliminary EFI support, improvements in SMP and networking performance under heavy load, and a full range of binary packages. 4.6.1 adds some security fixes and a power management daemon. 4.6.2 was a catchup release containing all 4.6 fixes before the 4.8 release.

The details of all commits between the 4.4 and 4.6 branches are available in the associated commit messages for 4.6RC, 4.6RC2, 4.6.0, 4.6.1, and 4.6.2.

Big-ticket items

Improved graphics support

The i915 driver has been updated to match the version found with the Linux 4.4 kernel. This gives us significantly better stability on newer cpus, Broadwell and Skylake in particular.

The radeon driver has been updated to match Linux 3.18, and controls for the backlight are available through drm.radeon.backlight.

NVMe support

We now have a NVMe driver (PCIe SSDs). It currently must be kldloaded with nvme_load="YES" in /boot/loader.conf. The driver uses all concurrency features offered by the chip and will distribute queues and interrupts across multiple CPUs to maximize performance. It has been tested up to around 1.05M IOPS @4K, and roughly 6.5 GBytes/sec @32K (random read from urandom-filled partition, physio, many threads), with the 2xE5-2620v4 (xeon) test server 78% idle in the IOPS test and 72% idle on the bandwidth test. In other words, we maxed out the three NVMe devices we had plugged in and the system still had plenty of suds left over.

Please note that a machine's ability to boot from an NVMe device depends on the BIOS, and not DragonFly. Most BIOSes cannot boot from NVMe devices and those that can probably only do it through UEFI. Info on device state is available with the new utility nvmectl.

NVMe support is not in the default kernel, yet. Put nvme_load=”YES” in loader.conf to use it.

SMP Performance significantly improved

SMP performance was already very good. As part of the NVMe driver work we revamped the buffer cache subsystem and a number of other I/O related paths, further reducing lock contention and IPI signalling overheads. We also put topology-aware cpu cache localization into the kernel memory allocator (primarily helps multi-socket systems and systems with high core counts).

The network subsystem also continues to receive significant improvement, with modest machine configurations now capable of handling upwards of 580K conns/sec.

Expansion of supported packages

DragonFly now supports over 24000 packages via its dports mechanism. This year a great deal of work went into improving library compatibility and improving the quality of the ports and the dependability of the bulk build system that backs them. Normal users can now do customized bulk builds of their own ports packages through a simplified build interface called "synth", which is itself available via the dports system (if desired, and you have sufficient machine resources and time).

Preliminary EFI support

DragonFly now supports (64-bit) EFI booting, but it has not been integrated into the installer. Following the manual steps of other users is the immediate solution.

Details

Checksums

MD5 (dfly-x86_64-4.6.0_REL.img) = ad06ed86ca67c23a54820ded123203c3
MD5 (dfly-x86_64-4.6.0_REL.iso) = 117b0a23044685067f23be6dd4297963
MD5 (dfly-x86_64-4.6.0_REL.img.bz2) = d544e81e1e24509edf44abcd7b8c562a
MD5 (dfly-x86_64-4.6.0_REL.iso.bz2) = aaaf3212b3922ed694324a6659fd2604

Upgrading

If you have an existing 4.4.x system and are running a generic kernel, the normal upgrade process will work. Change your local /usr/src to 4.6:

cd /usr/src
git fetch origin
git branch DragonFly_RELEASE_4_6 origin/DragonFly_RELEASE_4_6
git checkout DragonFly_RELEASE_4_6
git pull

And then rebuild: (in /usr/src )

make buildworld
make buildkernel
make installkernel
make installworld
make upgrade

Don't forget to upgrade your existing packages. 4.6 packages have already been built and are immediately available.

pkg upgrade

All changes since DragonFly 4.4

Kernel

Fix a namecache race (triggered by heavy tmpfs mount/umount rate)
Fix two unmount / access races when many mounts are present
Improve context switch hotpath
Cut pmap invalidations from buffer cache activity in half
Reduce certain potential interrupt stacking issues
Implement a new fixed memory pool for physio accesses, eliminating all SMP invalidations from the path
Refactor pmap invalidations to operate at a higher priority than critical sections, reduce response latency and greatly reducing multi-core stalls during heavy invalidation activity
Fix a 2-instruction SMP race during thread exit.
Add buffer cache repurposing to reduce TLB invalidations in high-I/O-thru-filesystem situations. Defaults to off (sysctl vfs.repurpose_enable).
Revamp the IPIQ (IPI messaging) mechanics to avoid deadlocks during heavy activity.
Revamp the IPI interrupt signalling mechanics to reduce unnecessary duplicate IPIs.
We no longer use the vm_map subsystem to dynamically allocate and deallocate KVM for the buffer-cache. Instead we pre-reserve the maximum amount of (unmapped) KVM for every buffer. This makes certain congestion-control algorithms for the buffer cache unnecessary and they have been removed.
Add a SMP %rip sniffer IPI, very useful for debugging.
Fix an issue where the reported real-time becomes inconsistent between cpu threads under heavy loads. This issue generally resulted in 'make' getting confused.
Refactor the VM prefaulting code to improve pipe-lining. This improves burst page-in performance when starting large applications such as chrome.
A lot of work to reduce noticable (but short) stalls in various subsystems when they have to work on large data-sets. For example such as when a process with 64GB of RSS exits.
Fix zfree()'s hysteresis, greatly reducing unnecessary IPIs.
Reduce stalls during major token contention. It used to be that a large multi-threaded program contending on a single lock across all cpus could create noticable stalls even if the program was running niced or at a high (aka low) dynamic priority. These situations have been fixed.
New threads should not inherit the sigalstack. Fixes issues with 'go'.
Fix MSI support issues.
Improve buffer flush and cluster_write linearity.
Allow drivers to register interrupts which are specifically meant to be high-frequency (disable the rate limiter for such interrupts).
Bring in ipfw3
implement a lockless in-kernel NAT for ipfw3 (note that PF's NAT is also mostly lockless).
Add ipfwsync for ipfw3, for synchronizing state between multiple machines.
Various procfs workarounds added to fix truss issues.
Implement PIE (place independent executables). Randomization be enabled with a sysctl kern.elf64.pie_base_mmap.
Implement ppoll() with precise microsecond timeout.
Fix VM issues with virtio.
Add EVFILT_FS support to kqueue.
Cut-down on the amount of BSS in the kernel binary. We ran over the loader's 64MB limit (with the initrd image and a few kld's also loaded), AGAIN!
modules.local mechanism added for non-base kernel modules.
Implement I2cSerialBus ACPI module support (used by IG4).
Implement ACPI's GPIO.
Work around HW VM bugs by aligning the kernel load's end address to 2MB, so there is no transition from 4K pages to 2MB pages for kernel text+data+bss.
Many, many HAMMER improvements.
HAMMER mount is now forced to noatime by default.
Do not allow the kqueue sleep timer to wrap.
User process scheduler adjustments to improve interactive responsiveness in situations where (essentially) batch jobs are syscall-heavy or VM fault-heavy.
Fix cpu selection skew in the scheduler with high fork/exec rates.
Fixed vfork()/getpid() bug in heavily threaded programs.
Refactor vm_page_alloc() and the pre-zeroing code to maintain cpu-v-cache affinity.

Graphics

The drm/i915 driver now matches Linux kernel 4.4, supporting Intel GPUs up to the Skylake and Cherryview generations
The drm/radeon driver mostly matches Linux 3.18 with specific fixes from Linux 3.19
UEFI framebuffer support

Networking

Fix an issue with IWMs packet tx.
Bring in expanded ipfw from FreeBSD (called ipfw3 in DFly)
Complete WIFI sync from FreeBSD. This gives us the new single-ifconfig mechanic that fixes all the locking issues that existed with the dual-ifconfig mechanic and allowed us to synchronize all FreeBSD wlan drivers. DragonFly and FreeBSD developers are thus able to combine efforts on bug fixes and new work.
IWM, IWN, ATH, usb-based, and other wifi drivers now have better support.
Reduce network maintenance IPI spamming (e.g. ipfrag_slowtimo()). These now only signal the cpus that actually need it.
hyperv/vmbus support added (ported from FreeBSD)
AUTOFS added (ported from FreeBSD), AM-UTILS removed.
Improve NFS handling of ESTALE.
Fix two inet6 issues.
Do not prematurely drop receiving-only TCP connections when mbufs are not available.
Make setsockopt() asynchronous for commonly used options. This significantly improves the connection rate.
Fix kernel panic when /dev/mapper/control is read or written (it isn't supposed to be readable or writable).
Improved power management for AHCI.
Improve DDB by making x/a work like in gdb. Add x/p.
Implement per-cpu lport cache for listen sockets. Greatly reduces listen/accept contention on a busy TCP server.
Move libedit and libncurses to /lib/priv to ensure that packages do not try to use them. They should only be used by base utilities.
Remove all references to libtermcap from ncurses.
The nrelease build can now build snapshots on tmpfs.
Rework the GCC Makefiles to reduce unnecessary re-building for incremental builds (quickworld).
Add kqueue support to NFS (fixes firefox issues w/NFS that cause firefox to corrupt itself).
Add EV_RECEIPT and EV_DISPATCH support to kqueue.
Add quirks for xhci (USB).
Significant work on IGB, IG, EMX, RAL, and many other NIC drivers.
accept(2)/close(2) performance has been [significantly improved](http://lists.dragonflybsd.org/pipermail/users/2016-June/249699.html0.

Other drivers

NVMe (PCIe SSD) driver added. This is a home-grown driver written from the ground-up for maximum SMP performance.
Syscons got some reworking to work better with GPU framebuffers.
Multiple SD/MMC card slots in a single device can be accessed simultaneously.

Userland

Revamp systat -vm display. MSI/MSI-x interrupts are aggregated a little, memory use reformulated, more page rates are shown in bytes (K,M,G, etc) instead of pages.
Revamp systat -pv display on wide terminals (add %rip sampling)
Fix mountd-caused system performance issue in mount-heavy systems
TRIM support refactored for newfs_hammer.
Align output from cat(1) when invoked with -be and -ne flags (from NetBSD)
Remove non-standard and headers from the system.
Turn SO_REUSEADDR on for mountd. Fixes startup errors when mountd is restarted or initially fails due to /etc/exports issues, and then restarted a little later.
Numerous wide-char improvements.
Port fstyp from FreeBSD
Add -E option to showmount
Add -o retrycnt= option to mount_nfs
Add -i and -q to sysctl, from FreeBSD
Add -x to chown/chgrp and sync with FreeBSD
fix -S option of ln (before it required -t to work)
Sync chflags, chmod, ctags, cpucontrol, kenv, pathchk, stat(1), ncal(1) with FreeBSD
Fix an issue with malloc() improperly setting errno.
Fix fscanf() format in atrun when reading the uid and gid from the job file.
Misc fixes to zic.
Synced ncal from FreeBSD
Add unzip(1) (if you can't beat 'em, join 'em).
Update ssh (now a private library)
libc - improve realloc() performance.
libc - Fix numerous open/dup/pipe descriptor leaks due to threaded races by using O_CLOEXEC in appropriate places. In particular, fixed bugs in popen().
nurses upgraded to version 6.0 and made private. As a result, functionality not needed in base (e.g. panel/form/menu libs) were removed as well as man pages.
libldns was privatized

Various tools have been upgraded in the base system:

Installer changes the inode/data space mix for the UFS /boot. We don't need 128K inodes on a 1GB /boot!
Installer no longer uses HAMMER PFSs, it causes too much confusion. Instead it separates out stuff that would normally not need to be backed up into /build and uses null-mounts.
Fix a couple of logic issues in (gcc, mpfr, tre). gcc and mpfr fixes pushed to upstream.
Import OpenSSL 1.0.1t

HAMMER improvements

Many bug fixes and improvements by Tomohiro Kusumi.
Fix an improper DATA CRC error that can occur sometimes when a block of data read just after being written.
The rescue image now includes the 'undo' utility for HAMMER1.

Other improvements

24000+ dports packages now build.
User utility "synth" to automate bulk building.
Many manual page improvements.
share/doc has had a significant number of papers added.

Hammer2 Status

HAMMER2 is still in-progress and will continue to be for some time. It is not ready for prime-time, but significant work has been done on its internals to improve backing-device separation, improve stability, and increase throughput in SMP environments. HAMMER2 now supports export via NFS.

Clang status

A starting framework has been added for using clang as the alternate base compiler in DragonFly, to replace gcc 4.7. It's not yet complete. Clang can of course be added as a package.

A Note on the Removal of 32-bit Support

DragonFlyBSD is now a 64-bit only distribution. It will no longer compile for a 32-bit target and most of the 32-bit support code has been removed over the past year. Some users have noted that this might be little premature, but the reality is that developers not only of DragonFlyBSD, but of other operating systems and more importantly of applications have begun to quickly shift over as well. From a kernel development perspective:

Several kernel subsystems now need to be able to assume a larger kernel virtual memory address space than is possible in 32-bit mode, allowing the implementation of significantly more efficient algorithms. The older code paths were conditionalized for 32-bit, but were becoming stale due to lack of eyeballs.
The 32-bit pmap code was becoming difficult to maintain for multiple reasons.
We need to be able to assume that a DMAP exists because non-DMAP code paths were no longer being tested or maintained. And certain algorithms just don't work without a DMAP. The 32-bit kernel didn't have enough address space for a DMAP.
We need to be able to assume a minimum SSE2 level for FP support moving forwards. Compilers are starting to assume this regardless.
There's virtually no point running a 32-bit server OR workstation any more. While some BSDs strive to maintain support for old (or ancient) hardware, the DragonFlyBSD team just doesn't have the man-power to do it. We are able to keep DFly relevant because we are able to focus on supporting contemporary hardware. That said, we do not expect to remove support for 'older' 64-bit hardware for many years as the 64-bit baseline is quite decent.