DragonFly BSD

release44

DragonFly BSD 4.4

DragonFly version 4.4 bring further updates to accelerated video for both i915 and radeon users, a new locale system, and a new default linker. Significant behind-the-scenes work has also been done, with symbol versioning, Hammer1 improvements, and other changes. Version 4.4.1 was the first release due to the late inclusion of OpenSSL update 1.0.1q. Version 4.4.2 was released to include OpenSSL 1.0.1r, plus updates to iwm(4) and other miscellaneous bugfixes. Version 4.4.3 includes OpenSSL 1.0.1s, I219 (Skylake) network support, some HAMMER1 bugfixes, and other updates.

The details of all commits between the 4.2 and 4.4 branches are available in the associated commit messages for 4.4RC, 4.4.0, 4.4.1, 4.4.2, and 4.4.3.

Big-ticket items

Improved graphics support

The radeon and i915 drivers are now updated to match the versions found with the Linux 3.18 kernel.

Collation support

DragonFly is the first BSD to properly support collation for named locales. This means data is sorted per the rules of the language and territory selected. Before this release, data would be sorted by the POSIX standard, basically in order of the character set. The lack of collation support manifested in various ways, e.g. French and German sorting would be completely wrong, but people would most often notice because collated databases would return query result sets in the wrong order. A well-known example is PostgreSQL databases with collation fields. The only way to get results in the proper order was to graft ICU to PostgreSQL, which was a non-default ports options that was not approved by PostgreSQL project. Now PostgreSQL and other databases that use libc provide proper results out of the box.

The collation defintions from CLDR (Unicode Common Locale Data Repository), and DragonFly has imported Release 27.01. The collation format, generation tool and libc handling was derived from the Illumos project.

Since this was a long-standing goal of FreeBSD, the collation code from DragonFly has already been imported into FreeBSD CURRENT and will be available for Release 11.0, so the boast of "DragonFly is the only BSD to support named locale collation" will only be for a limited time. There was significant collaboration between FreeBSD and DragonFly (bug discovery/fixing/RB Tree) and patches were fed back to Illumos, so this was a very positive FOSS story resulting in a common locale solution for three platforms.

Locales

The locale support on DragonFly has occasionally been synchronized with FreeBSD over the years, but FreeBSD's support itself had been static. This release brings a complete overhaul to the locale system. The data for all six locale categories (LC_CTYPE, LC_COLLATE, LC_TIME, LC_NUMERIC, LC_MONETARY, LC_MESSAGES) come from a single authoritarian source: the Unicode CLDR Project (http://cldr.unicode.org/). Releases of CDLR and Unicode are fairly frequent and are used to generate the POSIX-formatted input files. Keeping the locale data up to date in the future is intended to be regular maintenance.

While LC_COLLATE support was the most impactful change, locale users will immediately notice improvements in numeric, time and monetary string handling. For usability, the CTYPE defintions were merged into one map, including a very signicant hand-created data file to provide a comprehensive UTF-8 encoding that covers all supported (and many unsupported) languages rather than use the strict and narrow per-locale CLDR CTYPE definition.

Three-component locale names have been introduced here. Common examples are sr_cyrl_RS, sr_latn_RS, zh_Hans_CN, zh_Hant_TW, etc. We introduced "short codes", so now codes like "de_DE", "fr_FR", "en_US", "el_GR", etc. These short-codes are generally mapped to 8-bit character sets such ia ISO-8859-x, but sometimes they are mapped to UTF-8 if the traditional single-byte encoding doesn't adequately cover the locale anymore (e.g. the currency is not supported).

We took the opportunity to intelligently limit the locales supported. For example, a locale will support either ISO8859-1 or ISO8859-15 but not both. Generally this means ISO8859-15 is used in Western Europe and the rest of the locales remain with the older IS08859-1. Several older encodings were eliminated. Note that it was necessary to keep the zh_CN prefix due to compatibility issues with nvi and other 3rd party software.

Regex library replaced with TRE

As a consequence of the locale upgrades, the original regex library had to be forced into POSIX (single-byte) mode always. The support for multi-byte characters just wasn't there. We turned to the TRE Regex library written by Ville Laurikari. It was already being used by the MUSL library and the latest MacOS. The latter had been significantly modified to keep FreeBSD legacy support with specific tokens, and also added bracket pattern matching. The DragonFly regex library matches the Apple version in features and performance, and scores extremely high on the AT&T regex testsuite (much higher than the original library). The only regression is the recent addition of "\<" and ">/" token support, but the bracket equivalents of "?:<:" and "?:>:" work fine. TRE is faster, more capable, and supports multibyte characters, so it's a nice addition to this release.

Change in linker default.

DragonFly now uses the gold linker by default. The legacy "ld.bfd" linker is still available and can be permanently reselected through key definitions in the make.conf file.

Changes since DragonFly 4.2

Kernel

Graphics

Networking

Other drivers

Userland

Various tools have been upgraded in the base system:

Removed from the base system:

HAMMER improvements

Other improvements

Hammer2 Status

New in this release: Hammer2 supports root mounts, and live dedup. Hammer2 filesystems can be set up, for single-host testing. Bulkfree scans are currently not automated and out of space conditions are currently not recoverable (since everything is copy-on-write, additional mechanisms are needed to allow an out-of-space condition to be rectified). The WANT_HAMMER2 option has to be added to build HAMMER2 pieces. Do not experiment with it unless you have data you can lose! H2 is not ready for prime-time and the media format will continue to change in incompatible ways.

Work has generally progressed with the frontend essentially stable and passing most filesystem stress tests. A solution to the hardlink problem (related to renames of directories along the path to a hardlink breaking the hardlink) has been put in place. The physical hardlink target was previously stored in the most common parent directory vs all of the links. Now the hardlink target will is placed even higher in the directory tree, at or above the common directory at a directory with the 'xlink' chflag set. Renames across xlink points are not allowed.

Work continues on the backend, which is really where all the sophistication is going to be. The backend is now threaded, with one or more threads per physical drive. The frontend replicates and dispatches requests, collects results, and performs cluster operations such as quorum validation and is capable of detaching and aborting the remainder of a request once sufficient progress has been made. The idea is that a stuck physical drive does not cascade back to the frontend and cause the frontend to get stuck. This work is preliminary but the design feels solid so far.

Note that clustering functions are not yet ready for testing in any manner.

Clang Status

The Clang dport mostly works in DragonFly but is not officially supported for kernel or boot-loader builds until we track down and correct a number of issues with the clang-built boot loader. Ultimately the intention is to make Clang one of DragonFlys built-in compilers (DragonFly always ships with two which right now are gcc-4.7.* and gcc-5.2.1, with gcc-5.2.1 as the default).