KDE 4 gets more Hot New Stuff

This article was originally published in LWN on 2007-03-27.

Since KDE 3.3 was released in 2004, KDE applications have used the KDELibs library KNewStuff to implement a simple interface for downloading and installing content from a centralized server. Its successor, KNewStuff 2 (KNS 2), is gearing up to give KDE 4 users new options for uninstalling content, content synchronization, the ability to rate content directly from the application interface, a dramatically faster interface, and more. KNewStuff 2 will be compliant with the FreeDesktop.org GetHotNewStuff specification (GHNS), developed as a cross-desktop standard based on the original KNewStuff design.

Like its predecessor, KNS 2 will act as a sort of per-application package management system for content. HotNewStuff content differs by application, ranging from visual themes for Amarok to object position data for KStars. In KDE 3, Hot New Stuff can be installed in participating applications via a KNS 1 dialog which lists content by user rating scores, download rate, or release date.

Arguably the most noticeable improvement to KNS will be the content data caching feature, which should make the interface significantly faster. Content data primarily consists of preview pictures and metadata such as the name, description, and rating of each content item. In KDE 3, the user must wait for this content data to download before the KNS dialog appears. The KNS 2 dialog will open immediately with cached data and update the cache transparently in the background. The interface will be updated with new data as it arrives - though KNS’ exact protocol for updating its interface based on this data will not be finalized until the KDE 4 Usability and Accessibility Review period from May 8th to June 1st.

KNS 2 will introduce Desktop Exchange Service (DXS) support. KNS 1 presents content items in lists sorted by highest rating, most downloads, or release date. The ratings it uses come from user ratings made through web-interfaces such as KDE-Look.org. By using DXS, content providers will be able to integrate this rating system directly into applications by using a CGI-like script. If application developers support DXS, users will be able to upload content ratings and even leave text comments without having to open a web browser. Additionally, DXS integration will add support for content searching by keyword and allow users to “subscribe” to content and be notified of updates.

Content management will also be greatly improved in KNS 2. It will keep a record of content files and files extracted from content archives. KNS 1 is not capable of uninstalling content, but KNS 2 will be able to do so by simply referring to its record of installed files and deleting any associated with a content item. Like version one, KNS 2 will be able to recognize when content has been updated. Although KNS 2 is not capable of automatically downloading updated content, it will indicate to the user that the update is available.

KNS 2’s content management features will also resolve some distribution issues that occur with KNS 1. In the event that an application ships with some content data already installed, KNS 1 is not able to detect it and will indicate to the user that it is not installed. Moreover, it cannot indicate to the user whether or not this content is the latest version or not. KNS 2 will recognize these content packages and treat them like any other content it has installed.

In addition to being able to upgrade installed content through the KNS 2 interface, KDE 4 users will be able to automatically synchronize some content over the Internet or a private network. Content providers can offer automatic synchronization by listing references to files on a network rather than the files themselves. When the application attempts to access this quasi-installed content, it will have to fetch it from the server using KIO rather than a local directory. In this way, the content will always appear up to date to the user.

KOrganizer is among those applications that will benefit greatly from synchronization functionality. Users may use KNS 2 to install public calendar listings, for instance, holidays or company events which will then appear alongside the users’ own calendar entries in KOrganizer. If KNS 2 installs these calendars as remote links rather than local content, users will be kept aware of newly added events or other calendar changes.

KNewStuff2 will be included in KDELibs for KDE 4. Lead developer Josef Spillner is already well into the coding of KNS 2. He quips, “There is already more code for it than what its predecessor had, although more might not always be better.” He is currently working to integrate KNewStuff with the SVN development version of KDELibs 4, marking the implementation of basic downloading and uploading functionality. Spillner plans to release a new developer tutorial soon.

Spillner believes the developer tutorial for KNS 1, which walked KDE 3 application developers through implementing GHNS functionality, was partly responsible for his library’s fairly wide adoption. KDE 3 applications such as Korganizer, Kopete, SuperKaramba, Amarok, and many KDE Edutainment applications such as KStars all use KNS to deliver content such as interface themes and templates to users.

There is reason to believe that KDE 4 applications will adopt KNS on a wider scale than was seen in KDE 3. Spillner notes that several common hesitations that prevented application developers from adding KNS support in KDE 3 have been fixed. He writes, “The inelastic installation methods turned out to be the obstacle which probably prevented more apps from using it.” KNS 2’s uninstallation functionality should eliminate this obstacle. Moreover, he believes that developers will not have to write as much code to integrate KNS, as “simple configuration files” have replaced what previously required lines of code, in many places.

KNewStuff2 also retains two features from its predecessor that perhaps saw less use than they deserved: data verification and uploading. Since KDE 3.4, the KNewStuffSecure class has allowed application developers to implement md5 hash and gpg signature checks for Hot New Stuff. Unfortunately, this class is currently separate from the original KNewStuff. By integrating these verification features into KNS 2, it may prove more visible and popular among developers. KNS 1 is capable of handling user uploading without requiring a web interface, though there has been almost no implementation outside a few of Spillner’s own games. KNS 2 will also support uploading, and may switch from using the FTP protocol to the more secure WebDAV. HotStuff server administrators could then allow users to register to upload content or do so anonymously.

For KDE 4, Spillner is also removing what is perhaps the greatest barrier to KNS integration: server requirements. KNewStuff must get its content from a server running HotStuff. For KDE 3 applications, this often meant that application developers needed to build and maintain their own HotStuff server. Shortly, however, KDE will begin to offer a general HotStuff server and a web interface at data.kde.org. The Oregon State University Open Source Lab has agreed to host the server on a Xen box alongside the KDE news site, The Dot. Spillner still encourages developers to run their own HotStuff servers, but general distribution points such as data.kde.org and KDE-Look.org will be available for those developers who do not want to burden themselves.

Spillner is setting his sights on KDE Games and KOffice integration for KNS 2. Many KDE Games applications already implement KNS, but Spillner would like to see adoption throughout the entire Games module. At least one of KOffice’s many components, the database manager Kexi, already uses KNS. Spillner sees the remainder of KOffice as another excellent opportunity for KNS growth: “In organizations, there is often a need for keeping templates up to date regarding the corporate design, contact data and so on. KNewStuff could be of great use here.” Organizations could host a GHNS server for employees and synchronize templates among them using KNS 2’s remote content synchronization feature.

The GHNS specification is broadly defined by the “idea of collaborative work” statement posted on its website: “Free desktops empower their users to work together over the internet, and share their ideas, artwork, scripts and files. The missions of GHNS is to provide the necessary infrastructure on the client, the server and the protocols in between.” More specifically, the specification outlines the GHNS standard for distributing and centralizing data.

The GHNS project has developed or inspired both server-side and client-side implementations of this standard. Client-side GHNS software includes KNS and the SDLNewStuff Python library for games. Additional client implementations include the incomplete Get Hot New Stuff for Java and the unmaintained GNOME Art. Spillner notes that the SDLNewStuff library has had significant adoption among game designers for sharing content such as level designs and themes. He writes, “In Chess, for example, fiddly game positions are exchanged in the standard PGN format. Using KNewStuff or other GHNS client libraries, those can easily be retrieved from within the game.”

Server-side GHNS software consists of the HotStuff web interface and backend scripts, as well as the DXS service. Major content providers supporting the GHNS specification include KDE-look.org and KDE-apps.org. Spillner credits KDE-look.org’s historical support for KNewStuff as prompting the standardization of GHNS.

As with most FreeDesktop.org-endorsed specifications, Spillner intends for GHNS to have a cross-desktop appeal. In areas where desktop content overlaps, such as wallpapers - where nearly any desktop can use the same data files, the benefits of cross-desktop GHNS compatibility is obvious. With the lapse of GNOME Art, KDE has become - once again - the only desktop which has significant support for GHNS. Spillner would like this to change in the coming months. In his words, “My main focus right now is on development, but advocacy will certainly become more important again once KNewStuff2 is in a state where we can show off with it and trigger proponents of other desktops to rival it.”

Development on KNS 2’s content management dialog has only just begun. Spillner will be designing the interface in collaboration with KDE Usability experts. Spillner plans for the interface to be very customizable by application developers. As an example, he describes an interface for dynamically listing content in menus, like Live Bookmarks. Spillner will give a comprehensive presentation on KNewStuff 2 at the Akademy 2007 conference in Glasgow, Scotland June 30th to July 8th.