In part 1, I described the problem of software installation on Linux; in part 2, I’ll describe the solution we came up with at the recent LSB Packaging Summit.
After reading through the comments to part 1, let me first point out that our goal is to create a vibrant third party software ecosystem around Linux—you know, like the one Microsoft has built around Windows. No, it’s not about imitating Microsoft. It’s about being competitive. A platform is only as good as the applications that run on it.
Bottom line: Many third parties have built their businesses around proprietary software, and we can’t just ignore them. And “ecosystem” implies decentralized, which I argued in part 1 was a key tenet of open source development anyway, i.e., this should be playing to one of our core strengths. So, if your “solution” is to tell ISVs (independent software vendors) to give us their source code so the distributions can include it because that’s just how we do things, you can safely skip the rest of the post below. You’re simply not going to agree that any of this is a problem.
Ok. Assuming our goal is to create a vibrant third party software ecosystem (and everyone still reading agrees that’s a good goal, right?), we have the following challenges.
First of all, the distribution vendors are hugely invested in their existing package systems, and for the most part, those package systems work extremely well. As I said in part 1, “if [an application] is in your distro of choice, you’re only an apt-get or a yum install away from running it.” (I’m saying it again here because some of the commenters apparently didn’t see that. The tricky bit is “in your distro of choice”, which by definition is not the case with third party software.)
Furthermore, a variety of highly sophisticated systems management solutions are built above those package systems, such as Red Hat Network and Novell ZENworks. This is a pretty important consideration, because these days, the “software management problem” largely involves managing software across the entire infrastructure, not just a single system.
What problem then? This: ISVs have thus far been reluctant to use the native package systems. Why? THEY’RE hugely invested in “package systems” for other platforms, and every Linux specific thing they have to support costs money. In a world where decisions are ruled by cost vs. benefit, this is a pretty important consideration too.
How do ISVs handle this today? For the most part, they ignore the package systems on Linux and do their own thing. Trouble is, while doing their own thing gives ISVs the flexibility to work cross platform, it ultimately makes their products integrate poorly in the broader systems management context because the package systems know nothing about them.
What is needed is a way to bridge the gap between what the distros provide and what the ISVs want. But how?
First off, we have to understand what ISVs want. The answer is simple: ISVs want to treat Linux as a single platform, which means they want to offer a single package for Linux, much as they do for Windows. So, if one commenter is right that “[t]he problem is that people (including software distributors) believe there’s such [a] thing as ‘Linux’ as a target platform” and that “[i]f you’re distributing software for ‘Linux’ then it won’t be simple to install it, ever”, well, then Linux is destined to suffer the fate of UNIX. I’m not ready to give up so easily.
Several commenters suggested that what we need is a brand new package system. That’s a non-starter—for one thing, the distros aren’t going to be too keen on replacing something they’re hugely invested in, and if ISVs aren’t going for RPM today, why would they go for something different tomorrow?
No, to find a way forward, we need an evolutionary step from where we are today. From there, perhaps we can do more, but even the first steps can be quite valuable in their own right.
To help find those first steps, we put some of the leading minds in Linux packaging in the same room (including the maintainers of RPM at Red Hat and Novell and the authors/maintainers of APT, yum, alien, and klik) along with ISVs large and small, server and desktop.
The discussion pretty quickly converged on constructing a single API that could be implemented across the various package systems, because APIs make for nice evolutionary steps and can, done right, mask underlying implementation differences.
Question is, what do ISVs need in such an API? Limiting the scope is key here, because providing an API that spans all the functionality of, say, RPM and dpkg is overwhelming to the point of being unworkable, not to mention more work to implement, which in turn makes it less likely to get into the distros so that ISVs can count on it being there, the whole point of this exercise in the first place.
Fortunately, the ISVs don’t really need much. At the most basic level, an installer just needs to be able to query the system to see if it’s LSB compliant, and if it is, what version of the LSB it’s compliant with; and it needs to be able to “register” with the package system, so the package system knows about it, including what files it has installed. And that’s really about it.
Importantly, because we assume an environment that’s LSB compliant, we don’t have to worry about dependencies, because everything is covered by the single LSB dependency, and dependency management is 95% of the package systems right there. We still need minimal dependency support—components can extend the LSB, and applications can depend on those other components being installed—but we’re talking on the order of a handful of components, not the tens of thousands of components typical package systems have to deal with.
We only had a day, so we obviously don’t have a complete solution yet. For one thing, we barely touched on the issue of uninstallation (should we allow applications to register GUI uninstallers?), and the issue of how applications go about changing the system configuration still needs discussion (in a lot of cases, the assumption of LSB compliance means the installer is going to be well behaved, but there’s undoubtedly corner cases to be explored). And it’s going to take time for the API to be implemented and put into widespread enough use that ISVs can depend on it.
However, everyone in the room did come to consensus pretty quickly that this was an important problem, and that providing a simple API that provides the minimum necessary functionality was the way to go. Perhaps most exciting, the very people whose support are needed to put the API into widespread use were in that room, and active participants in the discussion too.
Everyone is certainly motivated: The distros get more applications, which makes the shared platform more attractive; and ISVs get lower cost, which tilts the cost vs. benefit equation in their favor, making Linux versions more economically attractive and potentially opening new markets.
Because this is just a start, the FSG is launching a Packaging workgroup to continue the discussion we started at the Packaging Summit. If you’re interested in this topic, I’d encourage you to join the mailing list and get involved. There’s still plenty to be done.
Update: One more thing: In addition to making it easier for ISVs to better support Linux by simply extending their existing installation scripts, the API approach really comes into its own when you imagine it implemented in things like Autopackage and InstallAnywhere. Let the market decide!
Sounds interesting – although the part about dependencies was quite surprising to me, but the arguments are right. And a dependency solving thing could be added in the API 2.0 – most important is to get the first version of the API now stable and into the wild.
I really hope that this is the begin of something new and that it will solve the problems.
This is pretty funny. On the one hand, almost every operating system except Linux does have the “single platform” aspect. For instance, FreeBSD is just FreeBSD, same layout and packaging system even across huge architectural gaps (4.x -> 6.x). And yet, because most of these vastly more consistent and sane platforms are incidentally much less popular than Linux (which is very probably as popular as it is *because* so many people are willing to change everything to appeal to an extra 1%), most ISVs don’t care to support them in the first place, even though it would be extremely easy if they already support anything approximately Unix-like.
So the platforms which can easily support third-party software don’t have any to support, and Linux, the one “platform” which is infinitely difficult to support uniformly, has an extensive collection of third party software almost nobody bothers installing because it immediately kills the health of the system and nulls out any hope of reliability and an upgrade path. A lot of people complaining that Ubuntu, one of the finest Linux distributions at the moment, would violently break dist-upgrades and hose the system, were found to be using too much third party software. Nobody is surprised by this.
A vibrant third party software ecosystem would bring along a host of security concerns, too. There are definite benefits for centralised packaging of software. I would like to point people to the following essay: http://linuxmafia.com/~karsten/Rants/spyware.html
I’d like to see a “safe” API for package management. I think besides ISVs and binary only distributed software this would be the right thing for people compiling stuff from source too.
I’d would be great if the api would allow pushing some package meta data (name, version, …) at it and then say a list of temporary files and pathnames where they should be installed. So a sysadmin can be sure nothing gets overwritten by a badly designed application installer. And for from source installs we would have a patched install(1) that just registers the stuff with the native package manager.
Best would that all this would work as a somewhat priveleadged non root user. (maybe something that can be started with su/sudo that takes the data and temp files and does all needed priveleadged stuff, KISS ;) )
In sort sane package management is one of the really strong points of the linux ecosystem, but we need to extend it to somewhat ad-hoc packages that are created on the fly; software like checkinstall is just an to ugly hack for the real world).
The time where enforcing distro packagers to have sane source packages was a good idea is hopefully gone, because they learned the benefits of it a long time ago
“ISVs want to treat Linux as a single platform, which means they want to offer a single package for Linux, much as they do for Windows.”
Uh-huh. Good luck with having them offer a single package that works on Linux/x86-32, Linux/x86-64, Linux/IA-64, Linux/PPC, Linux/Arm, Linux/Alpha, Linux/Sparc, etc, etc, etc…
Linux is *not* a single platform, and it’s *so* much more than Linux/x86-32.
I wish the Linux trademark holders would create a “Supports Linux” program, and only give out the mark if an application supported at least 5 of the above architectures. Supporting Linux/x86-32 is not “supporting Linux”.
I’m *tired* of having people say “Hey, use our Linux client; we support Linux” and it not running on my perfectly capable Linux system, just ‘cos I happen to have “unusual” hardware that is nevertheless fully supported by Linux itself.
Adam: while I agree with you that the ISVs do not carefully enough check which hardware they support, the attempt of creating an installation API is all about being realistic – and supporting hardware tha has no market share at all is not realistic.
Even ppc on the desktop has no bright future anymore because there is hardly an ppc desktop hardware left (yes, I know the PS3), and the other hardware is mainly used in embedded systems.
Supporting x86 and x86_64, yes, but not the rest.
Windows packages have three ‘platforms’: Win9x, W2k/XP/2K3 32bit, and XP/2K3 64bit. Try mixing drivers, software, etc. for these three and see how well they work.
Granted, Linux has a ton of issues to consider before it can think of ‘one platform’ installations, some of which Adam noted above. While being able to fork a project is one of the strengths of open source software, it seems that people are too ready to break off in their own direction rather than agree on a base API. many times it looks like people reinventing the wheel for ideological reasons, rather than practical or technical reasons.
> A vibrant third party software ecosystem would bring along a host of security concerns, too.
Why?
Is the cause really the ‘third party’ part or the OS itself?
IMO the problem is the amount of required privileges by the application and the installer, not the ‘third party’ part.
The application shouldn’t require root and neither should the installer, two things not true for most Windows apps.
In fact, both should be able to run in a much more restricted environment, but Linux (standard, by default) doesn’t support/use such an environment AFAIK.
> Importantly, because we assume an environment that’s LSB compliant, we don’t have to worry about dependencies, because everything is covered by the single LSB dependency, and dependency management is 95% of the package systems right there.
How does that work? Does LSB mean every single library that an app could depend on will be installed?
Or that the apps should ship other dependencies themselves, like in Windows?
And how will update management be done?
Although every Windows app has an installer, IMO a lot of the responsibilities of the installer should be in the OS, not in the installer.
>> Uh-huh. Good luck with having them offer a single package that works on
>> Linux/x86-32, Linux/x86-64, Linux/IA-64, Linux/PPC, Linux/Arm,
>> Linux/Alpha, Linux/Sparc, etc, etc, etc…
At the least have some sanity on the same HW platform, for example x86-32. Too much of vendor dependency when it comes to packaging. Why that?
I tried to use Kubuntu 6.06 and still I’m looking for a Firefox 2 package. I can not use it since Kubuntu thought it is not worth. Surely I can go to Mozilla and get it. Then what good your package manager or wonderful package repository is? You are totally at your distro-makers mercy. Or some volunteers who will package the thing for the exact distribution (with exact version) you have installed. Why there can not be a generic installer or package? I know the later Kubuntu release (6.10 I believe) has got firefox2 or openoffice.org 2.04. Can i just take these two and install in my version? I don’t think so and I didn’t try. Then you say that is distro makers fault. Yes, you are right. Then which one is the perfect distro that I can use?
Can you make a single XYZSoftware.rpm or XYZSoftware.deb which I can just download and install on any x86-32 linux box (be it Ubuntu, or Fedora or Suse.. or whatever) If you can, then it is excellent and that’s what the user want. Why it has to specially packaged for each of these distributions? Not even distributions, but when there is a version change even?
Changing version 1 to version 2 on a same distro makes you download the whole application suite. Why that? Why XYZSoftware.rpm built for version 1 will not run on version 2? Version of XYZSoftware software has not been changed!!!
Too distro specific packaging. Well, as the author mentioned earlier, if your package manager says it is there and can be installed, then really it is trivial (though I have seen apt messing up, but just one or two cases. in 99% cases it worked.). But if it is not there then most likely you will see trouble.
Another ugly thing is, the dependency. We talk about dll hell but from my experience rpm hell is something which I would like to forget. Well, it is not a problem generally when you are hooked to internet and got a fast connection. I live in India where around two years ago a fast connection was very expensive. I used to download a set of rpms in my office (That was Fedora time) , back at home only saw them failing as they depend on some other rpm which I have not downloaded. 95% cases it failed and I went back to office to get those dependencies. Again those failed… Why that? How am I supposed to know that it depends on others? I never had this kind of problem on windows. Yes, even with popular opensource software.
I came across so many good free software which are available for Linux and Windows. Mostly Windows thing came with an installer which worked out of the box but the linux thing came with a a few lines telling you how to compile it and what else (mostly hyperlinked to the sources again) you need to have before you start compiling this one.
Why software installation has to be that complicated?
On Linux, the quality of software is really good (I run a hell lot of linux apps on server boxes and they are excellent. Our full network infrastructure runs on Linux I must admin) and even on my Office Windows mahine I use a lot of free software which run great on Linux. It is just the installation which really sucks. For servers it may not matter, for desktop it does.
Make one package (rpm, deb or whatever) for an application which will install on popular distributions atleast. Then I think desktop Linux user base will increase like anything.
> our goal is to create a vibrant third party software ecosystem around Linux.
Ah, I see… Please bear in mind that open-source applications gained in quality by deploying them to platforms other than linux. This is also due to the type of users on windows and osx, which are indeed “users”, i.e. not willing/able to install from souce; they just use an application, if it is easy to install, looks good, does not crash, is secure, and-of course-useful. This is a challenge in its own right, and one that linux applications often overlook, on the basis that its users are willing to digg their hands into the guts of the penguin and perform surgery by themselves, all day long, every day. There are people who want to use a computer for own work, because-needless to say-they do not get paid to run linux and its applications. There is, therefore, an overall problem of usability within linux, and a problem of how to deploy open-source applications not only on linux, but also on other platforms. Now, the first difficulty with packages under linux is, that there are many different implementations of linux that ignore the LSB. A while ago there was a project to create a standard and gather willing project to adhere. This project has failed. What you are proposing here is to repeat that experiment. What makes you believe that, this time, the experiment will succeed? Each linux project has a mind of its own, and is not willing to stick to a standard. Some believe this is good. To cut short with this problem, Apple OSX allows three types of installation: /usr, /usr/local, and /Applications. The third type consists essentially in wrapping all the files of an application into a single folder, this has the benefit of ignoring the local world (the LSB in linux’s case), and the disadvantage of duplicate libraries. Now, if linux adopts the same solution, commercial vendors could install into any linux system, regardless of the specific base (LSB and otherwise). Still using the same approach, one could distribute open-source applications of a certain fame (firefox, thunderbird, eclipse, etc.), that happen to run on platforms other than linux and thus require a certain degree of freedom (being platform independent). Now, if one agrees on this, the next step is the one I described in my former comment, that is, give birth to a new project to deploy essentally the same package installer/updater application for linux, osx, solaris, bsd, windows, etc. This application can be like Microsoft’s “windows updates”, or Apple’s “Software Update”, or whatever.
This is my opinion. I think it is feasible, and albeit hard to swallow at first, I think it is the best one can do at this time.
I’ve managed a large (3000+) desktop site. The wide variety of Window’s installers in such an environment is a nightmare when building standard software images. It can take 2-3 staff weeks of mouse clicking and close following of installation procedures to set up a standard corporate desktop, work which needs to be repeated for each desktop variant or release.
One of the great advantages of Linux distributions is the use by applications of the distribution’s package manager. Since these can run unattended the standard desktop image can be built in 40 minutes or so. Even better, the package managers can be used to actually deploy the software, so the rather fragile concept of a “software image” can be done away with.
It is a great shame that there isn’t one package manager (and package names, etc) for the whole of Linux, as that would make a lot of issues go away. You raise the argument of sunk costs against this option, which is hardly relevant , as it’s current and future costs which matter. And the current suckiness of RPM+YUM mean that some vendors face significant future costs (30MB RSS, CPU in minutes, and saturation disk utilisation to check if software is current, WTF?)
You say developers want single API, well us customers want a single system. Our deployment costs well outweigh whatever your developers spend packaging software they know well (particularly since our usual response to unpackaged software is to package it as we use .deb or .rpm for updating the installed base of Linux software), and we won’t mention training or support. It’s a shame your meeting didn’t think to include some users.
I’d strongly argue that what you are doing is an exercise in cost shifting from software developers to software purchasers and administrators. Whereas the current situation puts the most of the cost of software deployment onto developers, which is nice since it leads to easily quantified deployment costs and users do not face a large and unexpected number of staff-hours after purchasing the software.
The obvious response to this attempt at cost shifting from Linux-savvy corporates will be to request use of the native packaging system and when evaluating software to add about 30 staff weeks when evaluating software delivered in another format. In which case what’s the point of a common API?
Finally, your article ignores other packaging issues. For example, Sun ship Java in a RPM. But that’s next to useless since that RPM doesn’t integrate well into the JPackage environment used by the majority of third-party Java packages. Given that experience I’ve got strong doubts that LSB compliance alone is adequate guidance for third-party installers.
Pingback: Planeta Fedora » 451 CAOS Links - 2006.12.20
Pingback: tecosystems / links for 2006-12-22
Pingback: Free Standards Group Weblog » Blog Archive » FSG launches Packaging workgroup
I don’t see why third party software wouldn’t sometimes have dependencies outside LSB, why should they not be covered by the API ? Not catering for dependencies might very well encourage vendors to botch installation of dependencies themselves, when they could be cleanly installed by the system.
It doesn’t seem a good idea either to add extra barriers to vendors who could use free software libraries. If the API let an installer query the availability of a dependency, it could use the API to install an available dependency or use it’s own method if the dependency is not available.
An API could indeed help things along in certain ways, but maybe also providing a standard package format could make things even easier. Imagine a vendor that just wants to place a few files on the system : must he write his awn installer or just create a file which lists the files he wants to install ?
Glen Turner seems to make some very valid points.
Olaf:
The LSB requires a certain set of interfaces, which means that application developers can depend on those interfaces being present (and with the expected behavior) on any LSB compliant distribution. When application developers need to go beyond that set of interfaces, they can bundle the interfaces with the application, either by static linking or shipping the dynamic libraries.
-ian
Bob:
I’m not sure where you come up with the assertion that “there are many different implementations of linux that ignore the LSB”.. Take a look at http://www.freestandards.org/en/LSB_Distribution_Status, and you’ll see that virtually every distribution you can think of is either LSB certified today or will be LSB certified in its next version. There are a few exceptions (e.g., Gentoo), but I’d hardly call those few “many”, and even with those few exceptions, the current list has to represent 99% or more market share in total. That’s pretty broad support by any measure.
-ian
Glen/Knox:
No question, there are opportunities for improvement here, and as I said in the original post, “[f]rom [here], perhaps we can do more, but even the first steps can be quite valuable in their own right”. One issue we briefly talked about at the Packaging Summit was the repo metadata project, which could conceivably make it possible to create a single repository format that APT or yum could consume. Another was better tools—if it were easier to create RPMs, and if it were easy (and possible) to create a single repository for APT and yum, would ISVs do it? The first steps have to be small. Adoption is key, and you don’t get adoption by ripping and replacing, no matter how good your replacement is.. The answer to that last question from the ISVs in the room, by the way, was “no”. There’s simply no interest in supporting Linux specific package systems today. Now, if enough software purchasers and administrators vote with their wallets that ISVs support these package systems, I suspect they’d change their answer pretty quickly. However, the fact of the matter is that it’s not a priority today, and our goal is to improve the situation given the current environment.
P.S. – We’d love to have end user participation—join the mailing list and get involved!
-ian
Pingback: Mike’s Thoughts » Linux on the Desktop - Different Ideas and Theorem
Thomas Leonhard said (in a response to the first part):
“However, Klik isn’t decentralised (there can’t be two different klik packages in the world both called ‘convert’ for example, and if the central server is down then nothing can be installed anywhere in the world).”
Thomas, that’s right — klik isn’t decentralized. Yet.
You can probably imagine that the klik client can be easily extended? So it attempts contacting not just one, but (in turn) two, five, hundred,… $any_number of different klik server mirrors for fetching a specific recipe until it succeeds?
What you point out is not a fundamental, eternally binding limitation of klik’s design, but just one in the current implementation of its basic principles (an implementation which is a “usable Alpha”).
If we agreed all for now, that the basic ideas and architecture of klik are great, we can have multiple klik servers implemented next week who would regularly synchronize their recipes and load balance their activities.
However, that agreement is probably not yet here. :-)
(Your other point, about two different packages named “convert” I didn’t quite get. You can’t be meaning that the same package name should be able to map to two different things [where f.e. one of the converts would be a “destroy”]…)
I have a simple solution: Gnome and KDE have the power to include a common standard interface for packaging systems. E.g. they include a YAST style interface and a yum style interface. The distributors can then provide their own backends if they want to or just take the standard tool.
I mean just the GUI and a reference implementation, say for Ubuntu or Debian and SuSe. So packaging systems for Suse and Debian can be used uniformly by users if they want to use the default component. Sure, distributions can ignore that and come up with their own version but then they have to justify these investments in what is already provided.
This also applies for system configuration. There were plans to port YAST to Debian. Tools like qtparted should be integrated into the standard toolchain of KDE. Variation and fragmentation here makes little sense.
LSB sounds promising but does not happen. But there are some things which need to be harmonised. For instance a standard directory for codecs.
install://software-098
—
Klik is great but I don’t want these 10 virtual drives in my directory.
—
Software description: We have a commercial standard format for these: PAD
http://www.asp-shareware.org/pad/index.asp
Open Source does usually not support it.
@Gerd:
You said, “Klik is great but I don’t want these 10 virtual drives in my directory.”
I’m not sure what you meant with “10 virtual drives”. Probably the loopmounted .cmg files? These are not in your home directory, they are currently mounted in /tmp/app or /tmp/mnt
In a not too distant future (when we can rely on FUSE [file systems in user space) to be present on most systems), we will probably use the fuse mounting abilities… (and, there are some more alternatives to explore).
Ian,
Re: http://www.freestandards.org/en/LSB_Distribution_Status
I count 12 “planned” and “not certified”, out of the listed 23, which makes up to 50%. The list is incomplete.
(I observe that ubuntu is certified, while debian is not.)
So, what was your point?
… To clarify my former point, I did not mean to move against the LSB; I just meant that many linux implementations are not LSB compliant (to date). I think the LSB standardization is useful, if and when it will be adopted. Managing linux implementations is like herding cats, so it is no wonder that LSB is taking time. (The similar problem occurs with the GUI, with gnome/kde/…; I think the best move here is think cross-platform, so that applications can be deployed not only on linux, but also on windows and osx. For this to be possible, it would be very nice if the linux community would agree to dump each and every dogma, and set to develop a single cross-platform toolkit.)
Bob:
Of the “12 ‘planned’ and ‘not certified'”, 10 are planned, which means that 21 of the 23 will be certified in the not too distant future (and keep in mind that many of the 10 “planned” haven’t even been released yet). 21 of 23 is over 90%, and as I said before, when you factor in market share, we’re even higher than that. There’s plenty to be critical of as far as the LSB goes (historically speaking—we’ve come a very long way in the past year!), but distro support isn’t one of ’em.
-ian