Saturday, April 13, 2013

Systemd: The New PulseAudio

by Dietrich Schmitz

In the name of progress.  Change is inevitable or is it?  In some cases change can be quite good.  In others, it can be quite contentious and there's no shortage of that on the subject of systemd, a wholesale replacement for the System-V init daemon. (Image credit: www.macho.com)

It is so noted in Wikipedia:

systemd is a replacement for the Linux init daemon (either System V or BSD-style). It is intended to provide a better framework for expressing services' dependencies, allow more work to be done in parallel (concurrently) at system startup, and to reduce shell overhead.

Thus far, there seems to be a division in adoption.  My Distro, Fuduntu, a fork of Fedora 14, does not employ systemd.  I tried to reach +Andrew Wyatt for comment who was not available, but was lucky to find on a Saturday +Shawn W Dunn one of the Fuduntu analysts who gave this comment regarding systemd:


Systemd, whether by design, or circumstance, is largely becoming non-optional. Inclusion of core technologies such as dbus and udev are reducing choice for linux users and developers, rather than expanding them--which is the very antithesis of the idea of Free/Open Source software.
[Edit-- Last minute, +Andrew Wyatt, Fuduntu founder, came through with this comment:
ConsoleKit + UDev + Syslog + DBus + Polkit + Sysinit + this + that.   RedHat Enterprise Systemd is the best product we've ever been force fed.  We are facing being forced to integrate it at Fuduntu because it's replacing so many core tools now that it's impossible to continue the project without it.
Those "idiots" that don't like binary logs aren't "idiots", some of us actually have some idea of what we are talking about, but what do I know..]

 This Wikipedia list shows some that have gone head-long into adoption:


Red Hat will incorporate systemd into RHEL 7.


In sampling the blogosphere, this reddit talkback includes discussion of the pros and cons of systemd:

Pros:
  • Uses parallelization, a lot of it
  • That means that some daemons are started simultaneously, which means boot time should be faster.
  • Has a convenient API
  • systemd supports DBus and sockets, so you can easily control it and talk to it from your own code
  • The unit syntax is way simpler
  • For most cases, all you need to do is start a daemon on boot and kill it on shutdown. Old bash-based init systems need a large piece of boilerplate code to do that, but systemd doesn't. A common unit syntax is also easier to work with for developers, because you only need to support one init system, and not tons of <something> init derivatives, OpenRC and whatnot.
  • Integrated logging
  • As an init binary, systemd knows more about other processes than, e.g. syslog, so it can log data in a more convenient way. For example, you can get logs for a specific process, unit or target. You can also add additional information to the log if your code uses systemd's library.
Cons:
  • Everything in one package
  • Currently, systemd has a lot of features in a single package. QR codes for log verification, a built-in HTTP server, json serialization, you name it. This means a lot of dependencies that are not actually needed. Lennart promised to split those out into separate packages later, but no one knows when 'later' is going to come.
  • Not POSIX compliant
  • systemd uses things that are exclusive to Linux, so it can't be used on *BSD systems. This makes *BSD people unhappy. If you use Linux, you can probably ignore this.
  • It is forced aggressively
  • As much as I like it (and yes, I like it), seeing GNOME enforce systemd as a strict dependency is just wrong. Also, see the previous point.
  • Lennart
  • I'm not sure if his personality is a valid point, but he seems to take a 'I'm right and fuck y'all' stance in some cases, and I don't really like it. Also it's quite common for his code to be really buggy (see early systemd/pulseaudio), but it's not really imporant any more now that a quite large team is working on systemd.


Marching to the beat of a different drum.


Apple chose to employ launchd in their Darwin BSD Unix derivative operating system, beginning with Mac OS X Tiger.  Canonical developed Upstart for their Ubuntu Linux Distribution.  Oracle's SUN Solaris employs Service Management Facility.  And finally, Gentoo Linux uses OpenRC.  It should be noted that recent newcomer ChromeOS is Linux-based and 'under the hood'  the compiler toolchain is Gentoo-based.  As to why Google chose Gentoo for ChromeOS is a subject for another story.

Conclusion

So, clearly, it's different strokes for different folks and while change is good, the contention over whether systemd is really a good thing remains hotly debated.  One thread concerning 'The Bad' in the Arch Linux community is fairly representative of the concern for adoption of systemd.  Arch Linux has a loyal following of pragmatic users who enjoy working at a component level because of how it allows one to truly learn the 'build your own' Desktop.  The result is a clean, lean system and their is purity in that.  So, they might be the most vocal of all critics and rightly so.

In a separate indictment from my esteemed colleague and friend +Aaron Seigo, he writes in my private Linux Advocates Google Plus Community the following:

As a perhaps relevant side-note, systemd is (imho) the new pulseaudio: it breaks things that were working (which is bad) to bring new features we couldn't get at with the old system (the intentions are good). It is not particularly easy to use, it is easy to get wrong and it is hard to debug when things go wrong. We've experienced all sorts of wonderful things during its short tenure inc triggering kernel panics.
Anecdote time: We use systemd in Plasma Active images (for various reasons I won't get into here). Recently we ran into an inexplicable set of crashes. Turned out one of our systemd service files had an empty line at the end of it, and systemd does not put up with that. It just simply skipped over this offending file and a critical service was not being started. There was nothing in the logs to note this was happening, or any hints in the documentation. We found the "solution" out the hard way, and I'm still baffled that a blank line should even matter.
Even more fun is how tightly welded specific versions of systemd are to specific versions of the kernel .. which on devices can be "fun" as kernel versions that are supported for a given SoC are often (sadly) limited.
The foundational ideas behind systemd, borrowed from another OS, are sound ... but the implementation is simply not there yet and I sense it is overreaching too early in the game in places (e.g. also taking on logging infrastructure).
I suspect it needs a "Colin Guthrie" type figure; Colin imho rescued pulseaudio, moving it out of the 7th level of hell into a slightly chillier level of hell .. pulseaudio is still a cranky bitch at times, but it works tolerably relative to prior incarnations since Colin began working to make it so.
So on the one hand, I'm rooting for systemd. On the other, I wish it would stop sucking .. and I really wish it had a different set of developers working on it.

As for myself, I am not convinced that systemd is a good thing.  Keep your powder dry.

-- Dietrich




Enhanced by Zemanta

38 comments:

  1. "Systemd, whether by design, or circumstance, is largely becoming non-optional. Inclusion of core technologies such as dbus and udev are reducing choice for linux users and developers"


    Wait i thought we wanted a standard for things? , Lennart said init had too many bugs, im not clever enough to know. But i like systemd. My distro Archlinux asked anyone in the community if they wanted to continue mainaining old initscripts so users could continue to have the option, nobody stepped up.
    see Lennarts talk on systemd here https://www.youtube.com/watch?v=TyMLi8QF6sw

    ReplyDelete
  2. There is one wrong thing in the Con's section. Gnome isn't forcing a dependency on systemd, or logind for that matter. They are forcing a dependency on the dbus interface that logind provides. Other login daemon can provide that same interface, other than logind. But the new standard interface is the same one that logind provides by default.

    ReplyDelete
  3. Cons:

    Everything in one package
    Yeah, the same way "coreutils" comes in "one" package or the kernel comes in "one tarball"
    Currently, systemd has a lot of features in a single package. QR codes for log verification, a built-in HTTP server, json serialization, you name it. This means a lot of dependencies that are not actually needed. Lennart promised to split those out into separate packages later, but no one knows when 'later' is going to come.

    The QR feature is optional, Systemd does not have a built-in http server, it uses the libmicrohttpd to provide a separate, optional component to browse the journal data
    Not POSIX compliant
    Is this an argument.?. it is not POSIX complaint because it is designed for linux,
    systemd uses things that are exclusive to Linux, so it can't be used on *BSD systems. This makes *BSD people unhappy. If you use Linux, you can probably ignore this.
    Well.. yeah, BSD people can take their own way, we should not under any circunstance stop producing linux software because it might upset BSD, or windows or Apple, that's a nonsencical argument.
    It is forced aggressively
    In the same sense that new phones, CPUs or monitors are forced, yes.
    Lennart
    Yeah, nice ad-hominem attack, when people have no arguments, no ideas and no clue they go to the person's characteristics

    ReplyDelete
  4. I spoke of just two things:


    1) FHS filesystem standard
    2) Grand Unified Package Manager


    That is all. Out!

    ReplyDelete
  5. "We've experienced all sorts of wonderful things during its short tenure inc triggering kernel panics." Do you know that kernel panics are bugs in the kernel and that anything in userspace triggering panics is a bug in the kernel and has NOTHING to do with systemd ?
    "Turned out one of our systemd service files had an empty line at the end of it," Ok, that's a bug, where is the bug filled ?
    "Even more fun is how tightly welded specific versions of systemd are to specific versions of the kernel .. which on devices can be "fun" as kernel versions that are supported for a given SoC are often (sadly) limited." WTF ? problems with the kernel or the lack of upstream kernel versions for SOCs have nothing to do with systemd, this is a totally fallacious non-sequitour argument.

    ReplyDelete
  6. "ConsoleKit + UDev + Syslog + DBus + Polkit + Sysinit + this + that. RedHat Enterprise Systemd is the best product we've ever been force fed. We are facing being forced to integrate it at Fuduntu because it's replacing so many core tools now that it's impossible to continue the project without it." Oh, dear, the victim mentality.. no one is force feeding anything to anyone, you are free to keep the old components and do further development on them, it is opensource! ConsoleKit is deprecated, udev can be used without systemd, DBUS has been here long before systemd, PolicyKit too.. more non-sequitours and bullshit please!

    ReplyDelete
  7. Please, tell me how I can compile udev without also compiling systemd? Also: http://lists.freedesktop.org/archives/systemd-devel/2013-March/009797.html

    Right, as you say we are welcome to continue to develop all of these various components ourselves. Like we had nothing better to do but support what we support today plus this extra stuff too. It isn't victim mentality, it's weighing the effort of what we maintain today + all of this additional work vs adopting systemd (hence force) but don't let logic derail you.

    RedHat Enterprise Systemd (+ Linux) is going to be great.

    ReplyDelete
  8. libsystemd-bus + kdbus plans : Yeah, that's a completely new effort, in the future systemd will use kdbus, a kernel-side dbus implementation of the dbus protocol, the reference userspace part will live in systemd, the kernel part is developed by Greg KH/the linux foundation.
    as you probably know, kernel interfaces are available to anyone that wants to develop their own thing, you are not forced to use systemd to use any kernel feature.

    ReplyDelete
  9. I was sceptical too until i used it. Its not too bad at all.

    ReplyDelete
  10. No doubt when everyone moves to wayland (except ubuntu) there will be the same murmerrings of discontent. Onward & upward.

    ReplyDelete
  11. Jonathan Fischer FribergApril 14, 2013 at 8:46 AM

    "which is the very antithesis of the idea of Free/Open Source software."

    The idea of such software is that it should be open source and/or free. Nothing else.

    On another note, pulseaudio is probably one of the best things that happened to linux. It had a lot of problems in it's infancy, sure, but right now it's definitely the best way to do audio on linux (for normal users, for people doing audio work jack is probably the better choice).

    ReplyDelete
  12. At least one set of bugs were in the cgroups code in the kernel, and systemd would trigger conditions that would cause the kernel panics. Otherwise, the system was rock stable. It was an ARM device with a slightly older, but not *that* old, kernel .. but it was old enough to not have these cgroup fixes. When we tried to move that image to using systemd, it simply failed and merging in the fixes was going to be a lot of work. So instead we ended up not using systemd there and having a rather different image profile for that device relative to others we were working on at the time.

    So the problem was in the kernel .. but the trigger was systemd. Which is fine: the kernel gets fixed and the world moves on. The problem is when such systems become mandatory too early, before the fixes can propagate.

    "Ok, that's a bug, where is the bug filled?"

    We would need to re-confirm on the same image but with a newer systemd, and that's unlikely to happen at this point (welcome to development on devices). With a bug report in hand, however, it would still smell like a system that was not tested well enough yet to become a mandatory key part of the OS stack.


    "problems with the kernel or the lack of upstream kernel versions for SOCs have nothing to do with systemd"



    You're missing the point, which is not "systemd has to fix the world" but that when Linux middleware gets changed and updated, it ought to be done with some real-world reality checks in place. It's all well and good to have systemd (it does many things right), but pushing it too hard and too early causes problems for many people trying to use the technology.


    This is precisely the problem we went through with PulseAudio: it was pushed too hard, too early with all sort of excuses for doing so being made. In the end, it was a true disaster for quite a while and now it simply only breaks sometimes.


    Not having audio is more than an inconvenience on a desktop or mobile system: it's a deal break. Making it difficult to have a reliable init system is even more deadly.


    Unfortunately, many of the Linux middleware decisions are made with what appears to be insufficient planning (how many versions of hardware awareness sub-systems have we received now?) and development methodology that is too lax for the critical nature of the components.


    It's one thing to do "develop and chuck it over the fence for bug reports" if you're writing a DVD burner app for the desktop or some such. If you're going to start reworking key middleware, though, I'd much rather see the sort of rigor and conservative behavior we typically see from projects focusing on filesystem or database systems.


    If we can agree that the revolving door of Linux middleware and the breakages our users suffer routinely when they are introduced too early into "stable" OSes, then maybe we can do something about it by improving how these things get done.


    Denial is not particularly helpful, however.

    ReplyDelete
  13. There is often a difference between theory and practice when it comes to these things. As more tools and components use or even outright rely on systemd, it becomes far less optional. There's also the matter of division of labor that allows more parties to participate: if you start ignoring the labor input of others, you end up supporting an enlarging fork of OS components (c.f. Canonical).


    Personally, I think that's all well and fine with systemd as the design is quite sane imho. What I don't find great is the deployment timeline relative to its maturity and its reaching into things like system logging before the other basics are rock solid and well adopted. It's a matter of planning and focus.

    ReplyDelete
  14. A better audio system is good.

    PulseAudio still fails far too often on far too many systems on far too regular a basis ("play with settings in pavucontrol until it starts working" is a far too common recommendation). Still, let's pretend it is currently near enough perfect to be undeniably awesome right now.

    The issue is precisely what you wrote: "It had a lot of problems in it's infancy". For whatever reason, a lot of Linux distributions are hellbent on thrusting work-in-progress technology on the users of their stable OSes.



    The result is simple: for many users (far, far too many) they have a happy working system (they don't know how or why, but it is) and then they get an OS upgrade and something new breaks. That gets better (usually), and then something else new breaks on another OS upgrade. These in-between cycles of suckitude piss off our users and give Linux a horrible reputation, esp since they don't happen all at once but are sprinkled here and there over the course of years .. so that Linux never feels quite "right" over any extended period of time.


    There is a theory that if you just shove crap down the throats of users, people will demand it gets fixed and that will cause others to fix it. (This reasoning was applied to PulseAAudio and, for instance, and various audio drivers.) This sometimes works .. sometimes it causes forks and new systems instead. When it does work, the people involved (users and fixers) get pretty unhappy in the process. It's a highly antisocial method of trying to get things improved.


    Now, I know the temptation to take this approach first hand. When KDE's Plasma Desktop first came out, many X11 drivers were horribly broken when it came to OpenGL and various X11 extensions ... and we were the first to seriously use some of these features which were (shock!) known to be broken. We stuck with it (including working with vendors so they'd fix their software), but we also maintained support for non-composited desktops that was ~100% transparent (excuse the pun? ;) to the user. They lost a little eye candy and some minor features, but things at least worked.


    In all of that fun, a number of distributions decided to push Plasma Desktop into their stable releases before it was ready. We said "it's not ready for daily usage by regular desktop users" but the mantra of "new is better, who cares about the results" held sway.


    This way of approaching integration of fundamental changes to the stack, even when absolutely needing to happen, needs to shift to something more responsible.

    ReplyDelete
  15. There's a difference: Wayland is being adopted by consensus by the affected and interested Free software projects and it is happening at a pace that allows us to offer a stable and reliable path from x.org to Wayland (and the two will co-exist for quite some time).


    It is not a question of "should we go forward" but "how should we manage the process of moving forward".

    ReplyDelete
  16. I agree with you, Dietrich, and with Aaron as well. If systemd is anything like pulseaudio I wouldn't touch it with a ten meter pole.


    My last three computers were this one, an Acer V3-771G, a Sony VAIO VGN-FW140, an Acer Aspire 521D (wife), and a couple of Acers and HPs that friends use. Pulseaudio worked poorly or not at all on any of them. My solution was to remove the critical pulse audio libraries and volume control GUI, and install ALSA and its utilities. ALSA never fails me and gives me full wrap around sound with Dolby.

    ReplyDelete
  17. Thanks Mon. Where have you been? :P

    ReplyDelete
  18. Sold house, dropped ISP, moved, new ISP, changed to a different set of favorite things, busy, busy, busy with wife and grandkids. Cleaned out all my circles and started over, just dabbling here and there on the Internet. Will probably cut out even more Internet activities and just reduce it to banking, shopping and emailing family.

    ReplyDelete
  19. I am glad you are okay, that's all. Was worried when you dropped off the radar unexpectedly. Take care.

    ReplyDelete
  20. "The problem is when such systems become mandatory too early, before the fixes can propagate." this is a problem of integrators, after all if you are integrating an init system into a distribution you surely need to know what are you doing..
    "You're missing the point, which is not "systemd has to fix the world" but that when Linux middleware gets changed and updated, it ought to be done with some real-world reality checks in place. " This is once again a task for integrators and distributions, expecting upstream to test stuff in $whoknows what environment is not reasonable, adding explicit dependencies at compile time and tests will help though.

    "Unfortunately, many of the Linux middleware decisions are made with what appears to be insufficient planning" , maybe, but if you have ever worked in large distribution you will know that software moves much, MUCH faster than any plan could possible account for.

    "If we can agree that the revolving door of Linux middleware and the breakages our users suffer routinely when they are introduced too early into "stable" OSes," the chicken and egg problem, "too early" do not include it --> never gets tested in real life, never gets any attention, never gets matures for inclusion..

    ReplyDelete
  21. "There's also the matter of division of labor that allows more parties to participate" , all distributions that use systemd are working upstream, contributing patches etc.. even companies that do not directly produce general purpose distributions such as Intel.. or embeeded system developers.
    Even canonical has contributed some code.

    ReplyDelete
  22. The biggest myths surrounding systemd http://0pointer.de/blog/projects/the-biggest-myths.html

    ReplyDelete
  23. Some systemd myths http://0pointer.de/blog/projects/the-biggest-myths.html systemd was adopted by my distro as a choice made by the community and the unpaid developers, not because someone held a gun to their head, and the developers seen that it was good, i trust their judgement. Im sure there will be systemd free offerings in the future just as there are now.

    ReplyDelete
  24. as for being stable , the transition to systemd was simple with no instability issues, have you actually used it ?

    ReplyDelete
  25. If god forbid you had used systemd you would know that system logging (journal) works in tandem with syslog-ng and logs can still be read from /var/logs, https://wiki.archlinux.org/index.php/Systemd#Journald_in_conjunction_with_syslog

    so that point is null & void

    ReplyDelete
  26. "Let's just switch to any crazy idea Lennart can come up with." (Patrick Volkerding, founder and maintainer of Slackware Linux, created in 1993.

    Full thread here: http://www.linuxquestions.org/questions/slackware-14/slackware-and-systemd-885228/


    One more reason why I'm 100% Slackware on servers and desktops.


    Cheers.

    ReplyDelete
  27. Yeah, except Arch works nice if you're an Arch developer.

    ReplyDelete
  28. Arch works pretty nice to me, and I and I even have a forum account.

    ReplyDelete
  29. I forgot to specify I'm not only using GNU/Linux privately, but mainly on production machines for clients. http://www.microlinux.fr. In this context, Arch is unfortunately unusable.

    ReplyDelete
  30. I remember when the feature wasn't there, and Lennart decided that it wasn't important or required (good luck passing an audit without it) and after a long winded argument between Lennart and I (leading to an unfounded implication that I attacked him) the feature was silently added.

    ReplyDelete
  31. I with you on this. Lennart is bad news I'm sad to say. Doesn't seem he can just work with others smoothly.

    ReplyDelete
  32. As I've said many times before, the pain felt during PulseAudio adoption is a result of two specific issues:
    1. Distros did a shit job on integration.

    2. A large part was down to hardware.


    Distro integration is a biggie. I put a lot of time and effort into making the integration in Mandriva first class and it showed. I've had numerous compliments for the smoothness with which we rolled it out. It touches so much of the stack you need to put a lot of care and attention into it. Ubuntu, on the other hand, sucked big time. This single detail set us back a lot upstream. If more care and attention had been put into that at the time - i.e. the same as I did with Mandriva, I suspect we would be in a very different situation now (in terms of feelings of resentment at least).


    PulseAudio, much like in some of your examples above, exposed errors in other areas of the stack. Does that make it to blame? No!! PulseAudio has become the defacto stress test for ALSA drivers (I know SoC people who say, "test it with PA and if it doesn't work you're drivers not ready"). Since PulseAudio has come out the ALSA drivers have had a lot of bug fixes. To their credit, Ubuntu (or rather Canonical) has made up for their previous mistake in the form of David Henningsson who has been a major player in getting these fixes pushed.


    Like it or not, these fixes could simply not be done in the lab. "PulseAudio" would NEVER have been at the point of "ready" in this context because, quite simply we didn't have the resources to get all the hardware in the world to fix all these bugs. Like it or not, the users had to suffer the pain so that we could get the needed feedback. We had to push it mainstream and we had to get the bug reports. It's as simple as that. Sometimes you need to push before things are at the almost mythical "ready" point.


    Sure, in a Utopian world where resources and time is infinite and all h/w works perfectly to a particular spec, then things would be different. But you know what? We don't live in that world.

    ReplyDelete
  33. The logging infrastructure is seen as an essential part of service management. I'm surprised you're stating on the one hand that things are hard to debug and then on the other hand criticising systemd for focusing on the logging infrastructure which massively improves debugging. It also makes debugging things very easy. As we've (pleasantly!) argued in the past, I find that systemd debugging is actually incredibly simple and adds massively to the overall clarity of the whole boot process.

    ReplyDelete
  34. I'm afraid that's just not true. Have you done the conference circuit, have you participated in mailing list and IRC discussions? Lennart works with a *lot* of people all the time - this statement just isn't borne out by facts. I see people make this statement, but I really can't help but feel people are just jumping on someone else's bandwagon here rather than having worked with him directly. Sure if you come in being aggressive or arrogant, then he'll likely respond in kind, but you know what? So do most people! If you come in open and willing to participate then he'll also likely respond in kind.

    ReplyDelete
  35. You can't *compile* udev without systemd. That's like saying tell me how I can compile coreutils without libc. udev uses the same libraries and utilities that systemd does. It means the same code is reused. You can happily compile the package and then use udev outside of systemd.

    ReplyDelete
  36. No it is nothing like that, coreutils and libc are two distinctly different pieces of software where udev and systemd are the same merged codebase. You can not build udev without also building systemd.

    ReplyDelete