Vale Unity

Its been almost six years since Ubuntu shipped with Unity as the default desktop and compiz as the underlying compositor. For every release since then, a similar software stack has shipped on every subsequent release up until 17.04 . Those ten releases make up about half of the Ubuntu desktop’s lifespan and certainly more than half of the person-hours invested into the project, so today’s announcement to wind it down is a pretty significant moment.

I haven’t worked on Unity or Ubuntu for a very long time now, but the announcement today has filled me with a couple of emotions. On one hand its disappointing to know that your legacy and a project that you personally invested a lot into is going away. Time truly is the destroyer of all things. I know Mark and a lot of other people at Canonical must be feeling particularly gutted right now. On the other, maybe its a sigh of relief. Unity certainly hasn’t been the most conflict free or easy project and maybe with this difficult decision the people involved finally have the freedom to move on without feeling guilty.

The Team who Built Unity

I have nothing but good things to say about the Desktop Experience Team at Canonical – later the Product Strategy Team and the associated Ubuntu community. Our core engineering staff was a killer combo. We had two people, Jason and Neil, who had amazing experience with graphical toolkits and a passion for Desktop GL. We had Mirco, Gord and Jay who knew how to extract every last ounce of performance from a GPU. We had Tim and Alan who were a marvellous leaders and a champion of stability and software maturity. We had Alex, Thomi and Thomas who championed automated testing. We had Neil again because he literally filled far more shoes, so he deserves more than one mention. We had Michal and Mikkel –  who were a masters of desktop search. We had Didier and Seb, who could get things shipped, no matter what. We had Robert – who knew how to put the pieces together for groundbreaking web-integration. We Marco and Andrea, who took the lead in maintaining the desktop even when nobody ask them to. We had Daniel who was a leader in desktop quality. We had John, Otto and Rosie who were amazing designers. We had Jorge and Jono who were personable and enthusiastic community managers. There were countless others making invaluable contributions. And then we had me – and I’m just humbled to have crossed paths with all these people.

We were co-workers but we were also friends. Not only friends, but very close, lifelong friends. We often talked about non-work related stuff to each other and really went the extra mile to support each other. We were under a lot of pressure to make the dream a reality and when you gel well together, that pressure makes friendships even stronger. We didn’t see each other in person much, but when we did, it honestly made up some of the happiest times of my life.

The Fun Moments

We had a lot of great moments with each other building Unity. There was a time when we had to instruct Mark on how to pull and build a new build back when we didn’t have stable updates in the alpha release. Mark was on a cellular connection and had to log out of IRC in order to restart his session. I remember Jason saying “whelp, if he doesn’t come back in 1 minute, assume we are all fired”. That got a few laughs.

There was another time when the DX team and the Design team had an ongoing prank-war with each other. This started with the “flies in the ice-cream” prank and eventually culminated with a sardine pizza being sent to Jason from England. I still have no idea how that happened. Then there was also the bug report.

We slipped all sorts of easter eggs into Unity. One, which when launched with UNITY_NEKO=1, replaced all images in the dash with images of cats. Another which almost made it in filled the screen with fire when the konami code was pressed.

The Low Moments

Of course, Unity’s development had its bad times. The development was, understandably not well received by the established community members who saw the project as fragmentation. We faced numerous quality control problems when trying to get the desktop out on such a tight timeline. The stacking bugs kept me up day and night and required a lot of personal sacrifice to get to a state where they stopped occurring so frequently. Some of our users were randomly hateful and/or toxic. For instance, someone in a Reddit AMA told Jason that they hoped he would be “hit by a bus”. That’s not a very nice thing to say. In the later half of the 11.10 cycle, I began to feel overwhelmed with matters going on in both my personal life and the status of the project and entered a deep state of depression. There was additional pressure in the company as Unity 2D was developed and the projects inevitably, though perhaps not intentionally, were pitted against each other.

Then there were the post 12.10 staff departures. It was really sad to see the team break apart almost as quickly as it formed. I left during this time as well after most of my friends on the team had left.

Unity 8

Unity 8 was a total reboot of the project, learning from the failures of Unity 7. There were a lot of high hopes. Canonical had recently been on a recruitment drive and hired a lot of smart people. On paper, Unity 8 looked technically groundbreaking. Unity was going to move away from an ageing, unmaintainable and limited display server architecture to one which was pervasively multithreaded, could take better advantage of mobile and desktop hardware, had high test coverage and really top-notch developers working on it. Global package management was going away in favour of isolated and atomically upgradeable containers known as ‘snaps’. The OS was going to make mobile hardware a first class citizen. Touch was going to become the new interaction paradigm. Everything was going to become more internet connected and integrated than before. Apps were going to become smarter on power and resource usage. The project was moving to a toolkit which at the time had an incredible backing and momentum behind it. Legacy apps were going to be supported. “We have to go deeper” was the motto of the team.

The pieces were in motion. A very real demo of this was ready in 2013 for announcement to the world. The technology was so ground breaking that it was going to continue under wraps for a while until it could reach perfection. And then we’d be in free software utopia. Convergence would become a reality. Ubuntu would win, by sheer force of being better than anything that came before it. The plan, idea and resources available to execute on that plan were perfect.

And then it was delayed.

And then it was delayed again.

And then it was delayed for a third time.

And now its not shipping.

An outsider’s postmortem

I’d be lying if I said I didn’t really see this coming, but not for the reasons that other people might say. The strategic decisions made were necessary – X11 was seriously limiting our ability to deliver on features and stability, without atomic updates, its hard to give people a secure and stable phone – but they were ultimately fatal. I remember back when we decided to rewrite compiz and completely underestimated the scale and effort to make that work in a bug-free way. It killed the project. Once you rewrite one thing, its very difficult to resist the urge to start rewriting other things while you’re at it. Porting compiz to C++ wasn’t enough; we had to split rendering into plugins; we had to support the non-composting case; we had to support reparenting; we had to support a replacement for function pointer hooking; we had to drop multi-screen and we had to move to a more up to date rendering model. All at once. We had to understand code that we hadn’t written and code where the original authors were long gone from the project. And to make it look like progress was made, we had to bolt on more features on top of the existing rewritten ones with fewer development resources than the ones who had already burnt out and quit.

About this time last year Unity 8 started to look an awful lot like compiz++ on a larger scale. Not only was it going to use Qt, it was going to use an entirely new display server! Not only was it going to use an entirely new display server, it was going to use an entirely new driver model! Not only was it going to use an entirely new driver model, it was going to use Android as the base for mobile! Not only was it going to use Android and Debian, but Debian would go away and we’d have containerised applications! Not only was it going to use containerised applications, but the applications would be all-new and written in Qt! And not only all that, but its going to be perfect!

The delays just kept on happening. And after ever delay, so as to motivate the base and show that Unity was still relevant, more scope was added. More scope led to more delays! More delays led to more scope! And its not going to ship until its perfect!

I became worried around this time last year that it was never going to ship. The reality was that “perfect” became a moving target. And the thing about aiming for perfection is that you can never hit it until you’ve had your imperfect product in front of users for a few years. Ironically, Unity 7 is now seen as “perfect” for a lot of people, because they don’t remember the change from GNOME 2 to Unity. And now there’s going to be another transition in moving to GNOME 3.

Community

Unity was all about turning a community made platform into a product. The goals were noble – we wanted to get this amazing bit of community goodness into the hands of so many other users. We wanted to preach the word of free software. We wanted to show people that they do have an alternative to the big four companies that control the software they use on a daily basis. We wanted to show people that free software could win! And the only way to do that was to turn it into a product that was so irresistible and incredible that you couldn’t not buy it.

And that’s where the problem was. The developers making it wanted a community but in order to get it to where it needed to be, it needed to become a product. There needed to be one direction instead of many. Linux was no longer about choice. It was about the masses. Unity had lost its soul.

Where its all going

For now, it looks like Canonical has folded on client to double down on server. That was a very difficult decision to make and I applaud the leadership for their courage and for their ability to recognise what was becoming sunk cost. Its sad for the developers, product people and community involved, but at the end of the day, you have to make money.

Free software on the desktop might enter a dark age soon. There aren’t very many companies in this space now.  Intel was out as of halfway through last year. Google is focused on Android and Chrome OS. Novell and Sun folded a long time ago. Nokia fell off a burning platform. IBM is nowhere to be found. And now Canonical is pivoting away too. Samsung’s Tizen is potentially an interesting player, but everyone knows its a plan B.  Red Hat and SUSE remain and so does Endless. An of course there is still Collabora, Codethink, Igalia and other free software contracting firms, though (edit) their participation depends on the re-use of our technology by larger players outside the desktop space. There’s room to grow, but there has certainly been a lot of disruptive change lately.

(edit 10/04/2017: This section got a little more traffic than I was expecting and I wrote it quite quickly. I’ve made some changes to better reflect the state of affairs based on feedback. I’ve replaced “their participation depends on the larger players” with “their participation depends on re-use of our technology by larger players outside the desktop space” and clarified that its not all doom and gloom! Just that there has been a lot of change lately and things aren’t pointing in the most golden direction right now)

A new hope

Lets not forget where the desktop free software revolution came from. It came from ordinary people, you and I, who wanted to make a difference. It came from people who didn’t want to accept the Microsoft and Apple duopoly on computing. Even if the resources start to dry up now, it doesn’t mean that free software is gone forever. Some of the best innovations came during the post-bubble period of 2000-2010 where the software world had become stagnant with Windows XP and IE6.

There will always be people who want to do something different. A community will form again. And once everyone has a clearer head, free software will rise again.

Advertisements

35 thoughts on “Vale Unity

  1. You made something awesome. With Compiz, bringing amazing eye-candy showed people that Linux could be beautiful, and that attracted lots of users. (And showed devs that users care about looks). Not only that, it looked so good without hoarding all the resources of the machine. Not only it was nice, it was good. For me it was convenient. I relayed on it so much. I loved Compiz. For a long while I maintained the packages in Exherbo Linux.
    It got recognized, and with Unity, it spread. It was a good change. Haters scream loud, sadly. But a lot of people loved (or ended up loving) it. A lot of people went to ubuntu because of it. Maybe they later changed distros or whatever, but thanks to what you started the community grew bigger.
    You changed Linux. Non-trivially. Forever. For good. You should be proud.

    Best wishes.

    1. Thanks @aldraia ! I’m still envious of what I and everyone else were able to achieve back in the day and I hope that one day I’ll finally be able to beat that with something better. But its really nice to hear the words of support. I’m glad I made your life better 🙂

  2. I think you missed someone from the team. There was this crazy-haired kid that never slept, took all bugs like as a personal insult (and therefore tired to fix them immediately), that could make application windows do some pretty crazy things – including not actually showing up 😉 ! He was an amazing person to work with, a great person to just hang out with, and I think someone that the rest of the team looked at like a younger brother (that had mild ADHD).

    Thank you for always being awesome, Sam – we all did something that few people have the opportunity to do and, while it didn’t stand the test of time, I think we made a real impact. We also managed to made some good friends along the way 🙂

    1. Aw, thank you for the kind words Neil! I really loved working in the old team, and I honestly believe that it was the people that made the experience. You helped define that culture. Nothing in software ever stands the test of time, but it was such an amazing experiment that I’m still honoured to have been a part of. I really do consider you and everyone else on the team to be lifelong friends even if we don’t get the chance to speak to each other that much these days. I hope you’re well.

  3. I hadn’t really thought about all of the scope creep that went into the ever-delayed Unity 8 launch. What you wrote makes complete sense, though.

    I’m a happy Unity 7 user. You and the rest of the team built something excellent. Best of luck in the future.

    1. People enter this state of a manic God-complex when they start building things from scratch. When you make one thing, you really do start to believe that you can make anything. Around the time Unity-Next was launched, as far as I could tell, nobody really seriously asked “can we actually do this?” People didn’t think about the complicated interactions between all the new components that were being built. Everyone just thought they were invincible. We could do anything because Test Driven Development had our backs. It was going to be perfect because the team was perfect and the process was perfect. We could wait for ages before putting it in front of users because we could make it perfect.

      It was so easy to drink the Kool-aid and be blind to where you were going. I did.

      1. Oh sure. I’ve seen the same mistake many times and made it many times in my day job.

        I think part of the engineering discipline in our field, when we remember to apply it, is designing big changes in small pieces. The pieces have to be self-contained and testable (manually or otherwise). If you’re trying to change two things at once, it’s a big gamble. Three or more at once is like setting your software on fire. I’ve never seen it work personally.

        1. Exactly! I takes painful experience to learn this lesson, but I think its one of the most valuable lessons one can learn.

  4. I miss you guys, I remember 11.04 being a zombie state of little-to-no sleep and some days it felt that only thing that kept it going was Didier’s shill willpower to make it all work. I’m still not getting a Mac though Neil, if you must know!

    1. Man, Didrocks really was the glue that held the ship together. Integration is the hardest part and whilst I didn’t appreciate it so much then, I definitely appreciate it more now.

    2. I really do miss everyone too! There’s really something to be said for the team gelling together well and its a shame that it all sort of just boiled over to the point where people needed to move on. Losing constant contact with friends is really really tough, but perhaps inevitable. I hope you’re well, Jorge 🙂

    1. Ah yes! Of course, SUSE is still around and growing in fact! I haven’t been watching it recently, but its very exciting. Of course when I say Novell is gone, Novell well and truly is gone and so was SLED which went with it. But of course, the community bounced back and SUSE found a new home, which is great news. I hope the same thing will happen, even though we’ve been seeing a few organisations in the space move recently. Its a very tough space to be in, but one which I think is very rewarding.

  5. Hi smspillaz! Long time no speak! Loved reading this post – it’s the most accurate history of Unity I’ve seen posted. And it was awesome working with you and the rest of the dev team, I treasure many fond memories of our time working together.

    Re. the pizza, that was my idea, we dug up Jason’s address and then googled for pizza delivery places near where he lived. The fish on the pizza were supposed to spell out a word (I can’t remember exactly what it was, might have been ‘happy birthday’) but unfortunately the fish moved a bit as the pizza was being delivered. I remember the pizza place we called up was quite surprised to have a call from the UK making such an odd request 😉

    To add one thing to your post, in retrospect the move from to QML combined with the ‘squads’ engineering management experiment was the ultimate culprit of the eventual failure (in my mind at least). Many of the key people on the engineering side seemed to leave as a result of these events, and with most of the key architects of the Unity7 technical architecture gone, Canonical seemed to launch into a quest to rewrite a much larger portion of the stack than we had ever tackled with Unity7. Yes the old stack was old and imperfect, but as you mentioned in your blog post it was working! In many cases better to incrementally improve software than attempt a massive ‘black hole’ rewrite. It’s a very typical cause of software project failure unfortunately…

    Anyhow, all my best regards 😉

    1. I remember the photo and the blog post that he posted about this! I think what happened was that all of the sardines had sort of drooped down to the bottom because the delivery person carried it vertically. It was pretty hilarious when Jason started posting in IRC about “wtf, how come I got this pizza with sardines on the bottom and I didn’t have to pay for it”. Took us ages to find out what had happened and then we all laughed pretty hard when we found out that it was your idea!

  6. It’s probably not as simple as to be described with a single phrase, but your post makes it sound like the ultimate cause was project mismanagement… It should’ve been the most important aspect to be learned from Unity development so far, IMO.

    In my subjective view, the biggest reason it was controversial, beyond the sudden disappearance of the all too familiar panels, is that Unity features were delivered piecemeal. Whatever features they shipped with worked well, more or less, but it was a complete DE until four releases down the line since the major desktop debut (after netbook incarnations).

  7. Nice post Sam! ❤

    I am amazed that we pulled off a desktop shell that went more than 5 years with only minor updates, and still keeping many users happy. I am really proud to have been a part of that. Sure there has been loud mouths on the internets, but when I meet Ubuntu users out and about they are generally content with Unity 7, oblivious to the debate online.

    I'm still a happy U7 user, and expect to be for some time yet! 😀

    /me goes back to nervously anticipating the future of the free desktop

    1. Its actually really funny – I’ve been meeting more and more people IRL who use Ubuntu and sometimes I drop into conversation “how funny, I used to work on that! I hope you haven’t had any problems with window stacking, but if you do, you’re welcome to direct your rage at me.” People really love it and people have really grown to love GNOME 3 too. Its interesting how people’s opinions change over time when norms shift, even if nothing happens to prompt those change in opinions.

  8. Loved reading your post, Sam Spilsbury!

    Do any of you, (former) insiders, know what is going to happen with Desktop maintenance at Canonical from now on?

    Mr Mark Shuttleworth isn’t really clear about this in his blog post …

    Will Canonical maintain it, re-package it, build upon it? Will there still be a team actively working in this field? …

    1. Hi Antoon,

      I can’t speak for Mark, though the most recent news I’ve read is that there will likely be some investment in GNOME, which is certainly encouraging to hear. The Mark I know is very passionate and while this would have been a very difficult decision, you can tell that he still wants to do what he can to make the Linux Desktop a reality, even its its not on the same scale or with the exact same vision.

  9. > We had Michal and Mikkel – who were a masters of desktop search.

    So who’s brilliant idea was it: to leak all the search requests to Amazon by default – Michal or Mikkel?

    > We slipped all sorts of easter eggs into Unity.

    Have you ever considered slipping in some actual functionality instead?

    > Free software on the desktop might enter a dark age soon.

    Oh, please! Can you be even more self-absorbed? The desktop GNU/Linux was fine before your NIH, and it will be equally fine after it. This sort of attitude towards community you’ve shown in this post is the root of unity’s problems.

    1. @Jack – Everyone can pick and mix their Canonical issues. But there’s no need to be abusive to Sam on his own blog. I wouldn’t come round to your house and yell at you about how you’re at fault for stuff your boss did five years ago. Please, if you can’t be a good commenter, don’t comment at all.

  10. […] hay augúrios de una nueva «edad oscura» para el escritorio de […] ” People really love it and people have really grown to love GNOME 3 too.

  11. As someone who honestly believed that Unity 7 was (and still is) the most beautiful and one of the most functional DEs/Window Managers on any OS (for example, I appreciate how your design took into account the issue of limited vertical space on many systems), and who had been looking forward to Unity 8, thank you for this fascinating, if sad, history of Unity!

  12. […] hay augúrios de una nueva «edad oscura» para el escritorio de […] ” People really love it and people have really grown to love GNOME 3 too. […] hay augúrios de una nueva «edad oscura» para el escritorio de […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s