Are Intel subverting MeeGo.com?

Well... there's a "challenging" blog title :)

Please read on and allow me to explain.

Intro

So what's making me risk offending one of the powerhouses behind MeeGo?


First off, let me say that I think the answer is "No"; but I do think there's a problem and it should be solved.


So having hopefully gotten your attention; what's the problem? Well, quelle surprise, it's that inconvenient "openness" again.


MeeGo is an open distribution; I don't think that's really in doubt. However there have been many discussions on what makes something open and I'd like to add another: "Inclusiveness".


For the record : In the remainder of this post I've included issues that I've seen raised that I may not personally support.

Issue

The question is ... who gets to use the builds that are produced at meego.com? Well, technically "anyone", but practically "only people who've bought recent Intel hardware".


Why? Because the MeeGo.com builds are optimised for the Intel Atom processor; not a generic x86.


So what's the impact of this?


Pros: Less CPU usage for some functions (3D and audio codecs AFAIUI get +20% performance on certain tasks)


.... that's it


Cons. Well machines with a non-Atom CPU running MeeGo netbook edition may boot up but often fail to run. Additionally non Atom machines can't easily develop for MeeGo (eg the image building app uses ssse3 instructions inside somewhere.)

Impact

If a developer wanders over to look at MeeGo and sees that she needs to buy an Atom netbook to run it then: "Oh, I need special HW to run it? Ubuntu just runs on my laptop. Bye MeeGo, hello Ubuntu."


Making it hard to use MeeGo is *not* inclusive.

Details

The way MeeGo is built any binary could contain SSSE3 instructions; however which ones actually do depends on the optimisation flags used, the compiler version and the source code and and a change in any of them can add or remove instructions.


Essentially it's "random" - or at least not humanly determinable if a given application will suddenly choke on non-Atom hardware.


In irc Thiago scanned the Qt binaries to try and find any SSSE3 instructions. When he ran a quick and dirty analysis he couldn't find any SSSE3 instruction in QtCore and QtWebKit; and found a total of 4 SSSE3 instructions on QtGui - that's many 10s or 100s of Mb of binary code and only a handful of instructions.


So given the entirety of Qt uses *4* instructions there was a challenge to the "performance benefit" it offers by using these options "across the board".

Postition

Given MeeGo.com is supposed to be open and to minimise barrier to entry this exclusion seems wrong.


MeeGo is managed by the Linux Foundation and I was under the impression that we had a Linux Foundation managed build service at build.meego.com; so why are we building for exclusivity and not inclusivity?

Fairness

My understanding of the business model is that Nokia and other vendors would have an internal OBS to allow them to tailor and optimise MeeGo for their commercial product


MeeGo's model is that Intel, Nokia and others (as vendors) have their own internal build service (OBS) to optimise for a commercial product so why does Intel get to optimise for Atom on the MeeGo.com build service?


Why do we optimise for only Atom machines on the MeeGo.com builder? Why isn't that done on build.meego.intel.com? Is Intel essentially using MeeGo.com to make "the Intel MeeGo" not "the community MeeGo"?


Obviously, as a fallback the community OBS is a potential place for this generic build but that was *not* the point. We have often been told that MeeGo *is* the community.

Counters

One point has been of the "cost" of the shared infrastructure and Intel employees having said that Intel should not be burdened with the cost of supporting other arch/platforms/builds.


There's some validity to this - and I don't think the community would object in the slightest to seeing optimisations for Atom made available *as well* as a generic build.


One question is: Is MeeGo an open and inclusive distro? Who dictates the commercial aspects of the "shared resources" and who gets to dictate what is and is not on the agenda.


One argument has been "Intel contributed the hardware and they want it optimised for their platform"; However that's not a community platform; that's a vendor specific build platform.


And are we seriously saying that a vendor producing a version of MeeGo for a non-Intel based x86 device is going to have a problem rebuilding with appropriate gcc flags?

Proposal

Add a 'generic x86' target to the MeeGo.com build architectures. This will form the basis for more inclusive MeeGo variants.


Allow the builder to create targetted binaries for some hardware; this is going to become a requirement as more hardware appears.


As we know: 'optimized for Atom' sells and attracts developers to the platform and a set of development hardware, etc..


But surely we can be "optimized for Atom" without excluding non-Atom users.


This means anyone with a non-Atom device could at least start MeeGo on eg an AMD desktop. As Arjan says: "I and several of my coworkers run MeeGo to develop MeeGo" "it's still very much intended to be a quite complete *client* PC linux"


http://lists.meego.com/pipermail/meego-dev/2010-April/001318.html
http://lists.meego.com/pipermail/meego-dev/2010-April/001392.html

Works for everyone

Well, isn't that the job of MeeGo.com?


MeeGo.com is supposed to be open and to minimise barrier to entry - lets help it do that

Conclusion

The point is that MeeGo should be doing a generic i586 build


Not Intel, MeeGo

Open Letter / Proposal to allow Maemo on the MeeGo Community OBS

This is an open letter to the whole MeeGo community and on behalf of the Maemo development community. The purpose of this letter is to ask the MeeGo community for their permission to bring Maemo build targets (currently Fremantle eventually Harmattan, Diablo, Chinook?) to the MeeGo Community OBS and to ask the Maemo development community for their support in this project.

*Please discuss on meego-community mailing list*

I would like to emphasise that this is a Maemo Community initiative and is not being pushed by Nokia.

At this point we are not aware of any similar initiatives related to the Moblin community but we would fully support any that arise.

The Maemo community has built up around Nokia devices which, in many ways, are amongst the most open devices available in their class. There is a passion for openness in the Maemo community and we know that the future for this family of devices lies with MeeGo.

Many of us are looking forward to MeeGo and are keen to transition as smoothly as possible.

However our devices are not fully open and developing for them has dependencies on vendor proprietary binaries which would need to be available on the build service. This would mean putting closed binaries on the MeeGo OBS and having a part of the MeeGo Community OBS functionality being 'restricted' to Maemo developers.

Naturally we recognise and respect that MeeGo is an open source project and there may be ideological issues in allowing closed binaries into the ecosystem (even though they're just for build/linking). We also recognise the risk of "opening the door" to closed binaries and suggest that this arrangement could be agreed as a one-time "grandfathering in" (http://en.wikipedia.org/wiki/Grandfather_clause) situation for the Maemo community.

However we also feel that the benefits of supporting a smooth transition for the vibrant Maemo development community would be worthwhile both for MeeGo and Maemo:

• developers would be able to use the OBS' natural ability to target Fremantle, Harmattan and MeeGo from a single location. This would bring more developers and their applications to MeeGo sooner.
• many of the same people in the Maemo and MeeGo community teams look after the Maemo Autobuilder and the MeeGo application OBS. Our limited volunteer time would be used more effectively on a single platform instance.
• resources earmarked for Maemo could be added to the MeeGo estate and would naturally be used at peak efficiency as Maemo demand decreases and MeeGo demand rises.
• new Maemo community Quality Assurance processes would evolve around the shared OBS and could assist the development of MeeGo QA processes.

and perhaps most important of all:

• users of existing devices could expect a significantly longer maintainable life from products built on a consolidated build service and could look forward to their applications being available on MeeGo as soon as devices become available.

The maemo.org buildservice already has a 'proof of concept' instance of the OBS which allows the Fremantle target to co-exist with a MeeGo target and we already intend to use this as a basis for the MeeGo community OBS.

The proposed solution *must* allow MeeGo community users to use the MeeGo Community OBS without any reference to Nokia closed binaries; this facet of the service should be entirely optional.

Equally the legal issues around the closed binaries require an EULA related to demonstrated possession of a relevant device. This can be handled in a similar manner to the Maemo Autobuilder service; ie registration of a serial# to a developer account.

The proposal therefore is:

• To provide the closed binaries as a build-target repository (probably DoD for those who know and care) on the community OBS.
• To grant ACL based access to this repository based on acceptance of an EULA
• To *not* require any such EULA for 'MeeGo-only' accounts on the service

I've run this by Tero Kejo in Nokia and he's very supportive of the idea.

From:

David Greaves / lbt
   Community Member and build systems guy.
Niels Breet / X-Fade
   maemo.org webmaster and autobuild owner
Carsten Munk / Stskeeps
   maemo.org distmaster
Andrew Flegg / Jaffa
   on behalf of the Maemo Community Council

I like crashes like that...

So I'm ssh'ed into my kerberos kdc server and at the same time I'm playing with nfsv4 and I clearly push the boundary somewhere and my NFS server has a kernel BUG and goes wibbly... ctrl-alt-sysreq S-U-B wibbly...

Now the kerberos kdc is a VM on the nfs server; which also holds the home directory for my desktop. So my desktop hangs hard.... toast.

I wander over to the nfs server and shut it down as best I can and reboot it.

It comes back... and as it does my desktop wanders into life, my ssh session to the VM is still active; nothing happened.

I like crashes like that :)

Kerberos and ssh

So you're setting up Kerberos and you've got a ticket with kinit

the target has a host/@REALM

but ssh fails with:

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context

Well the first thing to realise is that you're probably not using gssapi-keyex, you're probably using gssapi-with-mic (see http://www.ietf.org/rfc/rfc4462.txt if you want to know more).

Running:

/usr/bin/sshd -D -ddd -e

may give

debug2: input_userauth_request: try method gssapi-with-mic
debug3: mm_request_send entering: type 38
debug3: mm_request_receive_expect entering: type 39
debug3: mm_request_receive entering
debug3: monitor_read: checking request 38
debug1: Unspecified GSS failure.  Minor code may provide more information
Key table entry not found

The problem I has was a typical one in kerberos setups... DNS and name resolution has to work.
In this case a simple entry in the hosts file with a non-fqdn and there are very few clues.

So maybe you'll google some of that and it'll help... :)

OBS and Fremantle ... huge success :) and HELP!!!!

As you may know I recently volunteered to setup a community OBS in my "spare time" (hah, right!). This is a progress report because I've reached a significant milestone... the following 193 applications have been built from 'Extras' most on both ARM and X86:

accdisplay, acl, actman, adblock-plus-1.0, adflashblock-css, alienblaster, anarchism, attitude, attr, audiofile, avahi, battery-eye, belltower, blubbels, bluetooth-dun, bluezwitch, brainparty-data, burgerspace, catorise, clutter, clutter-gtk, copernicium, crimson, currencyconverter, datetoday-home-widget, dbus-python, de-lite, dialcentral, doom-wad-shareware, easy-deb-chroot, easypark, ejpi, fapn, feedingit, feedparser, flac, flatzebra, fm-boost, fmradio, freecell4maemo, fschedule, fuse, gboggle, glibmm2.4, glogarchive, gnome-python, gonvert, goocanvas, gpe-icons, gpodder-icon-theme, greekiradio, gst-plugins-good0.10-flv, gst-plugins-good0.10-mkv, gst-plugins-good0.10-ogg, gst0.10-python, gstreamer0.10-ffmpeg, gtk+extra2, gweled, headphoned, healthcheck, hex-a-hop, host, irssi, jamaendo, json-glib, kernel-power-settings, libart-lgpl, libcap, libdaemon, libevent, libffi, libgnomecanvas, libgnomeprint, libgtkhtml2, libliqbase, libmad, libmicrofeed, libmodplug, libraw-lite, librsvg, libsidplay, libsigc++-2.0, libsoup2.26-1, libxinerama, liqflow, lirc, lybniz, lzma, lzo2, madbomber, maemo-optify, mafw-lastfm, mancala, mastory, masudoku, mauku, mclock, meanwhile, mediabox, microfeed, microfeed-providers-unstable, microfeed-utils, milkyway-wallpaper, mirror, mplayer, mspede, mutagen, mygpoclient, n900-fmrx-enabler, n900fly, nako, ncalc, nmap, notify-python, omweather, openvpn, openvpn-applet, panucci, pedometerhomewidget, pexpect, polipo, poppler, proximityd, pycairo, pycurl, pygame, pygobject, pygtk, pygtkeditor, pyinotify, pymaemo-optify, pypianobar, pyrecipe, python-central, python-dbus, python-evolution, python-facebook, python-gdata, python-gtkhtml2, python-imaging, python-location, python-numeric, python-support, python-twitter, python-xml, python-ystockquote, qtsixa, quickbrownfox, quicknote, recaller, reflect, rfk, rmz-helsinki-wallpaper, rmz-senaatintori-wallpaper, rootsh, rsync, sdl-net1.2, sdl-ttf2.0, sdlgfx, sdlpango, sense, sgt-puzzles, shortcutd, sip4, sleeper, solarwolf, sopwith, sqlite, ssh-status, stockthis, storageusage, supertux-stable, systeminfowidget, tar, tennix, thai-ttf, tickstill, tor, touchsearch, tsocks, uqm-3do-data, uqm-data, uremote, ussd-common, ussd-widget, vim, vor, vultures, wget, witter, worldtv99, xkblayouts-rx51-ru, zoutube

There are a lot that don't build but I think we're at the point where I need help from some experts and it's time to open the beta service up to early adopters. You can see what I've done to setup Fremantle on the OBS.

As I said HELP
I now need to fix build issues like:

installing hildon-update-category-database
/usr/bin/hildon-update-category-database: I don't have write permission on /usr/share/mime.
Try rerunning me as root.
dpkg: error processing hildon-update-category-database (--install):
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 hildon-update-category-database
exit ...

which blocks a large number of applications

and package specific ones like this in libogg:

automake: configure.in: installing `./mkinstalldirs'
Makefile.am:3: require version 1.6, but have 1.4-p6
include/ogg/Makefile.am:6: invalid variable `nodist_ogginclude_HEADERS'
autoreconf: automake failed with exit status: 1
make: *** [configure-stamp] Error 1
dpkg-buildpackage: failure: debian/rules build gave error exit status 2

I can fix almost all of them myself but that's not the point - we need a team to make this work. Please get in touch and help.

Community Building for Maemo and MeeGo - What does the OBS *Do*?

In a closed environment you use what's in the SDK and you get your own little space; talking to others is discouraged and sharing and re-use outside the blessed area is frowned upon. This dictatorial style has the advantage of making life easier for the vendor. In an open world we have more interactions... and as students of networks know: increased connectivity brings increased complexity as well as increased benefits.  So this is an initial proposal for the organisation of OBS build projects and packages to support a QA process into an app-store / Extras or garage-like environment.  I'll introduce some basic OBS concepts and then describe how this might work. I would like this to raise awareness of some potential complexities that we may face and get some thoughts on how to deal with them. ...... Oh, and this is about both Maemo and MeeGo.

First an OBS Intro

1. It's a build system (fully local) and a build service (like the autobuilder).
   
2. You put source into it and say "use this repository" and it spits binary packages out.
   
3. It builds a minimal and "clean" SDK using the repository you told it to use
   
4. It has packages - a package corresponds to a tarball and a spec/dsc
   
5. It has projects - a project is like a directory with packages
   
6. Packages are 'published' into a corresponding repository (which can be shared with others)
   
7. The published repositories can also be used by devices to download binary packages.
   

There is a lot more to it; scheduling, dependency based building, multiple architecture support, home projects, package signing...

The 'trick' then, is to set up a structure of projects that we can use to support the publishing and sharing activities that we need.

So What are our Goals?
We want:

• to allow freedom for developers to develop;
  
• to provide a great build service and SDK
  
• to work with the core distro as much as possible
  
• to provide excellent quality assured applications for our "app store";
  
• and for MeeGo, to extend the core distro with a community supported area for libraries and packages that are depended upon by more than one application.
  

First thing we get: Individual Homes and PPAs
You'll start by uploading and building a package in your "home" directory (which can have a structure underneath it for multiple projects). This will allow you to build against any of the main distros; any group/community projects or even any other community member home projects. Oh, and they can use your code as a baseline too. Once built your code is automatically published to a repo on the
community downloads server.

This is a lot like the Ubuntu PPA solution.

At that point you can stop if you want. You have a complete set of repositories (1/subproject). No painful QA processes. No 'fragmentation'. But equally your repo(s) will needed to be manually added to a device in order for it to appear in any apt-get/zypper/yum etc.

A huge benefit here is that to get at a development version of an application you use a specific repository, not a mishmash of randomly unstable packages like Extras-devel.

For the pros: Community QA'ed Repository - Extras
If you'd like your application to appear in the Extras repo which is pre-installed on devices then you can submit to the QA process.

First you register for the package name, then submit your code to the distro promoter; there it is QA checked, copied to Extras:Testing, built, QA'ed again and becomes available in the Testing repository.

<here be dragons> The community testers then approve your code. </dragons>

(I know that's a contentious point - and I don't have an answer. I do however have confidence that we can solve the problem through collaboration and co-operation.)

Once approved the code is copied to the Extras:Stable repository for publication.

Conclusion

Much of this text is online at:
http://wiki.maemo.org/OpenSuse_Build_Service
(I really hate it when things get locked up in blog posts and forums)

How can you help?

Some tasks needed to get the OBS running:

• OBS Installation - a detailed guide on how the system was setup.
  
• Application QA Process on the OBS - a proposal
  
• Setting up Fremantle - WIP - post to follow
  
• Fixing the Fremantle SDK - mmmm
  
• Setting up cross-building - started, not imported yet
  
• An OBS Autobuilder queue - not started
  
• Reporting - not started
  

Also see:

• http://wiki.meego.com/Proposal_for_Community_Application_Support
  
• http://wiki.meego.com/Proposal_for_a_Repository_working_group
  

It was the Dawn of the 3rd Age...

...and not only is the Shadow War over but it looks like we will have a community OBS thanks to the team formerly known as maemo.org infrastructure :)

I'm really excited as I see this as a way for the Maemo and Moblin development communities to transition smoothly into a MeeGo future; we should eventually be able to offer an easy way to build applications for Fremantle, Harmattan, MeeGo and future device OSes too.


So what's happened?

A while ago the RWG proposal was begun and we've been pushing it ever since. Tero, Mike, Neils and others have been working on the CAT proposal as well as the RWG. We got together on irc and had a chat about the scope and how to proceed.

Last week the TSG agreed we need something (kinda!) - so we decided that we'd go ahead and do it.

We now (thanks to Tero & Nokia?) have a test machine behind a firewall and I've setup an OBS (more than you ever wanted to know... but you said to work in the open, right?) to prototype how we'd deploy it for community use. This has a link to the MeeGo OBS and successfully builds MeeGo packages.

Where are we going?

I don't know yet; but here's what I'm proposing (my ideas tempered with some input from others):

Initially we should offer a MeeGo OBS and a Maemo OBS. There are corporate issues around this and whilst my personal aim is a single build system able to build debs and rpms against any M[aemG]*o distro, that isn't the first step.

The MeeGo OBS will probably offer home directories and a way to build against the MeeGo weekly snapshots. We may be able to offer a 'live' feed if some OBS bugs are resolved and we have the build resources.

The Maemo OBS will provide an alternative to the autobuilder for Fremantle and, eventually, for Harmattan. This makes sense as our longer term future with MeeGo is clearly the OBS.

One thing I dread is having to go to each device vendor's build system to make build my app for their community app store; so maybe off into the future we'll even provide a neutral place where we can support multiple vendors: a single build system that can build community MeeGo apps for devices from any vendor that participate.

Another area of interest to me is something I call "Surrounds" (yes, Quim, I know... it needs work). This isn't my idea... lots of software products and distros have it. It's that chaotic area around a product where 'outsiders' or community people contribute their "unsupported" bits and pieces. Actually this deserves a post on its own. Anyhow, I think the OBS will make managing this a lot easier for the community.

There is still a lot to do...

We've asked a few community devs to join (and please come and talk to us if you feel you can help - bear in mind that we need documentation and build system process experience more than development skills).

A big task is the setup of an OBS build system for Fremantle. I managed to get this 80% done a while ago so that work needs digging up and finishing off. Once that's done we will need a lot of help validating all the Extras apps.

Yet more work is required on my cross-compilation system used by the OBS. That was deployed for Mer and we need to redeploy it for Fremantle too.

There is work on reporting systems and QA automation systems going on that we can re-use which will help in transforming the autobuilder promotion type processes and figuring out what is and isn't needed in the OBS world.

Speaking of which ... do we take this opportunity to tweak the Extras promotion policy?
I think we need something like the Devel->Testing->Stable process for Maemo apps - but maybe we should be working with the CAT guys to figure this one out?

Finally please don't ask me when you'll get to use it, we don't have a schedule yet :(

However, do feel free to ask me what we're working on or what we've done or what you can help with.