N900 and Mozilla on the BBC

Here's the BBC Technology story it's good that the N900 is being seen as an open system at this level.

snippets:

The browser, codenamed Fennec, will initially be available for Nokia's N900 phone, followed by other handsets.

 and

 Apple is very restrictive. It doesn't allow other browsers.

which is all promoting the core values the community loves about Maemo...

Android... is it Linux ?

This i just something that I saw on LWN (from Harald Welte) and felt needed a little more visibility here at maemo.

The presentation shows how Google has simply thrown 5-10 years of Linux userspace evolution into the trashcan and re-implemented it partially for no reason. Things like hard-coded device lists/permissions in object code rather than config files, the lack of support for hot-plugging devices (udev), the lack of kernel headers. A libc that throws away System V IPC that every unix/Linux software developer takes for granted. The lack of complete POSIX threads. I could continue this list, but hey, you should read those slides. now!

 The slides

As usual the commentary at LWN raises some interesting counters.

Ovi Maps... Really? Is this the best we can do?

I got caught in a nasty traffic jam tonight just a few miles from home.

There was a road that I've never been down - windy country lanes, right general direction...

I grin and whip out my N900 and give it to my wife to navigate us - just the ticket?

No.

An hour later we've been lost several times, argued and are generally very annoyed and unhappy. My wife thinks that *I* think she's stupid 'cos she can't use a map application. She hates that. I hate that.

The maps UI sucks. It is unusable to a novice. The GPS seems to update randomly every 10 mins or so. It blocks the screen with messages. It tries to connect to GPRS when there's no signal (and clearly no need). Mostly it just completely fails to meet its purpose.

As we hit a road we recognised I realised that I was so pissed off with this experience that if I were a consumer who'd paid retail for this device I would want my money back - and as someone who has breathed Maemo for almost a year now, that makes me very, very sad.

Maemo Security - Lockdown or Liberation?

At the Maemo2009 Summit Nokia shared a great deal of information about the security mechanisms that would be available and/or mandated in upcoming platforms.

The concepts outlined include well established favourites in the OSS world (like privilege management) as well as some that are rather less well regarded - such as relatives of the Trusted Computing Platform and DRM.

Inevitably there will be a significant amount of interest and concern about how this affects the open nature of the Maemo platform.

I'd like to highlight that there is a specific boot-path designed in to allow a community kernel to be booted with, as far as I could tell, no loss of functionality other than access to DRM content. That's amazingly positive.

For what it's worth I will say that I was massively sceptical as soon as I saw "Trusted Computing" and "DRM"; however after listening and thinking about it I came to the conclusion that this solution could be a huge benefit to the OSS community. Certainly Nokia can abuse it - but that's not the point. The point is do we - and should we - trust them.

We certainly need more security. Right now when I download an application from Extras-Devel it can do anything to my device; on an N800 that's not so bad - on an N900 that can incur significant cost and could conceivably (and almost trivially) be used to perpetrate fraud. I'd like to be able to say "no, scrabble game, you can't access my contacts data or make phonecalls - what on earth do you need to do that for?" An open security infrastructure would make me feel a whole lot more comfortable.

As a starting point to a rational response I think a sensible thing to do is to brainstorm the key questions we need to ask Nokia in order to gather the data needed to come to a conclusion.

The presentation was a great start - as was Elena's offer to continue the discussion on the mailing lists and respond to our queries and concerns (Thanks Elena).

So,I have started a Maemo Security page on the wiki which I hope can capture community questions (and, eventually I hope, Nokia's answers) about these issues.

The first questions are draft (I just landed and I'm tired!) to get people thinking. Certainly I want to re-watch Elena's presentation video several times before going much further.

I suggest we initially add questions to the  Maemo Security discussion page and  and once they've been refined and consolidated, we'll add them onto the main page.Obviously discussion needs to happen on talk, email and irc too.

I'll wrap up with a plea - please don't over-react  - jumping to conclusions and misleading the rest of the OSS community before we know the facts can only hurt what has so far been the most amazingly open phone device I could have dreamt of!

PS The Summit was brilliant!

Important Summit Announcement

Well, it's important if you arrive for the Summit on Thursday and want to meet up for a beer (or, if Copenhagen was anything to go by... an ice-cream) in the evening.

More details here:
  http://wiki.maemo.org/Maemo_Summit_2009/Schedule

Meet up at 19:00 (so after early registration closes).

There will be free beer available













if someone offers to buy it ;)

Want to PUSH... need a kickstart?

So one problem with the PUSH is that it's a bit tough for us software nerds to get into the real world. What would be cool is a nice easy "apt-get install HARDWARE" ... interested? read on...

A few years ago I bought a couple of these boards for about £25.

They provide:

• 5 digital input channels
• 8 digital output channels
• 2 analogue inputs
• 2 analogue outputs (8 bit resolution)
  

You can also have up to 4 of these plugged in via a usb hub.

So, I thought... how hard can it be... and actually it is quite easy. The source is on sourceforge and I'm packaging it so you can pull libvelleman from garage, plug in a board and PUSH :)


As you can see the test program implements K.I.T.T. for you...




So now all we need is someone with an N900 to help me test it...

(Warning: Don't get your hopes up - it appears that the N900 can't (yet?) act as a normal USB host; so no plugging in printers....)

The Summit: Git/Gitorious - Untracked talks: consider adding them...

I proposed a talk at the summit on DVCS and git and it has not been accepted (or rejected) by the talk selection group. This isn't a complaint  :)     . . . . .   however I do worry that it might worth pushing ... what do you think? (Breaking news... sounds like there has been an acceptance - also it may make sense to do a BOF session after any presentations). I still would appreciate feedback - what' important to you?

This message is just to solicit opinions and hopefully will validate their decision. If it causes a rethink then that's fine too.

Heh - I could do without the extra work involved!!

I personally thing think that as a development community with git on garage and gitorious.org we should be making efforts to understand how best to use DVCS processes to collaborate.

Maybe it's just me but I see a lot of devs who are new to DVCS and very few community guidelines on how to use DVCS. Qt uses it but, as we've recently been discussing, it could be going better.

Frankly I don't care which 'good practice' I use - I can go out and find lots of them. But it strikes me that as a community we should at least say "hey, quite a few of us are using this approach - if you don't have any strong preferences then you can use it too"

So why now?

Well, real soon now (I hope) we're going to have 3 different versions to support.

• Fremantle
• Diablo
• Mer

They'll each have different capabilities and some apps just won't care - they'll support Fremantle or Diablo and ignore the others. That's fine.
(Although being open-source you might find patches being sent in to add support for another flavour - how will you cope? Wouldn't you like some help in doing the boring stuff?)

Other apps will decide to embrace that diversity from the start.

So how do we (you) manage the build-variations (ie debian/* may well vary for each of them. Maybe ./configure too)? Do we use branches? How?

Now how do we manage adding features and back-porting simple bug fixes to the stable release whilst you work on that big new feature set.

How are contributions and teams handled?

It sounds horrendous - and it can be!


But actually this is all fairly simple stuff with DVCS once you have it explained and once you grok it - but it's bloody hard to figure out from scratch and it's also very unlikely that you'll arrive at the same solution as another team.

Which means if you're a member of multiple teams you might find they each have different approaches - "whoo hoo!" - not!

So anyway... I thought a talk would be a good idea.

Now at the time no-one had volunteered so I did - some of you may have noticed my name at the bottom of:
  http://www.kernel.org/pub/software/scm/git/docs/git.html
(yes, I'm proud of that contribution)

Now you may think that qualifies me to offer this talk - you'd be wrong ;)

I was hanging around the kernel lists at the time, got interested and acted as an 'editor' pulling together words of wisdom. Since then I've used git a little but it wasn't until we started to need it in Mer that I reviewed the state-of-play and tried to pull in other people's good ideas - so that's what I'd use as a base.

However we also have Johan Sørensen who wrote Gitorious.org - I think having him speak on using git and gitorious would be an opportunity that we shouldn't miss.

Equally there must be developers in Nokia/Trolltech who could say "we know this stuff and this is how we think you might want to do it."

So... speak now...

PS If anyone fancies collaborating and doing a multi-person presentation then I'm up for it.

GrandCentral for Mer... connecting the components

First, I'm sorry but this isn't an N900 post :) instead I've been looking at one of the areas that's holding back Mer. It's fairly simple conceptually: just handling interaction between various parts of the system; and by this I mean things like screen dimming; what happens when you press the power key; shutting down when the battery is low; waking up for an alarm... Things that should just work.  Well, I think I have a potential solution. (Oh, and some of you may be interested... I also found out how far eclipse has come in the UML modelling space too!)

Currently this is handled by powerlaunch; it is really not a bad bit of code and does most of its job well.

Sadly it really sucks at being configurable!

Here's a snippet:
global_key_press_power = timer_set _key_power_long $powerkey_longdelay; timer_set _key_power_medium $powerkey_mediumdelay; set powerkey_short 1; call _key_press_power_short
global_key_release_power = timer_cancel _key_power_long; timer_cancel _key_power_medium; if $powerkey_short _key_release_power_short key_release_power_long


I don't know what it does and the only way I'd be happy changing it would be after extensive research. Frankly it scares me!

Here's what I was thinking we'd do for a 'configuration' syntax in Mer:

# describe some action
def on_wifi_up():
  notify "The wifi is up"
  os.system("mail me@gmail.com -S 'wifi is up'")

connect(wifi, "Connection.up", on_wifi_up)

connect(camera, "Camera.active", takePhotoAfter10Secs)
connect(camera, "Camera.turn",   media.play, "/home/lbt/sounds/camera_click.wav")
connect(powerButton, "Button.longPress", powermenu.show_menu)


This uses a python signal library (like Qt or libsigc++) and provides a set of component objects that emit nice signals like "Connection.up".

The benefits of this approach:

• simple syntax is nice and readable
• interpreted so can easily be edited
• python understood by many so can be easily extended
• re-uses some powerlaunch code (eg powered)

Of course there are drawbacks:

• it's not a config file - it's code (but considering we want to configure functional behaviour then any config syntax needs to be a language... so use a decent one)
• python is quite large (but it's now in memory and .so are available)
• it's another re-write :(  (it only took a few days and I'm using powerlaunch as a prototype for config rules - if I can understand them!!)

So anyway, as I said, I found eclipse Modelling edition...

This says (well, I hope it says):

• There is a system and a user dispatcher
• They're based on a common Dispatcher
• Which contains components (such as Headphone, Menu etc) that can
• send signals (eg when the camera on the N800 is turned)
• take requests (dim the screen)
• provide state information (a headphone is connected)
• be accesed on D-Bus
• The Dispatcher creates Connections which
• listen() for Components to emit signals (also listening on D-Bus)
• run Scriptlets which might
• run arbitary code
• make requests to components
• emit signals that (other) Connections can listen() for
• Scriptlets are part of the Dispatcher

Oh, and the good news is that it's up and running... I'll put the code on gitorious RSN.

Do you take contributions? or is it "Beware of the Leopard"

Open source is about opening things up... right?
Well, yes.
But I think there's more to it than that.

Think about the kernel and how that works. A wide open mailing list (and source repository) with people getting on with it. Fedora? Ubuntu? The same. SGI's XFS filesystem; in fact any number of projects out there.

But what does that even mean? "Getting on with it?"

To me it means discussions, proposals, code, designs, bugs. If you care you can watch; you can even attempt to join in. You can certainly learn about what's happening and why.

So here's the question...

Where's the 'Contribute to Maemo' on maemo.org?

Fremantle is coming out - but it just appears on the download servers. There's no opportunity for the community to contribute. Where are the mailing list archives or the commit logs so I can find who committed that change, when and why?

Opensource means you can't be shy; open doesn't mean "behind closed doors"; it doesn't mean "you get it when we're done".

Open means "this is what we're doing". And "we" includes the community.

I hope we're doing it the right way in Mer.

irc is our main collaboration area and that's open and logged. Not terribly useful as an archive though. We have the chatter mailing list which gives a pretty good summary of key events.

What we do to activate incoming developers to make it easy for them to contribute? Well, we've adopted gitorious as our DVCS and I think the process and UI there makes that a good call. I've also put some effort into describing our 'internal' build processes and what we (ultimately) expect DVCS contributions to look like.
 Carsten has setup a neat little tracker of outstanding tasks so potential contributors can pick up things to work on.

We have some design statements on the wiki and we know that 'track Fremantle and predict Harmattan' are our tenets.

I wondered what Android does?
Hmm : http://source.android.com/roadmap

The Android Open Source Project will publish roadmaps for upcoming releases. The roadmaps will be developed with input from this community, Project Leads, and the Open Handset Alliance

Sounds good.. and the code is up there.

What about contributions?
Well... say no more! A random click got me that gem.

Nokia and Maemo5 .... well, to be frank it looks a lot more like Microsoft.

And that's not that harsh :)

Microsoft provide (I'm told) excellent documentation for developing OSS on their platform.

For Maemo there is some really good developer documentation (seriously - I think this is not shouted about enough); there's superb support for creating OSS on Maemo platforms; and that's absolutely crucial.

Contributing to Maemo... not so much.

Yes, the source is there for the released maemo packages; and there are random directories like this or this   (first one is circa 2005, 2nd April 2008)... browse around, we found some useful stuff in there!
As Carsten said "god knows how i found that first time". There are garage projects too (impossible to find the right ones though), recently we noticed some appearing on gitorious or  git.maemo.org or svn....

However I'm not really aware of anything other than bugzilla for communicating with the development teams; and I don't think that's something the community generally finds to be a 'shining example' :)

We recently added a note by way of an apology for Mer's lack of "apt-get source" but I think Maemo deserves it too:

`...You hadn't exactly gone out of your way to call attention to them had you? I mean like actually telling anyone or anything.'
`But the plans were on display...'
`On display? I eventually had to go down to the cellar to find them.'
`That's the display department.'
`With a torch.'
`Ah, well the lights had probably gone.'
`So had the stairs.'
`But look you found the notice didn't you?'
`Yes,' said Arthur, `yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of The Leopard".'

I really don't think it's doom and gloom though - Qt has opened up a lot since Nokia took over. The development has moved out into the open and there's a process to contribute. Gitorious is  being sponsored by Nokia/Qt and I hope the maemo-af team are leading the way.

I really hope that when Harmattan arrives the basement is light and airy; very, very noisy and the leopard has relocated to the lobby!

Accelerating OBS .... !scratchbox

So I've been working on accelerating the Open Build Service recently.

It's a great build environment for Mer and we love it. It uses qemu to emulate arm architectures and provides a pristine chroot build environment using native packages. This approach of using distribution tools rather than a toolchain provides a more consistent build system.

The problem is that this approach is slow.

Following the 80/20 rule, the OBS engineers (thanks Martin, Jan Simon) identified the applications that represented the greatest load on the build system:

• bash
• bzip2
• dpkg
• gzip
• m4
• make
• perl

And of course:

• gcc

So the intention was to accelerate these binaries in the emulator by using host machine code, not emulated code.

The emulator uses binfmt_misc to call qemu for foriegn binaries when it hits them so simply replacing the binaries with host binaries should work.

Well; almost.

The host binaries need their shared libraries; and they live in the same place as the emulated shared libraries... which may be used by non-replaced code. Hmmm.

(Before going any further, lets establish a naming convention: target is the architecture of the chroot and eventual target; host is the architecture of the underlying CPU. Typically for us target is armel, host is x86.)

The approach to solving this problem is to minimise changes from a pure arm build environment. This is achieved by installing both arm and x86 packages; to avoid collisions the x86 packages are installed into /lib-x86 (in the chroot) and of course, shared libraries will fall into /lib-x86/[usr/]lib and not overwrite the target .so files.
The final step is to move selected binaries in /bin and /usr/bin out of the way and replace them with symlinks to the i586 binaries in /lib-x86/[usr/]bin

Most x86 packages are unmodified; any paths they use will relate to the main / directory and the config/data files installed by the target package will be used. (This works in general although the eagle-eyed will spot issues with things like XS modules for perl - not a major problem for a simple build environment though).

Some packages are modified to minimise build times - eg to avoid 64 bit builds; others are modified (eg dpkg) to set the target architecture as a default.

Obviously the host binaries need to link to shared libraries which are not in the normal LDPATH. We don't want to have to rely on environment variables etc so 2 steps are used:

• modify ldlinux.so to look in /lib-x86/lib etc
• modify the rpath value in the ELF header of each replaced binary

This is done by creating a static build of patchelf and modifying the binaries as they are installed.

Finally a modified package has all postinst scripts removed and replaced with something like:

prefix=/lib-x86
$prefix/usr/bin/patchelf --set-rpath $prefix/lib $prefix/usr/bin/make
mv /usr/bin/make /usr/bin/make-orig-x86lib
ln -s $prefix/usr/bin/make /usr/bin/make

Note that some libraries have an rpath and need to be treated with patchelf too.

One final problem arises with fakeroot: it modifies the LD_LIBRARY_PATH to look for libfakeroot-XXX.so This is handled by editing the fakeroot script to append the /lib-x86/ paths in the same way the multi-arch paths are added.

All in all this is actually a very clean solution:

• the modified binary uses the exact same source as the one it replace
• the installation principles mimic biarch (and will, we hope use biarch when it's ready)
  
• only the specific binary executable is replaced - data and scripts come from the target package
• it can easily be switched on or off for a specific binary
• enabling it on the OBS is done in the enclosing project, not the package

Next steps, making it work on the OBS and then ... building a cross-gcc.

$ make distro

That's what the OBS does for us.

The openSuse builder  provides Mer with an incredibly powerful tool for building packages.

It gives us:

• over 150 build cores !!
  
• powerful monitoring and control tools - web and cli
• total access - it's all GPL
• proven capability - openSUSE and SLE are built using it
• distribution neutrality
  

OBS is a GPL, distro-neutral build system; it works by creating a totally empty virtual machine (or chroot), bootstrapping a minimal version of a distribution and then driving the distro's own build tools to pull in the build dependencies from the official distribution repository. Finally it calls the native tools (eg dpkg-buildpackage) to build the package

Think of it as a wrapper around the distribution's standard build tools which provides queue management and a good web interface.

What's good about that then?

• No wheels being re-invented - it uses the 'standard' tools
• Quality control - packages must specify exactly what they depend on to build; and a build installs clean packages and builds from clean source
• Management wrappers - more about this later
  

And it does this around multiple base distributions inclding Debian/Ubuntu/Redhat/Centos/Moblin and of course SUSE and openSUSE.
Just try building a Redhat package on the Debian build system - or vice versa.

What makes it particularly interesting for Mer is that it also does this with multiple architectures including ARM.

So what Mer essentially does is to feed it a large number (~200) standard debian format source packages and wait for it to spit out .deb files in an sources.list compatible repository.

Doing this correctly means it needs to analyse build dependencies and, make-like, consider the freshness of those dependencies to determine if a rebuild is required. So if a change is needed in  libhildonfm2 then all the packages that depend on it will be rebuilt - nice.

If you'd like to read a bit more about how Mer uses it then start on the Mer Build pages.