We arrived on Tuesday evening at hotel "The Grand", which is also the location where the 2010 Plone Conference is held. A really nice location, located in the city centre of Bristol.

After checking in we visited the Watershed, located near the harbour area of Bristol. Most of the conference visitors were also there so we enjoyed a cold beer together. Rob was having a meeting with who we call the "Plone rockstars", and lateron they joined us at the Watershed as well.

The next day the conference itself started with the keynote of Alexander Limi. The usual game "where is everybody coming from" was played, but somehow this always works to create a real international atmosphere, which the conference is famous for of course.

Right after the keynote Rob informed the community of the current status of Deco, which have been worked on during the Living Statues sprint. Mikko enlighted us about the current status of Web & mobile which has an official release right now. Really interesting if you want to have your Plone (or other python based) website to go mobile.

Matthew Wilkes gave a presentation about making high performance websites easy. Sane thinking and good caching including ESI can make a Plone site sreally fast, so you should check out his presentation.

Eric Steele talked about how we can improve Plone in the future, focussed on strategy, vision, responsibilities and ways to communicate in and outside the Plone community.

After that Martin Aspelli talked about how you should implement a Plone project the correct way by using a recent project he's been working at. Conventions and design choices were explained by good examples and arguments, and it was really good to see him back on the stage of a conference again.

After his first talk he had another presentation about project management, tools to use and how to make a success out of your project. Information everyone should know, explained in a easy and understandable way.

To finish of the day Richard Noble gave an excellent talk about the current attempt at the world landspeedrecord. Although it is a completely different project there are quite a lot of similarities. Especially the "no patent"-policy is really as open as you can get. Really nice to see somebody who is this enthousiastic about a project and motivating others to realize his dream.

In short : a very interesting first day at the conference with some really quality talks, so if you're not at the conference you should watch the video's!

The problem

When you remove a product, Zope does not know any more what the pickled objects in the ZODB are. Sometimes you do not notice any problems until you want to install a new product or want to change some setting in the plone control panel. Strange error appear like :

Module Products.GenericSetup.tool, line 1181, in _runImportStepsFromContext
 Module Products.GenericSetup.tool, line 1092, in _doRunImportStep
 - __traceback_info__: toolset
 Module Products.GenericSetup.tool, line 130, in importToolset
TypeError: 'NoneType' object is not callable

or you see messages in your Zope log like :

WARNING OFS.Uninstalled Could not import class <myclass> from module <mymodule>

Sometimes you get a ComponentLookupError by visiting the Plone site, even when you use the ZMI. Or some other strange error occurs (I've seen many different errors unfortunately).

Normally you can remove a local utility by using the unregisterUtility method, but at this moment you don't have the product installed, so importing interfaces etc is not an option. Reinstalling is not what you want, so what to do now?

What to do

The first thing to do is to find out which product was responsible for the broken persistent components so know where and what to look for. You probably remember which product you uninstalled, otherwise you can find some hints in the zope access log file. Look in the profile and zcml of the product what is registered using a generic setup profile. We use collective.myproduct for the rest of this blogpost.

Create a new zope instance with your broken Data.fs and start a debugging prompt with bin/instance debug so you can start investigating the ZODB.

Tip :You can use the .__dict__ property or the dir() method to inspect objects in the ZODB.

Disclaimer : Do not ever, ever do this in production environments unless you know what you are doing. One mistake can break your Plone website totally.

Sitemanager

In the zope component architecture, local utilities are stored in a local site manager. With Plone 3 and 4, you have a Sitemanager on your portal object. First thing to do is to get this Sitemanager.

sm = app.myportal.getSiteManager()

The sitemanager holds three types of local components : adapters, subscribers and providers.  (At least, that is what I found out). We should get them so we can see if our broken component is there somehow :

adapters = sm.utilities._adapters
subscribers = sm.utilities._subscribers
provided = sm.utilities._provided

Let's list (a part) of the current registered adapters :

>>> for x in adapters[0].keys():
...     print x
... 
<InterfaceClass plone.browserlayer.interfaces.ILocalBrowserLayerType>
<InterfaceClass zope.app.cache.interfaces.ram.IRAMCache>
<class 'collective.myproduct.interfaces.IMyProductConfig'>
<InterfaceClass Products.PortalTransforms.interfaces.IPortalTransformsTool>
<InterfaceClass plone.app.i18n.locales.interfaces.IContentLanguages>
...

In the list we can see that there is an adapter registered, and you can also see that the class differs from the others, because it's a broken object. You can verify that it's broken by printing the title of this object.

for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
        print "%s" % x.title
This object from the unknown product is broken!

So we found our broken object so we should delete it :

for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
        print "deleting %s" % x
        del adapters[0][x]
sm.utilities._adapters = adapters

After the reassignment the ZODB machinery does not detect that there is a change, so we should commit this change :

from transaction import commit
commit()
app._p_jar.sync()

The same story goes for the subscribers and providers, so here is all the code together :

sm = app.myportal.getSiteManager()
adapters = sm.utilities._adapters
for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del adapters[0][x]
sm.utilities._adapters = adapters

subscribers = sm.utilities._subscribers
for x in subscribers[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del subscribers[0][x]
sm.utilities._subscribers = subscribers

provided = sm.utilities._provided
for x in provided.keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del provided[x]
sm.utilities._provided = provided

from transaction import commit
commit()
app._p_jar.sync()

Now we got rid of our persistent components we don't want anymore, but some problems can still occur when (re)installing products using the Portal Quickinstaller or the Portal Setup tool.

Portal Setup

If you still have problems (re)installing products after you removed the broken local persistent components, you probably have to clean the Portal setup tool.You probably see something like this in the error log :

TypeError: 'NoneType' object is not callable

Check your product if it registers a tool using a toolset.xml file, like this :

<?xml version="1.0"?>
<tool-setup>
 <required tool_id="portal_myproduct" />
</tool-setup>

Fire up your debugger and get the portal_setup tool. It keeps its own registry for all registered tools which you can get using the getToolsetRegistry method :

setup_tool = app.myportal.portal_setup
toolset = setup_tool.getToolsetRegistry()

Check if a tool exists with the tool_id, which has been registered using 
<required tool_id="portal_myproduct" /> and remove it :

if 'portal_myproduct' in toolset._required.keys():
    del toolset._required['portal_myproduct']
    setup_tool._toolset_registry = toolset

from transaction import commit
commit()
app._p_jar.sync()

I'm pretty sure this is not the way to go, as I occasionally still see the OFS.Uninstalled warning, so I hope someone who reads this says "This is not the way to go, you should do this".

Quit your debugging session and start your instance with bin/instance fg. Browse to the portal_setup tool using the ZMI and go to the "manage" tab. (http://localhost/myportal/portal_setup/manage_stepRegistry). There you can delete any invalid import steps if they are still there.

Now you should be able to install products again, and have your Plone site up and running without problems.

Nice post, but this kind of ugly hacking shouldn't be necessary!

And I agree totally on this. So a note to all product developers is : Please write uninstall profiles! You can also remove persistent utilities in your uninstall profile like Roel described in a previous blogpost. And I' sure there is some development on this part with Generic Setup.

Feedback

Everything I described in this post I found out by trial and error, as there is little documentation on these issues. And if it's there: I would like to know! And if you have better, nicer or other ways of doing all these things : Also let me know! Thank you in advance.

When you are developing, you know that error messages will be there sooner or later. Most of the time when you don't know the answer immediately you copy and paste the message in your google searchbar and you will find the answer within a few minutes. If you first had your coffee you would have known immediatly what was going on, but hey, don't blame yourself, today it is a typical monday morning.

But on some very rare occasions you don't find an answer in a few minutes. You even won't find an answer after a few hours, and after spending some time with your very intelligent and smart colleagues you don't find anything which could point you in the right direction. And this always happens when you don't want it to happen offcourse.

An example error message of this is the following, on a Plone 3.3.2 website you are developing on for some weeks :

...
raise TypeError("ImplementedBy called for non-factory", cls)
File "../parts/zope2/lib/python/zope/interface/declarations.py", 
line 353, in implementedByFallback
TypeError: (<exceptions.TypeError instance at 0x936cb20>, 
<built-in function implementedBy>, 
(<RegistrationTool at portal_registration>,))

This happened by just visiting the Plone site, it even happened when visiting the ZMI.

Get back

We had to edit the code of zope/interfaces/declarations.py to something like this to get back in our Plone site  :

...
 else:
     try:
         bases = cls.__bases__
     except AttributeError:
         if not callable(cls):
             pass
# raise TypeError("ImplementedBy called for non-factory", cls)
         bases = ()
 bases = ()

Ok, after this very unwanted change it as time to find out what was going on. What we know now is  that is has something to do with implementedBy and that we used in in some code to check if an objects implements an interface.

Check the documentation

What does the documentation say about implementedBy ?

3.7. Checking Implementation

You can ask an interface if a certain class or instance that you hand 
it implements that interface. For example, say you want to know if 
instances of the HelloComponent class implement ‘Hello’:

>>> IHello.implementedBy(HelloComponent)

This is a true expression. If you had an instance of HelloComponent, 
you can also ask the interface if that instance implements the 
interface:

>>> IHello.implementedBy(my_hello_instance)

Ok, so it's only querying a class or an instance if it's implementing an interface. Nothing evil on that, right?

No clue where to look now so let's try to export some valuable content from the website by making an .zexp file. But while importing the zexp you get another error message :

Module ZODB.ExportImport, line 92, in importFile
 Module transaction._transaction, line 256, in savepoint
 Module transaction._transaction, line 253, in savepoint
 Module transaction._transaction, line 642, in __init__
 Module ZODB.Connection, line 1091, in savepoint
 Module ZODB.Connection, line 550, in _commit
 Module ZODB.ExportImport, line 178, in _importDuringCommit
 Module copy_reg, line 70, in _reduce_ex
TypeError: default __new__ takes no parameters

Right. The plone site is broken, we cannot export the content and even when you copy and paste content in the same website you get this error. And nobody on the world seems to have a similar problem.

IRC log FTW

Google is giving us a lot of hits on ZClasses for Zope versions of 2.9 and below. But we use Zope 2.10.9 and offcourse no ZClasses. Fortunately, there is one useful hint from an irc log by dixond :

<dixond> I had some code, which made the mistake of calling
ISiteRoot.implementedBy(context) ,which is (apparently) supposed to 
exception if context is not callable. I guess Archetypes objects are 
actually callable. Anyhow, it didn't Exception. Instead, it hosed the 
object such that import/export of that object no longer worked.

<dixond> thing is that it was a post-hoc thing. IE, once 
ISiteRoot.implementedBy(context) had been called once, it was stuffed 
from then on in. The object was just bung.

That's nice. When using implementedBy on an Archetypes object it permanently breaks it, according to dixond. Let's investigate the objects using a debug session to find out what is going on. It probably has something to do with __implements__ or __implemented__, as that is queried a lot in the implementedBy method.

On the broken plone installation (which could be run by patching the interfaces/declarations.py file) :

object = app.myportal.myfolder.myobject
object.__implemented__
<implementedBy mypackage.content.content.project.?>

That looks odd. A question mark. Let's see if we can find something on that. In the docstring of implementedBy :

Note that the name of the spec ends with a '?', because the `Callable` 
instance does not have a `__name__` attribute.

Hmm, Ok. we should invest some time by starting a pdb session to see what happens there in implementedBy, but we are running out of time as we want a working website now. Let's try, a wild guess, to remove the __implemented__ attribute :

del object.__implemented__
object.__implemented__
<implementedBy mypackage.content.content.project.Project>

That is looking different. By adding a fresh plone site and some new content we figured out that those new objects do not have question marks in their __interfaces__ either.

Fixing the Plone site

Ok, back to the start. We have a broken Plone site and some weird stuff in our __implemented__ attributes. We want a full working Plone site again, so we tried to remove all the __implemented__ attributes of the objects and committed it to the ZODB. Hopefully we could export our content again, and fortunately we could!

After replacing the implementedBy calls in our code with providedBy (we were testing if instances provide the interface anyway) the problems are now gone.

And yes, zope.interface is probably giving the right answer when to use which method :

def providedBy(object):
"""Test whether the interface is implemented by the object"""

def implementedBy(class_):
"""Test whether the interface is implemented by instances of the class"""

However, a query method should not alter your objects in my opinion. So my conclusion for now, until I fully understand what is going on is :

Think twice before using implementedBy in your Plone project and do not call it on Archetypes objects.

I hope some people with knowledge of this kind of issues have some useful comments on this. Until then you will probably find two useful hits when you search for it on google :)

We arrived on tuesday afternoon, together with our friends of Goldmund, Wyldebeast & Wunderliebe. Budapest is a great city to go out, so we had some drinks and food in the city centre.

On wednesday, the Plone Conference officially started with an excellent keynote of Alexander Limi about the current status and the future of Plone. Rob talked about the TinyMCE Editor integration in Plone. TinyMCE is one of the best WYSIWYG editors around and nowadays an excellent alternative for the default Kupu editor and it's compatible with all the 3.x versions of Plone. It will be the new default editor in Plone 4.

The video of the TinyMCE presentation

After the lunch and the talk of Alexander Limi about the Deco CSS grid system, Rob's talk started to demo the current version of Deco. Deco will be the next best thing in Plone : Instead of just editing content you defined in your content types you can edit the layout of your website as well. With Deco, you build up your page using tiles which you can drag & drop according to the grid which is defined. Tiles can be anything you want : a piece of text, an image or even an application.

The video of the Deco presentation

Last Christmas I left freezing Holland to spend the holidays in tropical Sierra Leone. Not many people will think of this country in West-Africa when you're talking about sunbathing on beautiful tropical beaches and hiking through lush forests. Instead, images of war and horrific bloodshed will flash through their heads. These images are indeed part of the country's recent past, but fortunately not anymore of its present. What we can find nowadays in Sierra Leone is a vital but poor and disrupted society, set in a country that beholds many opportunities and is blessed with abundance in natural beauty.

Since three years I'm working side by side with a group of volunteers in the Netherlands and Sierra Leoneans to improve education in rural villages. The group in Sierra Leone consists of three friends in their early thirties who have setup a community based organization called Joint Operation Youth (JOY) in the township of Lakka. When they were working as security guards for a big environmental NGO, they got inspired to do something to rebuild their country. Samuel, Abdul and Ibrahim aim to support and improve the existing educational institutions and where needed initiated new ones themselves. In doing this, JOY operates as an independent organization and respects and understands all written and the many unwritten laws of Sierra Leones society. From the Netherlands, we have the same goals, but instead of doing projects directly from a Western point of view, we support the initiative by letting JOY take the lead and help them in becoming more professional and structural as an organization. One of the things that we are working on is teaching the staff in using computers to write their proposals, estimates and to communicate with us through the internet.

In our search for sponsors, we found the Dutch based IT company Four Digits willing to help the Sierra Leonean team by donating them a portable laptop. During our recent visit we brought the laptop along and organized several computer courses to train the JOY members in using this modern machinery. In this article I will share with you my findings in using computers in Sierra Leone.

Choosing the right tools

Our Sierra Leonean team members are located in the town of Lakka, which is about 25 kilometers away from the edge of the capital city Freetown. The road between the village and the city is literally a pain in the ass and will take you about an hour to reach. Lakka has no electricity and obviously no internet. We figured that the best way to work would be to have a laptop that has a really long battery life so that documents and photos can be stored and prepared from their office in Lakka. Next we will supply the team with a USB stick, so that they can transfer the files to an internet cafe in town with or without carrying around the laptop whenever necessary.

Together with Four Digits we eventually picked a 10" Asus EEE PC with a 180GB hard drive to do the job. Ideally we would have chosen a solid state hard drive which would have been even more resistant against the bumpy Sierra Leonean roads, but by the time this hard drive would only come in 20GB, which would not be enough to store pictures on the long run. We also chose to supply an additional DVD player, since the input of the EEE PC is mainly focused around wireless communication and this is hardly found in Sierra Leone.

Other equipment that we've learned to be useful in a rough environment is a good waterproof cover for the equipment to fight the rain during the monsoon season, a voltage stabilizer for the power coming from generators or solar and spare USB cables because you never know.

Starting from scratch

After introducing the little machine to the JOY members, we started to give training in how to use the laptop. Knowledge about computers within the group was diverse. Ibrahim knew how to use email and how to create simple Excel and Word documents. Samuel and Abdul were pretty much newbie’s. Although Ibrahim knew how to use some software, he still didn’t understand the basic principle that lies underneath it all, which classically brought him in trouble every time he accidently pressed the “wrong” button.

Although I was pretty familiar with “African Logic”, I found it staggering to identify the difference in structural thinking between the African and Western world. Here in the West, we grew up with computers and computers grew up with us. I played simple games and wrote texts on my grandpa’s 8086 system when I was eight years old. As a teenager I settled myself for many hours behind the television screen at my friend’s place, playing Zelda. Later I got my own 486 pc on which I eventually figured out how to draw and create music. In short: my ability to use a computer increased almost simultaneously with the abilities acomputer could offer its audience. Even people from the older generation in the West, who often claim not to know how to use computers, are more or less familiar with programming orders into a machine, like a regular microwave or an alarm clock.

The personal computers that we know today, either Mac or PC, are advertized as being plug and play and easy to use for everybody and so on. In reality I came to realize that throughout the years they have actually become really complex magic boxes.

For example, take the Windows “Start”-button. If you actually lived through the years that Windows progressed from 3.11 to Vista, you could say: “Yeah makes perfect sense to start a new task with the "Start"-button in the down left corner of the screen”. But if you’re totally new to it all, this is not as logical as Microsoft might have thought it would be. My point is that in the beginning these machines look chaotic and confusing and after my first computer course I saw the pure horror on Samuel’s face. “You try it now”, I would invitingly suggest Samuel, on which he responded; “No no, let Ibrahim do it!”

Later that evening it stroke me: the stuff that I was teaching these guys was not fun or entertaining! Hell, the reason that I started to use a computer when I was young, was because it was fun and enhanced the things I was already interested in. Not because it allowed me to make practical to-do-lists or execute a variety of other dull-organizing-things-that-are-handy-but-also-boring. No way, it allowed me to compose electronic music without needing an expensive studio, it allowed me to create digital art and it entertained me with games. Next day I would try a new strategy...

Samuel is a DJ. With his towering speakers, two CD players and an old mixer he sets the groove for a party Sierra Leonean style! Could I describe this? Well, in short it adds up to loud and distorted reggae-ton beats, shaking booties and plenty of palm wine at room temperature in a seriously too small and steamy hot room. Trust me, it is good fun! Anyway, in my second attempt to teach him to use a computer, I asked him if it would be useful for him as a DJ, to make compilation CD’s that contain tracks that he often plays. “Yes, yes!” his reply was, “But how?” It was silent for a while and from Samuel I rolled my eyes to the computer. “Really, can you please teach me!?” he asked. “Yes, I can… yes, I can indeed.” I answered with a grin on my face. The bait had worked.

Later that day, long after the class had finished, I found Samuel and Abdul both sitting behind the computer, moving files, copying, pasting, renaming. In the evening they were still there and it reminded me of my own good old days of figuring out the magic box… The point that I would like to make, is that in teaching people from scratch how to use a computer, it is of uttermost importance to find out what it is that can link the personal interest of the person to the machine. If the goal is to teach somebody how to make estimates in Excel, start out with minesweeper... :-)

Maintenance

Our aim during the Christmas visit was to teach the members of JOY enough skills so that they could maintain the computer by themselves as soon as we would be back in Holland. Although we tried to focus on the controlling fundamentals of a computer, like the operating system, this aim still turned out to be too far out of reach. Alternatively for maintenance there are several computer stores in the capital city, but these would cost way too much money for what they would actually do. Therefore we contacted a friend from another NGO who I had been training in video production a couple of years ago. We found him willing to help out the guys from JOY whenever they get stuck or get returning error messages. We suggested the members to pay close attention when he is fixing the machine, so that they will be able to do these tasks themselves in the future.

Results

The project proposals and estimates that are communicated between Holland and Sierra Leone are nowadays well made in Excel or Word. This is a huge advantage and saves us a lot of time by for example not having to figure out if a character is indeed an '8' or perhaps a '0'. The only thing is that the internet is still very inaccessible. Since some time telecom providers in Sierra Leone also offer satellite internet, but with a monthly price tag that equals a local monthly salary, this is not a real option. All we can hope for is that the road from Lakka to Freetown will be finished any time soon and the internet in town becomes more accessible.

Thanks to Daan Velduizen for writing this excellent article.

On Tuesday, after the Zope Page Templates talk (18:30) we went to the Wellington pub having a beer with the guys from Netsight and Laurence Rowe. Birmingham is actually pretty nice in the evening as there are a lot of nice pubs and restaurants in the city centre.

Wednesday morning started with a talk from Bruce Eckel about programming language development and how Python was evolved compared to the other languages. Once again, check Reinout's weblog for details of this talk.

Reinout is writing everything down what is happening at the talks.

"We need to fail and we need to fail quickly". This was a talk about agile development. Bea Düring started talking about the two most widely used agile development methods : Scrum and Extreme programming. At their company, Open End, they developed a hybrid agile development method themselves. Important things like feature driven development and short iterations are featured in this hybrid method, including automatic testsuites. Interesting was the javascript testsuite they developed, you can find that one on PyPI.

"Tapping into the web of data" was a talk about semantic web data and RDF and what you can do with SuRF, a python library for RDF. Reinout, our hard working blogger, has posted a report about this.

Dr Sue Black of the University of Westminster was talking about the impact of Bletchley Park (Station X) on the world. On Bletchley Park the famous Enigma was cracked, the Colossus computer was invented, and a lot of other important code breaks as well. It is now a museum about code breaking, being an important museum in the UK.

People got really enthusiastic when showing a real Enigma.

We tried to have another conference call with Guido van Rossum again, but just like the day before it failed, so no Q&A with Guido this time. After the lightning talks, where Roel unfortunately couldn't talk about Plone as there were too many lightning talks, there was the excellent conference dinner.

More than 400 people at the conference dinner.

Right now Martijn Faassen is having a nice talk which probably will be reported on Reinout's blog. For so far the report from EuroPython 2009!

After the lunch, on the first day of the conference itself,  there was an excellent talk about copyrights and the impact of that for us as Python developers by Cory Doctorow. Reinout van Rees wrote an excellent report of this on his weblog.

Some effort was made to have a live video feed with Guido van Rossum. The facial expression of Guido on the stream was at least very funny, the conversation itself failed as we didn't had any audio. Next try is probably tomorrow.

Another interesting talk was about the FilterPype framework by Rob Collins, which is all about pipeline design patterns. Motivation of creating the framework was to simplify complicated processes into filters, like how the Unix piping system works. Implementation has been done using generators and coroutines, but it hides this implementation details for the users of the framework.

Pipelines have their own configuration language so that experts who need to create the pipelines do not have to program in Python. Check out the FilterPype website, as it is can be a nice framework for manipulating complicated data streams for example.

The Guardian has a interesting new approach for publishing their content so it's reusable by everyone who wants to use this content. One of the things they do now is that they publish data they collect (for example, all the bands and acts on Glastonburry) on a blog, so everybody can reuse that data as well. They also introduced a content API (for Python, but also for PHP and Ruby) so you can collect content in code pretty easily. There are some limits though, like you can query 5000 articles per day and you are not allowed to cache it longer for 24 hours. Anywho, other newspapers can take an example of at least trying to be innovative on the internet.

Last for today was the talk about Zope Page Templates by Matt Hamilton. There was some real interest in using the product, especially when people understood is could be used as a stand alone templating library without having the "Zope beast".

More coverage will follow tomorrow.

We arrived on monday evening and we met some Dutch Demoscene friends at the airport. After traveling to New street station, which is only 15 minutes by train from the airport, we checked in at our hotel and went for a walk to the city centre. We decided to meet in a pub called The Australian Bar and have a chat with the other EuroPython visitors. Discussions started immediately about Plone and it is clear that the Plone community is still seen as a separate group of people. Some work for Ploners to be done here!

It's good to see that our colleagues from Pareto are there too.

On tuesday, the first day of the conference itself, we were happily surprised by seeing a Plone logo over there.

Matt Hamilton from Netsight is running a Plone stand to promote it.

One of the nicest talks in the morning was from Simon Willison about the recent issues about the MP's expenses in the UK. He developed a django application so users can dig into the expenses and identify them, for example if they need further investigation.

See the website at http://mps-expenses.guardian.co.uk/

Interesting fact was that the application was developed in 4.5 days (!) and how he managed to make the website actual perform pretty good, and what lessons were learned from this project.

More coverage (hopefully) to follow soon!

Our requirements

Our goal was to build a HA (High Availability) server setup. In our case : If one physical machine fails, another should take over its tasks automatically. However, it's a waste if you have one machine doing nothing but waiting to take over, so it would be nice to have a combination of HA and load balancing, and a way to extend this setup with extra capacity if necessary.

Ganeti

Our first thought went to Ganeti, a tool developed by Google to manage virtual machines in a cluster with one or more physical machines. It doesn't support automatic failover yet, and the technology is very new so it would be risky to use this software for our setup. It's pretty neat software though,check out this PDF file for more information about Ganeti.

Traditional HA

A traditional setup is perfectly described here by the guys of Goldmund, Wyldebeast & Wunderliebe which is a HA setup which has proven to work. However, we wanted to extend this setup a little more, as we really liked the idea and flexibility using virtual machines as Ganeti does. It's easier to migrate this setup to a new machine, and you can move or copy virtual machines to another physical machine if necessary. You can also manage and configure the resources more easily, having the ZEO client VM having more memory and CPU power then the other VM's.

Our setup

The basis of our setup is almost the same as the HA cluster howto on plone.org : A master and a slave node. We added DRBD for data synchronization and Xen to have a virtual system for each layer in the setup. On each server  we have three (Debian) virtual machines : One for the Apache frontend (and caching if you want to), one for running the ZEO clients and one for the ZEO backend (and MySQL in our case). We mount a NFS disk for the actual data storage, but to keep it simple for now we assume the ZEO backend virtual machine stores the data. See the following image :

How it works

In a normal condition (when both nodes are active) the Apache VM and ZEO backend VM are running on the master node. The Apache VM has the public IP address (or floating address as mentioned in the HA cluster howto) to access the plone websites. The ZEO backend VM and the Apache VM are installed on a DRBD partition. This means these VM's are always being synchronized with the slave node. On both nodes there is a ZEO client VM which runs the ZEO clients. These are not installed on a DRBD partition (and not automatically synced) as they both need a different IP address (for the load balancer), and you may want to start a ZEO client in debug mode (for testing purposes for example). You can add new physical machines which run a ZEO client VM as well so you can extend capacity easily this way.

The main idea is actually the same : Heartbeat is configured on the slave node to monitor the master node. If the master server goes down it will start the Apache VM and the ZEO server VM on the second node. As these VM's are installed on DRBD partitions they are identical to the VM's of the first node. The Apache VM on the second node has the same IP address so it will take over all requests and the ZEO clients will reconnect to the 'same' ZEOserver, which is now active on the second node.

In Part II we will describe in more detail how we configured this setup, including our used configuration.

References

• HA loadbalanced cluster for Zope and Plone
• Google Ganeti
• DRBD
• XEN
• Heartbeat
• A simple introduction to work with LVM
• Setting up Xen on your Debian Etch box
• Xen and Heartbut Ubuntu Howto