Removing a persistent local utility part II

The problem

When you remove a product, Zope does not know any more what the pickled objects in the ZODB are. Sometimes you do not notice any problems until you want to install a new product or want to change some setting in the plone control panel. Strange error appear like :

Module Products.GenericSetup.tool, line 1181, in _runImportStepsFromContext
 Module Products.GenericSetup.tool, line 1092, in _doRunImportStep
 - __traceback_info__: toolset
 Module Products.GenericSetup.tool, line 130, in importToolset
TypeError: 'NoneType' object is not callable

or you see messages in your Zope log like :

WARNING OFS.Uninstalled Could not import class <myclass> from module <mymodule>

Sometimes you get a ComponentLookupError by visiting the Plone site, even when you use the ZMI. Or some other strange error occurs (I've seen many different errors unfortunately).

Normally you can remove a local utility by using the unregisterUtility method, but at this moment you don't have the product installed, so importing interfaces etc is not an option. Reinstalling is not what you want, so what to do now?

What to do

The first thing to do is to find out which product was responsible for the broken persistent components so know where and what to look for. You probably remember which product you uninstalled, otherwise you can find some hints in the zope access log file. Look in the profile and zcml of the product what is registered using a generic setup profile. We use collective.myproduct for the rest of this blogpost.

Create a new zope instance with your broken Data.fs and start a debugging prompt with bin/instance debug so you can start investigating the ZODB.

Tip :You can use the .__dict__ property or the dir() method to inspect objects in the ZODB.

Disclaimer : Do not ever, ever do this in production environments unless you know what you are doing. One mistake can break your Plone website totally.

Sitemanager

In the zope component architecture, local utilities are stored in a local site manager. With Plone 3 and 4, you have a Sitemanager on your portal object. First thing to do is to get this Sitemanager.

sm = app.myportal.getSiteManager()

The sitemanager holds three types of local components : adapters, subscribers and providers.  (At least, that is what I found out). We should get them so we can see if our broken component is there somehow :

adapters = sm.utilities._adapters
subscribers = sm.utilities._subscribers
provided = sm.utilities._provided

Let's list (a part) of the current registered adapters :

>>> for x in adapters[0].keys():
...     print x
... 
<InterfaceClass plone.browserlayer.interfaces.ILocalBrowserLayerType>
<InterfaceClass zope.app.cache.interfaces.ram.IRAMCache>
<class 'collective.myproduct.interfaces.IMyProductConfig'>
<InterfaceClass Products.PortalTransforms.interfaces.IPortalTransformsTool>
<InterfaceClass plone.app.i18n.locales.interfaces.IContentLanguages>
...

In the list we can see that there is an adapter registered, and you can also see that the class differs from the others, because it's a broken object. You can verify that it's broken by printing the title of this object.

for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
        print "%s" % x.title
This object from the unknown product is broken!

So we found our broken object so we should delete it :

for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
        print "deleting %s" % x
        del adapters[0][x]
sm.utilities._adapters = adapters

After the reassignment the ZODB machinery does not detect that there is a change, so we should commit this change :

from transaction import commit
commit()
app._p_jar.sync()

The same story goes for the subscribers and providers, so here is all the code together :

sm = app.myportal.getSiteManager()
adapters = sm.utilities._adapters
for x in adapters[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del adapters[0][x]
sm.utilities._adapters = adapters

subscribers = sm.utilities._subscribers
for x in subscribers[0].keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del subscribers[0][x]
sm.utilities._subscribers = subscribers

provided = sm.utilities._provided
for x in provided.keys():
    if x.__module__.find("collective.myproduct") != -1:
    print "deleting %s" % x
    del provided[x]
sm.utilities._provided = provided

from transaction import commit
commit()
app._p_jar.sync()

Now we got rid of our persistent components we don't want anymore, but some problems can still occur when (re)installing products using the Portal Quickinstaller or the Portal Setup tool.

Portal Setup

If you still have problems (re)installing products after you removed the broken local persistent components, you probably have to clean the Portal setup tool.You probably see something like this in the error log :

TypeError: 'NoneType' object is not callable

Check your product if it registers a tool using a toolset.xml file, like this :

<?xml version="1.0"?>
<tool-setup>
 <required tool_id="portal_myproduct" />
</tool-setup>

Fire up your debugger and get the portal_setup tool. It keeps its own registry for all registered tools which you can get using the getToolsetRegistry method :

setup_tool = app.myportal.portal_setup
toolset = setup_tool.getToolsetRegistry()

Check if a tool exists with the tool_id, which has been registered using 
<required tool_id="portal_myproduct" /> and remove it :

if 'portal_myproduct' in toolset._required.keys():
    del toolset._required['portal_myproduct']
    setup_tool._toolset_registry = toolset

from transaction import commit
commit()
app._p_jar.sync()

I'm pretty sure this is not the way to go, as I occasionally still see the OFS.Uninstalled warning, so I hope someone who reads this says "This is not the way to go, you should do this".

Quit your debugging session and start your instance with bin/instance fg. Browse to the portal_setup tool using the ZMI and go to the "manage" tab. (http://localhost/myportal/portal_setup/manage_stepRegistry). There you can delete any invalid import steps if they are still there.

Now you should be able to install products again, and have your Plone site up and running without problems.

Nice post, but this kind of ugly hacking shouldn't be necessary!

And I agree totally on this. So a note to all product developers is : Please write uninstall profiles! You can also remove persistent utilities in your uninstall profile like Roel described in a previous blogpost. And I' sure there is some development on this part with Generic Setup.

Feedback

Everything I described in this post I found out by trial and error, as there is little documentation on these issues. And if it's there: I would like to know! And if you have better, nicer or other ways of doing all these things : Also let me know! Thank you in advance.