Skip to content. | Skip to navigation

Four Digits | Jansbinnensingel 26, 6811 AL Arnhem, The Netherlands | info@fourdigits.nl
 

Transmogrify 100k AT objects from Plone 3 to Plone 4 Dexterity types

by Steven Looman Jan 18, 2013

Since a while we are working on our new support system. Our support system supports our workflows. It holds all our customer information, data, project information and support tickets are registered by us and our customers in our support system. Using the tickets, we can communicate with our customers and keep track of the status of the issues.

Support was built around five years ago, written in Plone3, based on Poi. It is time for an upgrade, since our working methods have evolved, and, we like to update to Plone 4. We don't want to lose our existing tickets, so we're migrating all the data from the old support system to the new support system.

We are exporting around 100.000 objects to JSON files from the old support system. We are using  collective.blueprint.jsonmigrator to do this. Each JSON file represents an ATContentType Plone-object, with all the properties we need (now or possibly in the future) to keep all the existing data. This process takes about 30 minutes to run. After this, we have 100.000 files including all data and blob files.

In the new support system we are importing the objects from the exported JSON files. For this we're using the collective.transmogrifier and transmogrify.dexterity packages. Several passes are performed on each JSON file to convert it to, for example Dexterity types, and change properties where needed. We read the Title of the object from the JSON file and store it in the newly created object.

Another step is changing the IDs of the tickets. In the old system, the IDs are simple integers, which are increased by one, each time a ticket is created. Each ticket belongs to a Tracker and there are several Trackers, at least one for each customer. In the new support system we are using a new ID-creation scheme where each ticket has a globally unique ID.

The new support system is based on bootstrap responsive layout and has the plone.app.toolbar for editing. All contenttypes are dexterity types, so it includes plone.app.contenttypes for folders and documents.

The whole import-process takes around 2 hours. So we are exporting and importing around 100.000 objects in less than 3 hours.

 
Steve McMahon

Customizations

Posted by Steve McMahon at Jan 18, 2013 05:53 PM
A great story! Did this require any customization of collective.blueprint.jsonmigrator or transmogrify.dexterity?
 
Steven Looman

Customizations

Posted by Steven Looman at Aug 19, 2013 03:14 PM
Hi Steve,

Sorry for the late reply. We had to change some steps from the original transmogrifier. For example, to keep the history (which I'm writing a post on right now) we had to either create a new object, or re-use the existing (older) object.

Due to the large amount of custom types, we needed to create several steps to handle the conversion from old types to new types. Fortunately, most of the types mapped one to one from old to new.

Also, some changes were introduced to owner IDs, which required some conversion as well.
 
Gil Forcada

jenkins

Posted by Gil Forcada at Jan 18, 2013 11:53 PM
Cool setup!

At der Freitag we had to migrate tiple your data (300k objects) and to really feel safe on *the day* we had a jenkins server doing migrations every single day during 5 months prior to the final migration.

You never know what users have entered in all the fields that will break the import, so having an automatic export from the live system and an automatic import of that data in a clean system was really helpful for us, without that we would be still debugging errors!
 
Steven Looman

jenkins

Posted by Steven Looman at Aug 19, 2013 03:18 PM
Impressive! And great advice, never trust your input data. It has caused a few headaches here... ;)

Maarten used to do several exports of the old site/imports to the new site a day. Later on, we only managed to do one export/import a day, due to the steps growing and growing.
 
Made by Four Digits based on Plone.
Made by Four Digits based on Plone.