Personal tools


Mar 26, 2010

"Future is bright, future is Plone"

Sometimes when you are doing a lot of Plone development and integrations you could miss the big picture: Plone is not just a CMS. It's a damn good CMS with almost unlimited possibilities of integration. However its 'unlimity' has started to be one of its biggest limitations.

I'm working right now on a project that uses ore.contentmirror to serialize Plone content data to postgresql which can be later reused in repoze.bfg. I have found several similar deployments in the Plone community which were quite inspiring.

Plone integrationsThe scenario is almost always the same:

  • to use Plone as a CMS (backend)
  • to have fast framework (frontend) to serve my backend data
  • to include in frontend layer Web2.0 functionality and some other dynamic stuff
  • to make benefit of external indexing server for full text search, for both frontend and backend


You can of course use solr with collective.solr for search engine but thanks to tsearch2 some of the search/ranking functionalities can be taken directly from postgresql (making the whole stack much smaller and easier to maintain).

Serializing Plone data to SQL opens many possibilities. But with Plone you can do much more.

Plone integration world

You can integrate other CMS's like Alfresco or Sharepoint using CMIS. You can connect your Plone instance to Salesforce. Instead of using tsearch2 or solr as an indexing engine - you can use Google Search Appliance inside your Plone instance. If you want to start online document collaboration you can even connect it to Google Docs.

External services and applications are not the end of the story. There is of course WSGI with collective.xdv and Deliverance and bunch of other interesting middlewares and we shouldn't forget about PAS plugins.

It's possible that I missed something interesting. Plone community is very creative, thought. It's hard to be always up to date. That's why I'm sure that I miss a place where integrators and developers can share they successful stories and discuss potential use cases. A place which will make Plone future bright.


Mar 25, 2010

Just a little pin prick for your ill pdb

Filed Under:

Some packages can compromises the pdb interactive shell by redefining sys.stdout

Disclaimer: Tested only on linux and mac

Symptoms: python readline does not work at all in the pdb shell.

If you find out that inside a pdb after pressing some keys, e.g. the arrow keys, some escape symbols appear on the line...

(Pdb) test^[[D^[[D^[[D

...and you cannot move the cursor to the desired position, probably some package screwed up your pdb!

This is very annoying to me because some times I have to spend a lot of time in the pdb to heal my buggy code: a misfunctional readline makes me lose too much time!

This probably happened because somewhere in the code the standard output (sys.stdout) has been redefined and this screws up the pdb. 

Check it out in the console:

(Pdb) import sys
(Pdb) sys.stdout

 If the output is not something like

<open file '<stdout>', mode 'w' at 0xb783d070>

than you can go on reading to find out how to fix it. 

Tip: the output can give you an hint of the code that is redefining the stdout.

Don't ask me why pdb does not like it. If you are able to identify the nasty piece of code that modifies the standard output and replicate it in a python console, this problem does not appear.

Therapy: fix the stdout!

In most of the cases this will give you back a fully functional console:

>>> import sys;sys.stdout=file('/dev/stdout','w')

Giving roles to visitors using HTTP headers

Filed Under:

In a recent project we need to provide different roles to users, basing this choice to host name used to reach the Plone site

What's up?

The Plone site I'm describing here is quite normal, but customer ask us to give some special additional permissions to users that reach the Plone site from an internal domain.

In facts the wanna be still anonymous (forcing no-one to authenticate) but be able to see some documents in a special "Published internally" state.

How tho give this permission to anonymous users?


I never used AutoRole before, but its clear that the idea behind is what we need. AutoRole is an interesting PAS plugin provide additional roles automatically using the IP of the client that is not what we really wanna there.
It also works well with anonymous users making some magic inside the plugin!


You can find on the Plone SVN our first attempt to use the AutoRole idea for our needs. Changing some lines of codes here and there we changed roles provided relying on HTTP_HOST used.

The HTTP_HOST works only when the client reach directly the Zope server (not exacly, but we have no controls on the Apache of that company)... and we wanna put Varnish in front of it.
What if tomorrow I need to give somewhere an additional role to users that use a specific browser, or something else? I can't spend all of my live developing AutoRoleFromSomething products!


The best choice we found is to look at HTTP Header in general, making what header and what value completely configurable.

We developed and released AutoRoleFromHostHeader. Similar to AutoRole, but  you can configure it like this:

HTTP Header;regexp;role,[role,]

To make it the most general as possible, the value of the header is used as a regular expression.

Using this you can reproduce some of the AutoRole features, but you can also make something like this:

HTTP_USER_AGENT;(MSIE|Internet\ Explorer);BrowserlessVisitor


Mar 22, 2010

Conditional Javascript choices relying on server side configurations

Filed Under:

Have you ever felt the need of make a Javascript choice, executing some client side actions, basic this choice on some server side configuration? Here an interesting way I recently taked, thanks to the powerful jQuery plugins structure.

Describing the problem

I'm near to release a new version of collective.flowplayer_toolbar (so it's only on collective SVN right now).

The first version of the product (see the old blog post about it) give a new Javascript controlsbar to Flowplayer, disabling the native Flash ones.

In a production environment we received some comments about this choice because when disabling the Flash controls you are not able anymore to use fullscreen mode... bad!
The only solution is to keep both control systems because is not possible (due to security issue) to control fullscreen from Javascript.

Have both controls give us the best freedom (features and accessibility)... but to be honest I find it really ugly!

Flowplayer instance with controlsbar

So, the best way is to give some configuration for switch from:

  • using only the Javascript controls
  • using Javascript and Flash controls

The collective.flowplayer choice

The same kind of problem has been overcome by collective.flowplayer.
Some of the configuration you can manage in the flowplayer_properties leave to changes in the client-side Javascript behaviour.

I'm not sure that this is the only motivation, but this problem is solved there providing the Javascript code not using a static file (so not giving a Javascript text source) but using a zope 3 view.
This view (collective.flowplayer.js) return a text/javascript content type:

    def __call__(self, request=None, response=None):
        """ Returns global configuration of the Flowplayer taken from portal_properties """
        self.request.response.setHeader("Content-type", "text/javascript")

The problem with this approach is that you must work with Javascript code written inside Python string... not really comfortable.

Of course, the problem is only for the developer. Client side the Web browser will not feel any difference.

Experimental approach

As far as Flowplayer works well with jQuery (and jQuery Tools, that is a requirement of collective.flowplayer using I'm trying a second approach to the problem. This because:

  • I don't like to write Javascript inside strings
  • I really like the power of jQuery plugins support, and this has become more clear to me after having read recently a book on this.

Following the server side collective.flowplayer approach I added a new property to the portal_properties tool (it's only a single property, so I added it to the same flowplayer_properties sheet).

This boolean property (toolbar_flash_controlsbar) must only controls this single line of Javascript code:


I need a way to execute this only when the flag is True.

The solution is simple: execute this line of code only when a Javascript variable is evaluated true/false:

if (!show_flash_controlsbar)

Now we only need a way to controls from the server this client variable, but first it's better to provide a default initialization.

As far as this control is executed inside a Flowplayer event, we can add the initialization before or after this piece of code, directly in the flow of page execution (so the order matters not):

var show_flash_controlsbar = false;

So the default is also the default of the toolbar_flash_controlsbar property (False) and we will not see that Flash controls.

Now we really need a way to react to a True value server-side.

The way used here is to include another (minimal) Javascript piece of code, this time after the ones above (so, if included, it must be after the initialization of the show_flash_controlsbar).

show_flash_controlsbar = true;

To keep controls on the inclusion of this we need to put this into a separate Javascript file and include it conditionally after the ones above.

The Generic Setup inclusion take this form:

 <javascript cacheable="True"

In the expression TAL we check and evaluate the server side property (and for security we also put a |nothing, to prevent all sort of errors if, for example, the property has been deleted).

The portal_javascript tool take care of providing this Javascript script or not, even when the it's merged with other and cached.

Don't you talked of jQuery somewhere above?

The only problem with this approach is the namespace pollution. We must be sure that the name of this variable is not used/taken from other Javascript code somewhere in Plone.

jQuery and its plugins technology help in this, and also help to keep things more clear and ordered.

With the syntax below we can make this definition available only inside the jQuery environment, preventing that non-jQuery Javascript code change/use this variable.

jQuery.show_flash_controlsbar = false;

Note the use of jQuery name, not $ or jq: this is how plugin development guidelines recommend to define and use jQuery object when developing plugins.

In our code we can continue testing the condition in this way:

if (!jq.show_flash_controlsbar)

This is better than before, but we can make it more robust.

This approach can fails only if another jQuery plugin define and use the same name for an internal variable of function (ok, this is paranoyc); if this become true we can get errors or unexpected results.

The best jQuery way is to define an internal namespace, making this problem more and more difficult to arise.

So finally we define the variable in this way:

jQuery.flowplayer_toolbar = {
	 * Some other script can put this to true to enable also the native controlsbar plugin
	 * This is the only way to get features like the fullscreen.
	show_flash_controlsbar: false

And we test it like this:

if (!jq.flowplayer_toolbar.show_flash_controlsbar)

The last benefit of using an internal jQuery namespace is to keep similar variables or function together. May be that some other variables (or global functions?) can be needed in future version of this product, so we can add them inside the same namespace.

Going back: is this approach better or not?

Aswer: I don't know. This works well with browser cache but as you can see it needs some more work than providing a dynamic Javascript source from the server using a view.

The most annoying things is the registration of an additional entry inside the portal_javascript tool only for put an additional line of Javascript code... this is not a problem for one script, but can you imagine this repeated for all collective.flowplayer configuration (or for other products too)?
The numbers of entries will explode!

Mar 16, 2010

PloneboardNotify: how tests will save my fat boy

Filed Under:

A new release of PloneboardNotify... no much more features, but now I can think about refactoring my too grown son.

The new release of PloneboardNotify doesn't give us many new features but is not this the real focus of this article.

This add-on was already available and used in production environment, and one of the first effort was to keep Plone 2.5 compatibility.
The same is for the new 0.4 release (and maybe Plone 2.5 compatibility will continue until all of our customer will drop this version).

So? What is the problem?

Thanks for testing

Sometimes when you develop something that is useful for a single site, it became magically useful for other... but sometimes you are too stupid (and lazy) to understand it.
Let's go back to Ploneboard: after all the notification e-mail after a new forum discussion/response is a wanted feature (worst... an expected feature), not available in Ploneboard (that, even if it isn't perfect, is the only mature choice available right now on Plone to have a forum).

PloneboardNotify is a good example of a bad way to start a product... developed fast but with no eye on good OOP, without thinking about make it extensible... not tested at all. It was only a stupid event script that send e-mail (luckily this old version isn't available or any public SVN so the Story will not judge me).

First public released versions were only more "user friendly", making it usable in any context where you have Plone and Ploneboard. Maybe they are good release but was clean the need of refactoring: but refactoring is someway a dangerous task (the code is working... why I must spend my time making it better only to get the same features and be forced to test all feature and behaviour from scratch? I'm sure to remember what I need to test?)!

The problem there was (and is also now) the "core" (please... don't look at it!). Tests at every release were done by me (and my name is not Funkload) manually playing with the browser, repeating the same operation on Plone 2.5 and 3. Time after time the core grow as new feature is added... Tests are every time more expensive and boring...

What is changed now? Nothing! PloneboardNotify is still a mess... but now it's a tested mess!


The primary effort of this release was adding a complete functional tests coverage of all features (for both Plone 2.5 and 3 versions). After that, thanks to the help of Nicolas Laurance, I also added the HTML e-mail feature.

Now I have no fear to make changes to the code core... also in this release I removed some stupid piece of code (after completing tests!) and introduce adapters. Having tests available now will speed up dramatically the develop process of new features.

Next steps will be to reach a better OOP and making the product more customizable by other developer in their sites (like customize the e-mail template in some simple way).

Lessons learned

  • Don't be lazy. You can make this code more reusable. Always.
  • Don't be lazy. Write tests! Better: you will became lazy if you don't write tests because testing manually is very boring and you'll begin skipping some action!
  • Don't be stupid. You can't make deep refactoring and expect that all will work after you've finished, so...
    make tests! ;-)