In one of our projects we are still using Plone 3.1.x. with a pack of customer specific content_types and some additional PAS plugins (Lotus). I have noticed recently that calling invokeFactory become VERY slow. After debugging we have found the problem - getDefaultAddableTypes - method that is provided by constraintypes.py (ATContentTypes package). Original implementation is more then 4 years old. What it does - it checks isConstructionAllowed for every content_type register in portal_types. If you have registered them more then 40 ;-) ...

In most of the cases folderish types are filtering allowed_types. Thanks to Vincent we have now a nice fix:

Modified lib/constraintypes.py:getDefaultAddableTypes method to check isConstructionAllowed only for allowed types, not for all content types in portal_types. isConstructionAllowed was called twice for each types.

This fix is in ATContentTypes version 1.3 and started to be used by Plone 3.3.1. For us it was a good argument to upgrade!

Let's suppose you have to concatenate several collections objects, e.g.:

>>> a=range(3)
>>> b=set('ale')
>>> c=(a,b)

One easy way to do that is to create an empty list and use the extend method

>>> cat=[]
>>> cat.extend(a)
>>> cat.extend(b)
>>> cat.extend(c)
>>> cat
[0, 1, 2, 'a', 'e', 'l', [0, 1, 2], set(['a', 'e', 'l'])]

But another elegant way do achieve the same result is to use the chain function from itertools

>>> from itertools import chain
>>> chain(a,b,c)

>>> cat=list(chain(a,b,c))
>>> cat
[0, 1, 2, 'a', 'e', 'l', [0, 1, 2], set(['a', 'e', 'l'])]

This is especially useful when you have big amount of data because the result of the chain function is an iterator

During a technical meeting here in RedTurtle, me and my colleagues were exposing fancy stuffs about software development and I was amazed that the attention of my team mates was triggered by me using a self made Pydev template to insert the encoding on top of Python files.

So here, I am reporting this to the whole planet, hoping someone else will appreciate this simple time saving tip :)

Adding a new template in Pydev is really easy, just go to Window -> Preferences... A configuration window should appear (see image below).

Navigate the tree panel on the left: Pydev -> Editor -> Templates and add click on the New button.

Another window will appear, where you can insert a name for your template (in my case utf8), a description (Insert encoding in the file) and the text to be inserted when this template is used (# -*- coding: utf-8 -*-).

With this template configured, insert che utf-8 encoding in to a file is just as easy as typing ut[CTRL]+[SPACE].

Here @RedTurtle we are heavily using funkload for acceptance and benchmarking tests. We have started using it in 2008 just after Bristol Performance Sprint. We have found even more useful with buildout recipe. The only missing part was the recorder. It's built-in Funkload itself, but it was not enabled in the recipe. Well - now it is ;-)

/ If you want to know how to include funkload in buildout project - check my previous blog /

Starting a recorder is quite easy:

$ ./bin/funkload record -p 8080 MyTest

for full usage call:

$ ./bin/funkload record --help

This will start proxy on 8080 port and save all your browser requests to MyTest funkload scenario. Now open your browser, change proxy configuration to localhost:8080 and click-through test case. When you finish - stop the proxy with Ctrl-C. Funkload will generate a test_MyTest.py file and notify you where you can find it. It should be now collective.funkload compatible - you can lunch:

$ ./bin/funkload test /path/to/test_MyTest.py

To test it.

Another small improvement is a PloneFLTestCase. It extends default funkload test case, implementing two additional methods helping with Plone content creation. Right now instead of using this approach:

folder_portal_factory = self._browse(server_url + "/coreloadtests/Members/" + self.user_id +"/createObject?type_name=Folder",
                                             method='get', 
                                             follow_redirect=False,
                                             description = 'Get folder portal factory')

folder_edit_url = folder_portal_factory.headers.get('Location')        
folder_id = folder_edit_url.split('/')[-2]

folder_created = self.post(folder_edit_url, params=[
    ['id', folder_id],
    ['title', 'folder'],
    ['description', ''],
    ['description_text_format', 'text/plain'],
    ['subject_existing_keywords:default:list', ''],
    ['last_referer', 'http://localhost:8080/coreloadtests/Members/' + self.user_id + '/view'],
    ['form_submit', 'Save']],
    description="Post /coreloadtests/Members/user...280843853/atct_edit")

new_folder_id = folder_created.url.split('/')[-2]

you can just do:

new_folder_id = self.addContent(
        server_url, 
        portal_type='Folder', 
        params=[['id', 'id'],
                ['title', 'testing title'],
                ['description', 'testing description'],
                ['description_text_format', 'text/plain'],
                ['form.submitted', '1'],
                ['last_referer', ''],
                ['form_submit', 'Save']], 
        description='Create folder')

It doesn't do much, but for sure it helps you keep you test case readable.

Following example returns full path to ATFile 'file' field on disk:

>>> field = self.context.getField('file')
>>> fss_info = field.storage.getFSSInfo('file',self.context)
>>> fss_info
<FSSFileInfo at test-file.pdf/>
>>> fss_strategy = field.storage.getStorageStrategy('file',self.context)
>>> fss_strategy
<iw.fss.strategy.DirectoryStorageStrategy instance at 0x21332c68>
>>> fss_props = field.storage.getStorageStrategyProperties('file',self.context,fss_info)
>>> path = fss_strategy.getValueFilePath(**fss_props)
>>> path
'/opt/buildout.plone/var/fss_storage_global/5b/5b09/5b09c322ae211207649e1803fe012cca_file'

The purpose of GDIP is to provide the Google Docs services to Plone users.

Advantages:

• Plone users can save their files on Google servers, instead of in the ZODB;

• Plone users have multiple access points to your files: they can manage their documents using two systems: Plone and Google Docs;

• Plone users can edit their documents directly from the Google Docs application, embedding that page in the current window of the Plone application;

The products collective.googleauthentication, collective.googlesystemstorage, collective.googlesharing and collective.googlemodifycontent provide the integration of Google Docs services in Plone.

Packages were released on pypi:

http://pypi.python.org/pypi/collective.googleauthentication

http://pypi.python.org/pypi/collective.googlesystemstorage

http://pypi.python.org/pypi/collective.googlesharing

http://pypi.python.org/pypi/collective.googlemodifycontent

and on plone.org:

http://plone.org/products/collective.googleauthentication

http://plone.org/products/collective.googlesystemstorage

http://plone.org/products/collective.googlesharing

http://plone.org/products/collective.googlemodifycontent

The system integrates Google Docs services in Plone using the gdata-python library provided by Google, which in turn uses the Google API.

GOOGLE AUTHENTICATION collective.googleauthentication

 To let the Plone application access the documents stored on Google servers, it is necessary to complete the authentication procedure for Web applications provided by Google Docs. The procedure allows Web applications to authenticate users through their Google accounts. For security reasons, the application acquires an authentication token which will later be used to dowload or upload documents from Google servers without explicitly providing the user's credentials. The Plone user is redirected to a Google page that invites him to insert his credentials. Once he logs in with his Google account, the user is asked to authorize the Plone application to access his documents. Then, if the user grants access, he is pointed again to the Plone site. The URL of the last redirection embeds the authentication token which, as mentioned above, allows the Plone application to access the user's documents on Google servers for the following requests. GA inititates the authentication procedure upon Google Docs immediately after the user has logged into the Plone application. The procedure will be executed just once, as when the Plone application obtains the authentication token, it will store as an attribute, google_token, in the user profile.

GOOGLE SYSTEM STORAGE collective.googlesystemstorage

GSS saves the document types supported by the Google Docs service on the Google servers, while storing all the other files in the local filesystem, so that Plone users to access their files from both the Plone application and their Google Docs account. GSS extends FSS through a meta ZCML directive will trigger the adoption of GSS as storage. GSS provides several utility methods: a method that takes an authentication token and returns a client for Google Docs, methods that take care of uploading, downloading and deleting the documents on Google servers, and a method that performs queries on documents stored on Google servers.

GOOGLE SHARING collective.googlesharing

GS manages the sharing of documents stored in the Google servers and their synchronization from the Plone application to Google Docs service. In this way, when a Plone user changes the roles of other users on a specific document, GS changes the document sharing attributes in the Google Docs service accordingly. So, if a Plone user assigns another user the Editor role on one of his documents, the other user will be able to read and modify that document through his Google account. To associate Plone and Google accounts, the system assumes that the email address attribute of Plone users corresponds to their Google account. The sharing attributes of documents stored in Google servers is managed through the feed access control list (ACL), a list that shows the Google users that have access to a specific resource. GS redefines the googlesharing view to perform the mapping of roles and the ACL feed retrieval and changing operations.

GOOGLE MODIFY CONTENT collective.googlemodifycontent

GMC extends the Plone content Edit function by adding the GoogleModify operation, which only applies to documents stored on Google servers. GMC embeds the Google Docs application inside the GoogleModify panel, allowing Plone users to edit their documents directly from the Google Docs application. GMC provides the new googlemodifycontent view, that takes care of discovering the specific URL of the Google Docs document function and of embedding that page in the current window of the Plone application.

One idea for improving the system is using the workflow, instead of directly manipulating roles and permissions associated with content.

The same bug! For the second time I found the same bug in an old piece of code... again the same silly bug!

Ok, that's my fault... again...
What I can say? I was young and zope.interface was a new entry in the Plone world... often when you begin playing with a new APIs you don't read documentation before... why read how to use a method called "getNameOfCurrentUser" or "convertToUnicode"?
I think that the name says all I need!

Even worst: you try the command and... it runs! You got the expected result, so why investigate further with documentation?

What I'm talking about? I'm talking of the Evil directlyProvides method!

Let's go back to those sad days

I have an object... I need the API that make possible to provide an additional interface... this is for sure a simple task for zope.interface library...

What I only know is this: a class implements an interface, and object provide it...

When I find an API called directlyProvides I try to understand it's meaning using it's name... may be calling this I'll make possible that my object provide my interface (why directly? I don't know... maybe because the interface is not inherited by the class... it's not implemented... I really don't know!).

Is my first idea right or not? Let's try. I can use this:

>>> from zope.interface import Interface, directlyProvides
>>>
>>> class IMyClass(Interface):
...     pass
...
>>> class MyClass(object):
...     pass
... 
>>> o = MyClass()
>>> IMyClass.providedBy(o)
False
>>> directlyProvides(o, IMyClass)
>>> IMyClass.providedBy(o)
True

So it works! I found my API!

The Devil, revealed

Going back to the real life... today I know the truth:

>>> print directlyProvides.__doc__
Declare interfaces declared directly for an object

      The arguments after the object are one or more interfaces or interface
      specifications (``IDeclaration`` objects).                            

      The interfaces given (including the interfaces in the specifications)
      replace interfaces previously declared for the object.

I'm not really sure of why this API exists (I hope for some good reason, of course...).
Today I also know that the real API I need is alsoProvides (you know... also this name is right... the problem is that I found directlyProvides before the last one...).

Only a doubt about the name itself... why they didn't called it like one of those?

• onlyProvides
• exclusivelyProvides
• willProvideThisAndNoMoreInterfaces

Lessons learned?

• Read the documentation! Check for the __doc__!
• Probably you don't need directlyProvides in Plone!
  

I'm working right now on a project that uses ore.contentmirror to serialize Plone content data to postgresql which can be later reused in repoze.bfg. I have found several similar deployments in the Plone community which were quite inspiring.

The scenario is almost always the same:

• to use Plone as a CMS (backend)

• to have fast framework (frontend) to serve my backend data

• to include in frontend layer Web2.0 functionality and some other dynamic stuff

• to make benefit of external indexing server for full text search, for both frontend and backend

You can of course use solr with collective.solr for search engine but thanks to tsearch2 some of the search/ranking functionalities can be taken directly from postgresql (making the whole stack much smaller and easier to maintain).

Serializing Plone data to SQL opens many possibilities. But with Plone you can do much more.

You can integrate other CMS's like Alfresco or Sharepoint using CMIS. You can connect your Plone instance to Salesforce. Instead of using tsearch2 or solr as an indexing engine - you can use Google Search Appliance inside your Plone instance. If you want to start online document collaboration you can even connect it to Google Docs.

External services and applications are not the end of the story. There is of course WSGI with collective.xdv and Deliverance and bunch of other interesting middlewares and we shouldn't forget about PAS plugins.

It's possible that I missed something interesting. Plone community is very creative, thought. It's hard to be always up to date. That's why I'm sure that I miss a place where integrators and developers can share they successful stories and discuss potential use cases. A place which will make Plone future bright.

Disclaimer: Tested only on linux and mac

Symptoms: python readline does not work at all in the pdb shell.

If you find out that inside a pdb after pressing some keys, e.g. the arrow keys, some escape symbols appear on the line...

(Pdb) test^[[D^[[D^[[D

...and you cannot move the cursor to the desired position, probably some package screwed up your pdb!

This is very annoying to me because some times I have to spend a lot of time in the pdb to heal my buggy code: a misfunctional readline makes me lose too much time!

This probably happened because somewhere in the code the standard output (sys.stdout) has been redefined and this screws up the pdb. 

Check it out in the console:

(Pdb) import sys
(Pdb) sys.stdout

 If the output is not something like

<open file '<stdout>', mode 'w' at 0xb783d070>

than you can go on reading to find out how to fix it. 

Don't ask me why pdb does not like it. If you are able to identify the nasty piece of code that modifies the standard output and replicate it in a python console, this problem does not appear.

Therapy: fix the stdout!

In most of the cases this will give you back a fully functional console:

>>> import sys;sys.stdout=file('/dev/stdout','w')

In early days on Plone I always used this API for logging:

context.plone_log("That's a log message")

This will give you a log message like this:

2010-02-15 09:56:32 INFO Plone Debug: 
That's a log message

I don't know or remember if the Python print command always worked on Plone, but right now you can use it freely.

I found that sometimes I used print, just because is simpler (and I don't need any context as plone_log).

Now the problem... use plone_log (or every other use of python logging system) is safe when writing functional doctests.

The print command isn't! If you are testing code that use print, you must be aware of its output writing test. For example:

This is a fake doctest.
Let's say that calling the button below will run some code that use Python "print" command.
For example for write "Hello".

    >>> browser.getControl('button').click()
    'Hello'

I don't like this... this is not a useful test!

This is very annoying!

So? It's better to not use print for logging messages, but rely on other logging APIs.