Personal tools

Andrew Mleczko

Mar 17, 2011

Munin timeout

Filed Under:

I had recently problems with one of munin plugins - it disappear from munin graphs. In my munin-node.log I have a lot of:

[21134] Service 'temperature' timed out.


How to change munin timeout value? There is nothing about it in munin documentation. But if you look into source code you will find that it exists. Adding:

timeout 60


to munin-node.conf or your plugin specific configuration does the tick!

Feb 10, 2011

Pyramid CRUD

I have been using Pyramid/BFG in several our projects. It really rocks. Probably you already know that. What I think could be an extremely useful add-on is a CRUD. For our internal usage (more as a proof-of-concept) we have developed one - traversal with SQLAlchemy support. Now we want to make it more generic and open-source. Interested in?

What we have is a working version of a traversal CRUD. General concept is similar to Sergey Volobuev Kelpie. Everything is based on repoze.bfg 1.2 (though it should work smoothly on 1.3).

Model definition is done using SQLAlchemy declarative approach.  This choice provoke us to use  FormAlchemy as a form generator (with fa.jquery widget support). It's easy to start with but difficult to extend later.  That's why it's the first thing we would like to change in the future.  Everything is glued together with bfg hybrid traversal approach

What we are planing to do (probably not exactly in that order):

  • check if somebody else didn't start similar project and maybe share concepts or do it together
  • check possibility of integrate existing solutions with Pyramid
  • reuse what can be reused from our bfg crud and move it to Pyramid
  • organize a python sprint that will work on these topics
 
CRUD section view:
 CRUD - section view
 
CRUD item view:
CRUD - item view
 
CRUD item edit:
CRUD - item edit
 

Jul 20, 2010

Aggregate zope munin graphs

Filed Under:

Munin with munin.zope is a handy tool if you want to monitor your Zope instance. But it starts to be annoying when you have too many zeoclients and too many projects on one server. Using munin aggregate functionality you can create nice, human readable graphs reusing your existing data.

With the newest version of munin.zope you have 4 different plugins:

  • ZServer threads
  • ZODB activity
  • Zope cache parameters
  • Zope memory usage

When you start using it in production environment you can end like this:

 

Munin
 

Using aggregation you can end like this:

Munin aggregation

What it does? It takes data from multiple zeoclients (in this case every project from A-E have 4 zeoclients) and renders only total amount per project.
Here is munin.conf which does this trick:

[Server1;projectA]
  address 127.0.0.1
[Server1;projectB]
  address 127.0.0.2
[Server1;projectC]
  address 127.0.0.3
[Server1;projectD]
  address 127.0.0.4
[Server1;projectE]
  address 127.0.0.5

[Server1;Aggregated]
  update no
  total_memory.update no
  total_memory.graph_category Zope
  total_memory.graph_title Aggregated Zope memory
  total_memory.graph_order \
    projectA \
    projectB \
    projectC \
    projectD \
    projectE

  total_memory.projectA.sum \
    projectA:projecta_zopememory_instance_1.VmSize \
    projectA:projecta_zopememory_instance_2.VmSize \
    projectA:projecta_zopememory_instance_3.VmSize \
    projectA:projecta_zopememory_instance_4.VmSize
  total_memory.projectA.label project A

  total_memory.projectB.sum \
    projectB:projectb_zopememory_instance_1.VmSize \
    projectB:projectb_zopememory_instance_2.VmSize \
    projectB:projectb_zopememory_instance_3.VmSize \
    projectB:projectb_zopememory_instance_4.VmSize
  total_memory.projectB.label project B

  total_memory.projectC.sum \
    projectC:projectc_zopememory_instance_1.VmSize \
    projectC:projectc_zopememory_instance_2.VmSize \
    projectC:projectc_zopememory_instance_3.VmSize \
    projectC:projectc_zopememory_instance_4.VmSize
  total_memory.projectC.label project C

  total_memory.projectD.sum \
    projectD:projectd_zopememory_instance_1.VmSize \
    projectD:projectd_zopememory_instance_2.VmSize \
    projectD:projectd_zopememory_instance_3.VmSize \
    projectD:projectd_zopememory_instance_4.VmSize
  total_memory.projectD.label project D

  total_memory.projectE.sum \
    projectE:projecte_zopememory_instance_1.VmSize \
    projectE:projecte_zopememory_instance_2.VmSize \
    projectE:projecte_zopememory_instance_3.VmSize \
    projectE:projecte_zopememory_instance_4.VmSize
  total_memory.projectE.label project E

 

For more information please check:
http://munin-monitoring.org/wiki/aggregate_examples
http://munin-monitoring.org/wiki/PercentGraphHowto
http://munin-monitoring.org/wiki/stack_examples

Jun 09, 2010

Expensive security check in constraintypes

Filed Under:

If you are using Plone 3.3 or older you are probably suffering an expensive security check when creating an Archetype. It can be EASILY fix.

In one of our projects we are still using Plone 3.1.x. with a pack of customer specific content_types and some additional PAS plugins (Lotus). I have noticed recently that calling invokeFactory become VERY slow. After debugging we have found the problem - getDefaultAddableTypes - method that is provided by constraintypes.py (ATContentTypes package). Original implementation is more then 4 years old. What it does - it checks isConstructionAllowed for every content_type register in portal_types. If you have registered them more then 40 ;-) ...

In most of the cases folderish types are filtering allowed_types. Thanks to Vincent we have now a nice fix:

Modified lib/constraintypes.py:getDefaultAddableTypes method to check isConstructionAllowed only for allowed types, not for all content types in portal_types. isConstructionAllowed was called twice for each types.

This fix is in ATContentTypes version 1.3 and started to be used by Plone 3.3.1. For us it was a good argument to upgrade!

Jun 01, 2010

Sorrento Sprint Summary - collective.amberjack progress report

RedTurtle is participating each year in Sorrento's Plone sprints. We were there also last week. It was an amazing time of brainstorming and coding. I will try to make a summary of what we have archived.

Symposium

We have started collective.amberjack presence in Sorrento with Massimo's presentation at the European Plone Symposium

 


Sprint

Then we had two sprinting days. We have started day one with brainstorming. We had three objectives/questions:

  • collective.amberjack sandbox
  • refactoring the code - how to simplify collective.amberjack tour definition and registration
  • translations - how to manage tour/steps translations

thanks to a fruitful discussion (garbas, dukebody, gborelli, miziodel, shywolf9982, sorry if I forgot your name ;-) we have right now answers to all of our dilemmas.


SanDbox

First idea of having a sandbox for collective.amberjack came up at last Plone Conference in Budapest. We have collected a lot of ideas for different use cases. We have had also some internal brainstorms in RedTurtle about it. Finally after confronting all the concepts - during Sorrento brainstorming - we will have a simple solution which should satisfy most of the users:

  • simple Plone site deployment (like demo.plone.org)
  • open registration for all users
  • collective.amberjack preinstalled and ready to use
  • user will try-out Plone using amberjack in his member folder
  • nightly restart with fresh Data.fs


Code refactoring

We have started refactoring collective.amberjack. There are 3 areas of interests: javascript, tour definition and tour registration. Luca was doing a great job trying to simplify javascript part. He also prepare a good startup for future enhancements. I was refactoring tour part. We have decided to use a 

configuration baseD tour definition,

which looks like that:

[amberjack]
steps = 
        0_create-a-new-folder
        1_fill-out-the-fields
        2_publish-the-folder
        3_all-done
title = Add and publish a Folder

[0_create-a-new-folder]
blueprint = collective.amberjack.blueprints.step
title = Create a new folder
url = /
text = Folders are one of ....
validators = 
        python: isManager
        python: isNotFolderCreated
microsteps = 
        0_0_microstep
        0_1_microstep
        0_2_microstep

[0_0_microstep]
blueprint = collective.amberjack.blueprints.microstep
description = If you don't want to perform the ...
automatically done by your browser.

[0_1_microstep]
blueprint = collective.amberjack.blueprints.microstep
idstep = menu_add-new
description = Click the [Add new...] drop-down menu.

[0_2_microstep]
blueprint = collective.amberjack.blueprints.microstep
idstep = new_folder
description = Select [Folder] from the menu.

This concept is well known across the community. I hope it will be also more human readable comparing to python dictionary ;-). We have a working version in the trunk. For new tours we have also 

new tour registration 

- comparing to old approach (registration only using ZCML) you can now use also TTW registration. Other important change is that tour doesn't need to be a python package any more. Because it's a simple .cfg file right now you can simply zip it into an archive (zip, tar) and register it. We have a working implementation for web source and local file system registration (which you can find in the trunk). Archive file can be shipped with multiple tours and with


translationS

which right now are still just a concept that need to be implemented. We would like to make a use of i18ndude tool but because it was designed for ZPT this could be a wrong approach. Overall idea is to keep all the files (tour and the translations) together, something like:

- mytours.zip
  |- tour1.cfg
  |- tour1_en.po
  |- tour1_it.po
  |- tour1.pot
  |- tour2.cfg
  |- tour2_en.po
  ...

 

Fixing bugs

Mirco and Federica were working together and did an enormous job fixing most of the 1.1a release bugs. Thanks to Rob we have solved also some TinyMCE problems that we have had. We have still some open issues, like 'new portlet creation' tour that need some attention so we still need your help!


What's next?

The current development focus is on 1.1 release, which includes:

  • new tour definition (python dictionary based tour will still work until 1.2)
  • new tour registration (old registration will still work until 1.2)
  • finish translation implementation
  • fix last bugs
  • deploy sandbox probably on demo.plone.org
  • collective.amberjack.windmill - which should become default interface for creating and editing tours; we have right now something more then just proof of concept: