Sunday, February 01, 2009

OpenERP running on Java (Jython) - ROUND 1

Hi folks,

Looking for an open source ERP that doesn't suck? Well there is at least one: OpenERP. Like it or not it's Python based. For most of the things this is a perfect fit, but let's face it, Python has less enterprise acceptance and the lower layers of that ERP could also have been powered by a powerfull virtual machine. And since building a new ERP can't really take less than 3 years, the situation will hardly evolve any time soon, so we better deal with it: I bet my money OpenERP will now take the market by storm, I don't see anything stopping it to take the lead, at least for the next 5 years.

So I wanted to show the world that it was possible to back OpenERP by the Java platform, much like it's now being admitted that Java is the best bed for Ruby on Rails applications thanks to JRuby. I built a proof of concept. Now I hope some get the message and join efforts to make it happen. I'll hardly finish that journey in my spare time, but hopefully, some people start to release how important this is and how close we are and help covering the extra miles to get it in production running on Java. Of course, OpenERP doesn't need to run Java to be the best ERP already, it is just that it could be even better. And if you want to help out, I'll be there too.

what about the claimed Java based ERP's?

I've been myself a Java programmer since around 2003, even created my first open source project on Sourceforge back in 2004 (EmSim http://sourceforge.net/projects/emsim/), later on I used those skills in a variety of advanced projects when working at Amadeus and Smile.fr. But, like it or not, after deep investigation, my conclusion is that the best open source ERP was coded in Python. I've looked deep inside Java based or claimed ERP's such as Ofbiz, Neogia, JFire, Compiere, Openbravo, Adempiere but those were too limited in my opinion. Basically, the trouble is that those 'Java' based ERP don't rely on true object oriented modeling since the ground up, some of them being even more SQL based then really Java based. I could explain in deep details why, but this is beyond the scope of this post.

So my conclusion was that even if I needed to learn Python, I would get the work done faster with OpenERP than with any of those existing Java based ERP's. My experiments proved me that I was right. With OpenERP, extending the relational model, the forms and making it fit to my needs was faster than with any of the other tried ERP's, even while being a Python noob. One year later, with a few successful implementations behind me, I'm only recommending it more than ever.


But wouldn't it be great to port OpenERP on the Java platform?

The benefits of porting OpenERP on Java are beyond the scope of this post. Still, here is a short list:

  1. Noticeable speed up. Even if the Python layer peaks up at 40% of the CPU load under high concurrent loads, getting twice as fast would be noticeable. If Jython get as optimized as JRuby which is 2 to 5 times faster than C Ruby, we will go there.
  2. Real multithreading and less memory use under huge loads (CPython scales using process and sharing nothing accross runtimes because it has a Giant Lock in its interpreter, much like PHP or Ruby). The expected benefit there is much like what benchmarks prove for heavily loaded JRuby on Rails applications.
  3. One click installation on lots of hardwares (imagine how easy it will be to sale to large companies when you will say it's only a .war you pu on your Tomcat server).
  4. Penetration in large companies where Java tends to rule more than Python
  5. Easier and solid cross database support thanks to the unified JDBC API.
  6. Larger community for OpenERP. OpenERP is certainly the open source ERP with the largest active open source community. But just imagine how large it could get when you managed to connect with the Java community, telling them: hey, we are somewhat Java based too. If you know Java you'll be able to leverage those skills to get the job done in OpenERP. Again, JRuby proved here that connecting communities was possible.
  7. Possibly larger fundings and exposure for OpenERP. Openbravo certainly had the large fundings it had by playing on its Java image which as a broad acceptance in the enterprise world.
  8. And the best of all: cross language implementation: by leveraging Jython, you would be able to call the best existing Java lib (there are some good ones like JDBC, JCR, JMX, JMS, ESB's, SOAP...), but also externalize some code pieces where speed is critical to Java, meaning an easier way to reach C speed (a warmed up Java6 runs at C speed and sometimes faster) than writing C bindings.
  9. Jython will probably ends up implementing the MOP (Meto Object Protocol) that makes it possible to share OBJECTS between JVM based languages such as JRuby, Javascript, Scala, Clojures, Groovy... Meaning that more people could adapt OpenERP to meet their needs using the language they already master. That means you could code OpenERP modules in Ruby for instance and that's not a small thing, while preserving backward compatibility. Of course, published modules would have to keep beeing closely controlled, but at least one shot veticalization modules could offer more options.
  10. Because it's fun and we can do it.

What does it take?
Well, Jython is really alive again. Notice that Microsoft hired the orginial Jython creator to create Iron Python on the .Net platform. Next, by 2008, Sun Microsystem hired the two Jython leads, Ted Leung and Frank Wierzbicki to make it Python a first class citizen on the Java platform again. Espcially, JRuby recently proved that the JVM was able to run dynamic languages faster than their native C interpreter that don't come up with such a sophisticated virtual machine. So in theory the future is bright. A 2.5 Python compliant Jython is expected by February and the current trunk largely reflecting it already.

Still, the trouble, comes with native libraries. Since Python tends to be somewhat slow, Pythonistas tend to back lot's of Python libraries by C extensions to provide extra speed. The trouble is that the Jython interpreter, sandboxed in its virtual machine environment can't invoke the same C extension, mostly because it doesn't work the same at the lowest levels.

In some places, OpenERP uses such libraries. Some of them are psycopg2, mx.DateTime, libxml2, libxslt. Overall there aren't too many fortunately so that's why this shouldn't be too hard to have a full blown OpenERP running on the JVM. The common strategy here is to set up a wrapper over existing Java libraries in place of those C extensions. Also notice that in the near future, Jython might also support CTypes, a standard way of building C extensions for Python. This is largely because Jython is benefiting here from the fantastic JRuby work of Wayne Meissner around JFFI (Java Foreign Fuction Interface). Other synergies exist, like the Da Vinci Machine, or Invoke Dynamic bytecode instruction that would help the JVM speeding up dynamic languages while optimizing memory usage.



Current status:

After some two days of heavy hacking, I'm getting:

  • OpenERP server starting powered by the last Jython Java based interpreter, backed by the regular JDBC Postgres driver over a standard Java JNDI database connection pooling, instead of Pyscopg2.
  • It answers most of the webservices calls, be it from the clients or from the webservice API. The largest thing that isn't working here is datetime operation that are using the mx.DataTime library that isn't yet fully supported.
  • Also, views don't work yet (the fields_view_get method). This is due the current XML and XPath limited support. Everything need is available on the Java plateform, but it should be wrapped properly to fake the Python API OpenERP is expecting there.



Next stages:

  • I guess that faking the required xpath and xml API would be great. That shouldn't be that hard given that OpenERP only use a few things in those API. We should only wrapp them over Java libraries such as Xalan.
  • As for the mx.DateTime trouble, the OpenERP team said they are ready to move away from it. Indeed mx.DateTime has a bug with dates before 1970. That's why the Tryton fork moved away form it, proving the thing is possible (they only use it marginaly for parsing, but alternative can be found). Advised by Jim Baker from Jython, it appears that the standard python datetime, datetime.timedelta and the http://labix.org/python-dateutil library might be used instead. So aside from helping getting rid of mx.DateTime, we probably won't have much work here.
  • Once all that is working, the Java based OpenERP server will already be in pretty good shape to be used in production.
  • Of course, ideally we would package the whole thing within a standard J2EE war package and serve it with a standard servlet container such as Tomcat or Glassfish. The idea there is to write a servlet connector that will use standard Java servlet instead of the python SimpleXMLRPCServer. Then we would route HTTP request to a pool of Jython runtimes and the appropriate OpenERP server layers, much like others do with Django on Jython or the Glassfish JRuby on Rails gem supported by Sun Microsystem.
  • Once performance starts to proves better than CPython (Jython and JRuby guys are expecting this to happen by 2009 already) and once large companies show interrest in deploying OpenERP over their existing Java stack, may be the OpenERP team starts to support it officially and provide alternative Java based layers to increase performance even more or provide extra features (like SOAP, JMS async messaging, ESB like Mule or ServiceMix, a REST layer, JMX remote monitoring, OpenTerracotta clustering...)



Steps to follow to test the current Jython powered version:


Requirements:

  1. you should know how to start a regular OpenERP v5 server
  2. you should have an existing OpenERP v5 database to test again
  3. you should have Java installed (1.6 advised)


First you have to grab a recent Jython interpreter. Make sure you have a recent version of Java installed (1.6 advised) and then grab Jython:
get a fresh version of Jython, and build it:


$ svn co https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython/ jython-dev
$ cd jython-dev
$ ant

now, put the jython-dev/dist/bin/jython command in your path.
now if you type

$ jython

that should bring you a Python 2.5, java based commande line interpreter.

Now grab my public branch of the OpenERP server:

$ bzr branch lp:~rvalyi/openobject-server/trunk-java-jython


As for the addons, you can use your regular OpenERP v5 addons, or grab them from here:

$ bzr branch lp:openobject-addons


Now open the trunk-java-jython/bin/tools/config.py and make sure you properly set up the database connection params and you addons path location.
May be it's better to leave the db_name to False so it will load all the features lazily when requested.


you can already see what happens if you:

$ cd trunk-java-jython/bin
$ jython openerp-server.py


You'll probably have errors because we didn't properly set up some required libraries.
So it's time to copy paste a few libraries from our regular Python path to our Jython path:
you can copy/paste the following directories:
pychart, pytz, reportlab
from
/user/lib/python2.5/site-packages/ (or whatever you Python sys.path)
to [...]/jython-dev/dist/Lib

There is one more catch though. Currently, there is bug in Jython preventing the reportlab lib to load fully. This is because font listings in the pdfbase/_fondata.py are just too large for the JVM spec to fit in a single Java method. The Jython folks are aware of that bug but it's low priority: see http://bugs.jython.org/issue527524
Meanwhile, helped by the fellow Charles Headius Nutter from the JRuby Sun team who faced similar issues implementing JRuby, I patched the file to make reportlab works in Jython. It's only a wrapping of large list instantiations and doesn't remove any feature. I submitted it to the reportlab developers and hope to see it included waiting for a Jython fix.
So in any case, for now, you should just move the _fontdata.py file provided in the trunk-java-jython branch and copy it in place of the reportlab/pdfbase/_fontdata.py file of your Jython reportlab installation.

Now you should be able to start you OpenERP server on Jython. Look at the following logs on my machine. Then I connected using XML/RPC with my GTK client (in that development version I'm printing all the requests, you could remove those print of course):

rvalyi@rvalyi-laptop:~/DEV/openobject_trunk/openobject-server-jython/bin$ jython openerp-server.py
[lun. févr. 02 2009 02:41:22] INFO:server:version - 5.0.0
[lun. févr. 02 2009 02:41:22] INFO:server:addons_path - /home/rvalyi/DEV/openobject_trunk/openobject-addons
[lun. févr. 02 2009 02:41:22] INFO:server:database hostname - localhost
[lun. févr. 02 2009 02:41:22] INFO:server:database port - 5432
[lun. févr. 02 2009 02:41:22] INFO:server:database user - openerp
[lun. févr. 02 2009 02:41:22] INFO:objects:initialising distributed objects services
WARNING; Python Imaging not installed, you can use only .JPG pictures !
[lun. févr. 02 2009 02:41:32] INFO:web-services:starting XML-RPC services, port 8069
[lun. févr. 02 2009 02:41:32] INFO:web-services:starting NET-RPC service, port 8070
[lun. févr. 02 2009 02:41:32] INFO:web-services:the server is running, waiting for connections...
list
()
-------------
[lun. févr. 02 2009 02:41:40] INFO:dbpool:Connecting to template1
executing the following DB query:
select datname from pg_database where datdba=(select usesysid from pg_user where usename='openerp') and datname not in ('template0', 'template1', 'postgres') order by datname
query passed!
[lun. févr. 02 2009 02:41:40] INFO:dbpool:Closing all connections to template1
['jython', 'openerp']
server_version
()
-------------
5.0.0



So the GTK client can connect to the Jython powered server and list the available databases for instance.
But if I try to really connect to the database and go further here I get an error unfortunately:

[...]
SELECT ir_act_window_group_rel.gid,ir_act_window_group_rel.act_id FROM ir_act_window_group_rel , res_groups WHERE ir_act_window_group_rel.act_id in (1) AND ir_act_window_group_rel.gid = res_groups.id order by res_groups.name offset '0'
query passed!
{'view_id': (1, u'ir.ui.menu.tree'), 'src_model': False, 'view_ids': [], 'context': u'{}', 'view_type': u'tree', 'auto_refresh': 0, 'usage': u'menu', 'res_model': u'ir.ui.menu', 'domain': u"[('parent_id', '=', False)]", 'name': u'Menu', 'id': 1, 'view_mode': u'tree,form', 'target': u'current', 'views': [(1, u'tree'), (False, u'form')], 'limit': 80, 'type': u'ir.actions.act_window', 'groups_id': []}
execute
('jython', 1, 'admin', 'ir.ui.view', 'read', [1], ['model', 'type'], {'tz': False, 'active_ids': [], 'lang': 'en_US', 'active_id': False})
-------------
executing the following DB query:
SELECT "model","type",id FROM "ir_ui_view" WHERE id IN (1) ORDER BY priority
query passed!
[{'id': 1, 'model': u'ir.ui.menu', 'type': u'tree'}]
execute
('jython', 1, 'admin', 'ir.ui.menu', 'fields_view_get', 1, 'tree', {'tz': False, 'active_ids': [], 'lang': 'en_US', 'active_id': False})
-------------
executing the following DB query:
SELECT arch,name,field_parent,id,type,inherit_id FROM ir_ui_view WHERE id='1' and model='ir.ui.menu'
query passed!
executing the following DB query:
select arch,id from ir_ui_view where inherit_id='1' and model='ir.ui.menu' order by priority
query passed!
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[01]: Traceback (most recent call last):
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[02]: File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 60, in wrapper
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[03]: except orm.except_orm, inst:
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[04]: File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 120, in execute
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[05]: cr.commit()
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[06]: File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/osv.py", line 112, in execute_cr
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[07]: File "/home/rvalyi/DEV/openobject_trunk/openobject-server-jython/bin/osv/orm.py", line 1082, in fields_view_get
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[08]: doc = dom.minidom.parseString(encode(result['arch']))
[lun. févr. 02 2009 02:44:30] ERROR:web-services:[09]: NameError: global name 'dom' is not defined




So yeah, not everything is working yet, and especially the view layer don't work yet because we didn't provide appropriate wrappers for the xml/dom and xml/xpath Python modules used by OpenERP.

The same goes withe the web-client (eTiny), it can list the databases but won't show any view properly yet.

Still, more something interesting is that webservices, which only care about the model and controller layers are already working quite well. See yourself, in an standard python or jython console, I can for instance read all the info related to a given product (here with id 1):

>>> sock = xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/object')
>>> sock.execute("my_test_base", 1, "admin", 'product.product', 'read', [1])
[{'warranty': False, 'property_stock_procurement': [5, 'Procurements'], 'supply_method': 'buy', 'code': False, 'list_price': 38.25, 'expected_margin_rate': 0.0, 'pricelist_purchase': 'Default Purchase Pricelist (0.00) : 25.50\n', 'incoming_qty': 0.0, 'weight_net': False, 'standard_price': 25.5, 'cost_method': 'standard', 'active': True, 'price_extra': 0.0, 'mes_type': 'fixed', 'uom_id': [1, 'PCE'], 'uos_id': False, 'ean13': False, 'default_code': False, 'type': 'service', 'property_account_income': False, 'qty_available': 0.0, 'sales_gap': 0.0, 'id': 1, 'expected_margin': 0.0, 'uos_coeff': 1.0, 'virtual_available': 0.0, 'seller_delay': 1, 'total_cost': 0.0, 'purchase_ok': True, 'date_from': '2009-01-01', 'property_stock_account_output': False, 'track_outgoing': False, 'company_id': [1, 'Espace Loggia'], 'product_tmpl_id': [1, 'Onsite Senior Intervention'], 'state': False, 'loc_rack': False, 'pricelist_sale': 'Public Pricelist (0.00) : 38.25\n', 'uom_po_id': [1, 'PCE'], 'price_margin': 1.0, 'price': 0.0, 'property_stock_inventory': [4, 'Inventory loss'], 'loc_case': False, 'sale_avg_price': 0.0, 'description': False, 'track_incoming': False, 'property_stock_production': [6, 'Production'], 'purchase_avg_price': 0.0, 'weight': False, 'supplier_taxes_id': [], 'volume': False, 'normal_cost': 0.0, 'outgoing_qty': 0.0, 'dimension_type_ids': False, 'date_to': '2009-12-31', 'procure_method': 'make_to_stock', 'sale_num_invoiced': 0.0, 'variants': '', 'partner_ref': 'Onsite Senior Intervention', 'loc_row': False, 'purchase_num_invoiced': 0.0, 'sale_ok': True, 'rental': False, 'packaging': [], 'sale_delay': 7.0, 'name': 'Onsite Senior Intervention', 'total_margin_rate': 0.0, 'description_sale': False, 'property_account_expense': False, 'categ_id': [8, 'All products / Sellable / Services / Onsite Intervention'], 'invoice_state': 'open_paid', 'property_stock_account_input': False, 'track_production': False, 'sale_expected': 0.0, 'lst_price': 38.25, 'taxes_id': [], 'dimension_value_ids': [], 'produce_delay': 1.0, 'seller_ids': [], 'description_purchase': False, 'turnover': 0.0, 'purchase_gap': 0.0, 'product_manager': False, 'total_margin': 0.0}]




OK, enough said for now. I'll try to post updates on the dedicated Launchpad blueprint: https://blueprints.launchpad.net/openobject-server/+spec/jython-support-as-jython-improves
Don't hesitate to contact me if you want to push this work further.

Raphaël Valyi.

14 comments:

Anonymous said...

Good job !

Anonymous said...

Hey ! Wake up & go on 4 ROUND 2 :-)

chanon said...

Good job, I was searching a project like this one... as I think OpenERP is pretty good and I am Java minded...

Anonymous said...

Excellent job!

It would be great to see this effort duplicated on the tryton project.

As a java developer, I found the openerp code to be very messy and difficult to understand. On the other hand, the tryton refactored code is much cleaner and nicer to work with...

Raphaël Valyi said...

@Anonymous,

you are right, the Tryton fork of OpenERP putted the accent on code quality and overall it's probably a bit better. Tryton is certainly the second open source ERP I would recommend to work with, but that's after OpenERP.

Code quality is one thing, OpenERP has way more features (saw v5?) plus it now have some 125+ parallel branches on Launchpad, meaning that the momentum is just huge and code quality is likely to improve and even probably pass Tryton on this point too. I think OpenERP is mainly missing docstrings and I hope those will be added. As for understanding the doc, OpenERP also has a lot of free dev doc, see yourself:
http://doc.openerp.com/developer/index.html#book-develop-link

In any case, I wish the too project could mutualize the effort their. Porting OpenERP or Tryton will be pretty much equivalent, once it's done for one it will be a piece of cake to port the other.


Ok, folks, I had no time to come up with something very finished, but continued pushing the idea forward. Now it seems that it will be pretty easy to actually get ROUND 2 done: OpenERP running is a standard Java server (Tomcat or Glassfish for instance), using the standard Modjy Jython servlet wrapper project.

Stay tuned.

Hendy Irawan said...

Excellent!

Indeed this is really big.. Hopefully Jython is improving steadily and also more cooperation from OpenERP.

I've to agree OpenERP is the most promising open source ERP out there.

Will it take SAP and Oracle markets?

GEM said...

Hello Raphael, it's GEM (openerp forum), does your server solution is portable for windows without use vitualisation ?
thanks for the answer.
A+

Anonymous said...

you have nice site. thanks for sharing this site. various kinds of ebooks are available here

http://feboook.blogspot.com

Anonymous said...

[url=http://www.blingforfun.com]hip hop jewelry[/url],[url=http://www.blingforfun.com/pendants/cat_9.html]hip hop pendants[/url],hip hop watches,[url=http://blingforfun.com/belts/cat_18.html]bling bling[/url] ,hip hop,[url=http://blingforfun.com/chains/cat_7.html]hip hop chains[/url],hip hop bling,[url=http://blingforfun.com/chains/cat_7.html]iced out chains[/url],[url=http://www.blingforfun.com/chains/cat_7.html]wholesale chains[/url]
hip hop jewelry
wholesale hip hop watches
hip hop rings

Anonymous said...

[url=http://tinyurl.com/getvpn][b]Click here to get VPN service![/b][/url]

[b]Anonymous surfing[/b]
Using our service you'll be fully anonymous in the Internet. Hide your IP address, and nobody will know that strange visitor from Germany ( Great Britain, Estonia and so ), is you.

[b]Full access to network[/b]
You can use any services, access any sites and use any software with us. BitTorrent, Skype, Facebook, MySpace, Twitter, Pocker .. we have no restrictions.

[b]Traffic protection[/b]
Don't worry, from this moment all you data will be protected using 256 bit Blowfish encryption algorithm. Nobody can access your internet data.

[b]Wide variety of countries[/b]
You can choose one of over twenty high speed servers located in different parts of the world, from South America coast to islands in Indian Ocean.

Related keywords:
anonymous surfing review
proxy server vpn
anonymous secure surfing
proxy vpn
anonymous vpn free
internet explorer vpn
vpn dial up
ssl vpn
Traffic protection
anonymous surfing freeware
anonymous surfing software
vtunnel
anonymous surfing vpn
best anonymous browser
surf the web anonymous
best anonymous surfing
anonymizer anonymous surfing review
firefox anonymous surfing
Virtual Private Networks
Free Vpn Client Software
anonymous surfing software
[url=http://dasbmw.ru] anonymous surfing software[/url]
[url=http://seobraincenter.ru] anonymous proxy[/url]
[url=http://carlwebster.com/members/Alexander-Pwnz.aspx]Buy Cheap Zoloft[/url]

sean said...

Is there a java library for talking to openerp over xmlrpc? There are some initial examples in the developer book, but what about a complete class? Can Jython provide this?

Anonymous said...

Great blog! Continue the good posts.

Anonymous said...

is their any ready framework that equal to openerp or tryton ???????????

sap upgrade automation said...

There are many opensource ERP and choosing the best one depends on anyones needs as same ERP can not suit for all requirements.