Mitch Pronschinske is a Senior Content Analyst at DZone. That means he writes and searches for the finest developer content in the land so that you don't have to. He often eats peanut butter and bananas, likes to make his own ringtones, enjoys card and board games, and is married to an underwear model. Mitch is a DZone Zone Leader and has posted 2573 posts at DZone. You can read more from them at their website. View Full User Profile

Impala: An Alternative Module Framework

12.21.2009
| 15338 views |
  • submit to reddit
Modular programming is a growing trend in application development.  OSGi is at the center of the modularity movement, however, it is a complex system that requires extra development overhead.  Phil Zoio wanted a less complex module framework that didn't depend on any third-party runtime environments, so he made his own.  He created Impala, a dynamic module framework for Java-based web applications based on the Spring Framework.  DZone recently spoke with Zoio in an exclusive interview about the current state of Impala and the inspiration behind it.

DZone: What is Impala?  What inspired you to create it?


Phil Zoio: I started Impala to address many of the practical problems I encountered when working on a large Spring-based project in around 2006 and 2007. The original developers on the project had attempted to structure the project in a modular way. However, because there wasn't any underlying runtime support for modularity, the attempt was a failure. Without any ability to partially deploy parts of the application in isolation, deploying each change required running a full multi-project build. The result was a slow build, deploy and test cycle, and poor productivity. Also, lack of modularity made it difficult to organize Spring configurations, and resulted in an unnecessarily high level of coupling between projects.

I realized that the solution was not to abandon modularity, but to leverage the technologies already at my disposal - Spring and class loaders - to create some simple mechanisms for supporting modularity at runtime. I considered using OSGi, but did not believe at the time that I would be able to achieve the same level of productivity using OSGi as I have been able to, using the approach I have taken. As of yet, I haven't been given any cause to regret this decision.

The result of subsequent efforts is Impala, a fully dynamic environment for creating and running modular Spring applications. Applications can be broken down in a way which supports and promotes loose coupling, with class loaders enforcing boundaries between modules. These modules can be added, removed and updated on the fly, even without any user intervention. This leads to a very smooth and enjoyable developer experience. Impala also makes it much simpler to write integration tests, because it allows test dependencies to expressed as a collection of modules rather than named Spring configuration files, and allows integration tests to be written in an interactive way.

DZone:  How does Impala fit into the ecosystem of related technologies?
 
Phil:  If you're used to Spring you'll know almost everything you need to know to use Impala. Impala uses Spring to configure the Impala runtime itself - it does not introduce any dependencies that wouldn't be present in the closest equivalent single module Spring application. So the easiest way to think of Impala is providing a simple route to dynamic, modular Spring applications. That being said, in its representation of modules Impala has an architectural layering which would allow for its use with other dependency injection containers.

Impala works for standalone applications as well as web applications running on standard JEE web containers. For web applications, you can of course use Spring MVC, but (with a very small amount of extra effort in some cases) Impala also works other web frameworks, such as Struts (1 and 2), Wicket, Tapestry, JSF, etc.

DZone: How does someone start using Impala?  


Phil:  For day-to-day development on an existing project, you would simply check out a set of Impala projects into an Eclipse workspace, then run up your application or tests. A vanilla Eclipse installation can be used - no additional plugins are required beyond the basic Java tooling.

To kickstart a new project, retrieve third party library dependencies, or build your project, the easiest thing to do right now is to download the Impala distribution, and use Impala's built-in ANT-based build system. This allows you to build a WAR or standalone Jetty-based distribution, run your automated test suites, and perform other tasks.

Support for Maven will be available in the next release milestone, which will available in the next few days.

DZone:  When people think of modularity, they mainly think of OSGi.  Why do you think that is?
 
Phil:  OSGi is a complete from the ground up Java modularity solution which has been around now for around 10 years, so it has had plenty of time to establish itself. The growing importance of modularity in Enterprise Java is a more recent phenomenon. As an established standard widely supported by vendors in the Java community, OSGi was well placed to move into this space as a default modularity choice.

That being said, because of a long separate evolution, bringing OSGi into Enterprise Java has (and continues to be) quite a challenge.

Impala is often positioned as an alternative to OSGi. While this is certainly true, at another level the situation is more complicated. Impala is not fundamentally incompatible with OSGi. There is even a sample of Impala running on OSGi, so full blown support of OSGi as an alternative deployment runtime for Impala applications is a future possibility.

DZone: How does Impala compare to OSGi?  What other technologies can be compared to Impala?  

 
Phil: OSGi on its own is a lower level technology than Impala, so the relevant comparison is really with OSGi in the broader sense, that is, OSGi plus an additional declarative services layer, provided for example by Spring Dynamic Modules.

Impala has evolved to become conceptually quite similar to OSGi, especially in the way that the service registry is used. The big difference is in the way that third party libraries are handled. This is very much part of the scope of OSGi, but not of Impala. Impala is about modularising an application, but not it's use of dependent jars.

The class loader model for Impala is relatively simple, designed to offer a practical solution to common issues. OSGi has a more sophisticated class loader model, capable of addressing pretty much any modularity issue. With Impala, you manage third party libraries in exactly the same way as you would for a regular application (for example, by placing third party libraries in the WEB-INF/lib directory). The advantage is that you also spared the overhead of having to provide fine grained package import and export declarations in your application. Of course, with Impala you don't get versioning and reloading of third party libraries within the application. The big question in view is whether very many applications really require this feature, or use it in anger.



Other differences are more practical than conceptual. Impala applications are easier to package. Testing is also much easier because with OSGi, integration tests need to be OSGi bundles running on an OSGi container. The Impala environment is simpler because of the absence of any third party runtime environment.

DZone: What is the current state of the Impala project?  

 
Phil: The most recent release is 1.0 RC1. Only minor changes are planned between now and the 1.0 final release, planned for early 2010. The most significant of these is to make sure that Impala works well with Maven.

DZone: Give me the project roadmap for Impala?  

Phil: It will definitely be an initial priority to make Impala more easily usable in a wider variety of common existing development environments, and also to make it easier to control and configure. An administration console which can be used to manage module configurations more easily will probably be introduced shortly after the 1.0 release. I'd like to see support for Impala in other IDEs (e.g. NetBeans, IntelliJ). An Eclipse plugin to create Impala projects, interactive with the WTP tooling and use the Impala interactive test runner would be great, too.

Looking ahead, there are also some very exciting but in some cases quite ambitious directions which the project could take in the coming months.

SpringSource are working on a project called Slices which is about making it possible to run typical Java/Spring web applications as separate modules (or slices) within a larger host application. Early versions run only on OSGi, but making this project work outside of OSGi in Impala is a very interesting possibility.
Support for modular Grails applications would be a great additional feature, bringing the modularity benefits of Impala together with the productivity benefits of Grails.

I'd like to add better support for multi-language applications, with different modules built in alternative languages (e.g. Scala or Groovy) according to their strengths, and sharing a common infrastructure.

First class support for OSGi would be another interesting option, which could be done in a number of ways. One is through adding support for Blueprint services specification. Having OSGi support would allow Impala to be used as a more gentle and accessible introduction to modularity, with migration onto OSGi as and when requirements of the project dictate.

There is no precise roadmap for Impala as priorities will depend on demands from users, as well as willingness for others apart from myself to come forward with contributions, so at this point it is impossible to put any precise timescale against any of these features.  Obviously, I would very much appreciate any help members of the developer community are able to provide in helping Impala to reach its potential.
_____
Phil Zoio is an independent developer, architect and consultant, and an active open source developer. In addition to creating Impala, he is the creator of Strecks, as set of Java 5 extensions for Struts. Phil has published articles in numerous leading journals and web sites.  He regularly speaks at Java user groups and conferences.

Comments

Christopher Brind replied on Mon, 2009/12/21 - 5:47am

" with Impala you don't get versioning and reloading of third party libraries within the application "

Ouch! How do you manage your 3rd party library dependencies then? e.g.

3rd party library X has a dependency on 3rd party library T version 1.0
3rd party library Y has a dependency on 3rd party library T version 1.4

(where the differences between 1.0 and 1.4 are that class definitions are slightly different, or even different behaviours within classes, e.g. Xerces)

Thanks,
Chris

Phil Zoio replied on Tue, 2009/12/22 - 4:43am in response to: Christopher Brind

Chris

You can ask exactly the same question about every Java application which does not use OSGi!

The question is whether the frequency of these types of problems occuring is sufficient to justify the extra cost of using OSGi. Based on my personal experience, I'm not convinced that it is, although perhaps if you have been bitten before you will disagree.

The point of Impala's approach is to bring the benefits of modularity where I feel it is needed most - in the application - without any extra pain, and to bring it with a productivity gain rather than a productivity drain.

Phil

Neil Bartlett replied on Tue, 2009/12/22 - 7:08am

"The question is whether the frequency of these types of problems occuring is sufficient to justify the extra cost of using OSGi."

I think this is the core fallacy in your entire argument. There is no extra cost of using OSGi, in fact in my experience it is cheaper than not using it (except for very small projects that neither need modularity nor need to be reused later). Given this, I see no compelling reason to use Impala, since you yourself admit that Impala is inferior to OSGi in all respects other than cost and productivity.

In fact I expect OSGi to be cheaper than using Impala, since you are still appear to be stuck using strictly hierarchical classloaders and Spring contexts. The problem with arranging modules in hierarchies is that, in order to share a module between two other modules (whether it is a third party library or what you call an "application" module), you need to push it up the hierarchy to make it into a common ancestor of both. This then *forces* all the other descendants of that module to use the classes in it rather than the ones they may prefer to use, because classloader delegation is always "parent first".

Incidentally, how do you decide what is a third party library and what is an application module? This distinction seems highly artificial and dubious to me. How would you classify JARs that come from the team down the corridor? In OSGi, everything is just a module, plain and simple.

Now, I understand that you have done some work to use Impala on top of OSGi, which should help to escape from the hierarchical modules problem. Can you explain what value Impala offers in that scenario? I.e. if you replace the module system in Impala, what is left? Please do not interpret this question as dismissive, I would genuinely like to know if Impala can be useful to me in my OSGi projects.

Finally, your statements about what can or cannot be done to test an OSGi application are misleading. For example, you stated that integration testing in OSGi requires a "third party environment", namely the OSGi Framework implementation... but why do you not count Impala as a "third party environment" for people who need to test their Impala applications? I do not see the difference, except that YOU wrote Impala and thus YOU do not consider Impala "third party"; unfortunately this is not the case for anybody else!

Christopher Brind replied on Tue, 2009/12/22 - 7:46am in response to: Phil Zoio

You can ask exactly the same question about every Java application which does not use OSGi!

Correct, so when someone comes along with a 'modularity' solution that does not address this core problem, I have to wonder why they've bothered.

In my experience (several years of OSGi, over a decade of Java), OSGi gives you a productivity boost and a comfort blanket from the start.

I recently finished an intensive 3 week project producing a platform for market data applications using OSGi and Flex (i.e. Flash front end). This included a preview of their first application on the platform as well. Two people, three weeks, and we produced a sophisticated rich internet application, nay *platform* that can be built on indefinitely. We decided it just would not have been possible if we hadn't used OSGi.

I suppose I could account for the lack of productivity drain from my point of view because of my experience, but the other chap didn't have that experience and literally whooped every time he found himself able to just get on with the job at hand.

It just seems like you had a quick brush up against OSGi, couldn't get on with it and then developed something that just isn't as good and doesn't appear to add any value (it just seems to be yet another wrapper around Spring). Maybe a few years ago the tools and availability of OBRs were a blocker for more widespread uptake, but (while there's still room for improvement) life is as easy or as difficult as you want to make it in the OSGi world these days.

Cheers,
Chris

Marcel Offermans replied on Tue, 2009/12/22 - 8:30am

I briefly wanted to comment on the architecture comparison you give. The Impala architecture from this picture seems nothing more than an OSGi framework where all third party libraries plus the extender bundle are made available through the system bundle. In OSGi terms that means you're defining your own execution environments for your components to run in.

Phil Zoio replied on Tue, 2009/12/22 - 10:33am in response to: Neil Bartlett

Neil, I'm not sure where the fallacy you are talking about comes in.

Regarding some of your specific points.

There is no extra cost of using OSGi, in fact in my experience it is cheaper than not using it (except for very small projects that neither need modularity nor need to be reused later).
The cost of using OSGi is adding declarations for every package which is imported and exported, and for having to maintain the OSGi manifest. Come on, even you can see that this involves work!
Given this, I see no compelling reason to use Impala, since you yourself admit that Impala is inferior to OSGi in all respects other than cost and productivity.

I don't believe that Impala is inferior to OSGi (your words not mine) - it is a different, and in my opinion, more productive solution to some of the problems solved by OSGi.

Since application frameworks are about lowering costs and raising productivity, I'm not sure what other criteria you have in mind.

In fact I expect OSGi to be cheaper than using Impala, since you are still appear to be stuck using strictly hierarchical classloaders and Spring contexts. The problem with arranging modules in hierarchies is that, in order to share a module between two other modules (whether it is a third party library or what you call an "application" module), you need to push it up the hierarchy to make it into a common ancestor of both. This then *forces* all the other descendants of that module to use the classes in it rather than the ones they may prefer to use, because classloader delegation is always "parent first".
Actually, Impala supports a graph of class loaders. The ability to support only hierarchical class loaders is a limitation which was present in early versions, but has not been a limitation for some time now. I don't suppose you've read the documentation or tried the project out recently?
Incidentally, how do you decide what is a third party library and what is an application module? This distinction seems highly artificial and dubious to me. How would you classify JARs that come from the team down the corridor? In OSGi, everything is just a module, plain and simple.

That for me is a problem with OSGi - I think there is a fundamental difference between application and library modules, which is a bit more blurred in the OSGi world. Third party libraries are typically (but not always) produced by other development teams, tend to change less often in terms of their use in your application, and be static (have no dynamic runtime components). Applications modules are created within the development team, change very frequently during the development of a project, and are used to configure the dynamic parts of your application.

I think it is intuitive and natural to think of third party libraries and application modules as different beasts, and to treat them differently. 

Now, I understand that you have done some work to use Impala on top of OSGi, which should help to escape from the hierarchical modules problem.
Ditto my previous earlier comment regarding graph-based modules.
Finally, your statements about what can or cannot be done to test an OSGi application are misleading...

My point here is that when you execute a test in OSGi you first need to bootstrap the OSGi container, which then needs to run the test in the OSGi environment. This makes writing tests in OSGi quite difficult in practice. Can I ask you, do you practice test driven development when writing applications using OSGi?

 

Christopher Brind replied on Tue, 2009/12/22 - 12:16pm

Can I ask you, do you practice test driven development when writing applications using OSGi?

Can't speak for Neil, but I do, yes - 100% (I presume Neil does too though) and this is easier than you might think.

I tend to write 'pojo' style components which are instantiated and have their dependencies injected by something else (usually declarative services, but there is a raft of options including iPojo, Peaberry, Spring and more). So for me this means I don't need an OSGi container when developing using TDD. It's just like developing for ant project that uses an IOC/DI approach.

With regards to managing imports and exports - use the right tools (I use Eclipse PDE) and this is extremely trivial - a negligible overhead for a powerful feature (package level versioning).

Neil Bartlett replied on Tue, 2009/12/22 - 12:30pm in response to: Phil Zoio

The cost of using OSGi is adding declarations for every package which is imported and exported, and for having to maintain the OSGi manifest. Come on, even you can see that this involves work!

Declaring the dependencies of a module is a development task, certainly, but it is a lot less work than the consequences of leaving dependencies undeclared and hidden inside the bytecode, ready to cause a NoClassDefFoundError or LinkageError some time later on. Also with those declarations in place and validated by the build tools and the runtime, I can use other tools to calculate the transitive requirements of a set of modules, provision them across a cloud, and so on. I could not imagine doing these kinds of tasks without the inclusion of some simple declarations somewhere, either in the manifest or elsewhere.

I don't believe that Impala is inferior to OSGi (your words not mine) - it is a different, and in my opinion, more productive solution to some of the problems solved by OSGi.

You did not use the term inferior, but you did state that OSGi does not suffer from some of the limitations that Impala does, especially in respect to what you term third-party modules. You then stated that these limitations are unimportant -- I disagree -- and that Impala is more "productive" -- I disagree again.

Actually, Impala supports a graph of class loaders. The ability to support only hierarchical class loaders is a limitation which was present in early versions, but has not been a limitation for some time now. I don't suppose you've read the documentation or tried the project out recently?

I admit that I had not; but in this case how can you claim that Impala is simpler or more compatible with legacy libraries than OSGi if you are doing essentially the same thing as OSGi?

99% of problems with libraries running under OSGi are caused by a small number of libraries that assume they live in a classloader hierarchy or that they have global visibility of all classes in the application -- though these problems are diminishing constantly, as library authors become more aware of the need to be good modular citizens. Unless you fudge modularity to provide global class visibility, Impala will suffer the same problems with the same libraries.

That for me is a problem with OSGi - I think there is a fundamental difference between application and library modules, which is a bit more blurred in the OSGi world. Third party libraries are typically (but not always) produced by other development teams, tend to change less often in terms of their use in your application, and be static (have no dynamic runtime components).

I think this is a rather old-fashioned view of software development, but nevertheless I accept that some developers may wish to work this way. As Marcel pointed out in his comment, your architecture is simply the same as providing all the third party libraries through the system bundle. We can do this in OSGi if we wish by simply placing those libraries on the classpath alongside the Framework itself. However this is not common practice, simply because it IS useful to manage the versioning, dependencies and lifecycle of ALL the modules in our applications, not just the ones we write ourselves.

My point here is that when you execute a test in OSGi you first need to bootstrap the OSGi container, which then needs to run the test in the OSGi environment. This makes writing tests in OSGi quite difficult in practice. Can I ask you, do you practice test driven development when writing applications using OSGi?

And when you do an integration test in Impala you need to bootstrap the Impala framework. Writing OSGi tests should be no harder in practice than writing Impala ones; I admit I have not written any Impala tests but I do not believe you have written any OSGi ones, so we are both arguing from ignorance.

Regarding your question, yes I practice TDD with OSGi. I wonder if you are conflating unit tests with integration tests? My OSGi project code is written largely as POJOs and I unit-test them entirely outside of OSGi; I expect you do the same when unit-testing your Impala projects. With integration testing we both have to fire up our respective frameworks, otherwise it's just not a real integration test. What's the difference?

Finally, a question for you: who aside from you is using Impala?

Phil Zoio replied on Wed, 2009/12/23 - 5:10am in response to: Neil Bartlett

Neil, Chris

Thanks for your feedback. Of course I wouldn't hope to expect to convince either of you that as a hardened OSGi convert you would be better off switching to Impala instead of OSGi. You have already spent the time and effort to acquire the understanding necessary to use OSGi effectively. You have put in the necessary effort to set up your tooling environment to minimise the overhead of using OSGi effectively. However, the reality of it is that for every early adopter of OSGi, there are 5 or 10 Java developers who either never heard of OSGi (believe it or not!), know very little about it, or have looked into it and shied away from it because they find it too complicated. A project like Impala offers a simpler, more gentle introduction to modularity which brings many of the benefits of modularity (in particular the dynamic service model, and dynamic redeployment of parts of an application) with very little pain. 

For most Java developers, the modularity question is really about whether to use OSGi or not. On the one side you have the "traditional" Java model (which most people use), and on the other side, you have OSGi. There is a very big gap between the two. The nice thing about Impala is that it is more accessible for the typical developer. It is a much smaller leap than going all the way to OSGi.

However, if you have already made that leap, then I can see how you would not want to look back to something which is simpler but more limited in scope than OSGi. Think of it for the typical Spring developer who is not using any modularity. They are much better off using Impala because it does allow you to modularise your application and enjoy automatic reloading of application code (ie applying changes without any build or deployment steps, even for web applications).

Ultimately, I would like to see Impala offering a simple migration path to OSGi, allowing developers the option of whether to use OSGi on a project by project basis, and providing an easier mechanism for integration testing (through provision of the dynamic services model) without having to run these tests within an OSGi container. It's too much work for me to do myself in the forseeable future, but I have done enough work to prove that it is doable.

Neil, on to a couple of your points.

99% of problems with libraries running under OSGi are caused by a small number of libraries that assume they live in a classloader hierarchy or that they have global visibility of all classes in the application -

If you are talking about the use of thread context class loader, that is a flawed usage pattern which is like it or not a very common usage pattern in enterprise Java.

I admit I have not written any Impala tests but I do not believe you have written any OSGi ones, so we are both arguing from ignorance.

Actually I have - for example I extended the Spring Dynamic Modules to make it work with Impala. This gave certainly gave me quite a good insight into the practical issues involved in doing integration testing with OSGi.

On this subject, my practice of TDD is not only limited to unit tests with mocks. I also use it with integration testing.

Finally, a question for you: who aside from you is using Impala?

Yes the user base of Impala is very small at this point. However, I do believe that the project offers significant potential benefits to a large number of users: namely users of Spring who are not using OSGi already but who could benefit from the increased modularity and productivity that Impala offers. If I didn't think that this was the case, I wouldn't bother promoting it or documenting it in a way which makes it easier for other to use. That being said, if it doesn't turn out to become "successful" in reaching a significant user base, I can live with this too, as I am still enjoying the benefits of using Impala every day in my own application development.

Cheers,

Phil

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.