Metadata Driven Applications

Several years ago we noticed that we could create an application that allowed the user to ‘create’ new features.  It would have a configuration tool that allowed the user to create the screens and then wire them up individually or in sequence to accomplish a specific task. The resulting screen descriptions would be saved to the database. When the main application started, it would load this metadata and generate the UI. Some customers were very exited when we released the initial version.

During subsequent development cycles the customers requested for more advanced features with the inevitable result of making the application more complex.  e.g. when they create a screen X with a field named gender, they wanted another screen Y to filter out gender specific lookups based on the gender selected in X.

We have noticed the following problems:
1.  Metadata chews up a lot of memory. Many of our larger customers have been complaining about it.
2.  The process of generating the screens and interpreting the collected data through the dictionary takes up a lot of processing. Many customers have complained that our application is too slow. 
3.  Due to the complexity of the software, and the fact that we use a lot of late binding, some bugs always find their way into the shipped product.
4.  Almost every new feature requires that we modify the configuration tool, coding it has almost become an end by itself.
5.  Sometimes, users modify the screens definition on screens that have already been used to collect data.  We have to maintain historical information regarding the screens to make the data collected using the earlier versions of the same screen meaningful.
6.  Reporting has been a big pain, we have to sing vodoo chants to get crystal/activereports produce  a simple report with any decent speed.

Most of the problems highlighted appeared when we tried to scale the application.  I have been thinking of solving the above problems by either:

a)  Re-designing the application so that it caters for the exact requirements without allowing the customer to ‘create’ any new feature.  This can be justified by the fact that the difference between customers in the domain is not very huge. 
b) Live with the current design and improve on it.  I do not know how to go about this!
c) Use the approaches (a) and (b) above.  However, the any new screens added will not support complex business rules other than simple data constraints.

It would be interesting to know whether any of you has ever worked on an application similar to the one described above.  Are there other mainstream products that are designed this way?  If we chose to retain the current desing , how can we make it faster and consume less memory?

Nam
Friday, April 16, 2004

Yep,

been there and I feel you pain. It always seems such a good idea at first, doesn't it.

One way out you can consider:
a> freeze the current "features" of your metadata based system. This amounts to a full stop on adding to your meta language. New whishes ouside the scope of the current language are implemented in a KISS way.
b> Write a compiler for the current meta language. This isn't as hard as it sounds. Most of the setups such as the one you seem to be describing can be reduced to a limited set of good templates. IOW, the target of the compilation is fairly course grained. This should take care of a lot of the "speed" issues and simplify the reporting issues tremendously.
c> (optionally) abolish the metalanguage and transfer full development to KISS (with reuse of the template code).

Just me (Sir to you)
Friday, April 16, 2004

Regarding compiling the metadata; how do you take care of the meta-metadata? I mean how do you handle the rules that say 'if field x = SomeValue then disable screen A'.  We currently store these rules as part of the metadata, I cannot visualize where this will go in the compiled metadata.

OOH, I would like to any references in the web that  discuss this topic.  I will appreciate any useful links.

Nam
Friday, April 16, 2004

Sorry Nam, I can not provide you with a web reference, and it is very hard to advize in more detail without being familiar with the particularities of your system.
For your rules, runtime braches remain runtime braches, but typically in interpreted meta-data driven systems, there is a heap of meta-data that specifies configuration. All the configuration can be compiled out to a specific instance of an application.

Just me (Sir to you)
Friday, April 16, 2004

I've always wondered about these "Meta" - Frameworks.

Do not the Hardware + OS + Language + Compiler/Interpreter/Libraries + (Optional) IDE + (Optional) Data Base, provide all the framework one needs? What is the point in building a framework on top of a framework on top of a framework (Hardware)?

I am genuinely interested in some clarifications.

KayJay
Friday, April 16, 2004

KJ,
The key benefit of using this approach was to shield the development team from having to make changes to code whenever (a) a user required the system to collect more data than it was already collecting  (b) a user required modification in the way the data was collected.

Nam
Friday, April 16, 2004

I've gone down this road in the past. In the end, I feel that everyone would have been better off if I had just built the base system in something like Access, given them some good training, and then provided on-going developer support and a willingness to do some of the hard bits myself.

The reason I feel that way is because I ended up doing a poor job of duplicating Access and still had to provide the training, developer support, and hard bits. Did I say "... a poor job of duplicating Access?" Make that a really crappy job.

Ron Porter
Friday, April 16, 2004

The primary reason to use a metadata approach for a system like this is to aid in your own development process.  If your app requires a lot of rapid iteration during development, metadata can help reduce the time in each iteration, since ostensibly you're working closer to the problem you're trying to solve.

The use of metadata to allow a lot of user-side configuration and modification could be seen as a fringe benefit, sure... but if you have to provide support for that kind of configuration, then you're not necessarily saving yourself any time there.  The big win in metadata is when you get to use it yourself, so you can get your product to market faster.

BadgerBadgerBadger
Friday, April 16, 2004

This approach is heavily advocated by books such as Pragmattic Programmer. It is said to simplify things and speed your development. The deadly tradeoffs you discuss are apparent to all who have tried it, but apparantly not to those like Hunt and Thomas who write those books and charge big money for consulting despite their extremely limited experience with actual development.

Leave this level of complexity to the developers of VB and Delphi.

Remove the metalanguage, and refactor your application to support the actual features required by your customers.

Dennis Atkins
Friday, April 16, 2004

BadgerBadgerBadger,

that first paragraph of yours is exactly the reasoning I followed when I proposed and dived into meta-hell.
The big problems:
- debugging is a nightmare: small changes can have the weirdest effects all over the place, and you will have to change as in any system there (a) will be bugs and misconceptions and (b) the demands will change
- Every little thing you add becomes an iteration through meta-land. Things that should be extremely simple e.g. the OP already identified adhoc queries and reports, become unnescessarily complex mini-projects in themselves.
- Due to the iterative roundtripping the simpelest of things bring your system to its knees performance wise, so you'll have to start an aditional caching layer etc. etc.

If you are not carefull, you soon find yourself on the way to either reinventing Lisp or Prolog, which wouldn't be too bad, exept that now you want the EnterpriseServices version of them, with persistence, transactions etc.

After the experience, I'm not too keen to go there again.

Just me (Sir to you)
Friday, April 16, 2004

Metadata driven applications reminds me of Greenspun's Tenth Rule of Programming:

"Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified bug-ridden slow implementation of half of Common Lisp."

;-)

R Chevallier
Friday, April 16, 2004

(Including Common Lisp)

Friday, April 16, 2004

looks like some folk beat me to it, but yes,

  L I S P

Tapiwa
Friday, April 16, 2004

"Just me",

Actually, I've found nothing but benefit from using metadata in this manner, although I admit the application context is a bit different (in my case, games), so that may be a factor.

I did initially have the "debugging nightmare" problem, but the key thing that made it work was understanding that the infrastructure must take advantage of the same foundation (the metadata) as the other systems will.  To put it another way, metadata isn't something you can use only in one "leaf" area (like UI configuration); it has to be exploited at the core of your application in order to be beneficial.

When you use metadata you're effectively moving your application to a higher "binding level" (in the sense of "coding time bound" -> "compile time bound" -> "run time bound"), i.e. you're changing the point when your abstraction becomes frozen.  Since standard debugging techniques are one binding level below what's needed for metadata systems, you have to add some more debugging support yourself, but this isn't a monumental task as long as you use that metadata to your advantage in that debugging infrastructure.  If you don't, and try to use your traditional debugging process in the context of your "new world", it won't hold up.

The same thing goes for performance issues; you have to use the metadata's implicit knowledge of your system's construction to your advantage, to counteract some of the mandatory overhead.  A metadata based system can actually be considerably faster than the alternative, once you fully grok all the possibilities available to you.

I'm not denying that "metadata hell" exists... but it's just a speed bump, not a brick wall.  It takes a while to think outside the box enough to where you get past that mental speed bump and start getting some programming ROI (but when you do get past that bump, you won't go back).

BadgerBadgerBadger
Saturday, April 17, 2004

I still do not see significant advantages of using

...<tables>
......tbl_one
......tbl_two
......tbl_three
...</tables>

...<action>
......retrieve
...</action>

...<filters>
......lastname='LastName'
......state='MyState'
...</filters>

along with a query generator in the Application over actually  allowing a full fledged sql query to be passed to the object requesting the information.

Yes, I am a very slow learner.

KayJay
Saturday, April 17, 2004

Thanks to all for your contribution. It's instructive to note that nobody has actively advocated  for the approach that we are using.

One problem thought, if we use the option (a) (see my earlier post),  we will have to re-write the application.  Fingers crossed!

Nam
Monday, April 19, 2004

  Recent Topics

  Fog Creek Home