The Old Joel on Software Forum: Part 4 (of 5)

Distributed Systems

Hi,

I am new here. I find this forum very interesting and educational.

I am studying a PhD in Information Systems Development at the University of Hull UK. I wonder if you guys could give me your opinions about Distributed technologies applied to Decision support Systems for multi organisations. I am thinking about a software framework of decision components that can be chosen on-the-fly by users as they are dealing with a situation. There would be components for communication, to access data, documents, models, to do calculations, to build reports, etc.

Those decision components would be generic pieces of software that can be arranged in any order depending on the situation. At the end, every decision process would have the appearance of a workflow. Every piece in that workflow would be an instance of a decision component and could be stored in any computer in the system. The system will be able to modify itself if necessary (adapt) to changes) and record a history of the activities performed so as to save them as knowledge for future decisions.

I am trying to avoid centralized databases or application servers. I am thinking more about a federation of pc’s connected by the system, each of them containing a component of the decision system.

Is that possible? Are distributed technologies going in that direction? There are a lot of issues to deal with… for example: network reliability, customization of components, adaptation to changes, evolution, etc….. what do you think about jinni or javaSpace or similar technologies?

I am very interested in your opinion… I’ve been reading all this academic stuff, about research projects, and theoretical frameworks, but to be honest I prefer the point of view of real developers, the people who can tell if something really works or not, the people who do the job …

Thanks

Cecilia Loureiro
Monday, May 24, 2004

Yes, entirely possible until you insert the first PHB in the middle of the process, then it goes all to heck.

old_timer
Monday, May 24, 2004

The reason centralized database, or application servers, work so well is that there always needs to be some central 'traffic cop' to organize the processing.

Even in distributed processes, some entity (person or machine) has to allocate the processes and the communication paths between them. When you focus on dynamic configuration of multiple tasks across multiple processors (at load time, or at run time) this becomes even more critical.

The centralized solution simplifies the role of the 'traffic cop', since the only questions are, what services are generic enough to be on every desktop (the browser), what services are on a database server, and what services implement business rules in code on the application server. (This is the 'classic three-tier' model).

A true distributed processing solution has no a-priori 'good solution' -- so you must specify the arrangement of processing and data that seems 'good' to you (under some evaluation metric) and implement that arrangement through the 'traffic cop'.

AllanL5
Monday, May 24, 2004

I recently worked with a proprietary distributed system which had some of the properties that you describe. But there was still a "traffic cop". However, the traffic cop could be any node in the system, so that if the master node became unavailable for some reason, the other nodes would "elect" a new one based on a preset algorithm.

I think an important feature to remember is the ability to have heterogeneous nodes. For example, imagine a distributed system with ten nodes, but you have only 5 MATLAB licenses. A complex statistical decision is constrained in where it can execute (even if those licenses can "float"). Perhaps this is well-covered in the theoretical literature...

I think that in any distributed system you have to have some level of central control. This is the only way (in my limited imagination, at least) to harness the power of the whole system to a single purpose.

Good luck to you.

Rob VH
Monday, May 24, 2004

I agree with old timer. The framework makes sense to someone with a PhD, but the average customer would probably look at the saleman like he has two heads.

Tom H
Monday, May 24, 2004

"look at the saleman"

make that "look at a salesman trying to explain the system"

Tom H
Monday, May 24, 2004

I'm not entirely sure that this is along exactly the same lines as you have in mind, or whether you've looked at it already, but it seems to me that distributed agents may be part of the way to what you're looking for.

If each work flow was defined in terms of some sort of plan (e.g. step 1: get data, step 2: analyse data, step 3: format report, step 4: distribute report ...) and each step could be performed by one or more agents that were aware of their own capabilities and context (e.g. I am a data analysis tool; I can perform analysis X; I can access file server S ...) and could communicate with each other across the (physical) network, then I'm not so sure that a 'traffic cop' process is actually needed. In fact if the work flow itself was an agent, then it could autonomously negotiate with the processing agents to complete itself in the most efficient manner.

David Roper
Monday, May 24, 2004

I'm stuck with the question of 'why bother?' I can't see that this would actually do anything useful in a significantly better fashion than existing systems, and would add lots of bizarre complexities.

Just because you _can_ do something (or might be able to) doesn't mean you _should_. The PC itself was an example of this: 'let's break away from centralised management and control and put anarchy on the desktops'. Now there is a definite trend back to centralised management and control.

I'd say a better PhD topic would be 'is this idea compellingly and sustainably better than the existing solutions'.

But I absolutely laud the OP's idea of asking non-theoreticians.

bah humbug
Monday, May 24, 2004

The original post from the PhD candidate is most probably a troll.

Neumonic
Monday, May 24, 2004

Holy Batman. And you're going to get a PhD in information systems? You don't know anything.

Tuesday, May 25, 2004

What is the problem you are trying to solve by going for this astrotecture?

Just me (Sir to you)
Tuesday, May 25, 2004

A PhD, apart from being a pain in the neck... is theoretical... it might not necessarilly be applicable in a straightforward manner to real situations.

Actually, my PhD is management... I see IS development as a business process, designed to create systems to serve other business processes. I am too an IS developer, I have been for 9 years.... and the idea for the PhD is based on my experience.... (and a lot of imagination).

What I meant guys is that I imagined a theoretical situation, in which people from multiple organisations need to make decisions.... they would need to access iniformation in different formats and from multiple locations, combine it, calculate it, analyse it, they would need to call other people to join them, they would need to meet in groups (virtual meetings), they would need to recall previos decisions and store all the knowledge learned from each experience.... everytime knowledge is acquired the system would adapt itself to evolve

that's from the management point of view.... but how can we tackle this situation from the IS point of view? I was thinking about distributed technologies because unlike centralised solutions (one database server, one application serve, etc) distributed technologies might emulate multi-organisations better... imagine a federation of computer applications interconnected....
any thoughts?

well that's what I wanted to say..... thanx very much for your comments... I enjoyed them.... mmmm... most of them.

Cecilia Loureiro
Tuesday, May 25, 2004

Cecillia,

it is still a bit unclear to me, but I'll have a go.
Can it be done? Certainly. I have done similar type of systems in the past. However, experience has thought me to be very wary of unnescessarily complex architectures.

First of all I'd make a clear distinction between the need for modularity and the need for distribution. Centralized single instance system architectures can be highly flexible and modular. In fact, some would argue that because (for most scenarios) they are less complex, they are more flexible in both development and operation. There is nothing I see in the sketch of your use case that nescessarily invalidates a centralized system.
Now the need for distribution can be a non-technical requirement. There might be operational restrictions, aquisition-control issues, policies, autonomy ... many things that force one into a "distributed" scenario where technically a central sytem would suffice. There can also be technical factors: offline scenarios, unique infrastructure demands, scalability ... that would necessitate distribution. But distribution is often a costly thing. You add a whole extra layer of complexity to your system that does not come cheap. Simply guessing "it might emulate multi-organisations better" does not in itself seem to support a good ROI for this.

Also I would urge if possible to look deeply into just how much unified abstraction you require. Certainly a very abstract metadata architecture has a "clean power" appeal, which is what lures most in this direction initially. But all this comes at a price. At the "framework" level, you will now go about specifying in terms of abstract "information" manipulators and "knowledge" and "information flows". Framework tools will all play at this abstract level. Again, while to your star-team members this might seem logical, you will find that not all your operations/development people are recruted out of the top 5% in the industry. If you are not carefull what could have been a simple A->B type of utility that Mort could do in an afternoon can get stranded in a myraid of implement IabstractInterfaceFoo and MustCatchExeption13546 that requires Elvis the best part of a month. It wouldn't be the first time the very features that where architected to make a system flexible in theory caused it in practice to become just the opposite.

Just me (Sir to you)
Tuesday, May 25, 2004