Fog Creek Software
Discussion Board

From Applications to "Loosely Coupled Systems"

I'm doing some research for a whitepaper I'm writing about the challenges in moving from enterprise applications to web service based distributed systems.

One of the areas of interest to me is the change in how programmers/architects have to think/work. Programmers at platform companies like MS or Apple have been desiging and shipping SDKs/APIs for years. Their 'user' is another developer, and they have to build software to support applications they have not conceived themselves.

Corporate developers have not done as much of this historically, and yet now we are asking them to become brilliant designers of objects/services which will be utilized by other departments, partners, etc. In the same way that Apple's early toolbox ( full of exposed data structures, not accessors) was a massive albatross when Apple tried to modernize the OS (Copland, etc.) are we heading for the same type of thing in corporate IT?

If you accept this premise that IT staffers are not well trained for SDK-esque development tasks, the next interesting question to me is how to proceed with a web service re-architecture project if you want to account for this learing, and not end up with a 'modern legacy' mess in a few years.

I'm interested in anyone's thoughts on either of these questions. Or, if you're working on a web services project, how are you aproachng API design?


Thursday, July 11, 2002

It's pretty sad if your developers don't know how to design interfaces.  Reminds me of a junior developer that thought function pointers were some sort of chicanery.  I suggest either buying everyone a copy of 'The Practice of Programming' or some other introductory CS textbook.

Either that, or running like hell.

I don't accept your premises
Thursday, July 11, 2002

Sounds like you only want to put down your corporate IT development staff to me.

As somewhat stated by "I don't accept your premises" your premises are incorrect.  You are merely repeating the folklore from some marketing person trying to sell a silver bullet as they have done in the past and will continue to do so in the future.

Let me give you just one example out of many.  The marketing folklore has stated (I am paraphrasing):  Mainframe programmers will find it difficult to adapt to the modern paradigm of event driven programming from their obsolete procedural programming ways.

Consider the fact that many mainframe systems have been implemented with IMS-DB/DC, which has probably been around for the last 25-30 years or so. 

In simple terms, an IMS program accepts input called a message from a message queue.  It processes it based on the type of message as determined from a "transaction identifier".  The processing program can insert messages back into the message queue and has control over the "destination" of the message (which can be another message processing program).

Is this any different from a C programmers view of the windows operating system?  No.

Your other assumptions are equally invalid as this one, including the lack of "SDK" or "API" experience... even mainframe operating system services are accessed through "API" calls. 

Changing terminology does not change or create a "new technology".  Your "web platform distributed system" approach has no functional differences from what they have been dealing with all along.  Just new buzzwords meaning the same old thing and a "requirement" of a host of multiple third party products to up the cost.

Most corporate IT departments focus on stovepipe applications, not of their choice or decision, but the choice of their user departments - to keep the fortress up around their empire building. 

This "social structure" and politics of your company will be the greatest hurdle, not the technical expertise of your IT department.  You should focus on that problem first.

Joe AA.
Thursday, July 11, 2002

Hmm, I don't think that Web applications per-se put any more requirements on developers to produce re-usable code. I certainly see a lot of shared code/databases in the (conventional) business systems that my employer uses.

You could possibly make the argument that if a company replaces a plethora of incompatible development platforms with a one-size-fits-all solution *as part of* their transition to more web-based solutions, then re-usability becomes more important.

If you replace 2 different 4GLs and 3 different database environments with 6 incompatible web development environments, you probably don't gain much.

Any effect you see of increased re-use of code is probably a side-effect of reducing the number of development platforms - "all new development will be done using WebSphere", or something.

I will say that developing a code library for someone else to use is HARD, in an entirely different sense than even a very complicated application design.


Mark Bessey
Thursday, July 11, 2002

Open Source is the solution.

For evey thing that I've needed to do at my current company, I have either found an open source technology that supports it (struts, iText, JBoss) or developed something that would be a killer open source app if my boss would let me release it (my  framework for generating reports from the DB to the web vi EJB and jsp)

for the afore mentioned stovepipe operations, either reuse code from sourceforge, or write something that willsuite your needs and release it into the wild.

You should have to write substantially little new code.  the biggest thing will be figuring out which of the various projects suite your needs.

Hell, there is even a jdbc driver for DBase outthere if you look hard enough.

Loosly coupled systems are dependant on open standards.  Programmers program to or implement open standards, and either side can be swapped.  See Bridge Pattern in the Design Patterns book. (Gof4)

Thursday, July 11, 2002

I would agree with Mark that application development and API development have little in common.

First of all, thinking of development as an upside down pyramid, the lower layers are the APIs and standards.  If these are not productive to use or have architectural flaws, applications developed on top of them will be resource intensive and inherently flawed.

An API developer also needs good communication and "sales" skills.  He needs to justify why things were done in the manner they were - and be able to back up his answers.  Staff needs to feel that things were done correctly, or else there's the tendancy to bypass the APIs.

Given the foundational aspects of API development, you want to have only the best engineers working on this stuff.  Good platform knowledge is essential.  Also, it's important to avoid the "kitchen sink" approach.  Small, clean interfaces often work best.

Depending on your scenerio, I'd be hesitant to distribute an API project widely.  This kind of thing may be better done by a small group of very talented people.

The bottom line is that API developers may effect your development process, for better or worse, by an order of magnitude greater than a "leaf" programmer.  Make sure these people know what they're doing.

Bill Carlson
Thursday, July 11, 2002

Adam wrote:
"Open Source is the solution."

I would rather say "Open Source is a symptom of the problem". The reason we need good API's is because we want to build an abstraction boundary. A good API is like a contract where the implementer of the API agrees to taking care of something so that the user of the API can effeciently outsource that part of the problem. Both sets of code should be able to develop asynchronously from that point on. This is idea is furter developed in technologies (DLL, CORBA, COM, WebServices ...) that try to loosen the binding sideeffects between the two sides of the contract.
Idealy there would be no sideeffects. The contract would be fully specified and closed. In practice this is never the case.
The source of the implementation of an API should never be part of the contract specification, since it can have no function. Al information in the source is implicit and never guaranteed to carry over to a different implementation of the API. If there is implicit knowledge derived from the implementation that should be part of the API, it should be explicitly documented in the API, and not left implicit in the implementation.

Just me (Sir to you)
Friday, July 12, 2002

I see where you are coming from.  I infer from your statment (Part of the problem) that the problem is when you need to see the source code tosee what is heppening, as opposed to the software doing what it is desgined to do.  Fair enough.

What I really meant is that alot of stovepipe applications rewrite way too much code, when, via Open Source (escpeciall LGPL libraries and GPL standalone applications) much of that functionality is available to you.  If you reuse open source components, contributing to those that you find bugs in, you will end up with a lot of avaiblesoftware that will solve big chunks of functionality for you.  For people in big organizations, it is a boon becasue you don't have to go through the requisition process. for smaller companies...same thing really.

The open source (And really, I mean RMS Free) software is a great boon to the code who just needs to get something done.  The source code beiung open means s/he can go through the code and figure out what it is doing.  We can focus on writing apps, as opposed to reverse engineering the pdf doc format.  If components we use have problems, we can contribute back.

I use JBoss, Tomcat, Apache, Postgresql running on linux.  I use iText, struts, ecs, tinySQL, junit, httpunit, and other libraries to get my job done.  Our code is comprised solely of our comapnies business logic (minus the reporting framework I described above)  and build/deploy scripts that represent our setup.

Oh, I forgot Ant.  I really like ant.

I think this may be straying from the topic though.  The application we are building is screen to databse stuff, collecting data from community care providers for the long term disabled.  It would  not be possible without the web.  The HTML/http/browser setup is what makes lhis kind of development possible. 

To turn it into a losly coupled system as describe in the top article, people can get reports from our systyem in many formats, one if which is a spread sheet.  We don't currently allow inports from spreadsheets, but that is possible.  I don't think I would try to write an app for the other end of the system, like say Java Application that hit a local database, but only due to the distribution requirements.  I'd rather have an imp[ort tool on the website, and a well published list of validation rules.  So if someone wnated to imnport their entire client  list (say) we would accept it as:

1)  Simple excell text format
2)  Columns names Last_Name, First_Name...

If it passed validation, we would automtically import it into the system.

Yes,I would prefer an XML format, but the tools to generate that from a Database or spreadsheet are not quite there...or arre they.  If so, that is a possibility.

Friday, July 12, 2002

In the recent project I was involved in designing I purposely made the middle ware very dumb and created a very simple command language.

The client connected to the middleware using the command language, apart from converting the syntax into stored procedure calls and some simple validation, the middleware just sent on the request.

The database stored procedure then returned whatever it returned in XML.  This wasn't just the data but the status and bookkeeping as well.  Not only that but if I needed the middleware to pass on information to the client itself, such as use _this_ ip address not me for the next call cos I'm too busy, then the middleware passed that onto the stored procedure which then returned it neatly wrapped in XML.

The back end database was SQL Server, but to get it to return the XML in a clean uncluttered way (not the way MS do it by default) took a deal of effort on behalf of the database designer.

So the answer is yes, you can get XML returns back from the database.

Simon Lucy
Monday, July 15, 2002

*  Recent Topics

*  Fog Creek Home