The Old Joel on Software Forum: Part 3 (of 5) - Enterprise Data Distribution: how best to do it?

Enterprise Data Distribution: how best to do it?

I'm looking for some "best practices" advice on distributing data to clients in the field. To take things to extremes to see what shakes out, I'll imagine a hypothetical company with lots of stores selling my artwork (prints, posters, greeting cards, etc.).

Let's say I have 50,000 cash registers spread across the country. I need them to have up-to-date price lists and other business data. I need to be able to tailor my data to a specific cash register, or store, or state, or region, or the whole country. I also want to know when the stores sell my work, so they should be able to report their sales back to me (daily? real-time?).

Updates may be daily or more or less often (maybe I won't create any new art for months, then do three pieces in one day). Since updates may need to occur during business hours, the method can't be a resource hog. I want to conserve bandwidth, so I'd like it if just deltas could be transmitted, not the complete price lists, etc. (If I have 1000 products, it would be dumb to send every store in the country a 1001-entry file when I make a new product.)

Some assumptions to make the problem easier to visualize: hardware will be Windows/PC cash register terminals at art stores, stand-alone kiosks in the mall, or hand-helds taking orders at an outdoor festival. Let's assume all the devices are standardized to a Microsoft OS like XP, and my central database is Oracle 9i. Let's also assume all the stores will be running the same cash register software.

I'd like it if the client machine was smart enough to check with the home office for updates, rather than having a massive co-ordinated push from a central location. Add the usual desired qualities like secure data, low cost, standards-based, fault-tolerance, etc.

I find myself thinking about web services, XML, bi-directional data synchronization, etc. But I don't know. If you could point me to relevant white papers or offer some advice, that would be great. (I don't expect you to do all my work for me; I just want to know how things are done these days.)

Greg O'Rear
Friday, October 24, 2003

I'm also looking at this kind of thing, to write a dedicated app that sends HTML to a CMS server via a call in XMLRPC. I need to find out how big a parameter can be, to check if this is valid way to do things. In you case, that would obviously be a call to _download_ data instead.

Otherwise, just make a call to an FTP server every so often, and let the client app look for updates?

Frederic Faure
Friday, October 24, 2003

There's no "one size fits all" best practice. You need to examine and prioritise your requirements to design the best tradeoffs.
- Sending updates only rather than complete price lists conserves bandwidth, but requires more synchronisation. Your case seems to be a small data file & low update frequency, so my initial reaction would be to send the whole file.
- Push requires central maintenance of client location, polling from the clients may require more bandwidth and has security implications.
- If your remote nodes are under your control you have more flexibility than if they are owned by 3rd parties

AH
Friday, October 24, 2003

Microsoft has a set of patterns describing methods for doing exactly what you are asking. If you go to MSDN you should be able to locate the PDF.

Geoff Bennett
Saturday, October 25, 2003

I'm not sure why "browser based" wouldn't work in your case. If you're talking about server push / client pull, you're already assuming these computers are all online 24/7. This is also implied in your 'several updates in one day' statement.

By "browser" I'm not limiting you to a web browser, you can use a fat client as well, but they pull live data from a centralized server. This is convenient as well for ordering and inventory.

If you only sync once a week, what happens when your painting is shown on on PBS and you have a run of that painting? 5,000 times the ordinary amount order that panting, and you're not warned until the end of the week, and at the same time the system allowed orders that whole time.

If it's mission critical these cash registers be up even if the internet is down, then they should be allowed to store both the entire catalog and orders for sync later.

PDA's I'd assume would have to sync only occasionally.

Perhaps you can use a model similar to many content management systems, but on a different scale.

The web server has a cache of commonly created pages - basically any page that's been called that it created, and it only calls the app server for new pages or updates, or when someone logs in to get personalized data, or initiate a transaction.

Any time someone calls up an old page, the web sever can ping the app server - not for the data on the page, but just with "has any of the data i have changed?" If it's a yes, it makes a full request for that data. So whenever someone hits a page for the first time it's a little slower, but from then on it loads pretty quickly.

In this case, the web server would be each terminal, and the app server would be you centralized system.

www.MarkTAW.com
Saturday, October 25, 2003