The Old Joel on Software Forum: Part 4 (of 5) - New project using XML & XSD schemas: hayulp!

New project using XML & XSD schemas: hayulp!

First, razz me all you want. Get it out of your system. I don't learn something until I have an application for it. Shoot me, then throw the carcass out to the coyotes...

Secondly: my client brought me a requirement to interface with an external system that uses XML for data interchange. The schemas for formatting the XML are supplied in the form of XSD files that define the XML format.

I found an excellent tutorial on XSD at http://www.15seconds.com/issue/031209.htm.

Great, so I know what XML and XSD are. Beyond that... ??? I am having trouble understanding the tool environment and predominate practices for dealing with XSD and XML.

So, here is what we need to be able to do:

- Generate correct XML from our internal data per the XSD schemas.

- Receive responses from the external system (formatted in XML) and parse out the data elements.

No need (right now, anyway) for: representing web pages in XML; for developing client applications nor web applications that use XML as a presentation layer.

The client's entire programming environment is Delphi.

Right off the bat I am seeing lots of discussion on the net that revolve around using Microsoft specific tools like an MSXML API, or SAX, or other stuff mostly written in Java or C#. Also, most of what I see is UI centric, and relatively little seems to be about data handling. So I am having extreme difficulty understanding what the conventional practice is in data-centric (not web based) applications using XML and XSD.

I think it's gonna be difficult to sell the client on changing part of their development environment just because someone at MS decided that everyone should do C#. Also, the inevitable problem is that I am coming up the learning curve on XML and I am the only one between us qualified even to research this stuff. A new language will make the process undoable as well as unsalable. (please don't argue the point, thank you.)

My Delphi/legacy instincts tell me that I "want" to somehow compile the XSD schemas down to static Delphi class definitions so that we wind up with Delphi code that is capable of emitting XML according to the schema. Likewise somehow for parsing the received XML. But I don't really understand what others find most productive. IE, this way may be laughable and wrongheaded.

Pointers and references to tutorials for dunderheads on data centric application development 101 using XML deeply appreciated.

Bored Bystander
Friday, February 6, 2004

MSXML is a COM API. I thought Delphi fully supported COM? In which case I'd say that's the way to go.

Friday, February 6, 2004

I need a starting point. Yeah, I'm aware of MSXML. I have no idea what I would do with it in code once I imported it.

Bored Bystander
Friday, February 6, 2004

Delphi works fabulously with MSXML 4. Back in the Delphi days we also bought and used XMLPartner from Turbopower (which was a great vendor, and their collapse was pretty much what led me away from Delphi), but the huge effort that Microsoft put into MSXML made it a second rate solution

Dennis Forbes
Friday, February 6, 2004

BB, welcome to the lost world where you are not alone.

Just been there
Friday, February 6, 2004

I don't know anything about Delphi and it's data access components, but if you could use ADO for data access you would simply read in the XML file using ADO and ADO would then produce a recordset from that XML. ADO will also write an XML file from a recordset when you are done adding/deleting/modifying the data.

If the XML file is originally written out according to the XSD schema then it should be no problem to read/write it as long you don't modify the it's structure. (i.e. add a column or change a datatype.)

I'm not sure if this helps you in anyway but thought it might give you some ideas.

Dave B.
Friday, February 6, 2004

Mapforce....(same comp that makes XMLSpy)
http://link.altova.com/products_mapforce.html

-creates "database to xml transforms"
db-> xml generations (it creates a schema and/or classes in c#, c++, java that you could port to delphi)

-and "xml to xml transforms"
take resulting xsd from above and use Mapforce to create a transform to the target xsd

apw
Friday, February 6, 2004

You're talking about deserializing a database/table structure or a memory object from an XSD schema, right? I don't know about Delhi, but .NET does provide several means of doing this. The best one amoung them being the xmlDataDocument object used in conjunction with the StreamReader object. You first read the XSD into the stream reader object saying something like

Dim StreamReaderObj as StreamReader = new StreamReader(StrMyXSDFilePath)

Then you have to call only one more method, ReadXMLSchema of the xmlDataDocument object this time. To this object you pass the stream reader object. Like:

Dim DataDocObj as xmlDataDocument = new xmlDataDocument
DataDocObj.ReadXmlSchema(StreamReaderObj)

Just doing this, .NET deserializes your XSD into a set of one or more tables with rows and columns, which you can traverse using the Tables collection of the xmlDataDocument object.

I haven't seen any serialization capabilities I know of in MSXML. I think your application has to do it itself. I may be wrong.

Sathyaish Chakravarthy
Friday, February 6, 2004

***I don't know about Delhi, but ***

I am damned.

Sathyaish Chakravarthy
Friday, February 6, 2004

Bored,

It's been a while since I've used Delphi, but here it goes...

http://www.chilkatsoft.com/ChilkatXml.asp is the best non validating parser I've worked with. Unfortunately, it is an ActiveX component.

http://www.destructor.de/xmlparser/ is VCL xml parser. Haven't used it myself, but I hear a lot of programmers like it.

You will probably be stuck with MSXML, since you wrote that your client has a XSD schema. Microsoft is the only one (that I know of) that can validate against XSD. Most other ones will validate against DTDs just fine.

http://www.perfectxml.com/soft.asp?cat=5 has a lot information on what sort of parsers are available out there.

Slartibartfast
Friday, February 6, 2004

I just found BetaXML which is an application written in Delphi that works with MSXML and XSDs. Might want to contact the author to see how he did it. This could be useful for debugging XML docs against the client's XSD.

http://www55.pair.com/betasoft/ (bottom of page)

Slartibartfast
Friday, February 6, 2004

BB, while MSXML will validate the XML against an XSD or DTD for you, it will not serialize/deserialize the object. As one poster said, you could use ADO to create datasets or recordsets out of the XML but that's that. I don't think MSXML has the capability to manufacture memory objects out of your schema. .NET has the cookie cutter.

Sathyaish Chakravarthy
Friday, February 6, 2004

Damn, this might just be my excuse to pump up my resume with C#. ;-)

Thanks for the tips, it's very helpful. One thing I am seeing is that two of the most useful Delphi XML wizards are only available in the $$$$$overpriced$$$$ Delphi Enterprise edition (we only use Professional, and I can just see my client jumping with joy at $3000 per developer seat.)

Bored Bystander
Friday, February 6, 2004

Bored:

If cost is a serious issue (and if you can afford to trade time for cost), then take a look at using the python XML libraries. All the python packages are free (both gratis and libre) and pretty comprehensive. There is a delphi / python connection at:

www.atug.com/andypatterns/pythonDelphiTalk.htm

I've never used it because I've never had to program in Delphi, but it purports to allow you freely to intermix delphi code (for the front end UI) with python (for the business logic).

From personal experience, the difficulty with the type of work you describe lies in mapping between the input and output schemata. My first choice would always be to use a tool like like Mapforce, but if it isn't feasible for technical or financial reasons, then your idea of converting the XSD data structure definitions into classes, building an object model from the XML and mapping the objects to whatever output structure is required is clearly the only way forward.

I would suggest, however, that a weakly typed scripting language with meta-class support (like python) is a much better candidate for writing - or at least prototyping - such software than a strictly typed, compiled language. I believe that there may already be a python package (PYbyPY? But don't hold me to it) that can read an XSD/XML files and create an object tree from them.

Non-trivial data transformation does not yet seem to have been widely discussed in the XML community. If your problem is really tough - like on the level of moving data between 3-D CAD packages using different geometry models - then take a look at the literature in the ISO STEP (STandard for Exchange of Product data). Long before XML was even a twinkle in W3C's eyes they had defined a data modelling language called EXPRESS and an associated general purpose schema transformation language (EXPRESS-X). There are good details at:

http://www.steptools.com

I'm not suggesting you should use this stuff, but in my opinion the literature on data transformation written by the STEP community more substantive than that in the XML arena. It seems to me that a lot of the XML work on platform independent data transfer is re-inventing wheels that STEP had already made circular.

David Roper
Saturday, February 7, 2004

Bored, forget all the MSXML/Delphi etc stuff.

Focus instead on what you're trying to achieve - get some data from A to B.

The XML/XSD debate is just a distraction. XML for this type of problem is just a transport mechanism and nothing more. The XSD is useful for validating the XML prior to input/display or for creating XML documents by hand.

Chances are the XSD was automatically generated from your client's DB schema or from an XML document. May have come from Biztalk, or may have been done by hand. Makes no difference. Basically its just a technical document from your client containing the format they expect.

What it does do though is provide you with a means of validating the XML you generate.

Speaking of generating XML, you can do this in one of several ways:

Using a mapping product, e.g. MS Biztalk
Using XML directly from your DB (not recommended for complex stuff, e.g. nested queries, aggregates etc)
By hand, i.e. using a handwritten function

I prefer the latter approach. I run a query against the DB and get an ADO recordset. I then pass this recordset to my XML converter object, which turns it into an xml document.

So where to start?
1) Figure out from the XSD exactly what data they want and where it lives in your DB, E.g. does their "supplier_id" = "customer_id" on your system, etc. Resolve these first and make sure you can find all the data. You can do this by writing a query. Tip: make sure all column names will make valid xml element/attribute names
2) Create simple component/library function to get ADO recordset
3) Create simple component/library function to turn ADO recordset into a string containing the XML document (means you're not tied into e.g. MSXML)
4) Find schema validator - there's quite a few free ones around - and check your XML against it.

All this assumes you don't have a data mapping product, of course!

You can find samples of ADO-XML code on the web, or mail me for VB source code to do this (Yes, I know you're probably not using VB, but it'll give you ideas).

Justin
Monday, February 9, 2004

Justin & David,

Thanks VERY much for the help, insights and overview.

Justin especially, your description of the problem arena is pretty much on-target and gives me much better insight into the nature of the task.

Looking at the free, trialware and payware Delphi XML components on the net has clogged my brain up with a lot of crap. Given the profusion of academic styled text on the subject, it's amazing anyone gets anything to actually work... I just don't have the luxury of turning one project into an "XML Career Choice" but that seems to be the orientation of the stuff out there.

Bored Bystander
Monday, February 9, 2004