Fog Creek Software
Discussion Board




Crazy ass question

If you want to achieve absolute speed and size is it reasonable to write your UI components in Win32 assembler? Where I work we use DBArtisan to manage our databases. The problem is, when queries get really big, the app starts to drag and responds very slowly. If the developers used Assembler instead of Delphi, would it be fast and light?

Paint in Black
Wednesday, July 07, 2004

If your programmers wrote everything in Assembler your company would probably run out of money before the next release.

If database access is slow, then there is something wrong with the design, either of the database or the query. Check out you have all the right fields indexed for a start.

Stephen Jones
Wednesday, July 07, 2004

The problem is not actually the DB itself. I'm talking about the sheer number of records returned being so big the app becomes unresponsive.

Paint in Black
Wednesday, July 07, 2004

I had the S&M inclinations to look at the codegen and I can tell you it's wonderfully optimized. But even that is completely besides the point.

99% of the time is spent inside Windows, where you can't change anything. You're just calling libraries, man, nothing to optimize here.

Alex
Wednesday, July 07, 2004

Then why is the app so damn slow?

Paint in Black
Wednesday, July 07, 2004

If your app is dragging because you're giving it too many records from your DB, then don't give it so many records. I can't give any specific advice since I don't know anything about your app, but there are lots of ways to efficiently present large amounts of data.

Brad
Wednesday, July 07, 2004

We use a commercial app, DB Artisan. It seems to have been made in Delphi and since it's a DB administration tool, I thought it was suppose to be able to handle very very very large amounts of data. It's not suppose to feel heavy...

Paint in Black
Wednesday, July 07, 2004

Could simply be the hardware. What kind of app is it? If it's a data mining app then you should consider denormalization.

However you really need to analyze it carefully, look at the SQL, see why it needs to serve up such a large recordset. There are loads of books written on database optimization, and some DBAs specialize in nothing else.

Stephen Jones
Wednesday, July 07, 2004

I think you are probably getting more data than can fit in the ram of your workstation. As a result, windows begins to swap it out to the harddrive. It probably has nothing to do with the program at all.

Eric Debois
Wednesday, July 07, 2004

How much would it cost to add a couple Gig of RAM compared to rewriting one query? Probably not even close.

Anony Coward
Wednesday, July 07, 2004

If you UI is dragging while you try to process large amounts of data, have you tried running the data processing in a separate thread?  I'm not familiar with your development environment, but in Swing that is what I do to keep the UI responsive while processing a lot of data.

name withheld out of cowardice
Wednesday, July 07, 2004

"How much would it cost to add a couple Gig of RAM compared to rewriting one query? Probably not even close."

Oh to be so ignorant and live in a Microsoft dominated computing world.

.net, the equivalent of MS Bob.
Wednesday, July 07, 2004

You need to profile your code figure out really where the slowdown is.  My guess is that you suspect the Windows display because it is taking forever to load some type of list- which makes the obvious- the more records you try to prefill a list with the longer its going to take.  If this is the case- do you really need to fill a list with thousands of records when the user is only going to be able to look at say 20 at a time??  Come up with some sort of "Virtual List" type solution.  How responsive is the query (does it take more time to get back the reults/recordset than fill the list)?  You need this data to make an informed descision (not to be too blunt- but assembly is not the informed decision and it is a crazy ass question).  Re-evaluate the 3rd party tool your using- just because you bought it doesnt mean it is 'industrial' or good for that matter...

Mike

MikeG
Wednesday, July 07, 2004

Are you trying to display a few million entries in a listbox or grid? If so, what use is that?
(sorry if this sounds like a crazy ass comment)

Just me (Sir to you)
Wednesday, July 07, 2004

Listen, I did write the damn thing. My company just bought it and I just run queries against it. Those queries are always on the heavy side, and having some theoretical knowledge of SW development (PHP...) and having heard that Assembly is smaller and runs faster than anything else, I decided to ask you guys how to do this.

Nothing more.

But the "Add more RAM" answer was quite on the mouche.

Paint in Black
Wednesday, July 07, 2004

"...sheer number of records returned..."

Listen.  Go back and read Stephen Jones answer.

If you're trying to present so much data to a human being that the *software* is struggling to cope with it, then you've got a profoundly stupid design.

You have a bad design.  You have a bad design.

I don't know why I'm bothering.  Nobody looking for assembly solutions ever acknowledges and fixes their real problem.

Any UI that presents more information than a listbox can easily accommodate is dumb, dumb, dumb design.  If you hear the words "virtual" and "listbox" together, you're hearing the wrong answer.  dBase and its ilk have inspired more savagery upon computer users than any third-world death squad could ever dream to accomplish.

Oh, never mind.

Torque
Wednesday, July 07, 2004

Paint in Black may forget, but I hear you. The strength of this forum is precisely these gems like yours. Can you explain why I mustn't exceed a list box in the amount of data?
And what influence did dBase have in all this?

RP
Wednesday, July 07, 2004

Pretty much the only time you need to concern yourself with rewriting code in assembler is after you've improved your algorithms as much as possible with higher level languages.  At that point, use a profiler and see where it's spending the most time.  That's the part to rewrite in assembler.  There is no point in rewriting the whole thing.

Aaron F Stanton
Wednesday, July 07, 2004

To answer the original question, DBArtisan is slow with tons of data because it wasn't designed to be used with tons of data.  It has nothing to do with the implementation language - if they'd used assembler and attacked the problem in the same way, you'd see roughly the same performance.  If they wrote it in Delphi and attacked the problem differently (that is, designed the program to be good at browsing large amounts of data), you'd see far better performance.

The question is kinda like "would my 60-horsepower car go faster with premium gas?".  It might by some small amount, but that wouldn't be the most effective way to make your car go faster.  :)

schmoe
Wednesday, July 07, 2004

Tell the developers to find all the places where they multiply by 2 and use this assmebly code instead:

SAL AX, 1;

Then find all the places where they multiply by 4 and use:

SAL AX, 2;

(assuming they move the value to AX first, of course)

This should easily speed up your code by a factor of 100.

Billy Joel on Software
Wednesday, July 07, 2004

That assembler development is so expensive is a myth. With a decent macro assembler it's quite efficient. Main problem is documenting your code, there's not really any "idioms" (don't know if that's the right word) as other languages have, so you kind of make up as you go.

I don't think you would gain any performance on a modern OS anyway, since most of your time is probably spent elsewhere (VM, GUI toolkits etc.), and the OS you interface with is probably written in a higher level language anyway.

To understand why your application is slow, you use a profiler. Google is your friend here. There are both free and proprietary profilers available, depending on the language and platform you used.

Jonas B.
Wednesday, July 07, 2004

What MikeG refers to above is often called "paging" -- ie, you load 50 records at a time into the UI and have previous/next buttons, rather than 200,000 records all at once.  This is easy enough to accomplish in the data tier by adding a row count column to the query (or an autonumber to the table) and filtering the results on it.

Multithreading (as suggested above) is also a good tool...in this case, the user would still be waiting for the query to complete, but the UI wouldn't freeze in the interim.  Ideally you'd also give them a Cancel button.

Of course, this is all assuming that the query itself is optimized and that the bottleneck isn't in the DB itself.  If it is the DB, then you only have two options: display less data, or get a new DB engine.

And for the poster who wondered why it's considered asenine to display more records than a human being can possibly manage: there's two reasons that come to mind.  One, you're performing unnecessary work which impedes performance.  And two, it's a bad UI design because the whole point of the app is for the users to manage the data, and they can't very well do that if they are overwhelmed by it.

Joe
Wednesday, July 07, 2004

"Can you explain why I mustn't exceed a list box in the amount of data?"

If your computer can't process that much data easily, you user is being inundated with too much too.  There are much better ways to let the user find what they want to work with than to show them everything at once.  A narrowing search, for example.

"And what influence did dBase have in all this?"

dBase and its clones offered a visual idiom of presenting in much the same way as a listbox the entire contents of a file (like a table in the modern terms).  Fine perhaps when you're presenting 185 brownie recipes, but absurd for a list of 23,000 clients.

Torque
Wednesday, July 07, 2004

What does all this have to do with asses?  Maybe I misunderstood the question...

5v3n
Wednesday, July 07, 2004

Is it the retrieval that is slow or the display? 
Assuming you have another method of running the query to completion, use it to see how long the query takes (SQL Profiler in MS SQL land or QTODBC come to mind).

If the query is the problem, fix the indexes etc. 

If the query is not the problem, its probably an issue in loading the data into a grid/listbox/listview etc for you to see.  There are methods to optimize each of these as well -- paging, virtual display, turning off sorting before adding the records, doing bulk adds, pre allocating storage in the list box or list view etc. 

But I also have to agree with the previous poster that in a lot of apps, display this much data to the user a once is just begging the user to hate your UI. 

Billy Boy
Wednesday, July 07, 2004

"and having some theoretical knowledge of SW development (PHP...) :


Now I understand why the OP is so confused.

A little knowledge is a dangerous thing.

Mr. Analogy
Wednesday, July 07, 2004

What everyone else said... a database application is almost always an "I/O bound" situation, meaning that the physical process of input/output is an order of magnitude slower than the code running on the client workstation's processor. Think about all the crap that has to happen to communicate the data back to the workstation: even a fast ethernet will be much slower than memory to memory copy. And you're dependent first of all on the network speed. 

Assembler makes the following kinds of applications and programs run visibly faster than the equivalent higher level language solution: games, graphics drivers, graphical rendering of complex (3D) objects, etc. It's really, really not going to help a database appear to be faster. Even in the early days of PCs, nobody bothered to write database applications in assembler.

I agree with the concensus that you have a lousy design. An arbitrarily huge returned dataset that exceeds the size of physical RAM still needs to go somewhere, no matter what language the app is written in. The dataset's size will cause the OS to start swapping physical RAM to disk, and this will reduce the apparent speed by a great amount.

My guess is that between physical network transmission speed and the workstation being brought to its knees by the dataset size, that is where the problem lies.

It has nothing to do with the application's language.

Bored Bystander
Wednesday, July 07, 2004

Assembler != Silver Bullet

Bill Rushmore
Wednesday, July 07, 2004

"and having some theoretical knowledge of SW development "

This is a great line.  I'm going to remember it for later use...  but you know, in the late 90's this was about all it took on your resume to get a high-paying job.

free(malloc(-1))
Wednesday, July 07, 2004

Hey, come on. I've been doing a bit of everything in IT, from PHP websites to help-desk. Having heard that assembler produced smaller and faster code (I have written this somewhere before) I asked the theoretical question of whether it would be better to write the UI in assembler.

I'm not going to do it. I'm not doing it. I'm not selling that app, I'm just asking a question. Gimme a break, will'ya?

Paint in Black
Wednesday, July 07, 2004

The answer to your question is NO.

END OF THREAD.

Bob
Wednesday, July 07, 2004

Last time I had optimizers and assembly in the same task, I was getting RID of the assembly in favor of C so I could rewrite the algorithm easier.  Worked great, ended up 300x faster.  How?  Mostly through not running code at all - caching approximate values instead of recalculating them for example.  That's the theme here: Use a profiler, don't do slow things, do necessary slow things less often, and finally optimize the most frequent code path.  Not a silver bullet, but it helps.

Mikayla
Wednesday, July 07, 2004

I would write assembler code that allocates a crap load of ram and runs very slow.  Your problem has nothing to do with the language.

This thread seems like a troll.

christopher baus.net
Wednesday, July 07, 2004

If you are doing a file extract (creating a file of data for some other purpose) DB Artisan may not be the best tool. It has to format the data for display etc. It might be faster to use a simplier program that will run the sql and create an output file, like SQL Plus (oracle) or isql (MS or Sybase).

John
Wednesday, July 07, 2004

Wordperfect was written in assembly, and was slow as a dog.

You'll just bury yourself even deeper in implementation details than in any other programming language, making it even harder to stay on top of general design issues.

I know of two common performance issues with data-centric applications written in Delphi. One is to load more data into memory than you need at any given time. The other is to execute the same code over and over again for no good reason in response to various events. Algorithms are rarely an issue with data-centric applications.

But don't believe any of this, profile your application and see for yourself.

Big B
Wednesday, July 07, 2004

Having compiled Word Perfect a few times, I think I can safely say that not all of Word Perfect was written Assembly.  In fact a very small amount of it was. 

christopher baus.net
Wednesday, July 07, 2004

"Wordperfect was written in assembly, and was slow as a dog."

The very first versions were, up to 4.0.  After that, it was written in C, or ported in some cases (funny to  see functions like uxwpd that mimic an assembly label.)

WP was slow because it used a split sequential buffer for *everything*.  Put a font code in, it went into the buffer. Select a TOC code, it went into the buffer. Rather than use appropriate data structures to represent the formatting and layout of text, they just dumped all into one buffer and then required that developers crawl through it to find what they needed.  Most of the formatting was controlled by globals that were set in callbacks when a code was interpreted from the buffer.  Also made for devious bugs when a developer crawled through the buffer themselves (and no, there wasn't a standard interface for document traversal) and got off by a byte or two and would stomp codes and corrupt documents.

Just my $0.02.  Back to your originally scheduled topic.

Former WP developer
Wednesday, July 07, 2004

Damn straight, whigga! <jk>

Crazy Ass Whigga
Thursday, July 08, 2004

*  Recent Topics

*  Fog Creek Home