|
.net benchamarking
Hi coders,
I'm a late adopter to .net and I'm astonished at how slow its execution code is.
To be fair it's a great IDE and, yes, it's a powerful and flexible environment to work in, but what good is that when the outputted code is so lame?
I was recently tasked with writing an analyser for IIS log files. It involves a lot of file reading of files, a bit of file writing and tons of string analysis.
I wrote it in VB6 first and was not suprised that the execution time was a bit slower than I'd have liked, but it did the job. I wondered if a .net version might improve things so I rewrote it in .net - a complete rewrite - not a VB6 to .net conversion.
Shock ! it takes, on average about 10% longer than the VB6 version to crunch the data files, and that's not counting the long wait for the app's GUI to make an appearance. on my desktop
I was interested now, so I dug out my (dusty) old copy of Delphi, installed it, struggled for hours with the syntax, 'cos I've forgotten most of the Delphi/Pascal I ever knew, but in the end I got it working and it is 3 to 4 times faster than the VB6; furthermore, I've not yet looked to optimize it, and as my Delphi skills are not great I suspect there is room for improvement in there.
DEV TOOLS
VB app in VB6
.net app in .net 2003 with Framewrok 1.1
Delphi app in Delphi2 - yes, version 2 - nearly 8 years older than my version .net !
HARDWARE
Pentium 2.6 with WinXP Pro
I'm interested to know if anyone has had similar experiences; there must be loads of us porting perfectly good VB6 code to work under the .net framework. Is it worth it?
Marx
Friday, November 21, 2003
I doubt it's fair to compare Visual Basic (.Net) to Delphi. Visual Basic does not produce machine code, while Delphi does.
If you happen to understand German, there's been a comparison of C/C++, C#, Delphi and Java lately, i.e. execution time of machine code/intermediate code, using the usual IDEs and compilers, with standard tasks (object creation, recursion etc.). Interestingly C# and Java did fall back behind the other two that far, sometimes being even faster than compiled code.
For what it's worth.
Johnny Bravo
Friday, November 21, 2003
Forgot to mention it was in c't magazine.
Johnny Bravo
Friday, November 21, 2003
CORRECTIONS
'...involves a lot of file reading of files...' should of course, read '...involves a lot of file reading...'
And I know how to spell 'Framework' - honest ;)
Marx
Friday, November 21, 2003
My experience has been the complete opposite -- code runs many times faster in VB.Net than VB6. Obviously, your milage may differ depending on what your code is doing and how it's implemented.
There really isn't any objective way to measure whether Language A is faster/slower than Language B, since any test will merely show how fast that particular code runs. (I remember one lovely flamewar between C# and Java folks over which was faster for making some obscure regular expression calls, and the results were completely skewed by how the code was implemented.)
I'd be willing to bet, however, that you could get dramatic improvements by profiling your code and optimizing the most time-consuming portions.
DevPartner Community Profiler (free):
http://www.compuware.com/products/devpartner/profiler/
"Performance Optimization in Visual Basic .NET"
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/vbtchperfopt.asp
"Writing High-Performance Managed Applications"
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/highperfmanagedapps.asp
Robert Jacobson
Friday, November 21, 2003
"I was interested now, so I dug out my (dusty) old copy of Delphi, installed it, struggled for hours with the syntax, 'cos I've forgotten most of the Delphi/Pascal I ever knew, but in the end I got it working and it is 3 to 4 times faster than the VB6; "
A buddy of mine had a similar experience in converting a small utility graphics application (originially created with delphi) to .NET. The .net app was 3 to 4 times slower at rendering then the previous application.
It might have something to do with the following :
The .NET Abstraction Pile
http://software.ericsink.com/Abstraction_Pile.html
Cletus
Friday, November 21, 2003
Hi Johnny, I appreciate your point, and thanks for the links.
But, that does not really address what I meant, which I should make clearer.
If you accept that .net is slow to run, then what is it good for? It might be nice to dev in, I like it, but the end-users won't be impressed.
Marx
Friday, November 21, 2003
Marx,
What framework classes are you using to read, write and manipulate the files?
Rick Childress
Friday, November 21, 2003
Cletus, the .Net Framework uses GDI+ for rendering, and the GDI+ drivers aren't hardware accelerated. Unfortunately, that means that a graphics-heavy application in VB.Net or C# for now may be more sluggish than one in another language.
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=eN32%23sBqBHA.2128%40tkmsftngp07
(Complete thread)
Robert Jacobson
Friday, November 21, 2003
"If you accept that .net is slow to run, then what is it good for? It might be nice to dev in, I like it, but the end-users won't be impressed. "
Oh, but you can tell your customers that the application is ".NET" compliant now :)
Cletus
Friday, November 21, 2003
.Net itself is not "slow". What exactly do you mean?
If you mean the intermediate code produced by Visual Studio .Net's IDE, or the free SDK compilers (which is the same) which then gets executed in the CLR: actually such code is optimized beforehand or during runtime so there should really be not a bottleneck there.
I know some peps here will jump on me for this example but take a look at the Quake 3 port from C/C++ to Managed C++ in .Net. It's frame rate dropped only by 10%, so obviously it's neither the .Net Framework's libraries nor the CLR which are "slow".
Perhaps you could give us some more details of the application you ported.
Johnny Bravo
Friday, November 21, 2003
That's a Quake 2 port. The Q3 source has not been released by the Great Carmack (yet).
Mitch & Murray (from downtown)
Friday, November 21, 2003
Rick,
I don't have the code with me, but I *think* I used the .net StreamReader. I don't think the file i/o is the bottleneck - but I've not profiled it. My gusess is it's the string analysis, I have to check for a substring many times per line, with a variable substitution for part of the string test.
Another point that might be of interest is that I'm using the ASP File System Object in the VB6 app, rather than VB's native file i/o statements and that improved the VB6 performance dramatically.
And, LOL Cletus, that is so true
Marx
Friday, November 21, 2003
Johnny wrote
>.Net itself is not "slow". What exactly do you mean?<
I mean that the runtime of the .net 'exe' is slower than I expected.
TYI I used Visual Studio .net
And, WOW, what a response, looks like someone (me) opened a can of whoopass - let's not have a flame now all you .net evangelists ;)
Marx
Friday, November 21, 2003
If your doing lots of string manipulation, one thing to look out for is to use StringBuilder, instead of String. Everytime you do anything with a string a new one gets allocated which can result in a huge number of objects being created.
nojuan@nowhere.com
Friday, November 21, 2003
Marx,
it might be viable to examine your code for possible replacement opportunities of intuitive string operations with regular expressions. Depending on the task at hand that might give you another performance boost as regexes can be compiled for faster repeated execution, and are lighter on execution overhead than general string methods.
Johnny Bravo
Friday, November 21, 2003
RegEx is a thought I don't know it would be faster than the InStr/IndexOf functions I'm using now, but all things being equal I could also use the RegEx functions in the VB6 app, I'm already referencing the ASP library, so I'd guess that any improvements would go across the board and apply equally to the .net and VB6 performance equally.
Marx
Friday, November 21, 2003
It would be interesting to see your code (and a sample data file). It's very possible your doing something that could be optimized. Profiling it first is an important step though, otherwise your taking shots in the dark.
Guy Incognito
Friday, November 21, 2003
Note that from what I've heard about the Quake 2 ".net port", it primarily consisted of a huge swath of unmanaged code with a small subsection of managed code, though someone really got their 15 minutes for that.
Dennis Forbes
Friday, November 21, 2003
As a sidenote, .net was never meant to be a barnburner and was really intended for the java/ASP/VB crowd where any performance is adequate (and where you can say "Uh...it has a recompilation optimizer..." and the minions will parrot it without ever doing a single empirical benchmark set to see for themselves. The overwhelming majority of .net advocates have never done any sort of benchmark, and just take claims of greatness at face value). Having said this, early on I was working on a performance critical project that basically acted as a message processor for MSMQ. Of course a .net evangelist coworker simple _believed_ (in the religious sense) that .net would represent all that was good in the world, and it would surely be the fastest thing since sliced bread (oh, and it has "native" access to MSMQ versus the "bulky" COM access...you know, apart from the fact that the "native" framework calls are simply stubs just like the COM stubs). I threw together two benchmarks, one in Delphi 5 and one in .net (C# using all best practices) and the Delphi 5 code absolutely trounced the .net code.
None of this is meant to deprecate .net: It truly is a great environment, and the functionality in the Framework is simply extraordinary (though there are still some amazing gaps, like most graph technologies). And of course comparing it to the interpreted ASP is night and day.
Dennis Forbes
Friday, November 21, 2003
Someone once told me that publishing .Net benchmarking (or benchamarking) results was a violation of the license.
Clutch Cargo
Friday, November 21, 2003
>Someone once told me that publishing .Net benchmarking (or benchamarking) results was a violation of the license.<
There's actually a clause to that effect in the Visual Studio EULA. It's completely stupid and almost certainly legally unenforceable.
Robert Jacobson
Friday, November 21, 2003
BTW, I might fall into the .Net evangelist camp, but it's silly for anyone to assert that any one language (or framework) is "the best" for all situations.
I used to work in the printing industry, where we had the adage "you can have it done quickly, done right and done inexpensively -- pick any two." Picking the right language for a job is the same situation -- you can maximize ease of development, professional results (fewer bugs, better GUI, etc.) or maximum performance, but not all at the same time.
If performance is your chief criteria, like if you're developing the latest 3D game or number-crunching scientific application, of course you should use a compiled language like (unmanaged) C++. If you're developing a typical business app where you want to get a product to market quickly and as professionally as possible, where performance isn't critical, C++ isn't the best choice.
Applications running on the .Net Framework aren't going to be quite as fast as a C++ app, but are certainly within the same ballpark -- which is good enough for most needs.
Robert Jacobson
Friday, November 21, 2003
Well, I did some small benchmarks of my own and found that C# 1.0 and Managed C++ were about 10-20% slower than unmanaged MS VC++ 7.0 on average, and around 50% slower in the worst case (numerical algorithms).
However, this only refers to computational performance. If you have lots of I/O going on (GUI, disk, keyboard) there won't be a noticeable difference.
Also, memory allocation is extremely fast in .NET. I haven't benchmarked it against unmanaged code but I did notice that avoiding memory allocations makes practically no performance difference in .NET.
Chris Nahr
Saturday, November 22, 2003
>Someone once told me that publishing .Net benchmarking (or benchamarking) results was a violation of the license.<
Crikey, will the MS Police be paying me a visit for speaking without license?
There's some interesting stuff been postedin here, on a bit of a sidenote, I got to wondering about the Delphi compiler(s) - did Borland continue tweaking.optimizing the compiler: is an app compiled in Delphi 5 'faster' (I'll put 'faster' in quotes since it looks like I'm not allowed to say it otherwise ;)) than one compiled in Delphi2 ?
Marx
Saturday, November 22, 2003
Denis wrote:
<As a sidenote, .net was never meant to be a barnburner and was really intended for the java/ASP/VB crowd...>
Denis I think we'd all agree that it's best to use the best tool you have for the job, allowing for the real-world considerations, for example my skills in Delphi are not great, my C is worse and my TASM/MASM non existant.
But back to VB, one of the reasons I posted was that I was amazed to see slow old VB out-performing the all new .net
I accept all the points made re profiling and optimization, perhaps .net's string builder would boost the .net app, and perhaps I've not used the best methods available under .net, but what I have ended up with is a near equivilance of code between the VB6 and .net apps with each using their 'native' functions - excepting that the VB6 app uses the ASP File System Object (which proved faster than VB's native file handlers). And, as I said slow old VB6 out-performed .net and Delphi2 a 7 year old tool out-performed both by upto 4 times.
Marx
Saturday, November 22, 2003
Marx, your experience is very, very atypical. Every benchmark I've seen, coupled with my own experience, shows that VB.Net is usually many times faster than VB6.
The bottleneck is probably with your code. If you optimize it you will see different results. Simply complaining that "my unoptimized code is slow" is a tautology.
Robert Jacobson
Saturday, November 22, 2003
Also, you can't compare "identical" code between VB6 and VB.Net. The danger of VB.Net is that it looks very similar to VB6, but is actually a completely different language -- and has to be opmized diferently. You can't make an "apples to apples" comparison If you would just show us your code, we could probably identify the bottlenecks very quickly.
Robert Jacobson
Saturday, November 22, 2003
Rob, fuck the science
Let's get real world - how do I make a SLOW .Net app beat a 7 yr old Delphi app ?
Marx
Saturday, November 22, 2003
Do tell me, I'd like to hear
Marx
Saturday, November 22, 2003
I switched to VB.NET last year and did some benchmarking on varioius small units of code. I did some particular investigation into the performance of VB Collections as opposed to arraylists and hashtables to determine whether my "super collection" class (one where the keys can be retrieved for example plus lots of other goodies) which actually uses an arraylist and a hashtable would be feasible or not...
I found .NET blistering fast. It was about 10 times faster than VB in the collection testing and in all other aspects in my client server development it's been all I expected and more.
Gwyn
Saturday, November 22, 2003
" did Borland continue tweaking.optimizing the compiler: is an app compiled in Delphi 5 'faster' (I'll put 'faster' in quotes since it looks like I'm not allowed to say it otherwise ;)) than one compiled in Delphi2 ?"
Marx,
Your best bet to get an answer for your question would be to post your question on the Borland Newsgroups. There are a lot of knowledgeable people on the newsgroups who could answer your question specifically.
Cletus
Saturday, November 22, 2003
Marx, your original question was "why is my VB.Net app slower than my VB6 app." Several people (including me) have given you pointers about how to optimize your code, and have even offered to look at your code to find the performance bottlenecks. Instead, you just keep complaining about VB.Net's supposedly poor performance while stubbornly refusing to do any to improve the your code's performance.
If you genuinely want help, email the code to me or post it online someplace. Otherwise please drop the subject -- you asked for our help, we offered it, and you ignored it.
Robert Jacobson
Saturday, November 22, 2003
If your looking for an in-depth benchmarking essay that compares C#, Java, and C++ (various compilers), check out this free download from Windows Developer magazine (free registration unfortunately required):
Does C# Measure Up:
http://www.wd-mag.com/wdn/webextra/2003/0313/
Guy Incognito
Sunday, November 23, 2003
Marx,
In Delphi 2, are you using an 8 bit ANSI character string (Char or ANSIChar) or a 16 bit Unicode character string (WideChar)?
Since you admit that your code is doing a lot of string parsing/manipulation, the differences in how strings are treated (allocated/referenced) between runtimes (managed/unmanaged) is going to play a significant role in the speed of each implementation.
Guy Incognito
Sunday, November 23, 2003
 Recent Topics
Fog Creek Home
| 
