Fog Creek Software
Discussion Board




Are bigger systems always more complicated?

In this thread http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=176798&ixReplies=35 , it is mentioned that SQL Server, while simpler, doesn't scale that well, and vice versa for Oracle.  Does anyone have a good example where the system that scales better or is "bigger" (allows more users/transactions/etc.) and is also equally or less complicated?

Furthermore, it doesn't seem to me like Oracle cares if their database server is complicated, so is making bigger systems less complicated even an issue, or should they be complicated to force people to think about what they're doing?

devinmoore.com
Wednesday, August 18, 2004

I've always wondered if making things more complicated isn't a marketing ploy when selling high tech products to techies.  If techies believe they are the only ones who can understand and use a particular technology, then that technology becomes the "best" technology to use. 

Ewan's Dad
Wednesday, August 18, 2004

+++If techies believe they are the only ones who can understand and use a particular technology, then that technology becomes the "best" technology to use.  +++

You mean like nearly every other industry?  Folks favor changes that make it more difficult for a laymen to approximate what they do.  It's economics.

muppet
Wednesday, August 18, 2004

muppet,

Yes, that's what I mean.  Complexity serves as a barrier to entry as well as leverage when negotiating salary.  That's a huge incentive to choose complexity.

Ewan's Dad
Wednesday, August 18, 2004

No matter what system you choose optimizing huge databases can't be reduced to a Next-Next-Finish Wizard. When you get beyond that stage aren't all systems going to be complicated enough to require trained professionals?

Ewan's Dad, you sound like the CEO who thinks he has to be an expert in all of his employees' skill areas.

Nate Silva
Wednesday, August 18, 2004

Nate -

It's fair to say that complexity is a necessary part, to a degree, of many technologies.

However, there are cases where complexity is unnecessary, and perhaps even added to make a product seem more viable than it is.

Personally I've always hated reporting tools like Crystal.  It seems to me an example of a tool which makes the job its designed for /harder/ instead of easier.

Now let's not have this thread devolve into a flamewar about Crystal Reports... :P

muppet
Wednesday, August 18, 2004

Nate,

I wish I made a CEO's salary! :-)  And, you're right, I struggle to overcome the need to know the technical details of things even if they're outside my realm.  I'm getting better, but I have a long way go.

Ewan's Dad
Wednesday, August 18, 2004

Yes sometimes complexity is just a smokescreen. Remember Novell NetWare? (yes I'm old :-)

Whether SQL Server is "less complicated" for small systems is not really relevant when comparing it to Oracle. I have no idea if the anecdotes from the other thread are true but if large Oracle systems require fewer DBAs than the equivalent SQL Server then isn't it *less* complicated?

Nate Silva
Wednesday, August 18, 2004

Shhh!  Don't point out muppet's logical fallacies.

 
Wednesday, August 18, 2004

Perhaps, instead of big system, you should say complex system.  Big sounds more like size or quantity of ….

Complex <> complicated. 

I got this from google, Complex = “systems that display properties that are not predictable from a complete description of their components, and that are generally considered to be qualitatively different from the sum of their parts.” 

Complicated is something difficult to analyze or understand.

Designing software for complex systems is difficult, but we make things worse if the design is complicated, i.e. does not have good quality.  This might be happening to the industry… we get caught by the complexity of systems, we don’t see the big picture and we are unable to create simpler designs from them.

Cecilia Loureiro
Wednesday, August 18, 2004

Bigger. Better.


Wednesday, August 18, 2004

==>SQL Server, while simpler, doesn't scale that well

Whoever said that is smoking crack.

Visit www.tpc.org and tell me it doesn't scale well.

I'm running a system with about 1/4 TB (Note the "T") that has roughly 5.5 *million* casual users, roughly 3,500 "heads down" users (8 hours a day, 7 days a week) and during peak days is running millions of transactions a day.

That's *very* small potatoes compared to the systems run through the TPC benchmarks. Visit the site for details.

Here -- read this: http://www.tpc.org/results/FDR/TPCC/hp_superdome_win64_030828.pdf

When you're done, come back and tell me it doesn't scale well.  Here's the highlights from that document:

64    Processors (Intel Itanium (64bit)
512  GB RAM (128x4GB) -- That's half a *TERA*Byte of RAM
1,854 Disk Drives for a total of over 34,586 GB of disk -- that's 34 *TERA*Bytes of disk.

This thing averages, over long testing cycles:

    786,000 transaction -- *PER MINUTE*
    Think about it. Thats over a *billion*
    transactions per day.

    
Moral: Don't let anyone ever tell you that SQL Server
doesn't scale. It does. That's a damned impressive
system outlined in that case study. How large do you
want to scale to? I'd say that the pdf describes a
system larger than 99.99999% of folks out there will
ever need.

My large system isn't 1/100th of what is highlighted in the above pdf, but so far, it's scaling with us just fine.

Sgt. Sausage
Wednesday, August 18, 2004

+++I have no idea if the anecdotes from the other thread are true but if large Oracle systems require fewer DBAs than the equivalent SQL Server then isn't it *less* complicated? +++


Err... no.  Oracle requires (required, in MY experience) fewer DBAs because it was more stable and required less frequent maintenance, not because it was simpler.

muppet
Wednesday, August 18, 2004

IBM was, in the bad old days, notorious for making their mainframe systems complicated.

The constant upgrades and incompatibilities kept everyone running in place just to keep the systems going, with no time to think about any alternatives.

And the IBM salesman would always be there to tell you the next thing you needed to buy.

"You need FORTRAN G for this project and for that you need to upgrade to a 360/67 with new disk drives and printers. Just sign here and you will get 500 hours of training credits for your staff."

frustrated
Wednesday, August 18, 2004


>Aren't all systems going to be complicated enough to
>require trained professionals

Yes, they have essential complexity.

frustrated
Wednesday, August 18, 2004

That's SQL server's 64-bit version running on the hp superdome, by the way, lest anyone think they're going to get 512 GB of ram on a desktop or single-cpu 32-bit system, which is where a lot of SQL Server's sit.  By the way, any stats on how many transactions/minute, etc. you'd get on that system with Oracle?  Sure, you can put SQL server on a large box, but the point isn't that it doesn't scale, it's that you have to throw 5 million bucks in hardware at it to get that kind of database throughput.  Personally, I'm shocked that anyone would put windows on a machine like that, given it's notoriety for being deluged with security issues.

devinmoore.com
Wednesday, August 18, 2004

Sorry for the double-post, and the previous rant... long day.  Anyways, what I was initially referring to is how complciated Oracle seems to be to setup and maintain in terms of "simpler" tasks -- i.e. the ones that could be next-next-finish.  Optimizing a database generally requires operator intelligence, I'm talking about the kinds of things that shouldn't require operator intelligence but are forced to by the app provider (I sympathize with that IBM story...)

devinmoore.com
Wednesday, August 18, 2004

"Sure, you can put SQL server on a large box, but the point isn't that it doesn't scale, it's that you have to throw 5 million bucks in hardware at it to get that kind of database throughput. "

Errr, no the claim was that SQL Server doesn't scale, which is patently untrue (yet the proof-through-repetition seems to have no end in sight). As a sidenote Oracle on largely the same machine is comparable in performance.

SQL Server really pioneered low-cost scaling, which is putting multiple SQL Server boxes in a loosely coupled clustered called a federation of servers. It is not the most ideal design that I can imagine (it requires planning and explicit partitioning, and may not equally share all loads), but it does effectively allow one to scale out at a low cost.

Dennis Forbes
Wednesday, August 18, 2004

The systems that people create tend to go through cycles of starting out simple, evolving greater and greater complexity, then a new simplifying principle is discovered, and we return to simplicity.

For example, the Model T was quite a simple car.  That basic approach got bigger and bigger, with more and more things bolted on (electric starters, alternators, automatic transmissions, air conditioning, pollution controls) until the poor engine had dozens of vacuum hoses and wires all over it.  Then we got computer control, and a lot of the wire and vacuum hoses went away.  Now we have a few sensors driving the computer, and the computer drives the gas and spark.

Now software has an essential complexity about it, which we try to manage through hierarchy of functions and limiting data communication between modules.  But currently there is limited science or methodology that supports doing that on a simple basis.  So our systems for doing this keep evolving in the direction of greater complexity.

Thus flowcharts led to Data Flow Diagrams and Structure Charts, which led to Booch and Rumbaugh and Jacobsen's 'UML' (a complicated ball of wax).  We'd like to simplify UML to the point where it we could generate code from the UML -- but currently UML has too little specificity of mapping between diagram and code for it to be used that way.

Ideally, we would work in broader 'chunks' of code.  In the old days, we worked with assembly instructions.  Then we worked with Fortran/Pascal/C language constructs.  Then we worked with Object constructs.

It 'would be nice' to have a language which supported 'bolting together' large chunks -- having a 'Sort' object bolted to an 'Input' object, writing to an 'Output' object, with a 'Transform' object where we actually have to do some work to specify what the transform is. 

The problem with the 'bolting' approach is that we STILL don't have a nice way to specify the interfaces between 'chunks', and it also looks like almost every 'chunk' needs to be customised and debugged every time we apply it to a new problem.  The C++ STL is a nice try toward a set of nice chunks, by the way.

IF we could get to the point where the 'chunks' are well-defined and easy to use, and the 'bolting together' is simple and straigtforward, in theory we could encapsulate the complexity enough to come up with simple, reliable solutions.  So far, we aren't there yet.

AllanL5
Wednesday, August 18, 2004

"Personally, I'm shocked that anyone would put windows on a machine like that, given it's notoriety for being deluged with security issues."

Ugh.

34 flaws found in Oracle database software
http://www.computerworld.com/securitytopics/security/holes/story/0,10801,95013,00.html

http://www.vnunet.com/news/1128907

http://software.newsforge.com/software/04/07/06/1812203.shtml?tid=78&tid=27&tid=31

The last one is interesting. I can quote from several sources reports that by the numbers, Linux isn't necessarily more secure than Windows. What I find intriguing is that even an obviously very pro-Linux columnist has to really hem, haw, dig and grind data to tip the scales just a little bit the other way, which yields *my* point of "it's really not that black and white, and the security of a properly administered Windows system is definitely comparable to that of a properly administered Linux system"

In other words, when you get to the level of counting chads, there is no way you can casually wave your hand and dismiss Windows with "well, it has security issues" unless you have a lot of good, recent numbers in your pocket.

Philo [Microsoft]

Philo
Wednesday, August 18, 2004

It's a strange day when trying to analyze answers and provide context for numbers is considered hemming and hawing.

Obviously pro-Windows folks have their own opinion.

Edward
Wednesday, August 18, 2004

Obviously pro-windows folks were dropped on their heads as small children.  If people are able to support the directions Microsoft is trying to go with DRM and Palladium and their ilk, and not feel even slightly chafed by the rampant civil rights abuses inherent to each, then there is something psychologically wrong with them.  I have constructed a hypothesis which says that Pro Microsoft almost always corrolates with Pro George Dubya.

muppet
Wednesday, August 18, 2004

And you'd be wrong.

Miles Archer
Wednesday, August 18, 2004

I said almost always.  Have you got hard data to refute me?  :)

muppet
Wednesday, August 18, 2004

"Does anyone have a good example where the system that scales better or is "bigger" (allows more users/transactions/etc.) and is also equally or less complicated?"

Linux clusters are replacing supercomputers.

Tom H
Wednesday, August 18, 2004

==>I said almost always.  Have you got hard data to refute me?  :)

And you'd be wrong here too.

Now you've got two "hard" datapoints, matched up with your "hypothesis" (hunch).

The way I see it, that's 2 to 0.

Sgt. Sausage
Wednesday, August 18, 2004

Three nothing.

Philo

Philo
Wednesday, August 18, 2004

*  Recent Topics

*  Fog Creek Home