The Old Joel on Software Forum: Part 4 (of 5) - DLL Hell

DLL Hell - Part 2

A while ago I posted a question about DLL Hell between Windows 2000/XP and Windows 98 and ME.

I have a follow up.

Someone is interested in getting it running on Windows 98 and recompiled the program and library and got the same problem, so DLL Hell it wasn't. He eventually tracked it down to what appears to be a difference between the behaviour of the C++ run time library between these versions, and Linux for that matter.

In my program I create an array of integeres (or shorts, or chars, I forget which) using new and then put data into most of the cells apart from some at the beginning. Under Windows 2000, XP, Linux and probably NT, the array was initialised to 0 which is a valid value for the program hence no problems. Under Windows 98 and probably ME it is left with unitialised gunk which caused a later stage of the program to have a headache.

I hold my hand up and say Mea Culpa, it is a bug, but one that shows up differences in the C++ run time libraries. What do the C++ standards say about new ? Should it initialise to 0 (like calloc()) or should it be like malloc() which explicitely makes no guarantees.

Thoughts gentelmen please. And please no cack arsed comments about Linux this time, this is a C++ run time question not a religious debate.

whattimeisiteccles
Tuesday, February 17, 2004

I am no guru, but this much I know. The new operator does call your constructors and destructors unlike malloc, so there's always a guarantee of initialization with new if you've written your constructor.

Sathyaish Chakravarthy
Tuesday, February 17, 2004

>>does call your constructors ***and destructors*** <<

Sorry for the typo!

Sathyaish Chakravarthy
Tuesday, February 17, 2004

You are right about constructors and destructors, of course. But that isn't the issue here, I am allocating an array of primitive types, not classes so there is no explicit constructor to work with. Therefore the question is different, otherwise it is a basic C++ question that I would be enbarrased to ask.

I would be living in a Citadel of Profound Cluelessness otherwise.

whattimeisiteccles
Tuesday, February 17, 2004

Calling new with primitive types does not guarantee zero initialization, unless you can point to it in the spec. This is not one of those things that you can simply guess about by trying it on different platforms.

If you want the array to start with all zeros, then initialize it as such.

Brad Wilson (dotnetguy.techieswithcats.com)
Tuesday, February 17, 2004

According to the standard (well, the draft standard, which is all I have conveniently to hand) ...

---------- begin quotation ----------
A new-expression that creates an object of type T initializes that
object as follows:

--If the new-initializer is omitted:

--If T is a (possibly cv-qualified) non-POD class type (or array
thereof), the object is default-initialized (_dcl.init_). If T is
a const-qualified type, the underlying class type shall have a
user-declared default constructor.

--Otherwise, the object created has indeterminate value. If T is a
const-qualified type, or a (possibly cv-qualified) POD class type
(or array thereof) containing (directly or indirectly) a member of
const-qualified type, the program is ill-formed;
----------- end quotation -----------

For an array of ints, shorts or chars, that would mean that T is *not* a non-POD class type; ints are about as PODdy as you can get :-).

So it looks to me as if your Win98 system is within its rights to leave the array uninitialized.

Gareth McCaughan
Tuesday, February 17, 2004

The explanation you seek lies deep down in the Windows NT/W2K/XP kernel.

In the Windows NT kernel base there is a background service thread called the "Zero Free Page" thread that is responsible for clearing (set to zero) all free pages in the "free-list" during idle time. Once a page of memeory has been cleared, it is moved from the "Free List" into the "Zerod List".

The kernels memory allocators always allocated memory pages from the "Zero List" first if available, and then from the _gunky_ "Free List"

The Kernel memory routines can be explicitly told to zero any pages it allocated before returning to the caller, so the "C" run-time can have an influence on this depending on how it is implemented; i.e. Release vs. Debug builds can be different.

The Windows 9x kernel base does not have (AFAIK) this "Zero Page" thread

I have no clue about Linux..

this "Zero Page" thread is part of the Windows C2 Government certification requirements... but that is another big ball of wax!

I'm writing this from memory so discrepencies may exist in the above.

Heston T. Holtmann, B.Sc.Eng
Tuesday, February 17, 2004

What purpose does the "Zero Free Page" thread serve? I can see why having uninitialised memory set to 0 is a good idea, but only if it's __guaranteed__ to happen.

From your description of the current setup, badly-written programs will work happily until Windows becomes stressed, at which point they'll start failing. Bit difficult to track down and fix bugs like that.

Adrian Gilby
Tuesday, February 17, 2004

That must be the shoddies programming style. Allocating memory and "hoping" it's 0, most of the time, sort of.

OP: if you don't want to worry about clearing memory after each "new", maybe you can wrap your primitive types in classes with constructors that do the job.

Say "Int" instead of "int".

Machine-code wise, it should compile to different code (a "loop" rather than "rep stos") but the speed should be roughly equal -- in both cases, it's just writing to memory.

Alex.ro
Tuesday, February 17, 2004

... or just overload the "new" operator :):):)

Alex.ro
Tuesday, February 17, 2004

It is not always a case of operating systems acting differently, sometimes the same operating system and compiler will behave differently with different optimization settings. On linux with gcc, compiling with no optimizations seems to set everything to 0, while higher optimizations put in the random junk that happened to be in the memory address at the time.

Solaris with gcc seems to always put in 0's regardless of optimization settings.

Oren Miller
Tuesday, February 17, 2004

The pages *are* guaranteed to be zeroed before the next process gets them. This is to protect information in the process which freed the memory, not for the convenience of the programmer who next allocates that page.

You may however free a page, leaving your own garbage in it, then get it back, still containing that garbage, so you can't guarantee that the memory initially contains 0's.

Edwin
Tuesday, February 17, 2004

Edwin is right. You CANNOT rely on the contents of any memory you just allocated with malloc(), operator new(), etc. Sometimes through coincidence it will be zero-filled, but not always.

If you really do need zero-filled pages, you can get those by mapping anonymous pages explicitly. (Linux does this with mmap() on fd -1, I'm sure there is a way in Windows)

The only thing that is guaranteed to be zero-filled is *static* data. e.g. if you have in your C source code:
int bar; int foo[100];
You are guaranteed that all bytes within foo[] and bar are zero at program startup.

In C++ if you have a constructor, that will be run inside of operator new(), so it may set the value to something. Constructors for static objects are called before main().

Dan Maas
Tuesday, February 17, 2004

Adrian,
If the zero page thread is indeed part of "Windows C2 Government certification requirements" (I'm not doubting Heston, but I have no basis for the fact myself, other than his word), then it's not about protecting programs from Murphy, but from Machiavelli. In other words, it's not meant to protect you from a poorly written program that doesn't initialize its storage, but from a program that tries to get sensitive information by allocating a large amount of memory and rooting around in it for fragments of information.

Edgewood
Tuesday, February 17, 2004

The 1998 ANSI C++ specification requires "default" initialization; it's described in section 8.5.

I'm fairly certain it was not required in older versions.

Standards compliance is fun.....

Jason McCullough
Tuesday, February 17, 2004

Yeah, but how is "default" initialization defined for int? I'm betting it's "whatever happens to be in memory at the time".

Chris Tavares
Tuesday, February 17, 2004

The only default initialisations that make any sense are all zeroes or complete garbage. I would favour the latter, in fact I would prefer garbage that includes lots of high values as that would have shown up my problem very quickly.

While I understand the W2K/XP zero page stuff, I don't understand why the Linux C++ run-time and/or OS also appears to put zeroes into the newly allocated memory. Perhaps it too has the concept of zeroing out of memory for security certification ?

I thought it was a bit unfair of one of the earlier posters to imply that I wrote shoddy code, has he never made a mistake ? If you don't make mistakes, you don't learn.

whattimeisiteccles
Wednesday, February 18, 2004

Man, I thought they are now teaching people to ALWAYS initialize their variables, no matter what...

Guess not.

T.J.
Wednesday, February 18, 2004

It's not you writing shoddy code, it's the style of new-ing and not zeroing that's shoddy.

Alex.ro
Wednesday, February 18, 2004

In an earlier incarnation of the software the array didn't need to be initialised since every cell was given a value later on. This was in a simple for loop that covered every cell, the data was non-trivial. In such cases initialisation is unneeded and waste of CPU and machine code.

Later I changed an algorithm that mapped two rows of data to one, and that meant that I had a row without data at the bottom, in my haste to get it working I didn't spot it.

I consider myself a very careful developer, but somethimes these things slip through, even with the best of intentions.

However I do wish you had kept your comments relevant to my question and not commenting on my or anyone elses ability or otherwise.

whattimeisiteccles
Wednesday, February 18, 2004

Hey man,

If you look at my post, there is a *general* remark that relying on new returning 0's is shoddy, and then a friendly comment to *you*, without any trace of criticism.

No worries, I'm pretty jumpy too :) And yes I write shitty code par excellence ;)

Alex

Alex.ro
Wednesday, February 18, 2004