Fog Creek Software
Discussion Board




Designing a huge documentation system [long]

Hi all, it's me again, the librarian of http://www.reactos.com/ . I've started designing our documentation (something equivalent to MSDN) in more depth, and here's what I've come up with (originally an e-mail to our coordinator), and I'd like to hear your comments and - especially - your criticism:

----

No numbering. Numbering is faster to implement, but weak. For example, URLs for our current RosDocs tutorials tend to break at every rebuild

The documentation should be, instead, based on multiple indexes, rather than hierarchy. This is to allow for a certain freedom of movement. For example, the documentation for the Win32 user interface could start as part of the documentation for the Win32 platform, and then gradually outgrow it, and need a whole section of its own. It will need to be moved out of Environments-> ROS32, into User Interface Development, but this shouldn't break links to its Overview or Examples pages, or worse to its Reference pages. Therefore, we need a mix between indexing and hierarchy. An example, with a simplified outline:

Environments
ROS32
  Graphical User Interface
  Overview
    Windows
    Message Queues
  Examples
    Creating a Message Loop
    Creating a Window
  Reference
    Functions
    CreateWindow
    GetMessage
    Structures
    CREATESTRUCT

Now, we add some indexes:

Topic                        Index[key]
-------------------------------------------------------------------------------
Environments                  section[environments]
ROS32                        section[ros32]
  Graphical User Interface    section[gui]
  Overview                  overview[gui]
    Windows                  overview[windows]
    Message Queues            overview[messagequeues]
  Examples                  example[gui]
    Creating a Message Loop  example[messageloop]
    Creating a Window        example[createwindow]
  Reference                  reference[gui]
    Functions                functions[gui]
    CreateWindow            reference[CreateWindow] keyword["CreateWindow"]
    GetMessage              reference[GetMessage] keyword["GetMessage"]
    Structures                structures[gui]
    CREATESTRUCT            reference[CREATESTRUCT] keyword["CREATESTRUCT"]

(note: all topics can have one or multiple "keyword" links - and they can, and will be generated automatically - I've merely shown the easy ones only. The "keyword" index is what you see in Windows Help's Index page)

The concept is that hierarchy is to be obtained through symbolic links - it should be merely presentational, rather than structural. With some limits, and grasp on reality, of course - religious wars are of no use. For example, each topic should have *at least* "parent", "first child" and "next sibling" links to its surroundings, using each topic's primary key. Since it's needed for the hierarchical view only (and the hierarchical view is not mandated) and since it can be easily calculated, this information doesn't need to be stored into the global database - it can be generated on the fly, upon installation. So is the whole "section" index, exclusively finalized to the hierarchical view, and whose topics' contents must be entirely auto-generated, save for a short introduction

A "section" entry contains a template specifying the section's own content, and where and what subsections are to be inserted. Every time a topic of any type is inserted in a super-section, its "parent", "next sibling", etc. links are generated, and the super-section's "first child" is generated too. If a topic is a child of many parents, all will be stored, but only the first used (i.e. shown to the user) - this is to allow deleting a section and automatically reparent its subtopics, if their reference count hasn't dropped to zero yet

Wow. OK, we need an example, and some implementative details. In pseudo-XML, the above example is something like this:

<!-- book/developer.xml -->
<page>
<title>The ReactOS Developer's Book</title>
<section key="environments"/>
</page>

<!-- section/environments.xml -->
<page>
<title>Environments</title>
<section key="ros32"/>
</page>

<!-- section/ros32.xml -->
<page>
<title>ROS32</title>
<description>ROS32 is a ReactOS environment compatible with Microsoft Win32</description>
<section key="gui"/>
</page>

etc.

<!-- functions/gui.xml -->
<page>
<title autogen="true"/>
<description autogen="true"/>
<sort type="alpha">
  <select index="reference" where="'gui' in sections">
</sort>
</page>

etc.

<!-- reference/GetMessage.xml -->
<page>
<!-- extracted from reactos/lib/user32/message.c -->
<title>GetMessage</title>
<sections>
  <section name="man3">
  <section name="gui">
</sections>
<!-- etc. -->
</page>

I know I'm terrible at designing file formats (for example, content and structure have to be further decoupled - the "sections" part of reference pages should be separate from actual content, for reasons I'll explain later), so don't worry, this is just to give you an idea. I'm studying DocBook (like we decided initially), because they'll surely have done a better work than me, and I want to settle with a well-known format. And the indexing could be done better than with files scattered in multiple directories, probably (on second thought, simplicistic as it is, it can be versioned flawlessly by CVS, since it doesn't require any renaming/moving. And power users will be able to customize the documentation by overriding sections, simply creating aptly-named files in the right directories. And it's easy to compile into a database)

I'll design the rest of the outline with this concept in mind, to see how it holds with an insane amount of documentation, like we'll eventually need to write

BTW. We get, on the user side, the best of both worlds: we can help newbies getting started, with a rational, easy to understand hierarchy; and we can help professionals to get their job done fast (think "man CreateWindow", "howto ROS-on-BOCHS", etc.), and cut disk space requirements if all they need is a plain reference

Also note how hard-core developers are completely shielded from the complexity of the documentation system, and you know how important is not to alienate developers, especially in an open-source project. They just write specially-formatted comments (that don't need to be in XML, a simpler markup will do) before functions, data structures, etc., that an automated tool later extracts, converts into DocBook XML and puts in the "reference" index. What place will the page actually have in the tree will be decided by someone else (namely, me), who will write a wrapper for the reference page (let's say GetMessage.meta.xml for GetMessage.xml), containing the cross-references ("see also" links, and a list of relevant sections where the function should appear) needed by the hierarchical help, and reference it from the appropriate sections

----

That's it. Do you like it? do you think it could be done better? suggestions?

KJK::Hyperion
Monday, December 16, 2002

> That's it. Do you like it?
> do you think it could be done better?
> suggestions?

Suggestions? Make your posts a *lot* shorter so that I might actually want to read them.

But seriously, while I do waste alltogether too much time at JoS, I don't really have the time to read detailed design documents.

Also, to be brutally honest, your post strikes me as the "please do my homework for me" type of post. I love to discuss general software development topics, but I'm not interested in doing a peer review of somebody's detailed design.

Hmm. That was a bit meaner that I wanted to be. But I'm going to let it stand.

Bill Tomlinson
Monday, December 16, 2002

I totally agree with Bill.

The only problem with comprehensive posts is that too many of us are really lazy, and do not want to slog through a long technical post (read my code anyone?).

My first instinct when I started reading your post, was to scroll down and see how other readers had responded, and what direction the discussion was taking.

Maybe another thread on the psychology of boards like this, and what the optimum length of a post, in order to stimulate discussion.

I notice that even when Joel posts a new long article, it is rarely discussed in its entirety. Different readers take pot shots at different bits.....

what you might want to do is have your article posted on a site somewhere, with generous use of whitespace (read better formatting). Then people will find it easier to read than they can here (no html). Until they read it, they will not respond.

Phew.. that was a long post.

tapiwa
Monday, December 16, 2002

Respectfully, and without any mean-spirit intended, I have to agree with Bill's post.

Requests for general direction or guidance appear to be more in keeping with this group's posting customs than requesting a review of even a partial tech spec you're going to use in your job.

Alternatively, it **might** fly here to post a request for folks who might be willing to do such a review off-line. Though you're still asking folks to invest some time, presumably for free, so you might not want to count too much on getting a significant number of 'pro-bono' volunteers.

Good luck in solving your problem, but as I said, I think Bill's correct.

anonQAguy
Monday, December 16, 2002


It's also expecting a bit much for outsiders to comment on your design when you haven't explained your requirements first.

Of course, if you don't have any requirements, then ignore this post and full speed ahead.

Bruce Rennie
Monday, December 16, 2002

Damn I got to the bottom, and found everyone already here.

Admittedly I haven't scanned more than the first half of the original, but the core seems to be at the beginning, no numbers, no structure except presentational.

Its going to be a bugger to maintain.

Simon Lucy
Monday, December 16, 2002

> Suggestions? Make your posts a *lot* shorter so that I
> might actually want to read them.

I come from an Usenet background - the Land of Whining, home to the pickiest Netiquette advocates on Earth. Putting "[long]" in the subject to me means "pack some snacks, it will be a long ride", and is considered perfectly acceptable even in the Land of Whining

I don't know a way to be more clear than that about the length of a post, and I didn't want to rewrite this in a "for dummies" edition - it was enough work to write it the first time

> Also, to be brutally honest, your post strikes me as the
> "please do my homework for me" type of post.

That's because you didn't read, but rather skimmed through it

Actually, what I like in my new role as the ReactOS librarian is that I only have to report directly to the project coordinator and nobody else, so I can do "homework" *my* way

> I love to discuss general software development topics,
> but I'm not interested in doing a peer review of
> somebody's detailed design.

OK, I'll be damned for having a clear idea, for once. Next time I'll try smoking some pot first, just to have a more blurry vision and please you

KJK::Hyperion
Monday, December 16, 2002

> what you might want to do is have your article posted on
> a site somewhere, with generous use of whitespace
> (read better formatting).

<whine>it's the forum's fault, not mine!</whine>. Seriously. I have written way longer, and more difficult to read and reply to, posts on comp.unix.programmer than this code-free and relatvely technicisms-free piece - and got actual replies. I didn't expect  this kind of reaction

I guess this is similar to the Slashdot RTFA (Read The Fine Article) syndrome, where people comment on the story (generally misinformed and poorly spelled flamebaits) rather than the linked article, because it's too long (I know - I'm affected too)

OK, maybe it wasn't a great idea, after all. Sorry to everyone

KJK::Hyperion
Monday, December 16, 2002

> Requests for general direction or guidance appear to be
> more in keeping with this group's posting customs than
> requesting a review of even a partial tech spec you're
> going to use in your job.

No job. I don't get any money for this (well, not until we go commercial - but it isn't going to happen soon, nobody has offered to sponsor us yet). Actually, I *lose* productive hours (still no job involved - I'm an unemployed student) on this

And you'll see it's not a specification, but rather a really verbose informal explanation

KJK::Hyperion
Monday, December 16, 2002

KJKJ, I agree with a lot of what you are saying, but you seem to miss the point a lot of people were saying. I too, do not like the slashdot first-post-syndrome, before one RTFA.

Having said that though, when faced with a daunting post, I would rather only do the work if I think the topic/discussion might interest me.Albert D Kallal rarely makes short posts to the forum. I do read his comments though, because it does not make me work. A cursory glance at your post though, said WORK.

I suppose one of the biggest differences between this forum, and usenet, is that with usenet, the audience was more specialized, and so if you posted on alt.librarians.technology?, you would find people keen to engage at that level, formating or not.

A lot of us here just pop in every now and then, to have a cheerful banter, and then knuckle back down to work. The less effort I therefore have to make here, the more I will engage in the discussions.

Just my £0.02

tapiwa
Tuesday, December 17, 2002

Well written, tapiwa.

FWIW, KJK, you're also being rather aggressive.  That doesn't help.

As with other folks, there's just too much information there for me to synthesize, and you're not too specific about the sort of advice you want.  It's a bit like asking game designers for advice, then talking about the game's hardware requirements.

Brent P. Newhall
Tuesday, December 17, 2002

KJK. Your post was entirely acceptable.  If I could offer substantive comments on it I certainly would. 

But how dare you FORCE your other respondants to waste their time.  What a bunch of old grannies.

Jeremy
Tuesday, December 17, 2002

KJK i feel has a point, [long] in the post means this type of post. The first 3 or 4 replies just beat him/her up for posting a 6 pager. I you all were not ready to read a [long] post then you should not have clicked the link. Sorry to be stale but wow. I do not think that was fair...

Jason Hawryluk
Saturday, January 04, 2003

*  Recent Topics

*  Fog Creek Home