The Old Joel on Software Forum: Part 4 (of 5) - Design Question (For a FTP Scheduler)

Design Question (For a FTP Scheduler)

I'm working on a little program which uploads
and download files using the FTP Protocol :

The 1st thing I do is that I build a list of files which must be downloaded/uploaded.

there are currently a dozen of different FTP server set up.

I've got a hard time to choose how to handle
the download/upload

1/ One file at a time
2/ connect to 12 different FTP (12 different threads) server and start downloding 12 files at the same time...

The Target platform is Windoze ....

What woud you advise me solution 1 or 2 ?

Linus Ericsson
Wednesday, April 7, 2004

The first principle of programming is to get the basic stuff working first and in that spirit you should choose option 1 and get it working first.

Option 2 would require multiple logins to the same ftp server and I am not sure that that will be taken very well unless it is your own FTP server.

Code Monkey
Wednesday, April 7, 2004

CodeMonkey :

Option #2 is connection to 12 different FTP servers
(not the same IP)

Linus Ericsson
Wednesday, April 7, 2004

Depends on your bandwidth requirements.

Otherwise, flip a coin.

Alyosha`
Wednesday, April 7, 2004

>Option #2 is connection to 12 different FTP servers
(not the same IP)

Ah! Then having 12 threads make sense and I think in that case it should be not that difficult as long as you make sure that the local filenames you are downloading to do not end up trampling on each other so perhaps having a thread id or ftp server ip as part of the filename might help there.

Lookup WaitForMultipleObjects and SetEvent to see how to syncronize the whole shebang.

Code Monkey
Wednesday, April 7, 2004

This question is more difficult than it sounds. It depends on the bandwidth available for each server, and the bandwidth available to the client.

If one download can fully utilize the available client bandwidth (say with a dialup), that starting multiple downloads concurrently will probably slow down the total time.

christopher baus (www.baus.net)
Wednesday, April 7, 2004

> What woud you advise me solution 1 or 2 ?

Another possibility is one thread/one connection per client, and if you want 12 connections then run 12 simultaneous client processes.

Christopher Wells
Wednesday, April 7, 2004

> Another possibility is one thread/one connection per client, and if you want 12 connections then run 12 simultaneous client processes.

Then of course the increasingly popular event driven, non-blocking, single threaded option. Thanks to regular poster here, Ori Berger, this is my current favorite.

christopher baus (www.baus.net)
Wednesday, April 7, 2004

Actually, this question also depends on the latency characteristics of the various connections.

The throughput of a TCP connection is
min(bandwidth, latency * window_size).

With broadband connections, quite often you are limited by the latency/window size factor. Although there are hacks to increase window size, real operating systems should not need them.

Point being, it is possible to bring up a connection to an FTP server, then bring up another connection to the same server, and not have the throughput of the first connection suffer at all!

You will need to monitor total throughput of all concurrent connections, and add load only as long as throughput is increasing. You can experiment with adding multiple connections to the same server, while discovering the latency/window size limits for that server. (Basically, if two connections to one server are slower than 2*throughput of one connection then you have reached the limit for that server.) Once you saturate your link, you stop adding connections.

Of course, as others have said, get the basics working first.

David Jones
Thursday, April 8, 2004

For relevant source code see

http://en.typsoft.com/index.php

http://www.codeproject.com/internet/serversocket.asp

Those are my principles, and if you don't like them... well, I have others." - Groucho Marx

Code Monkey
Thursday, April 8, 2004