In my scenario, users are able to upload zip files to a.example.com
I would love to create a “daemon” which in specified time intervals will move-transfer any zip files uploaded by the users from a.example.com
to b.example.com
From the info i gathered so far,
- The daemon will be an .ashx generic handler.
- The daemon will be triggered at the specified time intervals via a plesk cron job
- The daemon (thanks to SLaks) will consist of two FtpWebRequest’s (One for reading and one for writing).
So the question is how could i implement step 3?
- Do i have to read into to a memory() array the whole file and try to write that in
b.example.com
? - How could i write the info i read to
b.example.com
? - Could i perform reading and writing of the file at the same time?
No i am not asking for the full code, i just can figure out, how could i perform reading and writing on the fly, without user interaction.
I mean i could download the file locally from a.example.com
and upload it at b.example.com
but that is not the point.
8
Answers
you need some listener on the target domain, ftp server running there, and on the client side you will use System.Net.WebClient and UploadFile or UploadFileAsync to send the file. is that what you are asking?
It sounds like you don’t really need a webservice or handler. All you need is a program that will, at regular intervals, open up an FTP connection to the other server and move the files. This can be done by any .NET program with the System.WebClient library, doesn’t have to be a “web app”. This other program could be a service, which could handle its own timing, or a simple app run by your cron job. If you need this to go two ways, for instance if the two servers are mirrors, you simply have the same app on the second box doing the same thing to upload files over to the first.
Here is another solution:
HttpWebRequest
Links:
Problems you gotto solve:
How to determine which files to upload to server B. I would use
Directory.GetFiles
in aTimer
to find new files instead of using aFileSystemWatcher
. You need to be able to check if a file have been uploaded previously (delete it, rename it, check DB or whatever suits your needs).Authentication on server B, so that only you can upload files to it.
To answer your questions – yes you can read and write the files at the same time.
You can open an
FTPWebRequest
to ServerA and aFTPWebRequest
to ServerB. On theFTPWebRequest
to serverA you would request the file, and get theResponseStream
. Once you have theResponseStream
, you would read a chunk of bytes at a time, and write that chunck of bytes to the serverBRequestStream
.The only memory you would be using would be the
byte[]
buffer in your read/write loop. Just keep in mind though that the underlying implementation ofFTPWebRequest
will download the complete FTP file before returning the response stream.Similarly, you cannot send your
FTPWebRequest
to upload the new file until all bytes have been written. In effect, the operations will happen synchronously. You will callGetResponse
which won’t return until the full file is available, and only then can you ‘upload’ the new file.References:
FTPWebRequest
If both machines are in the same domain, couldn’t you just do file replication at the OS level?
DFS
Something you have to take into consideration is that a long running web requests (your .ashx generic handler) may be killed when the AppDomain refreshes. Therefore you have to implement some sort of atomic transaction logic in your code, and you should handle sudden disconnects and incomplete FTP transfers if you go that way.
Did you have a look at Windows Azure before? This cloud platform supports distributed file system, and has built-in atomic transactions. Plus it scales nicely, should your service grow fast.
I would make it pretty simple. The client program uploads the file to server A. This can be done very easily in C# with an FtpWebRequest.
http://msdn.microsoft.com/en-us/library/ms229715.aspx
I would then have a service on server A that monitors the directory where files are uploaded. When a file is uploaded to that directory or on certain intervals it simply copies files over to server B. Again this can be done via Ftp or other means if they’re on the same network.
set up keys if you are using linux based systems:
http://compdottech.blogspot.com/2007/10/unix-login-without-password-setting.html
Once you have the keys working, you can copy the file from system A to system B by writing regular shell scripts that would not need any user interactions.