Download Manager deletes download

Xolo · Post by **Xolo** » 13 Feb 2009, 15:32

L.S.

Today I was using the download manager on my personal account on Bubba|Two, and told it to download the following Mandriva Linux DVD ISO file:

Code: Select all

ftp://ftp.nluug.nl/pub/os/Linux/distr/Mandrivalinux/official/iso/2008.1/mandriva-linux-free-2008-spring-dvd-i586.iso

And the download starts, runs (can see the incomplete file build up in my download folder), and completes, but once completed the file isn't in the downloads folder anymore.. nor can I locate it anywhere on disk.

What gives?

I'd like to fix this.

Xolo · Post by **Xolo** » 13 Feb 2009, 19:30

This is very easy to reproduce.. at least on my end.
I restarted the same downloads I have done today by first deleting all files it downloaded, emptying the download list, and inputting them again.
At the moment the ISO I mentioned has already finished, and yet again the file has disappeared.
Torrents run and finish without problems, and so do JPG image files, as far as I can see.

Xolo · Post by **Xolo** » 15 Feb 2009, 05:27

Seems this is a more serious issue than I first imagined, see screenshot:

Apparently the download manager can't handle large (over 1GB) files?.
I hope it has not corrupted the downloads that were running, that would be a real pain as the ones still running were HTTP.. and Bubba doesn't do resume

EDIT:
Nope.. all HTTP and FTP based downloads are gone (no resume, item in download list deleted) after restarting FTD.

Post by **tor** » 15 Feb 2009, 10:33

Hi Xolo,

Sorry to hear about your problems with the DL manager.

I actually can confirm your problems. Downloading the specific URL you mention do really get deleted upon completion. Looking a bit deeper into this it seems like this is a bug in the DL manager. (We get a timeout for some yet unknown cause) Downloading the same file from another server works fine. We will investigate this further. In the mean time could you try another download server or use the torrent dowload of the same, http://torrent.mandriva.com/public/. (The spring edition is found under the old folder.)

Regarding size. The downloader should handle files over 2GB.

Regarding resume. Only torrent downloads has a resume function atm.

/Tor

Xolo · Post by **Xolo** » 15 Feb 2009, 14:28

Hello Tor,

Thank you for replying;
Glad it can at least be confirmed!

Is there any other data you may need?
Pity that it is so specific.. that particular FTP mirror is one local to my country and is much faster for me than torrenting (when invented BitTorrent was supposed to be faster, go figure

).

Is there a tiny chance resume/restart options will come to the Download Manager in the future?

Post by **tor** » 15 Feb 2009, 16:46

Hi again Xolo,

After reading up a bit on this it seems like its more likely that both you and me at home have a somewhat flawed router.

Could you please try to do the same download via http ie, http://ftp.nluug.nl/pub/os/Linux/distr/ ... d-i586.iso instead.

If my suspicion is correct and this works the reason to the previous problem is due to your router disconnecting one of the connections, the controller one, that makes up an ftp session. No data is sent over the control socket and thus your router shuts that one down.

This might also be considered a flaw in the underlaying library, Curl. I will se if we can do anything about this in the next update of the download manager.

Regarding resume/restart/pause etc this is all features that we really want to add to the downloader. But i cant give any promises on this due to many other priorities.

/Tor

Xolo · Post by **Xolo** » 15 Feb 2009, 17:37

Hi Tor,

I'm not sure where to place the router flaw; when I download directly to any computer off ftp.nluug.nl the transfer of one or many files run and complete without being aborted. Only Bubba seems to exhibit this behaviour at the moment.
As a side note I do not use any kind of download manager that sports resume functionality on the local computer.

I added the link you posted to the Download Manager on Bubba and it is currently running steadily; i'll let you know about the end result.

I'm not too familiar with Curl, so I cannot comment there.. I use Wget sometimes when camping in the terminal.

I see that Curl, or cURL as the website suggests.. seems to support more protocols and has more functions. I'll have to play with that sometime.

No problem on the priorities, having them is good. keeps things organised.

Xolo · Post by **Xolo** » 16 Feb 2009, 05:12

Small update:

The download is still running very slowly, which is unusual.
It has reached 54%, and is currently running at 57.9Kbyte/s.
Remarkable is that the torrents have slowed down to a complete halt.
When I touch the link you provided from the computer I write this on, the speed at which it runs averages 760Kbyte/s.
Bubba doesn't like mondays? hehe

Xolo · Post by **Xolo** » 16 Feb 2009, 17:42

Another small update;
It looks like the download is stuck, between the time of my previous post and this one the speed has dropped to a mere 28Kbyte/s and the progress still shows 54%.
The torrents in the list are also still halted.

Xolo · Post by **Xolo** » 17 Feb 2009, 05:26

Nope, it's still stuck at 54%, only now the speed has dropped further to 19KByte/s.
I'm going to go ahead and cancel it, then put it back into the list on Bubba.

Edit:
Umm, never mind that. It seems the download manager itself froze up, none of the cancel buttons work and I can't add anything to the list.
I will instead restart Bubba entirely instead of just the FTD daemon.

Edit 2:
After rebooting Bubba, the same download now averages 810Kbyte/s, quite a difference!

Xolo · Post by **Xolo** » 17 Feb 2009, 07:16

Okay, the download just finished.
The file this time can be found in File Manager! this is good.
Questions that remain are why there is a problem getting this file over FTP, and why this time the Download Manager froze entirely (without FTD dying in the background).
I will do another test for pure FTP via WinSCP and FileZilla on my laptop, and make sure they do not automatically resume on failure so I can see what happens.

Post by **tor** » 19 Feb 2009, 02:55

Hi xolo,

Sorry for the late reply.

Regarding the router flaw. Its correct that other ftp clients does not experience this. I assume that they work around this. The flaw is in our ftp/http library, libcurl, in conjunction with a router that terminates idle connections.

Regarding the freezes/slowdowns while downloading. This sounds like you are experiencing a bug that we have in the system. In rare occasions the network stack gets in some bad state and transfer rates drops dramatically. (This usually only happens when loading Bubba with multiple samba transfers from a windows pc. Which makes this a bit strange) We are currently working hard on a fix for it. If this continues to be a problem for you we would like to hear that. Please contact us on support@excito.com

Regards,

/Tor

Xolo · Post by **Xolo** » 19 Feb 2009, 18:09

Hello Tor,
Sorry for the late reply myself, I have been working and unable to reply till now

I was unable to find any setting related specifically to FTP in the router that was provided to me by my internet provider, even after updating it to the latest firmware available.
I do see QoS (Quality of Service) now available in it's configuration (default was no QoS at all), and I have tried to see if giving FTP priority helped or not, but was unable to find any difference at all with or without.
Unfortunately I will not be able to try a different router, as the model that was supplied to me is a multifunction ADSL Annex A VoIP-PBX/modem/router device I cannot easily find a replacement for without discussing such an action with the provider first.. a regular ethernet router will not work.

The two FTP clients I tried, FileZilla and WinSCP do manage to transfer the file, but dropouts are very rare, and FileZilla's resume functionality will kick in when this is too serious. WinSCP lacks resume but did not display any errors.
During the tests with FTP clients I used 'passive' mode to transfer files. I did not have enough time to also try 'active' mode unfortunately.

Maybe it is an idea to send keepalives? FTP servers support some form of keepalive command to help with transfers over firewalls.
As for Samba? I notice that when FTD is working on things, even when idle, it can regularly peak at 40%+ CPU depending on how large the download queue is.
At those moments there is only 1 file browser pointing to Bubba's samba share in idle (ie. not doing any browsing or transferring, etc.).
This high usage by FTD may be pushing other services down faster than Bubba can reallocate resources?

I'm not sure how often I will encounter issues with the Download Manager and/or FTD, so I will just have to see with time and use.
I will report back if it happens again to the address you provided (and here if so desired).
Meanwhile I will keep working and await official updates.

Thank you for your support