Bash download all files from partial url with forbidden

Active 5 years, 6 months ago. Viewed 4k times. Ani Menon Manish Manish 23 1 1 silver badge 8 8 bronze badges. Honestly, who has upvoted this awful question.

There is no specific problem, they are just asking someone to write a script for them — Add a comment. Active Oldest Votes. Also provide the command to run. Ani Menon Ani Menon I understood scheduling it on cron, I need the complete bash script.

So that I can run it in my terminal also for previous dates also. It only takes a minute to sign up.

Connect and share knowledge within a single location that is structured and easy to search. I only have bash. If you have bash 2. Paste the following code directly into a bash shell you don't need to save the code into a file for executing :. Source: Moreaki 's answer upgrading and installing packages through the cygwin command line?

Taking the " just Bash and nothing else " strictly, here's one adaptation of earlier answers Chris's , 's that does not call any external utilities not even standard ones but also works with binary files:.

We deal with NUL bytes with read -d ''. It reads until a NUL byte, and returns true if it found one, false if it didn't. Bash can't handle NUL bytes in strings, so when read returns with true, we add the NUL byte manually when printing, and when it returns false, we know there are no NUL bytes any more, and this should be the last piece of data.

Tested with Bash 4. The kB wget binary took about 5. In comparison, 's cat-solution finishes in less than 0. Not very surprising, really. This is obviously silly, since without using external utilities, there's not much we can do with the downloaded file, not even make it executable. So you can also use SSH to upload to it. Which is functionally equivalent to downloading of software packages etc. As shown in this answer , you would execute the following on your local machine to place a file on your remote headless server:.

The disadvantage of the above solution compared to downloading is lower transfer speed, since the connection with your local machine usually has much less bandwidth than the connection between your headless server and other servers. To solve that, you can of course execute the above command on another server with decent bandwidth. To make that more comfortable avoiding a manual login on the third machine , here is a command to execute on your local machine.

See the explanations below for the reason. The command will ssh to your third machine intermediate-host , start downloading a file to there via wget , and start uploading it to target-host via SSH. Downloading and uploading use the bandwidth of your intermediate-host and happen at the same time due to Bash pipe equivalents , so progress will be fast. For the -T -e none SSH options when using it to transfer files, see these detailed explanations.

This command is meant for cases where you can't use SSH's public key authentication mechanism — it still happens with some shared hosting providers, notably Host Europe. Viewed 2k times. PersianGulf 2, 4 4 gold badges 39 39 silver badges 61 61 bronze badges. Tom Iv Tom Iv 1 1 gold badge 3 3 silver badges 19 19 bronze badges.

Is there a realistic maximum? Why not just loop to a high number? If it fails, it'll fail quickly and you can just ignore it. Add a comment. Active Oldest Votes. Community Bot 1 1 1 silver badge. Bruce K Bruce K 5 5 silver badges 13 13 bronze badges. A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites.

Referer sic : when clicking a link on a web page, most browsers tend to send the current page as referrer. It should not be relied on, but even eBay failed to reset a password when this header was absent.

So yes, it may happen. It can be set in curl with the -u user:password or --user user:password option. User-Agent: some requests will yield different responses depending on the User Agent. This can be used in a good way providing the real download rather than a list of mirrors or in a bad way reject user agents which do not start with Mozilla , or contain Wget or curl. Improve this answer. Lekensteyn Lekensteyn Another really perverse possibility would be that the server for some reason was configured to return instead of on success.

This gave me the clue I needed. After trying cookies, I found the referrer to be the problem now, if only that could be spelt properly!!! If it is still failing in wget try adding --auth-no-challenge. Works like magic. The --referer option solved my problem. This is what worked for me too, but a shorter user agent name is also probably fine, e.

This helped immensely, especially the tools in Chrome. When I tried in firefox, the request header after the was all I could see. In Chromium I could see both and this gave me the info to solve the problem. Chrome has a similar option.