Web Downloading
[ Home ] [ Back] [ Next ]
How to download from the Web

Introduction 1. Introduction
Tools 2. Tools
Connecting 3. Connecting
Joining 4. Joining
Viewing 5. Viewing
Resources 6. Resources

1. Introduction :-
To download files from a website, you will need access to the website. So you'll need a program that will allow you to download files. The web browser is the most commonly used program for doing this, one of Opera, Internet Explorer, or Netscape with the security settings applied for surfing safe is fine. Also you can use a type of program known as a Downloader since it allows for the automatic scheduling of file downloads.

Files that you download will be one of three general types: Encrypted, Compressed, and Data.

Encrypted files are files that have been encoded in such a manner as to render their original contents invisible. Although there are many programs available for doing this. The accepted standard is PGP. Files in this format will have the extension .pgp.

Compressed files are files that actually contain many other files and are encoded in such a way as to reduce their overall size. Compressed files will need some sort of program to uncompress them. A compressed file may contain executables or data files. The most common type of compression is ZIP compression. Files compressed in the ZIP format will have a .zip extension.

Data files, i.e. text files(.txt), pictures (.gif and .jpg), movie files (.mpg and .rm), music (.wav and .mp3),etc... need some other type of program in order to be useful. In some cases this program may also need to be downloaded. In other cases you may already have the program on your computer. As an example, text files can be viewed with the program Notepad which comes with Windows 95

Downloaded files can each be possibly encrypted, or compressed or both. So recognizing that you may need to perform one or all of the the following operations Decryption, Joining or De-compression is part of the skill in successfully downloading and viewing files from the web.

2. Tools:-

Anonymizing Tools: There are many programs available to assist you in remaining anonymous. One such program is SocksCap32 which will allow you to anonymise yourself or sockisfy your connection to network news and other protocols such as irc ftp etc. It is a versatile and valid alternative to using proxy servers directly. Download and install it from here . How to setup Sockscap is to be found in the Socks Basics faq.

Downloader/Scheduler: A utility such as Getright from Headlight Software or AutoFTP from PrimaSoft that will let you line up your downloads to perform at a time when you dont need to do other things on your computer. Alternatively you can connect to a web page catalog its contents and selectively download items of interest leaving you free to pursue some other net business. Another valuable feature of these tools is "auto-resuming" so if your net connection is lost, these programs will pick up where they left off, avoiding losing the 50% of a large file that vanishes when the Windows downloader loses its connection.

Opera Browser This is one of the smallest and safest web browsers out there. Its not perfect then either are any of the others. Available here: Opera Browser

File Joining Tools: Many items uploaded to the web are split into parts, to reverse this process and join them some application is needed. A good program to do this is MasterSplitter which lets you Join split files. MasterSplitter is a simple to use program that allows the use of a batch file which can be used to automatically join the parts together again. Download and install it from here. MasterSplitter

Extraction Tools: Two programs that allow you to extract or inflate files that have been compressed and possibly password protected are:

  • WinZip
  • WinRar
You can download and install WinZip from here WinZip

You can view a WinRar download faq here.

Decryption Tools: The industry standard for distributed file encryption is PGP. This allows you to decrypt any password protected files that you have downloaded from the web. You can download and install PGP from here MIT's pgp distribution.


3. Connecting:-

Step 1: Anonymize your browser or downloader either via SocksCap32 or some web based proxy and check that you are anonymous!

Step 2:
Using your browser, link to the upload page, or paste it from the textfile into your browsers addressbar. This should display either the contents of a folder or an individual file. If the file is a picture it will be shown directly, similarly with movies they can play automatically, but it is better if you use a downloader for these. With files such as compressed .zip files the Windows downloader will automatically start downloading. Once you have the file then you can right click the picture or zip and save it to disk.

or

Paste your link into your Downloader url box as illustrated next with GetRight.

Connecting

GetRight will itemize the page and present all the files to you in a dialog, then you can pick the ones you wish to download and either proceed to download immediately or schedule it for downloading later.

GetRight has as a proxy field option that can be switched on or off, so if you wish to download anonymously enter a proxy (ftp or http etc) and switch it on.


4. Joining :-
How files are joined depends on the method used by the person who uploaded them initially. Files that are intended to be joined using another program tend to have digits appended to the filename ie filename.mpg.001.pgp, filename.mpg.002.pgp etc, and in some cases an additional extension is evident due to another operation being applied, here its .pgp since each part has been encrypted. Occassionally files are renamed and their extension altered, say from file.mpg to file.txt because the Host has banned .mpg uploads. So once all the articles have been retrieved, some common operations which can be performed to complete a download are Decryption, Joining, and/or Decompression and Renaming.

The Join Sequence:
This is normally determined by the reverse order of the file extensions on the target parts, running from right to left. And one or more of the following operations are normally used.

  • Decryption
  • Decompression
  • Joining
  • Renaming
So for a set of files that started with one named hanson.mp3.001.pgp Then running from right to left the Joining Sequence would be:
  1. Decrypt all the file parts using PGP
  2. Join all the files

Example Join Sequence:

Step 1. Decrypting the parts:

Highlight all the parts, then right click, selecting PGP using the Decrypt option.

Highlight the parts. Decrypting the parts.
The PGP password box should appear so just enter your password.

Enter 
the Password

Step 2. Joining the parts:

Run Mastersplitter and select the Join tab then using the "Browse" button search for the file with the .000 sequence number, here we search for "Hanson.mp3.000". Then click on the Join button to complete the file join. Joining the parts.

Now for a set of files that started with one named hanson.mp3.001.zip Then running from right to left the Joining Sequence would be:
  1. Decompress all the file parts using WinZip
  2. Join all the files
For a set of files that started with one named hanson.mp3.001.rar Then running from right to left the Joining Sequence would be:
  1. Decompress all the file parts using WinRAR
  2. Join all the files
A variation on this theme is for the poster to Encrypt or Compress the file first, then to Split the file into parts.

The simple case is that of a file being Split followed by Encryption. So for a set of files that started with one named hanson.mp3.pgp.001 Then running from right to left the Joining Sequence would be:
  1. Join all the files
  2. Decrypt the resulting file using PGP
Now for a set of files that started with one named hanson.mp3.zip.001 Then running from right to left the Joining Sequence would be:
  1. Join all the files
  2. Decompress the resulting file using WinZip
For a set of files that started with one named hanson.mp3.rar.001 Then running from right to left the Joining Sequence would be:
  1. Join all the files
  2. Decompress the resulting file using WinRar

Renamed Parts:
You can Rename all the files to eliminate the numerical suffix. An example might be hanson.mp3.pgp.001 renamed to hanson001.mp3.pgp. So for a set of re-named files that started with one named as hanson001.mp3.pgp Then running from right to left the Joining Sequence would be:
  1. Rename all the files in ascending numerical order i.e. from .pgp.001
  2. Join all the file parts
  3. Decrypt the resulting file using PGP
And for a set of files that started with one named hanson001.mp3.rar Then running from right to left the Joining Sequence would be:
  1. Rename all the files in ascending numerical order i.e. from .rar.001
  2. Join all the file parts
  3. Decrypt the resulting file using WinRAR
Finally for a set of files that started with one named hanson001.mp3.zip Then running from right to left the Joining Sequence would be:
  1. Rename all the files in ascending numerical order i.e. from .zip.001
  2. Join all the file parts
  3. Decrypt the resulting file using WinZip

Batch File Join:
Files posted in parts, are sometimes accompanied by a .bat file. This method allows you to join files without using a joining program.

As before download and save all the parts to a named folder on your hard disk. After identifying the Join Sequence . One of the files will have the file extension .bat. Highlight and Double-click on the .bat icon. A DOS window will open and the file will start to be joined. A message will appear in the DOS window once joining has been completed. Just close the DOS window. Apply any further Join Sequence operations, then all you need do is check that your file is working fine then you can delete the parts.

Custom Join:
Custom join usually applies when the first file, a .bat file, or possibly even the joining program is not available!

Use Another Splitter
The simplest solution is to download a copy of a splitter program. Then select an arbitrary file that is similar in size to the sum of the splitted parts. Next Copy and Paste this file renaming it with the same name as your splitted file. Then run the split program inputting your "copied and renamed" file as the file to split, select the same number of parts and request that a .bat file should be created. Now delete all the file parts and the copied file then use the resulting filename.bat to join your file parts.

Rename Another .BAT File
Alternatively use an ascii type editor to edit and change an already existing .bat file, Notepad is fine here, but one with a "search and replace" command is better. First rename the .bat file to one with a .txt extension i.e. "filename.dat.bat.txt" then open it with your ascii editor.

Add Join Lines
Now check that the .bat file contains the required number of join lines. If not then use simple copying and pasting of lines followed by a sequential renumbering till you have the required total.

Rename The Original File
Then if the name of the file within the unedited .bat file is say filename.dat.001 and your split file name is filename.mp3.001, then use the search and replace facility to replace "filename.dat" with "filename.mp3".

Save and Join
Check to see that everything looks ok, then save your filename.dat.bat.txt, next rename it to filename.mp3.bat, now run it to join all your parts, verify your file works then clean up before deleting all the parts, but keep your filename.mp3.bat, since this can be used in some future join excercise.

Manual Join:
Manual join usually applies when you have a list of uu encoded files, or encoded using some other method, usually from news, which refuse to be auto-joined or have syntax problems.

First try using Fastcode32 or WinZip which both allow the decoding of files encoded using mime, uu, binhex, etc. This helps sort out the valid parts.

Click on the MS-DOS Prompt shorcut or menu option and navigate to the folder containing the file parts. Now you can use the COPY command as in the following example:

COPY /B File1+File2+File3+FileN DestFile.ext Where DestFile.ext is the name of the destination file name with the appropriate extension (.MPG for MPEG movies, .MP3 for MP3 audio files, .JPG or .GIF for pictures, etc )

The option /B is necessary because you are joining Binary files. If you try to join binary files without the /B option, they will be treated as ASCII (pure text) files, and the copy process will end as soon as the computer will meet a byte with a value of 0, because it will consider it as an End-Of-File (EOF) marker.

If you have several chunks and you can't put all of their names in a single MS-DOS command line (there's a limit of 127 characters for a single command line), you'd better join them in larger chunks first, as in the following example:

COPY /B File1+File2+File3+File4+File5 File1-5

COPY /B File6+File7+File8+File9+File10+File11+File12 File6-12

Then ...

COPY /B File1-5+File6-12 Track.mp3

Missing Parts?
A quick fix for this is to nominate the previous file as the next file to be joined. So if filename.mp3.009 is missing then first copy filename.mp3.008 renaming it to filename.mp3.009 if neccessary, then join and verify as before.


Viewing Files :-

Playing:
Microsoft Windows MediaPlayer handles most audio/video formats, such as MPEG, AVI etc. Be sure to have the latest version installed.

Sound:
If the file is an MP3 you may want to use a specific player as WinAmp.

Pictures:
Pictures in almost any format can be opened with ACD-See32 (shareware) that will work from inside forte agent as your picture viewer of choice. For a larger selection of graphic formats you can try a professional application such as Corel PhotoPaint. PhotoPaint will allow you can to edit pictures and adjust colors, contrast, or sharpness.

In this case you should save the file with the same name, but adding an "r" as a suffix to the file name: is a good rule to never change the name of a picture, especially if it's part of a series.


6. Resources:-

Proxy Lists:
http://tools.rosinstrument.co m/cgi-bin/fp.pl
http://proxys4all.cgi.net/
http://www8.big.or .jp/~000/CyberSyndrome/pl1a.html
http://www.cyberarmy.com/lists/ proxy/

Proxy Checkers
http://checkpoint1.hypermar t.net/index.html
http://www.ren tal-web.com/~azuma/cgi-bin/env.cgi
http://mizuno -labo.cs.inf.shizuoka.ac.jp/~s5087/proxy.html
http://www.all-nettools.com/tool s1.htm
http://allfree.webprovider.c om/proxychk.html
http://www.tamos.com/netres/pr.cgi
http://www.all-nettools.com/

Country Check
http://www.all-nettools.com/tool s1.htm
http://www.arin.net/whois/
http://www.samspade.org/

Cracks:-

Programs:


Home Back Next