Web Excavator - Update History

UPDATE HISTORY

Version 1.30 - January 2, 2010

Added a minimum image width and height option to the Advanced Options dialog.
Fixed a bug where the save file could become corrupted if the application crashed.
Now better handle links that end in a forward slash.

Version 1.29 - January 2, 2008

Fixed a bug in the update duplicate files system and made some efficiency improvements.
Fixed a possible crash bug with a particular sequence of actions in the download list dialog.
Now when searching with nothing it displays the first set of URLs.
Fixed a bug with the list dialogs being unresponsive when saving.
No longer guesses site names with URL encoded characters (e.g. %3F, etc.)

Version 1.28 - October 28, 2007

Fixed a bug in the search algorithm (e.g. when searching for zza in zzza).
Fixed a bug when clearing the URL history while files are downloading.
Many improvements to the web page parsing algorithm.
Many improvements to the algorithm that converts web pages for offline viewing.
No longer guesses file names with URL encoded characters (e.g. %3F :)
When using the up or down arrow to get the previous search option the cursor is now defaults to the end of the phrase.
Now remembers the last ten search terms.
Added additional default sound and movie file types.

Version 1.27 - September 30, 2007

Fixed a bug with the 'Pause After Downloads' where it could turn itself off.
Save files are now slightly smaller (this change is backwards compatible)
An empty 'To Download' list dialog will now automatically reload the list when files are scheduled to be downloaded.
Also on the 'To Download' list dialog you can now 'Delete All' the downloads in the queue (if you are not performing a search - in which case the button deletes all the search entries found)
Pausing the program will now update the graphic in any open list dialog.
Added an Advanced option to save downloaded files with URL encoded files names.
Fixed a possible deadlock.
Added .xml to the default site types.
Fixed a bug with loading option (.ini) files.

Version 1.26 - September 1, 2007

Fixed a bug with duplicate files where deletion would occur if the same file was added again to the download list.
Fixed a bug loading file types from an ini file.
Within the list dialogs, Ctrl-S selects all displayed entries and Ctrl-D deletes displayed selected items.
You can now have both 'To Excavate' and 'To Download' list dialogs open at the same time.
Also both 'To Excavate' and 'To Download' minimize when Web Excavator is minimized.
Optimized loading the duplicates data file.
If 'Open Last Download' fails it now tries to open a previous download.
Fixed a bug in the Wizard with selecting 'Guess URLs'.
Minor UI improvements.

Version 1.25 - August 19, 2007

Fixed a rare crash bug.
Made improvements to the HTML parsing engine.
When downloading a web site, offsite images are now downloaded. More and more websites are storing their images offsite.
Made some optimizations, particularly when checking to see if a site is to be added to the download list.
Found an instance where Web Excavator could create a sub-directory when configured not to.
Now extracts links that do not have a corresponding closing quote.
Now displays the size of large files correctly.

Version 1.24 - July 14, 2007

Pressing the up arrow in the search field within the 'To Download' or 'To Excavate' dialogs, will now display the last phrase searched for.
Added 'Save' and 'Load As' toolbar buttons.
Saved files are now named with the site you started plus ".sav". Old saved files will still load.
The save files above also now have a corresponding ini file.
In the wizard when downloading files skipped phrases are applied to both Site URLs and file URLs.
Improved the handling of local 302 redirects.
Fixed a bug with checking for updates.
Now handles links that only have a directory and do not contain a file name.
When downloading web pages that do not have a file extension, Web Excavator now appends a file extension.
Fixed a bug when downloading within a domain, where it could download html files outside of the domain.
Fixed a bug where the stop command would not stop downloading if a particular situation occurred.
Fixed a bug where a URL to excavate could be added twice.
History data is now backed up to History.bak
Changed it so Web Excavator will not guess URLs if a duplicate file is found.
More minor UI improvements.

Version 1.23 - June 24, 2007

Minor UI improvements.
Zero byte length files are now automatically deleted.
As requested, added a 'Pause After Downloads Completed' menu option that, when activated, stops processing URLs and waits until the download queue is empty before pausing Web Excavator.
Last image download is now updated while 'To Download', 'To Excavate' or 'In Process' dialogs are open.
Fixed a bug where occasionally the first file downloaded would be saved into the same directory as Web Excavator.

Version 1.22 - June 2, 2007

Fixed a bug with updating Web Excavator that was introduced in version 1.21.
Added option to 'Download Dir' to store checksums/file size information about every file in the download directories so that when a new file is downloaded it compares the checksums/file size of this new file with the data it has stored. If it is found to be the same it deletes it. Also, if checked, all new downloads checksums/file size are added to this list.
There is a new menu item to delete this checksums/file size data file.
Improved file compare functionality.
Fixed threading related crash bug when removing files from download list.
Fixed bug with application not receiving the focus properly after restoring from system tray.
Added advanced option to resize the last downloaded displayed image so as to fit within the display window.

Version 1.21 - April 11, 2007

Now checks minimum file size upon receiving file header information.
Fixed rare bug where Web Excavator would not restore properly after being minimized.
If not saving web pages and audio/image/etc file is really an html file, then this file is not saved.
In the list dialogs the search term now defaults to active.
Version check file is now always downloaded to the same directory as the exe.

Version 1.20 - January 14, 2007

Added ability to search within the list dialogs and to remove all files containing the search term.
Reload within the list dialogs now starts at the given index.
Within the list dialogs the shift key now can be used to select multiple files .
Now displays the last jpg or gif image saved. Left click on this image to open in default viewer.
Added Advanced option to disable displaying the last image saved.
Send cookies (which defaults off) now also saves cookies too.
Fixed: Downloading from secure sites now updates the number of files downloaded correctly.
Fixed: 'Erase URL History' now deletes all ignore lists.
Delete key will now deletes selected items from list dialogs.

Version 1.19 - December 28, 2006

Added buttons to the application window to do common tasks.
Added download directory to main application window (after typing press enter to use new directory).
No longer guesses URLs on a redirect
No longer guesses file names when the file downloaded is below the minimum file size (set in options)

Version 1.18 - August 6, 2006

Fixed a bug with saving Sub-domain data to separate folders.
'Within This Domain Only' which downloads into sub-domains folders (see below) will now change the original code so that it points at these.
The above feature can be turned off within the advanced options by selecting 'Do Not Alter Web Pages On Download'.
Added an 'Log Valid URLs' advanced option to log all the valid downloads to a ValidURLs.log file located in the same directory as the executable.
Added an option to not save any files when 'Log Valid URLs' is checked.
Fixed a bug with not downloading after starting and stopping a previous download.
Fixed a bug with guessing download URLs and IP addresses.
Added option to apply any changes to the search keywords, or exclusion keywords to the appropriate lists immediately.

Version 1.17a - July 5, 2006

Added advanced option to define a custom user-agent.
Added .dll to the default list of web site types. Saves with dll extension. Do not run downloaded dlls.
Fixed a bug with extracting URLs from an unusual (to me) JavaScript coding style.

Version 1.17 - July 3, 2006

When minimizing Web Excavator to the system tray the 'File duplicates' dialog, if open, is now also minimized.
Properly handles web servers that do not allow disconnected downloads to continue.
Fixed a potential program lock up when exiting in the middle of searching for duplicate files.
Now handles style="background-image:url(URL)"
Fixed problems with setting "More Options" on the Wizard Dialog.
Fixed a bug with connecting to websites that do not use port 80.
Added jhtml to the web page types to search
'Within This Domain Only' will now also download from sub-domains (e.g www.example.com, images.example.com, scripts.example.com, etc.). Sub-domain data will be saved in a separate folder (e.g. scripts_example_com).

Version 1.16a - March 23, 2006

Now appends 'index.html' when a sub-directory ends in '/'.
Fixed a bug with regards to newlines when handling redirects.

Version 1.16 - March 11, 2006

Streamlined Wizard for downloading Single web pages
Improved downloading single web pages.
Fixed a bug in the wizard where you could not turn off guessing URLs
Fixed a bug where one of the previous buttons took you to the wrong place
Fixed a bug where it did not automatically prepend http when downloading files
In verbose mode it now displays if it failed to download a file
Reduced false positive rate when parsing JavaScript.
Now saves the 'Min. Downloads (below which search for more)' setting.

Version 1.15 - February 12, 2006

Fixed a couple of potential buffer overrun crash bugs.
Added option to search in the sub-directories of the 'optional directories' to see if a file has already been downloaded.
Added 'Download Dirs' to the options menu.
Now automatically appends a ';.' if the search types are all exclusions (!).
Fixed bug with setting defaults in the Options Dialog not resetting previous download file types.
Fixed a bug with not ignoring first excluded site file type.
Fixed a bug with using non-exclusions (!) with Keywords to skip.
Added Advanced Option to change the minimum number of downloads before it searches for more.
Minor interface changes/improvements.

Version 1.14 - February 2, 2006

Optimized the code that parses web files.
Fixed a bug where Web Excavator would, under a particular circumstance, parse non-web files.
Added ability to type a starting index when viewing the 'Download' and 'Sites To Scan' lists.
Made the 'check for updates' easier to use.
Now closes faster if having a connection problem.
On failure to connect to a site, it now checks against a known site to verify that the Internet connection is good.
Fixed a spelling mistake on 'Find Duplicate' dialog.
Now remembers the 'Search Sub Directories' setting between duplicate file searches
Fixed a bug where web pages did not have a extension when saved in 'Do Not Create Any Sub Directories' mode.
Clicking the exit button a second time will force the software to exit no matter what.

Version 1.13 - February 2, 2006

This one was unlucky ;)

Version 1.12a - December 11, 2005

Added Portable Application Descriptions XML file to the installation program.
Now sets all options to there original values after pressing the 'defaults' button.
Included additional exclusion defaults when downloading entire websites.

Version 1.12 - December 4, 2005

Fixed a download crash bug.

Version 1.11 - November 26, 2005

Made some optimizations to the code, in particular Web Excavator now shuts down faster.
Fixed a crash bug when parsing web page after installing Norton Internet Security.
In the 'In Process' view it now shows if it is trying to connect.
Added an option not to guess URLs after Web Excavator finds a question mark.
Fixed a display bug in the download list.
A few small interface changes.

Version 1.10 - October 27, 2005

Added a wizard to simplify the use of this application.
Added an option to the advanced options dialog to always start in 'classic view'.
Added ability to remove files from the download list while paused.
Fixed a bug when exiting while paused.
Optional 'Keywords In URL To Skip' is now checked when a save file is loaded.
Fixed a bug where it would sometimes not check all download directories.
Can now 'row select' on the Download List Dialogs. Also right clicking will now open that URL.
Fixed a bug where the optional 'history of sites visited' list was loaded twice.
Also, when using the above option, the program now exits faster.

Version 1.09 - August 13, 2005

Added menu item to search for duplicate files in any directory / sub-directories and optionally delete them.
Added advanced option to set the priority of the program. Set this to low if you want it to run as a background process.
Fixed a bug where it would not download files starting with '../'.
Minor tweaks to the parsing algorithms.
Added ability to copy a link to the clipboard from file or site download lists by right click or Ctrl-C.
Added an option to send a cookie to the site being downloaded from (see options dialog)
Added an option to view the files currently being downloaded.
Added two more optional download directories to check in to see if a file has been downloaded.
Some minor graphical and display tweaks.

Version 1.08 - June 22, 2005

Added advanced option to check for updates at start-up.
Fixed a bug in the download code.
Added a menu option to open the last file downloaded.
Within the view site and download lists it now displays the index range of the items being shown.
With 'Guess URLs' it now does not try to guess IP addresses.
Corrected a few grammar errors.

Version 1.07 - June 17, 2005

Fixed a possible crash bug when guessing URLs.
Added a menu option to remove all files to be downloaded.
Made many tweaks and improvements to downloading sites where a redirect file is needed.
Added an Advanced Option to set default redirect scripting language when file type is not supported. Defaults to PHP. Currently only ASP, JSP and PHP are supported.
Added a Open Debug file button to Advanced Options dialog.
If loading a saved state fails, it will now automatically attempt to load the back up.

Version 1.06 - June 5, 2005

Fixed bug handling URLs that are not in quotes.
Added option to not download files if their name contains a given keyword.

Version 1.05 - May 31, 2005

Now handles redirecting web pages. When handling PHP, ASP and JSP redirecting web pages it writes out a redirecting page in that language; all others are written out as ASP. Defaults to only using just one parameter.
Can now set the Maximum file size to download.
Can now set the number of URL download threads (see Advanced Options).
Many minor optimizations and improvements with the web page processing.

Version 1.04 - April 24, 2005

Now displays the number of files downloaded this session.
'Save and Exit' now shutdowns the program if download list is empty.
You are now prompted if the program is about to overwrite the save file with an empty one.
Fixed bug in download directories to check display and added two more.

Version 1.03 - April 18, 2005

Fixed bug when downloading single web page.
Fixed potential crash bug on exit.
Added additional optional directories to search in to see if a file has already been downloaded.
Optionally keep a list of files already downloaded so as not to download them again.
User deleted files are added to the 'already downloaded list' (if enabled).
Fixed bugs in download and site list display.
Fixed problem with resuming interrupted downloads.

Version 1.02 - March 13, 2005

If you specify a save directory that doesn't exist it will now create that directory for you.
Added option to specify other directories (up to 5) to search in to see if a file has already been downloaded.
Now saves the date when logging URL errors.
Added stop adding to URL History option menu option.
Added 'Erase URL History' option to the menu.
Added a 'fetch' individual file option on the main window.
Fixed a bug where it would not search the exclude list under certain circumstances.
Fixed a small memory leak with the file view dialog.
Some cosmetic changes, including making the file view dialog wider.
Added a reload button to the file view dialog.
Fixed a crash bug in the file view dialog if you selected or deleted from an empty list.
Some minor tweaks to the programming logic.

Version 1.01 - February 6, 2005

Made numerous changes to the interface, including:
1. Adding a 'Back' button and 'Select All' to the file view list.
2. Adding a Web Excavator graphic.
Can now optionally tell it to guess URLs. Currently, this only works for URLs that contain numbers. For example, if it finds a Neptune01.jpg it will look for a Neptune02.jpg and a Neptune10.jpg.
Added an option to search for keywords on the web page not just in URLs.
Added 'Save As' and 'Load As' current options to the File menu.
Added Menu item to 'Erase URL History' that clears the list of previously downloaded URLs.

Version 1.00 - January 30, 2005

Initial release.