Displaying huge text files

 Delphi  Comments Off on Displaying huge text files
May 292016
 

Sometimes programmers have to deal with larger than usual text files. Examples are log files or XML dumps of databases. For my tests, I used a dump of the English Wikipedia in XML format. This file is 48 gigabytes in size and as I found out only today contains 789,577,286 lines of text.

dzLargeTextViewer-Line708754410

If you google for “large text viewer” you get quite a few hits, but many of these are not what they advertise.

Others kind of work but have various shortcomings, e.g. Large Text File Reader which allows you to only display the first n lines of a file. But don’t try to enter too large a number because it will then just hang. Then there is Log Expert which was suggested by this answer on StackOverflow but apparently tries to load the whole file into memory, nearly crashing my Windows installation.

I found the following viewers which seem to work:

LTFViewer from the now defunct swiftgear.com site (Link therefore goes to archive.org). This works but seems to be rather slow in reading the file. Closed it after about an hour when it had read about 200 million lines.

glogg which is a log file viewer with searching and filtering. It’s multi platform which usually means that the Windows version is barely usable. Glogg is not too bad on this account. It took quite a while loading the file which ran in the background. Unfortunately while it is loading the file, you cannot browse it. It only displays the first page, scrolling is impossible. You can use the filter function though. It took about one hour to read the 48 GB file using 1 GB of memory in the process. Once it has loaded it, it supports e.g. incremental search, which, in a file of this size takes quite a while.

Another option is a file viewer written in JavaScript at www.readfileonline.com. I had my doubts about this but since it works locally and browses the file in batches of a given size, it can be very fast. It even has a search function, but of course that one has limits when talking about a file of 48 gigabytes.

Yours truly also wrote such a tool, back when I received a huge XML file from a customer, also a database dump, and had to resort to Linux and the less command line tool to view it. dzLargeTextViewer doesn’t have many features. It theoretically allows you to display files up to MaxInt64 lines. I have tested it with the aforementioned Wikipedia dump. The first version ran out of memory after around 75 million lines. The second version used 1 GB of memory and could index about 300 million lines. The third version now has read all nearly 790 million lines and displays them, using only 3.5 MB of memory. To achieve that, it created a 6 GB index file. It stores that file as <Filename>.LineIndex so it can be reused when the same file is opened again. My tool assumes ansi strings, though, while glogg reads the file as UTF-8.

Clearing a TTreeView

 Delphi  Comments Off on Clearing a TTreeView
May 282016
 

Note to self: If you want to clear the items that were added at design time to a TTreeView, you must make sure it has a handle. The following does not work (in Delphi 2007):

constructor TForm1.Create(Owner: TCoponent);
begin
  inherited;
  TreeView1.Items.Clear;
end;

Adding a TreeView1.HandleNeeded makes it work:

constructor TForm1.Create(Owner: TCoponent);
begin
  inherited;
  TreeView1.HandleNeeded
  TreeView1.Items.Clear;
end;

Known IDE Packages in Delphi

 Delphi  Comments Off on Known IDE Packages in Delphi
May 262016
 

Prompted by a comment in this Google+ post I had a look at what is actually listed in [HKEY_CURRENT_USER\Software\Borland|Codegear|Embarcadero\BDS|Delphi\#.0\Known IDE Packages] and found some interesting entries:

In Delphi 6 and 7 there is direct60.bpl and direct70.bpl which can probably simply be removed nowadays. The server used by “Delphi Direct” which was the equivalent of the current “Welcome Page” no longer exists. Removing this package results in the Environment Options dialog having one less tab:

DelphiDirect

I never understood the purpose of Delphi Direct. Even back in the 1990ies there wasn’t any interesting content delivered to it. In my opinion it was totally wasted by Borland.

Then there is delphiclxide60.bpl and delphiclxide70.bpl. These packages apparently contain the designer for CLX applications (anybody remember theses?). When removed, there are still some CLX items in the File->New->Other menu and these can still be used, but you will no longer be able to visually design these forms and dialogs. Also the File->New->CLX Application entry will be gone. In Delphi 6 this results in some designtime packages not being able to load, in Delphi 7 it doesn’t but that might just be because I removed them earlier.

CLXNew

Since I doubt that anybody is still developing CLX applications, it’s probably safe to remove these entries as well.

Moving on to the “modern” IDE, which was introduced with Delphi 8 (which I don’t own, so I’ll look at Delphi 2005 instead):

The startpageide90.bpl package is the equivalent of Delphi Direct mentioned above. It is still there in Delphi 10.1 Berlin as startpageide230.bpl. It has been suggested to remove this package because it is hardly useful. I haven’t done that because my delphi7help4bds expert uses it to display HTML content.

There is caliberide90.bpl or Borland.Caliber.IDE100.bpl as it is called in Delphi 2006. After removing it in Delphi 2006 I found nothing missing. Caliber apparently is a requirement management tool now owned by MicroFocus (who acquired Borland after they spun off their development tools as CodeGear). I’m not sure what exactly it does or did in Delphi 2005 and 2006 but it is no longer there in Delphi 2007 and later. The same applies to TGIDE90.BPL which is the “Together IDE integration”. It’s the UML modelling tool Borland acquired in 2003.

Another interesting entry is historyide90.bpl. It apparently implements the history tab (at the bottom of the editor, to the right of the Code and Design tabs).

HistoryTab

It’s still there in Delphi 10.1 Berlin.

Some other observation: Starting with Delphi 2006 the key “Known IDE Packages” has subkeys, one of them is “Delphi”, others, that were in use at some time are:

  • CSharp
  • DelphiDotNet
  • CBuilder

There might have been others but I recently stopped installing anything but the Delphi personality even though my (employer’s) subscription is on Rad Studio just in case.

I’ll stop here with listing those packages. Just some comment about enabling and disabling them:

Apparently they don’t get loaded if the entry does not have a value. Some of the packages have a value of (Untitled) which apparently means “nobody bothered to give them a name but we need a value so we added something generic”. If you remove this value (set it to an empty string) the package will no longer be loaded. But it’s probably not a good idea to disable them by removing the value. I did my tests by creating a new “unused” subkey and moving these packages there, including the description.

I’ll probably write another post on this topic at a later time. I’m also thinking about writing a tool to disable/enable entries in Known IDE Packages.

GExperts 1.38 experimental twm 2016-05-26

 Delphi, GExperts  Comments Off on GExperts 1.38 experimental twm 2016-05-26
May 262016
 

This is another test release before Erik is going to do an official 1.39 release. Please report any bugs you may find (preferentially in the GExperts community on Google+ or the bug tracker on SourceForge)

Again, I have built installers for each Delphi version. These installers should install everything that is necessary, including the files for the Code Formatter.

GExperts-Configuration-Editor-Experts

Here are the links:

Thank you, GExperts contributors

 Delphi, GExperts  Comments Off on Thank you, GExperts contributors
May 162016
 

I would like to thank those people who have contributed to GExperts during the time that I have been maintaining most of it. There have been various contributions ranging from feature suggestions with partial implementations, good bug reports, to bug fixes that worked out of the box.

Thank-You

You might have had the impression that the recent activity has been a one man show, which would not be too far from the truth, but I really appreciate the contributions of others.

dzDelphiPaths tool

 Delphi  Comments Off on dzDelphiPaths tool
May 152016
 

Somebody in the German forum Delphi Praxis asked for a tool to display the Delphi library paths. Apparently such a tool didn’t exist yet but it could sometimes come in rather handy, so I wrote it.

dzDelphiPaths

It doesn’t do much, only scans the registry for installed Delphi versions (Starting with Delphi 5, I am not sure about the registry path’s for the older versions. Feel free to send me patches.) and gives you a list of all entries in the Libraries branch. In addition to a list by Delphi version it also gives you a list by entry for all Delphi versions as you can see in the screen shot. That way you can easily compare the same path in different Delphi installations.

It’s on SourceForge.

Getting an old package wizard to work

 Delphi  Comments Off on Getting an old package wizard to work
May 082016
 

I tried to compile the SarchWiz project by Corbin Dunn, which he describes in the corresponding article which in turn Ondrey Kelle mentioned in his blog post More Subversion.

(The links to code central and edn above are the ones Embarcadero created from the original links to code central and bdn when they were still operated by Borland. Luckily they did not change the article IDs so it was only a matter of converting the urls to the new adresses. Allan Bauer’s example code is here, btw.)

The first problem I ran into was that the original wizard was written for Delphi 5 (yes 5, not XE5). So some package and unit names had changed.

  • The VCL/RTL etc. .dcp files no longer have the version sufix (only the corresponding .bpl files do), so the package requires vcl, vclie and inet rather than vcl50, vclie50 and inet50
  • The dsnide50 package is now called dsignide
  • The unit DsgnIntf no longer exists, but apparently nothing was used from it, so I just removed it from the uses list.

Then, there is TDockableForm, the ancestor of the TSearchWizForm class used in the wizard. Apparently the declarations of LoadWindowState and SaveWindowState have changed. In Delphi 5 the Desktop parameter was a TMemIniFile, nowadays it’s a TCustomIniFile.

// old
procedure LoadWindowState(Desktop: TMemIniFile); override;
procedure SaveWindowState(Desktop: TMemIniFile; isProject: Boolean); override;

// new
procedure LoadWindowState(Desktop: TCustomIniFile); override;
procedure SaveWindowState(Desktop: TCustomIniFile; isProject: Boolean); override;

The last change to get it to compile was changing the WebBrowserBeforeNavigate2 method. In Delphi 5 some parameters were declared as var, they now are const:

// old
procedure WebBrowserBeforeNavigate2(Sender: TObject;
  const pDisp: IDispatch; var URL, Flags, TargetFrameName, PostData,
  Headers: OleVariant; var Cancel: WordBool);

// new
procedure WebBrowserBeforeNavigate2(ASender: TObject;
  const pDisp: IDispatch; const URL, Flags, TargetFrameName, PostData,
  Headers: OleVariant; var Cancel: WordBool);

So, I compiled and installed the package without errors, and … nothing happened. Since it is an ancient wizard, I looked into the help menu and found Help -> Help Wizards -> Web Search, which is the menu entry added by the wizard. Clicking on it still didn’t do anything.

After putting a breakpoint in the wizard’s execute method (and setting the host application to the Delphi ide), I found that the window is actually being created and shown. It’s listed in the IDE’s Window menu and I eventually found it on the screen: It was far down behind the task bar (which fortunately in Windows 8.1 is transparent). So, switching to it and pressing Alt+Space gave me the system menu and selecting Maximize finally brought it somewhere sensible. It turned out that the problem was this code:

procedure TSearchWizard.Execute;
begin
  if SearchWizForm = nil then
  begin
    SearchWizForm := TSearchWizForm.Create(Application);
    SearchWizForm.Left := 10;
    SearchWizForm.Top := Application.MainForm.Top +
      Application.MainForm.Height + 10;
  end;

  FocusWindow(SearchWizForm);
end;

Application.MainForm is no longer the small window it was in Delphi 5 which contained only the menu, tool bars and component palette but the whole Window, so setting the form’s top position to MainForm.Top + MainForm.Height + 10 no longer positions it below the component palette but below the normal screen area.

After changing it to simply 10, the wizard window is displayed and can be used.

It’s very strange to see Altavista, Deja.com, Excite and Northern Light in the list of search engines. Apparently Altavista now gets redirected to Yahoo, deja.com is now Google groups, excite.com still exists, but search.excite.com does not work, and northernlight.com is no longer a search engine. I changed the excite entry to the new url and it worked (ignoring some JavaScript errors).

This is what it looks like when sarching excite for “delphi programming” when the wizard is docked to the Delphi 10 IDE:

SearchWiz

Now, you might ask why I went through all this, just to have a web search window in the IDE? It’s connected with my Delphi7Help4Bds expert. It currently uses the Welcome Screen to display its results when configured to call a url. I thought about creating my own browser window and – you know that if you looked at the code – I had started to build something based on some code I got from Simon Stuart before he stopped publishing stuff under open source licenses and shut down his blog in 2012. And since I was unable to get it to work, I googled for other stuff using similar techniques. So I came across Ondrey Kelles’s post and the links there took me to this ancient wizard code. Isn’t it strange what turns life can take?

Delphi IDE Explorer Expert for Delphi 6 to 10.1 Berlin

 Delphi  Comments Off on Delphi IDE Explorer Expert for Delphi 6 to 10.1 Berlin
May 072016
 

I wrote this Delphi IDE Explorer Expert when I was trying find the button for showing and hiding the background for the Firemonkey mobile form designer and turn the bloody thing off for good.

Later, I ported it back to Delphi 2007 to 2010 and also added support for Delphi 10 Seattle.

Then I started improving GExperts (or actually made GExperts improve the Delphi IDE dialogs), so I needed it also for Delphi 6 to 2006.

Now I have added support for Delphi 10.1 Berlin. You can find the sources on SourceForge.

Oh, did I mention, that it accepts user input even if the IDE shows a modal dialog? Very useful feature.

DelphiIDEExplorer-for-Berlin

Delphi7Help4BDS updated for Delphi 10.1 Berlin

 Delphi  Comments Off on Delphi7Help4BDS updated for Delphi 10.1 Berlin
May 072016
 

I have just updated my Delphi7Help4BDS expert to support Delphi 10.1 Berlin.

It allows to reconfigure F1 + any of the modifier keys Shift, Ctrl, Alt and Alt+Ctrl to call either a help file (you will need the old winhelp viewer that Microsoft dropped with Vista(?)), chm file or internet url. It comes preconfigured with a few example internet urls for searching with Google, Bing, the Embarcadero docwiki or MSDN.

This shows the result of Alt+F1 when the cursor was on the keyword "Integer":

Delphi7HelpForBdsExpert

To install it, get the sources from SourceForge, open the package for your Delphi version, compile and install it. You should then find a new entry in the IDE’s Help menu called “Configure Delphi7HelpForBds” which gets you a dialog where you can configure the actions for each of the key combinations.

Delphi7HelpForBdsWizard

Delphi Custom Container Pack updated for Delphi 10.1 Berlin

 Delphi  Comments Off on Delphi Custom Container Pack updated for Delphi 10.1 Berlin
May 072016
 

I just updated the Custom Container Pack sources to support Delphi 10.1 Berlin. It was mostly a matter of creating the packages for the “new” version. I also used the latest version of the Delphiversions.inc file.

It now compiles and installs. I have not tested it extensively.

Custom Containers Pack (CCPack) is an integrated tool and component mini-library to produce and maintain composite controls (or simply “composites”) and other containers (forms, data modules and frames). The process of building composite components looks like ActiveForm and Frame creating, but the result is the native VCL component. You can create new composites just as usual forms.

Here is the original documentation in RTF format.

It was originally developed by Sergey Orlik who posted the source code to code central