Nov 252017
 

You might have noticed that I haven’t blogged as much as usual and that there have been no recent updates to GExperts either. The reason of course was that I was busy doing other things. Some of them were programming related and one of these was writing a caching class wrapper for TStream descendants (e.g. TFileStream, but it should work with any other streams as well).

It is simple to use and totally transparent to the calling code because the cache itself descends from TStream:

procedure TestCache;
var
  fs: TFileStream;
  cache: TdzStreamCache;
  i: integer;
  by: byte;
begin
  cache := nil;
  fs := TFileStream.Create('C:\test.dat', fmOpenWrite);
  try
    cache := TdzStreamCache.Create(fs);
    // now, do any operation you usually do on the stream, but use the
    // cache instead
    // e.g. write 50 megabytes of random bytes
    randomize;
    for i := 0 to 50*1024*1024 do begin
      by := Random(256);
      cache.WriteBuffer(by, SizeOf(by));
    end;
    // Writing a single byte to a TFileStream will result in very
    // bad performance, which is why we usually copy the bytes to a
    // larger buffer instead and then write this buffer to the stream.
    // Using the TdzStreamCache you won't see much of a difference between
    // these two approaches.
  finally
    FreeAndNil(cache);
    FreeAndNil(fs);
  end;
end;

TdzStreamCache can also be used for read and read/write access to a stream. It supports linear and also random access. Performance is worst when reading a stream backwards a single byte at a time.

To get an idea how well it performs, I have benchmarked it against David Heffernan‘s excellent CachedFileStream and found that it compares very well. (I could not simply use David’s class because I needed caching for other types of streams, not just files.)

I have been using TdzStreamCache extensively in the last several weeks and I am pretty sure that I have weeded out most of the bugs. But of course, that does not mean that there aren’t any left, so there are no guarantees. If you use it, all problems are your own.

TdzStreamCache is released under the Mozilla Public License (MPL) 1.1 as part of my dzLib library, but the unit is pretty much stand alone. The only thing it requires is jedi.inc because Delphi 2007 declares NativeInt inconsistently to other Delphi versions. As with all my dzlib code, it is available from sourceforge as u_dzStreamCache.pas.

The code for the benchmarks I mentioned above is also available if you want to do the benchmarks yourself.

I would appreciate any feed back, especially if you find bugs or improve on it.

Nov 152017
 

So I don’t forget:

The following Linux command shows a list of all subdirectories that contain a least one *.xml file:

find . -type f -name '*.xml' | sed -r 's|/[^/]+$||' | sort | uniq

This is an adaptation of this answer on unix.stackexchange.com.

Nov 022017
 

There was a discussion about using the PE flag IMAGE_FILE_LARGE_ADDRESS_AWARE and whether this is equivalent to compiling a program to 64 bit Windows target (no it’s not) in the German Delphi Praxis forum. This prompted me to have a look at what kind of flags can be specified and how to do that for a Delphi program.

A quick Bing search for SETPEFLAGS and Delphi also found a blog post by Halvard Vassbotn from 2006 about the IMAGE_FILE_RELOCS_STRIPPED PE flag.

Since one of my programs has recently started to throw out of memory
exceptions (when displaying large(!) data files) and our executables tend to be rather large anyway, I decided to add these PE flags to it.

  • IMAGE_FILE_LARGE_ADDRESS_AWARE – App can handle >2gb addresses
  • IMAGE_FILE_RELOCS_STRIPPED – Relocation info stripped from file

With the flags it already used …

  • IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP – If Image is on removable media, copy and run from the swap file
  • IMAGE_FILE_NET_RUN_FROM_SWAP – If Image is on Net, copy and run from the swap file.

… this resulted in the following SETPEFLAGS compiler directive in the .dpr file:

{$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP
          or IMAGE_FILE_NET_RUN_FROM_SWAP
          or IMAGE_FILE_LARGE_ADDRESS_AWARE
          or IMAGE_FILE_RELOCS_STRIPPED}

The result was as I hoped: No out of memory error any more when displaying a file that always caused one before. The executable size also shrank, but not as significantly as I had hoped: 9698 KB where before it was 9987 KB, but every little helps.

But there is another side effect I found today: Setting IMAGE_FILE_NET_RUN_FROM_SWAP not only makes Windows load the executable into the swap file when executing it but apparently every time it accesses it. Simply selecting it in Windows Explorer resulted in the Explorer Window freezing for a short time (it feels like several seconds but probably is just about one), the same delay happens when opening the Properties dialog (e.g. to check the version info). That does not happen, when the flag isn’t set.

%d bloggers like this: