Exported GMail mbox contains GMail labels

 Google  Comments Off on Exported GMail mbox contains GMail labels
Oct 142018
 

Just in case it isn’t already common knowledge: The “Takeout” export of GMail will give you a single file in mbox format (which will be rather large).

That file contains the content of the “all mail” folder. And each of the messages in the file has a header like this:

X-Gmail-Labels: Archived,Important,Rechnung

So with the right tool it is possible to restore the labels you have set in GMail. Unfortunately I know of no such tool. And also I don’t know of any (Windows) email client that allows you to set user defined tags for emails.

EDIT: There is a tool called Google Takeout Data Converter that claims to do that. I haven’t tested it though.

EDIT2: Thunderbird apparently supports custom tags, not just the ones that are predefined.

EDIT3: Thunderbird also supports virtual folders e.g. based on tags.

Building less annoying user interfaces (Part 2)

 Delphi  Comments Off on Building less annoying user interfaces (Part 2)
Oct 142018
 

In Part 1 I was ranting about input validation, now it’s about form sizes.

In my previous post I already said that I am not a big fan of fixed size dialogs. They are fine as long as they are large enough to show all of their content, but as soon as there is a list that can grow to an arbitrary size, fixed size dialogs are a no go. Look at the Microsoft Printer installation dialog again:

It contains two lists, one of printer manufacturers and the other of printer models of the selected manufacturer. Of course, it could be even worse and have one list only (And even that wouldn’t have surprised me really. Microsoft has all sorts of bad UIs.). So the list of manufacturers is sorted and even has incremental search. It’s also not too long so it’s probably fine for most users. But the list of printer models is really stupid:

  1. All printer names are prefixed with the manufacturer name, so incremental search does not help.
  2. The list is not high enough to display all entries. Thus there is a vertical scroll bar. OK, for several hundred printers listed for HP alone no screen would have been large enough, so it can’t be avoided. But one page only shows 4 lines and the last one is even only half visible.
  3. And also, the list is not even wide enough to display the full text of the entries. So we get a horizontal scroll bar too which further reduces the number of lines visible.

In my opinion this is really bad UI design. That dialog could be improved a lot by simply changing it to be sizable, anchor both lists to the left, top and bottom and the right list to the right as well.

I faked that dialog with Delphi so I could show you what I mean:

I even reproduced the ridiculously large white areas around the lists. But this dialog can be sized and has even got a maximize button. Let’s see what it looks like if I make it just a little bit larger:

No horizontal scroll bar any more. And all visible lines are completely visible rather than only half of the last one. And if I increased the height even more, it would become much easier to scroll the printer list down to the right model.

OK, so we make all our dialogs sizeable and set the control’s anchors correctly. Easy as cake. Is that it?

No. With more freedom for the user we get more things he can screw up. Have a look at the next screen shot:

A sizeable dialog can also be made smaller and this is what it looks like. Eeek! So what can we do about it? We set size restrictions.

The easiest way to do that, is designing forms in the smallest size you want them to have and then set its minimum size to the designed size.

I have got two simple helper functions that do this for me:

type
  TControlConstraints = (ccMinWidth, ccMinHeight, ccMaxWidth, ccMaxHeight);
  TControlConstraintsSet = set of TControlConstraints;
const
  ccMin = [ccMinWidth, ccMinHeight];
  ccMax = [ccMaxWidth, ccMaxHeight];
  ccAll = [ccMinWidth, ccMinHeight, ccMaxWidth, ccMaxHeight];
  ccFixedHeightMinWidth = [ccMinWidth, ccMinHeight, ccMaxHeight];
  ccFixedWidthMinHeight = [ccMinWidth, ccMinHeight, ccMaxWidth];

procedure TControl_SetConstraints(_Control: TControl; _Which: TControlConstraintsSet);
begin
  if ccMinWidth in _Which then
    _Control.Constraints.MinWidth := _Control.Width;
  if ccMinHeight in _Which then
    _Control.Constraints.MinHeight := _Control.Height;
  if ccMaxWidth in _Which then
    _Control.Constraints.MaxWidth := _Control.Width;
  if ccMaxHeight in _Which then
    _Control.Constraints.MaxHeight := _Control.Height;
end;

procedure TControl_SetMinConstraints(_Control: TControl);
begin
  TControl_SetConstraints(_Control, ccMin);
end;

So, in the constructor I simply call:

constructor TMyForm.Create(_Owner: TComponent);
begin
  inherited;
  TControl_setMinConstraints(Self);
end;

Since we started out with a fixed size form, this does not have any negative effects. The form still has a minimum size that corresponds to the fixed size it used to have, so even users whose monitors are too small aren’t any worse off, and those who have large monitors can use all the space they want to.

Two more screenshots to just give you an idea of the difference:

This is the original dialog on a 24″ monitor with 1920×1280 pixel resolution:

It feels like a postage stamp.

And this is my fake dialog resized to about 1/4 of the monitor resolution:

(Of course here you see that I didn’t bother to add more lines to the lists than necessary to get my point across.)

Here is another example where a fixed sized dialog is not a good idea.

There is one entry field that takes a file name. In a fixed sized dialog you would usually end up with clipping that name because nowadays file names are very long since nobody has to type them any more. If you are a user, you still want to be able to see the full path. So, you simply make the dialog wider and tadaaa:

This is a special case where you probably don’t want the dialog to become any higher than the designed size. So what do you do? Simple, you also restrict the height:

constructor TMyForm.Create(_Owner: TComponent);
begin
  inherited;
  TControl_SetConstraints(Self, ccFixedHeightMinWidth);
end;

Now that user can resize it to make it wider, but the height will not change.

I’ll leave it at that for now. Expect more rambling in a future post.

And while you are waiting for that one, you can read what
David Millington said in his blog posts on How to Design a Great UI (part 2, part 3).

Some remarks on “How Design a Great UI”

 Delphi  Comments Off on Some remarks on “How Design a Great UI”
Oct 112018
 

David Millington has published his third blog post on How to Design a Great UI. While I agree with many of his points I disagree with some others. So here is what I disagree with and why:

Use BorderStyle bsDialog

This means several things:

  1. The dialog has a thicker border than usual
  2. The dialog has a close icon only.
  3. The dialog cannot be sized.

The last point is what I strongly disagree with for dialogs like the one in his example. It has got a list of chapters from a book that are to be exported. This list can be rather long so it is quite likely that it at one time get a vertical scroll bar. Also, chapter headings might be much longer than simply “Chapter 1” (and you want to display those headings so the user can make an informed decision on which chapter(s) to export.), so it is also likely that the list gets a horizontal scroll bar (or even worse, if the control you use does not support horizontal scrolling: The items get cut off on the right.). This reduces usability significantly. Just have a look at the Microsoft Printer installation dialog:

We are no longer living in the 1990ies, when you could count yourself lucky if you owned a 17″ Monitor with a resolution of 1280×1024 pixels. Today, there are users who have got a a 30 inch monitor or even several of those and with resolutions up to 8K, where this dialog shows only in the center with the perceived size of a postage stamp.
As a user I want to maximize this kind of dialog so I see more entries of the list and also read the whole description.
So, rather than using BorderStyle = bsDialog, use bsSizeable and make sure that you set the Anchors and Align properties of all controls in a way they resize and move sensibly.

Position: poOwnerFormCenter (or alternatively poMainFormCenter.)

Again, this was fine for the 1990ies. But nowadys the owner form can be huge (e.g. full screen on a 30″ monitor) and you clicked a button somewhere on that form (e.g. in the bottom right) to get to this dialog. Would you really like the new dialog to pop up centered on the owner form? Your mouse just moved to that button and your visual focus also did. Now, all of a sudden you have to move your focus (and your mouse) to the middle of the screen. My suggestion in that case would be to center the new dialog on the button the user clicked to open it. Of course this should still make sure that the dialog is fully visible and does not cross monitor borders.

InplaceExeWrapper for those tools that do not allow specifying an output file

 Delphi  Comments Off on InplaceExeWrapper for those tools that do not allow specifying an output file
Oct 072018
 

There are a lot of command line tools that are very useful, but have one flaw: They directly modify a file in place and do not allow you to specify an output file instead.

So, if you e.g. want to compare the modified file to the original, you have to make a copy first. Or, if you just want to view the modified file but keep the original or you cannot modify the file itself because it is immutable due to whatever reason (e.g. it resides on a read only medium like a cdrom), again you have to make a copy first.

My particular use case was the great tool DrpojNormalizer written by Uwe Raabe (and later replaced by ProjectMagician) which I wanted to use for showing changes in Delphi .dproj files in BeyondCompare. These tools are actually Delphi IDE plugins but both come with a command line program too. But unfortunately both modify the file in place so they are one example of the above mentioned category of tools.

Enter InplaceExeWrapper which called as

InplaceExeWrapper --expectfilenameonstdout c:\path\to\dprojnormalizercmd.exe input.dproj output.dproj

does the following:

  1. Create a temporary directory under %TEMP%
  2. Copy the input file input.dproj there
  3. Call the tool as
    dprojnormalizer tempfile.dproj
    
  4. Copy the modified file to the output file output.dproj
  5. Delete the temporary directory

So basically it replaces the missing functionality of specifying an output file for the tool it calls.

Here is the full help on command line parameters and options:

Synopsis: InplaceExeWrapper [options] Executable InFile [OutFile]

Parameters:
Executable        : Executable to call (if set to "copy" we only copy InFile to OutFile)
InFile            : input filename
OutFile           : output filename, defaults to infile

Options:
--CheckResult=value : If set, the executable must return the value given for this option (must be a number)
--debug           : if given, some debug output is written to error.txt in the temp directory and the temp directo
ry will not be deleted.
--ExpectFilenameOnStdout : if set, the output of the executable must contain the filename
--help
-?
-h
-H                : display parameter help
--ShowCmdLine     : Show command line as passed to the program.
--StartupLog=value : Write a startup log to the given file.
--TempDir=value   : directory to use for temporary files (must exist)
--toStdOut        : if set, output is written to stdout and InFile is not changed

The option –ExpectFilenameOnStdout can be used to detect if the called program actually worked. If the output does not contain the file name InplaceExeWrapper assumes that the call failed.

There is the special “executable” parameter “copy” which simply copies InFile to OutFile. I needed it to make the file editable in BeyondCompare. Here is the required configuration to be added to Tools -> File Formats:

Name: Delphi Project
Mask: *.dproj
Conversion:
Loading:

Path\to\InplaceExeWrapper.exe  --expectfilenameonstdout "Path\To\dprojnormalizercmd.exe" %s %t

Saving:

Path\to\InplaceExeWrapper.exe copy %s %t

InplaceExeWrapper has a project page on OSDN which includes a binary download as well as the source code.

List of my open source projects

 blog  Comments Off on List of my open source projects
Oct 062018
 

Since I keep forgetting which project is hosted where, here is a list of all(?) my open source projects as of 2018-10-06 with links to the pages where they are being hosted.

Most of them are written in Delphi with some Lazarus (for Windows) sprinkled in.

My old homepage still lists other stuff, some of it written in Perl or even ReXX.

Denkzettel Companion 0.0.5 released

 Delphi  Comments Off on Denkzettel Companion 0.0.5 released
Oct 062018
 

I have just released version 0.0.5 of my Denkzettel Companion tool. It is available from OSDN.

This version fixes a bug that prevented the program to start when the notes directory did not exist.
(Since nobody reported that problem, probably nobody but me has ever used the tool.)

Also, I have moved that project to OSDN now too.

Calling Application.ProcessMessages in a Delphi program

 Delphi  Comments Off on Calling Application.ProcessMessages in a Delphi program
Sep 292018
 

Using Application.ProcessMessages is being frowned upon by many because often it means that your program design is flawed. These people usually suggest you should use multi-threading instead, which then opens another can of worms.

First of all, let me make clear, that I am talking about Windows programs written in Delphi using the Visual Component Library (VCL). This is not about Android or IOS development and also not about the Firemonkey (FMX) framework and most certainly not about any dotNET language.

With that out of the way, let’s discuss what Application.ProcessMessages does and why it is being used:

Application is the global object declared by the VCL in the Forms unit which provides methods that are independent of forms. One of them is ProcessMessages. What it does is looking into the Windows message queue and processing any messages it may find there. Since Windows GUIs are message driven, this is important for a program to react to user input as well as other events, e.g. redrawing controls that have changed.

Normally this processing of messages is done automatically in the program’s main message loop that is executed all the time when it is idle. But if your program does some long running work after e.g. the user pressed a button, it might take a while until it gets back to executing the message loop and to the user it will seem to have crashed or hung.

To alleviate this, for many years Delphi programmers have added the occasional call to Application.ProcessMessages in their code. This usually looks like this:

proccedure TMyForm.b_Execute(Sender: TObject);
var
  i: integer;
begin
  for i := 0 to GetCount - 1 do begin
    DoSomeWork(i);
    Application.ProcessMessages;
  end;
end;

Where DoSomeWork does part of the total work and returns. And then the program calls Application.ProcessMessages to update the user interface.

Why is this bad?

  • Processing Windows messages may have some undesired side effects. E.g.:
    • Your form might have multiple buttons which do different things. ButtonA might start a long running calculation A while ButtonB might start a different calculation B. And it’s most likely that you don’t want to start calculation B while calculation A is still running.
    • Even if your form only has one button that starts an action you still don’t want that action to be started again while the first run hasn’t finished.[*]
    • Your calculation might access the user interface to retrieve some configuration. If you do that frequently and in between call Application.ProcessMessages, these settings may have changed, so you might start with one set of configurations and continue with a different one. The result will most likely not be what you expect.
    • The user might close the form while the calculation is still running. That could result in resources that the calculation needs being freed, which usually causes errors like Access Violations.
  • Each call to Application.ProcessMessages takes time which might slow down your calculations significantly.

So, should we not call Application.ProcessMessages? And how do we keep the GUI active?

As mentioned above, the answer you usually get is to use multi-threading. That is: Move the work into a secondary (or even many secondary) worker thread and let the main thread take care of the user interface. If done properly this works very well. But can you do it properly? Well, I for one have written my share of multi-threaded programs and I can tell you: It’s difficult to do it properly. And most of the time it isn’t necessary.

One of the main stumble blocks is debugging. The Delphi debugger is not particularly suited for debugging multi-threaded programs. It shows you a list of threads and the code which is being executed by the currently active thread (on a multi core system, there is more than one currently active thread, I have no idea how the debugger selects the one to display). You can switch to a different thread and look at the code it is executing. Even worse: The debugger will switch between the threads automatically for no apparent reason, so you press F8 to step over one command and find yourself in the code of a different thread all of a sudden. Other than the debugger, there are of course other debugging methods, e.g. writing to a log or showing message boxes. The latter can only be done safely in the main thread because the VCL is not thread safe. Writing a log will pose the challenge of accessing that log in a thread safe manner.

All that does not mean, you should not use multi-threading if you need it. It just means that you should consider the implications this will have on the complexity of your code and debugging it.

This blog post is not about multi-threading in particular but about Application.ProcessMessages, so I will not go down that route any further.

So, what can we do with a single threaded program to alleviate the above mentioned pitfalls of Application.ProcessMessages?

  • First of all: Once you start processing some data, stop the user from interfering.
    • Disable the controls in the user interface, including the button that just started the processing. This also gives the user feedback that the program is busy right now. He cannot start anything else and also not change the settings being used by the calculations. Don’t forget to enabled the controls afterwards (e.g. use Try…Finally to ensure them to be re-enabled.)
    • Prevent the current form from being closed. This can be done with an OnCloseQuery event.
  • Call Application.ProcessMessages as often as necessary, but not too often.

In addition you should provide a visual feedback of the progress and have an Abort button that allows the user to – well – abort the calculations.

Basically you do something like this:

procedure TMyForm.SetIsRunning(_Value: boolean);
begin
  // set a flag that can be checked elsewhere
  FIsRunning := _Value;
  // en-/disable the button that started this
  b_Execute.Enabled := not _Value;
  // en-/disable any configuration controls
  fe_InputFile.Enabled := not _Value;
  fe_OutputFile.Enabled := not _Value;
  chk_SomeSetting.Enabled := not _Value;
  // en-/disable the Abort button, note the missing "not" here!
  b_Abort.Enabled := _Value;
  // ensure that the Abort button has the correct caption
  b_Abort.Caption := _('Abort ...');
  // reset a flag that tells you to abort
  FAborting := False;
end;

procedure TMyForm.FormCloseQuery(Sender: TObject; var CanClose: Boolean);
begin
  Result := not FIsRunning;
  // you might want to add a message box here to ask the user if he
  // wants to abort
end;

procedure TMyForm.b_AbortClick(Sender: TObject);
begin
  // you might want to add a message box here to ask the user if he
  // really wants to abort

  // set the flag that the user wants to abort processing
  FAborting := True;
  // give him feedback that we are doing what he requested
  b_Abort.Caption := _('Aborting ...');
end;

procedure TMyForm.b_ExecuteClick(Sender: TObject);
var
  LastAPMCall: Int64;
  ThisAPMCall: Int64;
  i: integer;
begin
  SetIsRunning(True);
  try
    LastAPMCall := GetTickCount;
    for i := 0 to GetCount - 1 do begin
      DoSomeWork(i);
      // now, instead of calling Application.ProcessMessages every time
      // we check that the last call was more than 100 ms ago
      ThisAPMCall := GetTickCount;
      if (ThisAPMCall - LastAPMCall > 100) or (ThisAPMCall < LastAPMCall) then begin
        // GetTickCount is a 32 bit value that wraps around about
        // every two weeks. That's what the second check is for.
        Application.ProcessMessages;
        LastAPMCall := ThisAPMCall;
        if FAborting then begin
          // The user has pressed the Abort button, so we do what he wants
          SysUtils.Abort;
        end;
      end;
    end;
  finally
    SetIsRunning(False);
  end;
end;

That’s the basic principle which many of our internal tools use. It avoids the complexity of multi-threading while still keeping the UI responsive and allow the user to abort processing. I’m not claiming that it is perfect but it works, is simple to understand and also simple to debug.

Drawbacks are the following:

  • If the processing is more complex and should be moved to a procedure in a different unit, it’s easy to forget the Application.ProcessMessages calls. Also if you have these calls in your code, you should not call it in a secondary thread. Bad things will happen, if you do.
  • If you call Application.ProcessMessaging in multiple places, it might not be as easy as in the example to keep track of how long ago the last call was. One possible solution is to have a global TTimedProcessMessages object that does the time keeping for you.
  • Checking the Abort button is also a problem once your code is no longer in the form’s object. Again, a TTimedProcessMessages object can solve that.
  • You cannot use that code in Firemonkey programs and, most unfortunately, you cannot use it for cross platform programs (which would require Firemonkey anyway.)
  • Nowadys most computers have multiple cores. This is a single threaded program, so it will only use the processing power of one core. Depending on the amount of processing that needs to be done and whether it can be split into independent work packages, a multi-threading solution would be much more efficient.

Oh, you might ask, where this TTimedProcessMessages object I am talking about can be found. Here you go:

type
  PBoolean = ^Boolean;
  TTimedProcessMessages = class
  private
    FDummyBool: Boolean;
    FAbortedFlag: PBoolean;
    FLastCall: Int64;
    FMinTicks: Int64;
  public
    constructor Create(_MinTicks: integer; _AbortedFlag: PBoolean = nil);
    function Execute(var _WasAborted: boolean): boolean; overload;
    function Execute: boolean; overload;
  end;

constructor TTimedProcessMessages.Create(_MinTicks: integer; _AbortedFlag: PBoolean = nil);
begin
  inherited Create;
  FMinTicks := _MinTicks;
  if Assigned(_AbortedFlag) then
    FAbortedFlag := _AbortedFlag
  else
    FAbortedFlag := @FDummyBool;
end;

function TTimedProcessMessages.Execute(var _WasAborted: boolean): boolean;
var
  ThisCall: Int64;
begin
  ThisCall := GetTickCount;
  Result := (ThisCall - FLastCall > FMinTicks) or (ThisCall < FLastCall);
  if Result then begin
    Application.ProcessMessages;
    _WasAborted := FAbortedFlag;
  end else
    _WasAborted := False;
end;

function TTimedProcessMessages.Execute: boolean;
var
  DummyBool: Boolean;
begin
  Result := Execute(DummyBool);
end;

You either create a global variable (no, of course that’s bad) a singleton of this class

  gblTimedProcessMessages := TTimedProcessMessages.Create(100, @FAborting);

or create an instance and pass it around to all methods that need it. These methods then use it like this:

  if gblTimedProcessMessages.Execute(WasAborted) and WasAborted then
    SysUtils.Abort;

(Note: All the code above has just been typed in as I went along, so it might contain bugs. I haven’t even tried to compile it.)

There you go. Good thing I have disabled comments on this blog anyway, so there will be no way for people to call me unprofessional because I recommend using Application.ProcessMessages. 😉

But if you really want to call me names, here is the corresponding Google+ post.

Footnotes

[*]
As Lars Fosdal pointed out in his comment on my Google+ post:
You also need to worry about reentrancy. I.e. processing a new message before the processing of the previous one have completed, something that is particularly troublesome during UI updates.

Delphi LongWord is not always a 32 bit unsigned integer

 Delphi  Comments Off on Delphi LongWord is not always a 32 bit unsigned integer
Sep 162018
 

Did you know that LongWord in Delphi is not (any longer) always a 32 bit unsigned integer?

I wasn’t aware of this until I asked about what others use for that data type in the Delphi Developers Google+ community.

I turned out that the majority of the participants use Cardinal, followed by Uint32.

Only less than 10% use LongWord.

Stefan Glienke was the first to point out that LongWord is not always 32 bits, even though the 64-bit Windows Data Types Compared to 32-bit Windows Data Types entry in the DokWiki seems to suggest it. But it only covers Data types for Windows.

There is a better overview table on the Simple Types reference page. Even though I find it rather odd that they say that Integer and Int64 are aliases for NativeInt (I wouldn’t have had that “alias” column in the Platform Dependent Integer Types table at all.)

So, where does that leave me? I guess it’s best to switch to Int / UInt + bitness for those types where it is important how many bits an integer has (such as in records that define data written to a file). For the other cases it might be better to use NativeInt or NativeUInt, which translates to Int32 / UInt32 on 32 bit platforms and Int64 and UInt64 on 64 bit platforms. Unless of course they don’t.

NativeInt / NativeUInt type in various Delphi versions

 Delphi  Comments Off on NativeInt / NativeUInt type in various Delphi versions
Sep 082018
 

Just in case you are maintaining Delphi code for several older versions of Delphi: Be aware that the declarations of NativeInt and NativeUInt are wrong in some of them.

Delphi version SizeOf(Native(U)Int) Win32 SizeOf(Native(U)Int) Win64
1 to 6 not available not available
7 to 2007
8 ← this is wrong
not available
2009 to XE 4 not available
XE2 to XE8 4 8
10.x 4 8

I just stumbled over this in GExperts, which had a wrong IFDEF for overwriting that declaration.