Splitting and joining tar archives

 Linux  Comments Off on Splitting and joining tar archives
Apr 272015
 

This is just a note to myself about splitting a tar archive into multiple smaller files on the fly and joining them again. It’s based on this answer to this question on stackoverflow.

# create archives
$ tar cz . | split -b 100GiB - /mnt/8TB/backup.tgz_

This uses the tar command to create a gzip-ed archive of all files in the directory tree starting with the current directory. The output is written to stdout.
This output is then piped into the split command that splits it into multiple files of 100 Gibibyte and writes them using the given prefix, appending aa, ab, ac etc. The "-" tells split to process its stdin rather than a file.
The input amount is massive and contains very may hard links (it’s a daily dirvish backup dating back 2 years).

To restore from this backup, the following command will be used:

# uncompress
$ cat /mnt/8TB/backup.tgz_* | tar xz

This uses the cat command to pipe all input files to stdout, concatenating them implicitly in the process. This is then piped to the tar command again which extracts it as a gzip-ed archive to the current directory. (I have to try that yet, the backup is still running.)

Input validation in dzLib

 Delphi  Comments Off on Input validation in dzLib
Apr 152015
 

In a recent Google+ post Andrea Raimondi was mentioning the JVCL’s JvValidators components and asked whether we use/know them or not. Daniela Osterhagen mentioned that she preferred the input validation from my dzlib which led to a short discussion about how I implemented it.

I added input validation functionality to dzlib because I didn’t like the way JvValidators works. I’ll not go into the reasons but the most important one was performance, the programs using JvValidators felt like running on an old 286 computer. Instead I’m going to show here, how it currently works. Note that I am not satisfied with this yet, see below for the reasons.

First thing you do, is get an instance of the IdzInputValidator interface by calling the InputValidator function from unit u_dzInputValidator. That function takes two optional TColor parameters OKColor and ErrColor which default to clWindow and clYellow respectively.

procedure TMyForm.CheckInput;
var
  iv: IdzInputValidator;
begin
  iv := InputValidator; // we just take the default colours

This interface has a number of overloaded Check methods that take different types of controls, I have implemented Check methods for T(Custom)Edit, T(Custom)ComboBox, T(Custom)Checkbox and TJvDateEdit (but see below). These methods return specialized interfaces that are meant to validate the data in theses controls. I’ll show that using a TEdit for entering an integer number between 5 and 10 (including these limits) as an example.

   // continued from above
   iv.Check(ed_NumberInput).AsInteger.IsBetween(5, 10);

If it is allowed for the TEdit to be left empty, there is this alternative:

   // continued from above
   iv.Check(ed_OptionalNumberInput).AsInteger.IsBetweenOrEmpty(5, 10);

These are actually several function calls wrapped into one line:
Check, returns an IdzEditValidator interface, AsInteger returns a IdzEditIntegerValidator interface and IsBetween / IsBetweenOrEmtpy returns a boolean. I am using interfaces because I am a lazy bastard™. I don’t want the hassle of freeing the classes that are being created on the fly.

After you have validated the input of all these controls, you ask the IdzInputValidator interface for the result. There are two overloaded GetResult methods, one simply returns a boolean, the other also returns an error message. I am assuming here that there is a TStatusBar on the form which will display this error message and an OK button that should be enabled only if the input is valid.

  // continued from above
  b_Ok.Enabled := iv.GetResult(ErrMsg); // note: Declare ErrMsg: string
  TheStatusBar.SimpleText := ErrMsg;
end;

To make CheckInput do anything at all, you need to call it when the data in the form changes, e.g. in the OnChange events of the controls.

procedure TMyForm.ed_NumberInputChange(Sender: TObject);
begin
  CheckInput;
end;

Behind the scene IdzInputValidator and friends change the background colour of the input controls based on whether the input is valid or not. If it is valid, the colour is set to the OKColor, if not, it is set to the ErrColor. These colours were passed to the call to the InputValidator function right at the start and default to clWindow and clYellow. It’s a matter of (bad?) taste which colours you prefer. I like yellow because it still allows to read the input easily in contrast to e.g. red. Instead of colouring the controls it is also possible to show a blinking icon with an exclamation mark like JvValidators does but I am not going into that right now.

This implementation is already the second rewrite of the code. In the beginning IdzEditValidator used to have lots of methods like ValidateFloat, ValidateFloatBetween, ValidateInteger, ValidateIntegerBetween, ValidateDate etc. I changed it to have these As<someType> methods that return more specialized interfaces because it became unwieldy. I was pretty satisfied with this implementation until there was a requirement to validate additional controls.

The first one was TJvDateEdit. This required JvToolEdit to be added to the uses clause of u_dzInputValidator. Not every program uses the JVCL, so I added the conditional define NO_JVCL to exclude this code. Not pretty, but I could live with it.

The next one was TSigFilenameEdit (which is an internally developed control at work that is a bit similar to TJvFilenameEdit). It required an ugly hack because I could not simply add the unit that defines it to the uses clause of u_dzInputValidator.

It became worse when there was a requirement to not only validate the controls on a form but the validation was to span several frames which were put on this form.

Generally this problem is referred to as tight a coupling, in this case between the validator interface an the controls. There are a few ideas to resolve this issue:

1. I could create descendants of IdzInputValidator that know additional controls, so there would be additional Check methods that take a TSigFilenameEdit or even a Tfr_MyFrame parameter. This breaks down as soon as there are multiple, mutually exclusive controls to support. e.g. Support for JVCL controls and internal controls, because there is no multiple inheritance support in Delphi. (Hm, I’m not even sure whether mutliple inheritance could solve that problem. Maybe generics or rather my pseudo templates could? I’ll have to thing about this one.)

2. I could create overloaded functions ControlValidator that take a control and an IdzInputValidator interface like this:

function ControlValidator(_ed: TEdit; _iv: IdzInputValidator): IdzEditValidator; overload;
function ControlValidator(_ed: TJvDateEdit; _iv: IdzInputValidator): IdzJvDateEditValidator; overload;

These functions, in contrast to the methods of an interface, could be distributed over multiple units so I could include only those units that support the controls I am currently using. (This would be similar to the different IDatasetHelper implementations for BDE, ADO, ZEOS, tdbf etc. in dzlib.) This would sacrifice some convenience but would pretty much work. I don’t like the additional iv parameter, it just clutters the code. Also, the programmer would be responsible to add the unit containing the function declaration, e.g. for supporting TJvDateEdit add u_dzInputValidatorJVCL.

3. I could come up with some kind of registration mechanism for control validators that register validators for various kinds of controls with a central factory. That factory would then return the correct validator interface for a given control. That would add more complexity to the currently simple way of the validation mechanism. Also, how do I then call the actual validation methods? Have a generic interface that takes e.g. a string describing the check I want to do? No, too ugly.

4. I could revert to checking strings rather than control content. Maybe all that is required is a way to get the control’s content as a string, so TCheckbox.Checked would be converted to ‘Y’ and TCombobox.ItemIndex with IntToStr(…), so solution 3. could be simplified to supply an interface that returns a string which then gets checked. That could be even taken further to using regular expressions. (Remember that joke: You have a problem, you decide to solve it with regular expressions, now you have two problems.)

I am currently leaning towards solution 2. It allows the most flexibility without adding too much overhead. But I am going to publish this blog post on G+ to ask for feedback. Maybe somebody else has a brilliant idea?
Here is the G+ post.

Experimental GExperts Version 1.37 2015-04-11 released

 Delphi, GExperts  Comments Off on Experimental GExperts Version 1.37 2015-04-11 released
Apr 112015
 

The latest version is the first version that supports Delphi XE8.

There is nothing really new about the formatter code. But the new release can be installed even if you don’t have the official GExperts installer (yet). This is an extract from the readme file:

** Installing without an official installer **
   
With two Delphi Releases per year and Erik Berry being busy otherwise,
new GExperts releases have been lagging behind. So in case there is no
official GExperts installer yet, these are the steps to install the
experimental version by hand:

1. Create a suitable GExperts directory. It doesn't have to be a
   subdirectory of %ProgramFiles% but may be anywhere, even on a network
   share if you are sure this share will always be available.
2. Extract all files from the ZIP somewhere
3. Copy all files from the extracted directory to the GExperts directory.
   (Do *NOT* copy the subdirectories!)
4. Copy the appropriate GExperts DLL from one of the subdirectories
   EditorExpert or RegularExpert to the GExperts directory.
5. Copy the files from the subdirectory FromRegularRelease to the
   GExperts directory.
6. Copy the appropriate cmd from the install subdirectory. To the GExperts
   directory.
7. Make sure that the Delphi IDE is not running
8. Run the cmd file. It will register the GExperts dll with the Delphi IDE.
9. Start Delphi and check that the new GExperts DLL has been loaded by
   opening the GExperts About dialog.

In theory it is possible to install GExperts for all Delphi versions into
the same GExperts directory. But be warned: This has not been tested
thoroughly.

Head over to the Experimental GExperts page to download the new version.

Running SyncThing as a Windows service

 Windows  Comments Off on Running SyncThing as a Windows service
Apr 092015
 

Yesterday I blogged about SyncThing as an alternative to BitTorrentSync. Today a business partner asked me whether it is possible to run SyncThing as a service. The answer is definitely yes, but it requires an additional tool.

As per this StackOverflow answer you can use NSSM – the Non-Sucking Service Manager to create a service that works fine.

I had to add the following command line arguments to the call:

  • -no-console
  • -no-browser
  • -home=”c:\users\<myaccount>\AppData\Local\Syncthing”

(Mind the extra dash after the “no”!)
The latter was because I wanted to keep using the configuration which was generated by just starting the SyncThing console application. It’s perfectly possible to move this folder somewhere else.

Also, on the I/O tab, I added log files for stdout and stderr, so I am able to see any error messages it might write. I didn’t specify any file rotation yet, but that’s probably something you want to do.

After creating the service using NSSM, I could simply start the SyncThing service and connect to it with the browser.

Due to a restriction of our router (no UPnP), I also had to configure port forwarding for port 22000 as described here for it to be able to connect to other servers over the internet.

SyncThing as an alternative to BitTorrentSync (btsync)

 Linux, Windows  Comments Off on SyncThing as an alternative to BitTorrentSync (btsync)
Apr 082015
 

A while ago I blogged about using BitTorrentSync as a privacy conscious alternative to the more popular cloud services like Google Drive or DropBox.

BitTorrent recently released btsync version 2 which, apart from trying to sell you a so called PRO version, changed the user interface yet again and also changed the way you set up your installations significantly. Actually, there seems to be no upgrade path, you have to configure your peers all over again. And, just in case that’s not enough incentive for looking for an alternative, the new Windows version does no longer run on Windows server OSes.

One possible alternative is SyncThing. It’s also peer to peer and the configuration is quite similar. In contrast to btsync it is open source. It is available for most desktop OSes and also for Android.

I tested the (official) 64 bit versions for Windows and Linux and it worked so far. For our Ubuntu server I used the initd script from the SyncThing forum. On Windows I just started the program in the console.

Next step is the Android version.

Windows annoyance number 3 gazillion and 4

 Windows, Windows 7  Comments Off on Windows annoyance number 3 gazillion and 4
Apr 082015
 

We all know, that Microsoft software is great and that they are constantly striving to improve the user experience (best example: Windows 8 (Yes, that’s sarcasm)).

Windows 7, the best Windows of all times, has a great feature which is an improvement of the universally loved desktop cleanup we know from Windows XP. It regularly helps us to get rid of broken and unused shortcuts on the desktop, which are a serious scourge for humanity. To be even more helpful, it doesn’t ask you but just does its task. And to add to its usefulness, you can’t disable it (Yes, more sarcasm.).

This lead to my coworkers complaining that they lose all their icons on the desktop around every 4 weeks. Some of them accept it, restore them and live with the conveniences of the great Windows 7. Others call me and ask me what to do.

There is an entry in the Microsoft Knowledgebase about this feature.

What it says is basically this:

  • It’s not a bug, it’s a feature.
  • There is a hotfix for it, which you can request by email only
  • Once you have installed that hotfix, you can run a Fixit tool that changes two registry entries.

Unfortunately I can’t provide the hotfix downloads (there are two different ones, one for 32 bit Windows 7 and one for 64 bit Windows 7). The files are called Windows6.1-KB2642357-x64.msu and Windows6.1-KB2642357-x86.msu respectively.

What I can do, is provide you with a .reg file that changes those two registry entries. This will only work after you have applied the hotfix!.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\ScheduledDiagnostics]
"EnabledExecution"=dword:00000001
"IsBrokenShortcutsTSEnabled"=dword:00000000
"IsUnusedDesktopIconsTSEnabled"=dword:00000000

Put this into a text file, change its extension to .reg and open it on the computer that has the problem.

It worked for me. If it doesn’t work for you, don’t blame me, praise Microsoft.

Delphi Pipe

 blog, Delphi, Delphi Pipe  Comments Off on Delphi Pipe
Mar 282015
 

Since Delphi Feeds doesn’t list my blog any more and generally isn’t very fast in responding to requests for adding or removing feeds, I have created my own aggregated feed: RSS-02Delphi Pipe. It’s done with Yahoo Pipes and currently contains the following feeds in no particular order (*1):

Many of these are also on Delphi Feeds and there are probably many interesting feeds that are still missing. If you want to add a feed, please contact me via my Gooogle+ profile or post a comment to the announcement on Google+. I promise to do my best to maintain the pipe, but that doesn’t mean that I will respond immediately.

You can see the current content of the pipe to the right.

I have also created a static page and subdomain for it: delphipipe.dummzeuch.de

(*1: Meaning that I have yet to be able to create a meaningful sort order for the rss feed listing the sources.)

When const doesn’t mean const

 Delphi  Comments Off on When const doesn’t mean const
Mar 202015
 

Consider this code:

procedure SplitAt(const _Input: string; _Position: integer; out _Head, _Tail: string);
begin
  _Head := Copy(_Input, 1, _Position-1);
  _Tail := Copy(_Input, _Position);
end;

It’s meant to split a given string into two parts at the given position.

Nothing fancy, really, isn’t it?

Now, consider this call to the procedure above:

var
  s1: string;
  s2: string;
begin
  s1 := 'hello world';
  SplitAt(s1, 5, s1, s2);
  WriteLn('s1: "', s1, '"');
  WriteLn('s2: "', s2, '"');
end;

Which output do you expect?

The output I got certainly wasn’t what I expected:

s1: ""
s2: ""

I took me a while to understand what happened:

It has been brought to my attention, that the following explanation is wrong:

The compiler passes the parameters right to left, so the first parameter it passes is s2, then s1, then 5 and last s1 again as the input. Since the two rightmost parameters are *out* parameters it clears them before passing them, so s1 is already an empty string when it gets passed as input.

What actually happens is that regardless of the order in which the parameters are passed clearing of the out parameters happens. This clears s1 and it doesn’t matter whether it is was already passed as Input or not since strings in Delphi are always passed as pointers.

Not quite what I would have expected. It doesn’t help to remove the const modifier either.

Accessing the clipboard in a Firemonkey application

 Delphi  Comments Off on Accessing the clipboard in a Firemonkey application
Feb 282015
 

It just took me quite a while to find this information so I’ll put it here for future reference.

A Firemonkey application can not just access the clipboard, it needs to ask the platform whether it actually has one, then get the service interface and use that.

uses
  Fmx.Platform;

[...]
function TryGetClipboardService(out _clp: IFMXClipboardService): boolean;
begin
  Result := TPlatformServices.Current.SupportsPlatformService(IFMXClipboardService);
  if Result then
    _clp := IFMXClipboardService(TPlatformServices.Current.GetPlatformService(IFMXClipboardService));
end;

procedure StringToClipboard(const _s: string);
var
  clp: IFMXClipboardService;
begin
  if TryGetClipboardService(clp) then
    clp.SetClipboard(_s);
end;

procedure StringFromClipboard(out _s: string);
var
  clp: IFMXClipboardService;
  Value: TValue;
  s: string;
begin
  if TryGetClipboardService(clp) then begin
    Value := clp.GetClipboard;
    if not Value.TryAsType(_s) then
      _s := '';
  end;
end;

(This is for Delphi XE7.)

TEdit and TMemo have got a CopyToClipboard and PasteFromClipboard method. (Which begs the question: Why not implement a generic function for reading and writing a string rather than methods for two controls? I was tempted to use the Visual Basic solution: Put a hidden control on the form and use its methods. Bad memories awake …)