Splitting and joining tar archives

 Linux  Comments Off on Splitting and joining tar archives
Apr 272015

This is just a note to myself about splitting a tar archive into multiple smaller files on the fly and joining them again. It’s based on this answer to this question on stackoverflow.

# create archives
$ tar cz . | split -b 100GiB - /mnt/8TB/backup.tgz_

This uses the tar command to create a gzip-ed archive of all files in the directory tree starting with the current directory. The output is written to stdout.
This output is then piped into the split command that splits it into multiple files of 100 Gibibyte and writes them using the given prefix, appending aa, ab, ac etc. The "-" tells split to process its stdin rather than a file.
The input amount is massive and contains very may hard links (it’s a daily dirvish backup dating back 2 years).

To restore from this backup, the following command will be used:

# uncompress
$ cat /mnt/8TB/backup.tgz_* | tar xz

This uses the cat command to pipe all input files to stdout, concatenating them implicitly in the process. This is then piped to the tar command again which extracts it as a gzip-ed archive to the current directory. (I have to try that yet, the backup is still running.)

 Posted by on 2015-04-27 at 10:49

Input validation in dzLib

 Delphi, dzLib  Comments Off on Input validation in dzLib
Apr 152015

In a recent Google+ post Andrea Raimondi was mentioning the JVCL’s JvValidators components and asked whether we use/know them or not. Daniela Osterhagen mentioned that she preferred the input validation from my dzlib which led to a short discussion about how I implemented it.

I added input validation functionality to dzlib because I didn’t like the way JvValidators works. I’ll not go into the reasons but the most important one was performance, the programs using JvValidators felt like running on an old 286 computer. Instead I’m going to show here, how it currently works. Note that I am not satisfied with this yet, see below for the reasons.

First thing you do, is get an instance of the IdzInputValidator interface by calling the InputValidator function from unit u_dzInputValidator. That function takes two optional TColor parameters OKColor and ErrColor which default to clWindow and clYellow respectively.

procedure TMyForm.CheckInput;
  iv: IdzInputValidator;
  iv := InputValidator; // we just take the default colours

This interface has a number of overloaded Check methods that take different types of controls, I have implemented Check methods for T(Custom)Edit, T(Custom)ComboBox, T(Custom)Checkbox and TJvDateEdit (but see below). These methods return specialized interfaces that are meant to validate the data in theses controls. I’ll show that using a TEdit for entering an integer number between 5 and 10 (including these limits) as an example.

   // continued from above
   iv.Check(ed_NumberInput).AsInteger.IsBetween(5, 10);

If it is allowed for the TEdit to be left empty, there is this alternative:

   // continued from above
   iv.Check(ed_OptionalNumberInput).AsInteger.IsBetweenOrEmpty(5, 10);

These are actually several function calls wrapped into one line:
Check, returns an IdzEditValidator interface, AsInteger returns a IdzEditIntegerValidator interface and IsBetween / IsBetweenOrEmtpy returns a boolean. I am using interfaces because I am a lazy bastard™. I don’t want the hassle of freeing the classes that are being created on the fly.

After you have validated the input of all these controls, you ask the IdzInputValidator interface for the result. There are two overloaded GetResult methods, one simply returns a boolean, the other also returns an error message. I am assuming here that there is a TStatusBar on the form which will display this error message and an OK button that should be enabled only if the input is valid.

  // continued from above
  b_Ok.Enabled := iv.GetResult(ErrMsg); // note: Declare ErrMsg: string
  TheStatusBar.SimpleText := ErrMsg;

To make CheckInput do anything at all, you need to call it when the data in the form changes, e.g. in the OnChange events of the controls.

procedure TMyForm.ed_NumberInputChange(Sender: TObject);

Behind the scene IdzInputValidator and friends change the background colour of the input controls based on whether the input is valid or not. If it is valid, the colour is set to the OKColor, if not, it is set to the ErrColor. These colours were passed to the call to the InputValidator function right at the start and default to clWindow and clYellow. It’s a matter of (bad?) taste which colours you prefer. I like yellow because it still allows to read the input easily in contrast to e.g. red. Instead of colouring the controls it is also possible to show a blinking icon with an exclamation mark like JvValidators does but I am not going into that right now.

This implementation is already the second rewrite of the code. In the beginning IdzEditValidator used to have lots of methods like ValidateFloat, ValidateFloatBetween, ValidateInteger, ValidateIntegerBetween, ValidateDate etc. I changed it to have these As<someType> methods that return more specialized interfaces because it became unwieldy. I was pretty satisfied with this implementation until there was a requirement to validate additional controls.

The first one was TJvDateEdit. This required JvToolEdit to be added to the uses clause of u_dzInputValidator. Not every program uses the JVCL, so I added the conditional define NO_JVCL to exclude this code. Not pretty, but I could live with it.

The next one was TSigFilenameEdit (which is an internally developed control at work that is a bit similar to TJvFilenameEdit). It required an ugly hack because I could not simply add the unit that defines it to the uses clause of u_dzInputValidator.

It became worse when there was a requirement to not only validate the controls on a form but the validation was to span several frames which were put on this form.

Generally this problem is referred to as tight a coupling, in this case between the validator interface an the controls. There are a few ideas to resolve this issue:

1. I could create descendants of IdzInputValidator that know additional controls, so there would be additional Check methods that take a TSigFilenameEdit or even a Tfr_MyFrame parameter. This breaks down as soon as there are multiple, mutually exclusive controls to support. e.g. Support for JVCL controls and internal controls, because there is no multiple inheritance support in Delphi. (Hm, I’m not even sure whether mutliple inheritance could solve that problem. Maybe generics or rather my pseudo templates could? I’ll have to thing about this one.)

2. I could create overloaded functions ControlValidator that take a control and an IdzInputValidator interface like this:

function ControlValidator(_ed: TEdit; _iv: IdzInputValidator): IdzEditValidator; overload;
function ControlValidator(_ed: TJvDateEdit; _iv: IdzInputValidator): IdzJvDateEditValidator; overload;

These functions, in contrast to the methods of an interface, could be distributed over multiple units so I could include only those units that support the controls I am currently using. (This would be similar to the different IDatasetHelper implementations for BDE, ADO, ZEOS, tdbf etc. in dzlib.) This would sacrifice some convenience but would pretty much work. I don’t like the additional iv parameter, it just clutters the code. Also, the programmer would be responsible to add the unit containing the function declaration, e.g. for supporting TJvDateEdit add u_dzInputValidatorJVCL.

3. I could come up with some kind of registration mechanism for control validators that register validators for various kinds of controls with a central factory. That factory would then return the correct validator interface for a given control. That would add more complexity to the currently simple way of the validation mechanism. Also, how do I then call the actual validation methods? Have a generic interface that takes e.g. a string describing the check I want to do? No, too ugly.

4. I could revert to checking strings rather than control content. Maybe all that is required is a way to get the control’s content as a string, so TCheckbox.Checked would be converted to ‘Y’ and TCombobox.ItemIndex with IntToStr(…), so solution 3. could be simplified to supply an interface that returns a string which then gets checked. That could be even taken further to using regular expressions. (Remember that joke: You have a problem, you decide to solve it with regular expressions, now you have two problems.)

I am currently leaning towards solution 2. It allows the most flexibility without adding too much overhead. But I am going to publish this blog post on G+ to ask for feedback. Maybe somebody else has a brilliant idea?
Here is the G+ post.

 Posted by on 2015-04-15 at 16:33

Experimental GExperts Version 1.37 2015-04-11 released

 Delphi, GExperts  Comments Off on Experimental GExperts Version 1.37 2015-04-11 released
Apr 112015

The latest version is the first version that supports Delphi XE8.

There is nothing really new about the formatter code. But the new release can be installed even if you don’t have the official GExperts installer (yet). This is an extract from the readme file:

** Installing without an official installer **
With two Delphi Releases per year and Erik Berry being busy otherwise,
new GExperts releases have been lagging behind. So in case there is no
official GExperts installer yet, these are the steps to install the
experimental version by hand:

1. Create a suitable GExperts directory. It doesn't have to be a
   subdirectory of %ProgramFiles% but may be anywhere, even on a network
   share if you are sure this share will always be available.
2. Extract all files from the ZIP somewhere
3. Copy all files from the extracted directory to the GExperts directory.
   (Do *NOT* copy the subdirectories!)
4. Copy the appropriate GExperts DLL from one of the subdirectories
   EditorExpert or RegularExpert to the GExperts directory.
5. Copy the files from the subdirectory FromRegularRelease to the
   GExperts directory.
6. Copy the appropriate cmd from the install subdirectory. To the GExperts
7. Make sure that the Delphi IDE is not running
8. Run the cmd file. It will register the GExperts dll with the Delphi IDE.
9. Start Delphi and check that the new GExperts DLL has been loaded by
   opening the GExperts About dialog.

In theory it is possible to install GExperts for all Delphi versions into
the same GExperts directory. But be warned: This has not been tested

Head over to the Experimental GExperts page to download the new version.

 Posted by on 2015-04-11 at 17:55

Running SyncThing as a Windows service

 Windows  Comments Off on Running SyncThing as a Windows service
Apr 092015

Yesterday I blogged about SyncThing as an alternative to BitTorrentSync. Today a business partner asked me whether it is possible to run SyncThing as a service. The answer is definitely yes, but it requires an additional tool.

As per this StackOverflow answer you can use NSSM – the Non-Sucking Service Manager to create a service that works fine.

I had to add the following command line arguments to the call:

  • -no-console
  • -no-browser
  • -home=”c:\users\<myaccount>\AppData\Local\Syncthing”

(Mind the extra dash after the “no”!)
The latter was because I wanted to keep using the configuration which was generated by just starting the SyncThing console application. It’s perfectly possible to move this folder somewhere else.

Also, on the I/O tab, I added log files for stdout and stderr, so I am able to see any error messages it might write. I didn’t specify any file rotation yet, but that’s probably something you want to do.

After creating the service using NSSM, I could simply start the SyncThing service and connect to it with the browser.

Due to a restriction of our router (no UPnP), I also had to configure port forwarding for port 22000 as described here for it to be able to connect to other servers over the internet.

 Posted by on 2015-04-09 at 10:55

SyncThing as an alternative to BitTorrentSync (btsync)

 Android, Linux, Windows  Comments Off on SyncThing as an alternative to BitTorrentSync (btsync)
Apr 082015

A while ago I blogged about using BitTorrentSync as a privacy conscious alternative to the more popular cloud services like Google Drive or DropBox.

BitTorrent recently released btsync version 2 which, apart from trying to sell you a so called PRO version, changed the user interface yet again and also changed the way you set up your installations significantly. Actually, there seems to be no upgrade path, you have to configure your peers all over again. And, just in case that’s not enough incentive for looking for an alternative, the new Windows version does no longer run on Windows server OSes.

One possible alternative is SyncThing. It’s also peer to peer and the configuration is quite similar. In contrast to btsync it is open source. It is available for most desktop OSes and also for Android.

I tested the (official) 64 bit versions for Windows and Linux and it worked so far. For our Ubuntu server I used the initd script from the SyncThing forum. On Windows I just started the program in the console.

Next step is the Android version.

 Posted by on 2015-04-08 at 16:52

Windows annoyance number 3 gazillion and 4

 Windows, Windows 7  Comments Off on Windows annoyance number 3 gazillion and 4
Apr 082015

We all know, that Microsoft software is great and that they are constantly striving to improve the user experience (best example: Windows 8 (Yes, that’s sarcasm)).

Windows 7, the best Windows of all times, has a great feature which is an improvement of the universally loved desktop cleanup we know from Windows XP. It regularly helps us to get rid of broken and unused shortcuts on the desktop, which are a serious scourge for humanity. To be even more helpful, it doesn’t ask you but just does its task. And to add to its usefulness, you can’t disable it (Yes, more sarcasm.).

This lead to my coworkers complaining that they lose all their icons on the desktop around every 4 weeks. Some of them accept it, restore them and live with the conveniences of the great Windows 7. Others call me and ask me what to do.

There is an entry in the Microsoft Knowledgebase about this feature.

What it says is basically this:

  • It’s not a bug, it’s a feature.
  • There is a hotfix for it, which you can request by email only
  • Once you have installed that hotfix, you can run a Fixit tool that changes two registry entries.

Unfortunately I can’t provide the hotfix downloads (there are two different ones, one for 32 bit Windows 7 and one for 64 bit Windows 7). The files are called Windows6.1-KB2642357-x64.msu and Windows6.1-KB2642357-x86.msu respectively.

What I can do, is provide you with a .reg file that changes those two registry entries. This will only work after you have applied the hotfix!.

Windows Registry Editor Version 5.00


Put this into a text file, change its extension to .reg and open it on the computer that has the problem.

It worked for me. If it doesn’t work for you, don’t blame me, praise Microsoft.

 Posted by on 2015-04-08 at 10:54