For-in Enumeration

The “For in” style of enumeration in Delphi was introduced in Delphi 2005. The idea is to make enumeration over a collection of things easier and cleaner by removing the need for the iterator variable, removing the code for indexing and removing the code for boundary checking. If you think you know everything about enumeration, hold on to your hat. You may be in for a pleasant surprise.

We will start with the basics and then progress to custom enumerators. Seasoned “enumerophiles” may prefer to skip to the “Custom Enumeration over Objects” section. Let’s look at a simple example. Say we’ve got a list of strings, and we want to show them to the user. Here is how we would do it before and after this feature.

Enumeration by index

procedure EnumDemo1;
var
  SomeStrings: array of string;
  s: string;
  i: integer;
begin
  Populate( SomeStrings);
  for i := 0 to Length( SomeStrings) - 1 do
  begin
    s := SomeStrings[i];
    Dialogs.ShowMessage( s)
  end
end;

Enumeration by for-in

procedure EnumDemo2;
var
  SomeStrings: array of string;
  s: string;
begin
  Populate( SomeStrings);
  for s in SomeStrings do
    Dialogs.ShowMessage( s);
end;

Looking at the details of the above, you see that for-in enumeration is not exactly earth-shattering. We have dropped a variable declaration, the for-loop is a little cleaner and indexing is compiler-magiced away. But you’ve got the wrong idea if you think that the key selling point of for-in enumeration is these simple syntactic abbreviations. The real selling point is greater than that.

Stand back from the details of the code, and think about what we are doing at an abstract level. We are looping through a collection of strings and doing something with the members. From this perspective, the mechanism of indexing is a pesky irrelevant detail. What manner of collection it is (array, list, something-else) is a pesky irrelevant detail. Boundary checking? – another irrelevant artefact of implementation. What we have achieved in EnumDemo2() over EnumDemo1(), is the holy grail of language design – a closer alignment of semantics with syntax.
Alignment of semantics with syntax

Standard Enumerables

So what can we enumerate over, out-of-the-box? The following VCL types are for-in enumerable? Member types are as you would expect.

Fundamental and compound types – Enumerables

  • Dynamic arrays
  • Static arrays
  • string/ ansistring/ utf8string
  • sets

Sets are a special case. Enumeration only visits members that are in the set, as opposed to the base range.

Object Enumerables

  • TArray<T>
  • TStrings
  • TList
  • TList<T>
  • TObjectList
  • TObjectList<T>
  • TCollection
  • TQueue<T>
  • TStack<T>
  • TDictionary<T>
  • TInterfaceList
  • TObjectStack<T>
  • TComponent
  • TFields
  • TActionList
  • TTreeNodes
  • TListItems
  • TToolBar
  • TMenuItem
  • A large suite of RTTI properties.

Interface pointer Enumerables

  • IInterfaceListEx

What’s missing?

So what is missing from Delphi out-of-the-box for-in enumeration support?
What's missing
Perhaps in a future version of Delphi we will get enumeration support for:

  • Enumeration types. With some custom declarations, enumeration of “enumerated types” has been solved by Jeroen W. Pluimers
  • Set inversion. Currently, there is no way to for-in enumerate over members NOT in set. Wouldn’t it be handy if we had a generic function to invert sets.
  • Character Streams. Ever wanted to parse a text file character by character? Wouldn’t it be nice if we could use for-in?
  • Reverse order. Have you ever wanted to traverse in reverse order? How useful would this be?
  • TBits
  • TDataset – How many times have you written a while-not-eof loop? For us older developers, the number is probably astronomical.
  • Predicates. Ever wanted to traverse a collection, but only for those members that met some specified condition. Sure we can do this with a simple if statement, but this pattern is so common, the thought of an XSLT style syntax mechanism for predicates is enticing. Imagine if we could do something like FantasyEnum4()

Predicated enumeration as it is now…

procedure EnumDemo3( Collection: TObjectList<TSomeClass>);
var
  Member: TSomeClass;
begin
  for Member in Collection do
    if SomeCondition( Member) then
      Member.SomeAction
end;

Fantasy enumeration supporting predicates

procedure FantasyEnum4( Collection: TObjectList<TSomeClass>);
var
  Member: TSomeClass;
begin
  for Member in Collection ! SomeCondition( Member) do
    Member.SomeAction
end;

Generics and Enumeration

Generic classes and for-in enumeration are a good match-up. Take a look at the syntactic synergy in EnumDemo3 above. If we wanted to do the same in Delphi 7, we would probably write something like EnumDemo5() below…

procedure EnumDemo5( Collection: TObjectList);
var
  i: integer;
  Member: TSomeClass;
begin
  for i := 0 to Collection.Count - 1 do
  begin
    Member := Collection[i] as TSomeClass;
    if SomeCondition( Member) then
      Member.SomeAction
  end
end;

Custom Enumeration over objects

So much for the basics. Now here is where it gets really interesting. With some customisation you can for-in enumerate over any object of any class.

You can enumerate over objects, records and interface pointers. The enumerators also can either be objects, records or interface pointers. And the collection members of the enumeration can by any type including fundamentals, compounds, objects, interface pointers, anonymous functions, even class references and type references.

When the compiler sees a code like

for Member in Collection do

.. and Collection is an object, it compiles by the following algorithm..

  1. Look for a public function of name GetEnumerator(). The function return type must be a class, record or interface pointer type. Protected and below methods do not count. Class functions are also discounted.
  2. If class, Examine the declaration of the class. There must be public non-class function MoveNext(), with signature…
    function MoveNext: boolean;
    
  3. Also in this class, there must be a public property named Current. The read accessor method for Current may be any thing, even strict private.
  4. The type for the current property must be equal to the declared type of the enumerated value (‘Member” in our above example), or, if the type is an object type, the current class type may be a descendant of declared member class. If the member type is an interface type, then the current property interface type may be an interface descendant of the declared member interface type. It is not required for interface types to have syntactically bound guids.
  5. The MoveNext() and Current are utilised as follows…

With declarations …

TMyEnumerator = object
  strict private
    function GetCurrent: TMemberClass;
  public
    property Current: TMemberClass read GetCurrent;
    function MoveNext: Boolean;
  end;

TMyCollection = object
  public
    function GetEnumerator: TMyEnumerator;
  end;

…this procedure …

procedure EnumDemo6a( Collection: TMyCollection);
var
  Member: TMemberClass;
begin
  for Member in Collection do
    SomeAction( Member)
end

… is compiled as if we wrote instead…

procedure EnumDemo6b( Collection: TMyCollection);
var
  Member: TMemberClass;
  Enumerator: TMyEnumerator;
begin
  Enumerator := Collection.GetEnumerator;
  try
    while Enumerator.Next do
    begin
      Member := Enumerator.Current;
      SomeAction( Member)
    end
  finally
    Enumerator.Free
  end
end

If the Enumerator is an interface pointer, then instead of being freed, it is released (reference count decremented). If the Enumerator is a record, then instead of being freed, it is record finalized and de-scoped.

If you are making custom enumerators, you may find it convenient to leverage these base classes and interface pointer types from the VCL…

  IEnumerator = interface(IInterface)
    function GetCurrent: TObject;
    function MoveNext: Boolean;
    procedure Reset;
    property Current: TObject read GetCurrent;
  end;

  IEnumerable = interface(IInterface)
    function GetEnumerator: IEnumerator;
  end;

  IEnumerator = interface(IEnumerator)
    function GetCurrent: T;
    property Current: T read GetCurrent;
  end;

  IEnumerable = interface(IEnumerable)
    function GetEnumerator: IEnumerator;
  end;

  TEnumerator = class abstract
  protected
    function DoGetCurrent: T; virtual; abstract;
    function DoMoveNext: Boolean; virtual; abstract;
  public
    property Current: T read DoGetCurrent;
    function MoveNext: Boolean;
  end;

  TEnumerable = class abstract
  protected
    function DoGetEnumerator: TEnumerator; virtual; abstract;
  public
    function GetEnumerator: TEnumerator;
  end;

Custom Enumeration over records and interface pointers.

You can enumerate over records and interface pointers too. It works the same way as classes, except that the restriction that the GetEnumerator method be public is not applicable.

Custom Enumeration with helpers

There is a natural synergy between class helpers and custom enumeration. See the next section for examples.

Some incredibly useful examples

Parse a character stream, line by line

With a class helper and custom enumeration, we can conveniently pass a text stream, line by line (or character by character just as easily), like so…

procedure ParseLines( TextStream: TStream);
var
  Line: string;
begin
  for Line in TestStream do
    ParseLine( Line)
end;

One possible implementation to achieve this (with a UTF-8 encoded stream) might be….

TTextStreamHelper = class helper for TStream
  public
    function GetEnumerator: TEnumerator;
  end;

function TTextStreamHelper.GetEnumerator: TEnumerator;
begin
  result := TLineReader.Create( self);
end;

TLineReader = class( TEnumerator)
  private
    coTextStream: TStream;
    ciPosition: Int64;
    csCurrent: string;

  protected
    function DoGetCurrent: string; override;
    function DoMoveNext: Boolean; override;

  public
    constructor Create( poTextStream: TStream);
  end;

constructor TLineReader.Create( poTextStream: TStream);
begin
  coTextStream := poTextStream;
  ciPosition   := coTextStream.Position;
  csCurrent    := ''
end;

function TLineReader.DoGetCurrent: string;
begin
  result := csCurrent
end;

function TLineReader.DoMoveNext: Boolean;
var
  Ch, Ch2: Ansichar;
  sWorking: UTF8String;
  L: integer;
begin
  sWorking := '';
  result   := False;
  while coTextStream.Read( Ch, 1) = 1 do
  begin
    result := True;
    if (Ch <> #13) and (Ch <> #10) then
    begin
      L := Length( sWorking) + 1;
      SetLength( sWorking, L);
      sWorking[L] := Ch;   // Adds a code-unit NOT a character!
    end
    else
    begin
      if (coTextStream.Read( Ch2, 1) = 1) and
           (((Ch = #13) and (Ch2 <> #10)) or
            ((Ch = #10) and (Ch2 <> #13))) then
        coTextStream.Seek( -1, soCurrent);
      break
    end;
  end;
  csCurrent := UTF8ToString( sWorking)
end;

XML Navigation

See my previous blog entry Dances with XML

Data set

I haven’t yet developed a class helper for traversing a TDataSet, but it strikes me as something that would be incredibly convenient. I invite the reader to have a go. Tell us about your efforts in the comment stream.

Third party support

Mike Lischke’s TVirtualTree, (a tree and list view component), is probably the worlds’s most popular third party component, and for good reason. TVirtualTree supports for-in enumeration.

I suspect that most readers of this blog are also readers of Nick Hodge’s “A man’s got to know his limitations“ blog. In this entry, Nick introduces the Delphi Spring Framework’s extention to IEnumerable. The entry is worth reading, including Nick’s answer to the originating question on StackOverflow.

Restrictions

  • One would expect that the member variable would be syntactically required to be a local variable. Surprisingly this is not the case. However, I advise against non-local member variables, as this is contrary to the semantics.
  • One would expect that member variables should not be capturable by closures. And this too is not the case. Again, don’t do it. The outcome would be just too messy.
  • Within the loop, the member variable cannot be assigned to.
  • The member variable can be referenced, both before and after the loop. Unlike a for-to loop variable, it’s value is indeed defined after loop exit, provided that the loop has passed at least one iteration.

14 thoughts on “For-in Enumeration

  1. Set Inversion: There is a whole thread about that here: http://objectmix.com/delphi/402305-invert-set-members.html

    Character Streams: Yup, anything where you have a loop like `while not Expression do begin …. increment; end;` will be more readable with a for-in construct.

    Reverse order: which things would you like to reverse?

    TBits: Francois Piette wrote a nice article on that this month: http://francois-piette.blogspot.nl/2013/01/original-method-to-iterate-bits-within.html
    He also wrote a nice piece on enumerators for containers: http://francois-piette.blogspot.nl/2012/12/writing-iterator-for-container.html

    TDataSet: I also wrote a TDataSetEnumerator, just not blogged about it yet.

    Predicates and XSLT: I remember a blog about something like that, with incredibly smart code, just forgot who wrote it (;

    • About the reverse order: Consider the situation where you traverse a list and the visitation action on the list member may potentially result in the member being removed from the list. In this situation, it is usually considered safer and simpler to traverse in reverse order. There are a dozen examples in the VCL where lists are traversed in reverse order precisely because of this reason.

      About TDataSetEnumerator. When you do write the blog entry, please insert a link in this comment feed. I would be most interested.

  2. There are couple of issues to using for-in for line parsing though, the main one being that it falls apart as soon as you want to be able to do thing like reporting an error with the line number at which the error occurred…
    You end up either having to maintain the line number separately (bad) or exposing the current line number in the iterator (good), which means using the iterator class/interface directly in a while loop (and not for-in).

    Also using for-in to process items in a collection is effectively limited to simple cases: since you have no control over the direction of the enumeration, you are unable to perform tasks like deletion, insertions or exchange/move items.

    • With the out-of-the-box enumerators, your points are bang-on.

      With custom enumerators, all of these issues go away with at a surprisingly low cost. The critical question will always be: “When is it worth it, to invest in a custom enumerator?”. It’s a hard one to answer.

      • And then one could argue that a simple index-based loop or a while loop are more readable than a custom enumerator :-)

        Cool language features are those that improve maintainability and code readability, and don’t promote over-engineering, at some point Enumerator become just an extra layer of complexity rather than being facilitators.

        Also there can be something philosophically wrong to exposing as enumerations constructs that aren’t designed for sequential access (like dictionaries or trees).

    • >There are couple of issues to using for-in for line parsing though, the main one
      >being that it falls apart as soon as you want to be able to do thing like reporting
      >an error with the line number at which the error occurred…
      >You end up either having to maintain the line number separately (bad) or
      >exposing the current line number in the iterator (good), which means using the
      >iterator class/interface directly in a while loop (and not for-in).

      That’s a problem with Delphi moreso than iteration; specifically it lacks tuples and tuple unpacking. In Python you’d just do:

      for count, line in enumerate(in_file):

      where enumerate is a function that produces an iterator that counts the iterations and returns a tuple with the count and the value for each iteration. You could probably do the same thing in Delphi except you’d need to return a record with the count and the value and it gets messy with generics if you wanted a general solution.

      >since you have no control over the direction of the enumeration, you are
      >unable to perform tasks like deletion, insertions or exchange/move items.

      I’m not quite sure what you mean by needing control over the direction of enumeration. Filtering and mapping at least shouldn’t depend on direction. You don’t want to be me rearranging items on the same container you’re iterating though of course; as Raymond Hettinger once said, “If you do that… then you’re living in sin and deserve whatever happens to you!” In most cases iterating through a copy or building a copy avoids any problems.

  3. Looks like some of your <T> parts were removed fro mthe listing, probably by tge code formatter, thinking they were tags.

  4. By the improvement of Delphi, I see that it’s more likely to become an alternative of C# language. It changed from for-to to for-in and in the latest XE4, StringBuilder is recommended to do string handling rather than original functions. You’ll see string.toint() and int.tostring() in the future. TArray will replace Array of T.
    Frankly, if it comes closer to C#, why do I have to learn Delphi anymore?
    Just some thoughts.

  5. Great list of potential features, but I have to disagree with “set inversion” on principle. What Delphi (and many of its cousins) implements has no relation to the mathematical concept of a set. The inverse of a set would contain an infinite number of values. The perverse limitation to a subset of 255 int values is solely caused by using a bitmap internally to represent the values, which is crippling sets to the point of near uselessness for a supposed improvement in speed. Somehow I doubt set speed was ever the bottleneck in any code ever written. :-(

    You’d love Python; it does all of the things you wish Delphi would do with iteration and more.

    >Enumeration types.

    Enums are new in Python 3.4 (!) but they’re quite capable (although not as cleanly initialized as Delphi). There’s no problem iterating over them.

    > Character Streams. Ever wanted to parse a text file character by character?

    I’m not sure you can directly iterate over a text file character by character, but you can certainly do this:

    with open(‘somefile.txt’) as in_file:
    for line in in_file:
    for char in line:

    > Reverse order

    for element in reversed(some_container):

    > TDataset

    import sqlite3
    conn = sqlite3.connect(‘test.db’)
    cursor = conn.execute(“SELECT id, name, address, salary from COMPANY”)
    for row in cursor:

    > Predicates.

    What you want is a list comprehension.

    For instance just this evening I wrote code that looked like

    good_days = [day for day in wednesdays if day not in special_wednesdays]

    which filtered the list wednesdays to only include elements not in the lost special_wednesdays.

    If you had a list with numbers and only wanted to iterate over the even-numbered values you could write:

    for element in [num for num in numberss if num % 2 == 0]:

    The problem as I see it is that iteration was another case of Delphi adding a feature and then never spending the time to go back throughout the VCL and refactor everywhere that feature would have been useful. As a result, we got only a taste of iteration, which when implemented fully (along with filter/map/reduce functionality) can be a programming paradigm in and of itself. As you noted, simply looping though a container is the least of its functionality.

    Here’s a few favorites I’ve picked up from Python:

    chain – takes a variable number of iterators. When the first iterator is empty it begins serving up the elements of the second iterator, etc. For example I had two sets, one called “good_words” and “bad_words”. At one point I wanted to filter a list so that it contained elements from either list. I could just do all_words = chain(good_words, bad_words) and then later use all_words as an iterator.
    There are variations of this throughout the libraries of Delphi – for instance, a function that can read a set of file names from the command line and then will open/iterate over all of them, and an awesome dictionary function that takes multiple dictionaries. It will look for the key in each subsequent dictionary. An awesome use for this is a dictionary that contains values from an INI file, a dictionary with OS defaults and a dictionary with program defaults. You can use this function and then when you look for settings values you’ll get the most specific one (INI files) if it exists down to the most general (program defaults).

Comments are closed.