The “For in” style of enumeration in Delphi was introduced in Delphi 2005. The idea is to make enumeration over a collection of things easier and cleaner by removing the need for the iterator variable, removing the code for indexing and removing the code for boundary checking. If you think you know everything about enumeration, hold on to your hat. You may be in for a pleasant surprise.

We will start with the basics and then progress to custom enumerators. Seasoned “enumerophiles” may prefer to skip to the “Custom Enumeration over Objects” section. Let’s look at a simple example. Say we’ve got a list of strings, and we want to show them to the user. Here is how we would do it before and after this feature.

Enumeration by index

procedure EnumDemo1;
var
  SomeStrings: array of string;
  s: string;
  i: integer;
begin
  Populate( SomeStrings);
  for i := 0 to Length( SomeStrings) - 1 do
  begin
    s := SomeStrings[i];
    Dialogs.ShowMessage( s)
  end
end;

Enumeration by for-in

procedure EnumDemo2;
var
  SomeStrings: array of string;
  s: string;
begin
  Populate( SomeStrings);
  for s in SomeStrings do
    Dialogs.ShowMessage( s);
end;

Looking at the details of the above, you see that for-in enumeration is not exactly earth-shattering. We have dropped a variable declaration, the for-loop is a little cleaner and indexing is compiler-magiced away. But you’ve got the wrong idea if you think that the key selling point of for-in enumeration is these simple syntactic abbreviations. The real selling point is greater than that.

Stand back from the details of the code, and think about what we are doing at an abstract level. We are looping through a collection of strings and doing something with the members. From this perspective, the mechanism of indexing is a pesky irrelevant detail. What manner of collection it is (array, list, something-else) is a pesky irrelevant detail. Boundary checking? – another irrelevant artefact of implementation. What we have achieved in EnumDemo2() over EnumDemo1(), is the holy grail of language design – a closer alignment of semantics with syntax.

Standard Enumerables

So what can we enumerate over, out-of-the-box? The following VCL types are for-in enumerable? Member types are as you would expect.

Fundamental and compound types – Enumerables

Sets are a special case. Enumeration only visits members that are in the set, as opposed to the base range.

Object Enumerables

Interface pointer Enumerables

What’s missing?

So what is missing from Delphi out-of-the-box for-in enumeration support?
What's missing
Perhaps in a future version of Delphi we will get enumeration support for:

Predicated enumeration as it is now…

procedure EnumDemo3( Collection: TObjectList<TSomeClass>);
var
  Member: TSomeClass;
begin
  for Member in Collection do
    if SomeCondition( Member) then
      Member.SomeAction
end;

Fantasy enumeration supporting predicates

procedure FantasyEnum4( Collection: TObjectList<TSomeClass>);
var
  Member: TSomeClass;
begin
  for Member in Collection ! SomeCondition( Member) do
    Member.SomeAction
end;

Generics and Enumeration

Generic classes and for-in enumeration are a good match-up. Take a look at the syntactic synergy in EnumDemo3 above. If we wanted to do the same in Delphi 7, we would probably write something like EnumDemo5() below…

procedure EnumDemo5( Collection: TObjectList);
var
  i: integer;
  Member: TSomeClass;
begin
  for i := 0 to Collection.Count - 1 do
  begin
    Member := Collection[i] as TSomeClass;
    if SomeCondition( Member) then
      Member.SomeAction
  end
end;

Custom Enumeration over objects

So much for the basics. Now here is where it gets really interesting. With some customisation you can for-in enumerate over any object of any class.

You can enumerate over objects, records and interface pointers. The enumerators also can either be objects, records or interface pointers. And the collection members of the enumeration can by any type including fundamentals, compounds, objects, interface pointers, anonymous functions, even class references and type references.

When the compiler sees a code like

for Member in Collection do

.. and Collection is an object, it compiles by the following algorithm..

  1. Look for a public function of name GetEnumerator(). The function return type must be a class, record or interface pointer type. Protected and below methods do not count. Class functions are also discounted.
  2. If class, Examine the declaration of the class. There must be public non-class function MoveNext(), with signature…
    function MoveNext: boolean;
    
  3. Also in this class, there must be a public property named Current. The read accessor method for Current may be any thing, even strict private.
  4. The type for the current property must be equal to the declared type of the enumerated value (‘Member” in our above example), or, if the type is an object type, the current class type may be a descendant of declared member class. If the member type is an interface type, then the current property interface type may be an interface descendant of the declared member interface type. It is not required for interface types to have syntactically bound guids.
  5. The MoveNext() and Current are utilised as follows…

With declarations …

TMyEnumerator = object
  strict private
    function GetCurrent: TMemberClass;
  public
    property Current: TMemberClass read GetCurrent;
    function MoveNext: Boolean;
  end;

TMyCollection = object
  public
    function GetEnumerator: TMyEnumerator;
  end;

…this procedure …

procedure EnumDemo6a( Collection: TMyCollection);
var
  Member: TMemberClass;
begin
  for Member in Collection do
    SomeAction( Member)
end

… is compiled as if we wrote instead…

procedure EnumDemo6b( Collection: TMyCollection);
var
  Member: TMemberClass;
  Enumerator: TMyEnumerator;
begin
  Enumerator := Collection.GetEnumerator;
  try
    while Enumerator.Next do
    begin
      Member := Enumerator.Current;
      SomeAction( Member)
    end
  finally
    Enumerator.Free
  end
end

If the Enumerator is an interface pointer, then instead of being freed, it is released (reference count decremented). If the Enumerator is a record, then instead of being freed, it is record finalized and de-scoped.

If you are making custom enumerators, you may find it convenient to leverage these base classes and interface pointer types from the VCL…

  IEnumerator = interface(IInterface)
    function GetCurrent: TObject;
    function MoveNext: Boolean;
    procedure Reset;
    property Current: TObject read GetCurrent;
  end;

  IEnumerable = interface(IInterface)
    function GetEnumerator: IEnumerator;
  end;

  IEnumerator = interface(IEnumerator)
    function GetCurrent: T;
    property Current: T read GetCurrent;
  end;

  IEnumerable = interface(IEnumerable)
    function GetEnumerator: IEnumerator;
  end;

  TEnumerator = class abstract
  protected
    function DoGetCurrent: T; virtual; abstract;
    function DoMoveNext: Boolean; virtual; abstract;
  public
    property Current: T read DoGetCurrent;
    function MoveNext: Boolean;
  end;

  TEnumerable = class abstract
  protected
    function DoGetEnumerator: TEnumerator; virtual; abstract;
  public
    function GetEnumerator: TEnumerator;
  end;

Custom Enumeration over records and interface pointers.

You can enumerate over records and interface pointers too. It works the same way as classes, except that the restriction that the GetEnumerator method be public is not applicable.

Custom Enumeration with helpers

There is a natural synergy between class helpers and custom enumeration. See the next section for examples.

Some incredibly useful examples

Parse a character stream, line by line

With a class helper and custom enumeration, we can conveniently pass a text stream, line by line (or character by character just as easily), like so…

procedure ParseLines( TextStream: TStream);
var
  Line: string;
begin
  for Line in TestStream do
    ParseLine( Line)
end;

One possible implementation to achieve this (with a UTF-8 encoded stream) might be….

TTextStreamHelper = class helper for TStream
  public
    function GetEnumerator: TEnumerator;
  end;

function TTextStreamHelper.GetEnumerator: TEnumerator;
begin
  result := TLineReader.Create( self);
end;

TLineReader = class( TEnumerator)
  private
    coTextStream: TStream;
    ciPosition: Int64;
    csCurrent: string;

  protected
    function DoGetCurrent: string; override;
    function DoMoveNext: Boolean; override;

  public
    constructor Create( poTextStream: TStream);
  end;

constructor TLineReader.Create( poTextStream: TStream);
begin
  coTextStream := poTextStream;
  ciPosition   := coTextStream.Position;
  csCurrent    := ''
end;

function TLineReader.DoGetCurrent: string;
begin
  result := csCurrent
end;

function TLineReader.DoMoveNext: Boolean;
var
  Ch, Ch2: Ansichar;
  sWorking: UTF8String;
  L: integer;
begin
  sWorking := '';
  result   := False;
  while coTextStream.Read( Ch, 1) = 1 do
  begin
    result := True;
    if (Ch <> #13) and (Ch <> #10) then
    begin
      L := Length( sWorking) + 1;
      SetLength( sWorking, L);
      sWorking[L] := Ch;   // Adds a code-unit NOT a character!
    end
    else
    begin
      if (coTextStream.Read( Ch2, 1) = 1) and
           (((Ch = #13) and (Ch2 <> #10)) or
            ((Ch = #10) and (Ch2 <> #13))) then
        coTextStream.Seek( -1, soCurrent);
      break
    end;
  end;
  csCurrent := UTF8ToString( sWorking)
end;

XML Navigation

See my previous blog entry Dances with XML

Data set

I haven’t yet developed a class helper for traversing a TDataSet, but it strikes me as something that would be incredibly convenient. I invite the reader to have a go. Tell us about your efforts in the comment stream.

Third party support

Mike Lischke’s TVirtualTree, (a tree and list view component), is probably the worlds’s most popular third party component, and for good reason. TVirtualTree supports for-in enumeration.

I suspect that most readers of this blog are also readers of Nick Hodge’s “A man’s got to know his limitations“ blog. In this entry, Nick introduces the Delphi Spring Framework’s extention to IEnumerable. The entry is worth reading, including Nick’s answer to the originating question on StackOverflow.

Restrictions

12 Responses

  1. Set Inversion: There is a whole thread about that here: http://objectmix.com/delphi/402305-invert-set-members.html

    Character Streams: Yup, anything where you have a loop like `while not Expression do begin …. increment; end;` will be more readable with a for-in construct.

    Reverse order: which things would you like to reverse?

    TBits: Francois Piette wrote a nice article on that this month: http://francois-piette.blogspot.nl/2013/01/original-method-to-iterate-bits-within.html
    He also wrote a nice piece on enumerators for containers: http://francois-piette.blogspot.nl/2012/12/writing-iterator-for-container.html

    TDataSet: I also wrote a TDataSetEnumerator, just not blogged about it yet.

    Predicates and XSLT: I remember a blog about something like that, with incredibly smart code, just forgot who wrote it (;

    1. About the reverse order: Consider the situation where you traverse a list and the visitation action on the list member may potentially result in the member being removed from the list. In this situation, it is usually considered safer and simpler to traverse in reverse order. There are a dozen examples in the VCL where lists are traversed in reverse order precisely because of this reason.

      About TDataSetEnumerator. When you do write the blog entry, please insert a link in this comment feed. I would be most interested.

      1. I’m pretty sure that the TDataSetEnumerator code is in the example source code of my “Smarter Code with Databases and Data Aware Controls” for which the download links are: http://wiert.me/2009/04/24/spoken-coderage-iii-december-1-5-2008-on-delphi-database-and-xml-related-topics/

        I’m a tad occupied right now (our stove broke down, so need to bring foward of selecting and buying a new one), otherwise I’d have checked myself if the sources are there.

  2. There are couple of issues to using for-in for line parsing though, the main one being that it falls apart as soon as you want to be able to do thing like reporting an error with the line number at which the error occurred…
    You end up either having to maintain the line number separately (bad) or exposing the current line number in the iterator (good), which means using the iterator class/interface directly in a while loop (and not for-in).

    Also using for-in to process items in a collection is effectively limited to simple cases: since you have no control over the direction of the enumeration, you are unable to perform tasks like deletion, insertions or exchange/move items.

    1. With the out-of-the-box enumerators, your points are bang-on.

      With custom enumerators, all of these issues go away with at a surprisingly low cost. The critical question will always be: “When is it worth it, to invest in a custom enumerator?”. It’s a hard one to answer.

      1. And then one could argue that a simple index-based loop or a while loop are more readable than a custom enumerator 🙂

        Cool language features are those that improve maintainability and code readability, and don’t promote over-engineering, at some point Enumerator become just an extra layer of complexity rather than being facilitators.

        Also there can be something philosophically wrong to exposing as enumerations constructs that aren’t designed for sequential access (like dictionaries or trees).

  3. Looks like some of your <T> parts were removed fro mthe listing, probably by tge code formatter, thinking they were tags.

  4. By the improvement of Delphi, I see that it’s more likely to become an alternative of C# language. It changed from for-to to for-in and in the latest XE4, StringBuilder is recommended to do string handling rather than original functions. You’ll see string.toint() and int.tostring() in the future. TArray will replace Array of T.
    Frankly, if it comes closer to C#, why do I have to learn Delphi anymore?
    Just some thoughts.