Tcl/Tk and Expect

If you want to understand why TCL is in many cases more viable choice than, say, Perl, Python and several other languages you need to think about it as both scripting and macro language, not as a pure scripting language. Also its better suited to imbedding into larger programs (that was the design goal). And TCL was amazingly successful as a macro language for several important tools (for example in famous Expect), mail tools. Due to Tk toolkit TCL was and still is a popular tool for converting old command line application to GUI interface (Tkman), although "Webalization" (via HTML) is a better approach now.

At one point Sun was supporting the development of TCL on Solaris to the extent the it hired John K. Ousterhout. They produced a couple of new, better versions of TCL and the final quality was high enough for this product to be deployed on Solaris, but Sun became distracted by Java and John K. Ousterhout left Suu for its first TCL startup.

The killer applications for TCL proved to be Expect: a uniquely powerful sysadmin tool and TK a GUI development toolkit. for a nice introduction see Getting Started With Expect (HTML Format)

Expect is available for any flavor of Unix, including Solaris and Linus. Older free version of it is available for Windows too. ActiveState has a newer non-free version for Windows.

The killer applications for TCL proved to be Expect: a uniquely powerful sysadmin tool and TK -- GUI development toolkit

Expect is a great tool in a system administrators arsenal and can be used to easily automate tasks that require periodic user input. This can allow the administrator to make better use of their time than watching the application or utility to spot the next time it requires input. The idea of Expect was developed by Don Libes in 1987. The bulk of version 2 was designed and written between January and April, 1990. Minor evolution occurred after that until Tcl 6.0 was released. At that time (October, 1991) approximately half of Expect was rewritten for version 3. See the Expect history for more information. The HISTORY file is included with the Expect distribution. Around January 1993, an alpha version of Expect 4 was introduced. This included Tk support as well as a large number of enhancements. A few changes were made to the user interface itself, which is why the major version number was changed. A production version of Expect 4 was released in August 1993. In October 1993, an alpha version of Expect 5 was released to match Tcl 7.0. A large number of enhancements were made, including some changes to the user interface itself, which is why the major version number was changed (again).

The production version of Expect 5 was released in March '94. In the summer of 1999, substantial rewriting of Expect was done in order to support Tcl 8.2. (Expect was never ported to 8.1 as it contained fundamental deficiencies.) This included the creation of an exp-channel driver and object support in order to take advantage of the new regexp engine and UTF/Unicode. The Expect book became available in January '95. It describes Expect 5 as it is today, rather than how Expect 5 was when it was originally released. Thus, if you have not upgraded Expect since before getting the book, you should upgrade now.

TCL/Tk were originally developed at the University of California Berkeley, then at Sun by John Ousterhout. Here is a short biography sketch from his homepage

John K. Ousterhout is founder and CEO of Electric Cloud, Inc. He is also creator of the Tcl scripting language and is well known for his work in distributed operating systems, high-performance file systems, and user interfaces. Prior to founding Electric Cloud, Ousterhout was founder and CEO of Scriptics Corporation, Distinguished Engineer at Sun Microsystems, and Professor of Computer Science at U.C. Berkeley. He received a BS degree in Physics from Yale University and a PhD in Computer Science from Carnegie Mellon University. Ousterhout is a member of the National Academy of Engineering and a Fellow of the ACM. He has received numerous awards, including the ACM Software System Award, the ACM Grace Murray Hopper Award, the National Science Foundation Presidential Young Investigator Award, and the U.C. Berkeley Distinguished Teaching Award.

Here how he describes the events:

I got the idea for Tcl while on sabbatical leave at DEC's Western Research Laboratory in the fall of 1987. I started actually implementing it when I got back to Berkeley in the spring of 1988; by summer of that year it was in use in some internal applications of ours, but there was no Tk. The first external releases of Tcl were in 1989, I believe. I started implementing Tk in 1989, and the first release of Tk was in 1991.

TCL has a very simple structure. Each line starts with a command, such as dotask and a number of arguments. Each command is implemented as a C function. This function is responsible for handling all the arguments. It was really brilliant in its simplicity design -- a complete opposite to PL/1 style design of Perl. Actually in addition to TCL John Ousterhout developed a Tk toolkit that also became the most popular (and independent of TCL) GUi development toolkit for Unix. It continues the life of its own and actually became more popular than TCL itself. It is now ported to many languages. Now when we are talking about TCL we often mean TCL/Tk, but still we need to distinguish:

TCL is the Tool Command Language -- an embeddable scripting language

TK is the Tool Kit -- a graphical interface development tool.

Sun was the first and the only company that realized the potential TCL in Unix environment and John Ousterhout joined SunLabs as a Distinguished Engineer in 1993.

But TCL was not sexy enough and after Sun started its Java marketing campaign, TCL became an orphan in the corporation. Without major corporate sponsor TCL never was able to realize its full potential in Unix, where it could became a universal application-level macro language. That's really sad.

John Ousterhout left Sun and in 1998 founded Scriptic Corp. (see Tcl's inventor leaves Sun to form startup - SunWorld - February 1998). The life of a startup is a difficult one. In late 2000 it changed it name to Ajuba Solutions and then disappeared after the acquisition by Interwoven. Later he founded Electric Cloud, Inc.

TCL team was organized to drive further development of TCL/TK:

The Tcl Core Team (TCT) is a group of Tcl experts who collectively manage the development of the open source Tcl core, including the Tcl scripting language, the Tk toolkit, the Tcl Developer Exchange Web site, and other things yet to be determined. The Tcl Core Team was formed in August 2000 with an initial membership elected by the Tcl community. The TCT currently contains the following members:

Mo DeJong Andreas Kupries

Donal Fellows Karl Lehenbauer

Mark Harrison Michael McLennan

D. Richard Hipp Jan Nijtmans

Jeffrey Hobbs John Ousterhout

George A. Howlett Don Porter

Jim Ingham Brent Welch

As of this writing (early October, 2000) the TCT is just starting to get organized. It will probably take several weeks before we figure out the best way to work together and get shared resources such as the source repositories and Web site set up. More information will appear here as things develop.

From the computer science perspective Tcl looks like a more interesting development than Perl as it is the first open sourced embeddable language. IMHO Perl is a kind of mainframe (anti-Unix) style language that very much reminds me PL/1 ("all things for all people" kind of approach; there is always several ways to accomplish anything, etc. -- bonanza for consultants and book writers ;-), while Tcl is a more Unix-style language. It does well one thing well: it's a really decent command language for various tools. And that feature instantly attracted several talented authors. It's not only famous Expect was based on TCL (probably Expect can serve as an example of a tool that is more popular than the underling language ;-). AOLserver is based on Tcl! StoryServer is based on Tcl! Source Navigator is based on Tk! Tcl does awesome XML! Comanche is based on Tk! ...

It's very sad that Tcl never become prominent neither in Solaris nor in Linux environment. BTW that definitely attests Linux as a neo-conservative movement as Tcl has tremendous potential to lift Unix-style Os to a different level by integrating and streamlining command like and GUI as well as providing a common internal language to a million of obscure Unix utilities some of which outlived their usefulness. One needs to understand that despite their non-orthogonality and obscurity (I would like to see a person who really understands all the differences between find and grep in interpreting regular expressions ;-) the current set of Unix utilities represents quite an innovative, semantically rich set of commands for the operating system.

Actually there was one attempt in this direction -- here I would like to mention tksh -- a very interesting development that also failed to get a proper attention of the Unix/Linux community. Due to ability to use Tcl as tool macro language there are a lot of Tcl-based applications (see Top Applications), much more than in any other language (even Rexx) -- and that makes an important difference between TCL and other scripting languages. John Ousterhout views on scripting were best formulated in his famous paper Scripting: Higher Level Programming for the 21st Century. In the paper The salacious truth of the scripting paradigm shift to Tcl-Tk and Python Nicholas Petreley put it very well:

So expect scripting in general to take off soon. Of the scripting languages available, I predict Tcl/Tk and Python will eventually gain the popularity Perl now enjoys. Although it's somewhat of an apples and oranges comparison, both Tcl/Tk and Python are easier to learn than Perl. And like Perl, there is a version of Tcl/Tk and Python for practically every platform, including Windows 95 and Windows NT.

WWW has dramatically raised the visibility of Perl in recent years -- to a certain extent at the expense of Tcl and other Unix-based scripting languages. Due to the Web Perl has become one of major programming languages for Internet -- practically like VB dominates Windows arena, Perl dominates CGI scripting arena -- not because it is perfect for the purpose (string-handling capabilities of Perl are probably too much based on regular expressions -- compare for example with Lex-based approaches or with parse in REXX) -- it happened to be at the right place at the right time. But the main problem with Perl is that it cannot be easily used as an internal macro language. At the same time this is an extremely important problem area and here Tcl really shines. IMHO it will give Tcl a boost sooner or later. Although it's somewhat of an apples to oranges, I think most people agree that Tcl is easier to learn than Perl or Python. And in portability Tcl comes very close to Perl -- there is a version of Tcl/Tk for practically every major platform. Contrary to some Perl advocates statements, the speed of Tcl and Perl are very similar (of course, if both implementations use similar implementation techniques like conversion to p-code, JVM-like approaches, etc.). See Why Use Tcl for additional details.

Like Tcl itself TK is a simple and elegant toolkit that proved to be useful not only for TCL programmers but in other scripting languages including Python and Perl. The major benefit of Tk is its conciseness. The classic minimal Tk application is indeed tiny:

pack [button .mybutton -text Bye-bye -command exit]

Compare this with Java and you will understand that newer is not always better. This is a small but complete program in Tcl/Tk that puts a button on the screen and exits when the button is pushed, with proper behavior for iconifying, clicking, and other modern GUI conventions.

This economy of expression has two aspects. One is using TCL (or other scripting language) create a remarkably consider and at the same time powerful toolkit. Typical attempts to create a Java-based GUI application require a lot of learning of Java infrastructure, methods, classes, etc before a beginner even attempts to create a working application. That is not true for Tk -- you are productive almost instantly. That's why a number of programmers discover Tk by accident. There are plenty anecdotes of developers who choose Tk as a quick prototyping tool, then discover they're able to complete a full-blown application by refinement of the prototype.

Again it's important to stress that Tk is remarkably concise and that Ousterhout defined for Tk a very reasonable default values that greatly simplify programming. Tk programmers often write prototype implementations and fill in details as requirements emerge. Tk encourages this prototype-based, "scripting style" more than any other toolkit: scripting languages are comfortable with variable numbers of arguments, default arguments, and arguments of variable type.

As Cameron Laird and Kathryn Soraiz wrote in their excellent paper Tk sets the standard Tk can certainly hold its own against other GUI toolkits. It only takes an hour or so to begin producing meaningful applications, and there are plenty of books and tutorials to help.

However, the king of the hill now, at least temporarily, are WEB interfaces. It's probably that most hot fashion trend and the true toolkit of choice recently is not Tk, but DHTML in WEB browser environment. And as everybody who spend in software development two or more years knows (or at least suspect ;-) software market is driven by fashion. IMHO fashion can dominate this market for a long stretches of years. What is bad is that fashionable trends are not progressive in all respects. In some areas they can even be reactionary. For example Linux as one of the most powerful fashion trends of late 90th and early 200x does not help much TCL/TK. Probably because of Linus Torvalds limited knowledge of languages other than C, Linux was shaped into a pretty conservative Unix-style desktop that lucks flexibility and power of Amiga or Os/2 with REXX as both scripting and macro language. Both gnome and kde are essentially traditional style GUI that are not bound to a scripting language. That means that the bright future of Tk is not given and a the market can capriciously turns its back on Tk. In this case the toolkit's usage might become limited to a those bright developers and administrators who realize its indispensability for extremely rapid production of portable, intelligent, well-behaved GUI applications. But I hope this will not happen.