Pliant language design

Read this document with care: it is written by Pliant's author and focuses on the superior design of Pliant over other languages. But since Pliant is so new and different, we must wait a few years before the worst Pliant design flaws become clear.

Another very general introduction on programming languages which focuses on goals very similar to Pliant may be found in The Will and the Word.

This page has been initialy written by Hubert Tonneau in 1998, then corrected by Jerry Fass in 1999.

Pliant is designed to meet these goals:

general purpose, very expressive language
highly efficient
scalable up to very large applications running on supercomputers
strong (easy to debug)

Everyone knows that goals do not grant success. We now examine the technical choices and new features introduced by Pliant, to better match these goals than other languages do.

By `general purpose', we mean that Pliant is expected to provide everything that you may need to build your custom general purpose application. This includes:

A rich set of native types: big integers, arrays, lists, dictionaries, indexes, etc
Persistent objects: data may be automatically saved to disk when an application ends and reloaded when it restarts
A built in mechanism to exchange and access complex remote objects with no or little extra code.
A basic set of graphical widgets
An integrated application framework: word processor, spreadsheet, database and drawing tool

By `very expressive', we mean that solving a given problem should use, usually, less instructions in Pliant than in other languages. This is achieved by a new design of the language itself. Pliant is made of two different parts:

Compiler: lean and minimal executable written in C, about 100K only
the basic set of modules containing the Pliant Default Extended Environment (PDEE), a general purpose full programming environment

On the other hand, a classical language is made of:

Compiler: large and tricky (if optimized) with a medium set of built in features
a set of very poor libraries that cannot add any exciting features

The main difference between Pliant and other languages is that in other languages the most advanced features are in the compiler and cannot be written in a library:

if you had a C compiler with no 'for' operator, you would not be able to define that operator in a library because 'for' is a first class citizen (called a reserved keyword) of the C language, and you can only write second class citizens in libraries.
another example of the poor extension abilities of orthodox languages is that it is impossible to upgrade a C compiler to C++ by including only a new library.
even more modest extensions are not possible: if you write an instruction such as y=2*x ; where x and y are integers, any smart C compiler will rewrite it as y=x+x ; , but if you define a C++ complex class, you can't tell the compiler that the optimization is still possible. So with orthodox languages you can define new data types but can't define the optimization rules that apply on them.

Alternately, Pliant's compiler lacks many advanced features, but it lets you write very sophisticated modules that truly extend the language. If Pliant lacks a feature you love in language X, then you probably can add it as a Pliant module. (for more details, see the big chapter about meta programming

However clever a language designer is he cannot include all the features that each user may need as programming is a very complex and changing activity. So, if nothing else, the Pliant design is clearly better since any user is more likely to be able to write and add any extra advanced feature they need or want.

Note: Pliant development is progressing. The only part currently finished (probably with many bugs and maybe some severe design flaws) is the minimal executable, which is the heart of the language since it defines how modules can extend the language, optimize the compiler, and avoid extension conflicts.

Computing's short history shows that most large programs need, at least in part, to be written in an efficient language. This is surprising since computer power grows rapidly and so many experts have said that language efficiency would soon be less vital. But short term history shows that as computer power rises, programs are asked to perform new tasks that need yet more power and so efficient languages are always needed.

The Pliant implementation is, and will always be, a dynamic compiler. This means that a Pliant program is generally distributed as source code which is compiled into the native binary code of the host computer partly at install time and partly at run time. This will greatly simplify the distribution of software for two reasons:

First, as the computing industry ages, the number of platforms rises rapidly. Now, providing software binaries for every platform is a nightmare. Further, in sophisticated software, a large set of flags must be set at precompiler level (or make level if you prefer) to build a well optimized binary. To provide the most optimized program the author must endlessly compile a different version for nearly each user (the best example of this may be the Linux kernel). Lastly, most large programs are full of bugs and are provided as a highly optimized, and so unsafe, version which makes feedback far less efficient.
The Pliant language is designed so that compiling applications is fairly straightforward and nearly invisible to the end users. This is not at all true for C and C++ programs. All the flags that must be set at precompile time in orthodox languages may be set at run time (which is also compile time) in Pliant's environment and so each end user gets a fully optimized binary. Lastly, users can choose at runtime whether an application is safe for their use and so to compile it very optimized or, if it's not, and so to compile it in a safer, more verbose, way to improve feedback to developers.

Second, the Pliant language allows applications to improve the code generator itself by providing functions that will dynamically optimize the code at any stage (parsing, compiling, optimizing, assembly) of the compiling process.
This is very important, since in mature software, many bugs come from buggy rewrites of source code in attempts to make executables more efficient. In any language you can optimize by rewriting source code. In Pliant, you can also optimize, to some extent, by writing extra optimizing functions that rewrite the code on the fly at compile time. This helps you get the optimum binary code without adding ugly patches in the source code. For big applications this is also clearly a winning method since an optimization can apply at many places and be removed at any time.

Conclusion: For small applications, Pliant can yield code as optimized as any C compiler. For large applications Pliant can yield much better optimized code, since humans cannot hand optimize large programs, due to lack of time.

Note: this was not true for early Pliant releases since the internal optimizer was not very good for integer computations and only applied trivial optimizations for floating point computations. To get an optimizer that can compete with C compilers you must wait for a compiler hacker to provide a state of the art Pliant optimizer as an external module. For more details, please check Pliant projects status chapter.

Scaling an application mainly depends on two key features:

Functionality scaling: code must be written in reasonably sized modules with clear interfaces between modules, so changing one part corrupts few (or no) others.
Power scaling: programs must be able to use efficiently the power of larger hardware, even supercomputers.

Functionality scaling has been the main goal of object oriented programming. So the question that arises is:
Is Pliant an object oriented language?
The answer is no.

From object oriented languages, Pliant keeps these features:

A type definition also contains the code that defines how to properly perform basic operations on that type: create, destroy, copy, ...
Virtual methods allow for applying a function on a generic type without caring about the concrete type.

But Pliant omits these features:

Inheritance which, from the Pliant author's point of view, leads to overly clever and sophisticated programs that break the true scalability of the application because:
- Code for a single function is often spread over too many modules.
- True computations are hidden by an excess of interface code.
Thus, in most cases, the code ultimately becomes hard to read, or even incomprehensible.

If you want inheritance, you must find or write a module defining the kind of inheritance you want. Yes, you can do this too in Pliant!

Finally, Pliant corrects a general design flaw of object-oriented programming:

The design of object-oriented languages suggests that the human brain thinks from abstract to concrete, but it usually works the other way.
So does Pliant. Pliant programs are written from concrete to abstract, not abstract to concrete.
Example: In C++, when you define a new class, you must provide the set of all virtual methods that may apply to the class, so at the very start, you must think abstractly and guess any uses you or anyone may have for the new class you define.
Alternately, in Pliant, you first define the data type, and later you can add virtual methods to it, to use it more abstractly.

In functionality scaling, Pliant goes one step further than other languages in isolating various modules of an application to avoid side effects. In orthodox languages, there is one environment and namespace; though recent languages, Python for example, allow using the <module>.<object> notation to avoid name clashes.
In Pliant, each module runs in it's own environment: parser + namespace + optimizer. This means various parts of a program can also use varied syntax or optimizers. In Pliant, any module sees exactly what it asks for; when a module includes another one, the new module is included only in the caller module, not in all modules.

For power scaling, Pliant is designed as a platform independent language because big applications are built and run over several years or even decades, so they often change host environment several times. All other platform independent languages (as far as I known) rely on a virtual platform which is more or less the common subset available at the time they are designed. While still platform independent, Pliant has far more evolution abilities:

Size of integers and addresses is defined as at least 32 bits, but can be anything higher including 64 bits or even arbitrary numbers such as 103 bits.
File system and network interface are designed to allow machines with different integer sizes to exchange data transparently.
Memory allocation functions include an additional parameter that can grant that (until a given memory area is full) all parts of an object be allocated in the same memory area as the root of an object. This is a major issue and enables making supercomputers with more processors.

The Pliant model for supercomputers is a set of groups of processors with a memory area owned by each group. A processor can access the memory area in it's group at full speed, but can also access the memory in other groups via a global bus, likely more slowly. The ability to have all parts of an object stored in the same memory area will enable, most of the time, managing the object by a processor of the group that owns the memory area the object is stored in, and so managing the object at full speed.

This clearly means that Pliant is the most scalable general purpose language ever designed.

This idea is commonly poorly understood, but it may be the most important one.
When a program runs over a bug, it often does not stop immediately. Debugging the program consists of working backward from the point where the program stopped, to the point where the bug occurred. The longer the program keeps running before halting, the harder it is to find the bug. C may be the weakest language ever built, and so C programs are the hardest to debug. The hardest part of a language design is to make it efficient AND strong.

The better and only way to strengthen C code is to use 'assert' instructions, everywhere they might catch a possible error. The more assertions you use, the stronger the code grows. The object oriented language Eiffel is famous for using assertions, in a formalized system called Design by Contract (DbC). Pliant is no better in this trait, yet.

All languages yet have failed to be efficient and strong at the same time. So, since we cannot yet reduce significantly the time to find a bug, we try to reduce the number of bugs.

Object oriented programming tries to help build safer programs by providing more sophisticated, well written data types (this means making the language more expressive as defined earlier) that enable programmers to build shorter and simpler programs, since any complex data type internals are handled by the code in a library that defines the data type, and not by the program using it.
The limit of object programming is that any attempt to write general purpose efficient complex data types leads to inefficient code, or to unusable data types, due to an excessive number of methods to chose between, and rules to follow when using them. The basic trait missing in object languages that Pliant introduces, is that the module (or library if you prefer) that defines the new complex data types, also contains the code that recognizes how it is used by the user program and rewrites that program on the fly to make the code more efficient, but still safe, since the responsibility for rules is with the module which defines the complex data type, and no longer with the user.
In other words, with Pliant you can build a complex data type that is simple to use yet efficient, which you cannot do with normal object programming languages.

Memory management is a major design issue when introducing a new language. Pliant innovates here too.

State of the art in memory management:

Low level memory management (malloc/free in C or C++) lowers the strength of a language since bad memory use is one of the most common and hard to discover causes of bugs; all of C history demonstrates this.
High level memory management (garbage collector) is unacceptable for applications that handle large data sets and still must be responsive. Most languages that try to make programming safer (Smalltalk, Eiffel) implemented garbage collectors that prevented them from being general languages.

For low memory management, Pliant uses a modified version of the Doug-Lea malloc/free algorithm. It seems to be a state of the art low level memory manager (very fast and space saving), and is the default memory manager of GNU C library, release 2.

Pliant also provides a medium level memory management scheme called reference counts. This is a trivial mechanism consisting of adding a counter before each object, to store the number of pointers that point to that object, and to automatically delete the object when the counter falls to zero. The advantage of such a mechanism is that it is efficient and works fine for most applications that use a simple memory layout. But the drawback is that it fails to free memory when several objects point to each other in a loop. It is the preferred Pliant memory handling mechanism since it is efficient and as easy to use as garbage collecting when memory layout is simple.

To make the best possible use of reference counts even for complex memory layouts, Pliant has two kinds of pointers. Normal ones (Address data type in Pliant, which is the same as void * in C) that ignore the reference count mechanism, and the kind that automatically updates the reference counter of the pointed object (Arrow data type in Pliant). The reference count mechanism works well if and only if the graph of all the arrows is a tree (there are no loops). But when a complex application builds loop-locked objects (the Pliant compiler will give a message when exiting), there are two possible actions:

In many cases, it is possible to change some arrows to pointers that are obvious ways back in the arrow graph, to turn the graph to a tree.
In other cases, one must return to low level memory handling by freeing objects explicitly, as in C or C++.

Conclusion: Pliant provides the highest level of functionality and security that does not lower performance too much for large scale applications. In computing's short history, there have been several very successful languages that are awful, but there has never been one successful language that is not top speed. This will likely remain true in the future, so Pliant's design choices are obvious in fact.

Pliant applications tend to be fatter than their C/C++ equivalents.
The main reasons are:

Pliant's current optimizer is not state of the art.
This should be solved in the next few months.
Pliant applications also contain the development environment.
To omit the development environment in the final application, we must generate a C or assembly version of the code, then compile it with a normal tool.
This feature is not provided yet, and may well never be provided, since it is not so easy to find what's required and what's not. Moreover, dynamic compiling is a great feature that will enable writing much better applications. Eliminating the development environment disables this feature.
Pliant applications always contain some extra information about each function.
This is needed for debugging (which can be turned off), and to allow freeing functions that are not used anymore.

Conclusion: If you plan to develop large Pliant applications, you will need more memory than if the same application is developed using orthodox tools.

In some ways, a human brain is a two stage computer:

The conscious part works like a single processor: it computes sequentially and reliably, one operation at a time.
The unconscious part works like a true neural network: it computes in a highly parallel fuzzy, and not very reliable way.

Why does it work so ?

We can start by asserting that the only solution for truly increasing computational power is massively parallel computations
But when using such computational mechanisms, granting reliability becomes a nightmare, since first it requires all the parts to work reliably which may be an issue, and moreover it requires some kind of synchronizations which is an even bigger issue
So the human brain design is a clear solution: one part is full speed raw computing with potential errors, and the other part is checking the result.

This design asserts that on most algorithms, checking the result costs far less computations that finding it, which is certainly true because nature got to that point after many many years.

Pliant's design does not contain the basis to ease writing programs for massively parallel computers (thousands of processors).

The main reason for excluding these computers from Pliant's scope is that the number of processors (a few thousand) and so the computing power is far higher than on orthodox supercomputers, but still very low compared to what is in brains (billions).
So, this kind of computer, to be significantly more efficient than mainframe computers needs some algorithms that don't waste too much power. This means that, for each algorithm, one must find a very clever way to spread computations over many weakly linked processors and still compute the exact result because allowing errors would waste too much power.
The obvious conclusion is that applications don't scale on these kind of computers because programs are too hard to write.

On big projects, the weak part is always programmer abilities, so the closer a language is to the way a human brain works, the more efficient it is.
Thus, when we are able to build far more massive parallel computers, the most efficient computational model will likely be the two stage mechanism of the human brain. This is also beyond Pliant's scope.

It may be impossible to design a programming language to work this way until we get computers to run it on, since no programmer is able to build a non-trivial program without testing, and it will take a long time to get two stage electronics since we may need to wait for 3D circuits.

All alternatives to current sequential programming (logical programming, constraint programming, lambda calculus, ...) have failed, and likely will all fail since they don't scale well.
They may lead to very nice programs for some simple problems, but on larger problems, programmers fail because the programming model is too far from his own brain.

Pliant's author views the main steps in programming language evolution as:

No abstraction: Machine code
Human readable notation of hardware instructions: Assembly
Abstract instruction sets: COBOL, FORTRAN, Basic
Functions with prototype and private variables: Pascal, C
Types with embedded low level management: Object languages: Smalltalk, C++, Eiffel
Library can check and (re)write the program using it: Meta programming language: Pliant
Two stage programming model: Brain model languages

So, Pliant is likely not the ultimate way to program, but it may be as long as computer parallelism is far less than in brains.
Up to the limits of the sequential programming model, Pliant enables inventing new ways to program by including new modules rather than moving to another language. Far more important, various parts of an application can be written in various programming styles, and still work perfectly together. This is what Pliant truly does and truly is designed for.

The ability to make several pieces of code written with different programming styles (so different languages) work together was previously assumed by the linker, but it was a nightmare to use practically since an application is not only some binary code, but also a set of prototypes, data types, and constants that had to be written in every language. Pliant enables writing all these things in any of them and using them in all others, or even to generate them as a result of computations.
Rather than being a true language, Pliant is a very general language framework that can be customized very deeply to match specific needs. In Pliant, most parts are not in the compiler executable, but rather in the modules, so the right question for programmers may not be:
Should I use Pliant or another language ?
but rather:
What Pliant modules should I use ?

Hope you have been interested by these fairly abstract considerations and are now ready to study how Pliant truly works.