Revisited: No future for functional programming in 2008 – Scala, F#

FWIW, this was a busy year for Haskell.

* O’Reilly published “Real World Haskell”

* We reached 890 libraries and tools on http://hackage.haskell.org, up from 300 a year ago. The fastest growth in libraries seen so far, and on track to double the growth of 2007.

* The IRC channel reached a high of 566 users last week (compared with, say, 65 users in #scala)

* The Commercial Users of FP conference had the most Haskell speakers from industry so far, and the largest attendance. http://cufp.galois.com/2008/schedule.html

I’d consider this the strongest growth year to date for the Haskell community.

“And they should drop .NET support. It looks like nobody uses it.”

Personally, I want this statement to be true. My past experience with C# (admittedly, earlier versions) makes me like Java much better.

I wonder whether you came across meaningful (hopefully non-biased) data that confirm the statement.

Scala and Clojure are rivals mainly for buzz. They aren’t particularly rivals as programming languages.

Scala is making a beeline for the role of king of the JVM, a drop-in replacement for Java. As such it follows the JVM’s natural model a lot closer than Clojure, which really is a “scripting” language in the sense of parasitically orchestrating existing code. (Clojure is fast, but “scripting” is only slow by historical accident.)

Clojure is very powerful in consuming Java libraries but as far as I can tell it’s really not suited for creating new ones for general usage. This is in direct contrast with Scala, whose design is explicitly “the Scala runtime is nothing more than another dependency jarball”).

Clojure tries to float serenely above Java, Scala mixes into it. They can succeed without poaching each other’s users, because they’re tools for completely different tasks.

@Itay: “I wonder whether you came across meaningful (hopefully non-biased) data that confirm the statement.”

No, just reading blogs and some following mailing lists, and there are no vocal users for Scala on the CLR. I know I got a bloody nose already for that statement in the past, but that’s the impression I’ve got.

I also remember that poll on the Scala website about build tools – Java based ones.

@Julian: “Scala and Clojure are rivals mainly for buzz. They aren’t particularly rivals as programming languages.”

I think programming languages have a target market, otherwise they won’t be used. From my impression Clojure has two (judging from the blogosphere), a Lisp “implementation” and a Java successor. Scala as a main target has Java developers (others seem to prefer Haskell, OCaml or F#).

So they both are competing mainly (my impression again) for Java developers. And two functional (at least partly) languages which tout concurrency as one of their strenghs are rivals in my opinion.

Comparing languages by how many are on an IRC channel? ##java itself only has a couple hundred folks, and it’s the most popular programming language of all.

And about libraries, remember scala can use any java library.

I’m curious who’s positioning Clojure as a Java successor. It doesn’t look that way to me.

Saying they’re both after Java developers is too broad-brush and almost a truism.Java is the only serious player on the JVM right now. OF COURSE any new language wants to poach Java programmers. But Java programmers doing what?

It seems to me the Java world divides roughly into consumers and implementers of libraries in the form of prepackaged jarballs. Clojure only really targets the former (and it has an edge over Scala in simple directness), Scala targets both. Without doing that, how can you be a “Java successor”?

@Julian: Oh, I thought it was handled that way.
“I’m curious who’s positioning Clojure as a Java successor. It doesn’t look that way to me.” You’re shortening my argument quite a bit which was “From my impression Clojure has two (judging from the blogosphere), a Lisp “implementation” and a Java successor”.

“It seems to me the Java world divides roughly into consumers and implementers of libraries in the form of prepackaged jarballs.”

I’m using Java since 1.0 and I’ve never seen the Java world divide significantly into these two parts, what gave you that impression? Jar producers what you call them are primarily irrelevant. Most Java developers are consumers, this is what I call Java developers. And they write application code. So both Clojure and Scala target the same main demographic, people writing applications (currently in Java).

I wasn’t so much shortening it as not addressing the part in which we already agree. Yes Clojure is obviously a Lisp. But who’s putting it forward as a better Java?

Jar producers are so far from irrelevant that I’m shocked you can’t see it. Consumers are in the majority, but producers and their libraries are 99% of the reason anyone bothers using Java, and they have significantly different needs from consumers.

See also:
http://www.lispcast.com/drupal/node/55
http://memeagora.blogspot.com/2008/10/library-oriented-programming.html

“Jar producers are so far from irrelevant that I’m shocked you can’t see it.”

Jar producers are irrelevant as they are not a target of a Java successor. Ruby worked fine with C before people moved to Ruby for implementing libraries. Obviously people use Java because of the libraries, but not because they can create libraries for wider consumption. This is a fine difference. People will go on writing Java code consumable by a Java successor, independently of what the successor will be.

People writing applications, e.g. a Swing or web application, and who currently use Java can easily use Clojure as their implementation language. They do not care if their code is consumable by others.

Concerning your links – which I don’t agree with: I’m mainly on Rickards side, most languages should not be called OO but class-oriented.

@Julian – This presumption that Scala is somehow more suitable for library implementors is founded upon what – its object orientation? As Don Stewart’s earlier comment re: Haskell’s many libraries demonstrates, being object-oriented is not necessary.

Clojure is not a ’scripting language’ any more than is Common Lisp. Its transparent access to Java is layered upon a set of facilities that should allow it to excel everywhere Lisp has traditionally – AI, scheduling, bioinformatics, simulation, knowledge management, etc, anywhere you face a hard problem and don’t know the answer in advance. Transparent access to Java just keeps the trivial parts of those problems trivial.

Being Java’s successor is also an intriguing concept – successor at what? Java is likely to remain a key piece of implementation infrastructure, and its simplified object model will remain a common bridge point for language and library interop. To the extent Scala diverges from Java’s object model, its suitability for that role will diminish, i.e. it will be no different than any other language whose libraries can only be consumed by itself. From the Scala FAQ – “Using a Scala class from Java can get tricky”.

Clojure can produce all-binary Java-callable ‘jarballs’. All of its data structures implement standard Java interfaces. It does not have an object model other than Java’s. You can extend classes and implement interfaces in Clojure. Nothing is likely to succeed Java as a library substrate on the JVM, but consuming Clojure from Java (and therefor other JVM langs) is easy.

Application development dominates library development, and all JVM languages compete there. It’s true – Scala and Groovy will probably seem most familiar to Java devs. Clojure is for those looking to change the way they write software. All three can succeed without replacing Java, but if Java isn’t being replaced, consumability via Java’s object model will be a critical factor in suitability for library or mixed-language development.

Add to that the fact that Scala isn’t a purely functional programming language (i.e. referential transparency isn’t enforced).

Let’s all stop propagating the myth that there’s a divide between functional and object-oriented programming. There’s only implicitly vs explicitly knowing which functions your program is composed of. Programming is a single discipline.

@Rich Hickey – you wrote the thing, you know it. I defer to you. Please read this as curiosity and confusion, not argument. “I am only an egg.”

When I say Scala is better suited for library dev, I’m not thinking of OO (I’m no more a fan of that than you are), I’m thinking that Scala supersets the Java object model and extends in almost the same direction, while Clojure subsets it and extends in an orthogonal direction. (Again, this is not a preference ordering. I like Clojure’s fastidiously minimal contact with the Java object model.) That means coding a Scala library for Java interop is matter of adhering to Java conventions in your public classes, while coding a Clojure library results in a rather flat and spartan Java interface.

Examples: emitting a class is in Clojure core, emitting an interface and an enum is only in clojure contrib, and I’m not sure Clojure can consume or emit annotations at all. Scala can. (Also, Scala programs tend to be written in terms of objects – the impedance mismatch is lower, and the public interface makes internal sense in Scala.)

I’m also confused by your saying Clojure can produce all-binary jars. The only info I’ve found online is about producing jars full of binary classfiles that act as shims to included .clj files.

As for scripting, I think I phrased my comment badly – Clojure as a whole language obviously isn’t only for scripting. The Java interop layer, however, does seem designed for scripting, with a focus on consumption rather than production.

You think Java will always be the JVM’s lingua franca. I’m much less certain. Java the language seems to be hitting something of a brick wall in 1.7, caused by the fundamentally non-extensible syntax just having one too many features crudely nailed on. If Java stalls, some other language will get to define the next iteration of the JVM bytecode and its object model, and I’d not be surprised if that language was Scala.

@Julian – Clojure’s Java generation is still a work in progress. Ahead-of-time compilation is new, and creates for each Clojure namespace a Java class. You can declare the superclass and any interfaces that class implements, as well as additional methods, constructor signature etc. Defining methods is as simple as defining functions in the namespace. So, it is very idiomatic Clojure, and yields a perfectly normal Java class. If your methods return Clojure’s native data structures, they look like java.util.Map/List/Collection/Iterable/Callable etc to Java consumers, and for anything Clojure specific there is a corresponding normal Java interface. You gain a lot as well, such as the ability to redefine/fix methods on the fly in a running program, and the ability to create objects with transactional state. Making that experience Java-like on the Clojure side is not an objective.

Yes, some things are still in contrib, or not yet done, but the objective is to allow full access to Java’s model, without super- or sub-setting, on both the consumption and generation side, and I’m fully confident that will be reached.

There will always be somewhat of an impedance mismatch between functional and OO, but I think there is an impedance mismatch on the way for OO programming in the face of multi-core, and to the degree people are doing mutable objects in Scala, it’s coming their way as well.

I don’t know Scala very well, but I think the question implied by the quote from the FAQ remains: if you use the additional capabilities of the Scala object model, idiomatically, what does that look like from Java? And if you eschew them, where’s the benefit of Scala? Can you have your cake and eat it too? If you program in Scala in a functional style much as you might in Haskell, is that not as much of an impedance mismatch from the Java perspective as is Clojure?

As far as lingua franca, I think the importance of the stability and simplicity of the JVM bytecode and object model in the Java ecosystem’s success cannot be underestimated. Java-the-language’s ’stalling’ is a good, not bad, thing, as it becomes the C of the next generation. But much as C++ had to deal with C linkage, so too will Scala have to subsist on the JVM as-is for the indefinite future.

@Rich – It’s unfinished? Then I was mistaken. My bad.

Re: Scala libraries, there’s a 1:1 mapping between a subset of Scala and Java, so if you stick to that subset then the resulting compiled API will be trivially identical in Scala and Java. Of course, you only have to use this subset on the “outside”. Internally, code can use FP, mixins, tail calling, etc.

Re: the JVM, my point is that Java stalling doesn’t mean the JVM will stall. People will keep adding new stuff to the open source JVM, it just won’t be Java stuff. I seriously doubt the JVM will stabilize.

>Following one debate about Guidos post on the Scala debate list, >there are many counter points raised. Sadly the bigger number of >commentators want the language to stay the way it is. It seems only >David R. MacIver wants to change the language to make it easier. I >strongly agree with him and Guido that there are parts which are >inconsistent, which need to be changed to get Scala into the >mainstream.

hi stephan,

not sure if the statement above is true, please see:

http://www.nabble.com/Summary-and-response%3A-what-things-should-be-dropped-from-Scala–%28LONG%29-to20647368.html

@pk11: Perhaps I’m getting Martins summary wrong again, but it mainly looks like “Won’t change”. Which is not very surprising because he designed the language the way it is.

In every thread where the topic comes up, Martins answers seems to be “[...] , but I think the claims about complicated syntax have been somewhat exaggerated.”

Not very encouraging.

And from my feeling the syntax IS complicated in parts and confusing. I’ve easily used more than 10 languages in real projects and can easily adapt, but Scala is often hard to get compiled for beginners.
Some of the points raised in the discussion are really not helping, like the optional (), foo_ and the usage of () in for.

>Scala is often hard to get compiled for beginners.

could you please back this up?

we have a team of three and we are following a very java-like coding style (so we are not using foo_ for anonymous functions or allowing multiple syntax versions for “for” etc.). so far I have not had any issues with reading other devs’ code. In fact the only issues i had so far were mostly related to working with java libs (ie I did not know about the collection wrapper etc.)

as for martin’s point:
I think he is saying that the language grammar is not more complex than c# or java as per the spec (which does not mean that the syntax can not be confusing).

but he raises a good point though, scala is not python, it’s more similar to ruby — in that sense that there are a lot of ways to achieve the same thing. so developer conventions are really important (same true for ruby)

(btw i use ruby, python, java and scala at the same time (not for the same project))

@pk11: I also had no problems with Java like code. But going into deeper types, use the automatic setter/getter for attributes, leaving out braces etc. leads to problems.

@Rich: I’m afraid to say anything, because talking to you I’m quite out of my waters :-)

“Haskell’s many libraries demonstrates, being object-oriented is not necessary.”

Well, since Haskell is a niche language, you are actually making the point that being object-oriented is necessary, and that any language that doesn’t enable basic OO features is doomed to remain marginal.

As for Clojure: I have a certain fondness for Lisp and I applaud the effort to provide Lisp on the JVM, but realistically, Lisp has failed to gain any significant traction for decades and I’m not seeing any reason to think that this is going to change any time soon.

–
Cedric

“As for Clojure: I have a certain fondness for Lisp and I applaud the effort to provide Lisp on the JVM, but realistically, Lisp has failed to gain any significant traction for decades and I’m not seeing any reason to think that this is going to change any time soon.”

Dito. For almost 15 years I carry one book from employer to employer (and which I own much longer):

“(List 1.5 Primer (By (Clark Weissmann)))”, Dickenson Publishing, 1968 edition.

I will look into Clojure as I’m interested, but I’m quite sure I’ll never use it to earn money (Never say Never).

>@pk11: I also had no problems with Java like code. But going into deeper types, use the automatic setter/getter for attributes, leaving out braces etc. leads to problems.

http://www.nabble.com/Scala—Debate-f30218.html

looks like from the three things you mentioned two will be fixed in 2.8

as for scala’s killer app:

maybe that’s actually sun?
“Sun has no immediate plans to submit the JSR for Java SE 7″ http://tinyurl.com/55t4qm

seriously, i think the fact that there won’t be a new version of java (let alone one with language changes) within a few years will definitely increase the interest in scala.
Whether the current stigma ’scala is too complex’ will be the common agreement that remains to be seen.

the positive effects in this year were:
- books
- IDE support (eclipse, netbeans and intellij, personally i think intellij’s is the best)
- twitter
- some buzz in the blogsphere
- Java interoperability is probably the best among the JVM languages and it was getting better

the negative ones:
-all of the well-known googlers disliked scala
(crazybob, cedric, guido and yegge) – that’s certainly a bad barometer
- lack of support from well-known IT shops

i, for one, am a happy scala user

(we built a few webapps with scala while reusing many existing java libs & relying on very strict anti-magic, java like syntax conventions)

Good points:

“-IDE support (eclipse, netbeans and intellij [...])

- all of the well-known googlers disliked scala
(crazybob, cedric, guido and yegge) – that’s certainly a bad barometer”

But

“- Java interoperability is probably the best among the JVM languages and it was getting better”

I’d say Groovys integration is (much) better. For mainly three reasons:

The “as” syntax makes it easy to implement Java interfaces, especially one-method interfaces.

Scala pollutes the method space of classes with lots of ugly methods. This makes using this classes horrible for Java developers (my opinion and experience)

Traits generate Java abstract classes and interface extension chains which are ugly and not clear to the Java side of things.

>Scala pollutes the method space of classes with lots of ugly methods. This makes using this classes horrible for Java developers (my opinion and experience)

>Traits generate Java abstract classes and interface extension chains which are ugly and not clear to the Java side of things.

we always start with creating an interface like trait (which will compile to normal POJI and provide an easy entry point for java->scala calls)

http://www.ibm.com/developerworks/java/library/j-scala04298.html

what i mean by the best compatibility (ok, one of the best): being able to write scala classes where one can import java libs and call java methods which take scala classes as arguments). on top of that: annotations, circular references and generics are working back and forth (ie java->scala and scala->java).

also the fact that advanced areas can be confusing in a language is a point which is valid for probably every language (including java, generics anyone?)

I know both languages: Scala since 2.0 and Clojure since its inception. It may seem strange just looking at the syntax but I think Clojure has a better integration with Java than Scala. Hickey designed Clojure to be a JVM language and it shows. IMHO, I think Scala should drop CLR support and concentrate all its effort on the JVM like Clojure does.

This year saw unprecedented innovation from Microsoft in functional programming, as seven years of work on the functional foundations of .NET culminated in serious uptake of .NET 3.5, C# 3.0, LINQ and the TPL this year as well as the first signs of F# coming to life in mainstream programming.

Amazon alone list 59 new books on C# 3.0, all of which are likely to cover the first-class lexical closures that make C# 3.0 as much a functional language as Scala.

.NET 3.5 is the first release to make extensive use of first-class functions throughout the core libraries (e.g. System.Func).

LINQ brings functional programming to the world of database programming.

The Task Parallel Library (CTP, July 2008) uses functional programming to provide easy-to-use constructs for data and task parallelism, like the Parallel.For higher-order function.

F# is undoubtedly Microsoft’s boldest step in functional programming but it went way beyond “promises”, as you said. F# burst onto the scene following its September 2008 CTP and immediately gained as much traction as established functional languages like Haskell and even OCaml (see Google Trends or the job market).

F# has already seen the publication of the books Foundations of F#, Expert F# and F# for Scientists in 2008 as well as the announcements of F# in a Nutshell and Real World Functional Programming with examples in F#.

We decided to diversify our OCaml-only company in 2007. Our choice was between Scala, Haskell and F#. With the benefit of hindsight, we made absolutely the right choice with F#. I am glad to say that gambling entirely on F# is (literally) paying dividends for us now. I do not believe Scala would have been as lucrative and adopting Haskell would certainly have been a serious mistake.

thinking that something like scala and the playframework could easily take the world by storm here soon .. other than that — yeh scala does need a few other killer apps — they will come though — scala really is the java that should have been imo

I think there is a future in functional programming. Obviously cell phones are hot Really not the target market for functional programmers. Large scale server side programs this is where functional programs will live.
Of course that seems like an optimistic assessment. I program in main stream programs like C# , VB, and Python. (Lots of SQL)

Who is using functional programming?

Amazon uses Erlang to implement SimplDB, Yahoo,Facebook T-Mobile , Motorola and of course Ericsson all use Erlang.
If you have a single processor don’t use functional Programming if you have 32 processors why wouldn’t use all the cores?

It is a matter of scale!