You wouldn't run your test suite against a production database, so why would you develop in a REPL that touches it?

Why not?

Shapir-Whorf. Today, running something like that against production is unimaginable. Security, performance, ... But why not?

Bridge builders cannot experiment with live bridge for resource reasons. Yet all big construction efforts at some point or other built 1:1 replica. So why shouldn't we? It's much easier to do it with software than it is with bridges.

And yet - there's a work machinery company. Engineers have "REPL" for trucks. When they make a modification, they get high-end, supercomputer-based virtual truck they then put through paces in real (simulated) environments in real time. They went from drawing boards to instant feedback in 2 decades. They don't see a problem in testing on live data.

In software, we take a 10:1 scale of a wheel, then play with it on the desk and conclude that new truck is a good fit for 15 years of service in coal mine.

Why shouldn't I experiment with live data. Best way to ensure my work is correct is to use actual data set.

Common Lisp, Scheme, Smalltalk, Clojure, Ruby and Python[/quote]

First 3 are natural fit and have solved this problem long ago, there's little to say, it was so easy it was forgotten 30 years ago.

But majority of work today relies on external APIs which are not a good fit for most cases. For pure data mangling, where web apps fall into, it's somewhat easier. Doing desktop or native development is much more difficult for general case.

I assume you mean either scale to large programs or scale to large teams.[/quote]

Yes, as well as coping with project lifecycle. Stuff that was modelled upfront worked OK since it took limitations into consideration. But as requirements changed, they couldn't cope.

Simple examples - visual parts were eventually copied by hand and maintained by hand. So instead of doing find/replace/sed/grep/etc., people had to manually replicate connections on copy-pasted forms, since there was no natural mechanism to deal with it.

"Easy-for-nonprogrammers". It was. But as soon as real world requirements and constraints hit, it was much cheaper to just pay original team to implement the specs the old way. Even for trivial customization, none of it stood the trial of fire. It looks easy, just add this component, but in reality there were hundreds of edge cases.

Existing APIs. VA worked OK-ish for what was provided. But APIs (some as big as CORBA) weren't made for them. So for most part, they couldn't be used. Platforms - not languages matter. That is where the value lies.

In either case, I'd argue that textual languages don's scale particularly well either.[/quote]
Everything we have today is written in those.

Nothing of comparable scale is written in visual languages.

Regular 9-to-5 programmers these days work with 1MLOC codebases or larger without much effort. It's essentially a solved problem. There are inefficiencies, but work is getting done on schedule by several million people.

For reference, the project in question was probably some 400 people across multiple continents with some 300MB of code.
There are projects orders of magnitude bigger being in completely non-problematic production these days using conventional methods.
When I left the project, it probably involved around 1200 people working multiple releases, from test to production with codebase at 800MB. Core engineering team at the time was split into three parts with a total of 30 people, it simply became so much easier.

All of these things can be done equally well with visual languages.[/quote]

Can they?

flaws which could be fixed (like providing a richer set of data types).[/quote]

Could those fixes be made in visual language? If not, why isn't the platform itself written in these?

it quite literally was spaghetti code.[/quote]

Yes. This is how people program.

Before you go after improvements, first accept that as a fact.

Today's languages work with that, quite effectively.

Living in OSS and github world can give you a skewed perspective on what programming is. Vast majority of programs are broken to the degree that platform/compiler will allow it. Change one comma and it all falls apart.

Far from ideal or inspiring, but it's what keeps the world running. Good programmers write good code, bad programmers don't. Regardless of language.

What matters is that language works for both.

but I do feel it is something that could improve certain areas of programming for a lot of people and is worth exploring more.[/quote]

It is.

But also depends on the domain. Music production has traditionally been a good fit for alternative models. Much of more mundane work isn't.

It definitely helps if you can make your own little sandbox. Flash would be a mainstream example.

it quite literally was spaghetti code.

Yes. This is how people program.

Before you go after improvements, first accept that as a fact.

Today's languages work with that, quite effectively.

Living in OSS and github world can give you a skewed perspective on what programming is. Vast majority of programs are broken to the degree that platform/compiler will allow it. Change one comma and it all falls apart.
[/quote]

I find the argument that hiding shitty programming more easily makes it better for programmers odd. The last two paragraphs also seem to contradict each other.

I find the argument that hiding shitty programming more easily makes it better for programmers odd.[/quote]

Very few people are paid for programming. Majority is required to produce results.

If good programming practices helps that, after taking into consideration training, talent selection and similar, it will be used. For most part, it doesn't matter.

A lot of industries run on Excel spreadsheets. Languages today produce results.

OSS code, especially the handful of popular packages is exposed to many eyes. Majority of code in production however is written by one person once and that's it.

"Hard code" (C, Java, Ruby, etc.) is in minority. Majority of important code is written in arcane rule engines, Visio diagrams, proprietary config/declarative languages. It's just not advertised as code or programming, it's stuff people do to get the job done. Software Engineering doesn't apply here. Yet these are the people who can benefit most from programming.

But why not?

Why shouldn't I experiment with live data. Best way to ensure my work is correct is to use actual data set.

I completely agree. I don't think we're there yet, in the general case, but this is something we should be striving for. In a way, I feel like for in-memory data, functional programming and pure-functional data structures should help here: if your data is immutable, then you can work directly on the live data as long as you don't commit the changes back to the data store and have some kind of separation so that real users don't see your modified data. I think for databases a tiered approach "should" work - your test code hits the test db, if the item doesn't exist or is out of date, it hits the real db, which is read only to the test code.
I do some web development contract work for a recruitment site sometimes and the development server basically uses real data copied from the real database periodically (after anonymizing and other things to make sure test emails don't get sent out, etc). The test code then works on otherwise real data. Not quite what you're saying, but part way there.

Stuff that was modelled upfront worked OK since it took limitations into consideration. But as requirements changed, they couldn't cope.

People his this problem all the time, regardless of languages and tools used, visual or textual doesn't matter. The various design principles help, but overall there is no substitute for experience.

Can they?

In my experience, yes. For example, in Max/MSP, I can abstract and modularise code just like I can in Java or C++. I think it could be easier and I feel the reason that it isn't is that most of these visual languages are designed as domain specific sytems, so they often ignore general purpose programming aids which help modularity, extensiblity and reuse. Either that, or they are (or were?) not designed with textual language programming principles in mind. In any case, Max/MSP and Synthmaker got me part of the way there and I can see a clear path to improving the situation in both those languages. Even without these "improvements", you can write good code, its just that the improvements would make it easier to do.

Could those fixes be made in visual language? If not, why isn't the platform itself written in these?

Do you mean could the fixes be implemented in visual languages themselves? If yes, then the answer to both question is that its a bootstrapping problem: current visual languages are largely domain specific languages and lack all the general purpose features that I feel they need. In the case of Max/MSP, there is a lack of rich type system - you have ints, floats, symbols and lists of these. You cannot nest lists, lists of symbols emulate strings. Max/MSP is optimised for audio and video, but everything else is kinda inefficient. Clearly Max is not a good fit to implement itself.

But you could certainly design a language thats very like Max but supports, say, C's data types (C's primitives, structs, pointers/references for recursive structs) and a set of components to act on these and then I think you could rewrite the whole thing in itself. Or parts of it. If I were to build such a language, I'd use a haskell-like type system with algebraic data types and type classes. I'd have components for dispatch and pattern matching and I'd support functions (which could be simple expressions, or visual dataflow networks in themselves) as values which can be passed to higher-order components that act on sequences (where a sequence is a type class of which lists, vectors, maps, file systems, regexes, etc etc are members). I believe such a language would have similar pros and cons as a similar textual language - only thing changed is syntax (and possibly dataflow vs control flow considerations, just because dataflow is more obvious in visual syntax and control flow is more obvious in textual syntax).

Today's languages work with that, quite effectively.

I don't think they work with it as effectively as we would like. Codebases stagnate and become unmaintainable (the web contract work I mentioned is a testament for that - after passing through a few people before getting to me, the code is in serious need of refactoring, but the client does not want to pay for refactoring because.. it works).

I don't think this has anything to do with textual or visual syntax and I think visual languages can "solve" this just as much as textual languages. I also think that placing data flow at the forefront rather than control flow, and a functional style that isolates side-effects makes a huge difference here. I can see this when working on PrescienceAI code in Clojure and I feel that visual languages can do the same.
If they actually do or not, however, depends on the language designer/implementor.

But also depends on the domain. Music production has traditionally been a good fit for alternative models. Much of more mundane work isn't.

I would say Max and Synthmaker are signal processing DSL's. Flowstone is basically identical to Synthmaker as a language, but contains built in components for interacting with embedded systems for robotics - which is also signal processing really.

I think a reason that music has been a good fit for alternative models is because the users weren't (and still mostly aren't) programmers so traditional programming tools weren't going to hack it. This meant they were open to new ideas. Also, audio, and signal processing in general, is ideally suited to dataflow/stream programming since thats exactly whats going on and visual languages are very suited to this too because they can visually represent the "circuits" along which the signals are routed. This made it an obvious fit - but I think that general purpose programming, especially in a multicore world (because dataflow is similar to functional programing and both are better at multicore than imperative programming with mutable shared state), can also benefit from visual programming.

But again, let me say that I don't think we will ever (or should ever) replace textual languages - instead we should augment them with visual languages so that we can choose whichever suits ous needs best.

Majority of important code is written in arcane rule engines, Visio diagrams, proprietary config/declarative languages. It's just not advertised as code or programming, it's stuff people do to get the job done. Software Engineering doesn't apply here. Yet these are the people who can benefit most from programming.

Exactly and as visual programming is more approachable to non-programmers than textaul programming (the proof lies in how many non-programmers use the languages I listed before and stuff like spreadsheets), isn't this something that should be used by these people? Lower barrier to entry means more productive work done.

I'm not sure how anyone can look at history and think that visual programming languages will ever succeed for the general purpose. Can it be better for certain domains? Sure. Can it be better for tablets? Maybe.

Frankly, languages have been encoding data for millennia. If pictograms were good and unambiguous for information encoding we'd not be using text.

If pictograms were good and unambiguous for information encoding we'd not be using text.

There's some people in Asia that would like to draw you a few pictures. Those languages also demonstrate something interesting - they are ambiguous and open to interpretation, a currently very undesirable trait for machine languages, though that may change.

But you're right. Visual vs. textual is a matter of encoding.

Hence I mentioned problems with non-programmers. It's not the encoding they are not familiar with, it's the thought process behind it. Visual something doesn't simply solve this, unless it happens to match their thought process which also needs to overlap the general programming.


Visual stuff, for some definition of visual, requires a context. Mapping source 1:1 to some graph is definitely possible, but requires strong context. Electronics board drawings are a good example. There is a well-defined set of components, connected spatially.

Where visual programming (again, for some definition) falls flat is composition. With code, you start with assembly, it's then abstracted by C, then by Java and finally by JavaScript, for example - but all of these are read in precisely the same way. Left to right, top to bottom, statements, assignments. Each of these introduces a completely different context. Electronics schematics are similar to a degree. You might have gates at lower level, but then you abstract it into some chip and work with those gates. Instead of resistances you now work with logical connections.

But unlike conventional language representation, the level of abstractions is fixed, whereas one can easily use Ruby to write a DSL and then write a DSL in that and so on, each unlike the other, yet all based around identical alphabet and symbols. With schematics, you need to invent new concepts for each layer.