What evidence exists about brace- and indentation-based syntax for novice programmers?

A Stack Exchange question (Studies on learnability of braces vs. indentation for code blocks for beginners?) asked for information about what evidence existed about the impact on new programmers of different ways of delimiting block structure in code. This is the literature survey I composed in response, which finds there really isn't very much around at all.

I don't believe that there is any systematic pedagogical study on braces versus indentation specifically. Studies of actual learning are complex, time-consuming, and expensive, and I don't think any has been done (or likely would be) on that difference in isolation. The totality of human-factors studies comparing language design decisions likely numbers in the dozens (depending on your counting criteria, somewhere between 35 and 137 existed in 2012, per Antti-Juhani Kaijanaho's PhD thesis).

However, there has been work on program comprehension and reproduction under different indentation and delimiting modalities, and on learning performance in different languages and editing environments. These are more mixed on indentation than you might expect: it's widely accepted that correctly indenting to match structure is helpful, but in fact empirical studies often show limited or no impact in practice. My high-level summary of the state of the art for block structure for novice programmers is that

1. indentation is somewhat valuable, but they don't use it unless enforced;
2. keyword-based systems are more effective than symbol-based systems; and
3. redundancy enables enhanced errors and thus improves effectiveness.

None of these are strongly results. It is not clear that the benefits of keywords survive to expert programmers or with editor support for delimiting symbols, while indentation may well be more valuable to experts than novices. There has also been a lot of work with Lisps for novices, which are all delimiter and no indentation. It is likely that the ideal system for learners at this stage will both use delimiters and enforce that indentation is consistent with those delimiters (and this is the approach that recent novice-targeting languages like Pyret and Grace have taken, albeit without specific evidentiary basis).

I will point to some specific studies below, although they are mostly not as strongly-grounded as I would like. None of the indentation-focused ones are comparing to brace-based languages or to semantically-significant indentation, though there are some suggestive pointers. I think the most relevant result is from the Quorum studies, suggesting that braces in particular perform poorly.

• The effect of indentation on program comprehension, Thomas E. Kesler, Randy B. Uram, Ferial Magareh-Abed, Ann Fritzche, Carl Amport, H.E. Dunsmore, 1982.
• Program Indentation and Comprehensibility, Richard J. Miara, Joyce A. Musselman, Juan A. Navarro, Ben Shneiderman, 1983.

These studies looked at Pascal (keyword blocks, but all begin-end) programs with different indentation levels. The findings were that moderate indentation was beneficial for novice comprehension over both no indentation and greater levels of indentation. These are not brace-based programs and participants were already learning or programming in Pascal; the effect size is also surprisingly large for the difference between four- and six-space indents. There were a number of indentation studies around this time, with generally similar results, but not with novices in pure indentation-structured languages.

• Indentation: Simply a Matter of Style or Support for Program Comprehension?, Jennifer Bauer, Janet Siegmund, Norman Peitek, Johannes C. Hofmeister, Sven Appel, 2019.

This was a partial replication of the previous study, using curly-brace Java instead of Pascal, with students, and eight instead of six spaces. They found no impact from indentation at all on any of their measures: even zero indentation gave similar performance.

• Conditional statements and program coding: an experimental evaluation, Iris Vessey, Ron Weber, 1984.

This compares an IF-GOTO-based structure with (non-semantic) indentation with a variety of block-structured systems, using genuine novices in a controlled trial. The closest to braces is the begin-end variant of the language, and the programs in the GOTO version looked like an off-side-rule language, but the line labels were semantically significant; the other variants used IF-OTHERWISE and IF x-NOT x. The methodology had participants review a short tutorial on the language condition in use, and then write programs to implement a specific condition table. Indentation helped for all but one of their variants, but the version with distinguished start and end markers for each conditional performed worse. Participants had a lot of difficulty getting the indentation right in the GOTO case, but when they did their programs were no worse than others.

• Impact of maintainability defects on code inspections, Özlem Albayrak, David Davenport, ESEM 2010.

This study measured defect detection rates from inspection of Java code with mid-level students, and found that bad indentation impaired detection more than bad naming. This was inconsistent indentation, rather than none, which the other studies haven't touched. These programs would likely be static errors in an indentation-based language, but might misbehave instead.

• Indentation in Source Code: A Randomized Control Trial on the Readability of Control Flows in Java Code with Large Effects, Johannes Morzeck, Stefan Hanenberg, Ole Werger, and Volker Gruhn, in ICSOFT 2023.
• Indentation and reading time: a randomized control trial on the differences between generated indented and non-indented if-statements, Stefan Hanenberg, Johannes Morzeck, and Volker Gruhn in Empirical Software Engineering 29(5), 2024.

These studies look at the time taken to comprehend braced Java code (not edit it) with different indentation structures. The result is that indentation reduces the time taken for the reader to react to a prompt about the control flow of a piece of code substantially, incorrect responses were provided more often in unindented code, and higher TLX difficulty was associated with lack of indentation; the suggestion then is that indentation makes the code more understandable. All the programs were artificial nested-if-statement structures operating on integers, and all had braces (a similar set of authors also examined indentation of JSON instead of control flow, with comparable results).

There was an interesting inter-group analysis result: professional programmers were the most accurate group by a long way in the indented condition, and the least accurate group in the unindented condition, while undergraduate students were the fastest in both cases. This may be another point for reduced effect of indentation for learners in particular, but it's not clear what to take away from it, and it did not examine any effects on learning. There's also quite a high level of variation between individuals.

I think this last one is the highest-quality study that isolated indentation as a factor, and none of them approach your question directly. We can take some hints that indentation is probably useful and can be sufficient, but that delimited block structure can also be sufficient on its own, and learners seem if anything to get less benefit from indentation than others.

• An Empirical Investigation into Programming Language Syntax, Andreas Stefik and Susanna Siebert, 2013.

This is part of the Quorum project, a focal point for modern empirical language design for novices (and the source of the famous "randomly-generated language more usable than Perl" claim). Most of this relates to keyword choices and indentation was not studied specifically, but of relevance here was a finding that block constructions that did not use braces had better performance metrics than those that did, although that subgroup included both the indentation-only Python and the keyword-end Quorum and Ruby.

The best performers, very close together, were Ruby and then Python, with Perl and Java not statistically significantly different from Randomo, which was Quorum with braces and random ASCII keywords. Some issues writing the end keyword correctly were reported, but Quorum still uses that keyword today, so it wasn't found to be that bad of an issue.

In all cases, "correct" indentation was presented in the learning material, but I don't believe it was required in what was produced except for Python. Indentation is encouraged but not required in Quorum, I think to reduce unnecessary errors during editing. Follow-up work on Quorum has reinforced these results for it.

There are various other studies on teaching language pairs that I'm sure you've seen, but they don't generally get to this level of granularity and there are significant confounds for teasing out anything like this.

To conclude here, I don't believe such a study exists. I am confident that there isn't one prior to 2014, and I am not aware of and couldn't find any since either. I am sceptical that it would be worthwhile to run one for novices, because the value of that one factor is unlikely to be worth the substantial cost of running a genuine learning study, though a comprehension study that gets deeper than Hanenberg, Morzeck, and Gruhn is plausible.

The general direction of the literature is that braces are not highly effective for novices, although lexically substituting an explicit end-of-block delimiter word for } seems to do better, and omitting any opening delimiter has some support too. Semantic indentation isn't supported one way or the other by this, but there is some evidence that matching end delimiters presents a difficulty, which is a matter that doesn't arise for semantic indentation. I don't think there is sufficient evidence to draw a conclusion one way or another.