Most Web users have needs beyond what commercial web applications support. Automating common tasks, storing arbitrary data and performing calculations on them, tracking, are only a few categories of use cases.

Despite the Web originally being designed so that anyone could contribute, not just passively consume [1], these days the Web Platform¹ has grown tremendously in both complexity and power. It now takes years of training for someone to be in a position to create bespoke web applications, and even professional programmers with years of experience often lament the complexity of the modern web stack.

Even though trained programmers have the ability to create web applications for their own needs, the task is still so laborious, they rarely embark on it.

The goals of my research are three-fold.

1. The primary goal is to make web application development accessible to a wider audience and bring it within reach of everyone.

2. A secondary goal is to make it faster for any audience. If trained programmers can create prototype applications really fast, everyone wins.

3. Lastly, a tertiary goal is to contribute towards increasing the amount of machine-readable data on the Web and towards decentralization, not by attempting to convince users that these are worthy goals, but by creating technologies that incorporate them as a natural part of the interaction that does not require additional effort or even interest from the end-user.

1.1Why Is Web Publishing Still Hard?

Few questions fill web practitioners with more dread than a variation of this deceptively simple query:

"I want to publish a simple personal website and be able to easily edit its content. Nothing much, just a bio, a portfolio, and a contact form. I can’t afford to hire a web developer, but I’m a little technical, I think I could do it. What tools would you recommend?

The reaction is typically a deer-in-the-headlights look, as if having explain to a small child that puppies die sometimes. It is true that a multitude of tools and services exists, but answering the question is less about picking the best tool for the job, and more about scrambling to figure out the lesser of many evils.

Social media services (e.g. Facebook or Medium) are likely the lowest threshold (see Section 1.4.1) solution, but also come with a very low ceiling. They afford no control over presentation, and data schema and storage is entirely controlled by the service provider. Similar downsides apply to website builders like Wix or Squarespace, though to a lesser extent.

Content Management Systems (CMSes) are meant to be a middle ground between the lack of control of centralized services and the complexity of writing a web application from scratch but are associated with high levels of dissatisfaction [2]: they still require a lot of technical skill to set up and maintain, they are bloated and heavyweight for most use cases, yet still too rigid for many common use cases.

On the other end, the highest ceiling solution is to write a web application from scratch. However, even for more technical users, this is a daunting task. Even a deceptively simple website like the one described above would require a lot of code, and deep understanding of many technical concepts such as authentication, templating, sending and receiving HTTP requests, data binding, handshakes, asynchronicity, security, and many more.

Despite the Web being originally envisioned as a read-write medium [1, 3], web publishing today suffers from numerous usability cliffs (see Section 1.4.2).

1.1.1From Documents to Web Applications

Beyond publishing content, many users have data management needs that cross into the realm of web applications, requiring not just data binding, editing, and persistence, but also computation and interactivity. Examples abound: managing tasks, expenses, recipes, tracking life events, calculating interest rates and loan payments, or even more complex use cases like managing a small business or a community, to name a few.

Some of these use cases are common enough to make business sense for launching specialized commercial applications, but others are part of the very long tail of use cases that too niche to be served by commercial applications individually, yet vast in aggregate.

Even for use cases that on the surface appear to be well served by commercial applications, user needs are also varied and often not fully met by one-size-fits-all solutions. For example, let’s take a simple use case like tracking household expenses. Some families have joint finances, others keep them separate. Some of the latter split expenses evenly, others proportionally by income, and others anywhere in between. Some families only need to deal with one currency, others travel enough that currency conversion is a frequent concern. Prefabricated applications either only deal to the subset of these needs that are most frequently encountered (known in product management as the 80/20 Pareto Principle [4]), or grow to enormous complexity (feature creep) if they try to cater to all of them. While avoiding feature creep is generally good, it does mean that the resulting applications skew toward mainstream needs, and often leave minorities behind.

The main alternative to prefabricated applications is to build one’s own tools. Unlike the web publishing use cases, users rarely ask deceptively simple questions about this — they simply assume that building high fidelity tools for their needs is out of reach. When the delta between their needs and those catered by the prefabricated options, they typically try to adapt them to the tool. When it is too large, they resort to no-code tools such as spreadsheets, which do help with data management and lightweight computation, but are very limited in terms of presentation and interactions and many users struggle with authoring and debugging formulas [5, 6].

Creating websites and creating web applications is often treated as two distinct use cases, but the line between them is blurry. The need to manage structured data and share and display them on a webpage is very common. Consider a personal website displaying a portfolio, or a list of publications, speaking engagements, press mentions. Or a restaurant needing to manage and display their menu with dishes, prices, categories. Or a real estate website displaying listings of available properties. Or a wedding website that includes an RSVP. Or a conference website that includes a list of speakers, abstracts, and a schedule.

In all of these cases the data is structured, and cannot be managed (well) by interfaces essentially treating it as rich text. Furthermore, while these are often presented statically to end-users, end-users benefit tremendously [7] from the ability to interactively explore the data via filtering, sorting, aggregates, and other operations. The end-user need is so strong that there has been research in enabling such capabilities on websites not designed to provide it [8]. Making it easier for website authors to provide such functionality in the first place could provide tremendous value to end-users and have a ripple effect on the Web as a whole.

1.2The Mavo Ecosystem for Low-code Web Application Development

<body mv-app="todo" mv-storage="https://www.dropbox.com/…/todo.json">
	<h1>My tasks</h1>
	<p>[count(done)] done, [count(task)] total
	<ul mv-list property="task">
		<li>
			<input type="checkbox" property="done" />
			<span property="taskTitle">Do stuff</span>
		</li>
	</ul>
	<button mv-action="delete(task where done)">Clear Completed</button>
</body>

It was these recurring pain points around managing, sharing, and transforming data on the Web that led me to design the Mavo language and associated components.

Mavo is a novel low-code programming language that extends the declarative syntax of HTML to describe small-scale web applications that manage, store and transform data (henceforth referred to as data-driven web applications).

Authoring HTML does require some technical skill (although the ACM cites knowledge of HTML and CSS to be at the K-12 level of computer literacy [9]), but lowering the barrier of web application programming down to authoring HTML brings it within reach for everyone. Even for end-users who have never written a line of code, learning HTML from scratch is a much more manageable task than learning the entirety of modern web development concepts.

The Mavo language consists of four key components:

1. Formula², a hierarchical formula language designed from scratch to be easy to use and understand, even when working with deep hierarchical data structures.
2. Madata, a set of protocols and APIs which allow applications to read and store data either locally or to a variety of remote services, all with the same unified API.
3. The Mavo HTML syntax, which extends HTML with syntax to describe data-driven web applications and embeds reactive computation via Formula² expressions and unified storage via Madata URLs.
4. Data Update Actions, an extension to both Mavo HTML and Formula², which allows authoring data manipulation sequences that are triggered by user actions while largely maintaining the same low threshold (see Section 1.4.1) of Mavo HTML and Formula².

While there is great synergy between these four components, each of them is an independent contribution of this thesis, and is useful even without the others.

All four Mavo components share the same design principles, which are also key features that enabled the growth of the early Web:

• No installation, configuration, or maintenance. Anyone could “join the Web” simply by putting an HTML file on a web server. Similarly with Mavo, one only needs to put an HTML file on a web server capable of serving static files (no server-side code execution is required) and they can immediately take advantage of Mavo. This is a direct corollary of Section 1.4.1.
• Tinkerability. A web application’s entire logic is in its HTML file, and can be copied and tweaked. Furthermore, the data source of any Mavo app can be overridden by simply changing a URL parameter, which enables end-users to repurpose other people’s Mavo apps for their own needs even without copying them to their own file space.
• Incremental complexity. Authors can add additional functionality and complexity in small steps, never needing to swallow a whole new set of ideas in one dose (see Section 1.4.2).
• No network effect required. Unlike social networking sites, Mavo provides immediate benefits to its first adopter, regardless of others’ actions. It simplifies the author’s management of their data, and offers visitors using existing web browsers a better interface to that data than can be built by typical web authoring tools with the same effort.
• Robustness and fault tolerance. Fault-tolerance is one of the design principles that guided the design of these technologies (see Section 1.4.4). Reminiscent of the design philosophy of Scratch [10], Mavo components generally attempt to do something sensible with most input rather than failing with an error message.

1.2.1Formula²: A Human-centric Hierarchical Formula Language

A key part of Mavo is its formula language called Formula² (MavoScript in earlier literature) [11, 12]. Formula² expressions can be embedded almost anywhere in Mavo HTML by delineating them with certain syntactic tokens, or raw in certain attributes.

Formula² was designed with the explicit goal of reducing the amount of cognitive overhead around abstract data operations, and allow novices to write formulas that are closer to natural language, yet still unambiguous and easy to parse. To achieve this, it introduced several novel concepts, such as:

• Implicit reference semantics, where references are resolved based on the context of the formula, to alleviate users from complex mapping operations or long and fragile reference chains.
• Seamless list-valued operations, where operations on lists work just like operations on scalars, to reduce how much novices need to think about (or even know) the structure of their data.
• Robust and forgiving syntax in line with our design principles, which is unusual in the space of formula languages.

The contributions of Formula² are described in more detail in Chapter 4.

While originally developed for Mavo, Formula² has no particular dependence on Mavo concepts, and can be used to evaluate expressions against any arbitrary hierarchical data structure. That said, it is primarily useful for systems where the expression and the data have a natural mapping to elements in a visual layout, whose visual hierarchy largely follows the data hierarchy. Mavo is one such system, but so are most visual no-code systems.

1.2.2Madata: Facilitating Data Ownership by Democratizing Data Access

One of Mavo’s key features is its ability to store data remotely on a variety of cloud services, without requiring the author to register any OAuth [13] applications or write any authentication code. Storing and reading data remotely becomes almost as simple as storing it locally, and one storage service can be seamlessly swapped for another with the same capabilities without requiring any changes to the application code. All that users need to do is simply provide a URL that unambiguously identifies the storage location and Mavo takes care of the rest.

Originally hardcoded in Mavo HTML, after launching Mavo as an open source project in 2017, it quickly became clear that the potential reach of these concepts was broader than Mavo.

Reading and storing data is an integral part of many languages and systems. Yet, end-users typically have no control or ownership over their data. This is partly due to business reasons, but also because its is far easier for application developers to store data in a central location they control.

Madata makes it trivial to store data on any supported service, and swap out one service for another. Storage locations are specified by URLs, most of which can be easily obtained from the user interface of each service. Then, Madata takes care of the rest (authentication, data transformations, pagination, flags, etc.). Swapping one service for another is simply a matter of using a different URL, and requires no changes to the application code.

To ensure robustness and prevent centralization, extensibility is essential. Teaching Madata about new backends requires minimal JavaScript knowledge, especially for backends that follow certain known protocols (e.g. OAuth 2 [14]).

Madata frees authors from the need to procure servers that can run server-side code, a far more involved task. Nearly all of Madata runs client-side and interacts with APIs directly from client-side JavaScript.

There is one exception: Authentication. To facilitate experimenting with different storage locations without having to go through the hassle of registering applications, Madata introduces the concept of a federated authentication provider. This is a generalization of Mavo’s original ad hoc authentication server (auth.mavo.io), which is now simply another Madata authentication provider. These are servers that encapsulate API keys for supported services, and handle authenticating end-users and ensuring that users are not misled by malicious applications.

The European Union establishes data portability as a fundamental human right [15]. Madata prototypes a future where end-users can own their data and choose its location by simply entering a URL in the settings of the application they are using. If they later change their mind, and wish to store their data elsewhere, all they need to do is change the URL. This data portability affords a federated version of data ownership that places no additional (time or technical skill) burden on end-users than centralized architectures.

1.2.3Mavo HTML: Creating Web Applications by Authoring HTML

A key contribution of this dissertation is Mavo [11], a novel programming language that extends the declarative syntax of HTML to describe Web applications that manage, store and transform data (these will henceforth be referred to as data-driven web applications).

Using Mavo, authors with basic HTML knowledge define complex nested data schemas implicitly as they design their HTML layout. They need only a few HTML attributes and expressions to transform a static HTML template into a persistent, data-driven, access-controlled web application whose data can be edited by direct manipulation of the content in the browser. Mavo has been evaluated in lab studies, and in the real world, as an open source project.

Unlike current low-code/no-code approaches based on proprietary platforms, evolving the HTML language provides a solution that is universal and portable, with no dependence on any particular web infrastructure. By defining its syntax as an extension of HTML, all tools that process HTML — some of which do target end-users — can also process Mavo code.

This resulted in the following key ideas and primitives for Mavo HTML:

• UI First. User interfaces are less abstract than data, and thus require less technical expertise to reason about. With Mavo, authors are designing their interface with the tools they are used to; then they annotate where data goes in it. The data model is not specified separately, in the abstract; it is generated through these annotations. We believed that pointing to concrete places on a template is easier for novices than the abstract data modeling tasks that traditional software engineering requires and our lab studies validated that hypothesis.
• Editability. Creating a WYSIWYG interface for editing data in place is as simple as naming the data and choosing an appropriate HTML element for it.

Furthermore, embedding Formula² expressions and Madata URLs in Mavo HTML results in these additional primitives:

• Persistence. Data can be stored locally or remotely, on one of the many supported cloud services, by simply providing a storage URL. Mavo takes care of authentication, if needed. Access control is enforced by the remote service.
• Lightweight computation through a reactive expression language called Formula² similar to spreadsheet formulas but designed for nested schemas like those organically created in most Mavos. A key feature of Formula² is its novel reference mechanism: properties can be referenced from everywhere in the template, and the relative placement of the expression to the data affects what the named reference resolves to.
• Reactive defaults, which are essential to many very common use cases such as smart default values, or editable formulas.

1.2.4Data Update Actions

Originally, Formula² was purely reactive and side-effect free. However, we kept encountering use cases requiring programmatic data modification, triggerred by user actions. Often applications were almost entirely CRUD with lightweight computation and only one or two simple actions, but the inability to specify these actions made Mavo unsuitable for these use cases.

After exploring several alternatives, we decided to make these possible by extending Formula² with data update actions [12], which are only enabled in specific application-dependent contexts (e.g. an mv-action attribute in Mavo). We then did user research to ensure that our proposed syntax felt natural [16] to novices.

Our design adds minimal complexity but significantly expands the use cases that can be satisfied.

While our research focused on Mavo applications, the core concepts can be used to extend any reactive formula language with Data update actions (and since the publication of [12], some commercial no-code systems implemented similar ideas to great success).

In fact, data update actions do not even depend on hierarchical data structures, as this is a common spreadsheet user pain point. Perhaps this work could serve as a basis to address it.

1.3Lowering the Threshold to End-Users

Mavo is a low-code language, rather than a no-code system² and targets HTML authors rather than end-users. While we have made the argument that the effort required for an end-user to become an HTML author is minimal, and certainly orders of magnitude smaller than the effort required to become a fully-fledged web developer, any amount of syntax is a barrier to entry for a large group of people.

My later research explored the question If we eliminate HTML syntax, would end-users be able to use and understand Mavo concepts?.

We hypothesized that a domain-specific visual app builder would be more effective. Since personal tracking use cases are both very common, and a class of applications with minimal network effects, we decided to start by prototyping Lifesheets, a visual IDE for building custom Quantified Self [17] applications.

In addition to demonstrating that Mavo concepts can largely be understood by end-users with no technical skill beyond spreadsheets, Lifesheets introduces a novel architecture for empowering users of all technical skills to create web applications that are portable, malleable, and not dependent on any particular infrastructure.

1.4Design Principles

We conclude the introduction by describing a set of design principles that guided the development of the languages and systems presented in this thesis.

1.4.1Maximize the Distance Between Threshold and Ceiling

“Simple things should be easy, complex things should be possible” — Alan Kay (rumored)

Decades later after Alan Kay, Myers et al formalized this idea by introducing the concept of threshold and ceiling [18].

The threshold is how difficult it is to learn how to use a system³, i.e. its learnability; the ceiling is how much can be done using it, i.e. its expressive power.

Myers said that most successful systems are either low threshold / low ceiling (easy to learn but limited in expressiveness) or high threshold / high ceiling (hard to learn, but very powerful). In other words, most successful systems either trade off learnability for power or the opposite.

It seems clear that balancing a low threshold and a high ceiling would be ideal, but per Myers et al, it remains a challenge.

1.4.2Incremental User Effort Should Produce Incremental Value

While a low threshold and a high ceiling are certainly desirable, and establish a usability bar that a good majority of systems cannot pass, they are not sufficient.

Many systems today achieve a low threshold and a high ceiling by simply combining a low threshold / low ceiling solution with a high threshold / high ceiling one. When more power is desired than what the low-threshold solution affords, users are directed to the high-threshold solution. This introduces a “usability cliff”, a point where a small increase in use case complexity results in a disproportionately large increase in UI complexity.

<video src="myfile.mp4" controls></video>

The threshold and ceiling merely establish the two extremes of a spectrum, but many use cases are not at either extreme. For optimal usability, we want a smooth use case complexity to UI complexity curve, where UI complexity increases gradually with use case complexity. Incremental user effort should result in incremental value; there should be no sudden jumps in complexity. The rate of increase matters too; the flatter, more horizontal the curve, the better.

Essentially, this is a corollary of the Attention Investment Model of Abstraction Use [19], whose core idea is that programmers have a finite supply of time and attention to invest. For an investment to be worthwhile, the expected payoff must exceed the cost, unless the risk is too great. The cost of the investment is the amount of attention by the user that must be devoted to accomplishing a task. The expected payoff from that investment will be some saving of attentional effort in the future, such as by achieving a good abstract formulation to reduce the amount of effort required to cope with similar problems. The perceived risk is the extent to which the user believes the investment will not produce the payoff, or that it will lead to even more costs that are not yet apparent. A cognitive simulation of programmer behavior has validated that this simple investment model can model many of the actions and decisions made during programming tasks, both by professional software engineers and end-user programmers [19], and there is evidence that it is effective in practical language design [19].

A lot of the work presented in this thesis is about either reducing the threshold of web programming, or making the curve of use case complexity to UI complexity more gradual.

1.4.3Inference Should Be Escapable

Traditional programming languages often opt for explicit paradigms, where every parameter of the computation is specified by the programmer. Everything is clear cut, and there is no ambiguity, but to avoid potentially incorrect inference, this design offloads a lot of work to the programmer, increasing cognitive load.

This rigidity can be frustrating for novices, who are more familiar with the communication paradigms of natural language, which favor implicitness and ambiguity [20, 21]. In natural language, the receiver of a message will largely infer several concepts from context, and ask for clarification when needed. A compiler cannot ask for clarification, it can only produce errors. The “clarification” is essentially the programmer fixing the issue.

Heuristic algorithms that attempt to infer author intent from incomplete input can often improve user experience by reducing the amount of explicit input required and the amount of errors produced (which we know are discouraging). However, when the inference is incorrect, it is essential to provide a way for users to override the inferred behavior and provide explicit input.

The importance of providing overrides depends on the frequency and consequences of incorrect inferences. In some cases, the presence of alternative ways to solve the same problem can be a sufficient escape hatch.

In some ways, this is a corollary of Section 1.4.2: inference is making simple things easy, while escape hatches are making complex things possible.

1.4.4Be Liberal in What You Accept

HTML is possibly the most tolerant mainstream computer language. This is no accident; tolerance was one of its earliest design principles [23–25].

Eliminating error messages does not eliminate errors. However, when a program does something, even if it is not correct, it feels closer to working and is less discouraging than a program that does not run (or compile) at all [10].

Per [26], there are no errors; all operations are iterations towards a goal. Typing mistakes or illegal statements can be thought of as an approximation. The language’s job is then to aid the user in rapid convergence to the desired goal. In some cases, that may be achieved via inference (see Section 1.4.3), in others by failing gracefully. Notifying the user that there is a problem is important, but rarely requires complete and total failure.

This kind of resilience is especially important on the Web platform, where the environment is unpredictable and the user base is vast and diverse. There is no guarantee that when the error condition occurs, the user will be the website author. Thus, resilience ensures a better user experience for all Web users.

1.5Thesis Overview

Chapter 2 positions this thesis in the broader context of related research and tools that aim to make web application development easier.

From there, Chapter 3 to Chapter 6, and then Chapter 9 describe various languages and systems democratizing web application development and empowering data ownership from different angles:

• Chapter 3 introduces the Mavo HTML language and briefly describes Formula² and Madata and how Mavo HTML integrates them.
• Chapter 4 expands on the Formula² hierarchical formula language.
• Chapter 5 expands on the Madata JavaScript API and federated authentication architecture.
• Chapter 6 introduces Data Update Actions, a way to add programmatic data manipulation to reactive formula languages.
• Chapter 9 introduces Lifesheets, a domain-specific visual application builder for building Mavo applications for personal tracking.

These chapters present the latest design of each technology, which is often the result of multiple iterations following insights from user studies and deployments. They include results from formative needfinding studies, example use cases, descriptions of system specifications, and implementation details.

Then, Chapter 7 provides an overview of the various studies conducted to evaluate these systems, and provides context for the status of Mavo technologies at the time each study was conducted. These include results from lab evaluations, case studies, and wide deployments as open source projects. We decided to present them after the description of all four languages and systems, as many studies were evaluating more than one component.

Chapter 8 presents a series of case studies of Mavo applications showcasing all technologies in the Mavo ecosystem working together to produce high fidelity applications. Some were created by Mavo users, and some by the author. Some are included because they showcase interesting patterns for common use cases, and others because they push the boundaries of what is possible with Mavo. Each case study is accompanied by a description of key points from its architecture and implementation, as well as a list of limitations it exposes in the current Mavo ecosystem.

Last, Chapter 10 summarizes design lessons from these languages and systems, their user studies and their deployments, discusses current limitations, and proposes future research directions.

The thesis concludes in Chapter 11 by reviewing and summarizing the contributions of this work.

There is a multitude of systems that assist novice web developers and end-users with building dynamic, data-backed web applications, including research and commercial tools. This section provides an overview of the current landscape.

2.1Web Publishing Tools

2.1.1Social Media

In recent years, social media platforms for publishing web content have become a popular way for end-users to publish their content for free, without having to deal with any of the technical challenges of publishing a website they own.

Examples include Facebook Pages, blogging platforms (e.g. Medium, WordPress, Tumblr, etc.), or profiles on media-rich social platforms (e.g. Instagram, Flickr, TikTok, YouTube).

Out of all methods of publishing content on the web, this is certainly the one with the lowest threshold, which explains its popularity.

However, it comes with severe drawbacks and limitations on the type of content that can be published, what can be done with it, and how it can be displayed. These are typically designed around the most common, most generic use cases (e.g. a blog or a gallery of photos), which imposes severe limitations on the type of content that can be published, what can be done with it, and how it can be displayed. Because they are still part of the social platform that hosts them and need to maintain a consistent brand identity, they are usually limited in terms of personalization and customization. Custom functionality is typically not possible, and the platform may change or remove features at any time. Data is owned by the platform, and portability is hard or impossible. As a result, if the social media platform shuts down, content is (effectively) lost.

2.1.2SaaS Visual Website Builders

Visual website builders like Wix and Squarespace have revolutionized web development by enabling novices to create and manage websites that look professionally designed through direct manipulation interfaces, making it feasible for small businesses, freelancers, and individuals to establish an online presence quickly and affordably. Being designed as creative tools, these afford much better customization than the social media solutions discussed in the previous section, as the central focus is the creative artifact produced, not the connections between users.

However their how threshold typically also comes with a relatively low ceiling, or the bifurcation described in Section 1.4.2 of having a low threshold and a high ceiling by combining a low-threshold/low-ceiling solution with a high-threshold/high-ceiling solution. These platforms rely heavily on pre-designed templates that dictate much of the site’s layout and visual appearance. While these templates are often polished and professional, they are also quite rigid. Users can make changes within predefined sections and elements, but the overall structure is usually fixed. Advanced functionality is provided via predefined plugins, widgets, and integrations, which can be added to the site with a few clicks, but when these do not serve needs well, writing code is often the only escape hatch.

As a result, while these platforms excel in simplicity and speed for use cases that conform to the most mainstream of needs, they fall short to cater to the very long tail of specialized use cases that emerge in practice.

Additionally, since these are (usually proprietary) platforms, users have limited control over their data and content and transferring a website to another platform can be challenging.

2.1.3Content Management Systems (CMSes)

CMSes are possibly the most popular way for end-users to publish their content on a website they control.

These include platforms typically hosted on one’s own server which connect to some form of data storage (e.g. a database), and provide templating functionality and visual affordances for editing content. Examples of such systems include CMSes such as Wordpress, Drupal, or Joomla.

Previous work has explored the high levels of dissatisfaction with how rigid and heavyweight these are [27].

The drawback to many of these systems is that they often require using their own heavyweight authoring and hosting environments, and they provide pre-made plugins or templates that users can not customize without programming.

Another drawback is that they are structured around a very crude model of what is UI and what is data, typically consisting of a set of pages with content that is edited all at once, and no computation. While this model works for content-heaby websites, such as blogs or media portals, it does not work so well for displaying and editing structured data which is a lot more fine-grained than a single blob of text with a title and other metadata.

Displaying and editing structured data is a broad category of use cases that come up very frequently, even for content-heavy websites. For example, thing of the personal blog of a popular conference speaker, a textbook CMS use case. Displaying their list of talks, their list of publications, a list of press mentions, a list of interviews they have given, all of these are examples of structured data for which a CMS is not well suited.

2.1.4Static Site Generators (SSGs)

The dissatisfaction around CMSes bred the growing community around static site generators, such as Jekyll [28] and Eleventy [29]. These do not have a visual interface at all, content is typically stored in Markdown files and HTML templaes, and the final HTML is generated by invoking a terminal command.

While these are a lot more lightweight and afford tremendous levels of control, they practically target exclusively web developers, as they require significant technical expertise to configure, and offer no graphical interface for editing data.

Many uses of CMSes are merely to enable non-technical users to edit website content, a use case that static site generators do not accommodate.

2.1.5“Headless” CMSes

“Headless” CMSes are tools designed to bridge the gap between CMSes and SSGs, by combining the ease of use of the former with the control of the latter. However, these typically require the SSG to first be configured normally, and then its templates painfully annotated to tell the CMS where data should go and how to edit it.

Moreover, they tend to fare poorly at providing a WYSIWYG preview of the rendered website, since it’s not always clear to them what the content managed in the system will be used for in the end.

2.2Spreadsheet Extensions

So far, the types of website builders discussed focus on editing content and making it look good via templates. Any computation is added via plugins, and if no suitable plugin exists, it requires programming.

However, there is already a very successful paradigm for end-users to store their data and perform computations on it: spreadsheets. Because of the popularity of spreadsheet applications, many researchers and practitioners have explored eliminating the usability issues of spreadsheets and pushing the boundaries of the spreadsheet paradigm.

Common extensions to the spreadsheet paradigm include:

• Extending the formula language to named references [30–32]
• Allowing the user to define datatypes, defaults, and formulas for entire columns [30–32]
• Extending the formula language and input affordances to support hierarchical data [33–35]
• Allowing the cells to be arranged in layouts other than a grid [36]
• Extending the output to richer data types such as interactive graphics [36–38] or maps [39]
• Making it easier to correlate data across multiple tables via relations [31, 32]

While spreadsheets address the user need for lightweight computation, most spreadsheet systems (research or commercial) share the same limitations:

• They afford very little to no customization in terms of input UI
• They are typically not portable: the data is stored within the spreadsheet, and the functionality cannot easily be repurposed to handle different data or moved to a different platform.
• Many only target single-user local web applications and do not address the unique challenges that Web applications raise.

Because of the popularity of spreadsheet applications, some researchers have explored using a spreadsheet for end-users to define and manage their data. For example, Quilt is a system that allows users to link a Google spreadsheet with a webpage and provides simple syntax to bind GUI elements with particular cells in the spreadsheet [40]. Similarly, Gneiss is a live programming environment that incorporates a spreadsheet editor and allows users to create bindings between GUI elements and spreadsheet cells [33].

2.3Do-It-Yourself Database-driven Web Applications

A large class of web applications are purely CRUD (Create, Read, Update, Delete) interfaces to structured data. Databases allow storing and querying structured data, but integrating them into a web application is quite laborious even for professional programmers. A small study [41] found that the ratio of “plumbing” code to pure data code (business logic + SQL) in a web application was a whopping 24.4:1!

Therefore, many systems have focused towards making database systems more user-friendly for web application development, and/or building interfaces to easily display and edit database data.

One direction involved bridging the spreadsheet and database paradigms by exposing a database as a hierarchical spreadsheet [32, 35, 39, 42, 43].

SQL is a widely accepted data query and manipulation language, and its declarative nature means that relatively complex data queries and updates can be performed using even a single short line of SQL.

However, web designers with limited knowledge of databases might not be able to write SQL queries in order to make these edits programmatically. Several database-driven web application platforms have been developed to assist non-programmers to build web applications. WebML [44] presented a web modeling language that provided a graphical way of specifying the database schema and navigational structure of web application. However, WebML does not provide a mechanism to do programmatic updates to the data. A lot of work has been conducted on developing visual query languages [45–47]. These systems hide the SQL syntax from the users, but they still show the database schema and the relational tables, which could be overwhelming for non-programmers with limited knowledge of databases. They also do not offer any way for the user to create web pages on top of these visual query languages. Other systems have focused on creating form-based visual tools for creating queries, design database, and define views [48–50]. However, these tools do not offer a WYSIWYG environment and they similarly require the users to deal with joins across multiple tables, which has been shown to be unnatural for average users [51].

AppForge [52] tried to hide the complexity of building and editing databases by developing a graphical interface to navigate the database schema. And like our proposed extension to Mavo, it provided graphical primitives, in which developers can create and edit NRA views over the schema. Nevertheless, it exposes non-programmers to the complexity of databases. FORWARD [53] is another system that provides a powerful WYSIWIG environment for creating web applications, however, not only it requires writing SQL queries within HTML, but it also requires writing JavaScript if users need to create a custom visual layer. Other systems, like that presented by Kowalzcykowsi et al. [54] provide a WYSIWYG environment and do not require users to edit the database schema directly; nevertheless, they do not provide an abstraction for complex relationships, aggregation and nesting. Mavo [11] allows users with basic HTML knowledge to create Web applications that manage, store and transform data, and unlike some of the previous work it provides for nested data, but does not let you join one nested data blob to another, it also offers controls for adding, updating, and deleting individual items manually. However, the only data manipulation that Mavo presented is direct editing of a single item, although in many applications, even simple ones, there are more complex editing actions that need to be developed. In this work, we extended Mavo to support specifying such actions programmatically.

2.4WYSIWYG Application Builders

Visual application builders like app2you [54] and AppForge [52] allow authors to specify the design of pages by placing drag-and-drop elements into a WYSIWIG-like environment. However, this approach limits authors to only the building blocks provided by the tools and provides very little control over the specifics of the interface created.

Some systems have been developed to provide a WYSIWYG interface that allows non-programmers to create web applications, without dealing with the complexity of databases and SQL queries, by using spreadsheet as the a back-end. Dido [55] allows users to visualize, edit, and store editable data directly in their browser. It allows web designers to integrate Dido into any web design and made it independent of any back-end system. Another system is Quilt [40], which integrates web applications to a Google spreadsheet, allowing web authors with no programming skills to gain access to lightweight computation.

Gneiss [33, 34] is another interactive system that extended a spreadsheet. It lets users retrieve JSON data returned from web services to a spreadsheet interactively without programming, unlike some of the previous work, Gneiss supports hierarchical data. However, since it depends on spreadsheets as the back-end, it does not really provide a mechanism to update data. None of the previous work provided a mechanism for programatically allowing end-users to specify data updates, without them having to write SQL queries, which can get complicated for nested schemas [51] or scripting. Our work is building on top of Mavo to make it more powerful, allowing users to specify computational data updates that are not evaluated reactively, but are executed based on user interaction. Rather than limiting users to only manually data editing, we want to empower them to create richer data interactions and ultimately, to build more powerful web applications.

2.5HTML Extensions for Web Application Development

The idea of extending HTML to make it more powerful is not new; there have been many past attempts at extending it in different directions.

Many attempts to make HTML more powerful treat HTML as a shortcut for programmers to express programming concepts more succinctly. They focus on reducing the amount of programming code required, not its difficulty. One such system was FORWARD [41] which aimed to simplify the “plumbing code” needed to render and edit data stored in a SQL database into a web page. It was quite powerful, but required writing SQL queries within HTML. There are also several JavaScript frameworks with this philosophy, starting with AngularJS [56] in 2010 and more recently VueJS [57]. These adapt and extend HTML to present dynamic content through two-way data-binding that allow for the automatic synchronization of models and views, but require the user to be well versed in JavaScript to use them.

ConstraintJS [58] extended HTML with a templating syntax and reactively evaluated constraints, but required the user to understand and write JavaScript.

Exhibit [7] (and later Dido [55], based on Exhibit) were some of Mavo’s early influences. Exhibit extended HTML with language elements that visualized and stored editable data directly in the browser. This approach allowed a web designer to incorporate Dido into any web design and made Dido independent of any back-end system.

Quilt [40] was one of Mavo’s biggest influences. It extended HTML with a language for binding an arbitrary web page to a Google spreadsheet “back-end”, enabling web authors to gain access to lightweight computation without programming. While it afforded full creative freedom, and lightweight computation, like Mavo, it imposed the software engineer mindset of data modeling as a separate task, and UIs as views that need to connect to the data model as a separate step, which can be cognitively taxing for end-user programmers that tend to be goal-oriented. Additionally, it was by design limited to spreadsheets as the means for data storage.

2.5.1Web Components

Web Components is the colloquial term for a set of standardized technologies that allow developers to encapsulate reusable functionality in HTML elements, and provide an extensibility point to HTML by allowing the creation of custom elements. While Web Components require (fairly advanced) JavaScript to create, because they can be packaged and distributed, the theory is that novices can import them and use them just like native HTML elements.

Web Components are not an alternative to Mavo: first, novices can only use them, not create them, and second, they exist at a different level of abstraction than Mavo which focuses on facilitating data interactions, rather than encapsulating UI functionality.

However, Web Components are complementary to Mavo. Because Mavo leverages existing HTML elements as foundational building blocks, the addition of custom elements through Web Components broadens the range of functionalities available to Mavo authors. Well-designed Web Components extend Mavo’s capabilities while maintaining the same low threshold as native HTML elements, thereby expanding Mavo’s utility without increasing its complexity.

2.5.2HTML Transformation Languages

There are several languages designed around transforming data to HTML or simpler HTML to more complex HTML, in order to automate repetitive templating tasks.

Extensible Stylesheet Language Transformations (XSLT) [59] is a language designed for transforming XML documents into different formats, including HTML. It is primarily used to transform XML data into a presentable HTML format, applying styles and formatting rules that dictate how the content should be displayed in the browser. XSLT is powerful in environments where XML is the primary data format, allowing for the separation of content and presentation, but its syntax is complex and verbose, making it difficult for non-programmers to use.

Several templating languages also exist (e.g. Handlebars, Mustache, Jinja, etc.) that allow authors to write HTML templates with placeholders for data, and store the data separately in a structured format (e.g. JSON, YAML, etc.).

Cascading Tree Sheets [60] was a research language that essentially functioned as a templating language where both the data and the output were HTML. It allowed authors to write minimal HTML documents with only the elements required to hold their data, and add any superfluous presentational markup as transformations of that HTML, specified via CTS rules with a CSS-like syntax.

These languages are typically static one-time transformations that produce HTML from data, not dynamic data bindings.

2.6The Semantic Web and Web Data Extraction

There has been a great deal of work on both encouraging and extracting structured data on the web [61]. However, automatic scraping techniques often have errors because they must infer structure from unstructured or poorly structured text and HTML markup.

Several efforts have been made to define syntaxes and schemas, such as RDFa [62] and Microdata [63], for publishing structured data in web pages to contribute to the Semantic Web and Linked Open Data [64]. However, novice users have had little incentive to adopt these standards — sharing data rarely provides direct benefit to them — and find them difficult to learn, potentially contributing to their limited adoption on the web.

It appears that the approaches that work best for increasing adoption of semantic web technologies are those that provide immediate benefits to them, such as search engines displaying richer results for structured data, or tools using structured data to improve user interfaces and/or make prose more informative [65].

Mavo contributes to this line of work by using a standards-compliant syntax that is machine-readable, yet produces tangible benefits. With Mavo, authors expend effort because it makes their static website editable or creates a web application. As a side effect, however, they enrich the Semantic Web by producing structured data.

2.7Novice Mental Models & Natural Programming

In the last few decades, several research efforts have focused on how novice programmers or non-programmers struggle in learning how to program [66–68]. These studies showed that this is because of the mental models novice programmers build about the notional machine. Another study found that programming is more difficult than necessary because it requires solutions to be expressed in ways that are not natural for non-programmers [69]. The study examined the ways that non-programmers indicate solutions to common programming tasks, which are often vastly different than the ways programming languages require solutions to be expressed.

Natural Programming [16, 70, 71] is a research area that aims to make programming more accessible to non-programmers by studying what syntax and mental models feel most natural to them, and use these insights in designing languages and systems that allowing them to express solutions in ways that are more natural to them.

3.1Introduction

<body>
	<h1>My tasks</h1>
	<p>0 done, 1 total
	<ul>
		<li>
			<input type="checkbox" />
			<span>Do stuff</span>
		</li>
	</ul>
</body>

<body mv-app mv-storage="https://www.dropbox.com/…/todo.txt">
	<h1>My tasks</h1>
	<p>[count(done)] done, [count(task)] total
	<ul mv-list property="task">
		<li>
			<input type="checkbox" property="done" />
			<span property="taskTitle">Do stuff</span>
		</li>
	</ul>
</body>

Languages like HTML and CSS have characteristics that make them more natural [70, 71] to learn and use. They are declarative, reactive, robust and forgiving in terms of syntax. Authors assemble high-level concepts and constraints, rather than explicit instructions. Robustness is achieved in different ways across the two (HTML attempts to correct authoring mistakes, CSS to scope them tightly and ignore them), but both are designed with resilience and fault tolerance as a design principle.

These desirable properties have given rise to a large community of authors who are comfortable with HTML and CSS, yet not being comfortble with JavaScript or other traditional programming languages. While it is difficult to pinpoint the size of this community, it is likely large and growing. The ACM cites knowledge of HTML and CSS to be at the K-12 level of computer literacy [9].

Far more powerful than static pages are web applications that react dynamically to user actions and interface with back-end data and computation. Even a basic application like a to-do list needs to store and recall data from a local or remote source, provide a dynamic interface that supports creation, deletion, and editing of items, and have presentation varying based on what the user checks off. Creating such applications requires knowledge of JavaScript and/or other programming languages to support the necessary user interaction and to interface with a data management system, as well as understanding of some form of data representation, such as JSON or a relational database.

There are many frameworks and libraries aiming to simplify creation of such Web applications. However, all target programmers and still require writing a considerable amount of code. It is indicative that even implementing a simple to-do application similar to the one in Figure 3.1 requires hundreds of lines of code:

Many people who are comfortable with HTML and CSS do not possess additional programming skills¹ and have little experience articulating data schemas [72]. For these novice web authors, using a CMS (Content Management System) is often seen as their only solution. However, research indicates that there are high levels of dissatisfaction with CMSs [2]. One reason is that CMSs impose narrow constraints on authors in terms of possible presentation–far narrower than when editing a standalone HTML and CSS document. When an author wishes to go beyond these constraints, they are forced to become a programmer learning and modifying server-side CMS code.

The problem worsens when authors wish to present structured data [27], which CMSs enable via plugins. The interfaces for these plugins do not allow authors to edit data in place on the page; instead they must fill out forms. This loses the direct manipulation benefits that are a feature of WYSIWYG editors for unstructured content.

Finally, CMSs provide a heavyweight solution when many authors only need to present and edit a small amount of data. For example, out of the over 7,000 CMS templates currently provided in ThemeForest.net, a repository of web templates, 39% are for portfolio sites, while another 31% are for small business sites.

This chapter presents and evaluates a new language called Mavo² that augments HTML syntax to empower HTML authors to implicitly define data schemas and add persistence and interactivity. Simply by adding a few HTML attributes, an author can transform any static HTML document into a dynamic data management application. Data becomes editable directly in the page, offering the ability to create, update, and delete data items via a WYSIWYG GUI.

3.1.1Implicit Data Schema Definition

While programmers generally prefer to keep their data schema logic separate from presentation definition, end-users may not have the same preferences, and may instead be frustrated by the need to think about data in two separate places. Indeed, with a certain category of applications, including most CRUD applications, how the data is laid out on the page can easily translate to how the data should be organized. For end-users who are seeking to build these sorts of apps, it may be easier to define a proper schema in tandem with defining the layout.

Mavo authors never have to articulate a schema separately from their interface or write data binding code. Instead, authors add attributes to describe which HTML elements should be editable and how, unwittingly describing their schema by example in the process. With a few attributes, authors quickly imply complex schemas that would have required multiple tables and foreign keys in a relational database, without having to think beyond the interface they are creating. As an added benefit, Mavo’s HTML attributes are part of the HTML RDFa standard [62] and thus contribute to machine-readable data on the Web.

Mavo is inspired by the principle of direct manipulation [73] for the creation of the data model underlying an application. Instead of crafting a data model and then deciding how to template and edit it, a Mavo author’s manipulation of the visual layout of an application automatically implies the data model that drives that application. In addition, Mavo does not require the author to create a separate data editing interface. Users simply toggle an edit mode in their browser by clicking an edit button that Mavo inserts on their webpage. Mavo then adds affordances to WYSIWYG-edit whatever data is in view, with appropriate editing widgets inferred from the implied types of the elements marked as data. Mavo can persist data locally or outsource storage to any supported cloud service, such as Dropbox or Github. Switching between storage backends is a matter of changing the value of one attribute.

In addition to CRUD functionality, Mavo also embeds Formula² expressions, allowing users to perform complex calculations on nested data with a natural syntax. Formula² has been described in detail in Chapter 4.

In contrast to the hundreds of lines of code demanded by the popular frameworks, Figure 3.1 shows how an HTML mockup can be transformed into a fully functioning to-do application by adding only 5 lines of Mavo HTML.

Our approach constitutes a novel way for end-users to transform static webpages to dynamic, data-backed web applications without programming or explicitly defining a separate data schema.

From one perspective, this makes Mavo the first client-side CMS, where all functionality is configurable from within the HTML page. But it offers more. In line with the vision of HTML as a declarative language for describing content so it can be presented effectively, Mavo extends HTML with a declarative specification of how the data underlying a presentation is structured and can be edited.

Fundamentally a language extension rather than a system, Mavo is completely portable, with no dependence on any particular web infrastructure, and can thus integrate with any web system.

Similarly, existing WYSIWYG HTML editors can be used to author Mavo applications. We offer Mavo as an argument for the benefits of a future HTML language standard that makes structured data on every page editable, persistent and transformable via standard HTML, without dependencies.

We conducted a user study with 20 novice web developers in order to test whether they could use Mavo to turn a static HTML mockup of an application into a fully functional one, both with HTML we provided and with HTML of their own creation.

We found that the majority of users were easily able to mark up the editable portions of their mockups to create applications with complex hierarchical schemas.

3.2Related Work

Quilt [40] was a system that allowed users to link a Google spreadsheet with a webpage and provided simple syntax to bind GUI elements with particular cells in the spreadsheet. Similarly, Gneiss is a live programming environment that incorporates a spreadsheet editor and allows users to create bindings between GUI elements and spreadsheet cells [33].

Mavo combines ideas from three prior systems that addressed the downsides of CMSs. Dido [55] built on Exhibit [7], extending HTML with language elements that visualized and stored editable data directly in the browser. This approach allowed a web designer to incorporate Dido into any web design and made Dido independent of any back-end system. Quilt [40] extended HTML with a language for binding an arbitrary web page to a Google spreadsheet “back-end”, enabling web authors to gain access to lightweight computation without programming. Gneiss [33, 34] was a web application within which authors could manage and compute over hierarchical data using an extended spreadsheet metaphor, then use a graphical front end to interact with that data.

These three systems introduced powerful ideas: extending HTML to mark editable data in arbitrary web pages, spreadsheet-like light computation, a hierarchical data model, and independence from back-end functionality. But none of these systems provides all of these capabilities simultaneously. Dido had no computational capabilities, could not manage hierarchical data, and was never evaluated. Quilt was dependent on a Google spreadsheet back-end, which left it unable to manage hierarchical data. Gneiss was a monolithic web application that only allowed the user to construct web pages from a specific palette. It did not offer any way (much less a language) to associate an arbitrarily designed web page with the hierarchical data Gneiss was managing, which meant that a web author faced constraints on their design creativity. Gneiss and Quilt both required users to design their data separately from their web pages.

Mavo is a language that solves the challenge of combining the distinct positive elements of this prior work, which are in tension with one another. It defines a simple extension to HTML that enables an author to add data management and computation to any web page. At the same time, it provides a lightweight, spreadsheet-like expression language that is expressed and evaluated in the browser, making Mavo independent of any particular back-end. The editing and expression language operates on hierarchical data, avoiding this limitation of traditional spreadsheet computation.

The combination of these ideas yields a novel system that is particularly well-suited to authoring interactive web applications. In Mavo (like Dido), the author focuses entirely on the design of the web page, then annotates that page with markup describing data and computation. The web page implies the data model, freeing the author of the need to abstractly model the data, manage a spreadsheet, or describe bindings between the two. At the same time, our expression language provides lightweight computation (Quilt and Gneiss), even on hierarchical data (Gneiss) without relying on any external services (Dido). Because they are part of the document (Dido), Mavo expressions can refer directly to data elements elsewhere in the document, instead of requiring a syntactic detour through references to cells in the associated spreadsheet. Finally, because it is an HTML language extension (Dido and Quilt), Mavo can be applied to any web page and authored with any HTML editor, freeing an author from design constraints.

In sum, we believe that the combination of capabilities of Mavo align well with the needs and the preferred workflow of current web authors. In particular, the independence of the Mavo authoring language from any back-end system (or even from any particular front-end interpreter) means that Mavo prototypes a future for HTML and the web browser itself, where data interaction becomes as much a basic part of web authoring as paragraphs and colors.

3.2.1End-User Web Development

There are many systems that assist novice web developers with building dynamic and data-backed web applications. The drawback to many of these systems, however, is that they often require using their own heavyweight authoring and hosting environments, and they provide pre-made plugins or templates that users can not customize without programming. Examples of such systems include CMSs such as Wordpress, Drupal, or Joomla. The growing community around static site generators, such as Jekyll [28] is indicative of the dissatisfaction with rigid, heavyweight CMSs [27]. However, these require significant technical expertise to configure, and offer no graphical interfaces for editing data.

In the previous section, we described three systems—Dido [55], Quilt [40], and Gneiss [33]—from which we draw key insights. However, this work solves challenges in combining those insights into a single system, incorporates additional ideas, and contributes useful evaluation of the resulting system. Most importantly, Mavo demonstrates that the often-hierarchical data model of an application can be incorporated directly into the visual design on which a web author is focused, making the data modeling task an automatic side effect of the creation of the web design. Supporting hierarchical schemas is critical because they occur naturally in many data-driven apps on the web (53% according to [27]). Our evaluation studies users working with such hierarchical schemas.

3.2.2The Semantic Web and Web Data Extraction

There has been a great deal of work on both encouraging and extracting structured data on the web [61]. However, automatic scraping techniques often have errors because they must infer structure from unstructured or poorly structured text and HTML markup. Several efforts have been made to define syntaxes and schemas, such as RDFa [62] and Microdata [63], for publishing structured data in web pages to contribute to the Semantic Web and Linked Open Data [64]. However, novice users have had little incentive to adopt these standards—sharing data rarely provides direct benefit to them—and find them difficult to learn, potentially contributing to their limited adoption on the web. Mavo contributes to this line of work by using a standards-compliant syntax that is machine-readable. Authors typically do not care about theoretical purity and are motivated to add additional markup when they see a tangible benefit. With Mavo, they expend effort because it makes their static website editable or creates a web application. As a side effect, however, they enrich the Semantic Web.

3.3The Mavo Language

A description of the Mavo language follows. Its expression syntax, Formula² is described in detail in Chapter 4, so here we will focus on how Mavo HTML embeds formulas, and not their syntax. Similarly, its storage location is specified as a Madata URL (discussed in Chapter 5), so here we will focus on how Mavo HTML interfaces with Madata, and not the specifics of where data is stored.

We chose to use declarative, HTML-based syntax instead of new syntax for Mavo functionality because our target authors are already familiar with HTML elements, attributes, and classes and because HTML is inherently fault tolerant (Section 1.4.4) Whenever possible, we reused concepts from other parts of HTML. Using HTML5 as the base language also means a WYSIWYG editor for Mavo applications can be easily created by extending any existing WYSIWYG HTML editor — in fact, we discuss an attempt at this in Chapter 9. But as discussed previously, we consider it a key contribution of Mavo that it is a system-independent language. For example, we expect most Mavo authors to frequently take advantage of the ability to “view source” and work with arbitrary HTML.

View source is an essential methodology for learning and adopting new elements of web design. It permits authors to copy and tweak others’ designs (even without fully understanding them) without worrying about new or conflicting system dependencies [27]. Source editing is essential to let authors circumvent any limitations imposed by graphical editing tools.

Per [18] and our Design Principles (Section 1.4.1), we want a low threshold (cost to get started) while allowing users escape the low ceiling (maximum achievable power) of GUI-based tool builders.

3.3.1Creating a Mavo Application

To specify Mavo functionality on an HTML structure, the author places an mv-app attribute on the enclosing element. Its (optional) value provides an identifier that can be used to refer to this app’s data from other Mavo apps on the page, and is used in a variety of other places. If not specified, Mavo looks at the element’s HTML id attribute, and defaults to a generic identifier (e.g. mavo3) if that’s also not present.

3.3.2Data Persistence

By default, Mavo does not store data anywhere, which can be useful for calculator-type applications. Authors can specify a storage location via the mv-storage attribute.

The syntax of this attribute is a thin abstraction over a Madata URL (Chapter 5) to make these easier to specify for novices, and to provide sensible defaults, since Mavo has more information about the use case at hand.

For example, to store data locally in the browser, the Madata URL is local:foo, where foo is a unique identifier for the data store. To alleviate novices from having to understand what a custom protocol is or from the cognitive tax of finding a suitable key, Mavo uses a local keyword instead, and defaults the key to the app’s identifier.

Similarly, to store data in a remote service, e.g. GitHub, Madata expects a URL that identifies a specific storage location (e.g. a repository and file path). These do not need to exist — for storage services with predictable resource URLs (such as GitHub), Madata will create any resources needed (e.g. files, repositories, etc.). However, simply having to choose suitable values can still be taxing to the novices who are be using GitHub as a backend because of its capabilities, but do not necessarily understand its constituent concepts. To make this easier, Mavo includes a set of defaults for each backend that facilitate underspecified URLs. To reuse the GitHub example, authors can specify a URL to their GitHub profile (e.g. github.com/leaverou), and Mavo uses defaults for the repository name (mv-data) and file name (<app-id>.json). That said, not every backend is conducive to this — for example a Dropbox URL does not have a predictable structure, and thus for Dropbox to be used, authors first need to upload an empty file and get a link to it.

3.3.2.1Data Loading Overrides

By default, mv-storage specifices both the data source, and the data destination. If data does not yet exist, an empty dataset is rendered, and the data is created upon saving. However, there are certain common use cases that require more complex logic when reading data (storing data always goes to mv-storage).

Some applications need different backends for reads and writes. These include applications that transform data and store the result elsewhere, certain performance optimizations, or simply read-only applications. To enable these use cases, Mavo supports an mv-source attribute, whose syntax is identical to mv-storage, but it has precedence over mv-storage for loading data.

For many types of CRUD applications, starting from empty data provides a poor user experience. Using a certain default dataset if storage is empty can help users better understand the application’s purpose and functionality, or simply facilitate experimentation, while still allowing the user’s own data to take precedence once it exists. Mavo supports an mv-init attribute for this purpose, which is only used if the main data source (either via mv-storage or mv-source) is empty or unavailable.

3.3.3Data Definition

Our approach to data definition means that end-users define their data by defining the way they want their data to look on the page. This is in contrast to many systems which expect their users to define their data model first and then map their model into a view. In the spirit of direct manipulation, Mavo users are manipulating their data schema by manipulating the way the data looks. We believe that our approach is more natural for many designers, permitting them to directly specify their ultimate goal: data that looks a certain way.

Once Mavo is enabled on an HTML structure, it looks for elements with property or itemprop attributes within that structure in order to infer the data schema. These elements are henceforth referred to as simply properties. If the HTML author is aware of semantic Web technologies such as RDFa [62] or Microdata [63], these attributes may be already present in their markup. If not, they don’t need to understand either technology — authors are simply instructed to use a property attribute to name their element in order to make it editable and persistent. An example of this usage can be found in Figure 3.1.

When an element becomes a property, it is associated with a data value. This value is automatically loaded from and stored to the specified mv-storage location. For many elements (e.g. <span>), the natural place for this value to be “presented” is in the element’s contents. In others, such as <img> or <a>, the natural place for a value is a primary attribute such as src and href. RDFa [62] defines a small set of such attributes (mainly href and src), which Microdata [63] extends with many more, which Mavo adopts.

3.3.3.1Explicit Primary Attribute

There are many cases where the data value cannot be predicted by these heuristics. For example, consider the following Mavo HTML fragment:

<a property="twitter_username" href="https://twitter.com/[twitter_username]">leaverou</a>

Since this is an <a> element, its primary attribute is href, but the data value is in the element content. To handle such cases, both RDFa and Microdata use a content attribute as an escape hatch: If a content attribute is present, it overrides both the element contents, and the primary attribute. However, this requires duplicating data across where it actually lives and the content attribute.

Instead, Mavo adds an additional rule to these semantics: an mv-attribute which allows authors to specify the default primary attribute, with a none value for the element contents (which is what we would use above).

3.3.3.2Objects

Properties that contain other properties become grouping elements (objects in programming terminology); this permits a user to define deep hierarchical schemas implicitly, simply as a natural consequence of spatial containment. For example, an element with a student property can contain other elements with name, age, and grade properties, indicating that these properties “belong” to the student.

3.3.3.3Collections

A core value proposition of Mavo is automating the large amount of repetitive UI code that CRUD applications require to manage collections of data. Interactions like adding items, deleting items (with undo), reordering items via drag-and-drop, plus keyboard handling for efficiency and accessibility for all of the above, are all automatically handled.

To convert a property into a collection, all that authors need to do is mark what should be the collection item — i.e. the element that will be repeated — with an mv-multiple or mv-list-item attribute, or mark the collection itself with an mv-list attribute.

During editing, appropriate controls appear for adding and deleting new elements in the collection, as seen for the to-do items in Figure 3.1. Collection items can themselves be complex HTML structures consisting of multiple data-carrying elements and even nested collections. This enables the author to visually define schemas with one-to-many relationships.

To author a collection, the author creates one representative example of a collection item; Mavo uses this as the archetype for any number of collection elements added later.

As discussed earlier, this archetype can contain real data so it resembles actual output and not just a template, and can also provide default data values for new collection members.

3.3.3.3.1Collection Syntax

Originally, there was no HTML representation for the collection itself — mv-multiple was added to the collection item and was the only way to define a collection. This was not an issue for purely CRUD applications with very simple computations — it was just framed as mv-multiple makes an element repeatable.

Regardless, putting the attribute on the collection container instead was a common slip, although easy to self-correct from feedback (see Section 7.1). However, as noted in Section 7.2, when authoring expressions for hierarchical schemas the lack of an HTML element to host the entire collection created some confusion.

Additionally, this design created some conceptual inconsistencies. Other Mavo attributes (e.g. mv-default or mv-value) had to heuristically determine whether they were being set on the collection itself or its items, with awkward escape hatches.

As a result, Mavo later adopted a more explicit syntax involving two separate attributes: mv-list and mv-list-item. However, per Mavo’s philosophy of flexibility and fault tolerance, collections can be defined with only one of the two, and the other will be inferred. If necessary, suitable container elements to hold the collection or its items will be crated.

3.3.3.4Relationship to RDFa

The Mavo syntax for naming elements is based on a simplified version of RDFa we call Loose RDFa that is designed to prioritize learnability over generality.

Its main differences from standard RDFa [62] are as follows:

1. Objects are inferred from the property structure instead of requiring a separate typeof attribute. Mavo then adds any missing typeof attributes. Authors can additionally add typeof attributes to explicitly declare objects for cases where the inference is incorrect.
2. Plain identifiers (rather than URIs or CURIEs [74]) are allowed even when no vocab is set.
3. typeof values are optional, to declare an object with no specific type.
4. The way primary attributes are inferred (see above), which extends RDFa’s rather primitive heuristics.

3.3.4Data Editing

Mavo generates UI (user interface) controls for toggling between reading and editing mode on the page, as well as saving and reverting to the last saved state (if applicable), as seen at the top of Figure 3.1. In editing mode, Mavo presents a WYSIWYG editing widget (called the property’s editor) for any property that is (a) not already a form control, (b) not a computed property, and (c) not explicitly defined as read-only.

In line with its philosophy of implicit data schema definition, Mavo leverages available HTML semantics to both optimize the editing interface and extract data type information. For instance, a <time> element is expected to hold temporal data, and will be edited via a date or time picker (depending on its datetime attribute), whereas an <img> element will be edited via a popup that allows specifying a URL or uploading an image (Figure 3.3).

3.3.4.1Customization

These heuristics are intended to both reduce cognitive load on the author and to encourage the use of semantically appropriate HTML. However, per Mavo’s design principles, inferred information should be escapable (Section 1.4.3). Indeed, both the data type and the editing widget can be overridden.

The inferred data type can be overridden by using the datatype attribute. This is an RDFs attribute, and thus does not need an mv- prefix, though Mavo uses it more loosely by accepting plain identifiers like number rather than CURIEs like xsd:number.

The generated editing UI can be customized in a variety of ways. For small tweaks, such as overridding an attribute, authors can specify attributes with an mv-editor- prefix, and they are copied to the generated editing widget. For more extensive customization, authors can provide their own editing widgets by linking to an existing element anywhere on the page via the mv-editor attribute, whose value is a CSS selector. If no data type is explicitly specified, this new editor element will be used for inferring it, just like default editors.

For example, if a property only accepts certain predefined values, authors can express this by linking to a <select> menu, essentially declaring it as an enum. Any changes to the linked form element are propagated to the property editors. This way, authors can have dynamic editing widgets which could even be Mavo apps themselves, such as a dropdown menu with a list of countries populated from remote data and used across multiple Mavo apps.

3.3.5Computation

The aforementioned three attributes—mv-storage, property, and mv-multiple (or mv-list/mv-list-item) — are sufficient for creating any CRUD content-management application with a hierarchical schema and no computation. However, many CRUD applications in the wild benefit from lightweight computation, such as displaying a count of items, summing certain values, or conditionally showing text depending on a data value.

To accommodate these use cases, Mavo embeds reactive expressions in brackets ([]) in the HTML, as well as raw within certain attributes (mv-value, mv-if, etc.), called directives. Originally, Mavo supported two expression interpreters: (a) raw JavaScript (executed in a sandbox environment where properties become read-only variables), and (b) MavoScript, a Formula² precursor. Authors did not specify the flavor they were using. Instead, expressions would be first parsed as MavoScript expressions, and if that failed, as JavaScript expressions. This contributed to author confusion, and restricted the evolution of Formula², so it was quickly abandoned.

Our approach to expressions only partially meets the declarative, direct manipulation goal we described in our motivation. It is challenging to specify computation, an abstract process, entirely through direct manipulation. The expression language is similar to that in spreadsheets—fully reactive with no control flow, which nods towards declarative languages. The widespread adoption of spreadsheets provides evidence that this type of computation is within reach of a large population. Furthermore, placing the expression in the document, precisely where its value will be presented, as opposed to referencing values computed in a separate model “elsewhere”, fits the spirit of direct manipulation in specifying the view. During our several user studies many participants volunteered observations that this was effective.

3.3.5.1Embedding Formula² Expressions in HTML

There are two ways to embed a Formula² expression in HTML.

The first is to use an mv-value attribute, containing the expression. This is more verbose than the second method, but allows providing a fallback value (which is also used before the expression is evaluated) and allows Mavo to provide visible feedback when the expression is invalid, as there is an element to apply CSS to.

The second is to enclose it in square brackets ([]), which allows placing it anywhere inside the Mavo instance, including in HTML attributes (but not element names — yet).

To avoid triggering unrelated uses of brackets on individual elements, authors can use the mv-expressions attribute to customize the syntax or disable expressions altogether (mv-expressions="none"). The setting is inherited by descendant elements, unless they have a mv-expressions attribute of their own. For example, for the double-brace expressions common in many templating libraries, authors can use mv-expressions="{{ }}".

The choice of brackets for delineating expressions was based on the observation that non-programmers often naturally use this syntax when composing form letters, such as email templates. In addition, many text editors automatically balance brackets.

An earlier version of Mavo used a more spreadsheet-like =(expression) syntax, but we found from preliminary user studies that few users realized the spreadsheet connection and found it difficult to determine where an expression terminated due to parentheses also being used inside expressions.

3.3.5.2Named References

Instead of referencing mysterious row and column coordinates (a common source of errors in spreadsheets [75]), Formula² (and by extension Mavo) refers to properties by name. Every property defined in a Mavo instance becomes a (read-only) variable that can be used in expressions anywhere in the Mavo instance. The algorithm is described in detail in Section 4.4.1. These named references are necessary since Mavo has no predefined grid for row/column references. We consider this necessity a virtue.

We believe this will decrease bugs caused by misdirected references. Indeed, many spreadsheets offer named ranges to provide this benefit of understandable references (although no-one used them in our user studies, see Chapter 7). For spreadsheets, perhaps the main benefit of the row-column references is having formulas with “relative references” (e.g. to adjacent columns) to automatically update as they are copied down into new rows. But Mavo’s automatic duplication of templates in collections means copies are never made by the user, obviating the need for this benefit.

3.3.5.3Dynamic Collections

Formula² is not restricted to primitive values, but can also operate on lists and objects (and combinations thereof). Mavo harnesses this via the mv-value attribute. We saw earlier how mv-value can be used to specify scalar expressions with a fallback. However, mv-value follows the schema of the element it’s being used on (adapting the data if necessary, see Section 3.6.2). When used on a collection, it renders its value as collection data and reactively re-renders the collection items whenever needed. Combined with Formula² list-valued semantics, this can produce powerful results with very little code. Furthermore, this dynamic collection can also be named and referenced elsewhere, creating a basic but powerful abstraction mechanism.

For example, suppose our To-Do app had a priority property for each task, and we wanted to display the number of tasks per priority (a pivot table). It could be as simple as:

<dl property="tasks_per_priority" mv-list mv-value="task by priority">
	<dt property="priority">(No priority)</dt>
	<dd mv-value="count(task)">0</dd>
</dl>

3.3.6Reactive Defaults

It is worth noting that in collections, the example item can be filled with real data, which makes the template really look like the output, unlike other templating languages where the template is filled with visible markup. In addition, this example data can easily become default values, by using the mv-default attribute without a value. Specifying a value for mv-default will set the default value to that, which is useful for static defaults.

However, mv-default becomes very powerful when combined with expressions, creating reactive defaults. To our knowledge, Mavo is the first novice programming language to provide reactive defaults.

When a property is set to a reactive default, it gets updated whenever the default value changes, unelss it has been edited by the user. This effectively provides a data/formula hybrid which enables many use cases that are otherwise cumbersome or even impossible.

The more obvious use case is facilitating smart defaults that are progresssively refined as other properties are edited. For example, suppose we have a calendar application whose events have start_date, end_date, start_time, end_time properties. end_date could default to start_date if end_time is after start_time, and to one day after start_date otherwise. It would look like this:

<time property="end_date" mv-default="[ if(end_time > start_time, start_date, start_date + 1 * day()) ]">

However, there is a large set of use cases where fields are essentially computed via an expression, but there are some few exceptions where the user might want to override the computed value. Using mv-default rather than mv-value in these cases preserves editability and allows the user to override the computed value.

3.3.7UI Customization

We designed Mavo to be useful to HTML authors across a wide range of skill levels, including web design professionals. This presents a tension: Web designers want control, whereas novices want UI to be generated requiring as little involvement as possible.

The approach that Mavo follows is to generate a default UI, but expose hooks for customization at various levels of the ease-of-use to power curve. Smaller customizations can be done by simply styling the generated Mavo elements with CSS.

In addition, authors can provide their own UI elements that replace those generated by Mavo by using certain class names (such as class="mv-add-task" for a custom “Add task” button).

For the parts of Mavo most likely to be customized, such as the editing widgets or the toolbar, Mavo also provides specific attributes for high level customization with low effort (e.g. see Section 3.3.4.1 for customizing the editor).

3.4Implementation

Mavo is implemented as a JavaScript library that integrates into a web page to simulate native support for our syntax. On load, Mavo processes any elements with an mv-storage attribute and builds an internal Mavo tree representation of the schema (Figure 3.4). It also inspects every text and attribute node on or inside every element looking for expressions, and builds corresponding objects for them. For every expression, a JavaScript parser is used to rewrite binary operations as function calls in order to enable array arithmetic.

Any remote data specified in the mv-storage attribute is then fetched via Madata and recursively rendered. Every time an object is created during editing or data rendering, it holds a reference to its corresponding node in the Mavo tree (Figure 3.4), which it uses as a template. This improves performance by only running costly operations (such as finding and parsing expressions) once per collection.

When the data in an object changes — via rendering, editing or expression evaluation — expressions within it or referring to it are re-evaluated to reflect current values. This occurs in a special execution context where current object data and Mavo functions appear to be global scope. ECMAScript 2015 Proxies are used behind the scenes to conditionally fetch descendant or ancestor properties only when needed, to allow for identifiers to be case insensitive, to make the variables read-only, and to allow identifier-like strings to be unquoted.

3.4.1Extensibility

There are APIs in place for third-party developers to add new default editing widgets, new expression functions, and new storage backends. In addition, Mavo includes a hooks system for developing plugins that modify how it works on a lower level. For example, both the inline Mavo debugging tools and most directives are implemented as plugins and could be removed.

3.4.2Debugging

3.4.2.1Inline Debugging Tools

Mavo includes debugging tools that show the current application state, as expandable tables inside objects (Figure 3.5). This is enabled by placing a mv-debug class on any ancestor element or adding ?debug to the URL. These tables display current values of all properties and expressions in their object, and warnings about common errors. Expressions shown can be edited in place, so that users can experiment in real-time.

3.4.2.2Mavo Inspector

While inline debugging tools may appear to work better with novices, they do produce a lot of clutter. Furthermore, our first user study reported low levels of author engagement with those early debugging tools.

We later also implemented a Google Chrome devtools extension (Figure 3.6) for inspecting the values of Mavo Nodes and experimenting with expressions. The extension adds a “Mavo” tab to the Element inspector of the built-in browser developer tools that displays current values, and allows one-off execution of expressions and data updates. Since property references in Mavo expressions are context sensitive, we used this inspector in our evaluation to demonstrate context sensitivity to our subjects, and later on in the study to allow them to quickly experiment with expressions in training tasks.

Later user studies used the Mavo Inspector instead, both to explore data, and to allow subjects to quickly experiment with expressions in training tasks.

3.5Discussion

In this section we discuss various issues brought up in our design and study of the Mavo language, and other potential future directions.

3.5.1Data Formatting

Currently, data formatting is automatic. The only type that supports custom formatting is the <time> element, which can optionally nest an expression for presenting the date/time to override the default:

<time property="start">[date(start)] at [time(start, "seconds")]</time>

For example, numbers are automatically formatted in a locale-aware way, with thousands separators and two decimal places. However, there are cases where authors might want to override this formatting. For example, in a recipe app, one would typically want to list ingredients with fractional quantities like “⅜ cups”, rather than “0.375 cups”. This is already possible by redirecting the number value to an attribute, and using a custom expression as the element contents:

<span property="amount" mv-attribute="data-value">[custom_format(amount)]</span>

However, this is essentially a workaround, as its closeness of mapping [76] is poor. A more direct way to specify formatting would be beneficial, e.g. an attribute dedicated to this, alongside a richer set of built-in formatters.

3.5.2Direct Manipulation of Data Schemas

Mavo’s approach of designing schmemas by designing the presentation of the data from those schemas works well because the presentation of data usually reflects its schema. If we have a collection of objects with properties, we generally expect those objects to be shown in a list, with each object’s properties presented inside the space allocated to that object. This is understandable, as the visual grouping conveys relationship to the viewer. We are simply inverting this process, arranging for the visual grouping to convey information to the underlying data later. Mavo may not be suitable for creating presentations that conflict strongly with the underlying data schema, should such presentations ever be wanted.

3.5.3Target Users

Mavo is aimed at a broad population of users. There is no hard limit to what it can do, since JavaScript can be used as an extension point. However, this is not the primary target. Our focus is increasing the power payoff for a given investment of effort/learning that is accessible to novices (Section 1.4.2). Currently, even small web applications require substantial skill and effort to build. Too often, designers of essentially static websites are forced to deploy them inside CMSs, only so that their non-technical clients can update the site content. Mavo frees designers from these CMS constraints by providing an automatic WYSIWYG content management UI for plain HTML. Plain CRUD apps only need mv-* attributes “entirely in HTML” without application logic.

For users who want more, expressions add power: lightweight computation for application logic at a conceptual cost similar to spreadsheets. More complex functions provide more power, like advanced spreadsheet ones. Our user study traced out this ease/power curve and showed that most users can work with such expressions.

Although we have focused on Mavo as a tool to support non-programmers, skilled programers can also benefit from the ability to rapidly build dynamic CRUD interfaces. Even for programmers Mavo brings some of the benefits of data typing to the construction of the interface: declaring data types enables the system to provide appropriate input and data management without demanding that the developer write special purpose code for the typed content.

3.5.4Handling Large Amounts of Data

The existing Mavo backends save all data in a single JSON file. This is convenient for use cases involving small amounts of data, and allows using any popular file hosting web service as a backend. However, making multi-user apps possible will create a pressing need for handling larger amounts of data with Mavo, even to create entire social websites

Mavo already supports displaying and editing part of the data, and already keeps track of what data has been modified, to highlight unsaved changes. Therefore, it is easy to implement incremental saving for web services that support it. Implementing a backend adapter for a cloud database service (e.g. Firebase), will also allow for fetching partial data. Such backends usually also support server-sent events, which would enable incremental updates for true bidirectionality.

3.5.5Encouraging Semantic Web Best Practices

Mavo encourages semantic web practices by providing incremental value to authors when they add semantic markup. For example, using suitable HTML elements will produce more suitable editing widgets, and using good property names is essential for an editing UI that makes sense, since they are used in a number of places in the generated editing UI: button labels, tooltips, and input placeholders to name a few.

Additionally, its storage functionality typically produces structured, machine-readable data, contributing to the Semantic Web through another avenue. In fact, if a vocab attribute is used in the markup, Mavo will produce valid JSON-LD [77].

It could be argued that because of its looser syntax (Section 3.3.3.4), data produced by Mavo app users are not useful. However, we believe that reducing the burden of creating structured data and increasing the author perceived value of doing so is the only way to encourage widespread adoption of structured data practices, and a large quantity of imperfect structured data is more useful than a small quantity of perfect data created by the few users with intrinsic motivation to contribute to the Semantic Web.

3.6Future Work

3.6.1Separate Name for Collection and Collection Items

Using plural instead of singular nouns and vice versa is a common slip of Mavo authors, since currently the same name is used for both the collection itself, and each item.

As noted in Section 3.3.3.3.1, the current syntax for defining collections consists of two attributes: mv-list and mv-list-item. While that was not the motivation for the change, it opens up the possibility of defining a separate name to refer to the collection and its items (currently an authoring mistake handled by precedence rules).

This would allow using a plural name for the collection itself, and a singular alias for the items. This would also allow referencing the collection from within each item without having to do propName.all.

3.6.2Handling Schema Mutations

Mavo’s innovation of inferring schema from HTML presentation might be its Achilles’ heel. After Mavo is used to create data, changes to the HTML may result in a mismatch between the schema of the saved data and the new schema inferred from the HTML, which could lead to data loss (or perceived data loss, where data is retained but not displayed, creating the appearane of data loss). Similar mismatches can occur when connecting Mavo apps to third-party data with a pre-existing structure that was not created by the Mavo app. Mavo does automatically handle some changes, such as:

• When properties are added, the schema is automatically extended to include them.
• When properties are removed, corresponding data is retained and saved, but not displayed. This protects a user from data loss if they stop displaying a property then bring it back later. It also enables the creation of multiple Mavo applications operating on different parts of the same dataset.
• When a singleton is rendered onto a collection, Mavo heuristically either converts the single item to a collection of one item or renders a list-valued property of the singleton, depending on their structure.
• When a collection is rendered onto a singleton (e.g. by removing the mv-list/mv-list-item attributes), the data is retained so it can be brought back later but anything after the first item is not displayed and cannot be edited or referred to in expressions.

However, if the disparity between the data schema and the inferred schema is too great, Mavo will not be able to automatically reconcile them without author intervention. Such intervention includes:

• Using mv-alias to read data from a property with a different (older) name.
• Using mv-path to “fast-forward” to a property that is now nested deeper in the schema.

More complete handling of schema changes and/or better author feedback about how they are handled are key open questions for Mavo.

3.7Conclusion

This chapter presents Mavo, a system that helps end-users convert static HTML pages to fully-fledged web applications for managing and transforming structured data. Our user studies showed that HTML authors can quickly learn use Mavo attributes to transform static mockups to CRUD applications, and, to a large extent, use Mavo expressions to perform dynamic calculations and data transformations on the existing data.

4.1Introduction

Spreadsheets introduced reactive (aka continuous evaluation [36]), side-effect free formulas to the massses, and despite the well-known limitations of spreadsheets [78], no other end-user programming environment has managed to surpass their popularity.

Despite hierarchical data naturally occurring in more than half of data structures novices organically create [27], few formula languages support hierarchical data structures. Moreover, the few that do often prioritize compatibility with spreadsheets over natural programming and general HCI principles.

Formula² (MavoScript in earlier literature [11]) is a formula language designed from scratch to support hierarchical data structures in a way that is natural for novices and reduces the amount of abstract thinking about data required.

However, if the research question was simply How can we design a natural formula language?, the answer is simple — no structured language can beat natural language at that — almost by definition. But parsing natural language is resource-intensive and error-prone, so instead we designed Formula² to answer the question What would a formula language look like if it were designed from scratch today, with the explicit goals of prioritizing usability, natural programming principles [16, 70], and making common use cases easy, while still maintaining reasonable parsing and evaluation complexity?

Indeed, while our user studies repeatedly demonstrated that Formula² was easier for novices than alternatives, it can be parsed very efficiently with a simple Pratt parser [79].

While Formula² was originally developed for Mavo, it has no dependency on any particular host language or system. It could be used on hierarchical spreadsheet-like applications, compiled to JavaScript and used by JavaScript frameworks, or even used in a standalone environment against arbitrary data (see Figure 4.1). Consistency with JavaScript or other web technologies was not a primary design consideration for its syntax, but was used as a tie-breaker for otherwise equally valid design decisions.

4.2Related Work

Since LANPAR introduced reactive formulas in 1969, and VisiCalc augmented them them in 1979 with the A1 notation and most of the features we know today [78], a lot of work has been done to improve on their design.

4.2.1Hierarchical Data in Spreadsheets

Gneiss [34, 80] was a spreadsheet system supporting hierarchical data. However, the bulk of its contributions were in the user interface. Its formula language prioritized compatibility with spreadsheets and thus, included the minimum amount of changes needed to support hierarchical data. References to data still used the obscure A1 notation, just extended it for nested data.

4.2.2Reactive Formulas Operating on Databases

SIEUFERD [32] provided a spreadsheet-like hiearchical data interface to a SQL database, and thus had to design a formula language that could handle hierarchical data. While it used readable names rather than A1 notation, compatibility with spreadsheets was a primary design goal. More importantly, the primary means of querying list-valued data was its interface for visually constructing SQL queries, and thus the formula language is purposefully restricted to scalars as a return value.

4.2.3Reactive JavaScript Frameworks

Spreadsheet formulas are not the only syntax in wide use for reactive formulas. While spreadsheet formulas are the most popular end-user programming syntax for reactive formulas, this section would not be complete without mentioning the many reactive JavaScript frameworks, such as VueJS [57], Angular [56], or Svelte [81] These frameworks typically support embedding a restricted subset of JavaScript in HTML, and detect dependencies by parsing the JavaScript code.

4.2.4Other Dataflow Languages

There are many other dataflow languages, such as LabView [82], a visual programming language for dataflow programming. However, these are typically not designed for end-users, and are often not reactive in the same sense as spreadsheet formulas. The usability issues with LabView are well-documented [76].

Mashroom [83] was a dataflow language designed for mashups, and operating on nested tables. While some of its operators were similar to Formula², and it also centered around aggregation, it required a lot more technical knowledge to use and targeted different use cases.

4.3Syntax and Core Concepts

In this section we summarize Formula²’s core concepts and syntax. A more detailed design discussion can be found in Section 4.5.

4.3.1Separation of Concerns

We define the host environment as the environment in which Formula² is embedded. This could be a spreadsheet-like application, a no-code visual app builder, a web framework, or a standalone application that allows the user to directly import data, specify Formula² expressions, optionally select the context node (see Section 4.4.1), and see the result (Figure 4.1). We have already seen one host environment, Mavo HTML.

Since Mavo is the only host environment of Formula² that we have seen so far, it may be unclear where Formula² ends and the host environment begins. Formula² is responsible for (a) parsing and compling expressions, and (b) evaluating them against an arbitrary data tree. To maximize flexibility and make it easier to adopt by a variety of host environments, monitoring dependencies and re-evaluating expressions at the appropriate times is handled by the host environment.

It could be argued that this makes Formula² simply a functional, side-effect free language, since it is the host environment that may (or may not) implement it reactively. For example, the testing host environment shown in Figure 4.1 does not implement any reactivity. However, we believe this to be an implementation detail. The language design and semantics assume a reactive implementation. The fact that it can function (and even be useful) without reactivity is a testament to its flexibility.

4.3.2Syntax and Semantics

An explicit design goal was to minimize the number of syntactic primitives that need to be learned, and instead allow as many combinations of these primitives as possible.

• Literals: Numbers, strings (enclosed in single quotes, double quotes, or none (see below)), booleans (true/false), empty values (none)
• Identifiers: Any string of letters, numbers, or underscores. $ is also allowed, but is reserved for predefined Formula² identifiers.
• Function calls are invoked with the usual syntax of functionName(arg1, arg2, ...). Commas are optional, but encouraged.
• Operators, which can be unary, binary, ternary, or n-ary.

Complex data types such as lists and groups are also constructed using functions and operators.

4.4Core Contributions

In this section, we present the core concepts that position Formula² in the landscape of reactive formula languages.

4.4.1Implicit Scoping and Aggregation

Consider the use case shown in Figure 4.1. If we were using JavaScript, the expression could look like this:

tasks.reduce((acc, task) => {
	(acc[month(task.due)] ??= []).push(task.title);
}, {})

Going beyond syntactic differences, referencing in JavaScript (and most other programming languages) is shallow: we cannot reference any task properties without getting ahold of each task object, and we could do that without iterating over the list of tasks. There are two levels of indirection to go from the data root to the data we are interested in. This approach minimizes conflicts, but at the cost of verbosity and complexity.

In natural language, scoping is a way to optionally narrow, not a prerequisite for meaning. Any known object can be referenced, and the context of the conversation is implicitly used to resolve it. This process may even involve iteration or even a recursive walk through the hierarchy of known objects, yet humans handle it effortlessly.

We can say put all dirty dishes in the dishwasher without needing to specify whether the dishes are on the table, in the sink, or in the bedroom. We can optionally scope it further, e.g. put all dirty dishes from the dinner table in the dishwasher, but the scope is optional, and the dirty dishes concept has meaning without it, just broader.

In contrast, while programming languages also have a notion of context, references are typically more restricted. Context typically simplifies references to identifiers directly within the current scope, or in ancestor scopes, but that’s about it. Getting descendant or sibling data typically requires a series of recursive mapping operations. To word the dirty dishes example using the concepts of most programming languages, we would have needed to say something like:

“Visit every room in the house and do the following. If there are any dirty dishes on the floor, put them in the dishwasher. Look for objects with horizontal surfaces, then look for dirty dishes on them. If you find any, put them in the dishwasher. Now look inside all containers larger than a plate. If you find any dirty dishes, put them in the dishwasher. If you find other large enough containers, repeat the process for them too.”

Imagine how tedious communication would be if we had to speak like this! And yet, we have accepted that this is a reasonable way to communicate with computers.

This could explain why scoping and referencing errors are so common among beginner programmers [67–69]. The need to understand scoping rules, write out lengthy namespaces, or — worse — mapping operations when multiple values are desired are all barriers to entry. To alleviate this, Formula² uses a novel scoping algorithm that will prioritize explicit references over implicit ones, but will attempt to resolve any known identifier to the most reasonable author intent, taking into account both the structure of the formula, the data schema, as well as the placement of the formula relative to the data.

In Formula², any known data identifier can be used anywhere. However, the value(s) it resolves to depends on the placement of the formula. Every evaluation of a Formula² expression can be associated with a context node in the data.

This is in contrast with scoping in most programming languages, which is either lexical (derived from the code hierarchy) or dynamic (derived from the call tree). Typically the context node would have a spatial association with the formula in the host environment, for example in Mavo it would be the closest containing property. In a spreadsheet it could be the cell the formula is in.

A given identifier can resolve to a single value or a list of values, depending on the relative locations of the context node, the data it references, and the data schema.

Conceptually, to resolve the value of an identifier, we perform a breadth-first search¹ on the data to find the shortest path(s) from the context node to properties with the given identifier. We then resolve these paths to values. If there are more than one paths, or the paths cross any arrays (even if they only contain one element), the result is a list of values.

If the property is not found at all, then ancestor nodes are searched for the property.

While this algorithm may sound complex, in practice it affords a more natural way to reference data, where references largely just work and scoping rarely needs to be explicitly considered.

4.4.2Transparent List-valued Operations

Since lists are such common return types, built-in support for list operations is essential. But Formula² goes a step beyond that, and attempts to blur the line between scalar and list operations. Its answer to “Am I dealing with a list or a scalar?” is if you don’t already know, it shouldn’t matter, in the sense that the author providing a list or a scalar communicates intent, and that intent is generally honored, but whether a function’s return value is a list or a scalar should largely be irrelevant, except when it’s a predictable consequence of said author intent.

Part of this is a natural consequence of the scoping algorithm. Since foo.bar looks at descendant properties via a breadth-first search, the result will be the same whether foo is a single object with the foo property or a list containing said object as its only element (since Formula² lists don’t include arbitrary data properties).

Nearly all Formula² operations (functions and operators) are list-aware. Operations on lists, between lists, or between scalars and lists require no special syntax or functions. They largely just work.

The algorithm used is simple. Operations (including functions) are defined in terms of their scalar equivalents.

1. When both operands are scalars, the scalar operation is applied.
2. When only one operand is scalar, the result is a list where each list item is the result of applying the operation to the scalar and the list item.
3. When both operands are lists, the operation is applied element-wise. If the lists have different lengths, null is used for missing values.

Lists returned from operations are flattened, so that all data is either scalars or lists of scalars, i.e. authors do not need to deal with lists of lists. This was chosen to limit complexity, as lists of lists are less common in naturally occurring data structures [27], and create amgibuities for many operations (e.g. what should count(list) return if list is a list of lists? The number of lists or the total number of elements?).

4.5Detailed Design Discussion

We now present the design of Formula², focusing on its syntax and semantics.

4.5.1Fault-tolerance & Flexibility

The design principle of fault tolerance (Section 1.4.4), permeates many of the design decisions of Formula²:

• Case-insensitivity: Function names and identifiers are case-insensitive, so rating, Rating, and RATING will resolve to the same result.
• Graceful references: Using the narrowing (.) operator on an empty value, does not produce an error, just a result that is also an empty value.
• Unquoted strings: Unquoted strings are allowed, but are a last resort if an identifier cannot be resolved as data.

4.5.2Special Properties

Most formulas depend on the values of other data, and should update when those values change. However, some use cases require access to context data that may change even if no data has changed, and thus should influence when a formula is recalculated, essentially adding itself to its dependencies.

For example, a formula that displays the current time in hours and minutes should update at least every minute, even if no data has changed. In spreadsheets, this is expressed as a function, NOW(). However, function calls do not update dependencies, so NOW() has the confusing behavior that it does not actually reflect current time now, but the time the row was last edited (which is a useful piece of information in its own right, but needs a name that better describes its purpose).

In Formula², such computations are expressed as special, built-in properties beginning with $. For example, the current time is expressed as $now.

Other examples of special properties are:

• $mouse, which provides the current mouse position and updates whenever the mouse moves.
• $hash, which returns the current URL hash (without the # sign) and updates when the hash changes.
• $today, which returns the current date and updates at midnight.

There are also several tree-structural special properties, which are used to navigate the data tree. For example, $parent to reference the parent of the current context node, $previous and $next to reference its siblings, $all to go from an item to its containing list, or $index to get the index of the closest list item (or any list item, when used as a property)

While the $ prefix is designed to prevent clashes with author identifiers by partitioning the namespace, we noticed in user studies (see Chapter 7) that authors often forget to use it, presumably because this is an unnatural use for the $ symbol, which in natural language is used for currency. It remains an open problem how to partition the namespace in a way that is more intuitive to authors, without making it overly verbose (as in the case of a reserved prefix, e.g. f2_now).

In line with its design principle of fault tolerance (Section 1.4.4), Formula² will attempt to resolve special property identifiers without the $ prefix, but with a lower priority than those with the $ prefix, only if no data properties exist with these names. It will also correctly resolve author identifiers that begin with $.

4.5.3Data Types

Formula² supports the following data types:

• Numbers
• Strings, enclosed in single or double quotes. Unquoted strings are also supported if they only consist of identifier-compatible characters, but have lower precedence than data references. Since the schema can be mutated at runtime, this means their meaning could change if a new property is added anywhere in the data that matches the unquoted string.
• Booleans, true and false.
• Empty values (aka null)
• Lists (aka arrays or collections). Lists can contain any data type. There is no dedicated syntactic construct for lists, instead they are defined via a list() function.
• Groups (aka objects or dictionaries), which are sequences of key-value pairs. Groups are defined using a group() function and a colon to separate keys from values.

{
	"name": "Lea",
	"age": "32",
	"hobby": [
		"Coding",
		"Design",
		"Cooking"
	]
}

group(
	name: Lea,
	age: 32,
	hobby: list(
		Coding,
		Design,
		Cooking
	)
)

The colon (:) is actually an operator that returns an object with a single key-value pair. The group function merely merges these objects. This has the very nice consequence that when there is only a single key-value pair, the group() function can be omitted.

4.5.4Operators

Operators are essentially a nicer syntax to improve readability for certain functions, and to reduce the need for balancing nested parentheses. All operators are also implemented as functions (but the contrary is not true).

To improve learnability, only widely understood operators are symbols:

• common arithmetic operators: +, -, *, /
• common comparison operators: >, <, >=, <=, = (and ==), !=,
• concatenation: &,
• the key-value pair operator: :
• the range operator: ..

All other operators are words, including logical operators (and, or, not), a modulo operator (mod), filtering (where), grouping (by / as).

These are designed to eliminate many common types of errors across novice programming and formula languages [75, 84]:

• To avoid confusion between = and ==, = and == are both used for comparison.
• It is common for + to do double duty: addition or concatenation, heuristically determined by the operands. This usually leads to errors, when the heuristics predict author intent incorrectly. Moreover, such a heuristic does not improve ergonomics since addition and concatenation are fundamentally distinct operations. In Formula², + is only used for addition, and & is used instead for concatenation. This allows + to work even when numbers are stored as strings, sparing novices from having to think about data types.
• Unlike most programming or formula languages, comparison operators are n-ary. For example, 3 < foo < 5 is perfectly legal and equivalent to (3 < foo) and (foo < 5).

4.5.5Operators That Affect Scoping

In certain cases, Formula²’s innovation of implicit aggregation becomes its Achilles’ heel. A big class of these are operations whose operands are not independent, but one operand is implicitly scoped by the other.

4.5.5.1Narrowing Operator (.)

The most trivial such case is the narrowing (or dot) operator, which is used to access properties of objects and to narrow down overly broad references.

With most operators, operands are resolved independently. For example, in the schema shown in Figure 4.1, a formula like title & " " & due would evaluate title to all task titles, due to all task due dates, and would then concatenate them element-wise separated by a space.

However, if the same approach were followed with the dot operator, it would produce nonsensical results. Imagine if given a reference like students.name we evaluated students and name separately!

Instead, students essentially adds a constraint to the resolution of name: it says “only consider name properties that are descendants of students”. Whether this constraint is implemented by fetching all students objects and then looking within them, or by resolving name independently and then filtering it, is left up to implementations.

4.5.5.2Filtering and Grouping Operators (where and by)

A less obvious example is the filtering operator (where). To enable filtering of list-valued properties, Formula² supports a where operator and a corresponding filter() function. The filter() function takes two lists and returns a new list that contains only the elements of the first list for which the corresponding element in the second list is not empty, false, or 0. Any surplus elements in the second list are ignored per existing list operation semantics (Section 4.4.2).

However, a where b is not simply syntactic sugar for filter(a, b). There is one common ambiguity that where resolves: should properties referenced in predicates be resolved by the context node of the expression as a whole, or in a different way? This is best explained with an example. Consider the schema in Section 4.7.2. Assuming a decision item context node, what would you expect the expression pro where weight > 2 to return? Presumably, you would strongly suspect it should return pro objects that have a weight property over 2.

However, if where were naïvely rewritten, it would be equivalent to filter(pro, weight > 2). Like every function, each argument is resolved separately: pro resolves to all pros of the context decision, and weight > 2 resolves to a list of booleans …for all weights — including weights for cons! To get the expected result with the filter() function, we need to use the narrowing (dot) operator and write filter(pro, pro.weight > 2).

To prevent this terribly confusing behavior, where is implemented to apply special scoping rules, to preferentially resolve properties in the second operand as descendants of the first operand.

This is not the same as naïvely prepending identifiers with a narrowing operator involving the first operand, nor as using the first operand as a context node: When we write foo.bar, if bar is not found within foo and its descendants, the return value is empty². However, in the case of foo where bar > N, if bar is not found within foo and its descendants, we do want the search to continue as normal within the context node of the formula. It could be argued that using foo as a context node and continuing the search to its ancestors is also reasonable, but in practice that is rarely desirable and has similar issues as dynamic scoping. The closest rewriting would be (foo.bar or bar)

4.5.5.3Explicit Scoping Operator (in)

In some cases, it makes sense to evaluate an identifier using a different context node than the one of the whole formula. One area where this is needed is nested aggregates, i.e. aggregates of aggregates. We use nested aggregates when in a hierarchical schema we want to compute an aggregate within descendant lists and then aggregate the aggregates at a higher level of the tree.

This may sound obscure in the abstract, but use cases that require nested aggregates are quite common in data-driven applications. For example, assuming the data schema of a restaurant review website, average number of reviews is a nested aggregate: AVERAGE of COUNT.

To support this, Formula² supports an explicit scoping operator, in, which evaluates its first operand in the context of the second operand.

The in operator has slightly different semantics around list values than other operators (Section 4.4.2): if the second operand (the scope) is a list, the first operand is evaluated for each item in the list. This is because it actually affects what the first operand resolves to, so taking its shape into account would create a logical cycle.

The scoping operator facilitates nested aggregates using a concise, natural syntax. Returning to the restaurant review example, to calculate the average number of reviews per restaurant, we would write average(count(reviews) in restaurants).

It can also facilitate complex object mapping operations, though the syntax can be a little more awkward. For example, assume we have a list of right triangles (triangle) as objects with width and height properties. We can easily calculate a list of hypotenuses with sqrt(pow(width, 2) + pow(height, 2)), but what if we want to transform it to a list of triangles with width, height, and hypoteneuse properties? This would work, but a novice would be unlikely to write it:

group(triangle, hypotenuse: sqrt(pow(width, 2) + pow(height, 2))) in triangle

Due to the potential for confusion, its precedence is very high so that authors are forced to use parentheses to scope expressions with more than one term.

4.5.6Escaping the Scoping Heuristic

In many common cases, Formula²’s scoping heuristic can save authors time and effort. However, as with many heuristics, there are cases where the heuristic would incorrectly predict author intent. Per our design principles (Section 1.4.3), inferred author intent should be overridable.

For such cases, Formula² provides several ways to disambiguate references, either by using more narrow scoping, or by providing ways to tweak reference resolution:

• If ancestor values are undesirable, a narrowing operator (.) can be used to scope references to be within a specific object.
• Special properties such as $parent, $previous, $next, $item, $this can be used to navigate the data tree of a given reference.
• If the entire list of values is desired from a formula within a list, the $all special property can be used.

4.5.7Easier Temporal Computation

We identified temporal math as a common pain point when working with spreadsheets and later validated that our hypothesis was true (see Section 7.6). Yet, temporal computation is ubiquitous in so many types of web applications, from use cases like personal tracking and project management to simple things like showing the date a post was created, or how long has passed since.

Formula² provides a variety of functions to support manipulating and formatting temporal values across various points of the ease-of-use to power spectrum.

Dates, times, and date/times are represented as ISO 8601 [85] strings. If a mathematical operator is used with at least one string operand (which cannot be parsed as a number), Fomula² will check if the string is a temporal value, and will perform a temporal calculation if so.

This allows authors to specify temporal values by simply specifying them as strings, with no additional effort to convert them to a specific data type. Since no valid ISO dates are also valid numbers³, this is a safe heuristic.

A variety of built-in functions and special properties are provided to make such operations natural to read and write.

For example date($today + 1 * day()) will return tomorrow’s date and duration($now - "2019-07-12", 2) will return how long it has been since July 12th, 2019 in a human-readable format with two terms (e.g. “3 years, 2 months”).

Temporal expressions involving special properties such as $now should be updated by the host environment at a reasonable rate, up to display’s framerate, to ensure that they are always up-to-date. While Formula² does not specify optimizations for this (e.g there is no need to update readable_date($now) at 60 fps), host environments are encouraged to do so.

The ISO 8601 format also supports timezones, and these are honored in any calculations. However, it remains important future work to provide more primitives for their handling. If no timezone is specified, the user’s local timezone is assumed.

4.5.8Internationalization & Localization

Any functions that produce human readable output (such as many temporal functions) are locale-aware: the host environment can optionally associate a locale with certain data nodes (any node without a locale inherits the locale of its parent), and output follows this locale. For example, in Mavo, the locale is derived from the closest HTML lang attrbute.

4.6Architecture

Here we summarize some architectural considerations for implementing Formula².

4.6.1Expression Compilation

4.6.1.1Grammar

Ease and efficiency of parsing was a key design consideration. The entirety of Formula²’s syntax can be parsed by a simple Pratt parser [79]. In fact, all that is needed to adapt a Pratt parser intended for JavaScript expressions⁴ to a Formula² parser is to simply modify its operators.

To generate a Pratt parser for Formula², the operator associativities and precedences are shown in Table 4.1. On a high level⁵, an EBNF grammar for Formula² could look like this:

       Expression: Operand | Compound
          Operand: Literal          | Identifier | UnaryExpression
                 | BinaryExpression | MemberExpression | TernaryExpression
                 | CallExpression   | '(' Expression ')'
    UnaryOperator: 'not' | '-' | '+'
  UnaryExpression: UnaryOperator Operand
   BinaryOperator: ['<' | '>' | '='] '='? | '!=' | '*' | '/' | '+' | '-'
                 | 'mod' |  'and' | 'or' | 'where'
 MemberExpression: [ [MemberExpression | Identifier] '.' ]+ Identifier
 BinaryExpression: Operand BinaryOperator Operand
TernaryExpression: Operand 'by' Operand [ 'as' Operand ]?
   CallExpression: [ Identifier | MemberExpression ] '(' Compound ? ')'
         Compound: Expression [ Separator [ Expression | Compound ] ]+
        Separator: ',' | ';' | whitespace
                 | any other non-alphabetic, non-numeric character
          Literal: Numeric literals (e.g., 1, -42, 3.14)
                 | Single or double-quoted string literals
       Identifier: ['$' | letter | '_'] [ letter | digit | '_' | '$' ]*

4.6.1.2AST Transforms and Desugaring

After parsing, a number of AST transforms are applied to the parsed expression. These include:

• Flattening logical operators. For example, 3 < foo < 5 will be parsed as (3 < foo) < 5, which would produce an incorrect result. Only logical operators need to be flattened. Other operators are either correctly handled via normal precedence rules, or are associative and thus flattening would make no difference.
• Rewriting operators into their equivalent functions (with further rewriting for operators that affect scoping, see Section 4.5.5).
• If the left operand of : is an identifier, it is rewritten to a string literal.

4.6.2Expression Evaluation

4.6.2.1Building a Schema

For Formula² to be able to evaluate expressions against a data tree, it first needs to walk the data tree to understand its structure and to store pointers from data objects to this stored information.

This involves:

1. Walking the data tree to build a schema (unless this is precomputed by the host environment). This is not a detailed data schema; for example it does not concern itself with data types at all. It is simply a nested data structure of all possible property paths to help resolve identifiers.
2. Linking each node to its parent, so that traversal in any direction is possible from any node.
3. Linking each object to its corresponding schema node, so that traversal in any direction is possible.

Building a schema is not merely a performance improvement: it is essential for reasonable identifier resolution. Imagine if the meaning of identifiers changed simply because a collection happened to have no items at the moment of evaluation! Of course it is also a performance improvement — performing a breadth first search on the data tree for every identifier would be prohibitively slow even for modestly sized datasets.

The host environment needs to notify Formula² when the data tree changes in a way that affects the schema so it can be rebuilt, otherwise expression evaluation could be incorrect.

{
	task: [
		{
			taskTitle: "Code furiously",
			done: true,
			tags: ["coding", "fun"],
		},
		{
			taskTitle: "Run user study",
			priority: "P2",
			tags: "science"
		},
		"Have a life?"
	]
}

{
	task: [
		{
			taskTitle: true,
			done: true,
			priority: true,
			tags: [true]
		}
	]
}

4.7Comparison with Other Languages

We now present a few examples that highlight how Formula² provides improved ergonomics over JS, spreadsheet formula languages, and SQL. Each example schema has been chosen as a representative example of a common data shape. First, a flat table. Then, a schema with divergent one-to-many relationships with the same schema. Finally, a schema with a deeper hierarchical structure.

For the comparison with JS, we will use the scoping convention of many JavaScript frameworks, that identifiers directly below the context node can be referenced directly, as well as ancestor properties, but everything else requires explicit scoping.

To compare with spreadsheets and SQL, there is a dilemma: how to best convert a hierarchical schema to a tabular one to maximize the comparison fairness?

There are three main ways:

1. Denormalization: Flatten the data structure into a single table with repeating values, akin to the result of a SQL join.
2. Normalization: Split the data structure into multiple tables, with foreign keys linking them. 2NF ([86]) is probably sufficient for this.
3. Blank cells: Use blank cells where (1) would repeat values.

The latter is what is most readable to humans and what they often naturally gravitate to, as it does not require duplication (1), yet does not involve the complexity and indirection of relations (2). Effectively it is trying to visually emulate a hierarchy, and it works — for (sighted) humans. However, it is actively discouraged [87] as it breaks any type of data processing (e.g. formulas, filtering, sorting, etc.).

Repetition is a common way, albeit tedious to manage manually. However, repetition becomes very awkward when we have divergent one-to-many relationships [32], as is the case in one of the schemas below. This only leaves us one option: normalization.

In many cases, defining additional data in the host UI (e.g. computed properties in Mavo, or additional columns in a spreadsheet) would make some of these expressions a lot simpler, but to ensure a fair comparison, we will only use the data as it is defined in the schema. Additionally, we know from Section 7.6 that it is a barrier for novices when a computation requires auxiliary data to be defined in the UI.

For tasks that involve a context node, it is mentioned. Otherwise, the context node is assumed to be the root of the data tree.

4.7.1Flat Table: The To-Do List Schema

This is a variation of the to-do app shown in Section 1.2.

{
	task: [
		{
			taskTitle: "Code furiously",
			done: true,
			priority: "P2",
			due: "2022-07-12T12:00:00Z",
		}
	]
}

4.7.1.1Simple Aggregates: Percentage of Tasks Done

This is a simple aggregation task, so it mainly highlights syntactic differences across the four languages.

count(done) / count(task)

COUNTIF(B2:B, true) / COUNTA(B2:B)

task.filter(t => t.done).length / task.length

SELECT count(done) / count(*) FROM task

4.7.1.2Grouped Aggregate: Count Tasks by Priority

Here we want a data structure that will give us priorities and the number of tasks with that priority. Aggregate functions like count() have a special behavior with the results of grouping, and return a list with the counts of all groups. We can get the group names either as (task by priority).tasktitle or simply unique(taskTitle), since it is guaranteed to produce the same results, in the same order.

The colon operator can help us combine the two into a single object:

unique(taskTitle): count(task by priority)

This is not possible to do in spreadsheets as the result of a single formula. Spreadsheets have a dedicated UI feature for this called pivot tables, although no-one tried to use them in our study (Section 9.7). The closest we can get with the regular primitives is to add a column E to the table, place the formula =COUNTIF(C2:C, C2) in E2, and drag it down.

Since UI interactions are not acceptable in this comparison, we will not consider this option. We could get an arrayformula with a list of strings like “P0: 2”, “P1: 3”, etc. like this:

=TEXTJOIN(", ", TRUE, ARRAYFORMULA(UNIQUE(C2:C7) & ": " & COUNTIF(C2:C7, UNIQUE(C2:C7))))

In JS, it could look like this:

task.map(t => t.priority).reduce((acc, p) => {
	acc[p] = (acc[p] || 0) + 1;
	return acc;
}, {})

Contrary to the other two, these tasks are SQL’s bread and butter:

SELECT priority, count(*) FROM task GROUP BY priority

4.7.1.3Temporal: Display Time Left for Each Task

This is a simple temporal computation task. More than showcasing the strengths of Formula², this task highlights a weakness of the other three languages, in a task that is very common in data-driven applications.

(taskTitle): duration($now - due)

The parentheses are needed because otherwise Formula² would create an object with a single taskTitle property (rather than evaluate taskTitle) due to the special handling of the first operand in the colon operator (see Section 4.6.1.2).

Spreadsheets have the same issue as with the previous task: to use a simple formula, we need to perform UI interactions such as dragging down a formula. Otherwise, our only recourse is unwieldy arrayformulas.

But here there is a much bigger issue at play: there is no high-level way to express an interval in a human-readable way. Spreadsheet applications often provide UI for formatting a number as a duration, but this is not available in formulas and typically produces cryptic separated values of predefined units, so a duration like “2 months, 8 days” may appear like 936:00:00. Our Lifesheets user study explores this more (see Section 9.7).

Displaying values in any even moderately human-readable way is painful. Even if we only care about differences of a day or more, we need to do low-level wrangling like:

=TRIM(SUBSTITUTE(
IF(DATEDIF(D1, NOW(), "Y") > 0, DATEDIF(D1, NOW(), "Y") & " years, ", "") &
IF(DATEDIF(D1, NOW(), "YM") > 0, DATEDIF(D1, NOW(), "YM") & " months, ", "") &
IF(DATEDIF(D1, NOW(), "MD") > 0, DATEDIF(D1, NOW(), "MD") & " days", ""),
", ", ""))

It gets even worse if we want to show time units as well, as one normally would in a task manager:

=TRIM(SUBSTITUTE(
IF(DATEDIF(D1, NOW(), "Y") > 0, DATEDIF(D1, NOW(), "Y") & " years, ", "") &
IF(DATEDIF(D1, NOW(), "YM") > 0, DATEDIF(D1, NOW(), "YM") & " months, ", "") &
IF(DATEDIF(D1, NOW(), "MD") > 0, DATEDIF(D1, NOW(), "MD") & " days, ", "") &
IF(INT(NOW() - D1) = 0, IF(HOUR(NOW() - D1) > 0, HOUR(NOW() - D1) & " hours, ", "") &
IF(MINUTE(NOW() - D1) > 0, MINUTE(NOW() - D1) & " minutes", ""), "") &
IF(HOUR(NOW() - D1 - INT(NOW() - D1)) > 0, HOUR(NOW() - D1 - INT(NOW() - D1)) & " hours, ", "") &
IF(MINUTE(NOW() - D1 - INT(NOW() - D1)) > 0, MINUTE(NOW() - D1 - INT(NOW() - D1)) & " minutes", ""),
", ", ""))

The ARRAYFORMULA version of this is left as an exercise for the reader.

JavaScript also does not provide a built-in way to format durations in a human-readable way. In practice, this is often done with a library like moment.js or date-fns, but for the sake of comparison, we will use a simple, naïve implementation:

task.map(t => {
	let start = new Date(task.due);
	let end = new Date();
	let elapsed = now - start;
	let units = [
		{ unit: 'year', ms: 365 * 24 * 60 * 60 * 1000 },
		{ unit: 'month', ms: 30 * 24 * 60 * 60 * 1000 },
		{ unit: 'day', ms: 24 * 60 * 60 * 1000 },
		{ unit: 'hour', ms: 60 * 60 * 1000 },
		{ unit: 'minute', ms: 60 * 1000 }
	];

	let rtf = new Intl.RelativeTimeFormat('en', { numeric: 'auto' });

	return units.reduce((result, { unit, ms }) => {
		const value = Math.floor(elapsed / ms);
		if (value !== 0) {
			result.push(rtf.format(-value, unit)); // Negative value to represent the past
			elapsed -= value * ms;
		}
		return result;
	}, []).join(', ');
});

Standard SQL is not well-suited for this task either, as it is not designed for temporal computations. It is possible to do it, but it is also not pretty:

SELECT taskTitle, CONCAT(
	IF(YEAR(due) - YEAR(NOW()) > 0, YEAR(due) - YEAR(NOW()) & " years, ", ""),
	IF(MONTH(due) - MONTH(NOW()) > 0, MONTH(due) - MONTH(NOW()) & " months, ", ""),
	IF(DAY(due) - DAY(NOW()) > 0, DAY(due) - DAY(NOW()) & " days, ", ""),
	IF(HOUR(due) - HOUR(NOW()) > 0, HOUR(due) - HOUR(NOW()) & " hours, ", ""),
	IF(MINUTE(due) - MINUTE(NOW()) > 0, MINUTE(due) - MINUTE(NOW()) & " minutes", "")
) FROM task

PostgreSQL however has an age() function that can make this very elegant:

SELECT taskTitle, age(due, NOW()) FROM task

4.7.2Divergent 1-N Relationships: The Decisions App Schema

This is the (Mavo-inferred) schema from the decision-making application that was used in the first lab study (Section 7.1). It’s a hierarchical schema with two levels of nesting, where each decision item contains divergent one-to-many relationships.

{
	decision: [
		{
			decision: "Should I go to the party?",
			answer: null, // calculated
			score: null, // calculated
			pro: [
				{ argument: "Fun with friends!", weight: 3 }
			],
			con: [
				{ argument: "I have tons of work", weight: 1 },
				{ argument: "I will need to buy a present", weight: 1 }
			]
		},
		{
			decision: "Should I move?",
			answer: null, // calculated
			score: null, // calculated
			pro: [
				{ argument: "Reduced rent by $500", weight: 2 }
			],
			con: [
				{ argument: "Expensive moving costs", weight: 3 }
			]
		}
	]
}

4.7.2.1Calculate the Score of Each Decision

Context node: decision.*

This highlights aggregates that span two separate branches.

sum(pro.weight - con.weight)

sum(pro.weight) - sum(con.weight)

SUMIF(pro!A2:A, A2, pro!C2:C) - SUMIF(con!A2:A, A2, con!C2:C)

pro.filter(p => p.decision_id == decision.id)
   .reduce((acc, p) => acc + p.weight, 0)
-
con.filter(c => c.decision_id == decision.id)
   .reduce((acc, c) => acc + c.weight, 0)

Since SQL does not have a context node concept, we calculate all scores:

SELECT decision.id, sum(pro.weight) - sum(con.weight)
FROM decision
LEFT JOIN pro ON decision.id = pro.did
LEFT JOIN con ON decision.id = con.did
GROUP BY decision.id

4.7.2.2Filtered Aggregate: Count of Good (Score > 0) Decisions

count(score > 0)

COUNTIF(score!A2:A, ">0")

decision.filter(s => s.score > 0).length

SELECT count(*)
FROM decision
WHERE score > 0

4.7.2.3Nested Aggregate: Average Number of Arguments per Decision

This compares nested aggregates across different languages.

average(count(pro) in decision + count(con) in decision)

average((count(pro) + count(con)) in decision)

=(SUMPRODUCT((pro!A:A=decision!A2:A3)+0) + SUMPRODUCT((con!A:A=decision!A2:A3)+0)) / COUNTA(decision!A2:A3)

decision.map(d => d.pro.length + d.con.length) / decision.length

SELECT
    d.id,
    d.decision,
    AVG(argument_count) AS average_arguments
FROM
    (SELECT
        p.did AS id,
        COUNT(p.argument) + COUNT(c.argument) AS argument_count
    FROM
        pro p
    LEFT JOIN
        con c ON p.did = c.did
    GROUP BY
        p.did) AS argument_counts
JOIN
    decision d ON d.id = argument_counts.id
GROUP BY
    d.id, d.decision;

4.7.3Deep Nesting: The Restaurant Reviews Schema

This is the schema from the restaurant review log that was used in the first lab study (Section 7.1). It’s a hierarchical schema with three levels of nesting.

{
	"restaurant": [
		{
			"picture": "https://www.toscanoboston.com/common/images/thumb-cucina.jpg",
			"url": "http://www.toscanoboston.com/beacon-hill",
			"name": "Toscano",
			"visit": [
				{
					"date": "2016-03-15",
					"title": "Date night!",
					"dish": [
						{
							"name": "Filet mignon with black truffle and foie gras",
							"dishRating": 4
						}
					]
				}
			]
		}
	]
}

4.7.3.1Nested Aggregate (2 Levels): Restaurant Rating

Context node: restaurant.*

A restaurant’s rating is the average of all visit ratings, and the visit rating is the average of all dish ratings.

average(average(dishRating) in visit)

AVERAGEIFS(
    dish!C:C,
    dish!A:A,
    IF(
        ISNUMBER(MATCH(visit!A:A,
            IF(restaurant!A:A = A2, visit!A:A, ""),
            0
        )),
        visit!A:A,
        ""
    )
)

visit.map(v => v.dishRating.reduce((acc, c) => acc + c, 0) / v.dishRating.length) / visit.length

SELECT
	r.id AS restaurant_id,
	r.name AS restaurant_name,
	AVG(vr.average_dish_rating) AS restaurant_rating
FROM restaurant r
JOIN visit v ON r.id = v.rid
JOIN
	(
		SELECT  vid, AVG(dishRating) AS average_dish_rating
		FROM dish
		GROUP BY vid
	) vr ON v.rid = vr.vid
GROUP BY
	r.id, r.name;

4.7.3.2Filtered Nested Aggregate: Count of Good Restaurants

We define a restaurant as good if its rating is above 3. This is a nested aggregate with three levels of aggregation, and a filter.

count((average(average(dishRating) in visit) in restaurant) > 3)

SUMPRODUCT(
    IF(
        MMULT(
            (visit!A:A = restaurant!A2:A100) *
            TRANSPOSE((dish!A:A = visit!A:A) * dish!C:C),
            TRANSPOSE((visit!A:A = restaurant!A2:A100) * (visit!A:A <> ""))
        ) > 3,
        1,
        0
    )
)

restaurant.map(r => r.visit.map(v => v.dishRating.reduce((acc, c) => acc + c, 0) / v.dishRating.length) / r.visit.length).filter(r => r > 3).length

WITH restaurant_ratings AS (
	SELECT
		r.id AS restaurant_id,
		AVG(vr.average_dish_rating) AS restaurant_rating
	FROM restaurant r
	JOIN visit v ON r.id = v.rid
	JOIN (
			SELECT
				vid,
				AVG(dishRating) AS average_dish_rating
			FROM dish
			GROUP BY vid
		) vr ON v.id = vr.vid
	GROUP BY r.id
)
SELECT COUNT(*) AS good_restaurant_count
FROM restaurant_ratings
WHERE restaurant_rating > 3;

4.8Discussion & Future Work

4.8.1Limitations

Formula² was designed to make the kinds of computations that are common in small-scale data-driven applications easier to express by novices. There are numerous use cases arising in general application development that are currently either awkward or impossible to express in Formula².

4.8.1.1Complex Mapping Operations

While Formula² can express simple mapping operations very easily, such as mapping objects to a descendant property or a combination of descendant properties, many of the kinds of arbitrary mapping operations that a developer can accomplish in an imperative language via a loop are not possible, or awkward to express with Formula² alone.

A large class of such use cases is augmenting objects, such as the triangle hypotenuse example in Section 4.5.5.3, which can be expressed but the formula to express them is beyond the capabilities of most novices.

4.8.1.2String Parsing

One big category is parsing tasks: producing structured data from a string of text, for example color components from a serialized color, or SVG path segment information from an SVG path string (see Section 8.2.1).

Perhaps a novice-friendly way to express simple patterns could be developed and exposed in Formula² via suitable functions, but currently this remains an open question.

4.8.2Is Flattening Always the Right Choice?

Possibly one of the most questionable design decisions of Formula² is its flattening of multi-dimensional arrays. E.g. list(1, list(2, 3), list(4, 5)) is flattened to list(1, 2, 3, 4, 5). The intent behind this was to simplify the number of distinct cases that need to be handled, since these structures did not naturally occur in Mavo apps, and since every multi-dimensional array can be expressed as an array of objects.

However, since the original design, some cases have emerged where this flattening is undesirable. The main example is grouping by nested lists. Consider a list of people (person), each of whom has a list of hobbies (just a list of strings) (hobby). Currently, person by hobby would not produce a reasonable result, since it depends on element-wise matching.

Perhaps a compromise solution could be to use a data structure that behaves like a flattened array, but is not actually flattened. Or the opposite: a flattened array that preserves metadata about the boundaries of its constituent arrays.

4.8.3Dot Notation for Function Calls

We opted for a functional syntax, as it appears to be easier for novices to understand, and may be familiar from spreadsheets. We hypothesized that for example, count(rating) is easier to understand than rating.count(). However, it does come with the drawback of authors having to manage nested parentheses, a common authoring mistake. Additionally, if some functions are available as functions and some as methods, it increases the cognitive load on the author to remember which is which.

In line with our design principle of robustness, it appears an optional dot notation could greatly increase the efficiency and safety of Formula², without compromising its learnability.

Rather than authors having to remember which functions are methods, every function would be available as a method and vice versa. arg.foo(arg1, arg2) would be equivalent to foo(arg, arg1, arg2).

This would allow authors to use the syntax that is most convenient for their use case, and feels most natural to them, without introducing additional error conditions or cognitive tax.

4.8.4Notation for Unrestricted Identifiers

We have repeatedly seen in our user studies that novices struggle with the concept of identifiers having a restricted syntax. Furthermore, imposing restrictions on identifier names makes it awkward to work with data created by others, which may not conform to these restrictions.

To prevent syntactic ambiguity, allowing unrestricted identifiers would require a different syntax to tell the parser that a series of symbols is actually an identifier, and delineate where it begins and ends.

It appears that brackets ([]) could be a good fit for this purpose, and have some precedent ([32]). However, this would require substantial changes in Mavo, which uses brackets to embed Formula² expressions in literal text.

4.8.5Quantities as a First-class Citizen

For functions returning a numerical value with no arguments, and constants, it is often more readable to be able to express multiplications with a fixed number as a number with a unit.

For example, date($today + 2 * days()) will return the date two days from now. But it might be a lot more natural if we could write date($today + 2days). Or, rather than calculating a circle’s circumference as radius * 2 * pi, it may be more readable to write radius * 2pi.

Formula² could implement this as a general language construct: a number followed by an identifier is syntactic sugar for the multiplication of the number and the value of the identifier if the identifier is a number, or the value of calling the identifier as a function with no arguments if the identifier is a function. This could be a property of specific functions, since it only makes sense for a relatively small percentage of functions. Alternatively, the number and type of quantity could be retained in an object that can be coearced to a number when needed. This could allow expressing a lot more values as first-class citizens, e.g. currencies, temperatures, measurements, etc.

4.8.6Optimizing Frequent Lookups

While performance optimizations do not affect the language itself, in practice they can have a significant impact on the user experience. Currently, the only performance optimization that the prototype implementation of Formula² employs is the use of a schema to precompute all possible property paths. Even caching the result of an expression so that expressions are not re-evaluated if the data has not changed, is relegated to the host environment.

However, given the lack of abstractions in the formulas novices write (see [88] and Section 9.8), more elaborate caching mechanisms are essential. By moving caching within Formula², individual operands can be independently cached, so that for example task by priority and count(task by priority) only needs to compute the grouping once.

Additionally, a common pattern we have observed (see Chapter 7) is for certain properties to be repeatedly used to filter the same structure, often acting essentially as implicit primary keys.

For an illustrative example, consider a collection of books and their metadata (possibly fetched from a remote API), and a reading log with a list of books read, ratings, and notes, maintained by the app user. The app author could display a dropdown of books to associate each entry with, and store the selected book ID in a bookId property.

Then, to display book metadata next to each entry, the author would have to run a filtering operation such as books where id = bookId. The problem with that is that this would iterate over the entire books collection for each entry, making lookups O(N²).

These types of tasks are the bread and butter of relational database systems: you simply declare bookId as a foreign key, and assuming book.id is a primary key, the database will take care of the rest. However, this is nontrivial for novices and end-user programmers, who struggle to think in terms of relations [51] and are goal-oriented, and therefore averse to preparatory work such as setting up schemas and indices.

However, perhaps end-users don’t need to do this work. Assuming Formula² provided a mechanism for declaring primary keys and indices, creating them could be handled by the host environment, which has a lot more information about what expressions may be evaluated and how frequently. To use Mavo as an example, a simple heuristic would be to automatically create indices for any property used in a where or by clause within a collection.

Another (not mutually exclusive) direction could be for Formula² to automatically optimize expressions based on a combination of factors such as how frequently they are evaluated, and how large the collections they operate on are.

4.8.7Short-circuit Evaluation

Most common programming languages support short-circuit evaluation for logical and conditional(ternary) operators. Later operands are only evaluated if earlier operands do not suffice to determine the value of the expression. For example, in a and b, if a is false, b is not evaluated, since we already know that the overall value must be false.

Due to its handling of list-valued operations, Formula² cannot support short-circuit evaluation, as all operands need to be examined to determine the shape of the result.

This is not an issue for its use as a reactive formula language, but becomes one once side effects are introduced (such as by Data Update Actions, see Section 6.3.4).

4.9Conclusion

In this chapter, we presented Formula², a formula language designed for end-user programming in data-driven web applications, and optimized for hierarchical schemas.

Our user studies (described in Chapter 7) have shown that Formula² is easy to learn for novices, who often did not believe that expressions they wrote could work.

While Mavo is currently the only deployed Formula² host environment, we believe its potential is much broader, and look forward to seeing additional implementations in the wild.

5.1Introduction

One of the big innovations of the Web [3] was that it unified the process of accessing a remote document and encoded all the information needed for the (previously) multi-step process of conncting to the right FTP server, navigating the remote filesystem to find the right file, downloading the resource, and opening it in the right application into a single step: loading a URL in the browser. However, when it comes to writing data to the Web, we still need to deal with complex, multi-step processes and disparity between APIs and data formats.

As a result, most systems that store user data do so either locally or (more frequently) on a single cloud service they control. Users have little to no data portability; effectively locking them into the service. Part of this is the common need for services to control user data. However, another component is simply that supporting user selection for data storage is really hard.

While reading data from remote sources is generally easy, persisting data remotely is one of the big usability cliffs of the web platform. Despite it being such a common need, implementing it requires understanding of many programming concepts which are non-trivial for programmers and entirely out of reach for beginners.

Worse, even after investing the effort to understand and use a specific authentication and storage mechanism, interfacing with a different service requires learning a completely new API.

Madata is a set of simple protocols and JavaScript APIs¹ that allows web applications to read and store data in a variety of locations, serialized in a variety of formats, all with the same unified API. Programmers can support additional services without any changes to their application code, as Madata abstracts differences between services away into a single API.

Since one of the primary goals is portability, inspired by the usability innovations of the Web, Madata introduces the concept of a storage URL. The storage location can be uniquely identified via a URL, from which Madata infers which service to use, the location of the data within the service, and how to access it. This prototypes a future where users can decide where their data is stored by simply entering a URL in the settings of the application they are using.

Most remote services require authentication, which is a complex process that requires registering an OAuth [13, 14] application and writing code to handle the authentication flow which involves a multi-step handshake. Madata simplifies this process by introducing the concept of a federated authentication provider or FedAP. A FedAP is a server that securely stores API keys for specific OAuth appliations in its supported services. Instead of requiring developers to register a new application to experiment with a new API, FEDs allow several developers to share the same OAuth application. Developers have the option to use their own API keys that are not shared, but they don’t need to.

This flow also provides user experience benefits to end-users: once they have logged in to a FED, they can log in to any app using the same FED with two clicks. Any server can become an authentication provider by implementing a simple API (Section 5.5.3).

While the Madata client library that requires programming to use, it has been designed to minimize the gulf of evaluation [89] and to maximize closeness of mapping [76] between the user’s mental model and the API. It is indicative that Mavo [11], which targets non-programmers, provides an HTML-based API which is a thin abstraction layer over Madata objects and components.

While portability in terms of storage location is the core focus of Madata, it also allows for data portability in terms of data serialization format. Portability of format is essential for data longevity, as it allows data to be migrated to new formats as old ones become obsolete. While defaulting to JSON, Madata seamlessly parses and serializes data in a variety of formats, including CSV, YAML, TOML, BiBTeX, and more.

The European Union establishes data portability as a fundamental human right [15]. By making it easier for developers to offer end-users data portability than not to, Madata prototypes a future where data portability (and the data ownership it begets) are not a rare exception, but the norm.

5.2Related Work

WebDAV [90–93] was a protocol with very similar goals to Madata. It was developed in the late 1990s as an extension to HTTP to enable users to collaboratively edit and manage files on remote web servers. Despite its promising features, WebDAV failed to achieve widespread adoption beyond certain niche domains. Some of the reasons were related to its high complexity which created performance issues, and its proneness to network effects, as it required web server support. Instead of attempting to replace existing protocols, Madata is designed to pave the cowpaths by reducting the friction of interfacing with them.

The Solid Platform [94, 95] has very similar goals as Madata. Solid is a decentralized platform for social Web applications. Like Madata, user data is managed independently of the applications that create and consume this data. User data is stored in a Web-accessible personal online datastore (or pod). Like Madata, Solid allows users to store data in many different providers, and easily switch between providers. However, the solution Solid is proposing is more heavyweight and involves higher complexity and more cognitive burden for end-users. Solid’s approach requires users to understand concepts such as Pods and Linked Data, which can be complex for non-technical users, compared to Madata’s simple URL-based approach, which imposes no requirements or demands on the data being exchanged. But most importantly, Solid requires adoption by the storage providers, thus being subject to network effects, while Madata can work with any service that provides a Web API.

Two very relevant projects that came after Madata are Scrapir [96] and Shapir [97], tackling similar goals of standardizing and democratizing access to disparate Web APIs. Scrapir [96] takes an assisted collaborative approach, with somewhat technical users adding support for new services via a GUI, so that non-technical users can then use those services. Shapir [97] also tackles similar goals of standardizing various Web APIs by mapping them to schema.org [98] entities that can then be read and written by modifying regular JavaScript objects. While there is some intersection, both of these are focused around reading and writing third-party data, while Madata is focused on reading and writing arbitrary user data.

5.3Main Concepts

5.3.1Backend Classes

A backend class (or for short, backend) tells Madata how to interface with a specific type of storage location. This is often a remote service (e.g. GitHub, Google Sheets, Dropbox), but it can be any I/O mechanism that supports hierarchical data. For example, there are backends like:

• Local for storing data locally in the browser (localStorage object)
• Element for “storing” data as another element’s content (mainly useful for debugging)
• URL for “storing” small amounts of data as parameters of the current URL.

Not all backend classes provide the same capabilities. Backend classes declare which capabilities they support (write, login, upload). Applications using Madata can then read this information and adjust their UI accordingly or communicate to the user that their selection of backend is unsuitable for the current operation.

Backends are organized in a hierarchy, with common patterns implemented in base classes. For example, the Google Sheets backend and the Google Drive backend both share the same parent backend, which defines authentication for many Google™ services (Figure 5.2).

5.3.2Backend Objects

A backend instance encapsulates a specific storage location within a specific backend class. For example, a specific sheet in a Google Sheets spreadsheet, a specific file on Dropbox, or a specific key in the browser’s localStorage.

In some cases the boundaries of what should be an object in a hierarchical data structure vs a separate backend object can be blurry. For example, should a spreadsheet backend object represent the entire spreadsheet, by returning an object with keys for every sheet, or a single sheet? The current Google Sheets backend has opted for the latter, as many use cases only involve a single sheet, and thus having to deal with an extra level of nesting would be cumbersome. However, both options are defensible.

5.3.3Storage URL

As a design principle, it should be possible to identify storage locations by specifying a URL. While a URL should unambiguously identify the storage location, the same storage location may be described by multiple URLs. The URL should be either easy to compose, or easy to obtain from the service itself, and ideally both.

For example, one of the supported URLs for GitHub is the URL shown in the browser when viewing a file on GitHub. Or, the URL for a Dropbox file is the URL obtained when using its Share file UI feature.

Storage URLs are merely a portable way to represent the information needed to access a storage location. They do not need to be URLs browsers can natively load or actually resolve to the resource, altough both of these are desirable properties, as they assist with debugging.

However, in some cases, especially with non-typical backend classes, no HTTP URL pattern is a good fit. A last resort is using a custom protocol. For example, to save data in the browser’s local storage using mykey as the key, the Madata URL is local:mykey.

5.3.3.1Test URLs & Known URLs

URLs are defined as URL patterns [99], a standardized syntax for concisely specifying a set of URLs (for an example, see Section 5.5.1).

Each backend defines a set of zero or more test URLs and zero or more known URLs. Test URLs are those that are used to test if a given URL should resolve to a given backend. They need to be more precise, to avoid false positives. Known URLs are those that the backend knows how to process, but should not necessarily cause it to be selected.

Madata stores a list of all backends topologically sorted by their inheritance. To resolve a given storage URL to a backend, Madata iterates over the list of backends, trying their test URLs in order, until one matches. If none match, it falls back to the default backend, whose only capability is reading data from URLs.

These URL patterns perform double duty: their named groups are used to extract the necessary information from the URL, so that a second parsing step (which may be beyond the capabilities of many novice programmers) is not necessary.

5.3.4Federated Authentication Provider (FedAP)

There are various authentication protocols in use today to facilitate secure access to third-party APIs. These authentication protocols are designed to be very secure, but they are also complex and require a lot of boilerplate code to use.

As an illustrative example, consider OAuth 2.0 [14], one of the most popular authentication protocols today. For a developer to use OAuth 2.0, they usually need to:

1. Register an application with the service they want to access, by describing what they intend to do and obtain a secret API key.
2. Obtain a server with the capability to run server-side code. Register a domain name, then provide a “calback URL” to the service they want to access.
3. From their client-side application, open a popup to a certain URL, so users can authenticate with the third-party service.
4. The popup asks the user to log in to the third-party service and authorize the application to access their data. Then, the popup redirects to the callback URL, with a temporary code in the URL.
5. The server-side code residing at the callback URL sends a POST request to a special URL at the third-party service (e.g. https://github.com/login/oauth/access_token for GitHub), providing the temporary code as well as its secret API key (which shouldn’t be shared).
6. If everything went well, the response should (finally!) contain an access token. Extract that access token.
7. Now communicate the access token back to the client-side application, so it can use it to access the third-party service.

OAuth does support an easier implicit grant flow, but it is considered less secure and not supported by all services. Thus, even if the rest of the API is CORS-enabled [100], the authentication handshake often requires server-side code.

We introduce the concept of a Federated Authentication Provider (FedAP), to abstract all this complexity away into a single string: The FedAP’s domain name.

FedAPs are servers that store API keys for specific OAuth applications in their supported services, take care of the authentication handshake, and communicate the resulting access token to Madata, all with no involvement from the application developer. Developers can change their authentication provider (from the default auth.madata.dev) by simply setting a static property on Backend objects. For example, to be able to take advantage of existing logins to Mavo applications, one would need to use the Mavo authentication provider:

Backend.authProvider = "https://auth.mavo.io";

Or, they could specify multiple, for redundancy:

Backend.authProvider = ["https://auth.mavo.io", "https://auth.madata.dev"];

At the end of the authentication handshake, the FedAP presents the user with a confirmation dialog (Figure 5.4). This step is essential for preventing malicious use. Without a confirmation, a malicious application could trick users into visiting the page, and then would get unfettered access to the user’s data on all services they have used the FedAP to authenticate with.

If the user confirms, the FedAP then communicates the access token to Madata, and no further server interaction with the FedAP is necessary. Since the FedAP is only involved in the authentication handshake, this is generally a very low resource operation. As one data point, the author has used a very early precursor of this approach on a website [101] that served 50-100k users per month for five years (2013-2018) with no issues.

FedAPs can be used independently of Madata. When using a FedAP (without Madata), the above process looks like this:

3. From their client-side application, open a popup to a certain URL, so users can authenticate with the third-party service.
4. The popup asks the user to log in to the third-party service and authorize the application to access their data. Then, the popup redirects to the FedAP callback URL.
5. The FedAP takes care of the rest (steps 4-7 above), and communicates the access token back to the originating application by using the Window Cross-Messaging API window.postMessage() to send the token back to the originating application…

When using a FedAP with Madata, the process is even simpler:

3. From their client-side application, the developer calls backend.login(), which takes care of the rest.
4. The developer can await the result, which will be the user information (if the login was successful) or just listen to the login event.

FedAPs follow introspection; visiting a FedAP’s root domain displays the list of services it supports (Figure 5.5). The same information can be obtained programmatically via /services.json which should be CORS-enabled.

5.4Usage Examples

To create a backend object for a specific storage location, all that is needed is to call Backend.from() with the storage URL as the sole parameter, for example:

// Import Madata and all supported backends and formats
import Backend from "https://madata.dev/src/index.js";

let backend = Backend.from("https://github.com/leaverou");

From that point onwards, common data operations are a single function call away. We provide a few examples below.

5.4.1Authentication

Creating a backend will automatically log in a previously logged in user without showing a login prompt (passively).

To show a login prompt when the user is not logged in, we can call backend.login(). Backend objects emit login and logout events so that the UI can be updated accordingly.

Assume we have the following HTML:

<button id="login_button">Login</button>
<button id="logout_button" hidden>Logout</button>
<img id="avatar" alt="">
<div id="username"></div>

All the JavaScript we need to write to make it work is this:

globalThis.backend = Backend.from("https://github.com/leaverou/repo/data.json");

backend.addEventListener("login", evt => {
	login_button.hidden = true;
	username.textContent = backend.user.username;
	avatar.src = backend.user.avatar;
});

backend.addEventListener("logout", evt => {
	logout_button.hidden = true;
	username.textContent = "";
	avatar.src = "";
});

login_button.onclick = event => backend.login();
logout_button.onclick = event => backend.logout();

Note that while this is may appear like a nontrivial amount of code, it is nearly all UI (DOM) code: setting up events, updating the UI, and handling button clicks. The actual data interaction with the data layer has been reduced to a single line of code for each operation.

5.4.2Editing and Reading Data

Now suppose we want to show a number of upvotes for the current page, and allow any webpage visitor to see the same number, and any logged in user to add one or more votes.

It only takes a few additional lines of code:

<div id="upvote_count">0</div>
<button id="upvote_button">👍🏼</button>

let upvotes = await backend.load();

if (upvotes > 0) {
	upvote_count.textContent = upvotes;
}

upvote_button.onclick = event => {
	upvote_count.textContent = ++upvotes;
	backend.store(upvotes);
};

5.4.3Uploading Files

Now assuming we have a simple HTML file upload UI with an image to display the uploaded file:

<input type="file" id="uploader" accept="image/*">
<button id="upload_button">Upload</button>
<img id="uploaded_image">

This is all the JavaScript needed to make it work:

upload_button.onclick = async event => {
	let file = uploader.files[0];

	if (file) {
		let url = await backend.upload(file, {path: "images/"});
		uploaded_image.src = url;
	}
};

These 6 lines of code take care of uploading the file to the right location, fetching a URL that can be used to display it, and updating the image element with it.

5.5Extensibility

For a paradigm like Madata to be successful, extensibility is key, not simply a nice-to-have. Like Mavo, the prototype implementation of Madata supports arbitrary extension points via hooks, but here we focus on the three core extensibility points: adding a new backend, adding a new format, and adding a new authentication provider.

5.5.1New Backends

While adding support for a new backend requires writing JS, this does not mean that superfluous complexity is acceptable. Madata follows a class hierarchy where common patterns are implemented on base classes, so authors need only specify the backend-specific details, such as the specific authentication URLs, storage location URLs, or API calls needed. The less divergent the backend is to existing standards and patterns, the more declarative the code will be.

As an illustrative example, here is the full code² to add support for GitLab, a popular code hosting service,

import { OAuthBackend } from "madata";

export default class Gitlab extends OAuthBackend {
	static capabilities = { auth: true, put: true, upload: true };
	static defaultPermissions = { read: true };
	static fileBased = true;
	static apiDomain = "https://gitlab.com/api/v4/";
	static oAuth = "https://gitlab.com/oauth/authorize";

	static urls = [
		"http{s}?://gitlab.com/:id(.+)/-/blob/:branch/:path(.+)",
	];

	static api = {
		user: {
			get: "user",
			fields: {
				username: "username",
				name: ["name", "username"],
				avatar: "avatar_url",
				url: "web_url",
			},
		},
		file: {
			get: ref => `projects/${ ref.id }/repository/files/${ ref.path }?ref=${ ref.branch }`,
			put: ref => `projects/${ ref.id }/repository/files/${ encodeURIComponent(ref.path) }`,
		},
	};

	async put (data, {ref = this.ref} = {}) {
		return super.put(data, {
			branch: ref.branch,
			content: this.stringify(data, {ref}),
			commit_message: this.constructor.phrase("updated_file", ref.path),
		}, "PUT");
	}
}

The urls field specifies the URL patterns that should be recognized as GitLab URLs when Backend.from() is called. While not shown here, there is also a knownUrls field that specifies additional URL patterns that are recognized, but do not participate in the URL matching algorithm.

5.5.2New Formats

Adding support for a new format entails specifying which file extensions and MIME types [102] should automatically be recognized as the format, plus defining two methods: parse(string, options) and serialize(string, options), which are usually wrappers around a library that does the actual parsing and serialization.

For example, here is the complete code to add support for TOML [103], a generic configuration format language:

import { Format } from "madata";
import toml from "smol-toml";

export default class TOML extends Format {
	static extensions = ["toml"];
	static mimeTypes = ["application/toml"];
	static parse = toml.parse;
	static stringify = toml.stringify;
}

Note that because the format’s parse and stringify methods simply pass their arguments directly to the library, we did not even need to implement these two methods, we simply assigned them.

5.5.3New Authentication Providers

By definition, creating a new authentication provider requires a server. However, all it requires is copying the code (forking) of a template repository and deploying it to a server. Then, the server administrator would need to register OAuth applications with the services they want to support, and add their API keys to the server’s secret configuration file (.secret.json), and their public metadata to a public configuration file (services.json).

5.6Discussion

5.6.1Security and Threat Model

The Madata ecosystem has many parties: the developer, the user, other users, the FedAP, the web site hosting the Madata-using app. An important question is, how much does each entity have to trust other entities? And what power does a malicious version of each party have?

As described above, a mailicous website that tricks users into visiting the page, will not be able to access any user data, since the user will not confirm the authentication. However, a malicious website that tricks users into authenticating (such as a phishing attempt), can do a lot more damage. While this is a risk, it is not unique to Madata; it is a security risk on par with the implicit (client-side) OAuth grant flow — Madata simplicy extends it to the explicit (server-side) OAuth grant flow.

FedAPs need to be chosen carefully, as they require a high level of trust. A malicious FedAP could do a lot of damage, as it can access all user data on all services the user has used it to authenticate with. FedAPs do not need to store any user data (although nothing prevents them from doing so); once the access token is communicated to the client application, the FedAP’s job is done. This means that even if a FedAP is compromised, the damage is limited to the access tokens of users who authenticate while it is compromised. Passive authentication (where the user has already previously logged in) is not affected, since the FedAP is not involved in that process.

Some of the risks could be mitigated by allowing FedAPs to register multiple OAuth applications for the same service, and distrubuting them to Madata-using applications based on a one-way hash of their URLs, so that users who have previously authenticated with one Madata application do not need to authorize more OAuth apps, but other Madata applications using the same FedAP do not need to share the same API keys with all other applications using the same service on the same FedAP.

OAuth does provide a mechanism of scopes³ so that each OAuth application does not get unfettered access to user data. Using this mechanism here is tricky, because the scopes are shared across all Madata apps using the same service on the same FedAP, and thus currently Madata requsts very broad scopes when authenticating. However, perhaps the FedAP could start conservatively, and expand scopes as needed when users authenticate with a new Madata app.

It should be noted that Madata does not require use of FedAPs. For increased security, app developers can use their own authentication server that handles communicating the access token to their application, and still take advantage of Madata’s unified API for data access. However, this requires more work, as the OAuth handshake needs to be implemented on the server side. Perhaps Madata could make this easier by allowing users to register their own OAuth applications and securely store their API keys with the FedAP, which would take care of authenticating only a whitelist of app URLs with these API keys. This would still require trust in the FedAP, but it would prevent Madata applications from being prone to phishing attacks.

Another avenue is user empowerment. Many third-party services provide GUI for users to obtain access tokens directly with the service (see Figure 5.6 for an example). A planned improvement is for Madata to provide an optional way for more technical, privacy-conscious users to directly enter an access token, so that they can set their own scopes and access limits, and revoke access at any time. It could even make this easier by storing the link to the relevant settings page for each service in the FedAP’s metadata.

5.6.2Accountability

Madata facilitates simplicity and ease of use, but that does not come for free. Some tradeoffs were described in the previous section. Another potential tradeoff is accountability.

In the traditional OAuth model, application developers register their own OAuth applications to procure API keys, and are thus accountable for their actions. With Madata FedAPs, they do not need to register anything — they simply start using the service. FedAPs do have access to the app’s requesting URL and thus the ability to block bad actors, but this is after the fact, since there is no review step involved.

However, it is questionable whether the review step in OAuth is effective at preventing bad actors, or whether it simply adds friction to the development experience for little benefit.

5.6.3Lowering the Floor with Web Components

Madata is purely about the data layer. It does not construct any UI, which is left up to the application developer. While this makes it more flexible, it can also make it tedious to use, as a lot of interactions with the data layer are repetitive.

Mavo HTML is certainly a solution to this problem, but it is not a perfect one. While Mavo provides a very high abstraction level, the loss of control can be frustrating for programmers. To bridge this gap, Madata implements a set of Web Components⁴ that encapsulate specific UI interactions such as authentication, or autosave with throttling.

For example, a <madata-auth> custom HTML element can be used to display user information and authentication controls with just a single line of HTML:

<madata-auth src="https://github.com/leaverou/somerepo"></madata-auth>

5.6.4More Declarative Syntax

When Madata begun, as a Mavo component, it was a lot more imperative. To support a new backend, authors had to implement low-level get(), put(), login(), upload(), getUser(), etc. methods. While it did provide abstractions that made it palatable to implement these methods, it was still a lot more boilerplate. Over time, as more backends were added, common patterns emerged, and their code was abstracted away into base classes that only set static class fields as inputs.

It is an open question how far this process can go. Could we reach a point where adding a new backend can be done entirely by specifying metadata, without requiring any imperative code? And if so, would that bring it within reach of non-programmers?

5.6.5Dynamic, Decentralized Backends

The discerning reader will have noticed that the promise of storage URLs is not yet fully realized. For Madata to support a given service, it needs to know about it in advance, and for a storage backend to exist and have been imported for that particular service. For true decentralization, there should be a standard protocol to enable services to declare all the necessary information that Madata needs to know, so that the necessary backend can be generated on the fly.

There is already such a mechanism: well-known URIs, defined in IETF RFC 8615 [104]. A well-known URI is a URI [RFC3986] whose path component begins with the characters /.well-known/. While the registry of well-known URIs is maintained by IANA⁵, nothing prevents services from using well known URIs that are not registered, and those in widespread use would likely later become standardized. One could imagine a URL like /.well-known/madata or /.well-known/madata.json that would contain all the necessary information for Madata to interface with the service.

Additionally, authentication providers could also provide the necessary code or metadata for Madata to interface with a new service.

This is not without new risks. When allowed backends are imported by the programmer, they are in control of what code runs in their application. If backends can be generated on the fly, this control is lost. And if URLs are also user-supplied, an adversarial user could easily host malicious code on their own server, and then use a URL from said server.

The main way to mitigate this risk is to turn code exchange into data exchange. This could happen either at the point of interfacing with the backend, by only allowing entirely declarative backends from third-party services (see previous section), or at the point of data I/O, by running all untrusted third-party code in a sandboxed environment, where it can only access its inputs, and nothing else. Then, the sandbox communicates with the main application by exchanging plain data, which is generally safe.

Of these, declarativeness is a preferable approach; not only due to the benefits of declarative languages in general, discussed in previous chapters, but also because it is less likely to negatively affect ergonomics.

5.7Conclusion

In this chapter, we presented Madata, a set of protocols and client-side APIs, designed to facilitate data ownership by democratizing data access. By unifying the processes of reading and writing data across diverse storage services and formats through a single API, Madata addresses not only one of the biggest usability cliffs in modern web development, but also suggests a model that could facilitate data portability more broadly.

<body mv-app="todo" mv-storage="https://www.dropbox.com/…/todo.json">
	<h1>My tasks</h1> <p>[count(done)] done, [count(task)] total
	<ul>
		<li property="task" mv-multiple>
			<input type="checkbox" property="done" />
			<span property="taskTitle">Do stuff</span>
		</li>
	</ul>
	<button mv-action="delete(task where done)">Clear Completed</button>
</body>

6.1Introduction

Many systems and languages exist for assisting novice programmers to manage information, and/or create CRUD applications for this purpose. They range from the well known commercial spreadsheet systems to more complex application builders [54, 80] or simplified declarative languages [11, 40].

These usually generate an editing interface for elementary data manipulations (editing a data unit, inserting items, deleting items) and a mechanism for lightweight reactive data computation. As an example, in spreadsheets the editing interface is the grid itself, and the computation is the spreadsheet formula.

These tools typically offer only direct editing of specific data items by the end-user. Affordances may also be provided for aggregating certain kinds of commonly needed mass modifications. A few examples of these would be selecting multiple items for deletion or move, adding multiple rows or columns, or the spreadsheet fill handle. However, the set of potential data mutations is infinite, and it is not practical to predefine controls for every possible case. For more complex automation of data edits, users are typically directed to scripting or SQL queries. Learning a scripting language is almost as hard as learning a programming language, and SQL queries quickly become complicated when nested schemas are involved, which are represented by multiple tables and foreign keys [51].

Mavo [11] is an HTML language extension for defining CRUD Web applications by annotating a static HTML mockup to separate UI from data. A property attribute indicates that an element is data, and an mv-multiple attribute makes it repeatable i.e. turns it into a collection. Based on this markup, Mavo generates a suitable editing interface. Mavo stores its data locally or on a cloud service.

Many applications benefit from presenting values computed from their data, so Mavo implements a reactive expression language called Formula², similar to what can be found in spreadsheets, for lightweight computation. Expressions can be placed anywhere in the HTML and are denoted by square brackets ([]) or certain attributes(mv-if, mv-value etc). An expression can reference properties, with its evaluation depending on the location of the expression relative to the referenced properties in the data tree. Referencing a multi-valued property on or inside a collection item resolves to its local value on that item, whereas referencing that property name outside the collection resolves to all values. All operators and most functions can be used with both single values and lists. This makes many common expressions concise.

Our previous work [11] provided evidence that Mavo empowered users with no programming experience to author fully functional CRUD applications. But Mavo offered only direct editing of individual data items, while many applications call for richer, programmatic modification of large collections of data simultaneously. For example, while a simple To-Do list could be easily implemented with a few lines of HTML+Mavo, clearing all completed items was not possible.

Mavo did offer controls for deleting individual items, but no way to specify such actions that programmatically delete certain items.

6.1.1Our Contribution

In this work, we extended Mavo with a new HTML attribute, mv-action, for specifying programmatic data updates. Its value describes the data mutation as an expression, and it is placed on the element that will trigger the action by clicking. The expression leverages Formula²’s existing expression syntax as much as possible, but adds functions that modify the data. Formula²’s filtering operator, as well as its data specification syntax (group() and list()) were also originally defined to facilitate actions. A short example implementing bulk deletion can be seen in Figure 6.1.

Given the small user bases of most research systems, there is little data on any kind of user requests. However, when it comes to spreadsheets, there is a large volume of user questions indicating a clear need for a way to programmatically manipulate data. It is indicative that searching for button to change value excel, which is considerably more specific, both in terms of task and in terms of system, yields at least 400 relevant questions on stackoverflow.com alone!¹. Most askers did not mention their exact use case, but the abstract task they were trying to accomplish, which was primarily being able to press a button to change the content of one or more cells.

Our hypotheses are that (a) the set of primitives we have chosen is expressive enough to meaningfully broaden the class of data management applications that can be created without undue complexity, and (b) novice web authors can easily learn to specify programmatic data mutations. To examine the first hypothesis, we list a number of case studies of common interactions and how they would be expressed with our data update syntax. To examine the second hypothesis, we conducted a user study with 20 novice web developers writing a variety of data mutations, using first their own imagined syntax and then ours (Section 7.2).

We found that the majority of users were easily able to learn and apply these data mutation expressions with 90% of our participants getting two thirds of the questions right on first try, with no iteration. We even found that in many cases, our syntax was very close to their imagined syntax.

Although the presentation of information is an important part of Mavo, our work here focuses on and extends the computational power of Formula². We believe our work suggests more broadly a way to increase the power of such functional reactive programming environments, including spreadsheets, for novice users.

6.1.2End-User Reactive Programming

Although the presentation of information is an important part of Mavo, our work here focuses on and extends the computational power of Formula², an expression language with power similar to spreadsheet functions. And while we study Mavo, we believe our work has broader implications for such functional reactive programming languages.

As in spreadsheets, all Formula² expresions are reactive, updating immediately if their arguments change. But Mavo diverges from spreadsheets in two potentially useful ways. First, instead of referring to arguments through a grid-based coordinate system, Formula² uses the names defined in the data (through the property attribute in Mavo HTML).

Unlike the table model of spreadsheets, Mavo provides a hierarchical data model of objects containing properties and other objects, as well as collections. This data model permits storage and reference of more complex objects than spreadsheets. Nesting offers a limited amount of the power that database joins do, to connect relationships between multiple tables. The nested presentation structure of HTML makes it natural and easy for authors to define these nested data models by the way they look on the web page.

One of the primary reasons we chose to extend Mavo with this functionality is Formula²’s property reference mechanism (Section 4.4.1): Every property can be referenced from anywhere and the value depends on the location of the expression relative to the property in the data tree. Referencing a multi-valued property on or inside a collection item resolves to its local value on that item, whereas referencing that property name outside the collection resolves to all values. All operators and most functions can be used with both single values and lists. This makes many common expressions concise, a feature that extends to our mutations as well.

6.2Related Work

Our work extends the Mavo language [11]. A full discussion of related work can be found there. In summary, many platforms and systems have been developed over the past few decades to help web authors build web applications, many of which are presented in Chapter 2. Some of these tools target web developers with limited programming and database knowledge [44, 52], allowing them to make programmatic changes to the data using SQL queries.

Others were developed for novice web designers who are interested in rapid development of web applications [27, 55, 105], with spreadsheets or a variation of spreadsheets as a back-end, but with limited or no mechanism to make data updates programmatically. Two lines of prior work are especially relevant, one on creating systems for graphically designing database schemas as well as building web page content, depending on SQL queries to make automated data updates (Section 2.3), and the other on developing WYSIWYG tools for creating web pages with spreadsheets as the back-end, with limited data update capabilities.

6.3Mavo Data Update Language

6.3.1The mv-action HTML Attribute

Data updates are specified via an mv-action HTML attribute, which can be placed on any element. Its value is an expression that describes the action that will be performed. The action is triggered by clicking (or focusing and pressing spacebar, for keyboard accessibility), as that appears to be the most common way to invoke an action on most types of GUIs.

6.3.2Data Mutation Functions

We extended Formula² with four data modification functions (brackets indicate optional arguments):

• set(reference, value)
• delete(ref1, [, ref2 [, ref3, ...])
• add(collection [, data] [, position])
• move(from, to)

The first three are analogous to the SQL primitives UPDATE, INSERT, DELETE, whereas the latter is a composite mutation (delete, then add). Per [106], we used simple English words that have a largely unambiguous meaning. These functions are only available in expressions specified with the mv-action attribute. Regular Mavo expressions remain side-effect free.

We kept these functions minimal, to delegate selection and filtering logic to Mavo expressions. This maximizes the amount of computation specified in a reactive fashion, which is easier for novices to work with [107, 108].

6.3.2.1Changing Values with set(ref, newValue)

The set() function only accepts two arguments: what to set, and the new value(s). However, both arguments can be multi-valued, in which case they are applied pairwise, in the same way as Mavo operators, i.e. if the lengths are different, the operation is only applied to the corresponding items, and the extra items are just returned with no modification.

6.3.2.2Deleting Items with delete(ref1, ref2, ...)

The delete() function deletes all collection items passed to it via one or more parameters.

6.3.2.3Adding Items with add(ref [, data] [, position])

The add() function adds one or more new items, optionally pre-filled with data (if the data parameter is used) and optionally at a specific position in the collection (if the position parameter is used). The first parameter can be a reference to a collection, or a collection item. In the latter case, the target defaults to the collection containing the referenced item, at the position of that item. This for example makes it easy to replicate Mavo’s own “Duplicate item” behavior by invoking add(name, name).

6.3.2.4Moving Items with move(from [, to] [, position])

This function moves items to a different position within the same collection or to a different collection. Originally we did not plan to provide a move() function, since it can be simulated with a delete() and then add(). However, delete() displays UI for undoing the deletion which would be jarring in the case of moving items. Furthermore, even if that were not a problem, a separate move() function provides semantic clarity in the author’s code.

6.3.2.5Implementing References to Mavo Nodes

Previously, there was no way to infer which Mavo node had produced which data from within a function. The data passed to functions was essentially a combination of plain JavaScript objects, arrays, and primitives. This had the advantage that authors who were comfortable with a little JavaScript could easily extend Mavo by writing their own functions in plain JS. However, with data actions, it is essential to be able to trace values back to the Mavo node that produced them, so that the UI can be updated accordingly.

We added a non-enumerable Symbol² property to objects that correspond to Mavo Node data. This way, these node references do not interfere with normal expressions, but Mavo data mutations can still retrieve the nodes referenced and manipulate them. We also needed to convert all expression data to objects instead of primitives so they could have properties attached to them and used a special Null object for empty values, instead of the JavaScript null value (which also could not have properties).

Furthermore, we had to modify the implementation of several Formula² functions so that they do not inadvertently break these reverences by creating new data.

6.3.3Multiple Sequentially Executed Function Calls per Formula

<div property="diceHistory" mv-multiple
     class="dice-[diceHistory]"></div>
<div property="dice" class="dice-[dice]">6</div>
<button mv-action="
	add(diceHistory, dice),
	set(dice, random(1,6))">Roll dice</button>

Update actions often consist of multiple elementary updates, executed sequentially. The Dice Roller in Figure 6.2 demonstrates an example (add(), then set()). To facilitate this, we extended Formula² to support multiple function calls in the same expression. These can be separated by commas, semicolons, whitespace, or even nothing at all.

6.3.4Using if() with Data Updates

Formula² provides an if() function that works similarly to the IF() function in spreadsheets, except it can also be applied element-wise if one or more of the arguments are arrays. In early pilots of the user study, we realized that subjects may try to use if() to filter the target of a data update. For example, in the data update “Rename every person older than 40 to Chris”, it would be natural to use an expression like if(age > 40, set(name, ’Chris’)). However, given the way Formula² works, this would not filter the target of the update (name)), but merely the result of the set() function, a behavior which – while consistent with how functions work – is highly unlikely to meet author expectations.

To enable such expressions to produce the expected result, we defined a new set of data mutation functions: setif(), addif(), deleteif(), moveif(), whose first argument is a condition, and otherwise work the same way as their aforementioned counterparts. Then, we rewrite data update calls inside if() expressions to use these functions instead.

6.3.5Multiple Function Calls per Expression

Since regular Formula² expressions have no side effects and only produce one output, there was no way to specify more than one sequential function call (and no reason to do so). However, with data mutations, there may sometimes be a need to perform more than one update as part of the same action. Figure 6.2 demonstrates an example of this. To enable this, we modified Formula² to allow multiple function calls, which can be separated by commas, semicolons, whitespace, or even nothing at all.

6.3.5.1Implementation

We noticed that Formula²’s expression parser already parsed adjacent function calls as AST nodes of type “Compound” but serializes such tokens to JavaScript with no separator, which would result in an error when the JavaScript code produced is executed. Therefore, to enable multiple function calls we modified Formula²’s serialization to serialize Compound tokens into comma-separated values, as commas are an existing JavaScript operator that merely returns the last value.

Since the same Compound token was produced regardless of whether the function calls were separated by commas, semicolons, whitespace, or even nothing at all, providing this kind of flexibility essentially came for free. Another fortunate side effect of this modification was that this also enabled function arguments that are separated by whitespace.

6.4Example Use Cases

Part of our argument is that Data Update Actions have sufficient power to easily specify a broad range of programmatic data manipulations in applications. To support that argument, we outline a few common interactions below. None of these can be implemented with the original Mavo alone. In each case, we show the source code, primarily HTML and original Mavo syntax; and we highlight the code leveraging data actions.

We first demonstrate how a few lines of Mavo can be used to define a number of frequently used general-purpose UI widgets, then present a few specific applications such as the ubiquitous shopping cart.

6.4.1Generalizing Existing Mavo Data Updates

All Mavo data update controls except drag and drop can be expressed concisely as data actions, which facilitates UI customization. The following examples assume the updates are modifying a collection with property="item". Note that index (or $index) is a built-in Mavo variable that resolves to the index of the closest collection item, starting from 0.