There is a multitude of systems that assist novice web developers and end-users with building dynamic, data-backed web applications, including research and commercial tools. This section provides an overview of the current landscape.
In recent years, social media platforms for publishing web content have become a popular way for end-users to publish their content for free, without having to deal with any of the technical challenges of publishing a website they own.
Examples include Facebook Pages, blogging platforms (e.g. Medium, WordPress, Tumblr, etc.), or profiles on media-rich social platforms (e.g. Instagram, Flickr, TikTok, YouTube).
Out of all methods of publishing content on the web, this is certainly the one with the lowest threshold, which explains its popularity.
However, it comes with severe drawbacks and limitations on the type of content that can be published, what can be done with it, and how it can be displayed. These are typically designed around the most common, most generic use cases (e.g. a blog or a gallery of photos), which imposes severe limitations on the type of content that can be published, what can be done with it, and how it can be displayed. Because they are still part of the social platform that hosts them and need to maintain a consistent brand identity, they are usually limited in terms of personalization and customization. Custom functionality is typically not possible, and the platform may change or remove features at any time. Data is owned by the platform, and portability is hard or impossible. As a result, if the social media platform shuts down, content is (effectively) lost.
Visual website builders like Wix and Squarespace have revolutionized web development by enabling novices to create and manage websites that look professionally designed through direct manipulation interfaces, making it feasible for small businesses, freelancers, and individuals to establish an online presence quickly and affordably. Being designed as creative tools, these afford much better customization than the social media solutions discussed in the previous section, as the central focus is the creative artifact produced, not the connections between users.
However their how threshold typically also comes with a relatively low ceiling, or the bifurcation described in Section 1.4.2 of having a low threshold and a high ceiling by combining a low-threshold/low-ceiling solution with a high-threshold/high-ceiling solution. These platforms rely heavily on pre-designed templates that dictate much of the site’s layout and visual appearance. While these templates are often polished and professional, they are also quite rigid. Users can make changes within predefined sections and elements, but the overall structure is usually fixed. Advanced functionality is provided via predefined plugins, widgets, and integrations, which can be added to the site with a few clicks, but when these do not serve needs well, writing code is often the only escape hatch.
As a result, while these platforms excel in simplicity and speed for use cases that conform to the most mainstream of needs, they fall short to cater to the very long tail of specialized use cases that emerge in practice.
Additionally, since these are (usually proprietary) platforms, users have limited control over their data and content and transferring a website to another platform can be challenging.
CMSes are possibly the most popular way for end-users to publish their content on a website they control.
These include platforms typically hosted on one’s own server which connect to some form of data storage (e.g. a database), and provide templating functionality and visual affordances for editing content. Examples of such systems include CMSes such as Wordpress, Drupal, or Joomla.
Previous work has explored the high levels of dissatisfaction with how rigid and heavyweight these are [1].
The drawback to many of these systems is that they often require using their own heavyweight authoring and hosting environments, and they provide pre-made plugins or templates that users can not customize without programming.
Another drawback is that they are structured around a very crude model of what is UI and what is data, typically consisting of a set of pages with content that is edited all at once, and no computation. While this model works for content-heaby websites, such as blogs or media portals, it does not work so well for displaying and editing structured data which is a lot more fine-grained than a single blob of text with a title and other metadata.
Displaying and editing structured data is a broad category of use cases that come up very frequently, even for content-heavy websites. For example, thing of the personal blog of a popular conference speaker, a textbook CMS use case. Displaying their list of talks, their list of publications, a list of press mentions, a list of interviews they have given, all of these are examples of structured data for which a CMS is not well suited.
The dissatisfaction around CMSes bred the growing community around static site generators, such as Jekyll [2] and Eleventy [3]. These do not have a visual interface at all, content is typically stored in Markdown files and HTML templaes, and the final HTML is generated by invoking a terminal command.
While these are a lot more lightweight and afford tremendous levels of control, they practically target exclusively web developers, as they require significant technical expertise to configure, and offer no graphical interface for editing data.
Many uses of CMSes are merely to enable non-technical users to edit website content, a use case that static site generators do not accommodate.
“Headless” CMSes are tools designed to bridge the gap between CMSes and SSGs, by combining the ease of use of the former with the control of the latter. However, these typically require the SSG to first be configured normally, and then its templates painfully annotated to tell the CMS where data should go and how to edit it.
Moreover, they tend to fare poorly at providing a WYSIWYG preview of the rendered website, since it’s not always clear to them what the content managed in the system will be used for in the end.
So far, the types of website builders discussed focus on editing content and making it look good via templates. Any computation is added via plugins, and if no suitable plugin exists, it requires programming.
However, there is already a very successful paradigm for end-users to store their data and perform computations on it: spreadsheets. Because of the popularity of spreadsheet applications, many researchers and practitioners have explored eliminating the usability issues of spreadsheets and pushing the boundaries of the spreadsheet paradigm.
Common extensions to the spreadsheet paradigm include:
While spreadsheets address the user need for lightweight computation, most spreadsheet systems (research or commercial) share the same limitations:
Because of the popularity of spreadsheet applications, some researchers have explored using a spreadsheet for end-users to define and manage their data. For example, Quilt is a system that allows users to link a Google spreadsheet with a webpage and provides simple syntax to bind GUI elements with particular cells in the spreadsheet [14]. Similarly, Gneiss is a live programming environment that incorporates a spreadsheet editor and allows users to create bindings between GUI elements and spreadsheet cells [7].
A large class of web applications are purely CRUD (Create, Read, Update, Delete) interfaces to structured data. Databases allow storing and querying structured data, but integrating them into a web application is quite laborious even for professional programmers. A small study [15] found that the ratio of “plumbing” code to pure data code (business logic + SQL) in a web application was a whopping 24.4:1!
Therefore, many systems have focused towards making database systems more user-friendly for web application development, and/or building interfaces to easily display and edit database data.
One direction involved bridging the spreadsheet and database paradigms by exposing a database as a hierarchical spreadsheet [6, 9, 13, 16, 17].
SQL is a widely accepted data query and manipulation language, and its declarative nature means that relatively complex data queries and updates can be performed using even a single short line of SQL.
However, web designers with limited knowledge of databases might not be able to write SQL queries in order to make these edits programmatically. Several database-driven web application platforms have been developed to assist non-programmers to build web applications. WebML [18] presented a web modeling language that provided a graphical way of specifying the database schema and navigational structure of web application. However, WebML does not provide a mechanism to do programmatic updates to the data. A lot of work has been conducted on developing visual query languages [19–21]. These systems hide the SQL syntax from the users, but they still show the database schema and the relational tables, which could be overwhelming for non-programmers with limited knowledge of databases. They also do not offer any way for the user to create web pages on top of these visual query languages. Other systems have focused on creating form-based visual tools for creating queries, design database, and define views [22–24]. However, these tools do not offer a WYSIWYG environment and they similarly require the users to deal with joins across multiple tables, which has been shown to be unnatural for average users [25].
AppForge [26] tried to hide the complexity of building and editing databases by developing a graphical interface to navigate the database schema. And like our proposed extension to Mavo, it provided graphical primitives, in which developers can create and edit NRA views over the schema. Nevertheless, it exposes non-programmers to the complexity of databases. FORWARD [27] is another system that provides a powerful WYSIWIG environment for creating web applications, however, not only it requires writing SQL queries within HTML, but it also requires writing JavaScript if users need to create a custom visual layer. Other systems, like that presented by Kowalzcykowsi et al. [28] provide a WYSIWYG environment and do not require users to edit the database schema directly; nevertheless, they do not provide an abstraction for complex relationships, aggregation and nesting. Mavo [29] allows users with basic HTML knowledge to create Web applications that manage, store and transform data, and unlike some of the previous work it provides for nested data, but does not let you join one nested data blob to another, it also offers controls for adding, updating, and deleting individual items manually. However, the only data manipulation that Mavo presented is direct editing of a single item, although in many applications, even simple ones, there are more complex editing actions that need to be developed. In this work, we extended Mavo to support specifying such actions programmatically.
Visual application builders like app2you [28] and AppForge [26] allow authors to specify the design of pages by placing drag-and-drop elements into a WYSIWIG-like environment. However, this approach limits authors to only the building blocks provided by the tools and provides very little control over the specifics of the interface created.
Some systems have been developed to provide a WYSIWYG interface that allows non-programmers to create web applications, without dealing with the complexity of databases and SQL queries, by using spreadsheet as the a back-end. Dido [30] allows users to visualize, edit, and store editable data directly in their browser. It allows web designers to integrate Dido into any web design and made it independent of any back-end system. Another system is Quilt [14], which integrates web applications to a Google spreadsheet, allowing web authors with no programming skills to gain access to lightweight computation.
Gneiss [7, 8] is another interactive system that extended a spreadsheet. It lets users retrieve JSON data returned from web services to a spreadsheet interactively without programming, unlike some of the previous work, Gneiss supports hierarchical data. However, since it depends on spreadsheets as the back-end, it does not really provide a mechanism to update data. None of the previous work provided a mechanism for programatically allowing end-users to specify data updates, without them having to write SQL queries, which can get complicated for nested schemas [25] or scripting. Our work is building on top of Mavo to make it more powerful, allowing users to specify computational data updates that are not evaluated reactively, but are executed based on user interaction. Rather than limiting users to only manually data editing, we want to empower them to create richer data interactions and ultimately, to build more powerful web applications.
The idea of extending HTML to make it more powerful is not new; there have been many past attempts at extending it in different directions.
Many attempts to make HTML more powerful treat HTML as a shortcut for programmers to express programming concepts more succinctly. They focus on reducing the amount of programming code required, not its difficulty. One such system was FORWARD [15] which aimed to simplify the “plumbing code” needed to render and edit data stored in a SQL database into a web page. It was quite powerful, but required writing SQL queries within HTML. There are also several JavaScript frameworks with this philosophy, starting with AngularJS [31] in 2010 and more recently VueJS [32]. These adapt and extend HTML to present dynamic content through two-way data-binding that allow for the automatic synchronization of models and views, but require the user to be well versed in JavaScript to use them.
ConstraintJS [33] extended HTML with a templating syntax and reactively evaluated constraints, but required the user to understand and write JavaScript.
Exhibit [34] (and later Dido [30], based on Exhibit) were some of Mavo’s early influences. Exhibit extended HTML with language elements that visualized and stored editable data directly in the browser. This approach allowed a web designer to incorporate Dido into any web design and made Dido independent of any back-end system.
Quilt [14] was one of Mavo’s biggest influences. It extended HTML with a language for binding an arbitrary web page to a Google spreadsheet “back-end”, enabling web authors to gain access to lightweight computation without programming. While it afforded full creative freedom, and lightweight computation, like Mavo, it imposed the software engineer mindset of data modeling as a separate task, and UIs as views that need to connect to the data model as a separate step, which can be cognitively taxing for end-user programmers that tend to be goal-oriented. Additionally, it was by design limited to spreadsheets as the means for data storage.
Web Components
is the colloquial term for a set of standardized technologies that allow developers to encapsulate reusable functionality in HTML elements,
and provide an extensibility point to HTML by allowing the creation of custom elements.
While Web Components require (fairly advanced) JavaScript to create, because they can be packaged and distributed,
the theory is that novices can import them and use them just like native HTML elements.
Web Components are not an alternative to Mavo: first, novices can only use them, not create them, and second, they exist at a different level of abstraction than Mavo which focuses on facilitating data interactions, rather than encapsulating UI functionality.
However, Web Components are complementary to Mavo. Because Mavo leverages existing HTML elements as foundational building blocks, the addition of custom elements through Web Components broadens the range of functionalities available to Mavo authors. Well-designed Web Components extend Mavo’s capabilities while maintaining the same low threshold as native HTML elements, thereby expanding Mavo’s utility without increasing its complexity.
There are several languages designed around transforming data to HTML or simpler HTML to more complex HTML, in order to automate repetitive templating tasks.
Extensible Stylesheet Language Transformations (XSLT) [35] is a language designed for transforming XML documents into different formats, including HTML. It is primarily used to transform XML data into a presentable HTML format, applying styles and formatting rules that dictate how the content should be displayed in the browser. XSLT is powerful in environments where XML is the primary data format, allowing for the separation of content and presentation, but its syntax is complex and verbose, making it difficult for non-programmers to use.
Several templating languages also exist (e.g. Handlebars, Mustache, Jinja, etc.) that allow authors to write HTML templates with placeholders for data, and store the data separately in a structured format (e.g. JSON, YAML, etc.).
Cascading Tree Sheets [36] was a research language that essentially functioned as a templating language where both the data and the output were HTML. It allowed authors to write minimal HTML documents with only the elements required to hold their data, and add any superfluous presentational markup as transformations of that HTML, specified via CTS rules with a CSS-like syntax.
These languages are typically static one-time transformations that produce HTML from data, not dynamic data bindings.
There has been a great deal of work on both encouraging and extracting structured data on the web [37]. However, automatic scraping techniques often have errors because they must infer structure from unstructured or poorly structured text and HTML markup.
Several efforts have been made to define syntaxes and schemas, such as RDFa [38] and Microdata [39], for publishing structured data in web pages to contribute to the Semantic Web and Linked Open Data [40]. However, novice users have had little incentive to adopt these standards — sharing data rarely provides direct benefit to them — and find them difficult to learn, potentially contributing to their limited adoption on the web.
It appears that the approaches that work best for increasing adoption of semantic web technologies are those that provide immediate benefits to them, such as search engines displaying richer results for structured data, or tools using structured data to improve user interfaces and/or make prose more informative [41].
Mavo contributes to this line of work by using a standards-compliant syntax that is machine-readable, yet produces tangible benefits. With Mavo, authors expend effort because it makes their static website editable or creates a web application. As a side effect, however, they enrich the Semantic Web by producing structured data.
In the last few decades, several research efforts have focused on how novice programmers or non-programmers struggle in learning how to program [42–44]. These studies showed that this is because of the mental models novice programmers build about the notional machine. Another study found that programming is more difficult than necessary because it requires solutions to be expressed in ways that are not natural for non-programmers [45]. The study examined the ways that non-programmers indicate solutions to common programming tasks, which are often vastly different than the ways programming languages require solutions to be expressed.
Natural Programming [46–48] is a research area that aims to make programming more accessible to non-programmers by studying what syntax and mental models feel most natural to them, and use these insights in designing languages and systems that allowing them to express solutions in ways that are more natural to them.