Cakelisp » Blog

New blog post: Evaluating integrated development environments

Macoy Madson — 1 day, 13 hours ago

This article is also available on my blog.

I spend a lot of time thinking about development tools. In this article I want to discuss integrated development environments. These are programs that provide text editing (usually code) as well as a variety of features tailored to making the software development process smoother.

Why you should care

Better tools can make the development process more efficient. This doesn't just save you time, but also changes what you are willing to do.

A good text editor is at the core of writing code, but it is only the start. The more integration you can perform, e.g. by making your editor aware of your language of choice and its symbols, building the program directly from the editor, and even debugging and visualizing program execution in the editor, the more seamless and natural development can become.[^1]

You also shouldn't settle for "good enough". We need to always be improving our tools.

Finally, I have derived a lot of enjoyment from noticing something annoying or inefficient, then solving that problem by improving my tools. It is deeply satisfying to use an environment that is increasingly tailored to you. Much like a kitchen or workshop where every tool is right where you expect it to be, a well-tuned environment lets you focus on building the product rather than constantly annoy you or slow you down.

IDEs in the game industry

If you asked a random group of gameplay programmers "What IDE do you use?" (IDE stands for integrated development environment), you would almost definitely get the answer "Visual Studio". Specifically, Visual Studio 20XX Professional etc., not Visual Studio Code, which is a quite different thing.

Visual Studio is dominant in the game industry, and it is evident by what the large engines choose to support. Unreal has a guide and Microsoft has a dedicated page showcasing Unity support (this is no surprise given Microsoft's involvement with bringing C# to life; most Unity games are written in C#[^2]).

One should not use a program just because it is popular like Visual Studio. In this article I hope to show how IDEs could be even better than Visual Studio, with the motive that someone might read this article and be inspired to implement more of these features in their IDE. I want every programmer to have the best tools they can, and having a wide variety of attempts at making better tools I think will lead to a healthier marketplace and increase innovation.

Better IDEs

Because IDEs like VSCode, Visual Studio, XCode, and Android Studio are so popular, I want to present a few examples of relatively esoteric IDEs. The more exposure developers have to these, the better. You don't have to use them, but you should at least understand how they push the frontiers of the tools we use to make programs.

Emacs

This is the editor I have the most experience with.

Emacs is a great example of an IDE for modifying itself. It can profile, debug, and provide documentation on its own features.

Emacs is one of the oldest pieces of software being actively developed and used today. Imagine the advantages and drawbacks of a program with 46 years of continuous development. The big disadvantage is lots of cruft. The advantage is 46 years of refinement of features.

Emacs has some incredible features that are still unmatched in modern IDEs: it's easier to modify, easier to debug, has fantastic git integration via magit, has fantastic diff integration (via e.g. ediff, but also tools like SMerge that automatically detect inline merge conflict formatting in files), has an absolutely unparalleled markup and note-taking experience, a magical remote editing experience (don't just edit, but also stage and commit with magit practically without even realizing you are remote), and more. There are definitely rough edges, but the sheer number of killer apps and synergies between them in Emacs make it still stand above modern editors in my mind.

Emacs performance isn't so bad as to be unproductive, but I do want something more lightweight. The biggest drawback to Emacs for me personally is how frustrating it is each time I want to set it up on a new computer. My Emacs configuration is several thousand lines and requires a few dozen plug-ins. Something always breaks when I set up with new version or a new machine. The next evolution of IDE for me is something significantly simpler. That said, Emacs has raised the bar for me for what to expect from software in general, not just IDEs.

Pharo

Pharo came from the Smalltalk world. Its features constitute a majority of what I desire. This kind of environment is the most cutting-edge I have seen.

This kind of environment blurs the line between using a system and modifying it, which I think is a powerful idea. It is fundamentally different than the precompiled, closed-source (or "takes more than a few clicks to obtain"-source), complex tool-chain world we live in on Mac, Linux, and Windows.

Closely related, the Glamorous Toolkit includes and furthers many of the same ideas as Pharo. The Glamorous demos showcase the power of custom, easy-to-write visualizers for making data explorable and shareable. It demonstrates how an object model can be more user-friendly, more robust, and more powerful than the Unix "everything is text" idea.

Bret Victor's ideas

Bret Victor's Learnable programming essay doesn't only apply to learning programming--even long-time experts would benefit from IDEs with timeline views and temporal decoupling. His Inventing on principle talk also talks about these ideas.

It is important to start from "In a perfect world, how would I try to understand and modify this software?" and work backwards from there. This is one way to come up with ideas that wouldn't be obvious from a "I have an IDE, what's the next incremental change to improve it" approach, which can lead to local maxima. In addition, radical changes might require much tighter integration with the user's program, even to the point of forcing a language like Pharo. This sort of change would likely never be on the table with an incremental improvement approach, but might be the only way (or the most efficient way) to reach the next level.

Project Oberon

Project Oberon is "a design for a complete desktop computer system from scratch". There are many great ideas in the project, but I'm only going to mention the ones I am aware of that are relevant to this article.

Similar to Pharo, the entire system is written in the same language. Oberon avoids the distinction of processes, instead using the idea of modules. As an example, you can import the graphics application's module in your code and call its functions naturally. This is much cleaner than the fragile and messy Unix way of using pipes or sockets to communicate. In Oberon, leveraging functionality across the system is a natural part of the language, not a complex runtime negotiation.

Oberon's text-based interface has a brilliant feature. Runnable commands are simple to create--the user merely types the command name and arguments into any text input, then middle-clicks to run the command. This radically simplifies the creation of user interfaces, because the interfaces are just lists of commands in text files. Window button bars work exactly the same way. This also makes the system more learnable, because the user can see the action every "button" takes and can simply copy paste and tweak that button's command into their own versions. If they then want to run several commands (a.k.a. programming!) that's a clear next step, not a giant leap.

The trade-off with a text-based interface is that it might look less conventional or "pretty"[^3], but the amount of value gained from this transparency and consistency is worth it. Emacs is another good example of just how powerful and usable a text-only interface can be.

Critical feature: easy modifiability and extensibility

The difficulty in modifying the development environment itself is important for many reasons.

The difference in difficulty of modifiability is partly demonstrated by Visual Studio when compared to Visual Studio Code. Visual Studio is modified via Visual Studio extensions. Extensions are written in C#, though it is likely possible to write them in C++ as well. The Start developing extensions in Visual Studio guide demonstrates how to write an extension.

Contrast Visual Studio with Visual Studio Code's guide.

Finally, compare those to this Emacs Lisp tutorial. Creating new editor functionality is typing Emacs Lisp anywhere in the editor, highlighting it, then running a single command eval-region. That's pretty hard to beat, and compared to the Visual Studios, there's no contest.

There is of course a cost to learning Emacs' custom language, Emacs Lisp, to configure it. You need to get a hang of it, but once you do, creating new editor functionality is a natural part of the development experience, and done over time results in extremely tightly tailored environments custom for your preferences and problems. I believe the only way to reach perfection is allowing and encouraging deep user customization, because no single product could meet every developer's needs out of the box.

A point system for evaluating IDEs

For fun, here's a point system you can use to evaluate IDEs. The features and amounts are what I care about; everyone has different preferences, so I expect your feature list and point amounts to be different. I also expect I've forgotten things, or didn't get the relative values right, but this should at least provide a coarse view. I scored points relative to the other entries in the feature groups. I only partially scored things relative to all other groups, so group-wide multipliers may make sense to help e.g. fundamental features always dominate the score when they are not found.

Fundamental expected features

These features are in almost every editor. This is not an exhaustive list, but should serve to be a "basic functionality" checklist that e.g. Windows Notepad might not pass.

Easy to create custom language highlighting = 200
Easy to create custom language formatting[^4] = 100
Easy to re-theme (dark, light, user-generated, etc.) = 50
Easy to re-bind keys = 300
Find text in file = 500
Find text in files in directory = 500

Modern comforts

These features are common in modern IDEs, but noticeably absent in development environments before e.g. Sublime Text.

Command palette to easily search for and run functionality via keyboard = 200
Fuzzy search on all selection fields = 100
Go to file by name (goal here is to beat File Explorer's slow folder navigation) = 500
Go to symbol by name (list all symbols and filter to desired) = 300
Go to definition of symbol at cursor = 300
Possible to tie key bindings to specific languages/"modes" = 200
Macro system for bulk text editing or complex cross-file automation = 250
Multiple-cursor support for bulk text editing[^5] = 250
Possible to integrate with version control (native, plug-in, or CLI sub-process) = 300
Existing plug-in ecosystem/community with in-editor browsing/downloading = 400
Extensions can be written and loaded interactively (no closing of editor necessary) = 100

Integrated development

The following features ensure the "environment" truly becomes an integral part of the development cycle. Not only should it edit text, but it should also offer control of the end-to-end process from editing text all the way to building, debugging, and visualizing the developed program execution.

Debugger which lets you single-step code in the same editor = 800
Profiler which lets you quickly find performance hotspots = 300
Run other processes trivially from the editor[^6] = 500
Build and run the program being developed in one keypress = 500
View build and run output in editor = 300
Jump to error file and line from build or output in editor = 400
Integrated difference visualization (remove need for external diff tools) = 200
Integrated notes so I can e.g. write my to-do list and easily jump to referenced code[^7] = 500
Extensions written in same language as editor core = 100
Integrated help and documentation on extending the editor itself = 200

IDEs of the future

These features are absent in most IDEs. I would consider an IDE with all of these to be cutting-edge. If you have ideas for IDEs of the future, I want to hear them! Email me!

Easy to write simple graphical visualizations (lines, rectangles) = 200
Easy to write complex graphical visualizations (3D shapes, node graphs) = 100
Possible to use developed program code to enhance editor too (e.g. inline render) = 300
Indexed file search (something faster than grep) = 150
Trivial and natural to modify and extend (Emacs' eval-region sets the bar to hit) = 500
Integrated debugger used when extending the editor itself = 100
Integrated profiler used on the editor itself = 100
Temporal visualization (see Bret Victor's timeline ideas) = 500
Remote editing (files are on remote machine) = 150
Remote debugging = 300

Optional features

The following can be added in if they are things you care about. These are important enough to include, but not important to everyone. Not all of these features are important to me.[^8]

Vim emulation = 100
Free and Open Source = 800
Free (as in beer), even for commercial development = 200
If open source: simple to build IDE itself from source (simple tool-chain) = 100
100% keyboard support (I shouldn't ever need a mouse to browse, debug, or edit text) = 400
Extensions written in same language as your programs (depends on your programs) = 500

Example scores

Here are some editors I'm familiar with, scored according to my point system. Optional features sum appear in parentheses.

Emacs scored 7550 (1900)
Visual Studio 2022 scored 6750 (500)
Visual Studio Code scored 6250 (1000)
Sublime Text scored 5950 (400)

There is some ambiguity or debate that could happen on whether a feature counts. If the feature must be included via plug-in, but the plug-in is commonly installed, I count that as the editor itself having the feature. I do not count plug-ins that only partially implement the feature such that relative to the other editors the feature does not meet the minimum quality bar. I am not counting from the "out-of-the-box" experience because even a small time investment can introduce many of the important features in my lists to the editor.

This list is not meant to say one of these programs is fundamentally better, and that you shouldn't ever use the others. It is meant only to be a demonstration of how one can compare these programs with how it approaches the "future of development environments". Additionally, it has the false assumption that these programs even set out to create development environments. All of these programs have different goals and target audiences, which means one should not expect they do well at whatever arbitrary criteria you throw at them. There's some axis that each these individually beats the others at, for example.

I did not count Emacs as having a debugger because the Grand Unified Debugger with gdb do allow step-debugging, but it doesn't quite reach the quality relative to Visual Studio 2022 where I feel it deserves to be counted. I do have a C/C++ focus, so e.g. Emacs would count as having a debugger if you have a Lisp focus. Similarly, Visual Studio Code's debugger pales in comparison to Visual Studio 2022, so I also didn't count it.

Visual Studio 2022's score is bolstered by its inclusion of a great debugger, which is indeed its killer feature. It is behind the rest in regard to ease of extension especially, and lacks features like multiple cursors that I believe make a big difference. (And no, Visual Studio's block-based multiple cursors does not count because the column nature of it is far too limiting).

Finally, I do not want this to be read as "Emacs is the best editor, and everyone should use it!" I'm only trying to show that Emacs does well under the given criteria. I expect an IDE engineered to meet all these criteria would easily defeat the competition, simply because it is easier to optimize for a specific test than just set out to create an IDE without knowing the goals.

Cakelisp and IDEs

Where does my programming language, Cakelisp, fit in with all of this?

The Beef programming language ships the language and an IDE (including debugger!). I think this greatly lowers the barrier to entry for the language. I want to do something similar, with the goal of replacing my existing IDE (Emacs) and making a better development experience than could be provided by any existing IDE.

The path to this is not completely clear to me yet. My working idea is to fork a C compiler like Tiny C Compiler and modify it to provide the dynamic environment I want. The next step would be to port Cakelisp to C, which it can already export, but itself requires C++. I would then start building the editor in this environment both as a dogfooding experiment as well as to ensure the editor can be modified seamlessly at runtime. This would radically simplify the tool-chain for building Cakelisp programs, because it would be a single executable that can compile, run, and eventually debug the programs. Currently, I am dependent on the standard C++ compiler tool-chain as well as 3rd-party debuggers. I want Cakelisp to be a vertical integration-type system.

Conclusion

There is still plenty of room for improvement in development environments. If you plan on building an IDE, make sure to look outside of the common, popular examples to find much more powerful ideas.

[^1]: One should be careful as always of vendor lock-in. It is natural for vendors of closed-source or SaaS software to force your project to be organized with their tools. This usually means the service is more convenient, but it comes at a huge price. Companies are known to raise their prices, move to subscription models, or even remove functionality in the name of simplicity to appeal to a different target audience. You should always know what your valuable data actually is (source code, configuration, etc.), where it is stored (important for making your own backups anyways; do not trust cloud services to exist forever or not ban you for some arbitrary reason), and how the end-product is built from the data (i.e., the build process). If you don't know these things, you are being locked in, and you should consider your data (and by extension, your business) at risk. The steps you should take in the right direction are owning the storage of your data and being able to build and run the program(s) completely locally, without some software that needs to phone home asking whether you have permission to do so.

[^2]: The Unity engine itself is mostly C++, but most indie to mid-size developers build Unity games with C# exclusively. The Unity C++ source is closed and costs significantly more to access.

[^3]: By lowering the bar for how difficult it is to create UI, you can focus on the usability and power instead. The same holds true for video games: if your game can tolerate art assets that are "low quality" but cheaper to make, you can make new gameplay much faster. While it is of course nice to have things look good, there is a definite price for aesthetics vs. speed of iteration/new feature development which should be weighed carefully. A game like Dwarf Fortress would be infeasible to make with the graphical fidelity of a game like Red Dead Redemption 2, for example. Every icon, wireframe, and custom layout required reduces the likelihood of rapid and "risky" experimentation of new UIs and feature sets.

[^4]: This could be in the form of running clang-format on the highlighted region, Emacs-style "electric indent" (when I press tab, it does what I want it to according to my alignment preferences, not just inserts a tab character), or any "smart" automatic formatting.

[^5]: Many editors have macro systems, which similarly accomplish bulk editing. However, the immediacy of multiple cursors (popularized by Sublime Text) as well as the ease of initiating them feels different from macros and I end up using them much more.

[^6]: This is a game-changer when it comes to integrating the editor with the ecosystem at large. Every build process, compilation process, version control system, authentication scheme, etc. should have a command line interface for easy integration in this way. Writing a plug-in to have tighter, more "native" integration is usually superior, but it is usually good to support sub-process execution as a transition or stopgap.

[^7]: The absolute best form of this I've ever seen is org-mode. It's no contest--Org has so many killer features like heading folding (so you can can have a huge amount of complex notes without it getting cluttered; most programmers I've seen naturally end up with an outline-style notes structure, and Org is built to support that), inline code highlighting, URLs to both local files and web URLs, export to a wide variety of formats for publishing (this article is itself written 100% in org-mode), smart table/spreadsheet authoring (you can even do Excel-like formulas on the cells, and it's all still plain text!), and many more features I haven't even explored. Unfortunately, the only true implementation is in Emacs; any emulation I've seen lacks the critical features like heading folding that really make org a killer app.

[^8]: I don't care about Vi-style modal editing, for example. I use the Kinesis Advantage2 and heavily customize and rebind things, so Vim doesn't actually appeal to me. I know people who are used to Vi(m) are very tied to it, which is why it deserves an optional category here. If you are building a text editor/IDE, you should at least have some idea how to please the ever-growing Vim crowd.

Wheel Reinvention Jam

Latest News

Around the Network