HUMBUG»Blog
David Butler
I really hate to do this, but its been over two months since I've been able to do any work on this. I would have put the project into hiatus sooner but I was expecting things for me to lighten up. Hopefully I'll be back in a couple months but who knows. -- Take Care
David Butler
Hello Everyone:

Not a real update, but just wanted to remind people that I haven't dropped dead, and I am still working on the project. Been very busy in my personal and professional life, so I don't have time to write the blog post that you all deserve :( For what it is worth, i'm done prototyping all the GUI code that I was going to do and I have started re-implementing the humbug UI with my new stuff.

David Butler
Hello Everyone:

Again, this month I haven’t had much time to devote to HUMBUG. <INSERT LAME EXCUSES HERE> I did however get a couple good days and have inched forward on getting the GUI code done. I have a fairly full featured text box widget. I am pretty happy with it, it does undo, mouse select, and fully functional line editing. I was initially going to try to use STB’s text edit, but I actually decided that it would be much simpler for me to just implement what I needed from scratch. I also have scrolling panels now that work much better for me than what Dear ImGUI was doing for me before. The simplest way to use the immediate mode GUIs for list type things, is to render the entire list in a loop and it clips to the visible region. However for me I was having prohibitively large numbers of items in my lists and it was contributing a noticeable lag without some workarounds. My new way of doing the lists is to have the panel tell me which item to start rendering and I can just skip to that one. I also of course have buttons. I don’t really think that HUMBUG will really require much more than that, as I am not really using anything more than that in the current GUI.

I think moving forward I will not need movable/floating panels, so I think Im going to skip implementing that for now. I don’t think that the floating panels really contributed to the user experience, and I was only using them because that was the easiest way to get started with DearIMGui, I think I would be interested in maybe doing resizable/dock-able type things that you see in visual studio, but I think for now I’l just try something fixed, probably a 3 panel layout with the source code as the largest with watch and stack panels on the bottom and right. Other things like break-points and console I could just have show up in momentary modal dialogs… Anyone have thoughts on this?

I just have a few things left to do like keyboard focus and turning the GUI prototype code into more of a library. I am hoping that those items shouldn’t take very long and I can start re-implementing the HUMBUG soon. Hopefully I can get some good feedback from testers and then assess if its time to expand the testing pool or not.

PS Random question: I am using glScissor to trim up the edges of my panels, but I find that it being in window coordinates is a little awkward… is this something that I should consider putting into my shader? I’d be interested in seeing some example code for that…
David Butler
Hello Everyone :)

I hope you all are keeping warm this winter, nestled cozy near a warm hearth while listening to the long anticipated new episodes of JACS. I am just dropping in to say that I am still here and I'm sorry for the abrupt radio silence (and lack of streams). Things got busy very quickly after my last update and I haven't had much time/energy reserved for HB development. It’s most likely that it will be the middle of January before I start streaming like I was again. I have to be honest, the break is nice, but I'm getting a little worried that I have lost all of my momentum, and starting up in January is going to be tough ... We'll see... I wish everyone a happy handmade holiday season.

--Take Care :)
David Butler
Hello Everyone:

TL;DR: I have been hard at work over the past few weekends trying to get HUMBUG to a releasable state. Skip to the last paragraph for a brief summary of accomplishments. Hopefully I’ll be done with user interface stuff by the time I do the next update… well see :)

Long version:

Most all of the open complaints that I have from testers are issues with the the user interface. This is primarily because I didn't spend a lot of time trying to make a good user interface, but instead I just wanted to make something that was just better than using GDB in the terminal.

I had been using DearIMGui since the beginning but decided that it had some inherent limitations that would prevent it from being my long term solution. So I embarked on making a new UI toolkit that would be tailored to my needs. But before that I figured I'd take a quick side quest and try to get rid of SDL while I was at it.

I had wanted to get rid of SDL for a number of reasons, in brief: Runtime link dependancies on SDL(and Xlib), also also their runtime overhead; and also to have lower level control of event dispatching (i.e. just wanted to throw everything my application would wait on into a single epoll_wait call and dispatch on that).

Before I started on this journey I didn't know much about the internal workings of the X11 protocol or the state of Xlib or XCB. I did know that it was a client/server api and that XCB was a newer effort to try to make a lower level interface to X11. Fun fact: Xlib is now built on top of XCB, even though XCB is newer than Xlib.

I decided that for my goals it would be best to use XCB. It already was a very thin layer over the socket protocol. There were some things that I did not like about it, (mainly that it basically does malloc(s) for nearly every bit of traffic). Even with these issues I thought that I could just grab the source and begin to semantically inline the small bits of XCB that I was using and over time convert its implementation to something I would like better.

That was very foolish thought and...

Beware, there be dragons here.

Things started fairly well, and after a couple hours, I had a basic event loop where I could get mouse and keyboard input events. I got all this to work by following a tutorial on www.x.org. Things started to get hairy when I wanted to convert keyboard events into UTF-8 input. In the tutorial there is a line that simply reads: "TODO: Talk about getting the ASCII code from the key code."

I suspect that tutorial has not been updated for a very long time. This should have been a sign that I should have just given up. But I told myself, "But I've gotten this far fairly quickly, how hard could it be? Worst case, I'll just look at what Xlib does and reverse engineer it..." Also a second bad sign was that in the XCB tutorial, everything is given in terms of Xlib's api, again I simply brushed this off by thinking "This is probably just for people converting from Xlib, and surely there exists real reference and complete documentation.”

I won't go into all the little details but I'll briefly explain the process for which character input works in Linux. When you press a key on your keyboard the X11 server will send a packet over a socket to your X11 client and this packet will have a field called "detail" that has the keycode for the key that was pressed. This keycode (which is just a number) has no simple pre-defined relationship to any idea of what the key actually was (ie the KeySym, like this is the A key) or the character that key should generate if it were pressed (like "A").

There is a protocol that describes how one would convert those keycodes into characters. The process starts off by you telling the X server that you want to enable an extension called "XKB". Yes that's right, modern keyboard input is an "extension"... This extension was made in 1995. Also fun fact: Part of setting up XKB is the X11 server telling you what ID numbers the events for this extension start on, because for some reason we can't just have a header file with some defines in it to tell us this. After doing this, you then need to ask it for the keymap which will give you all the information needed to do the conversions. Also, you have to tell it that you want to receive events for situations where that keymap will change so you will know when to ask for it again.

Side note: You'd think at this point we might have considered just having the Xserver just give us reliable KeySyms and their related characters... Also another related fun fact: XKB is also the keyboard protocol for Wayland.

Luckily, I eventually found a library called xkbcommon, which can work on top of XCB and does most of this work for you. It even has relatively decent documentation to get you started. I found this merely by chance after lots of googling trying to get all this stuff to work.

A quick side note: getting an OpenGL context with XCB is tricky, if at all possible. At this point I had settled on just using GLX via XLib to get a GL Context. I had originally had hopes of getting that to work in the same spirit as the XKB stuff, and also was thinking about trying to get EGL to work. In the XCB code there is glimmers of hope that GLX can be achieved, but I was not brave enough to continue down that path. Morale was getting low for me at this point.

I am a very stubborn person, and I am not usually willing to admit defeat.

I had one last piece to get working before I could say that I had a platform layer for HUMBUG: Copy and Paste. This is its own flavor of insanity. This involves the XServer essentially serving as a generic Key-Value store, and you have to send messages between the windows in X11 to negotiate the Key and the format for the Value and also who "owns" things that are called "selections" which isn't strictly standardized, but upon popular consensus everyone uses a selection called "PRIMARY". Fun fact, when you want to specify things as strings (like "PRIMARY") in X11, you can’t just pass a string, you have to register that string as an "ATOM" and then use the ATOM's id as the string.

I had nearly everything written that I would need for a platform layer, but I still needed to clean some stuff up. For example, one specific thing I had to fix was that if you copied more than 4096 characters then I would silently truncate it. I’m probably missing some subtle detail here, but I was struggling to divine the correct procedure to get X11 to let me know the length of string ahead of time. And I was struggling to care.

This was my breaking point.

I was playing with my test code and I realized the cold hard truth: After spending almost two weekends working on this, I was not any closer to my goal. SDL and Xlib have their technical issues, but their actual "real world" impact on user experience is negligible and I had no more will left to continue working on this, and especially not to repeat this process on each platform.

I think this is a real clear case of underestimation and pre-mature optimization and its folly.

I still do one day want to revisit my XCB based platform code, but not any time soon.

After abandoning the platform code, I was able to get my lower level event code (via epoll) working the way I wanted by just working around the SDL code. Also I was able to build a static lib that had most of the dependancies in it. I got all that working how I wanted in just a couple hours. And then I moved on.

The rest of the month was much more productive. I spent a day making a simple build system that replaces a bunch of common code that I have in my build scripts. Some time I'll probably do a write up on it, it’s neat and makes it really easy for me to start quick one-off projects. The build system stuff was more of a quality of life improvement for me and my other projects than it really was for HUMBUG, but it was fun to work on and it got me going again. I also finally broke down and made a string library. Also learned how to use modern OpenGL (3.3). And I’ve made a decent amount of progress in getting the foundation for my GUI library. I’ve got some high hopes for what I can do with my own GUI that I could not do before.