handmade.network » Forums » Work-in-Progress » Pixie Game System
Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11055 Pixie Game System
8 months, 3 weeks ago

nothings
I only had one strpool.h in my single-file-library list, so I added the others you've posted about here to https://github.com/nothings/single_file_libs


Oh cool - I didn't think of it. I can let you know when I release more libs, should you want to consider them for inclusion in the list :)

nothings
Some of this looks pretty great.

Thanks :)

nothings

If it weren't for the brace style I might actually use them! :)

Ha ha, yeah, I know that whitesmith is not the most common bracing style... but: http://i.imgur.com/sQNn6Fm.jpg
Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11163 Pixie Game System
8 months, 2 weeks ago

So, this weeks single-header library is a smaller one: a few variations of pseudo random number generators.

https://github.com/mattiasgustavsson/libs/blob/master/rnd.h

It includes four algorithms: PCG, WELL, GameRand, and XorShift. I didn't come up with them - the libs is basically just a repack of public domain implementations. I couldn't find any existing ones which were good, clean and structured in the way I wanted them. So since I have them, I might as well share.

It includes support for generating both unsigned 32-bit numbers using all the bits (unlike the default rand() of visual studio, for example, which only outputs 15 bits), generating floats in the range 0.0 <= N < 1.0 (which proved to be a little non-straightforward to do in a good way which doesn't affect the random distribution, and which never has a rounding error accidentally returning 1.0) and a helper function for returning an integer in the specified range (again, without skewing the distribution, which happens if you use a simple modulo).

Usage is pretty straightforward - here are examples using the PCG generator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
    #define  RND_IMPLEMENTATION
    #include "rnd.h"

    #include <stdio.h> // for printf
    #include <time.h> // for time
    
    int main( int argc, char** argv )
        {
        (void) argc, argv;

        rnd_pcg_t pcg;
        rnd_pcg_seed( &pcg, 0u ); // initialize generator

        // print a handful of random integers
        // these will be the same on every run, as we 
        // seeded the rng with a fixed value
        for( int i = 0; i < 5; ++i ) 
            {
            RND_U32 n = rnd_pcg_next( &pcg );
            printf( "%08x, ", n );
            }
        printf( "\n" );

        // reseed with a value which is different on each run
        time_t seconds;
        time( &seconds );
        rnd_pcg_seed( &pcg, (RND_U32) seconds ); 

        // print another handful of random integers
        // these will be different on every run
        for( int i = 0; i < 5; ++i ) 
            {
            RND_U32 n = rnd_pcg_next( &pcg );
            printf( "%08x, ", n );
            }
        printf( "\n" );


        // print a handful of random floats
        for( int i = 0; i < 5; ++i ) 
            {
            float f = rnd_pcg_nextf( &pcg );
            printf( "%f, ", f );
            }
        printf( "\n" );

        // print random integers in the range 1 to 6
        for( int i = 0; i < 15; ++i ) 
            {
            int r = rnd_pcg_range( &pcg, 1, 6 );
            printf( "%d, ", r );
            }
        printf( "\n" );

        return 0;
        }


Now, PRNG's are interesting things. I'd been programming for a long time before I realized how they actually worked. Sure, I'd called the "RND" function in various BASIC languages, but when I wrote demos and games in assembler, I had no idea what to do - so I used to generate lists of random numbers from BASIC, to include in my assembler programs, and then use a clock register modulo the number count, to randomly seed my sequence. It sort of worked at the time :-)

The first time I had reason to actually look more closely at how random number generation works, was many years later, when I was working on a prototype for a GTA style game on the (then new) PS2, and we wanted to add recordings (mostly for debugging purpose) by recording user input and reapplying them when playing it back. For this to work, your game needs to have predictable behavior, and random numbers becomes part of this. By switching away from default rand(), I gained better control of the RNG - most importantly, I could save the state of the RNG, and continue where I left off - this was very helpful, and I also found it to be indispensable when doing procgen stuff.

I could also do things like using a faster RNG for things like particle systems, where speed was more important than quality, and use a better RNG for gameplay code.

If you are interested in RNGs, I recommend watching this video:

https://www.youtube.com/watch?v=45Oet5qjlms

It is a good summary of RNGs in general, and concludes with presenting the PCG family of generators.

Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11275 Pixie Game System
8 months, 1 week ago

Last weeks release was admittedly somewhat on the smaller and more basic side. So this time, let's have something a but more meaty: a library for loading game assets, from a folder on disk and/or a compressed zip file.

https://github.com/mattiasgustavsson/libs/blob/master/assetsys.h

The main thing here, is that it is possible to "mount" any number of folders or archives (zip files), optionally give each mount a prefix, and load/enumerate files and subdirectories as if all mounts were the same virtual file system.

Let's have a look at some code for how to use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    #define ASSETSYS_IMPLEMENTATION
    #include "libs/assetsys.h"                                                                                                                                                           

    #include <stdio.h> // for printf

    void list_assets( assetsys_t* assetsys, char const* path, int indent )
        {
        // Print folder names and recursively list assets
        for( int i = 0; i < assetsys_subdir_count( assetsys, path ); ++i )
            {
            char const* subdir_name = assetsys_subdir_name( assetsys, path, i );
            for( int j = 0; j < indent; ++j ) printf( "  " );
            printf( "%s/\n", subdir_name );

            char const* subdir_path = assetsys_subdir_path( assetsys, path, i );
            list_assets( assetsys, subdir_path, indent + 1 );
            }

        // Print file names
        for( int i = 0; i < assetsys_file_count( assetsys, path ); ++i )
            {
            char const* file_name = assetsys_file_name( assetsys, path, i );
            for( int j = 0; j < indent; ++j ) printf( "  " );
            printf( "%s\n", file_name );
            }
        }

    int main( int, char** )
        {
        assetsys_t* assetsys = assetsys_create( 0 );
    
        // Mount current working folder as a virtual "/data" path
        assetsys_mount( assetsys, ".", "/data" );

        // Print all files and subfolders
        list_assets( assetsys, "/", 0 ); // Start at root 

        // Load a file
        assetsys_file_t file;
        assetsys_file( assetsys, "/data/readme.txt", &file );
        int size = assetsys_file_size( assetsys, file );
        char* content = (char*) malloc( size + 1 ); // extra space for '\0'
        assetsys_file_load( assetsys, file, content );
        content[ size ] = '\0'; // zero terminate the text file
        printf( "%s\n", content );
        free( content );

        assetsys_destroy( assetsys );
        }


I guess this is pretty straightforward. It shows how to mount a folder, enumerate all files/subfolders, and load a file from it. The documentation in the library itself should be fairly exhaustive.

A good thing about multiple mounts, and one I often use in my own programs, is that I can mount a folder to load assets from during development, and a zip file to load from at release. It is easier to modify files in the folder as I go, and when I come to release the game, I just compress the folder into a zip archive, and then delete the folder. And the code just automatically uses the zip file, as the folder is no longer there - without me having to change the code.

An additional benefit of using assteys, is that it enforces the same path separator (forward slash) on all platforms, and also ignores upper/lower case on all platforms, unifying the way I access files, which can be a great help.

Now, this library differs from the previous ones in two ways. First off, it uses the open source "miniz" library to read zip files. This library is embedded inside assetsys.h, so you wouldn't know it when just using it. There's a way to have assetsys.h exclude the miniz definitions, if you'd rather include miniz into your project yourself.

Second, it has a dependency on another of my libraries, "strpool.h", which I released a few weeks ago. This one is not embedded though, so in order to use assetsys.h, you need to have strpool.h in the same path. I could have embedded it I guess, but I found this to be a bit awkward when continuing to develop my libraries.

So is this still a single header library? I guess technically it isn't, but I still choose to view it like that. It follows the same patterns as my other single header libs though, so at least it should feel familiar.

I am open to feedback on this. Maybe it would be better to embed strpool.h as well (with an option to use an external definition)? It is not unlikely I will consider this once the libraries have settled a bit.

Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11368 Pixie Game System
8 months ago

This weeks library is for basic HTTP requests (POST and GET commands)

https://github.com/mattiasgustavsson/libs/blob/master/docs/http.md

I have restructured by github a bit, making a copy of the documentation for each library into a separate .md file, to make it easy to browse the documentation. From now on I am linking to the documentation file, rather than directly to the library source code, but a link to the source code is at the top of each documentation file.

So, an example of how to use the lib:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
    #define HTTP_IMPLEMENTATION
    #include "http.h"                                                                                                                                                            

    int main( int argc, char** argv )                                                                                                                          
        {                                                                                                                                                       
        (void) argc, argv;
 
        http_t* request = http_get( "http://www.mattiasgustavsson.com/http_test.txt", NULL );
        if( !request )
        {
            printf( "Invalid request.\n" );
            return 1;
        }

        http_status_t status = HTTP_STATUS_PENDING;
        int prev_size = -1;
        while( status == HTTP_STATUS_PENDING )
        {
            status = http_process( request );
            if( prev_size != (int) request->response_size )
            {
                printf( "%d byte(s) received.\n", (int) request->response_size );
                prev_size = (int) request->response_size;
            }
        }

        if( status == HTTP_STATUS_FAILED )
        {
            printf( "HTTP request failed (%d): %s.\n", request->status_code, request->reason_phrase );
            http_release( request );
            return 1;
        }
    
        printf( "\nContent type: %s\n\n%s\n", request->content_type, (char const*)request->response_data );        
        http_release( request );
        return 0;
        }


Maybe it exists, but I've not seen any http libs with as simple an API as this. Make a request with a url, keep calling "process" until it is done, then do some cleanup. I mean, I get why you would often need more control, especially for more heavyweight web use, and for those things there are plenty of options - all of them quite bulky for my taste, but I guess that can be hard to avoid.

But I have often wanted, especially when making little games, to have something small and simple to just be able to do things like download an image or a "latest news" type of text, or to upload high score to a simple highscore table, things like that. And I couldn't find anything that I felt was small enough.

Also, I am not a fan of libs that use blocking sockets - it means I *have* to use a dedicated thread for each request. My lib is written as a single-header lib, no third party dependencies, and uses non-blocking sockets. You could still put it on a thread if you like, and just run process in a loop until it is finished, but you could also do the process on your main thread, or have a single background thread which can handle multiple requests in progress.

The lib only handles HTTP, not HTTPS. The amount of code needed for the TLS is just crazy, and all the libraries I've seen, like OpenSSL, seems really bloated. Ideally, I would want to, at some point, make a version of this lib which supports HTTPS too, but I suspect that might be a really large file. It might still be worth it though. Just the other day I came across a TLS lib called "tlse", which seemed less bloated, and only consisted of three files. I might embed that and make a https.h lib, but I think it would clock in at around 45,000 lines, which feels a little to large. But I guess you need a lot of code for that crypto stuff (which I have no desire to get into myself :P )

Anyway, if you just need some basic HTTP get/post functionality, this is probably just about as easy as it gets. Just don't expect to be able to build your own browser on top of it or anything :)
mmozeiko
Mārtiņš Možeiko
1503 posts
1 project
#11369 Pixie Game System
8 months ago Edited by Mārtiņš Možeiko on March 19, 2017, 8:56 a.m.

Windows API provides very simple API for handling http (and https, ftp) requests - wininet API. It can do a lot of things, but for basic http requests you need just 4 functions - InternetOpen, InternetOpenUrl, InternetReadFile and InternetCloseHandle. Easy to use also in unblocking mode. It will handle https as well and all the other http features, like persistent connection, redirects, etc...

While using http is good, nowadays it gets more complicated, because more and more sites are simply redirecting http to https (which they should), and you will need to use https. And TLS is a non-trivial subject. Properly to implement TLS requires a lot of skill - programming, cryptography and security wise. It's super easy to make mistakes implementing it. It will look like everything works, but later when somebody depends on your library for security (TLS purpose is security), somebody will find mistake and exploit it.

If you're OK depending on system support for TLS you can use Secure Channel API on Windows that can handle TLS protocol. You can still provide sockets, read data out, and schannel will handle TLS parsing and providing data out from it.

tlse is very bloated library :) It embeds libtomcrypt which is pretty big one.

For open-source alternatives to OpenSSL I can suggest looking to mbed TLS. It has a nicer API and is more sane than OpenSSL.
Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11373 Pixie Game System
8 months ago

I agree about http vs https - this library has its origin back when http was more of the norm. Still, can be useful for just doing quick tests against your own server where you can control if http is used. Possibly :)

I wasn't aware of the InternetOpen API - it seems quite straightforward. However, I'd want something cross platform. Possibly I could check for similar APIs on other platforms, and just write a common wrapper.

Thanks for pointing out mbed TLS, I hadn't looked into that before. However, a quick look seems to suggest it is about twice the size of tlse+libtomcrypt (57kloc vs 27kloc if not counting blanks/comments, 85kloc vs 43kloc if including them). So tlse is still the smallest I've found...
nothings
Sean Barrett
16 posts
1 project

former pro game developer (Thief etc. @ LookingGlass)
current indie game dev (Promesst etc.)
creator of stb libraries (stb_image.h etc.)

#11375 Pixie Game System
8 months ago

Goddammit windows.

1
2
3
4
5
6
7
8
9
#pragma warning( push )
#pragma warning( disable: 4127 ) // conditional expression is constant
#pragma warning( disable: 4365 ) // 'action' : conversion from 'type_1' to 'type_2', signed/unsigned mismatch
#pragma warning( disable: 4574 ) // 'Identifier' is defined to be '0': did you mean to use '#if identifier'?
#pragma warning( disable: 4668 ) // 'symbol' is not defined as a preprocessor macro, replacing with '0' for 'directive'
#pragma warning( disable: 4706 ) // assignment within conditional expression
#include <winsock2.h>
#include <ws2tcpip.h>
#pragma warning( pop )

None
mmozeiko
Mārtiņš Možeiko
1503 posts
1 project
#11380 Pixie Game System
8 months ago

The good part of mbed TLS is that it is very configurable - you can configure it with #define's to have functionality you only need. Then it gets very small. I haven't used it myself, but I know it is used in a lot of real life products, so I would trust its TLS implementation. I have experience with libtomcrypt, and it is not a good one. It had pretty obvious bugs in some of their crypto implementation.

When including Windows headers I just disable all warnings :)
1
2
3
4
#pragma warning( push, 0 )
#include <windows.h>
#include <...>
#pragma warning( pop )
Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11381 Pixie Game System
8 months ago Edited by @Mattias_G on March 19, 2017, 7:13 p.m.

nothings
Goddammit windows.

1
2
3
4
5
6
7
8
9
#pragma warning( push )
#pragma warning( disable: 4127 ) // conditional expression is constant
#pragma warning( disable: 4365 ) // 'action' : conversion from 'type_1' to 'type_2', signed/unsigned mismatch
#pragma warning( disable: 4574 ) // 'Identifier' is defined to be '0': did you mean to use '#if identifier'?
#pragma warning( disable: 4668 ) // 'symbol' is not defined as a preprocessor macro, replacing with '0' for 'directive'
#pragma warning( disable: 4706 ) // assignment within conditional expression
#include <winsock2.h>
#include <ws2tcpip.h>
#pragma warning( pop )


Ha, yeah, sorry about that, but what can you do :P Part of it is my own insistance on compiling with practically ALL warnings enabled, and windows does trigger some (the stb libraries also trigger some, so I have them wrapped in a couple of disable's too, but it's not a problem - they are generally more well behaved than windows).

You should see the list I have to add if I were to include any STL headers :P Luckily, I am never that wreckless.
Mattias Gustavsson
@Mattias_G
29 posts

Amateur (ex-pro) game dev, making retro styled games and small public domain C/C++ libs.

#11383 Pixie Game System
8 months ago

mmozeiko
The good part of mbed TLS is that it is very configurable - you can configure it with #define's to have functionality you only need. Then it gets very small. I haven't used it myself, but I know it is used in a lot of real life products, so I would trust its TLS implementation. I have experience with libtomcrypt, and it is not a good one. It had pretty obvious bugs in some of their crypto implementation.
[/code]


Yeah, I can imagine that if it is for embedded systems, it would be quite small. I haven't checked how much it would add to my exe. I did try tlse though, and it seemed to add around 150kb, which is a good chunk, but I can easily live with that for my areas of usage.

Your security concerns are more worrying though. Can you remember what kind off issues you encountered? I saw this on the tlse github, which seems to suggest sufficient security rating: https://raw.githubusercontent.com.../tlse/master/tlslayer_ssltest.png

Right now, I'm actually considering making a https.h with the same interface as http.h, but which embeds tlse/libtomcrypt in all its 43kloc glory. For the times when I want a drop-in single header HTTPS lib...
mmozeiko
Mārtiņš Možeiko
1503 posts
1 project
#11384 Pixie Game System
8 months ago Edited by Mārtiņš Možeiko on March 19, 2017, 9:04 p.m.

Supporting TLS features doesn't guarantee good TLS security.

Here's another discussion we had on crypto: https://handmade.network/forums/f...769-helping_with_the_site/2#10010
Even if you implement everything correctly from unit-testing perspective (all valid/invalid input data produces correct output), it is still potentially exploitable with side-channels.

I don't remember all libtomcrypt issues, but here's one minor issue on ECDSA that was reported by my colleague in 2011 and which was ignored for a long time: https://groups.google.com/forum/#!topic/libtom/gCFGsnhZ7WU But eventually it was fixed in 2014.
babygun
Lin
1 posts

None

#11745 Pixie Game System
6 months, 4 weeks ago Edited by Lin on April 24, 2017, 7:32 a.m.

Thanks for sharing those single libs!

I need some way of doing string interning in my pet project and I learned a lot from your strpool.h lib. I have one question though, why do you need `strpool_internal_handle_t` to reference `strpool_internal_entry_t`? Can you directly store the counter in `strpool_internal_entry_t` and return entry index and its counter instead?

I might've missed something (I only read through the code, didn't test it), but at this line `base_count` will always be zero, right?

None