This project may seem abandoned but I'm actually still working on it. I just am not really making any progress.
I got real time screen capture and input sending working pretty quickly but then it turned out that is not the hard part. I was planning to do a "pure" AI that would watch the screen to infer the gamestate but it's turning out that this is a very hard thing to do. I went down a computer vision rabbit hole for a while but there isn't really any good way to do this and I'd much rather spend my time writing the bot, not becoming a machine learning expert.
I'm currently trying to find different approaches like inspecting the game memory to infer where things are.
At some point I think I'll do a bot for HMH (once there is more of a game to play). Doing this for HMH is going to be wonderful because of the clear separation between platform and game, It'll be easy to just inject some code there than can inspect the screen, make decisions and send the input for the next frame. Heck I can even write an AI only platform layer and run 1000's of games at once, use genetic algorithms to evolve AIs, log all the results to be analyzed later. It's going to be a lot of fun I think.
I wish all games were designed as nicely as HMH.
I got real time screen capture and input sending working pretty quickly but then it turned out that is not the hard part. I was planning to do a "pure" AI that would watch the screen to infer the gamestate but it's turning out that this is a very hard thing to do. I went down a computer vision rabbit hole for a while but there isn't really any good way to do this and I'd much rather spend my time writing the bot, not becoming a machine learning expert.
I'm currently trying to find different approaches like inspecting the game memory to infer where things are.
At some point I think I'll do a bot for HMH (once there is more of a game to play). Doing this for HMH is going to be wonderful because of the clear separation between platform and game, It'll be easy to just inject some code there than can inspect the screen, make decisions and send the input for the next frame. Heck I can even write an AI only platform layer and run 1000's of games at once, use genetic algorithms to evolve AIs, log all the results to be analyzed later. It's going to be a lot of fun I think.
I wish all games were designed as nicely as HMH.
I've just pushed the first bits of code!
high speed screen capture on osx
I'm on osx so pretty much all of this code is osx specific platform layer stuff. I've written some code to hook into an osx api that you can pass a callback function that gets called every time the screen is updated and lets you access the framebuffer. It's not the greatest api, since you pass it a callback which it calls on one of it's own threads so communicating the data back to my main program required a threadsafe buffer. I built one using atomics but I don't have much experience doing that so if anybody wants to take a look and give any feedback that would be great.
Another small optimization that seems to speed things up a lot is that I take advantage of the fact that the game has pixel art and compress the data to be 1 actual pixel per logical art pixel. I'm not sure if this will work in all cases but viewing the preview of the game the only downside seems to be choppy animation. This seriously lowers the amount of data I have to run through scene recognition stuff so should help make that fast.
Next step is doing some scene recognition!
high speed screen capture on osx
I'm on osx so pretty much all of this code is osx specific platform layer stuff. I've written some code to hook into an osx api that you can pass a callback function that gets called every time the screen is updated and lets you access the framebuffer. It's not the greatest api, since you pass it a callback which it calls on one of it's own threads so communicating the data back to my main program required a threadsafe buffer. I built one using atomics but I don't have much experience doing that so if anybody wants to take a look and give any feedback that would be great.
Another small optimization that seems to speed things up a lot is that I take advantage of the fact that the game has pixel art and compress the data to be 1 actual pixel per logical art pixel. I'm not sure if this will work in all cases but viewing the preview of the game the only downside seems to be choppy animation. This seriously lowers the amount of data I have to run through scene recognition stuff so should help make that fast.
Next step is doing some scene recognition!
Aissac is a first experiment in writing programs that can play video games. The program should watch the screen and submit user input the same way
that a human player would and not have any direct access to the game's internal state. It will be written from scratch in a handmade style with
minimal libraries. This is definitely a program I do not know how to write so much of the experience will be learning and I hope to write about
problems and solutions as they come up.
I chose "The Binding of Isaac" as a first game because I understand how to play it well. It is not however a toy example but as a real time action
game will hopefully hit many of the topics I wish to dig into such as AI/planning, constraint solving, control theory and machine vision all with the
necessity of real time performance.
Hopefully in the future we can expand into other games, 3d games, puzzle games, different genres, etc. And who knows, maybe the tech we develop will
be useful in other areas for other projects. If any of this sounds interesting stick around and maybe together we can learn something.
that a human player would and not have any direct access to the game's internal state. It will be written from scratch in a handmade style with
minimal libraries. This is definitely a program I do not know how to write so much of the experience will be learning and I hope to write about
problems and solutions as they come up.
I chose "The Binding of Isaac" as a first game because I understand how to play it well. It is not however a toy example but as a real time action
game will hopefully hit many of the topics I wish to dig into such as AI/planning, constraint solving, control theory and machine vision all with the
necessity of real time performance.
Hopefully in the future we can expand into other games, 3d games, puzzle games, different genres, etc. And who knows, maybe the tech we develop will
be useful in other areas for other projects. If any of this sounds interesting stick around and maybe together we can learn something.