Working on my own JS game engine

Posted on 26/01/2017

Not the first time I've had a go at building a reusable full engine from scratch. But it's something that one of these days I'm bound to settle on one and finish it. I know what I want from the engine, though I'm still hitting kinks when something doesn't quite work as well in practice as it did in my head. The engine itself is a JavaScript library primarily for HTML5 canvas games. It will be multi-platform with the help of Apache Cordova, which basically can put your HTML/CSS/JS/etc into a stand-alone web view that runs as a native app. It's pretty neat.


This project is influenced quite a bit by the Love2D project. I follow the same base structure of the user overwriting 3 main functions, these are; load(), update(dt) and draw(). The player creates these 3 functions and the engine loops through them automatically. The object system is more inline with that of GameMaker, where you create an object and it has a set group of variables that are used to automatically update the object "behind the scenes", such as; gravity, movement, physics and animation. Though on top of that, you can create normal "stripped" JS objects and put them into the system manually for more detailed control.

As you can see in the image, nothing is quite settled yet! I think most things will function like the GAME.textDraw() function does and accept all arguments as a JS object. I find this approach very neat and flexible; using one function and only filling in the bits I need. The alternative is the GAME.primitiveRectangle() function here, this can actually be assigned to a variable, but in this case I just want to draw it directly so it calls the method right after the rectangle is defined. But all it's parameters are fixed, and so it's not very flexible.

Will post more details on this as the engine grows up, it;s still a bubba right now. :)

New website

Posted on 25/01/2017

I have been planning to rewrite my website for a long time, now I've finally got around to it. Previously I was using WordPress, but became frustrated at trying to do even the simplest things and not really having control. This new site I wrote from scratch, following a similar design but with everything a little more compact and clearer, not to mention faster and more convenient. I got a couple more websites that I need to update, and they'll likely follow the same design as this.

I plan to do more frequent updates on this site, in regards to the blog, filling up my games section and continuing to build my portfolio section. I may even try an update-a-day as I'm working on a lot of pretty cool, at least I think so, things right now. So, we shall see! :)

Optimizing noise generation in Terrablox

Posted on 22/09/2016

Part of the challenge I set myself with Terrablox is to create it in pure GML, not known for it's speed as it's a high-level language, so no external files that could do the job easier or faster or anything like that. The point of this is that I believe if some piece of code is slow, it's your fault. And finding a way round it by using another tool that is inherently faster is not solving the problem, I think. So it's largely a practice challenge I guess!

On that note, there's two main things that are "slow" in Terrablox, meaning in this case they take a lot of processing time, and they are world generation and geometry modelling. These are also the two most important things to the game, so finding ways to make them faster is always ideal.

Today I implemented a solution to make the world generation faster. The world generation is done using a noise function, passed different parameters for different levels of detail on the terrain (oceans/continents, mountains/rivers, hills/holes) and then the results are combined to create the final terrain at any given point. While noise functions are generally slow when used a lot, it itself was not the issue. The fact that I was using it a lot was the issue. For each 16x16 chunk it was doing the 3 noise functions for each and every cell, this adds up to 768 noise function calls per chunk and took about 12000 microseconds on average to complete. That's 0.012 of a second, which at 60fps is about 75% of what happens in a second, which is a lot for a single aspect of the game!. What I have been doing to alleviate that demand is split the chunk generation over n number of game steps, which divides the demand n times leaving more spare time in a second which helps maintain a good framerate (at the cost of it taking longer for chunks to appear). It worked ok-ish, and I'll still maintain that optimization.

But my new solution is better and simpler and, sadly, blindingly obvious. So, these noise functions work on quite a large scale, to the point there's very little variation in height within a single chunk. So I wondered why the hell am I calling noise functions for every single cell? Surely just calling the noise functions for each corner, and just interpolating those values per cell would be better? And yeah, it is, there's no major visible difference to the terrain in game and it's only costing 8000 microseconds per chunk now. That's good, it's 33% faster. So now it's only doing 12 noise function calls per chunk (3 for each corner). But there are still some noise functions in there to deal with biomes, and other things taking up time, which I shall work on replacing in the same way. I'll talk more about that when I do an article on my method towards biomes.

So in code, this is what was done per cell (repeated 256 times):
var noise1 = SimplexGetHeight( xPos, yPos, 0.5, 0.004, scale );
var noise2 = SimplexGetHeight( xPos, yPos, 1.0, 0.002, scale );
var noise3 = SimplexGetHeight( xPos, yPos, 1.0, 0.00025, scale );


And what happens now (before we get into the cells, done only once);
var height1a = SimplexGetHeight( chunk[X], chunk[Y], 0.5, 0.004, scale );
var height1b = SimplexGetHeight( chunk[X]+16, chunk[Y], 0.5, 0.004, scale );
var height1c = SimplexGetHeight( chunk[X], chunk[Y]+16, 0.5, 0.004, scale );
var height1d = SimplexGetHeight( chunk[X]+16, chunk[Y]+16, 0.5, 0.004, scale );
var height2a = SimplexGetHeight( chunk[X], chunk[Y], 1.0, 0.002, scale );
var height2b = SimplexGetHeight( chunk[X]+16, chunk[Y], 1.0, 0.002, scale );
var height2c = SimplexGetHeight( chunk[X], chunk[Y]+16, 1.0, 0.002, scale );
var height2d = SimplexGetHeight( chunk[X]+16, chunk[Y]+16, 1.0, 0.002, scale );
var height3a = SimplexGetHeight( chunk[X], chunk[Y], 1.0, 0.00025, scale );
var height3b = SimplexGetHeight( chunk[X]+16, chunk[Y], 1.0, 0.00025, scale );
var height3c = SimplexGetHeight( chunk[X], chunk[Y]+16, 1.0, 0.00025, scale );
var height3d = SimplexGetHeight( chunk[X]+16, chunk[Y]+16, 1.0, 0.00025, scale );


And now per cell (repeated 256 times, no noise functions, just simple lerps):
var r1, r2; r1 = lerp( height1a, height1b, _x/16 );
r2 = lerp( height1c, height1d, _x/16 );
var noise1 = lerp( r1, r2, _y/16 );
r1 = lerp( height2a, height2b, _x/16 );
r2 = lerp( height2c, height2d, _x/16 );
var noise2 = lerp( r1, r2, _y/16 );
r1 = lerp( height3a, height3b, _x/16 );
r2 = lerp( height3c, height3d, _x/16 );
var noise3 = lerp( r1, r2, _y/16 );


The lerp method can actually be optimized further. As the cells are scanned using 2 for loops, one for x and one for y, we can get the r1 and r2 variables in the x loop before entering the y loop. Which means that we can reduce the lerp function calls from 2304 (16x16x9) down to 864 (16x6 + 16x16x3).

Anyway, I think it goes to show that "it's always your fault" stands true. I've had this code the same for so long and that was clearly a mistake now, seeing as such an easy optimization was staring at me for so long. But, it happens. :)

Culling

Posted on 25/02/2015

Finally implemented proper frustum culling into Terrablox. Now, it works perfectly as intended; it knows precisely whether a chunk is inside the camera projection and I even check against the maxHeight inside a chunk, so even if you're inside that chunk area, if you're looking away from it's actually visible areas if counts as outside the projection. It's pretty neat. But the way it checks for visibility atm is actually slowing it down more, it does the check for every chunk in existence, which in the below example is 1024, but even then 757 are visible and that's still a big number. Without culling this world runs at >400 FPS, with it runs at 80. Wut...



So, the processing to lower the drawing is costing A LOT more than the drawing ever did. I did have ideas for optimizing the culling; like only drawing from the current chunk and moving to it's neighbors, using culling to check if the loop should continue. And I think I can get it a little faster by pre-defining a lot of things that I input to the culling function that don't actually change, like chunk position. But even then, the difference between these numbers doesn't bode too well.

I think the usual way to solve this problem is with octrees. While I know how they work I've never had the opportunity to implement them in anything before. But basically the idea is to look at groups of chunks contained in bigger boxes, if the bigger box is totally in view then we know all contained chunks are visible and we don't need to check them individually. If a group box is only partially in view then we know some of the contained chunks may be visible and we check them. And we can have boxes inside boxes to make even broader checks. I don't think implementing this will take long, but I hope for a good result when it is!

Not colorblind

Posted on 23/02/2015


Eww! What is that?! OK, I know it isn't pretty... I didn't spend time prettying it up. But this is the engine running in colored block mode, as opposed to textured block mode. This is designed to create worlds like in the game; Cube World. Now, you could have achieved this with textures, and the cool gradient color thing spread over the world that Cube World does could also be applied to textures. So, why bother doing this? Well, it's also handy for saving memory. For a texture to be applied to a vertex it needs to have 2 floating-point numbers, whereas a colored only block doesn't need them at all! For comparison; the world you see above (around 33m voxel spaces) when textured takes up 220,000K memory, and when it's NOT textured it only takes 170K. Now those numbers represent the whole program's memory usage, which means that 50,000K was used to hold texture coordinates alone! That's nearly 25%. And at the moment, the world isn't that complex, when we start adding trees, caves and things like that, that difference becomes exponential. So it's safe to say, that a color only world has more leg-room for sheer size. And that's why it's a mode worth having.

Switching between these modes is currently being done via constants. This is because GameMaker: Studio now has a cool little feature where the compiler checks against constants and won't compile anything from the following block if that constant isn't flagged. You could have 100 if statements checking against constants and it'd have no effect on performance in the final build if none of the constants were true as that code wouldn't be there at all! Nifty. This does however mean you can't switch mode mid-game, but I don't consider that an issue, as you'd make a game one way or the other. Though if you did want it for some reason, you can simply go through and replace that constant with a variable, however you will lose the optimization.

Another thing I worked on is progressive generation. I already had done this for modelling. What I mean by progressive is that a chunk's data or mesh can be created over several steps, which will help the engine maintain a playable frame rate. It's helpful for big worlds and testing as now I don't need to wait for generation to finish before it runs. I just run it and the world builds, still needs optimization though.

Sense of scale

Posted on 21/02/2015


So to show the sort of scales we can work with now I threw together a world with height set to the current max (128). So the mountains are a little grander, though I also decreased the frequency of the noise generation so they weren't all pointy! So this world has over 134,000,000 voxel spaces, and the reason the fps is low here, at 126-ish, is because each chunk is only 16 x 16 x 128, so there's a lot of them for the renderer to iterate through and none of them are even culled! This is an area that'll need a lot of work. Though a quick fix would be to just have bigger chunks. In fact, the new system actually allows us to join chunks together; so if you generate a bunch of them and they are frequently in view, we can join them up to cut down the number of rendered instances. In theory, we also shouldn't need to break up the chunks even if we're editing them. I'll talk more about how that will be done in another post.

tbsource3
I'm also now basing the generation on some fixed values, such as height of the sea floor, height of sea level and top height of the terrain. Those are just raw numbers. I'll need to alter what goes into the noise generation so the world scales up evenly and in a way that won't mess up biomes that should always remain the same, like swamps and stuff, which should always be patchy lowlands even if they're next to massive scaled up mountains.

Rivers

Posted on 19/02/2016

So I mentioned in the previous post that this current method of generation provides some nice meandering rivers. Well, I added a blue block to actually show these areas up and this is the result: tbsource2
As you can see, it's pretty neat! I also didn't expect that cool little island cluster in the corner, but I'll take it! (Over 33,000,000 voxel space, for those interested in progressive numbers, in this case it's about half a kilometer squared.) FPS is lower because there's over a 1000 chunks being drawn.

It'd be nice to add height below the rivers, in order to make them actually flow downhill. The problem with this would be finding the beginning or the end of a river as they are purely generated from noise. One way around it would be to just cut rivers off when they reach a certain height, then they'd flow from the sides of mountains. But this wouldn't work in all cases, as when no part of a river would reach that height, you'd have rivers that span over hills and stuff. However, that could also be fixed post-generation, when the water starts flowing it'd know it's flowing in 2 opposite directions, at that point we could bung some rocks in there and make it look like a spring.

Another idea I've had for liquids is having them completely separate from from the world's set of voxels. Which makes a bit of sense, really, but they would need to have their mesh created in a separate pass, which also might not be a bad thing. But having liquids separate from the solid world would mean we could have a whole different set of rules for it. I've been toying with the idea of having "bodies" of water, so all voxels that form a stable lake would only have 1 volume value between them... And when one voxel has to flow to another space, it actually only detracts a fraction of volume from the much larger body.

The big problem with all this talk of flowing liquids however is that the world is potentially procedural and infinite. What if we only generated half a lake, or half a river... Would it know it was only a small part of something else? One way to do it would be to make a liquid block, with no neighboring generated chunk, an endless volume block, ie; it's volume never goes down when it flows. Also, is an MC style "source" block a good idea for this, I mean, you wouldn't want a river to run dry now, would you? In this case I think it could be solved with the "bodies" idea, if a body is large enough and it's not losing too much volume at once, just assume it can't lose volume...

Not to mention what the hell I do with an ocean!

Anyway, this is all theory, but I thought it might be interesting.

And I wrote too much again! :)

Terrablox Source

Posted on 29/10/2014

So as a break from ignoring Terrablox I've decided to write it again from the ground up. This is my usual way of implementing real changes. Every version I've ever written has had vast improvements, but it quickly becomes dead weight again as I learn more. But this time, I'm coming at it with a different intention; I'm going to make just a core engine and release the source (which, at least for learning purposes and contributions, will be free). Later on, I will get back to the GAME side of it, using this new engine. So that's the goal.

As this is an engine, foremost, it'll have features that people have wanted and generally expect in this kind of system. So, procedural generation is a key feature. Onto this, a loose chunk system; chunks are simply stand alone objects, so they can be placed anywhere and in any configuration and will keep the terrain seamless if you fill in the gaps later. Memory is something I've already brought down using buffers with 8 bits per value, rather than GM's ds_grids which are 64 bits per value.

I plan to take memory usage further down by using RLE (Run-Length Encoding), as I've so far had no need to access every voxel thanks to my new mesh building system. When building a mesh, previous Terrablox done it in layers, I always knew stacks/columns would be faster and now I'm doing it that way. The thing with columns is you can quickly find where the first visible block is going to be (I track the population of each stack) and start the scan from there, continuing to the point where the last visible block will be. So, at least on a featureless terrain thus far, it's super fast to build the mesh as it reads the minimal number of voxels. And with RLE, even with complex features, it will be able to skip over the gaps, and when comparing to adjacent columns knowing exactly where everything actually visible starts and ends.

So the previous Terrablox, at my last reckoning could hold around 20,000,000 voxels in memory I think, before it hit the GM limit of about 1.7GB, it could render more but it'd just be a wrap of the world. This engine so far, without RLE, the most "physical space" I''ve had is 170,000,000 voxels (and there was plenty of room left). It took a while at the start to generate but once it had it built the meshes 1 chunk at a time at 100+fps, while drawing them all, I think since then I've optimized it further.

But enough of that nonsense, here's a picture:

tbsource1
This is only just over 8,000,000 voxels I think, the world is 256 x 256 x 128. But the reason I'm showing this one is to do with the procedural generation. This terrain is made using simplex noise, I've made a few area types, like swapland and mountains. This is more of a river/valleys place, as it produces very nice river-like meanders just from the in-between bit of the positive and negative areas of noise (you make them all positive and you're left with these twisting seams just at position 0, good for rivers!). Later there will be separate passes for features like lakes and canyons. So producing the height is one thing, later on I'll be trying to blend all these different types of terrain together, biome style. Another thing I'd like to mention about this, and it's just something I'm playing with, is the density of dirt in the lower areas. My thinking here was that dirt and sediment would naturally fall down and end up lower, and it's pretty trivial to determine this if you know the max height of your region. And as you can see from the cross-section I think it makes visual sense and looks quite natural.

Anyway, that's that. This post was a lot longer than I expected it would be, I guess that's not a bad thing, but there you go. :)

Dartmoor Finch - GMC Jam #16

Posted on 29/10/2014

The second bit of the stressy past week was the main GMC Jam. This went a little smoother and I was able to make a game that more closely resembled "finished". I even managed to implement online highscores using Scoreoid. I wan'ted to include more of a story for this game but there wasn't time, though I did get around to setting up a page for it on itch.io so you can check it out there.

Get To Da Choppa! - GMC Warmup Jam #1

Posted on 29/10/2014

So last week is all a bit of a blur, I entered two game jams! The first was meant to be a week long "warm up" to the to the main 72 hour GMC Jam. I originally thought that I could just take it easy and do it while still relaxing, I even joined up with someone I hadn't worked with before or even really knew. This someone was another GMC member called HayManMarc. The jam didn't quite work out as I expected it to; Marc and I seemed to work too well together and our game was becoming too good. Relatively speaking of course! :) We didn't manage to perfect the game as well as we envisioned, but hopefully when the Jam smoke clears we can come back to it and give it the attention it deserves. The game itself is basically a clone of an old game called "Choplifter", but with a twist: Zombies.

To read more about the game, Marc wrote up a very detailed dev blog here.

I haven't set up a page or anything for the game yet, but you can grab the jam submitted build here.

Week Of Awesome 2

Posted on 30/09/2014

So last week I entered a game jam on GameDev.Net. It was a lot of hard work but I'm really happy with the result. I made it with my artist friend, Francis. We had to keep a journal during the process of making our games.



You can check out my journal here: gamedev.net/blog/1938-jack-franciss-week-of-awesome-ii/

And you can download my game from here: Mindfield.zip

Terrablox Devblog 31

Posted on 08/09/2014

Every week I try and do a development video for Terrablox. In this one, I show how far I've gotten so far with dynamic Depth Of Field.

Terrablox - Alpha 0.2.9

Posted on 04/09/2014

So I guess I'll start talking about my main project here. Terrablox. It's a 3D Voxel-based, Geometry rendered sandbox adventure game. In very early stages still. There's a website and forum for it at: terrablox.com and I do a YouTube series on it's development which can be seen here: YouTube which has been going for probably about a year now! But today, I managed to get out a new release which had some cool stuff in it, like; Depth Of Field, graphics settings, hats, etc. Cool stuff. You can check it out here: Forums or download it directly here: Download

I hope to do a lot more posts about Terrablox! And I may make a page for it here too. :)

The first one

Posted on 26/08/2014

Alright so, I got the domain name a while back and was making slow progress with doing anything with it! So I decided to just start it out as a WordPress blog (FUTURE EDIT: This is no longer WordPress). Which might turn out for the best anyway, as doing enough content for my YouTube videos is proving very difficult so maybe this can serve as a good fallback! So far I only done a brief "About Me" page which tells you as much about me as I'd tell my enemies, so I'll try and fill that out more at a later point. I also added a "Games" page, seeing as that's one of my main focuses which currently links to my mostly old stuff, but expect that page to be filled a little quicker! :)