well 16 bytes for the index, 8 bytes for the number itself
so 24 overall. However, this doesn't seem entirely right, either. I tried out code that filled up a 6,000,000 size array for the single index, and it used up 300+ mb off the bat, when it should be 137. However, using the tricks we figured out over the past few pages, I've already got my program's memory usage down to manageable amount, even if its not perfect, so I think I'm good to go (I'm storing as array[i][j][k], too).

But I dont understand how you are getting such high memory usage when I have a table that is 500x500x12 in size in LUA, with this duplicated in an array in MMF and my memory usage is nowhere near that. Given that my array will never be completely full because empty slots I fill with 0 values, but that shouldnt make a difference? Are you still storing strings or only integers?

lua uses sparse tables, so even if you do something like:

table[1] = 1
table[1000000] = 2

it's only going to use space for those two "cells" (although the second will use more than the first because it's thrown into the hash table)

mmf's array object allocates space for whatever you set your XYZ for, whether you're using all of it or not; however, a given cell in mmf will use less memory than one in lua (because lua's have various overheads)

also from what i've seen, mmf's arrays won't show their memory usage in the debugger; you'll have to use task manager to see it (for example a 1000x1000x30 mmf array uses ~120MB)

Yep, use the task manager to see your memory usage.
With my 3 dimensional, 'fill as I go' array[x][y][z] method of filling it with increasing integer indexes, I'm getting about 54 MB base usage on the array, spiking up to 200 MB in the worst case. Predefining a 1000x500x12 one dimensional array in lua took up 300 MB.

Interesting result. I'll have to some further testing myself, since that seems to buck common logic if your worst case scenario is storing the same amount of data as your flat array. Unless your pre-definitions are themselves discarded as garbage, which is possible.

Ah! I will have to check task manager, I have been checking the MMF debugger all this time! I will check it out when I get home and post back with my results. Nevertheless I have learnt a lot from this thread.

EDIT: OK, it seems a 500x500x12 array in LUA uses ~84mb of ram by using map[i][j][k] as my indexes. If I also add a value into an MMF array at the same x,y,z indexes, the total ram usage only goes up to about 100mb. This was figured out by a quick blank frame i just made using some code:



So that means that the MMF array only uses about 16mb, compared to Lua which uses 3-4 times that ammount? Would I be better to store my map in one big flat array in LUA by multiplying the 3 indexes into one large integer instead of having 3 seperate indexes?

Nobodies gonna die from 84 MB of RAM usage, though, so its really not worth the hassle for you. Integer stacking is only really worthwhile if your data fulfills two qualities:
A) Its gotta be small numbers; low digits
B) It has to have a definitive size range

So storing 0-128 can be easily stacked, whereas 0-999999 is not worthwhile. Remember that you can stack numbers off any number, not just powers of 10- you could multiply it as: 16384x + 128y + z.

But yes retriever its an odd result, well above the projected amount of memory usage, for a single 'linear' array. If I had to guess, I'd say its possible that the exponential memory allocation is resulting in it allocating far more memory than the array actually needs? Perhaps I'll try it again, forcing garbage collection immediately after instantiating it, and see what happens. But I'm happy with the setup I have now.

Well most of the values are 0-16 range. Some values may be as high as about 4000 which is the reference to images within the background images object. That would be worst case scenario if I have to use that many images.
So stacking numbers would probably be a good idea in my case. I guess I can do a test and see what the difference is.
But either way, its much more than a MMF array. I understand it uses doubles so the memory will always be higher but I didnt realise it was that high.
Overall the RAM usage isnt that bad, but I still have to allow for all the potential images I want, plus character animations, pathfinding, etc I need to watch where I allocate resources to.

http://hocuspocus.taloncrossing.com/rii/fixedarray.zip

That's a link to a C library I just wrote for a generic fixed-size array of different storage types. It can create fixed-size storage for bytes, shorts, longs (ints), floats, or doubles. It stores 1000*500*12 doubles in 45MB, and as many shorts in 11MB. It avoids any problems with garbage collection.

Compared against native lua tables, averaged writes are anywhere between 15% - 400% faster, but reads are generally 50% - 80% slower. The library takes a hit on reads because of the C protocol it needs to return values (and the native table lookups conversely are very optimized).

To use it, add the line:
require "fixedarray"

to your source code, which will put the function "fixedarray" into the global table.

fixedarray can take 1 or 2 parameters. The first is the number of elements to allocate space for, and the second is the data type, which can be either numeric constant 0 - 4, or the strings "byte", "char", "short", "long", "int", "float", or "double". Any invalid type is automatically a double.

Calling fixedarray returns an object table, which contains 4 methods:

- at(index) [returns the value stored at the given index.]
- set(index, value) [saves value to the given index.]
- reset() [a fast operation to reset all elements to 0]
- destroy() [destroys the internal state, frees the memory, and empties the "self" table so you can't accidentally call functions on an invalid array]

Indexes are 1-based. Functions are tolerant of invalid indexes.

Usage example:


The methods do not need to use the : operator, but it will tolerate the operator if you use it (just be a tad bit slower).

Oh oh! Thank you very much, that looks like exactly what I needed. You should definitely think about including that with your standard release, it could definitely prove useful to more than just the two of us. One problem- I don't quite understand how Lua's library loading thingy works. How do you load the library in the lua file? I know lua has some funkiness with file directories. Thought I just had to alter package.path to tell it to search the proper directory, but I seem to be having troubles