Sometimes we need to store a lot of data in memory, but we find two restrictions:
1) VB6, as any 32 bits process, has a limit of using 2 GB RAM. It can be extended to 4 GB setting LARGEADDRESSAWARE but no more than that.
2) Some arrays (most) need to have available contiguous memory, and you usually hit that limit around 700 MB.
Here is a set of classes that use File Mapping.
It uses the name "File" but we are using it to create memory "files", to be able to use RAM memory outside our local 2 GB of our process.
This project is somewhat based in a very related project by Elroy. Thank you Elroy.
These classes can replace normal arrays with some code modifications.
Of course they are quite slower than normal arrays, but you can store a lot more data.
Sometimes it is not about surpassing the 700 MB or the 2 GB limits, but you may want to take out of the local memory some arrays to make room for other RAM consuming data that you need to handle in your in-process memory.
In this version there are 6 classes, that were the ones that I needed :cool:, but it is not difficult to add a new type not currently supported (such as Double or Boolean). If you need them you could modify the code from for example cLargeArrayDat.
Pay special attention to the Const cItemSizeBytes, it must the the size in memory of the data type in bytes. Also change all "As Date" to "As Boolean" (for example).
How to use them? for example:
All can be accessed like they were true arrays because the Item property was set to be the default property, with the exception of the UDT one, that VB6 does not allow to make Public properties of UDT and then I had to make it Friend, but doing so I lost the ability to set it as default, so for the UDT you need to access the elements like MylarArrayUDT.Item(1000).
Also regarding UDT. You can't access members of directly as with normal arrays, I mean:
will not work, you need to copy the UDT to a temporary local variable and make the changes there, and then assign it to the large array again, like this:
Also you will have to change all MyUDT occurrences to the actual name of your UDT.
For the cLargeArrayStrFix, you need to provide the string length at the time you first call ReDimArray.
Optionally you can change it at the ReDimPreserve or if the the first Redim is a ReDimPreserve it is then required.
The cLargeArrayStr does not allocate memory for all elements upfront because the elements lengths are variable, but it increases the memory size as needed.
When an element changes, it stores a new element at the end, and the space of the old element no longer in use is not reused. In other words: the FileMap always grows in size, it does not matter additions or modifications, they are all additions (regarding map size).
That worked well for my case because I didn't need to modify the strings once stored, but the code could be modified to reuse old string space when the new string is the same size or smaller than the old one.
Anyway there is a function ConsolitareRAM that can be used to get rid of all memory "holes" at once (the old, unused string spaces). But it takes some time to run.
The cLargeArrayStr by default also uses some local process memory, because it needs to keep the size and positions of all strings in the FileMap, so it has a couple of VB6 arrays to store that data.
But when these arrays grow too much and can no longer allocate space for them in the local memory, then it starts to use instead a cLargeArrayVar and a cLargeArrayLng to store that data. But then the operations become slower.
That is handled automatically, but it also provides a property LocalMemMode for the mode to be set explicitly.
In the RedimArray and RedimPreserve methods of all classes, there is a parameter nErrorModeSilent.
By default it is False. When there is not enough memory available then a normal error will be risen.
But if set to True, no error is risen when the operation failed and then you need to check the ErrorOnCreateMap property to know whether the operation was successful or not.
They have a property MapName that is not currently used by this code but that could be used to know the map name and be able to open the array from other process.
1) VB6, as any 32 bits process, has a limit of using 2 GB RAM. It can be extended to 4 GB setting LARGEADDRESSAWARE but no more than that.
2) Some arrays (most) need to have available contiguous memory, and you usually hit that limit around 700 MB.
Here is a set of classes that use File Mapping.
It uses the name "File" but we are using it to create memory "files", to be able to use RAM memory outside our local 2 GB of our process.
This project is somewhat based in a very related project by Elroy. Thank you Elroy.
These classes can replace normal arrays with some code modifications.
Of course they are quite slower than normal arrays, but you can store a lot more data.
Sometimes it is not about surpassing the 700 MB or the 2 GB limits, but you may want to take out of the local memory some arrays to make room for other RAM consuming data that you need to handle in your in-process memory.
In this version there are 6 classes, that were the ones that I needed :cool:, but it is not difficult to add a new type not currently supported (such as Double or Boolean). If you need them you could modify the code from for example cLargeArrayDat.
Pay special attention to the Const cItemSizeBytes, it must the the size in memory of the data type in bytes. Also change all "As Date" to "As Boolean" (for example).
| cLargeArrayLng.cls: | Array of Long |
| cLargeArrayVar.cls | Array of Variant |
| cLargeArrayDat.cls | Array of Date |
| cLargeArrayUdt.cls | Array of UDT |
| cLargeArrayStrFix.cls | Array of Fixed size String |
| cLargeArrayStr.cls | Array of variable size String |
How to use them? for example:
Code:
Dim MyLargeArrayDat As cLargeArrayDat
Set MyLargeArrayDat = New cLargeArrayDat
MyLargeArrayDat.ReDimArray 10000000
MyLargeArrayDat(1000) = Now
MyLargeArrayDat.ReDimPreserve 20000000
Debug.Print MyLargeArrayDat(1000), MyLargeArrayDat(20000000)Also regarding UDT. You can't access members of directly as with normal arrays, I mean:
Code:
MyLargeArrayUDT.Item(100).Field1 = 5
MyLargeArrayUDT.Item(100).Field2 = TrueCode:
Dim TempVarMyUDT As MyUDT
TempVarMyUDT = MyLargeArrayUDT.Item(100)
TempVarMyUDT.Field1 = 5
TempVarMyUDT.Field2 = True
MyLargeArrayUDT.Item(100) = TempVarMyUDTFor the cLargeArrayStrFix, you need to provide the string length at the time you first call ReDimArray.
Optionally you can change it at the ReDimPreserve or if the the first Redim is a ReDimPreserve it is then required.
The cLargeArrayStr does not allocate memory for all elements upfront because the elements lengths are variable, but it increases the memory size as needed.
When an element changes, it stores a new element at the end, and the space of the old element no longer in use is not reused. In other words: the FileMap always grows in size, it does not matter additions or modifications, they are all additions (regarding map size).
That worked well for my case because I didn't need to modify the strings once stored, but the code could be modified to reuse old string space when the new string is the same size or smaller than the old one.
Anyway there is a function ConsolitareRAM that can be used to get rid of all memory "holes" at once (the old, unused string spaces). But it takes some time to run.
The cLargeArrayStr by default also uses some local process memory, because it needs to keep the size and positions of all strings in the FileMap, so it has a couple of VB6 arrays to store that data.
But when these arrays grow too much and can no longer allocate space for them in the local memory, then it starts to use instead a cLargeArrayVar and a cLargeArrayLng to store that data. But then the operations become slower.
That is handled automatically, but it also provides a property LocalMemMode for the mode to be set explicitly.
In the RedimArray and RedimPreserve methods of all classes, there is a parameter nErrorModeSilent.
By default it is False. When there is not enough memory available then a normal error will be risen.
But if set to True, no error is risen when the operation failed and then you need to check the ErrorOnCreateMap property to know whether the operation was successful or not.
They have a property MapName that is not currently used by this code but that could be used to know the map name and be able to open the array from other process.