ZAP and PACK command for DBF files - error Win 11

[k-insis]

> even if Alaska released a 64bit version it would help me

If that will happen at all.

It is possible that it is caused by an older version of Xbase++, I have a release 554 of Xbase++ Foundation edition 2.0.
It is possible that this is caused by an older version of Xbase++, I have a release ,
I didn't buy a newer update because I only maintain my old programs and it's not worth it to me to invest almost 1000 Euros in a new version, even if Alaska released a 64bit version it would help me, but because of the RAM usage limitations of 2GB it wouldn't help me much.

[SlavkoDam]

Hi,

For PACK it would be something that is part of "topdown" library FUNCTION tdPack (there is full source available). It does pack with copying data to new file and later renaming it into original while original is left as backup.

K-insis, the same method is used by the PACK command in Xbase++/Clipper. Xbase++ creates a temp file "xppxxxxx.tmp", where "xxxxx" are random characters. The same method is used by the PACK option in the old Clipper DBU program. So, "topdown" library did not invent anything new, its already included in Xbase++/Clipper.

Victorio, I understand you programming concept and its o.k. 100 work areas and dbs is quite a number. How do you work, do you call SELECT 0 100 times, and then open dbfs in specific numbers optionally? I don't think its 100% reliable to select by number and not by name/alias. When you write:

SELECT 24
use &nazovdbf24 ALIAS TEMPVLA EXCLUSIVE

are you sure that work area is closed, you don't check it. Maybe that is the problem, that you USE ... EXCLUSIVE in an empty area, and Xbase++/Clipper don't "like" that.

I don't want to break down your programming concept. I want to suggest to try alternative way by using USE NEW EXCLUSIVE and SELECT name/alias for PACK/ZAP, and to check if your problem still persists. Maybe Xbase++/Clipper "prefer" that method for PACK/ZAP.

Slavko

[Victorio]

> even if Alaska released a 64bit version it would help me
If that will happen at all.

I mean , really no, because Xbase++ have small programmers community to force the Xbase++ developer to rework the system into a 64bit version...


Jimmy : thanks, I know about Harbour, also have downloaded it, but I haven't gotten around to trying it yet.

[Victorio]

Slavko, ok,
I can try it if the problem occurs again, the problem is that the problem occurs randomly with some client, or with me, but if, for example, I restart the PC, then it is already running, or sometimes not. I can't simulate it to crash all the time.
thx

[Auge_Ohr]

Jimmy : thanks, I know about Harbour, also have downloaded it, but I haven't gotten around to trying it yet.

you just need

Code: Select all

SET DBFLOCKSCHEME TO DB_DBFLOCK_HB64

to use DBF > 2 GB under Harbour

Sample for Harbour 64 BIt

Code: Select all

#include "dbinfo.ch"
REQUEST HB_GT_WIN_DEFAULT  // Console Output

proc main()
local i, n := int(2^22) + 10, nStep := int ( n / 50 )

   SET DBFLOCKSCHEME TO DB_DBFLOCK_HB64

   if !file("tst4GB.dbf")
      ? "Building tst4GB.dbf..., be patient ..."
      ? REPLICATE ( Chr ( 177 ), 50 )
      DevPos( ROW(), COL()-50 )
      // Note : Harbour 64 Bit can use Type C > 256 Length
      dbcreate("tst4GB.dbf",{{"F1","C",1023,0}, {"id","+",4,0}})
      use tst4GB exclusive
      for i:=1 to n
         dbappend()
         replace F1 with str(i,10)
         IF i%nStep = 0 
	      ?? Chr ( 219 )
         endif
      next i
      close
      ?? " 100% done"
   endif
   use tst4GB shared alias tst1 new
   use tst4GB shared alias tst2 new
   ? "records:", tst1->(lastrec()), ;
      ", size:", tst1->(header())+tst1->(lastrec())*tst1->(recsize())

   ? "tst1 goto:", lastrec()-1, tst1->(dbgoto(lastrec()-1))
   ? "tst1 recno():", tst1->(recno()), "->", alltrim(tst1->(fieldget(1))), "->" , tst1->(fieldget(2))
   ? "tst2 goto:", lastrec()-1, tst2->(dbgoto(lastrec()-1))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))
   ? "FLOCK tst1: ", tst1->(flock())

   n := int(2^20+10)

   ? "tst2 goto:", n-1, tst2->(dbgoto(n-1))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))

   ? "tst2 goto:", n-2, tst2->(dbgoto(n-2))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))


   n := int(2^21+10)

   ? "tst2 goto:", n-1, tst2->(dbgoto(n-1))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))

   ? "tst2 goto:", n-2, tst2->(dbgoto(n-2))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))

   n := lastrec()

   ? "tst2 goto:", n-1, tst2->(dbgoto(n-1))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))

   ? "tst2 goto:", n-2, tst2->(dbgoto(n-2))
   ? "tst2 recno():", tst2->(recno()), "->", alltrim(tst2->(fieldget(1))), "->" , tst2->(fieldget(2))

   close all
   inkey ( 0 )
return

[Victorio]

ok, Jimmy, thanks for advice, but I have not problem with 2GB DBF but with more than 2GB use of memory , for arrays and also variables.
because I read large TXT files and store its content to variable or array, and processing it in memory for maximal speed.
now I have on server 24 CPU, and 48GB RAM, this is enough for multiprocessing , on every CPU running separate process, but can not use more than 2GB RAM.
Because this I must processed TXT file divide to smallest parts and processing it invidiually. Works fine when parts are about 100MB.
After processing I must join results.
Modify system to use 3GB RAM solve anything.

Without this 2GB limit source code wound be simplier.
But my application work on 80 offices around country more than 30 years, I mean it is enough, and prospectively, I don't want to invest a lot of effort and money into it
I considered converting it to C++ again, where it used to be, and reworked it into Xbase++, while it was very, very fast in C++, I just don't know what the limits are in C++, moreover, I would only make external modules demanding processing speed in C++

[k-insis]

If I remember correctly - Victorio there is setting in xbase that controls how much of that 3GB accessible RAM
is used for different things by compiled exe and they left well below 1.5GB for usual variables.

I remember seeing it and it splits memory for variables, buffers, stack and such, but I am not sure where and when; it even might be post here on this forum or german xbaseforum. It was in context of with large address awereness or 3GB patch.

Long term it might be worth converting such routine to python and put it onto REST server side for fast computing.

ok, Jimmy, thanks for advice, but I have not problem with 2GB DBF but with more than 2GB use of memory , for arrays and also variables.
because I read large TXT files and store its content to variable or array, and processing it in memory for maximal speed.
now I have on server 24 CPU, and 48GB RAM, this is enough for multiprocessing , on every CPU running separate process, but can not use more than 2GB RAM.
Because this I must processed TXT file divide to smallest parts and processing it invidiually. Works fine when parts are about 100MB.
After processing I must join results.
Modify system to use 3GB RAM solve anything.

Without this 2GB limit source code wound be simplier.
But my application work on 80 offices around country more than 30 years, I mean it is enough, and prospectively, I don't want to invest a lot of effort and money into it
I considered converting it to C++ again, where it used to be, and reworked it into Xbase++, while it was very, very fast in C++, I just don't know what the limits are in C++, moreover, I would only make external modules demanding processing speed in C++

[Auge_Ohr]

hi Vitorio,

you can use "Vitual" File-System under 32 Bit to open File with 7 000 000 Lines

Code: Select all

*   vfFileRead Class                                              *
*******************************************************************
#include "hbclass.ch"
#include "fileio.ch"

#define vf_DEF_READ_SIZE  	4096
#define vf_DELIMITER		hb_eol()
#define vf_REC_MAP_STEP		100000

CREATE CLASS vfFileRead

what you need are Function to "Support Unsigned Long Long"
like : BIN2ULL and ULL2BIN

[Victorio]

yes, main problem for me is limitation to array elements, when number elements has about 1 milion, there is problem (at this moment I do not know one or 10 milions, because i had several arrays in proces.
but my solution is control numbers of elements and processing by parts.

[Victorio]

Jimmy, I have bigger files , more than 12 mil. rows sometimes file can be more than 20 milions.
better way is processing it by parts, processing is faster when working with smaller files, it's just that programming is more complicated