Is it possible to use non-DBF databases in Alaska+Express?

[Eugene Lutsenko]

I'm looking for a big challenge like this.
I need something interesting to do this winter.

Now we have to convince the Taxi Management company to go down this road with us.
They are inclined to want to invest in a web-based application written in a more mainstream language.

We don't think that it makes any sense.
They still need the big back office program and don't understand it because they are new investors and only see the world of the web.

Good luck to you, Roger! I would also like to make a professional version of the Eidos system over the winter. This version of the system should not have strict restrictions on the amount of data processed and the dimensions of the models that the current version has. I'm hoping to do this with a local portable database server like ADS or PGDBE (in the ISAM interface). I hope for your advice on this.

[Eugene Lutsenko]

I tried to make a program for experiments on creating large databases in the traditional way. I want to improve it later so that it works with ADS or PGDBE. But it does not catch errors of adding records when the database is larger than 2 GB. Also, when trying to create databases with a number of fields greater than 2048, there is also no error. Can you tell me what the reason is?

Code: Select all

     
     PROCEDURE AppSys
     // Рабочий стол остается окном приложения
     RETURN
     
     ********************************************************************************
     FUNCTION Main()
     
     LOCAL Getlist := {}, oProgress, oDialog
     LOCAL aSay[30], Mess97, Mess98, Mess99               // Массив сообщений отображаемых стадий исполнения (до 30 на экране)

        DC_IconDefault(1000)

        SET DECIMALS TO 15
        SET DATE GERMAN
        SET ESCAPE On

********************************************************************

mNField  = 2048
mNRecord = 1000000000


mNField  = 10
mNRecord = 10000000

@1,1  DCGROUP oGroup1 CAPTION 'Задайте параметры базы данных:' SIZE 40.0, 3.5

@1,2  DCSAY "Количество полей:"                  PARENT oGroup1 
@1,20 DCSAY "" GET mNField  PICTURE "##########" PARENT oGroup1 

@2,2  DCSAY "Количество записей:"                PARENT oGroup1 
@2,20 DCSAY "" GET mNRecord PICTURE "##########" PARENT oGroup1 

DCREAD GUI;
      TO lExit ;
      FIT;
      ADDBUTTONS;
      MODAL;
      TITLE 'Эксперименты с ADS и PGDBU'
      IF lExit
         ** Button Ok
      ELSE
         QUIT
      ENDIF
********************************************************************

   Wsego = mNRecord

   // Отображение стадии исполнения. Будет написано прямо в окне Progress-bar
   d = 0
   @0,0 DCGROUP oGroup1 CAPTION 'Стадии исполнения процесса' FONT "6.Helv" SIZE 105+d, 2.5 PARENT oTabPage1
   @4,0 DCGROUP oGroup2 CAPTION 'Прогноз времени исполнения' FONT "6.Helv" SIZE 105+d, 5.0 PARENT oTabPage2
   
    s = 1
   @s++,1 DCSAY " " SAYSIZE 100 SAYOBJECT aSay[ 1] FONT "10.Helv"
    s++
   @s++,1 DCSAY " " SAYSIZE 100 SAYOBJECT oSay97   FONT "10.HelvBold"
    s++
   @0.2+s++,1 DCSAY " " SAYSIZE 100 SAYOBJECT oSay98 FONT "9.Helv Bold" COLOR GRA_CLR_BLUE
   @1.5+s  ,1 DCSAY " " SAYSIZE 100 SAYOBJECT oSay99 FONT "9.Helv Bold" COLOR GRA_CLR_BLUE
   
   @s  ,1 DCPROGRESS oProgress ;
          SIZE 95,1.5 ;
          PERCENT ;
          EVERY 1;                          // Кол-во обновлений изображения
          MAXCOUNT Wsego; 
          COLOR GRA_CLR_CYAN                // Цвет полосы
   @s++,97 DCPUSHBUTTON  CAPTION '&Cancel' ;
          ACTION {||lOk:=.T.} OBJECT oButton ;
          SIZE 7,1.5
   DCREAD GUI ;
          TITLE 'Создание базы данных "BigData.dbf"' ;
          PARENT @oDialog  ;
          FIT ;
          EXIT ;
          MODAL

   oDialog:alwaysOnTop = .T.       // Окно открывается на переднем плане
   oDialog:show()

   ****** Обработка ошибки ******************
   bError := ErrorBlock( {|e| Break(e)} )           // установить новый кодовый блок обработки ошибок
   BEGIN SEQUENCE                                   // код нормального исполнения

         *** код нормального исполнения

         aStructure := { { "NumbRecord"  , "N", 19, 0} }
         
         FOR j=1 TO mNField
             FieldName = "F"+ALLTRIM(STR(j,19))
             AADD(aStructure, { FieldName  , "N", 19, 7 })
         NEXT
         DbCreate( "BigData.dbf", aStructure )

   RECOVER                                          // код обработки ошибки
         aMess := {}
         AADD(aMess, "Возникла ошибка при попытке создания БД с числом полей: "+ALLTRIM(STR(mNField)))
         LB_Warning(aMess)
         MsgBox('')
         QUIT
   ENDSEQUENCE 
   ErrorBlock( bError )                             // переустановить старый кодовый 
   ******************************************

   // Начало отсчета времени для прогнозирования длительности исполнения
   Time_progress = 0
   // Прошло секунд с начала процесса
   // Процесс может идти больше суток, поэтому для определения
   // во всех случаях вычисляется время, прошедшее с начала года
      T_Mess1    = "Начало:"+" "+TIME()            // Начало
      Sec_1      = (DOY(DATE())-1)*86400+SECONDS()
      PUBLIC T1 := (DOY(DATE())-1)*86400+SECONDS()        // Время предыдущей индикации процесса исполнения
      PUBLIC T2 := (DOY(DATE())-1)*86400+SECONDS()+1      // Текущее время (1-й раз оно заметно больше T1 чтобы было отображение)
      PUBLIC T1tp := T1
      PUBLIC T2tp := T2
   *********************************************************************************

   ****** Обработка ошибки ******************
   bError := ErrorBlock( {|e| Break(e)} )           // установить новый кодовый блок обработки ошибок
   BEGIN SEQUENCE                                   // код нормального исполнения

         *** код нормального исполнения

         aSay[ 1]:SetCaption('Идет процесс создания базы данных: "BigData.dbf"')

         USE BigData EXCLUSIVE NEW

         FOR r=1 TO mNRecord

             APPEND BLANK
             REPLACE NumbRecord WITH r

             lOk = Time_Progress (++Time_Progress, mNRecord, oProgress, lOk )

         NEXT

   RECOVER                                          // код обработки ошибки
         CLOSE ALL
         aMess := {}
         AADD(aMess, "Возникла ошибка при попытке добавления в БД записи N=й: "+ALLTRIM(STR(r)))
         LB_Warning(aMess)
         MsgBox('')
         QUIT
   ENDSEQUENCE 
   ErrorBlock( bError )                             // переустановить старый кодовый 
   ******************************************

   CLOSE ALL

   oSay97:SetCaption(oSay97:caption)
   oButton:SetCaption('&Ok')          // Деструктурирование окна отображения графического Progress-bar
   oButton:activate := {||PostAppEvent(xbeP_Close,,,oDialog)}  //<<<<<< Add This
   DC_AppEvent( @lOk )
*  PostAppEvent(xbeP_Activate,,,DC_GetObject(GetList,'DCGUI_BUTTON_OK'))       // Роджер
   oDialog:Destroy()

   aMess := {}
   AADD(aMess, 'База данных: "BigData.dbf" успешно создана')
   LB_Warning(aMess, 'Эксперименты с "BigData.dbf"')

   RETURN NIL

   ***********************************************************************************************************************

   FUNCTION LB_Warning( message, ctitle )
   
     LOCAL aMsg := {}
   *  DEFAULT cTitle TO ''
     IF valtype(message) # 'A'
       aadd(aMsg,message)
     ELSE
       aMsg := message
     ENDIF
     IF LEN(ALLTRIM(cTitle)) > 0
        DC_MsgBox(10,10,aMsg,cTitle)
     ELSE
        DC_MsgBox(10,10,aMsg,'Эксперименты с "BigData.dbf"')
     ENDIF
   
   RETURN NIL

   ***********************************************************************************************************************

***********************************************************************************************************
****** Графический прогресс-бар (на основе примера XSample_14() xdemo.exe)
***********************************************************************************************************
FUNCTION Time_Progress(Time_Progress, Wsego, oProgress, lOk )

    LOCAL nMaxCount := Wsego
    xtime     = Time_Progress

    ** Отображение занимает очень много времени, поэтому показывать прогресс не чаще чем через 0.1 секунды (как в PercTimeVisio())

    T2tp = (DOY(DATE())-1)*86400+SECONDS()          // Текущее время
    IF T2tp - T1tp > 0.1 .OR. xtime = Wsego         // Время в секундах или 100%

*      aSay[mPTVnumb]:SetCaption(mPTVmess+' '+ALLTRIM(STR(mNumPP/Wsego*100,15,7))+'%')

       *** Индикация времени исполнения

       ***** Процесс может идти больше суток, поэтому для определения
       ***** во всех случаях вычисляется время, прошедшее с начала года
*      T_Mess1 = "Начало:"+" "+TIME()           // Начало

       ***** Прошло секунд с начала процесса
       PUBLIC T_Mess2 := "ch:mi:se"
       Sec_2   = (DOY(DATE())-1)*86400+SECONDS() - Sec_1
       ch2 = INT(Sec_2/3600)                    // Часы
       mm2 = INT(Sec_2/60)-ch2*60               // Минуты
       cc2 = Sec_2-ch2*3600-mm2*60              // Секунды
       T_Mess2 = "Прошло:"+" "+ALLTRIM(STRTRAN(T_Mess2,"ch",STR(ch2,19)))
       T_Mess2 = STRTRAN(T_Mess2,"mi",STRTRAN(STR(mm2,2)," ","0"))
       T_Mess2 = STRTRAN(T_Mess2,"se",STRTRAN(STR(cc2,2)," ","0"))
       *@19,2 SAY T_Mess2+" всего: "+ALLTRIM(STR(Sec_2,17))+" сек."

       PUBLIC T_Mess3 := "ch:mi:se"             // Осталось
       Sec_3 = Sec_2*Wsego/xtime                // Прогн.длит.исп. в секундах
       ch3 = INT(Sec_3/3600)                    // Часы
       mm3 = INT(Sec_3/60)-ch3*60               // Минуты
       cc3 = Sec_3-ch3*3600-mm3*60              // Секунды
       T_Mess3 = ALLTRIM(STRTRAN(T_Mess3,"ch",STR(ch3,19)))
       T_Mess3 = STRTRAN(T_Mess3,"mi",STRTRAN(STR(mm3,2)," ","0"))
       T_Mess3 = STRTRAN(T_Mess3,"se",STRTRAN(STR(cc3,2)," ","0"))
       *@20,2 SAY T_Mess3+" всего: "+ALLTRIM(STR(Sec_3,17))+" сек."

       PUBLIC T_Mess4 := "ch:mi:se"             // Окончание
       Sec_4 = Sec_1 + Sec_3 - (DOY(DATE())-1)*86400
       ch4 = INT(Sec_4/3600)                    // Часы
       mm4 = INT(Sec_4/60)-ch4*60               // Минуты
       cc4 = Sec_4-ch4*3600-mm4*60              // Секунды
       T_Mess4 = "Окончание:"+" "+ALLTRIM(STRTRAN(T_Mess4,"ch",STR(ch4,19)))
       T_Mess4 = STRTRAN(T_Mess4,"mi",STRTRAN(STR(mm4,2)," ","0"))
       T_Mess4 = STRTRAN(T_Mess4,"se",STRTRAN(STR(cc4,2)," ","0"))
       *@21,2 SAY T_Mess4+" всего: "+ALLTRIM(STR(Sec_4,17L())+" сек.с нач.суток")

       PUBLIC T_Mess5 := "Средн.время обработки 1-го объекта: ch:mi:se"
       Sec_5 = Sec_2/xtime
       ch5 = INT(Sec_5/3600)                    // Часы
       mm5 = INT(Sec_5/60)-ch5*60               // Минуты
       cc5 = Sec_5-ch5*3600-mm5*60              // Секунды
       T_Mess5 = ALLTRIM(STRTRAN(T_Mess5,"ch",STR(ch5,19)))
       T_Mess5 = STRTRAN(T_Mess5,"mi",STRTRAN(STR(mm5,2)," ","0"))
       T_Mess5 = STRTRAN(T_Mess5,"se",STRTRAN(STR(cc5,2)," ","0"))
       *@22,2 SAY T_Mess5+" всего: "+ALLTRIM(STR(Sec_5,17))+" сек."

       PUBLIC T_Mess6 := "ch:mi:se"             // Осталось
       Sec_6 = Sec_3 - Sec_2
       ch6 = INT(Sec_6/3600)                    // Часы
       mm6 = INT(Sec_6/60)-ch6*60               // Минуты
       cc6 = Sec_6-ch6*3600-mm6*60              // Секунды
       T_Mess6 = "Осталось:"+" "+ALLTRIM(STRTRAN(T_Mess6,"ch",STR(ch6,19)))
       T_Mess6 = STRTRAN(T_Mess6,"mi",STRTRAN(STR(mm6,2)," ","0"))
       T_Mess6 = STRTRAN(T_Mess6,"se",STRTRAN(STR(cc6,2)," ","0"))
       *@23,2 SAY T_Mess6+" всего: "+ALLTRIM(STR(Sec_6,17))+" сек."

       Mess98 = T_Mess1+SPACE(142-LEN(T_Mess1)-LEN(T_Mess4))+T_Mess4  // Начало, окончание (прогноз) 145
       oSay98:SetCaption(Mess98);oSay98:SetCaption(oSay98:caption)

       Mess99 = T_Mess2+SPACE(144-LEN(T_Mess2)-LEN(T_Mess6))+T_Mess6  // Прошло, осталось (прогноз)  146
       oSay99:SetCaption(Mess99);oSay99:SetCaption(oSay99:caption)

       DC_GetProgress( oProgress, Time_Progress, Wsego )              // Отображение графического Progress-bar

       DC_AppEvent( @lOk, 0, .01 )

       T1tp = T2tp
    ENDIF

RETURN lOk

[rdonnay]

Why do you have the error trapping code in your sample?

[Eugene Lutsenko]

Hi, Roger!

I would like to set such parameters of the database being created that are not possible for DBF in Alaska. I was hoping that if too many fields were set, for example 100,000, or such a number of database records that when adding another record, the database size would become larger than 2 GB, then an error would occur. But for some reason, the error situation is not handled correctly. A 2 GB database with a broken structure is silently created. When trying to create a database with an unrealistically large number of fields, a database with a smaller number of fields is created (for example, 1808) and the error does not occur. I was hoping that when this program starts to catch errors correctly and issue messages about them, then in the next step it would be possible to try to use ADS/PGDBE to create databases with such parameters that are generally impossible for DBF. This program was conceived as a developing training example for mastering the basics of ADS/ PGDBE (in the ISAM interface). I would like to master the creation of databases of very large dimensions, as well as their indexing and filtering, and searching for them by index, as well as other operations (deleting tagged records, etc.)

[Auge_Ohr]

hi Eugene,

have you think about "compress" Data

Integer use "less" than Float Number so use Integer Number
big Integer Number can be "store" as "binary" ( W2BIN / BIN2W )

instead of Type "N", 19 you can use Type "C",4 to store "binary" 32 Bit numeric Value
you can build a "large" String and find Part with SUBSTR(cString,nstart,4 )
so you can hold much more "Data" in same Size of DBF

[Eugene Lutsenko]

hi Eugene,

have you think about "compress" Data

Integer use "less" than Float Number so use Integer Number
big Integer Number can be "store" as "binary" ( W2BIN / BIN2W )

instead of Type "N", 19 you can use Type "C",4 to store "binary" 32 Bit numeric Value
you can build a "large" String and find Part with SUBSTR(cString,nstart,4 )
so you can hold much more "Data" in same Size of DBF

Yes, I've thought about it and I know it. But this is inconvenient for programming.

[rdonnay]

Back in the days of Clipper, Nantucket documented their database engine, which they called an RDD (Replaceable Data Driver).
Many 3rd party developers took that spec and wrote their own database engine.
In fact, this spawned the Advantage Server RDD by a company named Extended Systems, here in Boise, ID USA.
That company was sold to Sybase which was sold to SAP , which uses that technology in house and is why it is still supported in the 32-bit community.
Sybase created the SQL technology for ADS.

Alaska emulated that concept of the RDD by creating DBE's, however they never gave us the tools to create our own.
They believe that it is too complicated for 3rd party developers to understand it.

It may be possible to use the concept of "virtual fields", but I don't know how practical this would be.
Indexing would be difficult.

The basic idea of virtual fields is creating Data Objects and storing them as binary data using Var2Bin() and Bin2Var().
I don't think there are any limitations on the number of elements in a Data Object.
Those objects can then be store in a MEMO field.
I'm not sure how practical this would be or what to expect in performance, but I like the idea because I think it would be a fun exercise.
Unfortunately, you cannot use any of the SQL features of ADS with these virtual fields.

If I can find some spare time, I may write a small sample program.

[Eugene Lutsenko]

Thank you, Roger, for your willingness to help. But I prefer to use more understandable and standard tools. That's why I chose Alaska + Express, not Harbor. I would like to develop the example of creating databases that I gave above. I would like to try to remove some DBF limitations by using ADS or PGDBE.