From a5817d2410f65c3a055e4c1ec212270aed50186d Mon Sep 17 00:00:00 2001 From: Mhd Sulhan Date: Sun, 27 Jul 2014 20:22:12 +0700 Subject: Add documentation. --- doc/dev/GOAL | 263 ++++++++++ doc/dev/NOTES | 122 +++++ doc/dev/TODO | 10 + doc/dev/slog | 206 ++++++++ doc/dev/test | 1 + doc/dev/vos-sketch.odg | Bin 0 -> 29491 bytes doc/dev/vos.test.create.log | 113 +++++ doc/dev/vos.test.create.mem.log | 50 ++ doc/dev/vos.test.join.log | 39 ++ doc/dev/vos.test.sort-00.log | 65 +++ doc/user/example/album.data | 5 + doc/user/example/artist.data | 5 + doc/user/example/create_artist_album.vos | 19 + doc/user/example/filter_artist_album.vos | 21 + doc/user/example/join_artist_album.vos | 16 + doc/user/example/sort_album.vos | 9 + doc/user/example/sort_artist.vos | 9 + doc/user/image/create_stmt.png | Bin 0 -> 13751 bytes doc/user/image/field_clause.png | Bin 0 -> 11335 bytes doc/user/image/filter_clause.png | Bin 0 -> 7554 bytes doc/user/image/join_rules.png | Bin 0 -> 5929 bytes doc/user/image/join_stmt.png | Bin 0 -> 21461 bytes doc/user/image/load_stmt.png | Bin 0 -> 6493 bytes doc/user/image/set_stmt.png | Bin 0 -> 1949 bytes doc/user/image/sort_stmt.png | Bin 0 -> 12914 bytes doc/user/style/page.css | 68 +++ doc/user/vos_user_manual.html | 809 +++++++++++++++++++++++++++++++ 27 files changed, 1830 insertions(+) create mode 100644 doc/dev/GOAL create mode 100644 doc/dev/NOTES create mode 100644 doc/dev/TODO create mode 100644 doc/dev/slog create mode 100644 doc/dev/test create mode 100644 doc/dev/vos-sketch.odg create mode 100644 doc/dev/vos.test.create.log create mode 100644 doc/dev/vos.test.create.mem.log create mode 100644 doc/dev/vos.test.join.log create mode 100644 doc/dev/vos.test.sort-00.log create mode 100644 doc/user/example/album.data create mode 100644 doc/user/example/artist.data create mode 100644 doc/user/example/create_artist_album.vos create mode 100644 doc/user/example/filter_artist_album.vos create mode 100644 doc/user/example/join_artist_album.vos create mode 100644 doc/user/example/sort_album.vos create mode 100644 doc/user/example/sort_artist.vos create mode 100644 doc/user/image/create_stmt.png create mode 100644 doc/user/image/field_clause.png create mode 100644 doc/user/image/filter_clause.png create mode 100644 doc/user/image/join_rules.png create mode 100644 doc/user/image/join_stmt.png create mode 100644 doc/user/image/load_stmt.png create mode 100644 doc/user/image/set_stmt.png create mode 100644 doc/user/image/sort_stmt.png create mode 100644 doc/user/style/page.css create mode 100644 doc/user/vos_user_manual.html diff --git a/doc/dev/GOAL b/doc/dev/GOAL new file mode 100644 index 0000000..7578e06 --- /dev/null +++ b/doc/dev/GOAL @@ -0,0 +1,263 @@ +Vos Goals +---------- + Taken from CoSort Technical Specifications. + + +Legend: +- : unimplemented ++ : implemented += : on going/half done +? : is it worth/why/what is that mean + + +Ease of Use +----------- + +- Processes record layouts and SQL­like field definitions from central data + dictionaries. + +- Converts and processes native COBOL copybook, Oracle SQL*Loader control + file, CSV, and W3C extended log format (ELF) file layouts. + +- SortCL data definition files are a supported MIMB metadata format. + +- Mix of on­line help, pre­runtime application validation, and runtime + error messages. + +- Leverages centralized application and file layout definitions (metadata + repositories). + += Reports problems to standard error when invoked from a program, or + to an error log. + +- Runs silently or with verbose messaging without user intervention. + +- Allows user control over the amount of informational output produced. + +- Generates a query­ready XML audit log for data forensics and privacy + compliance. + += Describes commands and options through man pages and on­line documentation. + + it's half done because the program is always moving to a new features. + it's not wise to mark this as 'done'. + +- Easy­to­use interfaces and seamless third­party sort replacements preclude + the need for training classes + + +Resource Control +---------------- + ++ Sets and allows user modification of the maximum and minimum number of + concurrent sort threads for sorting on multi­CPU and multi­core systems. + + using PROCESS_MAX variable. + ++ Uses a specified directory, a combination of directories, for temporary work + files. + + using PROC_TMP_DIR variable. + ++ Limits the amount of main and virtual memory used during sort operations. + + using PROCESS_MAX_ROW variable. + + Since input file size is unpredictable and a human is still need to + run the program, the amount of program memory still cannot decide by + human. What if it's set to 1 kilobytes ?. + ++ Sets the size of the memory blocks used as physical I/O buffers. + + using FILE_BUFFER_SIZE variable. + + +Input and Output  +---------------- + += Processes any number of files, of any size, and any number of records, + fixed or variable length to 65,535 bytes passed from an input procedure, + from stdin, a named pipe, a table in memory, or from an application program. + + - TODO: from stdin + - TODO: from a named pipe. + - TODO: from a table in memory. + - TODO: from an application program. + +? Supports the use of environment variables. + + for what ? + += Supports wildcards in the specification of input and output files, as well + as absolute path names and aliases. + + - TODO: supports wildcards in the specification of input files. + ++ Accepts and outputs fixed­ or variable­length records with delimited field. + +? Generates one or more output files, and/or summary information, including + formatted and dashboard­ready reports. + +- Returns sorted, merged, or joined records one (or more) at a time to an output + procedure, to stdout (or named pipe), a table in memory, one or more new or + existing files, or to a program. + +- Outputs optional sequence numbers with each record, at any starting value, for + indexed loads and/or reports. + + +Record Selection and Grouping +----------------------------- + += Includes or omits input or output records using field­to­field or field­constant + comparisons. + + TODO: field-to-field comparisons + +- Compares on any number of data fields, using standard and alternate collating + sequences. + ++ Sorts and/or reformats groups of selected records. + + using SORT and CREATE statement. + ++ Matches two or more sorted or unsorted files on inner and outer join criteria using + SQL­based condition syntax. + + using JOIN with '+' or '-' statement. + +- Skips a specified number of records, bytes, or a file header or footer. + +- Processes a specified number of records or bytes, including a saved header. + +- Eliminates or saves records with duplicate keys. + + +Sort Key Processing +------------------- + ++ Allows any number of key fields to be specified in ascending or + descending order. + + using SORT x by x.f1 ASC; or + using SORT x by x.f1 DESC; + ++ Supports any number of fields from 0 to 65,535 bytes in length. + + almost unlimited, the limit is your memory. + ++ Orders fixed position fields, or floating fields with one or more + delimiters. + +- Supports numeric keys, including all C, FORTRAN, and COBOL data types. + +- Supports single­ and multi­byte character keys, including ASCII, EBCDIC, + ASCII in EBCDIC sequence, American, European, ISO and Japanese timestamps, + and natural (locale­dependent) values, as well as Unicode and double­byte + characters such as Big5, EUC­TW, UTF32, and S­JIS. + +- Allows left or right alignment and case shifting of character keys. + +- Accepts user compare procedures for multi­byte, encrypted and other + special data. + +- Performs record sequence checking. + ++ Maintains input record order (stability) on duplicate keys. + +- Controls treatment of null fields when specifying floating + (character separated) keys. + +- Collates and converts between many of the following data types (formats): + --- + + +Record Reformatting +------------------- + ++ Inserts, removes, resizes, and reorders fields within records; defines new + fields. + +- Converts data in fields from one format to another either using internal + conversion. + +- Maps common fields from differently formatted input files to a uniform sort + record. + += Joins any fields from several files into an output record, usually based on a + condition. + + using JOIN statement. current support only in joining two input files. + +- Changes record layouts from one file type to another, including: Line + Sequential, Record Sequential, Variable Sequential, Blocked, Microsoft Comma + Separated Values (CSV), ACUCOBOL Vision, MF I­SAM, MFVL, Unisys VBF, VSAM + (within UniKik MBM), Extended Log Format (W3C), LDIF, and XML. + +- Maps processed records to many differently formatted output files, including + HTML. + +- Writes multiple record formats to the same file for complex report + requirements. + +- Performs mathematical expressions and functions on field data (including + aggregate data) to generate new output fields. + +- Calculates the difference in days, hours, minutes and seconds betweeen + timestamps. + + +Field Reformatting/Validation +----------------------------- + +- Aligns desired field contents to either the left or right of the target + field, where any leading or trailing fill characters from the source are + moved to the opposite side of the string. + +- Processes values from multi­dimensional, tab­delimited lookup files. + +- Creates and processes sub­strings of original field contents, where you can + specify a positive or negative offset and a number of bytes to be contained + in the sub­string. + +- Finds a user­specified text string in a given field, and replaces all + occurrences of it with a different user­specified text string in the target + field. + +- Supports Perl Compatible Regular Expressions (PCRE), including pattern + matching. + +- Uses C­style “iscompare” functions to validate contents at the field level + (for example, to determine if all field characters are printable), which can + also be used for record­filtering via selection statements. + +- Protects sensitive field data with field­level de­identification and AES­256 + encryption routines, along with anonymization, pseudonymization, filtering + and other column-level data masking and obfuscation techniques. + +- Supports custom, user­written field­level transformation libraries, and + documents an example of a field­level data cleansing routine from + Melissa Data (AddressObject). + + +Record Summarization +-------------------- + +- Consolidates records with equal keys into unique records, while totaling, + averaging, or counting values in specified fields, including derived + (cross­calculated) fields. + +- Produces maximum, minimum, average, sum, and count fields. + +- Displays running summary value(s) up to a break (accumulating aggregates). + +- Nreaks on compound conditions. + +- Allows multiple levels of summary fields in the same report. + +- Re­maps summary fields into a new format, allowing relational tables. + +- Ranks data through a running count with descending numeric values. + +- Writes detail and summary records to the same output file for structured + reports. diff --git a/doc/dev/NOTES b/doc/dev/NOTES new file mode 100644 index 0000000..92bf86c --- /dev/null +++ b/doc/dev/NOTES @@ -0,0 +1,122 @@ + sometimes i forgot why i write code like this. + -- S.T.M.L + +- follow linux coding style + +- priority of source code (4S) : + + stable + + simple + + small + + secure (this option does not need for this program) + +- keep as small as possible: + + remove unneeded space + + remove unneeded variable + +- write comment/documentation as clear as possible + +- learn to use: + + if (1 == var) + +- learn to avoid: + + (i < strlen(str)) + on loop statement because strlen() need temporary variable. + try, + l = strlen(str); + while (i < l) { ... } + +- use function in libc as much as possible; if not, wrap it! + + + +001 - I/O Relation between Statement +----------------------------------------------------------------------------- +LOAD is an input statement. + +SORT, CREATE, JOIN is an output statement, but it can be an input. +i.e: + + 1 - load abc ( ... ) as x; + 2 - sort x by a, b; + 3 - create ghi ( x.field, ... ) as out_x; + +file output created by sort statement in line 2 will be an input by create +statement in line 3. + + +002 - Why we need '2nd-loser' +----------------------------------------------------------------------------- + +to minimize comparison and insert in merge tree. + + + +003 - Why we need 'level' on tree node +----------------------------------------------------------------------------- + +list of input file to merge is A, B, C contain sorted data : + + A : 10, 11, 12, 13 (1st file) + B : 1, 12, 100, 101 (2nd file) + C : 2, 13, 200, 201 (3rd file) + +if we use tree insert algorithm: + + if (root < node) + insert to left + else + insert to right + +after several step we will get: + +B-12 + \ + C-13 + / +A-12 + +which result in not-a-stable sort, + + B-1 C-2 A-10 A-11 B-12 A-12 ... + +they should be, + + B-1 C-2 A-10 A-11 A-12 B-12 ... + +Even if we choose different algorithm in insert: + + if (root <= node) + insert to left + else + insert to right + +there is also input data that will violate this, i.e: + + A : 2, 13, 200, 201 (1st file) + B : 1, 12, 100, 101 (2nd file) + C : 10, 11, 12, 13 (3rd file) + + +004 - recursives call + thread + free on SunOS 5.10 +----------------------------------------------------------------------------- + +i did not investigate much, but doing a recursive call + thread + free cause +SIGSEGV on SunOS 5.10 system, but not in GNU/Linux system. This odd's found +whee testing on Solaris and by using dbx the SIGSEGV "sometimes" catched in +str_destroy, + + if (str->buf) + free(str->buf); <= dbx catch here + +and "sometimes" below that (but not in vos function/stack). + +i.e: + list_destroy(**ptr) + { + if (! (*ptr)) + return; + list_destroy((*ptr)->next); + free((*ptr)); + } + +and no, it's not about double free. diff --git a/doc/dev/TODO b/doc/dev/TODO new file mode 100644 index 0000000..3dc6001 --- /dev/null +++ b/doc/dev/TODO @@ -0,0 +1,10 @@ +>> +- add set variable + set process_compare_case_sensitive; (default) + set process_compare_case_notsensitive; + + set process_tmp_dir "/path/to/tmp/dir"; + set process_tmp_dir "/another/tmp/dir"; +<< DONE + +- Produces maximum, minimum, average, sum, and count fields. diff --git a/doc/dev/slog b/doc/dev/slog new file mode 100644 index 0000000..72e2185 --- /dev/null +++ b/doc/dev/slog @@ -0,0 +1,206 @@ + i have a odd habit: checking code every time + i get bored, which result an error some time. + this file prevent me to over checking it. + -- May Benot + +--- format --- ++ function_name +@check : XXXX XXXX +@last-check : year.month.day (last check) +@auditor : thisman@thatserver.com (last auditor) +@desc : fix algorithm +--- tamrof --- + + +vos_String +----------------------------------------------------------------------------- + ++ str_create +@check : X +@last-check : 2008.12.17 +@auditor : ms@kilabit.info + ++ str_append_c +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ str_append +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info +@desc : fix len increment + ++ str_detach +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ str_rtrim +@check : XX +@last-check : 2009.01.25 +@auditor : ms@kilabit.info +@desc : removed + ++ str_prune +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ str_destroy +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ str_raw_copy +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ str_raw_randomize +@check : XX +@last-check : 2009.01.25 +@auditor : ms@kilabit.info +@desc : 'x' should not be replaced + ++ str_raw_hash +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + + +vos_File +----------------------------------------------------------------------------- + ++ file_open +@check : X +@last-check : 2008.12.19 +@auditor : ms@kilabit.info + ++ file_read +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_write +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_fetch_until +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_skip_until +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_skip_space +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_destroy +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_raw_get_size +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + ++ file_raw_is_exist +@check : X +@last-check : 2009.01.25 +@auditor : ms@kilabit.info + + +vos_LL +----------------------------------------------------------------------------- + ++ ll_add +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ ll_link +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ ll_print +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ ll_destroy +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + + +vos_Field +----------------------------------------------------------------------------- + ++ field_soft_copy +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ field_add +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ field_print +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ _field_destroy +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + + +vos_Record +----------------------------------------------------------------------------- + ++ record_new +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ _record_cmp +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ record_add_field +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ record_add_row +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ record_prune +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ record_destroy +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + ++ record_print +@check : X +@last-check : 2009.01.26 +@auditor : ms@kilabit.info + diff --git a/doc/dev/test b/doc/dev/test new file mode 100644 index 0000000..ff6cde3 --- /dev/null +++ b/doc/dev/test @@ -0,0 +1 @@ ++ Accepts and outputs fixed­ or variable­length records with delimited field. diff --git a/doc/dev/vos-sketch.odg b/doc/dev/vos-sketch.odg new file mode 100644 index 0000000..45b86a6 Binary files /dev/null and b/doc/dev/vos-sketch.odg differ diff --git a/doc/dev/vos.test.create.log b/doc/dev/vos.test.create.log new file mode 100644 index 0000000..027c53f --- /dev/null +++ b/doc/dev/vos.test.create.log @@ -0,0 +1,113 @@ +2009.10.12 + +Comparing vos create process time by setting process max row and buffer size +============================================================================== + + is it disk or algorithm ? + +- file input size : 257,985,910 byte (~ 250 MB) +- format of input field: + + '\'':field01:'\''::';', + +- format of output field: + + '':field01:''::'|', + +- number of field in input & output : 11 field +- process max : 2 (this option does not effect process actually) + + +system copy time +============================================================================== + + real 0m2.906s + user 0m0.010s + sys 0m0.747s + + +vos load+create +============================================================================== + +test 000 +-------- +o process max row : 100,000 +o file buffer size : 8192 + + real 0m30.243s + user 0m55.680s + sys 0m1.567s + + +test 001 +-------- +o process max row : 100,000 +o file buffer size : 1,024,000 + + real 0m30.296s + user 0m55.536s + sys 0m1.790s + + +test 002 +-------- + +o process max row : 200,000 +o file buffer size : 1,024,000 + + real 0m30.115s + user 0m55.956s + sys 0m1.500s + + +test 003 +-------- + +o process max row : 100,000 +o file buffer size : 51,200,000 + + real 0m29.924s + user 0m55.443s + sys 0m1.563s + + +test 004 +-------- + +o process max row : 500,000 +o file buffer size : 51,200,000 + + real 0m32.795s + user 0m57.013s + sys 0m1.697s + + +(source change) +before: +- int record_read_filtered(struct Record **R, struct File *F, + struct Field *fld); +after: +- int record_read_filtered(struct Record **R, struct File *F, + struct Field *fld, struct String *str); + + +test 005 +-------- + +o process max row : 100,000 +o file buffer size : 8,192 + + real 0m29.783s + user 0m54.253s + sys 0m1.867s + + +test 006 +-------- + +o process max row : 100,000 +o file buffer size : 51,200,000 + + real 0m30.364s + user 0m56.000s + sys 0m1.570s diff --git a/doc/dev/vos.test.create.mem.log b/doc/dev/vos.test.create.mem.log new file mode 100644 index 0000000..f5db2f6 --- /dev/null +++ b/doc/dev/vos.test.create.mem.log @@ -0,0 +1,50 @@ + + How much vos load+create use memory +------------------------------------------------------------------------------ + +o input file size : 51,521,908 +o input rows : 501,000 +o input fields : 11 +o output fields : 11 +o process max row : 100,000 +o process max : 2 + + +2009.01.14 - test 000 +------------------------------------------------------------------------------ + +0 file buffer size : 51,200,000 +o bytes allocated : 296,833,772 +o allocs : 11,000,473 +o running time (w/o memcheck) : + + real 0m6.187s + user 0m10.789s + sys 0m0.363s + + +2009.01.15 - test 001 +------------------------------------------------------------------------------ + +o with new vos_process_create algorithm +0 file buffer size : 51,200,000 +o bytes allocated : 166,820,858 (~ 3 * input file size) +o allocs : 3,500,662 +o running time (w/o memcheck) : + + real 0m4.565s + user 0m8.026s + sys 0m0.327s + + +2009.01.16 - test 002 +------------------------------------------------------------------------------ +o file buffer size : 8192 (default) +o bytes allocated : 64,437,110 (~ 1.2 * input file size :) +o allocs : 3,500,652 +o running time (w/o memcheck) : + + real 0m4.361s + user 0m7.763s + sys 0m0.283s + diff --git a/doc/dev/vos.test.join.log b/doc/dev/vos.test.join.log new file mode 100644 index 0000000..d3557f3 --- /dev/null +++ b/doc/dev/vos.test.join.log @@ -0,0 +1,39 @@ + How fast vos_join is and how much memory does it's used +------------------------------------------------------------------------------ + +o input file size 1 (already sorted) : 40,499,908 +o input file size 2 (already sorted) : 40,499,908 + +o input rows : 501,000 +o input fields : 11 +o output fields : 22 + +o process max row : 100,000 +o process max : 2 +o file buffer size : 8192 bytes + + +2009.01.18 - test 000 +------------------------------------------------------------------------------ + +o allocs : 24,048,866 +o bytes allocated : 417,724,740 (~ 5 * inputs file size) +o running time (w/o memcheck) : + + real 0m9.118s + user 0m8.483s + sys 0m0.237s + + +2009.01.18 - test 001 +------------------------------------------------------------------------------ + +o with new vos_join algorithm +o allocs : 542 +o bytes allocated : 42,134 (~ 0.2 * inputs file size) +o running time (w/o memcheck) : + + real 0m5.336s + user 0m4.833s + sys 0m0.333s + diff --git a/doc/dev/vos.test.sort-00.log b/doc/dev/vos.test.sort-00.log new file mode 100644 index 0000000..d297e74 --- /dev/null +++ b/doc/dev/vos.test.sort-00.log @@ -0,0 +1,65 @@ + How much vos load+sort use memory +------------------------------------------------------------------------------ + +o input file size : 51,521,908 +o input rows : 501,000 +o input fields : 11 +o output fields : 11 +o sorted fields : field03 +o process max row : 100,000 +o process max : 2 + + +2009.01.16 - test 000 +------------------------------------------------------------------------------ + +o file buffer size : 8192 (default) +o allocs : 24,048,740 +o bytes allocated : 417,820,691 (~ 8 * input file size) +o running time (w/o memcheck) : + + real 0m12.341s + user 0m15.849s + sys 0m0.627s + + +2009.01.16 - test 001 +------------------------------------------------------------------------------ + +o file buffer size : 51,200,000 +o allocs : 24,048,751 +o bytes allocated : 1,185,697,974 (~ 23 * input file size) +o running time (w/o memcheck) : + + real 0m12.341s + user 0m15.849s + sys 0m0.627s + + + +2009.01.16 - test 002 +------------------------------------------------------------------------------ + +o with new sort_process algorithm +o file buffer size : 8192 +o allocs : 18,624,738 +o bytes allocated : 332,184,755 (~ 6 * input file size) +o running time (w/o memcheck) : + + real 0m10.314s + user 0m13.059s + sys 0m0.583s + + +2009.01.17 - test 003 +------------------------------------------------------------------------------ + +o with new sort_process & vos_sort_merge algorithm +o file buffer size : 8192 +o allocs : 6,600,924 +o bytes allocated : 123,352,391 (~ 2 * input file size) +o running time (w/o memcheck) : + + real 0m6.936s + user 0m9.379s + sys 0m0.560s diff --git a/doc/user/example/album.data b/doc/user/example/album.data new file mode 100644 index 0000000..af667d6 --- /dev/null +++ b/doc/user/example/album.data @@ -0,0 +1,5 @@ +'You Forgot it in People' 1 +'Burn' 5 +'Get Lifted' 4 +'The Joshua Tree' 2 +'Broken Social Scene' 1 diff --git a/doc/user/example/artist.data b/doc/user/example/artist.data new file mode 100644 index 0000000..0178395 --- /dev/null +++ b/doc/user/example/artist.data @@ -0,0 +1,5 @@ +1,"Broken Social Scene" +2,"U2" +3,"Led Zeppelin" +4,"John Legend" +5,"Deep Purple" diff --git a/doc/user/example/create_artist_album.vos b/doc/user/example/create_artist_album.vos new file mode 100644 index 0000000..7e530a8 --- /dev/null +++ b/doc/user/example/create_artist_album.vos @@ -0,0 +1,19 @@ +# +# example of create statement +# +LOAD "artist.data" ( + :idx : ::',', + '"':name:'"':: +) as artist; + +LOAD "album.data" ( + '\'':title :'\''::, + :artist_idx: :28:28 +) as album; + +CREATE "create_artist_album.data" from artist, album ( + :artist.idx : ::'|', + '"':artist.name :'"'::'|', + :album.artist_idx: ::'|', + '[' :album.title :']' :: +); diff --git a/doc/user/example/filter_artist_album.vos b/doc/user/example/filter_artist_album.vos new file mode 100644 index 0000000..5e989c5 --- /dev/null +++ b/doc/user/example/filter_artist_album.vos @@ -0,0 +1,21 @@ +# +# example of using create with filter +# +LOAD "artist.data" ( + :idx :::',', + '"':name:'"':: +) as artist; + +LOAD "album.data" ( + '\'':title :'\''::, + :artist_idx: :28 :28 +) as album; + +CREATE "filter_artist_album.data" from artist, album ( + :artist.idx :::'|', + '"':artist.name :'"'::'|', + '[':album.title :']':: +) FILTER ( + ACCEPT artist.idx = 1, + REJECT album.artist_idx != 1 +); diff --git a/doc/user/example/join_artist_album.vos b/doc/user/example/join_artist_album.vos new file mode 100644 index 0000000..d8d2e2c --- /dev/null +++ b/doc/user/example/join_artist_album.vos @@ -0,0 +1,16 @@ +# +# example of join statement +# +LOAD "artist.data" ( + :idx : ::',', + '"':name:'"':: +) as artist; + +LOAD "album.data" ( + '\'':title :'\''::, + :artist_idx: :28 :28 +) as album; + +JOIN artist, album INTO "join_artist_album.data" ( + artist.idx = album.artist_idx +); diff --git a/doc/user/example/sort_album.vos b/doc/user/example/sort_album.vos new file mode 100644 index 0000000..18c0286 --- /dev/null +++ b/doc/user/example/sort_album.vos @@ -0,0 +1,9 @@ +# +# example of sort statement with two field +# +LOAD "album.data" ( + '\'':title :'\''::, + :artist_idx: :28:28 +) as album; + +SORT album BY artist_idx, title INTO "album_sorted.data"; diff --git a/doc/user/example/sort_artist.vos b/doc/user/example/sort_artist.vos new file mode 100644 index 0000000..39fdda8 --- /dev/null +++ b/doc/user/example/sort_artist.vos @@ -0,0 +1,9 @@ +# +# example of sort statement by descending order +# +LOAD "artist.data" ( + :idx :::',', + '"':name:'"':: +) as artist; + +SORT artist BY name DESC; diff --git a/doc/user/image/create_stmt.png b/doc/user/image/create_stmt.png new file mode 100644 index 0000000..8c6ae84 Binary files /dev/null and b/doc/user/image/create_stmt.png differ diff --git a/doc/user/image/field_clause.png b/doc/user/image/field_clause.png new file mode 100644 index 0000000..ca5c387 Binary files /dev/null and b/doc/user/image/field_clause.png differ diff --git a/doc/user/image/filter_clause.png b/doc/user/image/filter_clause.png new file mode 100644 index 0000000..ff0d5a0 Binary files /dev/null and b/doc/user/image/filter_clause.png differ diff --git a/doc/user/image/join_rules.png b/doc/user/image/join_rules.png new file mode 100644 index 0000000..4b564ba Binary files /dev/null and b/doc/user/image/join_rules.png differ diff --git a/doc/user/image/join_stmt.png b/doc/user/image/join_stmt.png new file mode 100644 index 0000000..5dad5a0 Binary files /dev/null and b/doc/user/image/join_stmt.png differ diff --git a/doc/user/image/load_stmt.png b/doc/user/image/load_stmt.png new file mode 100644 index 0000000..401d391 Binary files /dev/null and b/doc/user/image/load_stmt.png differ diff --git a/doc/user/image/set_stmt.png b/doc/user/image/set_stmt.png new file mode 100644 index 0000000..a604664 Binary files /dev/null and b/doc/user/image/set_stmt.png differ diff --git a/doc/user/image/sort_stmt.png b/doc/user/image/sort_stmt.png new file mode 100644 index 0000000..1f45546 Binary files /dev/null and b/doc/user/image/sort_stmt.png differ diff --git a/doc/user/style/page.css b/doc/user/style/page.css new file mode 100644 index 0000000..18b3d67 --- /dev/null +++ b/doc/user/style/page.css @@ -0,0 +1,68 @@ +body +{ + font-family : sans-serif; +} + +h2 +{ + background-color : black; + color : white; + padding : 4px 4px 4px 16px; + border-bottom : solid; + border-color : silver; + margin-top : 4em; +} + +h3 +{ + background-color : silver; + padding : 4px 4px 4px 16px; + border-bottom : solid; + border-color : black; + margin-top : 2em; +} + +h4 +{ + padding : 4px 4px 4px 16px; + border-bottom : dashed; + border-color : black; + border-width : 1px; + margin-top : 2em; +} + +.quote +{ + font-family : serif; + font-style : italic; + text-align : right; + margin-left : 50%; +} + +.script +{ + font-family : monospace; +} + +.box-script +{ + background : #F0F0F0; + border : solid; + border-width : thin; + font-family : monospace; + padding : 8px; + margin-left : 2em; +} + +.list-desc +{ + margin-left : 2em; +} + +.image +{ + border : solid; + border-width : 2px; + border-color : gray; + margin : 1em; +} diff --git a/doc/user/vos_user_manual.html b/doc/user/vos_user_manual.html new file mode 100644 index 0000000..ed5618c --- /dev/null +++ b/doc/user/vos_user_manual.html @@ -0,0 +1,809 @@ + + + +Vos User Manual + + + + +

Vos User Manual

+ +

Table of Contents

+ + +

Introduction

+Vos is a program to process formatted data, i.e. CSV data. +Vos is designed to process a large input file, a file where their size is +larger than the size of memory, and can be tuned to adapt with your machine +environment. +

+Vos currently has four main features, + + +

+Building and Compiling Vos

+

+Vos was developed on GNU/Linux system, so any prerequisite below only valid +on system that running GNU/Linux system. Usually, any Unix like system +could compile the source, it just does not fully tested yet. +

+ +

+Software Requirements

+ +

This software/tools below is used in developing Vos, therefore we +recommended you to use the same or greater version when building Vos from the +source.

+ + + +

+Compiling Vos From Source

+

+This step assume that you already get the source and saved into your machine. +

+ +
+	$ tar jxvf vos-xxxx.xx.xx.tar.bz2
+	$ cd vos/src
+	$ make
+
+ +

+Where xxxx.xx.xx is Vos version (depend on which version that has been +downloaded). When running "make", make program will create directory "build" +in "vos" directory, vos executable is placed in there (vos/build). +

+

+For later use, you should copy Vos executable to your PATH directory. In +example: +

+
+	$ pwd
+	/home/johndoe/tmp/vos/src
+	$ echo $PATH
+	/home/johndoe/bin:/usr/local/bin:/usr/bin:/bin
+	$ cp ../build/vos /home/johndoe/bin
+
+ +

+Running Vos

+

+Vos program only have one parameter: vos script. +

+
+	vos < vos-script >
+
+

+vos-script is a file contains +vos statements that will be executed and processed. +

+ +

Vos Environments

+

+Before running Vos program, there are severals environment variables that you +can set to change behaviour of program while running. Some of the environment +variable also can be set at the vos script using Vos +variables. +

+ + + +

Vos Script

+

+To illustrate on how Vos script work, we will use two input files as an example +here, "artist.data" and "album.data". +

+ + artist.data +
+1,"Broken Social Scene"
+2,"U2"
+3,"Led Zeppelin"
+4,"John Legend"
+5,"Deep Purple"
+
+ + album.data +
+'You Forgot it in People'   1
+'Burn'                      5
+'Get Lifted'                4
+'The Joshua Tree'           2
+'Broken Social Scene'       1
+
+ + +

Vos Variables

+

+Vos variable is used with "SET" statement. +

+

+Vos variable is used to adapt with the environment where Vos will be running. +For example, let say that you have a machine with 8 processor and 16 GB of +memory and you want to sort 20,000,000 rows of data with it's size maybe +about 2 GB. Instead of using default maximum row (which is 100,000) with two +thread you can set maximum row to 2,500,000 and maximum thread to 8, which +will decrease processing time. +

+

+There are two method to set Vos variable, first, by explicitly defined it on +vos script by using SET statement; second, by +defined in environment variable using shell set or +export. +

+ + + + +

Vos Statements

+

+Vos script is not case sensitive, "Load" is equal with "LOAD". +

+ +

Set Statement

+vos set statement
+For example on how to use Set Statement and list of variable see +Vos Variable. + +

Load Statement

+vos load statement
+ +

+Example on using Load Statement: +

+
+LOAD "artist.data" (
+	   :idx :   ::',',
+	'"':name:'"'::
+) as artist;
+
+LOAD "album.data" (
+	'\'':title     :'\''::,
+	    :artist_idx:    :28:28
+) as album;
+
+ +

Sort Statement

+vos sort statement + +

+Example on using Sort Statement: +
+
+This script will sort artist.data by + name (second field) on descending order, +

+
+LOAD "artist.data" (
+	   :idx :   ::',',
+	'"':name:'"'::
+) as artist;
+
+SORT artist BY name DESC;
+
+

If you run the script the output would be like this,

+
+2|U2
+3|Led Zeppelin
+4|John Legend
+5|Deep Purple
+1|Broken Social Scene
+
+ +

+This script will sort + album.data by + artist_idx (second field) then by + title (first field) and save the output to a +file album_sorted.data . +

+
+LOAD "album.data" (
+	'\'':title     :'\''::,
+	    :artist_idx:    :28:28
+) as album;
+
+SORT album BY artist_idx, title INTO "album_sorted.data";
+
+

If you run the script the output would be like this,

+
+Broken Social Scene|1
+You Forgot it in People|1
+The Joshua Tree|2
+Get Lifted|4
+Burn|5
+
+ +

Create Statement

+vos create statement +

+Create statement is used to create a new data with new format or with +different field output order.
+Create statement also can be used to combine several input file into one file. +

+ + + +

+Example on using Create Statement, +
+
+This script will combine artist.data and + album.data into one file, fields will be +separated by '|'. +

+ +
+LOAD "artist.data" (
+	   :idx :   ::',',
+	'"':name:'"'::
+) as artist;
+
+LOAD "album.data" (
+	'\'':title:'\''::,
+	    :artist_idx::28:28
+) as album;
+
+CREATE "artist_album.data" from artist, album (
+	   :artist.idx      :   ::'|',
+	'"':artist.name     :'"'::'|',
+	   :album.artist_idx:   ::'|',
+	'[':album.title     :']'::
+);
+
+ +

If you run the script the output would be like this,

+
+1|"Broken Social Scene"|1|[You Forgot it in People]
+2|"U2"|5|[Burn]
+3|"Led Zeppelin"|4|[Get Lifted]
+4|"John Legend"|2|[The Joshua Tree]
+5|"Deep Purple"|1|[Broken Social Scene]
+
+ +

Join Statement

+vos joint statement +vos join rules +

+Join statement is used to combine two input file into one file, like create +statement, but using specific fields as a matching rule. +

+ + + +

+Example on using Join statement, +

+
+LOAD "artist.data" (
+           :idx :   ::',',
+        '"':name:'"'::
+) as artist;
+
+LOAD "album.data" (
+        '\'':title     :'\''::,
+            :artist_idx:    :28 :28
+) as album;
+
+JOIN artist, album INTO "join_artist_album.data" (
+        artist.idx = album.artist_idx
+);
+
+ +

If you run the script the output would be like this,

+
+1|Broken Social Scene|You Forgot it in People|1
+2|U2|The Joshua Tree|2
+4|John Legend|Get Lifted|4
+5|Deep Purple|Burn|5
+
+ +

Field Clause

+vos field clause + + + + +
Priority of quote vs position vs separator
+ +

+First, when reading field data start-position is have a higher priority than +left-quote. In example, suppose that input data is like this, +

+ +
+'You Forgot it in People'
+
+ +

and you defined field like this,

+
+	'\'':field00:'\'':4:22:
+
+ +

+Vos will always read from position 4, not from first character of +left-quote, which result " Forgot it in Peopl". +

+ +

+Second, while reading field data end-position have a higher priority +than right-quote, and riqht-quote is have a high priority than +separator. +

+ +

Filter Clause

+vos filter clause + +

+Example of using filter: +
+
+This script will only write artist and album where it's field +idx value is 1. +

+ +
+LOAD "artist.data" (
+	   :idx :::',',
+	'"':name:'"'::
+) as artist;
+
+LOAD "album.data" (
+	'\'':title     :'\''::,
+	    :artist_idx:    :28:28
+) as album;
+
+CREATE "filter_artist_album.data" from artist, album (
+	   :artist.idx	:::'|',
+	'"':artist.name	:'"'::'|',
+	'[':album.title	:']'::
+) FILTER (
+        ACCEPT artist.idx = 1,
+        REJECT album.artist_idx != 1
+);
+
+ +

If you run the script the output would be like this,

+
+1|"Broken Social Scene"|[You Forgot it in People]
+|""|[Broken Social Scene]
+
+ +

Vos License

+
+Copyright (C) 2009 M. Shulhan (ms@kilabit.info) All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. All advertising materials mentioning features or use of this software must
+   display the following acknowledgment:
+   "This product includes software written by M. Shulhan (ms@kilabit.info)"
+
+4. The names "M. Shulhan" or "Vos" must not be used to endorse or promote
+   products derived from this software without specific prior written
+   permission.
+
+5. Products derived from this software may not be called "Vos" nor may "Vos"
+   appear in their names without prior written permission of the author.
+
+THIS SOFTWARE IS PROVIDED BY SHULHAN "AS IS" AND ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
+EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ + -- cgit v1.3