Skip to content

Commit 6f36c3d

Browse files
GGabiGGabe73
andauthored
Updating IDB docs draft (#664)
* Updating IDB docs draft --------- Co-authored-by: GGabi <[email protected]>
1 parent 09cfe1d commit 6f36c3d

1 file changed

Lines changed: 55 additions & 2 deletions

File tree

docs/Processes.md

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1052,8 +1052,8 @@ Intraday Database (IDB)
10521052

10531053
The Intraday Database or IDB is a simple process that allows access to
10541054
data written down intraday. This assumes that there is an existing WDB
1055-
process creating a DB on disk that can be loaded with a simple load command.
1056-
As of now default and partbyenum WDB writedown modes are supported.
1055+
(and HDB) process creating a DB on disk that can be loaded with a simple
1056+
load command. As of now default and partbyenum WDB writedown modes are supported.
10571057
The responsibility of an IDB is therefore:
10581058

10591059
1. Serving queries. Since partbyenum writedown mode is done by enumerated
@@ -1064,6 +1064,59 @@ The responsibility of an IDB is therefore:
10641064
2. Can be triggered for a reload. This is usually done by the WDB process
10651065
periodically.
10661066

1067+
### Considerations
1068+
1069+
The IDB is connected by design and operates in a space between the RDB, WDB and HDB.
1070+
As such, the following are some considerations that must be made when creating an IDB:
1071+
1072+
1. The IDB requires both a WDB and HDB to be operational.
1073+
1074+
2. On startup, the IDB will automatically attempt to connect itself to a WDB
1075+
to facilitate its initialisations steps. These are:
1076+
- Attempt to connect to the WBD to retrieve required information:
1077+
- WBD location.
1078+
- HDB location.
1079+
- Current partition.
1080+
- Current writedown mode.
1081+
- Register itself with the WDB process.
1082+
- Loads the WDB and the HDB's sym file.
1083+
1084+
3. The IDB will have data overlapping with the RDB, so gateways and users should
1085+
be mindful when issuing queries.
1086+
1087+
4. The IDB is designed to fail fast so it doesn't affect surrounding components.
1088+
It will self-terminate as soon as an error is encountered, such as when
1089+
attempting to load a corrupted DB.
1090+
1091+
### Querying an IDB
1092+
1093+
The IDB can be queried just like any other HDB. If writedown mode partbyenum is used it has a useful "maptoint" function which can be used.
1094+
```
1095+
neg[gwHandle](`.gw.asyncexec;"select from trade where int=maptoint[`GOOG]";`idb);gwHandle[]
1096+
```
1097+
1098+
### Scalability
1099+
1100+
The IDB exists on disk and the TorQ gateway will automatically load-balance between
1101+
multiple IDB processes linked to the same IDB on disk. This, alongside the implications
1102+
of shifting data from an RDB to disk while maintaining availability makes the IDB a
1103+
component key to TorQ's scalability.
1104+
1105+
### Database Reload
1106+
1107+
The IDB can be triggered for an intraday reload, which will automatically detect
1108+
any changes to do the least amount of work possible:
1109+
1110+
1. The sym file will only be reloaded if it has changed size on disk.
1111+
1112+
2. In partbyenum writedown mode, the IDB will only be reloaded if the number
1113+
of partitions has changed.
1114+
1115+
3. A reload can be forced in cases where the above conditions are not guaranteed
1116+
to be met.
1117+
1118+
The WDB process triggers the intraday reload on all IDBs registered to it.
1119+
10671120
![IDB diagram](graphics/idb.png)
10681121

10691122
<a name="tick"></a>

0 commit comments

Comments
 (0)