Physical storage: pages, areas and db-keys

From whole database down to a single slot

What one shelf looks like inside

An IDMS page is a slotted page, the same family of design as a Db2 or PostgreSQL page, with some specifically IDMS features.

Some structural points that matter later:

• Space management pages (SMPs) are interspersed through an area at a fixed interval, recording how much free space each nearby page has. A STORE that finds its target page full consults the SMP to choose an overflow page instead of blindly probing.
• Page reserve holds back a declared number of bytes per page so that records which grow (variable-length, compressed, or with an OCCURS DEPENDING ON) can expand in place instead of being relocated.
• System record types (SRn) occupy lines like user records. The two you will meet by name are SR7, the owner record of an index, and SR8, an index block holding sorted key entries and pointers. Others handle internal bookkeeping.
• The page footer repeats the page number. If header and footer disagree, the page was written partially and the DBMS knows the page is broken rather than silently serving corrupt pointers.

The address of a record

A db-key is a 32-bit value that concatenates a page number and a line index. The split point is the db-key radix, declared per segment in the DMCL as the number of low-order bits reserved for the line number.

db-key   = (page number × 2^radix) + line number

radix 8  → 255 lines per page max, ~16.7 million pages addressable
radix 4  →  15 lines per page max, ~268 million pages addressable
radix 3  →   7 lines per page max, ~536 million pages addressable

page     = db-key ÷ 2^radix      (integer division)
line     = db-key MOD 2^radix

The radix is a real design trade-off. Small pages full of small records want a high radix; very large databases with few records per page want a low one so that the page-number field can address more pages. And it is embedded in every stored pointer, so changing it means unloading and reloading the area.

Why page numbers are not unique, and what that stops you doing

Every area belongs to a page group. Page numbers are unique within a page group, not globally. This lets one database contain areas whose page ranges overlap numerically, which is how very large databases get built.

The same rule tells you something important for replication: a db-key on its own does not identify a record in the database. It identifies a record within a page group. Any surrogate key built on db-keys must therefore include the page group, or the segment, or both.

Deciding which shelf a record goes on

CALC

The DBMS hashes ("randomises") the declared CALC key to a target page within the area, then stores the record on that page if there is room. This gives single-I/O access by key with no index to maintain.

LOCATION MODE IS CALC USING ( EMP-ID-0415 )
    DUPLICATES ARE NOT ALLOWED
WITHIN EMP-DEMO-REGION AREA.

• DUPLICATES ARE NOT ALLOWED — the DBMS enforces uniqueness. This is a real constraint and a real natural key.
• DUPLICATES ARE FIRST / LAST — duplicates permitted, chained off the same target page, new arrivals placed at the head or tail of the synonym chain.

When the target page is full, the record goes to a nearby page and is linked into a synonym chain rooted on the target page. Retrieval then costs one I/O for the target page plus one per hop. An area whose synonym chains have grown long is the classic "the database got slow and nothing changed" situation — nothing changed except that it filled up.

VIA

LOCATION MODE IS VIA <set> SET stores the record as physically close as possible to the owner occurrence it will be connected to in that set. If the owner's page has room, the member lands on the same page.

LOCATION MODE IS VIA EMP-EMPOSITION SET
WITHIN EMP-DEMO-REGION AREA.

*> a displacement pushes members a fixed distance from the owner,
*> used to stop a hot owner page becoming a contention point
LOCATION MODE IS VIA EMP-COVERAGE SET
    DISPLACEMENT IS 4 PAGES
WITHIN INS-DEMO-REGION AREA.

VIA is the highest-leverage physical decision in an IDMS database. Walking a 30-member set whose members are clustered VIA costs perhaps three page reads. Walking the same set with members scattered by CALC costs thirty. The set definition is identical; only the placement differs.

DIRECT

The program supplies a suggested db-key and IDMS stores the record at or near it. Legitimate uses are narrow — loading a database in a controlled physical order, or record types whose access is always by saved db-key. It hands physical placement to application code, which means the application now owns a responsibility that normally belongs to the DBA.

What happens when a record outgrows its space

A record that grows beyond the space available on its page cannot simply move — dozens of pointers in other records address it by db-key. IDMS handles this by keeping the record's root at the original db-key and storing the overflow as a fragment elsewhere, linked from the root.

The consequences:

• The db-key of a record is stable for the life of the record in that physical layout.
• A fragmented record costs an extra I/O every time it is read.
• Areas accumulate fragmentation over time; that is what an unload/reload fixes.
• An unload/reload assigns new db-keys. Everything is renumbered.

When and why a reorganisation happens

The utility that tells you which of these you have is IDMSDBAN, the database analysis utility: it walks the physical structures and reports chain lengths, fragmentation, space utilisation and broken pointers. When a DBA says "DBAN says the EMP area is 94% full with an average synonym chain of 3.2", that sentence now means something to you.

Worked example — reading a physical design

*> ---- schema (logical) ----
RECORD NAME IS EMPLOYEE
    LOCATION MODE IS CALC USING ( EMP-ID-0415 ) DUPLICATES ARE NOT ALLOWED
    WITHIN EMP-DEMO-REGION AREA.

RECORD NAME IS EMPOSITION
    LOCATION MODE IS VIA EMP-EMPOSITION SET
    WITHIN EMP-DEMO-REGION AREA.

*> ---- DMCL (physical) ----
SEGMENT NAME IS EMPDEMO
    PAGE GROUP IS 0
    MAXIMUM RECORDS PER PAGE IS 255.

AREA NAME IS EMP-DEMO-REGION
    PRIMARY SPACE MANAGEMENT INTERVAL IS 32
    PAGE RESERVE IS 200 CHARACTERS.

FILE NAME IS EMPFILE
    ASSIGN TO EMPFILE
    PAGE SIZE IS 4276 CHARACTERS
    PAGE RANGE IS 71001 THRU 71100.

Read that as an engineer and you can already state, without touching the data:

1. Employees are reachable in one I/O by employee number, and employee number is unique.
2. Positions live on or near their employee's page, so "all positions for this employee" is cheap.
3. The area is 100 pages of 4276 bytes — about 428 KB, which tells you this is a demonstration database, not production.
4. 200 bytes per page are reserved for growth, so usable space is nearer 4076.
5. Radix is 8 (255 records per page), so db-keys are page × 256 + line.
6. A space management page appears every 32 pages.
7. Everything is in page group 0, so sets may freely span the areas of this segment.

What this means for replication

• Db-keys are physical, not logical. Use them as a within-run correlation handle if you must, never as the target primary key, unless you have a written procedure that triggers a full refresh after every reorganisation.
• A db-key is only unique within its page group. Any surrogate you derive must be qualified by segment or page group.
• Area boundaries are your consistency boundaries. An initial extract that spans two areas quiesced at different times is not a consistent snapshot; you need either a single quiesce window or a journal position to reconcile to.
• Reorganisations are replication events. Ask to be told about them in advance and put them on the change calendar. A reload can rewrite every page in an area, which — depending on how the capture works — either floods the change stream or bypasses it entirely. Both need a plan.
• Fragmented records may be journalled in pieces. Know whether your capture reassembles the root and its fragments, or whether you will receive partial images.

Checkpoint