Module 09

Getting data out: unload, extract, encoding

Two separate problems share one name. The first is how do I read all of it. The second is what do these bytes actually mean — and the second one is where migration projects genuinely lose weeks.

~50 min2 simulators10-question checkpoint
What this module is really about

Getting the data off the machine, and making the bytes mean the right thing once it is off.

Text on a mainframe is not stored the way it is anywhere else. Numbers are packed two digits into each byte. A negative number can arrive looking like a typing mistake. Some records change length depending on what is in them, and some bytes mean different things depending on a value stored elsewhere.

None of these fail loudly. Every one of them converts silently and wrongly if you get it slightly off, which is why the last item in the checklist here is to verify a handful of known records byte by byte before trusting anything.

New words on this page, one line each
EBCDIC
The mainframe’s character encoding. Not ASCII, and its letters are not in one continuous run.
packed decimal (COMP-3)
Numbers stored two digits per byte with the sign in the last half-byte. Reading it as text destroys it.
zoned decimal
Numbers stored one digit per byte with the minus sign hidden inside the last byte.
sign overpunch
The reason a stored -123 can arrive as the text “12L”.
OCCURS DEPENDING ON
A repeating group whose length varies per record, so everything after it shifts position.
REDEFINES
The same bytes meaning different things depending on context. What decides usually lives only in the program code.
unload / reload
Rewriting a storage area to tidy it up. Renumbers every address.
image copy
A physical backup of a storage area, and a legitimate offline source for a first bulk copy.
quiesce
Stopping activity briefly so a consistent copy can be taken.

The five ways to get data out

The tools by name

Utility names and invocation vary by release and platform — confirm against your site's runbooks.
OperationPurposeNote for you
BACKUP / image copyA physical copy of an areaThe consistent starting point for a rollforward, and a legitimate offline source for an initial load
RESTOREPuts an image copy backA restore is a replication event — the area's contents changed without journal traffic you can use
UNLOAD / RELOADLogical unload and reload, used for reorganisation and schema changeAssigns new db-keys. Always. Plan for it.
PRINT PAGEDumps a page in hex with structural interpretationThe forensic tool when you need to prove what is actually stored
MAINTAIN INDEX / index rebuildRebuilds or reorganises index structuresGenerates journal activity that is structural, not user data
IDMSDBANAnalysis: space, chains, fragmentation, broken pointersAsk for its output before sizing anything
IDMSBCFBatch command facility — SQL and utility statements from JCLHow you run most of the above from a job

A copy of the data — but as of when?

The question in plain terms

“Extract the data” is an incomplete instruction. A database that is being changed while you read it does not give you a snapshot — it gives you a mixture of before and after.

There are only three honest answers. Stop the system briefly and copy it. Copy it live, note exactly where the change log had got to, and replay everything from that point afterwards. Or copy last night’s backup and replay from there.

The middle one needs no outage, which is why it is usually the answer — but only if you started recording changes before you began copying.

"Extract the data" is under-specified. Extract it as of when? An initial load has to hand over to change capture at a defined point, or you get either gaps or duplicates.

Quiesce and copy

Vary the areas offline, take an image copy or run the extract, restart. Perfectly consistent, needs an outage window, and the window scales with data volume.

Copy live, reconcile forward

Extract while the system runs, record the journal position at the start, then replay every change from that position. No outage; requires capture to be running before the extract begins.

Copy from a backup

Use last night's image copy plus the journals since. Zero production impact, and the "since" has to be exact.

The handover rule Start change capture before the initial extract, not after. Overlap is recoverable — re-applying a change you already have is idempotent if your apply logic is written as an upsert. A gap is not recoverable by anything except doing the whole load again.

What the bytes actually mean

Picture it

The cards are written in a different alphabet, and some of the numbers are written in shorthand that saves space by cramming two digits into the space of one.

Photocopy the cards and translate every character, and the words come out perfectly readable — while every shorthand number turns into gibberish. Worse: it turns into plausible gibberish, so the copy looks like it worked.

Everything below is a real failure mode that has silently corrupted real migrations. Work through the workbench as you read.

EBCDIC

Mainframe character data is EBCDIC, most commonly code page 037 or 1047 in US installations, with national variants elsewhere. Three specific traps:

  • The alphabet is not contiguous. In EBCDIC, a–i are X'81'–X'89', then there is a gap before j at X'91'. Any code that assumes c - 'a' gives an alphabet index is wrong.
  • Collating sequence differs. In EBCDIC lower case sorts before upper case, and digits sort after letters. A range predicate or a sort applied on the wrong side of the conversion gives different answers — which matters enormously if you chunk an extract by key range.
  • National variants move punctuation. Square brackets, currency symbols and accented characters differ between code pages. Converting with the wrong page corrupts exactly the characters nobody tests.

Packed decimal (COMP-3)

Why this is the number one cause of a broken migration

A transfer that translates characters across a whole record destroys every packed number in it — and leaves every text field looking perfect.

So the first thing you look at says the transfer worked. The damage is in the columns nobody eyeballs. The only safe approach is to convert field by field, driven by the record layout, never record by record.

Two decimal digits per byte, with the final nibble holding the sign: C positive, D negative, F unsigned. A PIC S9(7)V99 COMP-3 occupies 5 bytes.

packed decimal, byte by byte
PIC S9(7)V99 COMP-3, value -128000.00

digits  0 1 2 8 0 0 0 0 0    (9 digits, implied decimal after 7)
sign                      D  (negative)
bytes   X'012800000D'        (5 bytes)

*> the same 5 bytes read as EBCDIC characters produce mojibake,
*> and read as a number in the wrong endianness produce a plausible
*> but completely wrong value. Both failures are silent.
Never code-page-convert a binary field If a transfer applies EBCDIC-to-ASCII translation across a whole record, every packed, binary and floating-point field in it is destroyed — and character fields look fine, so the transfer appears to have worked. This is the number one cause of "the data is wrong but only for some columns". Field-level conversion driven by the copybook is the only safe approach.

Zoned decimal (DISPLAY) and the sign overpunch

The tell-tale symptom

If a numeric column is correct for most rows and wrong only for negative ones, this is why, every time.

The minus sign is not stored as a character. It is folded into the last digit’s byte, and a careless character conversion turns that byte into a letter. In a financial system this corrupts precisely the rows people care about.

A signed PIC S9(3) without COMP-3 stores one digit per byte and encodes the sign in the zone nibble of the last byte. So -123 stores as X'F1F2D3', and that last byte, X'D3', is the EBCDIC letter L.

A naive character conversion turns -123 into the string 12L. Every value in the column arrives looking like a typo, and only for negative numbers — which in a financial system means only for the rows anyone cares about.

Binary (COMP) and floating point

  • COMP / BINARY is big-endian two's complement, 2, 4 or 8 bytes depending on the digit count. Little-endian consumers must byte-swap.
  • COMP-1 and COMP-2 are traditionally hexadecimal floating point, not IEEE 754. Converting them as IEEE gives numbers that are wrong in the low-order digits and occasionally wildly wrong. Modern compilers can also produce IEEE binary floating point — you must find out which your data uses.

Records that change shape

If you know fixed-width file parsing

You know the assumption these break: that field five always starts at byte 40.

A record with a variable-length repeating group inside it shifts everything after that group. Records where the group happens to be full parse perfectly — which is exactly why a test file built from full-sized records passes, and production does not.

the three structures that break flat mapping
       02  DEPENDENT-COUNT       PIC 9(2).
       02  DEPENDENT-TABLE OCCURS 1 TO 20 TIMES
              DEPENDING ON DEPENDENT-COUNT.
           03  DEP-NAME          PIC X(25).
           03  DEP-BIRTH-DATE    PIC 9(8).

       02  POLICY-DETAIL         PIC X(40).
       02  LIFE-POLICY  REDEFINES POLICY-DETAIL.
           03  FACE-AMOUNT       PIC S9(9)V99 COMP-3.
           03  BENEFICIARY       PIC X(33).
       02  AUTO-POLICY  REDEFINES POLICY-DETAIL.
           03  VEHICLE-VIN       PIC X(17).
           03  COVERAGE-CODE     PIC X(3).
           03  PREMIUM           PIC S9(7)V99 COMP-3.
  • OCCURS — a fixed repeating group. Maps to a child table with a sequence number, or to N columns if the count is small and stable. Choose deliberately; the child table is almost always the better answer.
  • OCCURS DEPENDING ON — the record is variable length and you must read the count field to know where the following fields start. A parser that assumes fixed offsets will read garbage for every field after the table, in every record where the count is not the maximum.
  • REDEFINES — the same bytes have several interpretations, and nothing in the record necessarily says which one applies. The discriminator is usually another field, and the mapping from discriminator value to layout is usually only in the application code. Finding it is archaeology, and it cannot be skipped.

Dates

Stored asLooks likeTrap
PIC 9(8) yyyymmdd2018050700000000 means "none" — not year zero
PIC 9(6) yymmdd180507Century inferred by a windowing rule that lives in application code
Julian PIC 9(5) yyddd18127Day-of-year, not month-day
Packed PIC 9(7) COMP-3X'0201805 7C'Must be unpacked before it is a date at all
Days since an epoch, COMPX'00005A3F'Which epoch? Only the code knows.

Compression

IDMS supports record compression (the Presspack option and comparable facilities). Compressed records are stored in a form that is meaningless without the decompression routine. If you are reading pages or journal images directly, you must know whether compression is active for the record types in scope — the bytes will not tell you.

The extraction checklist

  1. Obtain the schema element list and the COBOL copybooks; diff them. Discrepancies are real.
  2. For every field, record: type, length, scale, sign convention, and whether it can hold a sentinel.
  3. List every REDEFINES and find its discriminator in the application source. Document the mapping.
  4. List every OCCURS DEPENDING ON and its count field.
  5. Confirm the source code page, per field if national data is involved.
  6. Confirm whether compression is active.
  7. Confirm floating-point format if any COMP-1/COMP-2 fields exist.
  8. Agree the sentinel-to-null mapping with the business, in writing.
  9. Decide the consistency strategy and the capture handover point.
  10. Build a byte-level test: take twenty known records, extract them, and compare against a PRINT PAGE dump field by field before trusting anything.

What this means for replication

  • Field-level conversion, always. Copybook-driven, never whole-record translation.
  • Test negative numbers and maximum-length strings specifically. Both hide bugs that positive short values do not.
  • Trailing spaces are data until you decide otherwise. A PIC X(15) surname is space-padded; whether the target stores 'ARAKI' or 'ARAKI ' changes every join and every comparison. Decide once, apply everywhere.
  • Variable-length structures need a decision, not a workaround. Child table or flattened columns — pick before the model is built, not when the first long record fails.
  • Reconcile on values, not counts. Row counts match happily while every packed field is wrong. Checksum a numeric column and compare it across the boundary.
Exercise 9.1 — Decode by hand

Use the workbench above to answer:

  1. What are the stored bytes of PIC S9(5)V99 COMP-3 holding -42.50?
  2. The bytes X'0000125C' are a COMP-3 field with two implied decimals. What is the value?
  3. A CHAR column in your target contains 12L where you expected a negative number. What happened, and what is the original value?
  4. Why does a record containing OCCURS DEPENDING ON break a parser that uses fixed offsets, and in which records specifically?

Answers: (1) X'0004250D' — seven digits plus sign, four bytes. (2) +1.25. (3) Zoned decimal with a sign overpunch, character-converted: the last byte X'D3' is both "digit 3, negative" and the letter L; the original value is -123. (4) Because every field after the variable table shifts. It breaks in exactly those records where the count is not the maximum — so a test file of full-sized records passes perfectly.

Checkpoint