Module 10

Rocket Data Replicate and Sync on IDMS

Everything in the previous nine modules exists to make this one make sense. A replication product is not magic: it reads a change log, decodes the bytes using a description you gave it, and writes rows somewhere else. Every hard problem in the project is one of the behaviours you have already studied, arriving at the wrong moment.

~65 min2 simulators12-question checkpoint
What this module is really about

Putting it together. The software doing the copying has four jobs, and almost every incident you will ever have is one of those four.

Capture — read the change log as changes happen. Decode — turn mainframe bytes into real values, using the record layouts. Stage — hold changes safely, grouped by transaction, until they are known to be final. Apply — write them to the target in the right order, in a way that is safe to repeat.

Knowing which of the four is failing is most of the diagnosis. This module gives you the architecture, the modelling decisions, and a failure catalogue you can work down when the lag graph starts climbing.

New words on this page, one line each
CDC
Change data capture — continuously reading the change log to copy changes as they happen.
bulk transfer
Reading the whole database once to populate the target the first time.
watermark
The exact position in the change log where the bulk copy ended and continuous copying must start.
staging
A durable buffer between reading and writing, where changes are assembled into whole transactions.
idempotent
Safe to repeat. Applying the same change twice leaves the same result as applying it once.
lag
How far behind the copy is, in unapplied changes or in time.
back-pressure
What happens when the buffer fills: reading has to stop, and the change log becomes your only remaining buffer.
reconciliation
Actually checking that the copy matches, rather than trusting that nothing errored.
full refresh
Throwing the copy away and rebuilding it, because the changes needed to catch up no longer exist.
Scope of this module Rocket Data Replicate and Sync is the productised form of tcVISION, which Rocket Software acquired in 2021. This module teaches the architecture and the engineering problems at the level you need to design and debug an IDMS pipeline. Component names, screen names, parameter names and IDMS-specific capture options differ by release — verify every one of them against the Rocket documentation for your installed version before writing a runbook.

The four jobs the software has to do

z/OS — source IDMS central version online + batch run units Journals BGIN BFOR AFTR CKPT ENDJ Capture agent reads journal, filters, ships Bulk extract initial load / full refresh RACF/ACF2/Top Secret authorises both Processing layer Metadata repository schema, copybooks, mappings Decode & transform EBCDIC → UTF-8, COMP-3 → decimal, sentinels → NULL Staging durable buffer, transaction assembly, commit ordering Compare & sync drift detection, delta repair Targets Relational DBs Kafka / streaming Cloud warehouses Files / object storage apply in commit order, idempotently
Four responsibilities: capture (get the change), decode (make the bytes mean something), stage (buffer durably and order by commit), apply (write idempotently). Almost every incident is one of these four, and knowing which one is most of the diagnosis.

Four ways to move data, and when each is right

If you know Debezium or logical replication

The architecture is the one you already know: read the log, decode, buffer by transaction, apply on commit. Two things differ.

First, decoding is not free — the log holds raw mainframe bytes and needs the record layouts supplied separately, so a source schema change that nobody told you about produces wrong data rather than an error.

Second, the log is not kept indefinitely. It is recycled on a fixed schedule sized for disaster recovery, not for you, so an outage longer than that window is unrecoverable without a full reload.

ModeWhat it doesUse it forCost
Bulk transfer Reads the source in full and loads the target Initial load, full refresh after an unrecoverable gap, rebuilding after a reorg Highest mainframe CPU and elapsed time; may need a quiesce window
Log/journal CDC Reads the journal stream and replicates committed change continuously Steady-state replication — everything after the initial load Low and roughly proportional to change volume, not data volume
Compare & sync Compares source and target and generates the delta to repair the difference Validation, drift detection, cheap recovery from a small gap A full read of both sides; schedule it, do not run it casually
Real-time / exit-based capture Sees changes as they happen rather than as they are journalled Lowest achievable latency where the platform supports it Tighter coupling to the source; availability and support implications to confirm

A real project uses at least three of them: bulk for the initial load, CDC for steady state, and compare for periodic assurance. Design all three from the start — retro-fitting reconciliation after go-live is how teams end up unable to answer "is the target correct?" with anything better than an opinion.

The record layouts are the product

Why this is the part that silently breaks

The software cannot decode a single change without being told that record type 415 is an employee, that bytes 5 to 14 are a first name in mainframe text, and that bytes 60 to 64 are a salary packed two digits per byte.

If someone adds a field to the source and nobody updates that description, the decoder happily keeps reading the old byte positions. No error is raised. The data is simply wrong from that moment on. Generate the description from the blueprint, keep it in version control, and compare every regeneration.

The pipeline cannot decode a single journal image without knowing that record ID 415 is an EMPLOYEE, that bytes 5–14 are EMP-FIRST-NAME-0415 in EBCDIC, and that bytes 60–64 are SALARY-AMOUNT-0420 packed with two implied decimals. That knowledge lives in the mapping repository, and it comes from the artefacts you gathered in Module 04.

Source artefactFeeds
Schema punch (DISPLAY SCHEMA … WITH ALL)Record types, record IDs, elements, pictures, sets, membership options
COBOL copybooksPhysical layouts including REDEFINES and OCCURS that the schema states less explicitly
DMCLPage groups, radix, area-to-file mapping, journal configuration
SQL catalog (if the option is installed)An independent cross-check on column types and lengths
Application sourceREDEFINES discriminators, sentinel meanings, ERASE options actually used
Version the mapping repository like source code A schema change on the source that is not reflected in the mapping produces silently wrong data, not an error, because the decoder happily reads the old offsets. Treat the mapping as a versioned artefact, regenerate it from the dictionary rather than editing it by hand, and diff every regeneration.

Turning pointer-linked records into tables

This is where the CODASYL model meets the relational one, and it is a series of decisions with no default right answer. Work through them.

The decision rules, collected

Source constructTarget constructDecision you must make
CALC record, DUPLICATES NOT ALLOWEDTable with a natural primary keyNone — take the key
CALC record, duplicates allowedTable with a non-unique indexWhat actually makes a row unique? Often owner + key.
VIA recordChild tableKey strategy: composite (owner + sort key) if the sorted set forbids duplicates, otherwise a generated sequence assigned at capture
One-to-many setForeign key on the member tableNullable if OPTIONAL, NOT NULL if MANDATORY
Junction record (member of two sets, owns none)Associative tableComposite key from both foreign keys, plus payload columns
Multi-member setSeveral child tables, or one with a type discriminatorWhether consumers want them unified
Recursive structure recordSelf-referencing FK or a closure/edge tableDepth of query the consumers need
Set order that carries meaningAn explicit column: sequence, valid-from, current flagWhich semantic the order was encoding
Orphaned optional membersRows with a null FK, or excluded, or attached to a synthetic parentWhat the business says an orphan means
OCCURS groupChild table with a sequence numberWhether a wide flattened row is genuinely better for the consumer
Sentinel valuesNULL, or a preserved distinct valueSigned off by the business, in writing, per field
Db-keyOptional technical column, never the primary keyWhether you need it at all for debugging

The first copy, and the handover point

Picture it

You are photographing a filing cabinet that people keep using while you work. By the time you reach the last drawer, the first one has changed.

The fix is not to work faster. It is to have a colleague writing down every change from the moment you started, and then apply that list over your photographs afterwards.

Which is why the rule is: start recording changes before you start copying. Overlap costs nothing if applying a change twice is harmless. A gap cannot be fixed by anything except doing it all again.

The most delicate five minutes of the project. The sequence that works:

  1. Start capture first. It begins accumulating change into staging; nothing is applied yet.
  2. Record the journal position at which capture started. This is the watermark.
  3. Run the bulk extract. It reads a source that is being modified concurrently, so what it produces is fuzzy — a mixture of states.
  4. Load the target.
  5. Replay staged change from the watermark forward, applying every operation as an upsert.
  6. Reconcile. Once replay catches up, run a compare on a sample or on everything.

The fuzziness in step 3 is cured by step 5 precisely because the changes are re-applied over whatever the extract happened to catch. This only works if apply is idempotent: an insert of a row that exists must become an update, a delete of a row that is absent must be a no-op. Write it that way from the beginning.

Why the copy falls behind

Lag has exactly one cause

For some period, the source produced changes faster than the slowest stage could drain them.

That is the whole explanation, and it means the useful question is never “why is replication slow?” but “which stage is the bottleneck, and was it sized against peak or against average?” The simulator below lets you create each failure shape deliberately.

Lag is the number the business will ask about, and it has exactly one cause: for some interval, the source produced change faster than the slowest stage could drain it. Play with the pipeline and find the failure shapes.

Read the simulator's behaviour as four distinct situations:

  • Steady state. Apply throughput exceeds change rate; lag hovers near zero and recovers from spikes.
  • Batch spike. A nightly job produces more change in ten minutes than the day does in eight hours. Lag climbs, then drains. Acceptable if it drains before the next spike — that is your real capacity test, not the average.
  • Target outage. Capture keeps reading, staging absorbs. This is what a durable staging area is for. When staging fills, back-pressure stops capture and the journal becomes your buffer — and now journal retention is the clock you are racing.
  • Structural under-capacity. Apply is simply slower than the change rate. Lag grows monotonically and no amount of waiting fixes it. The remedies are all upstream: filter what you replicate, batch the applies, parallelise by table, or reduce source change.

What goes wrong, and what to check first

SymptomLikely causeFirst check
Lag climbing steadily, no spikeApply under-capacity, or a target index making every write expensiveApply rate versus source change rate; target write latency
Lag flat then a cliffOne enormous uncommitted source transactionLong-running run units on the source; ERASE … ALL cascades
Target row counts diverge, no errorsDb-key-based keying and a reorg; or orphans excluded by an owner-down extractWhether an unload/reload happened; the key strategy
Some columns wrong, others fineWhole-record code page conversion destroying packed and binary fieldsConversion configuration; test with a negative packed value
Values wrong only for negative numbersZoned decimal sign overpunch mishandledThe field's PIC and whether it is COMP-3 or DISPLAY
Fields wrong after a certain position, in some rows onlyOCCURS DEPENDING ON parsed at fixed offsetsThe count field and the parser's variable-length handling
Rows that were later rolled back appear in the targetApplying on arrival rather than on commitTransaction assembly in staging
Capture cannot resume after an outageRequired archive journals have been recycledRetention policy versus outage duration; declare a full refresh
Unexplained divergence after a source incidentCold start, area restore, or a local-mode unjournalled jobThe source operations log for that window
Enormous change volume with no business activityStructural journal activity — index maintenance, pointer updates from sorted-set reorderingWhether capture filters structural images

Proving the copy is actually right

Three checks, and one that is not technical

Row counts catch gross failures and prove nothing about correctness. Column checksums catch the byte-level decoding bugs that counts never will. A full comparison catches everything — expensively.

Then add one more: run five reports the business already trusts against both systems and compare numbers a human recognises. That is the only check that catches a modelling mistake, where every technical test passes and the answer is still wrong.

"The pipeline reports no errors" is not evidence. Build three layers of assurance and run them on a schedule.

1. Counts

Row counts per table, per day. Cheap, continuous, catches gross failures — and catches nothing about correctness of values.

2. Checksums

Sum or hash a numeric and a character column per key range. This is what catches packed-decimal and code-page bugs that counts never will.

3. Full compare

A compare-and-sync run across everything. Expensive; schedule it weekly or after any source incident, and always after a reorg or cold start.

Add one non-technical layer: a business check. Pick five reports the business already trusts, run them against source and target, and compare the numbers a human recognises. That is the check that finds modelling errors, which no checksum can.

Cutover patterns

  1. Read-only replica. The target serves reporting and analytics; the mainframe remains the system of record. Lowest risk, and often the whole project. One-way replication, no conflict resolution.
  2. Parallel run. New applications are built against the target while the mainframe still runs. Both are live for reads; only the mainframe is written. Divergence is visible and cheap to fix.
  3. Phased write migration. Ownership of specific entities moves to the new platform, one at a time. Requires replication in the other direction for anything the mainframe still reads, and therefore requires conflict rules.
  4. Bi-directional sync. Both platforms write. Only do this with a defined ownership model per entity or per field, loop prevention, and an agreed conflict resolution rule. It is the most capable and by a wide margin the most dangerous configuration.

The non-functional conversation

  • Mainframe CPU. Capture consumes it, and mainframe CPU is expensive and metered. Ask what portion of the work is zIIP-eligible and get a measured figure from a proof of concept, not a brochure figure.
  • External run units. Anything connecting as an ERU counts against MAXERUS. Get an allocation agreed.
  • Network. Sustained throughput and, more importantly, behaviour under a link failure — does capture stall gracefully and resume from its position?
  • Security. Journal and database access is authorised by RACF, ACF2 or Top Secret. The capture identity needs specific, minimal, documented authority. Encryption in transit is a requirement, not an option.
  • Auditability. Some regulated environments need to prove what was replicated and when. Decide early whether the pipeline is in scope for audit, because it changes what you retain.

The project checklist

Everything this course says to establish before design freeze
  1. Complete list of segments, areas and record types in scope, with the DBNAME that groups them.
  2. Schema punch and DMCL source, with version numbers recorded.
  3. COBOL copybooks, diffed against the schema element list.
  4. Every REDEFINES discriminator, traced in application source.
  5. Every OCCURS DEPENDING ON and its count field.
  6. Sentinel-to-null mapping, per field, signed off by the business.
  7. Key strategy per record type, especially every VIA record.
  8. Set-to-relational mapping, with nullability derived from membership options.
  9. Orphan policy.
  10. Which ERASE options the applications actually use, and the worst-case cascade size.
  11. Inventory of every writer: online, batch under CV, local mode, ADS, ASF, SQL. Which journal.
  12. Journal retention, archive cycle, and the agreed minimum.
  13. Notification agreement: reorg, cold start, restore, schema change, DMCL change.
  14. MAXERUS headroom and the allocation for replication.
  15. Consistency strategy for the initial load and the watermark mechanism.
  16. Idempotent apply semantics, written down.
  17. Reconciliation: counts, checksums, full compare, business reports — each with a schedule and an owner.
  18. Full refresh procedure, timed on real data volumes.
  19. Latency target, alert thresholds set well inside journal retention, and an escalation path.
  20. Cutover pattern chosen, with the conflict model if writes ever go both ways.

Checkpoint