Capstone

Replicate EMPDEMO end to end

Five stages, no hand-holding. You are the engineer on a project copying a personnel database off the mainframe into PostgreSQL. Everything you need is in the primer and Modules 01–10. Nothing here is a trick question.

~75 min4 stagesSelf-assessed
What this capstone is for

Everything so far has been explained to you. Here nothing is. You get a realistic brief with realistic constraints, and you make the decisions.

The four incidents in Stage 4 all really happen on projects like this. Each one is diagnosable from evidence you now know how to look for — and in three of the four, nothing has errored. That is the point. The failures that matter in this work are silent ones.

New words on this page, one line each
target model
The tables you will create on the receiving side, and the keys and constraints you choose for them.
watermark
The point in the change log where your first bulk copy ended and continuous copying begins.
orphan
A record that has been detached from its parent but still exists. Owner-down extracts miss these entirely.
cascade
One source delete removing many child records, arriving as a burst that must be applied together.
drift
The copy and the source quietly disagreeing, with no error anywhere to tell you.

The brief

Situation. A general insurer runs personnel and payroll on CA IDMS. The HR analytics team needs the data in PostgreSQL with under five minutes of latency. The mainframe remains the system of record for at least three years.

Source. Schema EMPSCHM version 100, segment EMPDEMO, page group 0, radix 8. Areas EMP-DEMO-REGION (71001–71100), ORG-DEMO-REGION (75001–75100), INS-DEMO-REGION (76001–76100). Record types, sets and membership options are as used throughout this course.

Operations. Journals archive every four hours, archives are retained 48 hours. A nightly local-mode batch job recalculates salary bands. Areas can be quiesced for a maximum of 20 minutes, once, on a Sunday.

Constraint. MAXERUS currently peaks at 84 of 100.

Stage 1 — Design the target model

What Stage 1 is testing

Whether you can look at a set of pointer-linked records and decide, for each one, what identifies a row and which links are allowed to be empty.

Both answers are already in the source design — you are copying its rules, not inventing your own. The one that catches people is the record with no unique field anywhere in it.

Before opening anything, write your CREATE TABLE statements for DEPARTMENT, EMPLOYEE, EMPOSITION, OFFICE, JOB, SKILL, EXPERTISE, COVERAGE and INSURANCE-PLAN, including nullability and keys. Then answer these.

Model answer — the shape a reviewer would accept
target DDL — the decisions that matter are the keys and the nullability
CREATE TABLE department (
  dept_id        NUMERIC(4)   PRIMARY KEY,          -- CALC key, DUPLICATES NOT ALLOWED
  dept_name      TEXT         NOT NULL,
  dept_head_id   NUMERIC(4),                        -- not a set: a soft reference in the data
  src_commit_lsn TEXT, src_op CHAR(1)
);

CREATE TABLE office (
  office_code    CHAR(3)      PRIMARY KEY,
  office_city    TEXT, office_state CHAR(2), office_phone TEXT
);

CREATE TABLE job (
  job_id         NUMERIC(4)   PRIMARY KEY,
  title          TEXT NOT NULL, min_salary NUMERIC(9,2), max_salary NUMERIC(9,2)
);

CREATE TABLE employee (
  emp_id         NUMERIC(4)   PRIMARY KEY,          -- CALC key
  first_name     TEXT, last_name TEXT NOT NULL,
  status         CHAR(2),                           -- spaces -> NULL, agreed with business
  start_date     DATE NOT NULL,
  termination_date DATE,                            -- 00000000 -> NULL
  dept_id        NUMERIC(4) REFERENCES department,  -- NULLABLE: DEPT-EMPLOYEE is OPTIONAL
  office_code    CHAR(3)    REFERENCES office       -- NULLABLE: OFFICE-EMPLOYEE is OPTIONAL
);

CREATE TABLE emposition (
  emp_id         NUMERIC(4) NOT NULL                -- MANDATORY member of EMP-EMPOSITION
                   REFERENCES employee ON DELETE CASCADE,
  start_date     DATE       NOT NULL,               -- sorted set key, DUPLICATES NOT ALLOWED
  finish_date    DATE,
  salary_amount  NUMERIC(9,2), salary_grade NUMERIC(2),
  job_id         NUMERIC(4) REFERENCES job,         -- NULLABLE: JOB-EMPOSITION is OPTIONAL
  is_current     BOOLEAN    NOT NULL DEFAULT FALSE, -- set order was carrying this meaning
  PRIMARY KEY (emp_id, start_date)
);

CREATE TABLE expertise (                            -- junction record
  emp_id         NUMERIC(4) NOT NULL REFERENCES employee ON DELETE CASCADE,
  skill_id       NUMERIC(4) NOT NULL REFERENCES skill,
  skill_level    CHAR(2), expertise_date DATE,
  PRIMARY KEY (emp_id, skill_id)
);

CREATE TABLE coverage (                             -- member of TWO sets: two FKs
  emp_id         NUMERIC(4) NOT NULL REFERENCES employee ON DELETE CASCADE,
  plan_code      CHAR(3)    REFERENCES insurance_plan,   -- OPTIONAL: nullable
  selection_date DATE NOT NULL, coverage_type CHAR(1),
  PRIMARY KEY (emp_id, selection_date, coverage_type)
);

The five decisions a reviewer is looking for:

  1. employee.dept_id and office_code are nullable, because both sets are OPTIONAL.
  2. emposition is keyed (emp_id, start_date), justified by DUPLICATES ARE NOT ALLOWED on the sorted set — not by a db-key and not by a UUID.
  3. emposition.emp_id is NOT NULL … ON DELETE CASCADE, because membership is MANDATORY and the application uses ERASE PERMANENT.
  4. is_current exists because ORDER IS SORTED … DESCENDING was encoding "current position first". Losing set order without materialising the semantic loses information.
  5. coverage carries two foreign keys; dropping either loses a real relationship.

Stage 2 — Write the extract navigation

Write, in the interpreter, the navigation that produces one row of a flattened "employee with current position and department" extract for employee 0035. Then generalise it in your head to a full sweep.

Model answer — and why the naive version is wrong
one row of the extract
BIND RUN-UNIT
READY ALL USAGE RETRIEVAL
MOVE '0035' TO EMP-ID-0415
OBTAIN CALC EMPLOYEE
ACCEPT DB-KEY                       *> save it — you will need to come back
OBTAIN FIRST WITHIN EMP-EMPOSITION  *> sorted DESCENDING, so this is the current one
OBTAIN OWNER WITHIN JOB-EMPOSITION  *> the job title
OBTAIN DB-KEY IS 71048:1            *> back to the employee
OBTAIN OWNER WITHIN DEPT-EMPLOYEE   *> the department

The naive version omits the ACCEPT/OBTAIN DB-KEY pair and tries OBTAIN OWNER WITHIN DEPT-EMPLOYEE straight after the JOB. That fails, because currency of run unit is now a JOB and currency of DEPT-EMPLOYEE is stale. Run both versions in the interpreter and read the status codes.

For a full extract, do not drive this from OBTAIN CALC per employee. Sweep EMP-DEMO-REGION in db-key order (Module 06), which reads each page once, and chunk by page range for parallelism.

Stage 3 — Read the journal

Picture it

You are reading the warehouse day book after the lights went out mid-shift.

Some entries describe work that was finished and confirmed. Some describe work that was half done when the power failed. Your job is to say which is which — because the copy you are maintaining must contain the first kind and must not contain the second.

The CV abended at the point marked in the stream below. Before pressing anything, decide what the target should contain after recovery, and which of these changes your pipeline must and must not apply.

Stage 4 — Four production incidents

Read the symptom before the explanation

Three of these four incidents raise no error at all. Everything reports healthy and the data is wrong.

That is deliberate. Getting good at this work is mostly about knowing which silent symptom points at which cause — a row count that is too high, a column wrong only for negative values, a lag graph that spikes on the same night every week.

Each of these happened. Diagnose, then read the analysis. Partial credit is not a thing here — the point is whether you reach for the right evidence.

Stage 5 — Write the operational agreement

The deliverable, and the specific numbers this brief forces

Given the brief at the top of this page, your runbook has to state at minimum:

  • Maximum tolerable capture outage: well under 48 hours. Archives are retained 48 hours, so alert at, say, 8 hours of lag and escalate hard at 24 — you need time to fix a problem, not just to notice it.
  • The nightly local-mode job is a blind spot until proven otherwise. Confirm whether it journals. If it does not, salary-band changes will never reach the target and no error will ever be raised. Either get JOURNAL=ON added, or schedule a targeted compare-and-sync after it runs every night. Do not let this be discovered later.
  • One 20-minute Sunday quiesce is not enough for a bulk load of any size. Use the start-capture-first watermark pattern and take the extract live; keep the quiesce window in reserve for cutover, not for the load.
  • MAXERUS headroom is 16. That is the entire budget for capture, validation queries and any ETL. Get an explicit allocation and get the ceiling raised before, not after, production batch starts failing.
  • Notification list: reorg or unload/reload, cold start, area restore, schema change, DMCL change, journal reconfiguration. With lead time, in writing.
  • Reconciliation schedule: counts continuously, checksums nightly, full compare weekly and after every event on the notification list.
If you can do all five stages You can read an IDMS schema and predict its physical behaviour, write and debug navigational DML, reason about the journal as a change stream, design a defensible relational target, and run the operational conversation with a mainframe team. That is the whole objective of this course.