Capstone
Replicate EMPDEMO end to end
Five stages, no hand-holding. You are the engineer on a project copying a personnel database off the mainframe into PostgreSQL. Everything you need is in the primer and Modules 01–10. Nothing here is a trick question.
Everything so far has been explained to you. Here nothing is. You get a realistic brief with realistic constraints, and you make the decisions.
The four incidents in Stage 4 all really happen on projects like this. Each one is diagnosable from evidence you now know how to look for — and in three of the four, nothing has errored. That is the point. The failures that matter in this work are silent ones.
New words on this page, one line each
- target model
- The tables you will create on the receiving side, and the keys and constraints you choose for them.
- watermark
- The point in the change log where your first bulk copy ended and continuous copying begins.
- orphan
- A record that has been detached from its parent but still exists. Owner-down extracts miss these entirely.
- cascade
- One source delete removing many child records, arriving as a burst that must be applied together.
- drift
- The copy and the source quietly disagreeing, with no error anywhere to tell you.
The brief
Situation. A general insurer runs personnel and payroll on CA IDMS. The HR analytics team needs the data in PostgreSQL with under five minutes of latency. The mainframe remains the system of record for at least three years.
Source. Schema EMPSCHM version 100, segment
EMPDEMO, page group 0, radix 8. Areas EMP-DEMO-REGION (71001–71100),
ORG-DEMO-REGION (75001–75100), INS-DEMO-REGION (76001–76100). Record types,
sets and membership options are as used throughout this course.
Operations. Journals archive every four hours, archives are retained 48 hours. A nightly local-mode batch job recalculates salary bands. Areas can be quiesced for a maximum of 20 minutes, once, on a Sunday.
Constraint. MAXERUS currently peaks at 84 of 100.
Stage 1 — Design the target model
Whether you can look at a set of pointer-linked records and decide, for each one, what identifies a row and which links are allowed to be empty.
Both answers are already in the source design — you are copying its rules, not inventing your own. The one that catches people is the record with no unique field anywhere in it.
Before opening anything, write your CREATE TABLE statements for
DEPARTMENT, EMPLOYEE, EMPOSITION, OFFICE,
JOB, SKILL, EXPERTISE, COVERAGE and
INSURANCE-PLAN, including nullability and keys. Then answer these.
Model answer — the shape a reviewer would accept
CREATE TABLE department (
dept_id NUMERIC(4) PRIMARY KEY, -- CALC key, DUPLICATES NOT ALLOWED
dept_name TEXT NOT NULL,
dept_head_id NUMERIC(4), -- not a set: a soft reference in the data
src_commit_lsn TEXT, src_op CHAR(1)
);
CREATE TABLE office (
office_code CHAR(3) PRIMARY KEY,
office_city TEXT, office_state CHAR(2), office_phone TEXT
);
CREATE TABLE job (
job_id NUMERIC(4) PRIMARY KEY,
title TEXT NOT NULL, min_salary NUMERIC(9,2), max_salary NUMERIC(9,2)
);
CREATE TABLE employee (
emp_id NUMERIC(4) PRIMARY KEY, -- CALC key
first_name TEXT, last_name TEXT NOT NULL,
status CHAR(2), -- spaces -> NULL, agreed with business
start_date DATE NOT NULL,
termination_date DATE, -- 00000000 -> NULL
dept_id NUMERIC(4) REFERENCES department, -- NULLABLE: DEPT-EMPLOYEE is OPTIONAL
office_code CHAR(3) REFERENCES office -- NULLABLE: OFFICE-EMPLOYEE is OPTIONAL
);
CREATE TABLE emposition (
emp_id NUMERIC(4) NOT NULL -- MANDATORY member of EMP-EMPOSITION
REFERENCES employee ON DELETE CASCADE,
start_date DATE NOT NULL, -- sorted set key, DUPLICATES NOT ALLOWED
finish_date DATE,
salary_amount NUMERIC(9,2), salary_grade NUMERIC(2),
job_id NUMERIC(4) REFERENCES job, -- NULLABLE: JOB-EMPOSITION is OPTIONAL
is_current BOOLEAN NOT NULL DEFAULT FALSE, -- set order was carrying this meaning
PRIMARY KEY (emp_id, start_date)
);
CREATE TABLE expertise ( -- junction record
emp_id NUMERIC(4) NOT NULL REFERENCES employee ON DELETE CASCADE,
skill_id NUMERIC(4) NOT NULL REFERENCES skill,
skill_level CHAR(2), expertise_date DATE,
PRIMARY KEY (emp_id, skill_id)
);
CREATE TABLE coverage ( -- member of TWO sets: two FKs
emp_id NUMERIC(4) NOT NULL REFERENCES employee ON DELETE CASCADE,
plan_code CHAR(3) REFERENCES insurance_plan, -- OPTIONAL: nullable
selection_date DATE NOT NULL, coverage_type CHAR(1),
PRIMARY KEY (emp_id, selection_date, coverage_type)
);
The five decisions a reviewer is looking for:
employee.dept_idandoffice_codeare nullable, because both sets are OPTIONAL.empositionis keyed (emp_id, start_date), justified byDUPLICATES ARE NOT ALLOWEDon the sorted set — not by a db-key and not by a UUID.emposition.emp_idisNOT NULL … ON DELETE CASCADE, because membership is MANDATORY and the application usesERASE PERMANENT.is_currentexists becauseORDER IS SORTED … DESCENDINGwas encoding "current position first". Losing set order without materialising the semantic loses information.coveragecarries two foreign keys; dropping either loses a real relationship.
Stage 2 — Write the extract navigation
Write, in the interpreter, the navigation that produces one row of a flattened
"employee with current position and department" extract for employee 0035. Then generalise it
in your head to a full sweep.
Model answer — and why the naive version is wrong
BIND RUN-UNIT
READY ALL USAGE RETRIEVAL
MOVE '0035' TO EMP-ID-0415
OBTAIN CALC EMPLOYEE
ACCEPT DB-KEY *> save it — you will need to come back
OBTAIN FIRST WITHIN EMP-EMPOSITION *> sorted DESCENDING, so this is the current one
OBTAIN OWNER WITHIN JOB-EMPOSITION *> the job title
OBTAIN DB-KEY IS 71048:1 *> back to the employee
OBTAIN OWNER WITHIN DEPT-EMPLOYEE *> the department
The naive version omits the ACCEPT/OBTAIN DB-KEY pair and tries
OBTAIN OWNER WITHIN DEPT-EMPLOYEE straight after the JOB. That fails, because
currency of run unit is now a JOB and currency of DEPT-EMPLOYEE is stale. Run
both versions in the interpreter and read the status codes.
For a full extract, do not drive this from OBTAIN CALC per employee. Sweep
EMP-DEMO-REGION in db-key order (Module 06), which reads each page once, and chunk by page
range for parallelism.
Stage 3 — Read the journal
You are reading the warehouse day book after the lights went out mid-shift.
Some entries describe work that was finished and confirmed. Some describe work that was half done when the power failed. Your job is to say which is which — because the copy you are maintaining must contain the first kind and must not contain the second.
The CV abended at the point marked in the stream below. Before pressing anything, decide what the target should contain after recovery, and which of these changes your pipeline must and must not apply.
Stage 4 — Four production incidents
Three of these four incidents raise no error at all. Everything reports healthy and the data is wrong.
That is deliberate. Getting good at this work is mostly about knowing which silent symptom points at which cause — a row count that is too high, a column wrong only for negative values, a lag graph that spikes on the same night every week.
Each of these happened. Diagnose, then read the analysis. Partial credit is not a thing here — the point is whether you reach for the right evidence.
Stage 5 — Write the operational agreement
The deliverable, and the specific numbers this brief forces
Given the brief at the top of this page, your runbook has to state at minimum:
- Maximum tolerable capture outage: well under 48 hours. Archives are retained 48 hours, so alert at, say, 8 hours of lag and escalate hard at 24 — you need time to fix a problem, not just to notice it.
-
The nightly local-mode job is a blind spot until proven otherwise. Confirm whether it
journals. If it does not, salary-band changes will never reach the target and no error will ever be
raised. Either get
JOURNAL=ONadded, or schedule a targeted compare-and-sync after it runs every night. Do not let this be discovered later. - One 20-minute Sunday quiesce is not enough for a bulk load of any size. Use the start-capture-first watermark pattern and take the extract live; keep the quiesce window in reserve for cutover, not for the load.
-
MAXERUSheadroom is 16. That is the entire budget for capture, validation queries and any ETL. Get an explicit allocation and get the ceiling raised before, not after, production batch starts failing. - Notification list: reorg or unload/reload, cold start, area restore, schema change, DMCL change, journal reconfiguration. With lead time, in writing.
- Reconciliation schedule: counts continuously, checksums nightly, full compare weekly and after every event on the notification list.