Compare commits
5 Commits
devel
...
82bff2d341
| Author | SHA1 | Date | |
|---|---|---|---|
| 82bff2d341 | |||
| 44e71fd3a5 | |||
| 93b6cd136c | |||
| 3f98fa4843 | |||
| c4c5a3c7bf |
125
README.md
125
README.md
@@ -1,19 +1,20 @@
|
||||
# TrueMigration
|
||||
|
||||
A Python CLI tool for migrating TrueNAS configuration to a live destination system. Designed for systems integration teams working in pre-production deployment environments.
|
||||
A Python CLI tool for migrating SMB and NFS share configuration to a live TrueNAS destination system. Designed for systems integration teams working in pre-production deployment environments.
|
||||
|
||||
## What It Does
|
||||
|
||||
TrueMigration reads configuration from a source and re-creates it on a destination TrueNAS system via its WebSocket API. It also provides a destination audit mode for inspecting and cleaning up existing configuration before migration.
|
||||
TrueMigration reads share configuration from a source and re-creates it on a destination TrueNAS system via its WebSocket API. Two source types are supported:
|
||||
|
||||
**Supported source types:**
|
||||
- **TrueNAS debug archive** — the `.tgz` produced by **System → Save Debug** in the TrueNAS UI (SCALE and CORE)
|
||||
- **TrueNAS debug archive** — the `.tgz` produced by **System → Save Debug** in the TrueNAS UI
|
||||
- **CSV files** — customer-supplied spreadsheets for migrating from non-TrueNAS sources
|
||||
|
||||
**Supported migration types:**
|
||||
**Currently supported:**
|
||||
- SMB shares
|
||||
- NFS exports
|
||||
- iSCSI (extents, initiator groups, portals, targets, target-extent associations)
|
||||
|
||||
**Planned:**
|
||||
- iSCSI (targets, extents, portals, initiator groups)
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -24,7 +25,7 @@ TrueMigration reads configuration from a source and re-creates it on a destinati
|
||||
|
||||
### Interactive Mode (recommended)
|
||||
|
||||
Run with no arguments. The wizard will guide you through the full workflow.
|
||||
Run with no arguments. The wizard will guide you through source selection, destination configuration, per-share filtering, a dry run preview, and final confirmation before making any changes.
|
||||
|
||||
```bash
|
||||
python -m truenas_migrate
|
||||
@@ -36,41 +37,6 @@ or
|
||||
python deploy.py
|
||||
```
|
||||
|
||||
At startup the wizard presents two top-level options:
|
||||
|
||||
```
|
||||
1. Migrate configuration to a destination system
|
||||
2. Audit destination system (view and manage existing config)
|
||||
```
|
||||
|
||||
#### Option 1 — Migrate
|
||||
|
||||
Walks through:
|
||||
1. Source selection (archive or CSV)
|
||||
2. Destination host, port, and API key
|
||||
3. Migration scope (SMB / NFS / iSCSI, or all)
|
||||
4. iSCSI portal IP remapping (destination IPs differ from source; MPIO supported)
|
||||
5. Check for existing iSCSI config — offers to remove it before migration
|
||||
6. Per-share selection (choose a subset or migrate all)
|
||||
7. Dry run preview — shows what will be created, flags missing datasets or zvols
|
||||
8. Optional auto-creation of missing datasets and zvols
|
||||
9. Final confirmation and live apply
|
||||
|
||||
#### Option 2 — Audit
|
||||
|
||||
Connects to the destination and displays a full inventory:
|
||||
- SMB shares (name, path, enabled status)
|
||||
- NFS exports
|
||||
- iSCSI configuration (extents, initiators, portals, targets, associations)
|
||||
- ZFS datasets (with space used)
|
||||
- ZFS zvols (with allocated size)
|
||||
|
||||
After displaying the inventory, offers selective deletion by category. Deletion safeguards:
|
||||
- SMB shares / NFS exports / iSCSI: standard `[y/N]` confirmation
|
||||
- Zvols: requires typing `DELETE` — data is permanently destroyed
|
||||
- Datasets: requires typing `DELETE` — all files and snapshots are permanently destroyed
|
||||
- A final confirmation gate is shown before any deletions execute
|
||||
|
||||
### Command Line Mode — Archive Source
|
||||
|
||||
```bash
|
||||
@@ -84,7 +50,7 @@ python -m truenas_migrate \
|
||||
--api-key "1-xxxxxxxxxxxx" \
|
||||
--dry-run
|
||||
|
||||
# Live migration (all types)
|
||||
# Live migration
|
||||
python -m truenas_migrate \
|
||||
--debug-tar debug.tgz \
|
||||
--dest 192.168.1.50 \
|
||||
@@ -96,13 +62,6 @@ python -m truenas_migrate \
|
||||
--dest 192.168.1.50 \
|
||||
--api-key "1-xxxxxxxxxxxx" \
|
||||
--migrate smb
|
||||
|
||||
# Migrate SMB and iSCSI, skip NFS
|
||||
python -m truenas_migrate \
|
||||
--debug-tar debug.tgz \
|
||||
--dest 192.168.1.50 \
|
||||
--api-key "1-xxxxxxxxxxxx" \
|
||||
--migrate smb iscsi
|
||||
```
|
||||
|
||||
### Command Line Mode — CSV Source
|
||||
@@ -118,7 +77,7 @@ python -m truenas_migrate \
|
||||
--api-key "1-xxxxxxxxxxxx" \
|
||||
--dry-run
|
||||
|
||||
# Live migration — SMB only from CSV
|
||||
# Live migration — SMB only
|
||||
python -m truenas_migrate \
|
||||
--smb-csv smb_shares.csv \
|
||||
--dest 192.168.1.50 \
|
||||
@@ -134,11 +93,13 @@ Copy and fill in the templates included in this repository:
|
||||
| `smb_shares_template.csv` | One row per SMB share |
|
||||
| `nfs_shares_template.csv` | One row per NFS export |
|
||||
|
||||
Each template includes a header row, annotated comment rows explaining valid values for each column, and one example data row to replace. Lines beginning with `#` are ignored by the parser.
|
||||
|
||||
**SMB columns:** `Share Name` *(required)*, `Path` *(required)*, `Description`, `Purpose`, `Read Only`, `Browsable`, `Guest Access`, `Access-Based Enumeration`, `Hosts Allow`, `Hosts Deny`, `Time Machine`, `Enabled`
|
||||
|
||||
**NFS columns:** `Path` *(required)*, `Description`, `Read Only`, `Map Root User`, `Map Root Group`, `Map All User`, `Map All Group`, `Security`, `Allowed Hosts`, `Allowed Networks`, `Enabled`
|
||||
|
||||
Boolean columns accept `true` or `false`. List columns (`Hosts Allow`, `Hosts Deny`, `Security`, `Allowed Hosts`, `Allowed Networks`) accept space-separated values.
|
||||
Boolean columns (`Read Only`, `Browsable`, etc.) accept `true` or `false`. List columns (`Hosts Allow`, `Hosts Deny`, `Security`, `Allowed Hosts`, `Allowed Networks`) accept space-separated values.
|
||||
|
||||
Valid `Purpose` values: `NO_PRESET`, `DEFAULT_SHARE`, `ENHANCED_TIMEMACHINE`, `MULTI_PROTOCOL_NFS`, `PRIVATE_DATASETS`, `WORM_DROPBOX`
|
||||
|
||||
@@ -148,63 +109,45 @@ Valid `Security` values: `SYS`, `KRB5`, `KRB5I`, `KRB5P`
|
||||
|
||||
In the TrueNAS UI: top-right account menu → **API Keys** → **Add**.
|
||||
|
||||
## iSCSI Migration Notes
|
||||
|
||||
iSCSI configuration involves relational objects with IDs that differ between systems. TrueMigration handles this automatically:
|
||||
|
||||
- **Creation order**: extents and initiator groups first (no dependencies), then portals, then targets (which reference portals and initiators), then target-extent associations
|
||||
- **ID remapping**: old source IDs are mapped to new destination IDs as each object is created; downstream objects are updated accordingly
|
||||
- **Portal IPs**: the wizard prompts for destination IP addresses for each portal. Enter multiple space-separated IPs for MPIO configurations
|
||||
- **Zvols**: DISK-type extents reference ZFS zvols. The dry run checks whether the required zvols exist on the destination. If any are missing, the wizard prompts for their size and creates them before the live run
|
||||
- **Existing config**: if the destination already has iSCSI objects, the wizard detects this and offers to remove them before migration begins
|
||||
|
||||
## Conflict Policy
|
||||
|
||||
TrueMigration never overwrites or deletes existing configuration on the destination. Conflicts are skipped:
|
||||
TrueMigration never overwrites or deletes existing configuration on the destination. Conflicts are silently skipped:
|
||||
|
||||
| Type | Conflict detected by |
|
||||
|------|----------------------|
|
||||
| SMB share | Share name (case-insensitive) |
|
||||
| NFS export | Export path (exact match) |
|
||||
| iSCSI extent | Extent name (case-insensitive) |
|
||||
| iSCSI initiator group | Comment field (case-insensitive) |
|
||||
| iSCSI portal | Set of listen IP addresses |
|
||||
| iSCSI target | Target name (case-insensitive) |
|
||||
| iSCSI target-extent | Target ID + LUN ID combination |
|
||||
| Type | Conflict detected by |
|
||||
|------------|----------------------------------|
|
||||
| SMB share | Share name (case-insensitive) |
|
||||
| NFS export | Export path (exact match) |
|
||||
|
||||
Always run with `--dry-run` first to preview what will and won't be created.
|
||||
|
||||
## Archive Compatibility
|
||||
|
||||
| Source version | Archive format | Notes |
|
||||
|----------------|----------------|-------|
|
||||
| SCALE 24.04+ | ixdiagnose (lowercase dirs) | Combined JSON plugin files |
|
||||
| SCALE (older) | ixdiagnose (uppercase dirs) | Per-query JSON files |
|
||||
| CORE | freenas-debug / fndebug | Plain-text dumps with embedded JSON |
|
||||
| HA bundles (25.04+) | Outer .tgz + inner .txz per node | Active node archive selected automatically |
|
||||
| Source version | Archive format | Notes |
|
||||
|----------------|-------------------------|---------------------------------------------|
|
||||
| SCALE 24.04+ | ixdiagnose (lowercase) | Combined JSON plugin files |
|
||||
| SCALE (older) | ixdiagnose (uppercase) | Per-query JSON files |
|
||||
| CORE | freenas-debug / fndebug | Plain-text dumps with embedded JSON blocks |
|
||||
| HA bundles | Outer .tgz + inner .txz | Active node archive selected automatically |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
deploy.py # Entry point shim
|
||||
smb_shares_template.csv # SMB CSV template
|
||||
nfs_shares_template.csv # NFS CSV template
|
||||
smb_shares_template.csv # SMB CSV template for customers
|
||||
nfs_shares_template.csv # NFS CSV template for customers
|
||||
truenas_migrate/
|
||||
__main__.py # python -m truenas_migrate entry point
|
||||
__main__.py # python -m truenas_migrate
|
||||
colors.py # ANSI color helpers and shared logger
|
||||
summary.py # Migration summary dataclass and report
|
||||
archive.py # Debug archive parser (SCALE + CORE)
|
||||
summary.py # Migration summary and report
|
||||
archive.py # Debug archive parser
|
||||
csv_source.py # CSV parser for non-TrueNAS sources
|
||||
client.py # TrueNAS WebSocket API client and utilities
|
||||
migrate.py # SMB, NFS, and iSCSI migration routines
|
||||
client.py # TrueNAS WebSocket API client
|
||||
migrate.py # SMB and NFS migration routines
|
||||
cli.py # Interactive wizard and argument parser
|
||||
```
|
||||
|
||||
## Safety Notes
|
||||
|
||||
- **Never destructive by default** — the migration path only creates, never modifies or deletes existing destination config
|
||||
- **Dry run first** — always preview with `--dry-run` before applying changes
|
||||
- **Audit deletions require explicit confirmation** — zvol and dataset deletion requires typing `DELETE` and a final confirmation gate
|
||||
- SSL certificate verification is disabled by default (TrueNAS systems commonly use self-signed certs). Use `--verify-ssl` to enable it
|
||||
- Targets the TrueNAS 25.04+ WebSocket API endpoint (`wss://<host>/api/current`)
|
||||
- Exit code `2` is returned if any errors occurred during migration
|
||||
- SSL certificate verification is disabled by default, as TrueNAS systems commonly use self-signed certificates. Use `--verify-ssl` to enable it.
|
||||
- The tool targets the TrueNAS 25.04+ WebSocket API endpoint (`wss://<host>/api/current`).
|
||||
- Exit code `2` is returned if any errors occurred during migration.
|
||||
|
||||
@@ -386,24 +386,21 @@ async def _migrate_iscsi_targets(
|
||||
summary.iscsi_targets_skipped += 1
|
||||
continue
|
||||
|
||||
# Filter out groups whose portal or initiator could not be mapped (e.g. portal
|
||||
# creation failed). Warn per dropped group but still create the target — a
|
||||
# target without every portal group is valid and preferable to no target at all.
|
||||
valid_groups = []
|
||||
# Verify all referenced portals and initiators were successfully mapped
|
||||
missing = []
|
||||
for g in target.get("groups", []):
|
||||
unmapped = []
|
||||
if g.get("portal") not in portal_id_map:
|
||||
unmapped.append(f"portal id={g['portal']}")
|
||||
missing.append(f"portal id={g['portal']}")
|
||||
if g.get("initiator") not in initiator_id_map:
|
||||
unmapped.append(f"initiator id={g['initiator']}")
|
||||
if unmapped:
|
||||
log.warning(" %s dropping group with unmapped %s",
|
||||
_yellow("WARN"), ", ".join(unmapped))
|
||||
else:
|
||||
valid_groups.append(g)
|
||||
missing.append(f"initiator id={g['initiator']}")
|
||||
if missing:
|
||||
msg = f"iSCSI target {name!r}: cannot remap {', '.join(missing)}"
|
||||
log.error(" %s: %s", _bold_red("SKIP"), msg)
|
||||
summary.iscsi_targets_failed += 1
|
||||
summary.errors.append(msg)
|
||||
continue
|
||||
|
||||
payload = _iscsi_target_payload({**target, "groups": valid_groups},
|
||||
portal_id_map, initiator_id_map)
|
||||
payload = _iscsi_target_payload(target, portal_id_map, initiator_id_map)
|
||||
log.debug(" payload: %s", json.dumps(payload))
|
||||
|
||||
if dry_run:
|
||||
|
||||
Reference in New Issue
Block a user