From 16428671365d30687f012243b7bf986a0ead4bab Mon Sep 17 00:00:00 2001 From: Klaus Post Date: Wed, 20 Oct 2021 10:12:46 -0700 Subject: [PATCH] Add documentation for debugging tools (#13484) Move `xl-meta` so it can be installed out-of-repo with a single command. --- docs/debugging/README.md | 93 ++++++++++++++++++- .../xl-meta.go => debugging/xl-meta/main.go} | 0 2 files changed, 92 insertions(+), 1 deletion(-) rename docs/{bucket/versioning/xl-meta.go => debugging/xl-meta/main.go} (100%) diff --git a/docs/debugging/README.md b/docs/debugging/README.md index 426afd8dd..a62c93643 100644 --- a/docs/debugging/README.md +++ b/docs/debugging/README.md @@ -23,7 +23,6 @@ To trace entire HTTP request and also internode communication mc admin trace --all --verbose myminio ``` - ### Subnet Health Subnet Health diagnostics help ensure that the underlying infrastructure that runs MinIO is configured correctly, and is functioning properly. This test is one-shot long running one, that is recommended to be run as soon as the cluster is first provisioned, and each time a failure scenario is encountered. Note that the test incurs majority of the available resources on the system. Care must be taken when using this to debug failure scenario, so as to prevent larger outages. Health tests can be triggered using `mc admin subnet health` command. @@ -57,3 +56,95 @@ mc: Health data saved to dc-11-health_20200321053323.json.gz ``` The gzipped output contains debugging information for your system + +### Decoding Metadata + +Metadata is stored in `xl.meta` files for erasure coded objects. +Each disk in the set containing the object has this file. +The file format is a binary format and therefore requires tools to view values. + +#### Installing xl-meta + +To install, [Go](https://golang.org/dl/) must be installed. + +Once installed, execute this to install the binary: + +```bash +go install github.com/minio/minio/docs/debugging/xl-meta@latest +``` + +#### Using xl-meta + +Executing `xl-meta` will look for an `xl.meta` in the current folder and decode it to JSON. + +It is also possible to specify multiple files or wildcards, for example `xl-meta ./**/xl.meta` will output decoded metadata recursively. + +It is possible to view what inline data is stored inline in the metadata using `--data` parameter `xl-meta -data xl.json` will display an id -> data size. +To export inline data to a file use the `--export` option. + +### Remotely Inspecting backend data + +`mc admin inspect` allows collecting files based on *path* from all backend drives. + +Matching files will be collected in a zip file with their respective host+drive+path. + +A MinIO host from October 2021 or later is required for full functionality. + +Syntax is `mc admin inspect ALIAS/path/to/files`. This can for example be used to collect `xl.meta` from objects that are misbehaving. + +To collect `xl.meta` from a specific object, for example placed at `ALIAS/bucket/path/to/file.txt` append `/xl.meta`, for instance `mc admin inspect ALIAS/bucket/path/to/file.txt/xl.meta`. +All files can be collected, so this can also be used to retrieve `part.*` files, etc. + +Wildcards can be used, for example `mc admin inspect ALIAS/bucket/path/**/xl.meta` will collect all `xl.meta` recursively. +`mc admin inspect ALIAS/bucket/path/to/file.txt/*/part.*` will collect parts for all versions for the object located at `bucket/path/to/file.txt`. + +`xl-meta` accepts zip files as input and will output all `xl.meta` files found within the archive. +For example: + +``` +$ mc admin inspect play/test123/test*/xl.meta +mc: File data successfully downloaded as inspect.6f96b336.zip +$ xl-meta inspect.6f96b336.zip +{ + "bf6178f9-4014-4008-9699-86f2fac62226/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":4,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF"}}}]}, + "fe012443-6ba9-4ef2-bb94-b729d2060c78/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":1,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]}, + "5dcb9f38-08ea-4728-bb64-5cecc7102436/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":2,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]}, + "48beacc7-4be0-4660-9026-4eceaf147504/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":3,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","x-amz-object-lock-legal-hold":"OFF","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-mode":"COMPLIANCE"}}}]} +} +``` + +Optionally `--encrypt` can be specified. This will output an encrypted file and a decryption key: + +``` +$ mc admin inspect --encrypt play/test123/test*/*/part.* +mc: Encrypted file data successfully downloaded as inspect.ad2b43d8.enc +mc: Decryption key: ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54 + +mc: The decryption key will ONLY be shown here. It cannot be recovered. +mc: The encrypted file can safely be shared without the decryption key. +mc: Even with the decryption key, data stored with encryption cannot be accessed. +``` + +This file can be decrypted using the decryption tool below: + +#### Installing decryption tool + +To install, [Go](https://golang.org/dl/) must be installed. + +Once installed, execute this to install the binary: + +```bash +go install github.com/minio/minio/docs/debugging/inspect@latest +``` +#### Usage + +To decrypt the file above: + +``` +$ inspect -key=ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54 inspect.ad2b43d8.enc +Output decrypted to inspect.ad2b43d8.zip +``` + +If `--key` is not specified an interactive prompt will ask for it. + +The file name will contain the beginning of the key. This can be used to verify that the key is for the encrypted file. diff --git a/docs/bucket/versioning/xl-meta.go b/docs/debugging/xl-meta/main.go similarity index 100% rename from docs/bucket/versioning/xl-meta.go rename to docs/debugging/xl-meta/main.go