MVP. Storage policy support for Ozone#9807
Conversation
|
Thanks for the patch @greenwich. If this is something you are working on, it would be great to have a bit more info on the context, use-case and goals of this PR. Also if you have any reference JIRA for this with the relevant info, it'd be great |
|
|
||
| HOT(StorageType.SSD, StorageType.DISK), | ||
| WARM(StorageType.DISK, null), | ||
| COLD(StorageType.ARCHIVE, null); |
There was a problem hiding this comment.
what is StorageType.ARCHIVE in this context? if disk = HDD, what do we use for slower storage type?
There was a problem hiding this comment.
Yeah, good that you pointed it out; it's not needed here. I guess ARCHIVE comes from the ancient HDFS code. As in our team, we use the following storage types: DISK, SSD, NVME.
From my perspective, it should be:
- HOT -> NVME
- WARM -> SSD
- COLD -> DISK
Technically, NVMe is an SSD, but they are much faster, with different throughput and performance profiles, and we want separate layers for each. So, within our team, we would need to define separate storage for them.
I didn't want to change the policies at this point, but we should. What's your thought?
Also, as a user, I would appreciate the ability to define and configure my own storage policies and storage types, too. We missed it in HDFS, but it might be useful because we use multiple SSD types with different sizes, performance, etc. I would set them to different individual storage types with specific storage policies.
| */ | ||
| public enum OzoneStoragePolicy { | ||
|
|
||
| HOT(StorageType.SSD, StorageType.DISK), |
There was a problem hiding this comment.
how would we call e2e NVMe solution?
There was a problem hiding this comment.
Those things definitely need refinement - I responded to your comment above.
Please note it's a Draft MR.
|
Hi @greenwich, I'm not sure all the design/requirements for this feature have been completed to the point where we are ready to add code. Right now it looks like we should continue discussion in #6989 or open a new PR. I have pinged the contributors on that change for the best way forward. |
|
Thanks, everyone, for having a look! I am very sorry, but this MR isn't intended to be public or in the Open state. My bad - I'm moving it to Draft. I explained my motivation and urgency here: #6989 (comment) cc @errose28 |
greenwich
changed the title
greenwich
force-pushed
the
storage_policy
branch
from
a4a28c3 to
8e9c641
Compare
greenwich
force-pushed
the
storage_policy
branch
2 times, most recently
from
e28622f to
67504c0
Compare
greenwich
force-pushed
the
storage_policy
branch
from
67504c0 to
4004e4a
Compare
4 participants
What changes were proposed in this pull request?
Provide a one-liner summary of the changes in the PR Title field above.
It should be in the form of
HDDS-1234. Short summary of the change.Please describe your PR in detail:
perspective not just for the reviewer.
the Jira's description if the jira is well defined.
issue investigation, github discussion, etc.
Examples of well-written pull requests:
What is the link to the Apache JIRA
Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
(Please replace this section with the link to the Apache JIRA)
How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)