settingsLogin | Registersettings

[openstack-dev] Can swift leverage os page buffer?

0 votes

Hi,
I'm new to openstack swift, I'm a HPC user, by several days of exploration
of swift, I have some naive questions:
1. Can swift, e.g., PUT, leverage OS' page buffer?; Is there a Linux kernel
module for swift? What are the existing optimizations for caching the write
in order to gain better bandwidth on spin disks?
2. Does swift support asynchronous PUT/Get?

Also please let me know if the dev list is good for me to ask this kind of
questions.

Best,
Jialin


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 12, 2017 in openstack-dev by Jialin_Liu (240 points)  

1 Response

0 votes

On Wed, Oct 11, 2017 at 3:46 PM, Jialin Liu jalnliu@lbl.gov wrote:

Hi,
I'm new to openstack swift, I'm a HPC user, by several days of exploration
of swift, I have some naive questions:
1. Can swift, e.g., PUT, leverage OS' page buffer?;

Sure, but perhaps to a limited degree depending on what you're expecting?
We try pretty hard to fsync everything before we return success:

https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L1517

^ and just below that is finalizeput

Is there a Linux kernel module for swift?

No.

What are the existing optimizations for caching the write in order to gain
better bandwidth on spin disks?

... um ... that's a pretty broad question, swift has been around for quite
a while, it would require some research. You can read through the diskfile
modules; most of the bits that touch disk at the object layer are in
there. There's a number of tunables that effect how Swift will treat the
filesystem - and the low-level specifics of what they do are in the code.
Any results of rigorous classifications you perform and want to publish are
always interesting and welcome in the community. There's been a few
interesting things published over the years - but there's no been no
community effort to collect them into a single home that I'm aware of;
you'd have to track them down - I recall Seagate did some interesting
analysis a while back. RedHat's performance group is always doing stuff,
Intel did some stuff. Current community efforts are focused on further
refining erasure code storage - which have a positive impact on medium and
large object uploads both by reducing the total number of backend bytes to
store (compared to replicated object) and also by fanning that object data
+ parity out to larger numbers of spindles on the backend. On the other
end of the spectrum OVH is leading an effort to further improve performance
for lots of small files.

  1. Does swift support asynchronous PUT/Get?
    >
    >
    I don't know what that means, so I'll say no. I might hazard a guess it
    has something to do with a PUT only storing some reduced redundancy unsafe
    staged data and then doing something to improve durability after it already
    promised the client their write was "safe" - which is not something Swift
    does.

Also please let me know if the dev list is good for me to ask this kind of
questions.

The ML is nice in that people can give more detailed responses and the
archives tend to be a bit more searchable - the trade off is a longer
latency on responses. You can also jump on IRC - swift folks hang out in

openstack-swift on freenode.

Best Regards,

Clay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 12, 2017 by Clay_Gerrard (5,800 points)   1 2 2
...