Subject: Sub-surface protocol
Date: Wednesday 5th December 2012 14:32:43 UTC (over 3 years ago)
Hi all, I am currently looking into sub-surfaces, first to sketch the protocol extension, and I have some open questions. I decided to write an exhaustive document, so we would all be on the same page, and also to clarify my own thoughts. Introduction Sub-surfaces are additional wl_surface objects that are tied to a single application window. The "main" part of the window is the parent wl_surface, and there can be any number of sub-surfaces. The aim of sub-surfaces is to let the compositor do the pure compositing work, that the application (a client) would otherwise have to do itself. Letting the compositor do it should be more performant and convenient. One of the most important use cases is a video player in a window. It has XRGB or ARGB window decorations, usually the video content in YUV, and possibly an ARGB overlay for subtitles etc. Currently, the client has to color-convert the video, and merge it with the decorations and subtitles, before sending the grand ARGB buffer to the compositor. This prohibits the use of hardware video overlays, which would offer fast and high quality color conversions and scaling. If the YUV content was in a buffer of its own, and the compositor would be in charge of compositing it with the other window components, the compositor could choose to use a hardware overlay. The use of hardware overlays is especially important on embedded hardware. Other possible use cases are GL widgets, and webgl and flash in web browsers. Sub-surfaces would essentially allow different rendering APIs to work in isolation while still efficiently producing the pieces of a single window. Previous Art Jørgen Lind's extension in Qt http://qt.gitorious.org/qt/qtwayland/blobs/master/src/extensions/sub-surface-extension.xml Daniel Stone's experimental extension http://cgit.collabora.com/git/user/daniels/wayland.git/commit/?h=video&id=1dc6c2307ea53699023ea0f0ce25fe62811961a3 http://cgit.collabora.com/git/user/daniels/weston.git/commit/?h=video&id=6435c41fe71e07891e71992b34169ad540d98121 Anything else? Clipping The term sub-surface sounds like a sub-window, which may cause one to think, that it will be clipped to the parent surface area. I do not think we should force or even allow such clipping. Forcing the clipping would waste memory: for every pixel in sub-surfaces there would have to be a pixel in the parent surface. For instance, drawing window decorations in four sub-surfaces around the main content would not only waste memory, but also not solve the problem of GL applications, where one needs to reserve a margin for the decorations. (Hello, window decoration libraries? :-) Merely allowing clipping is just not the Wayland style. If the *client* wants to clip something, it should not draw it in the first place. Committing changes Essentially we have a set of surfaces: several sub-surfaces and the one parent wl_surface. We have the wl_surface.commit request, that atomically updates one surface's state. This needs to be extended to sub-surfaces, and there are several approaches. * The pending state of all sub-surfaces is applied only on the parent surface commit. Sub-surface commit is a no-op. This will nicely guarantee atomicity. The set of surfaces forms a single application window, and all the surfaces must be updated in sync to avoid flickering a bad composite. Resizing will absolutely require this to work reliably and without glitches. The downside is, that e.g. a video sink component cannot just happily push buffers to its own sub-surface. It has to signal the application to commit the parent surface for every single frame. Some flash, plugin, or video APIs might not support that. Do we care? * The sub-surface commit works like a normal surface's commit. The video sink would be happy with this. It can just spin in its own decoding loop and push the video into the surface, without calling back into the application. This works fine, until something in the surface configurations (geometry) changes. It will break horribly on resize. Also think about flash or webgl in a browser when you scroll the page. Therefore, we would probably want the best of the above two approaches, which raises the following two suggestions. * Implicit fallback to requiring parent commit on sub-surface configuration change. (by daniels) This means, that sub-surface commit works like a normal surface's commit, if the sub-surface configuration (position, buffer size) does not change. When the configuration changes, the compositor will apply the new sub-surface state only on the parent surface commit, and until then it will not complete the sub-surface commit. This will allow a video sink to run on its own, as long as the sub-surface size or position, e.g. via wl_surface.attach, does not change. Resizing can be handled by the application first signalling a resize to the video sink, and then repainting and committing the parent surface. Both the parent and the sub-surface will be updated atomically to the screen. For debugging, if the video sink does something unexpected, like pass non-zero x,y to wl_surface.attach, the problem will be clearly visible as the video will stop until the parent surface updates. The downside is, that the client cannot force a sync to the parent surface commit without changing the sub-surface configuration. Therefore sub-surfaces animating in sync will be very awkward to implement, though one can argue, that such applications should not be using sub-surfaces to begin with. * An explicit request toggling the behaviour of a sub-surface, whether it requires the parent surface commit, or commits on its own. This gives the client explicit control on what happens on sub-surface commit. A video player can have its video sink as autonomous, and switch to synchronized to parent during resize. However, there might be downsides to be discovered. What commit behaviour should we choose? Map and unmap Mapping and unmapping a sub-surface follows the normal wl_surface mapping rules, subject to the sub-surface commit behaviour, and conditional to the parent surface: a sub-surface can be mapped only if the parent surface is. Would we need explicit map/unmap requests? It might depend on the sub-surface commit behaviour with some corner cases, but I don't think we need them. Sub-surface configuration/geometry control All operations about positioning and sizing sub-surfaces take place in the parent surface's coordinate frame, and are limited to integer pixels. No fractional pixels are allowed, and no transformations apart from translation and scaling(?) are supported. We probably need a wl_subsurface.set_position(x, y) request. The initial position could be defined to be 0, 0. The position is affected by wl_surface.attach request's x,y parameters, as usual. The size of a normal wl_surface if defined by the wl_buffer last attached to it. This might be insufficient for sub-surfaces, since it does not allow using the scaling features of hardware overlays. Should we have a request, that can set a different size for a sub-surface, apart from the wl_buffer size? If we did, it should probably be double-buffered state like everything else, and follow the sub-surface commit behaviour. What if the wl_buffer size changes without a corresponding sub-surface size change request? Do we just scale to the set surface size regardless of the buffer size? How would it interact with the sub-surface commit behaviour? Stacking order As a client has the parent surface, and several sub-surfaces, it needs to be able to control the z-order of all the surfaces forming the window. Krh's proposal for the re-stacking requests was, IIUC: - wl_subsurface.place_above(sibling wl_surface) - wl_subsurface.place_below(sibling wl_surface) Where the wl_surface associated with the wl_subsurface would be placed immediately above/below of the given other wl_surface. The other wl_surface must be either the parent or one of its sub-surfaces, but not the one of the wl_subsurface. Protocol sketch My current idea is below. The wl_subsurface object is an additional interface to the wl_surface object it was created for (not the parent). It is similar to how wl_shell_surface is an additional interface into wl_surface. Destroying the wl_subsurface object does not make the wl_surface a normal surface again, it will stay as a sub-surface until it is destroyed. If the parent wl_surface is destroyed, the sub-surfaces will become inert. The only effective request for inert objects is destroy.