Can I become who I want to be?

That's a tough question but thankfully, our team is on it. Please bear with us while we're investigating.

Have you had a chance to answer the previous question?

Yes, after a few months we finally found the answer. Sadly, Mike is on vacations right now so I'm afraid we are not able to provide the answer at this point.

For a general explanation on Volume Rendering see

In Scenery volumes are represented by Volume nodes. But, unlike regular meshes, those volumes are not rendered directly, but together at the same time. This is coordinated by the VolumeManager, which is both Hubable and a Node at the same time.

The sampling of each volume is done in the same shader. The shader is auto generated and specific to the amount and type of used volumes. Therefore, adding a volume may trigger a shader rebuild, which is automatically handled by the framework. For more info, see .

Prerequisites

For developing with scenery, or scenery itself, it's quite useful to have an IDE that supports you in your coding tasks. We recommend using IntelliJ as IDE, which is available as . In case you are an Eclipse user, there is a Kotlin plugin available in the Eclipse market place that can be used for development with scenery.

scenery and IntelliJ require an installed Java Development Kit (JDK), with version 21 and upwards being supported. scenery is fully compatible with OpenJDK, which you can download at .

Meshes are, as you probably know, a collection of polygons. If you have never heard of the concept, do not worry, we will explain it on the fly. In case you still want to learn more, here is a very visual article for you: https://conceptartempire.com/polygon-mesh/ – also virtually any introductory book on computer graphics contains this topic.

Lets jump right into it with an example! Say you would like to render the Pyramid of Cheops. Fortunately, we don't need thousands of slaves (or aliens) to do that. More useful in our case is a class which inherits from the Mesh class, for example like this:

class Pyramid: Mesh("Pyramid of Cheops") {}

Before we can fill the Pyramid with useful information, we need to take a look at the Mesh class itself. Mesh inherits both from HasGeometry and Node. Essentially this means that Mesh is both a Geometry and a Node in the scene graph. About the latter we do not need to worry until the scene setup. HasGeometry, however, is essential so lets have a closer look. Of all its attributes, vertices is the most important for now, because it stores the, you guessed it, vertices of our Polygon Mesh in a float buffer. For our Pyramid the vertices look like this:

Lets store these coordinates first as vectors:

Now we need to make a Polygon Mesh out of these. Scenery works by default with triangle meshes. So, our pyramid needs to be stored in a construct similar to:

These triangles are stored vertex by vertex in counterclockwise direction, at least if you want to render your geometry with a normal triangle geometry. Lets have a look at our first triangle: A, B, C.

It does not matter in which order you store the vertices as long as the order is counterclockwise from outside the object. Consequentially, [A; C; B], [B; A; C], and [C; B; A] are the options you have in this case. Lets do this for all our triangles:

Now we are almost there. In the next step we will allocate and fill our vertices buffer with our vertices and then calculate a normal vector for each triangle. Fortunately, this is done by the function recalculateNormals(), note that the vertices must be stored in the right (counterclockwise) order for it to work.

Congratulations! You just wrote your first Mesh. If you wish to render the Pyramid in a different manner, consider the enum GeometryType which gives you a lot of options, e.g. rendering only the vertices. Otherwise, there it is, the mighty Pyramid of Cheops:

What is this about?

Here we describe scenery, an extensible framework for scientific visualisation of mesh and volumetric data that supports VR and AR. scenery supports a lot of input modalities -- apart from mouse and keyboard -- such as VRPN controllers, OpenVR/SteamVR controllers, gamepads, and eye trackers.

scenery runs on top of the Java VM and can be used on Windows, macOS and Linux. In case you dislike Java, fret not, for scenery is written in Kotlin, a new language that got rid of a lot of the boilerplate needed in Java, and provides a much nicer developer experience.

Who is this for?

scenery is a framework and therefore intended as the basis for other visualisation tools. One such tool is its sister project , a plugin for the popular, that makes all of scenery's features available via a user-friendly interface.

If you are a developer and want to develop your own scenery-based application, you've come to the right place! If you are already familiar with sciview, and want to extend its capabilities, you are right here as well! And in case you are an end-user looking for a visualisation tool you can use without any knowledge of programming, head over to the , or the for information how to get started with sciview in Fiji.

How to get started?

For developing scenery, it's useful to know the basics of Kotlin. A great starting point for learning are the , a set of small tutorials to get familiar with the language.

To get started with developing with scenery, head over to the Getting Started page, or if you're all set up already, start with .

Slicing and Cropping allow to view otherwise hidden, inner parts of a volume without tinkering with the transfer function or the data.

Slicing and cropping of a volume rendering is done by a SlicingPlane which is assigned to the Volume. Up to 16 SlicingPlane can be assigned to a volume via the addTargetVolume method. One SlicingPlane can be assigned to unlimited volumes. How those planes interact with the volume is governed by the slicingPlaneEquations property of the volume. The possible exclusive options are:

What are Bounding Boxes?

You want to check off the box on bounding boxes, so let's cut the bad puns and dive right into it. To understand bounding boxes, consider this stock photo of the sherlock holmes among penguins and its bounding box:

In short, a bounding box is the smallest box that contains every feature of an object. This makes them a powerful tool to realize intersections.

How are bounding boxes handled in scenery?

There are many ways to define such a bounding box mathematically, in scenery, we do it via a min vector and a max vector:

Here is a visual explanation of what this looks like:

Note that both of these vectors are local coordinates so that the bounding box remains flexible when translating or rescaling a node.

Accessing the bounding box of a node is rather easy:

In case you are dealing with a more sphere-like object, e.g. a biological cell, you should consider using a bounding sphere instead. Simply use the function:

User inputs are handled by the InputHandler, which is part of SceneryBase and Hubable. Adding inputs can be done in an overwrite of the inputSetup method of SceneryBase. First a Behavior needs to be added before a key can be assigned.

Adding a Behavior

A behavior defines an action which is executed when the corresponding keys are pressed. There are currently the following base behaviors from which can be inherited: ClickBehavior, DragBehavior, ScrollBehavior.

In most cases a ClickBehavior is used. Example:

In cases where the x and y screen positions of the cursor are not needed, they are just ignored. Adding a behavior to the InputHandler is straightforward:

inputHandler should be available from an overwrite of the inputSetup method.

Assigning Keys to Behaviors

Assigning keys to a behavior works in a similar fashion:

This code assigned the earlier added behavior to the first mouse button.

Does the same but for the "M" key.

For more info on the available keys and combinations thereof see

Instancing is a very cool feature, added to make the process of rendering a large number of objects a lot quicker, by submitting the object geometry only once and rendering it many times in a single draw call: Imagine you're making an educational movie and you want to animate thousands of blood cells moving through vasculature. If you would like to render all of the thousands of blood cells separately, which basically look the same, it would take a great amount of draw calls and therefore, resources. So instead, you create only one copy of each type of blood cells and render many instances of it in one go.

On this page, we describe an example on how you can do it in scenery. The complete example can be found at src/test/tests/graphics/scenery/tests/examples/advanced/BloodCellExample.kt.

First, we need to create a mesh:

val erythrocyte = Mesh()
erythrocyte.readFromOBJ(Mesh::class.java.getResource("models/erythrocyte.obj").file)
erythrocyte.material = ShaderMaterial.fromFiles("DefaultDeferredInstanced.vert", "DefaultDeferred.frag")
erythrocyte.material.ambient = GLVector(0.1f, 0.0f, 0.0f)
erythrocyte.material.diffuse = GLVector(0.9f, 0.0f, 0.02f)
erythrocyte.material.specular = GLVector(0.05f, 0f, 0f)
erythrocyte.material.metallic = 0.01f
erythrocyte.material.roughness = 0.9f
erythrocyte.name = "Erythrocyte_Master"
erythrocyte.instancedProperties["ModelMatrix"] = { erythrocyte.model }

scene.addChild(erythrocyte)

Important here is the, you guessed it, instancedProperties. So lets have a look at what this actually is. instancedProperties is a parameter of the node class (please do not get confused, the mesh class inherits from the node class):

As you can see, we store the model matrix of our erythrocyte in the instanced properties. The model matrix is the matrix governing how this object is positioned in 3D space. Now we can proceed. We will map through 40 erythrocytes and make them children of a cell container:

Two things are of main interest here: First we have the instancedProperties again. But this time with the world matrix, which brings our individual model into world space. Then secondly, instances which is a CopyOnWriteArrayList of Nodes. Meaning, it will copy the mesh data from the master object (erythrocyte) to each of the erythrocytes 0..40. Now all that's left to do is adding our erythrocytes to a scene:

Now you should be able to render the objects in much less time.

scenery has been tested with a number of different systems and GPUs. If you have a setup that is not listed in the following table - or marked as untested - please submit a PR to this documentation with the setup added. Please note that the OpenGL-based renderer was recently deprecated, so the following table only shows compatibility with the Vulkan renderer.

✅ Works ⛔ Does not work ⬜ Untested 🚫 Unsupported configuration (e.g. no driver support)

Scenery uses the (BBV) for rendering volumes. Currently, there is no documentation on BBV, therefore we try to explain both a bit in this chapter. However, this chapter will only provide a brief overview with a focus on shader generation and uniform access.

Shader Construction

As mentioned in , the Shader of the VolumeManager is auto generated according the type and amount of to-be-rendered volumes.

Axis and Buttons

Gamepad axis and buttons are handled differently in scenery. Axis are used for analog input and can e.g. lead to movement, or rotation of the camera, or of an object. Buttons in turn can trigger simple behaviours.

GamepadClickBehaviour

A GamepadClickBehaviour can be used to e.g. toggle functionality:

This snippet has been taken from ProteinComparisonExample. When this behaviour is triggered, another object in the scene will be highlighted. In order to bind this behaviour to a button on the gamepad, run the following:

This snippet adds the toggleProteins behaviour defined above to the inputHandler, gives it the name "toggleProteins", and binds it to the right directional pad button. In order to remove the behaviour again from the input handler, use

GamepadMovementControl and GamepadRotationControl

These behaviours can be used to either move or rotate nodes. With scenery's default key bindings, the left-hand stick is bound to movement in the plane, while the right-hand stick is used to look around. These bindings of course can be modified:

This snipped, again from ProteinComparisonExample, removes the default gamepad_camera_control behaviour, and adds a new behaviour bound to the RX and RY axis, that'll rotate the node bound to activeProtein. Movement and rotation controls are always active and should be bound to GamepadButton.AlwaysActive.

Vertical movement is not part of the default input bindings, but can also be easily added:

This snippet binds vertical movement to the Z axis of the controller.

Button Mappings

Buttons printed in italics are controller axis, they cannot be used for GamepadClickBehaviours, but only for GamepadMovementControl or GamepadRotationControl.

Xbox/Xbox One Wireless Controller

Overview

Nodes with attribute type Spatial can be positioned in the world coordinate system. By default, the spatial properties position, rotation, and scale are used:

These properties are used by (Default)Spatial::composeModel (or any class overriding this function) to construct the Node's model matrix. Model matrices are 4x4 matrices that determine the Node's transformation within the world coordinate system, and they use (if you want to know more about homogenous coordinates and their general application in computer graphics, ).

If the Node has any parent objects, the world matrix of it is updated by (Default)Spatial::updateWorld(). If a Node does not have a parent, the world matrix will simply be its model matrix. If it does have a parent, the world matrix will be the model matrix, multiplied with the parent's world matrix, such that hierarchical transforms become possible.

Update cycle

An important aspect to know about model and world matrices is the update cycle. Updates to the world and model matrix are handled asynchronously by scenery. This means any changes e.g. to the position, rotation, and scale properties will not be reflected immediately in the model and world matrix. Both matrices are updated in the background by scenery such that they are available to the renderer with the next frame rendered after the change of properties occurred:

Any manual changes to the world and model matrix will be overwritten by composeModel().

Again, do not rely on the model and world matrices to be up-to-date after changing transform properties.

Overriding default matrix composition behaviour

As stated above, scenery by default takes the position, rotation, and scale properties into account when constructing a Node's model matrix. These properties were chosen as they are the most common ones used. However, you might feel the need to introduce additional transforms into the world matrix, such as skew. This is possible in two ways:

1. Setting wantsComposeModel = false: This will cause scenery to not run the composeModel() routine, but it will use whatever matrix you have provided as the Node's model matrix property. The world matrix will still be the parent's world matrix times your custom model matrix.

2. Overriding composeModel(): This is possible when introducing a new type, and will integrate your custom composeModel() routine within scenery's default update cycle.

Manually updating the model and world matrices

As said, the model and world matrices are only updated before the next frame is rendered. This behaviour can be overridden as well, but is discouraged, as it goes outside of scenery's expected update cycle (inconsistencies should not be expected though, as the updateWorld() method is only called as ). Only use if strictly necessary, or for debug purposes.

The above example then changes to:

Which is what would be expected if updates to the matrices happed immediately.

Using Remotery

scenery includes support for the Remotery profiler. Remotery is a simple profiler that can be used from a browser, either on the same machine, or remotely. Here's the series of steps required to profile a scenery-based application:

Clone the Remotery repository and open vis/index.html in a browser. This is the client that connects to the application and visualises profiling results.

In scenery, set up profiling by either handing the (SceneryBase-derived) application the scenery.Profiler=true system property on startup, or adding the a new Remotery instance to the Hub:

A certain piece of code can then be wrapped in begin()and end() blocks of Remotery. The profiler object itself can be queried from the Hub, if available in that routine:

When connecting to the Remotery instance on the web browser, you'll see the profiling results, which are updated in real-time:

Using a 3rd-party Profiler

To profile scenery, a 3rd-party profiler of choice can be used as well. Good candidates are IntelliJ's integrated profiler and Java Flight Recorder. Since Oracle has changed the licensing terms of the JDK recently, we advise to use OpenJDK, and an open-source build of the Java Flight Recorder, which can be used from Java 11 onwards. A good tutorial how to set that up can be found .

The network synchronization of scenery has two targets. Firstly, for CAVE setups, the scene needs to be rendered by multiple nodes from different perspectives, and secondly, multi user volume viewing sessions using networked computers.

How to Start

The networking capabilities of scenery are enabled via VM parameters. The server has to be started with -Dscenery.Server=true and the client needs to be pointed to the server via -Dscenery.ServerAddress=tcp://127.0.0.1

The server can be pretty much any scenery scene (maybe with some adjustments, see below). For a simple client use [SlimClient].

Additional VM Parameters

• For server and client

  • scenery.MainPort - Port for the main channel. Default: 6040

Relevant Examples

• [SimpleNetworkExample]

• [NetworkVolumeExample]

• [SlimClient]

How it works

Once the server application is started, it begins scanning and registering the scene graph for objects which implement the [Networkable] interface. By default these are all nodes connected to the root node and their attributes.

When a client requests a resync, all registered objects are serialized and sent to all connected clients. Upon receiving an object, the client checks whether it has seen one with that ID before, and if not, adds its to its scene. If the object's parent has not been synced yet, it is put in a waiting queue and added once the parent has been synced. If the object is already there, the existing object is simply updated with the values of the new object.

If the server registers a change in a registered object, it sends an update with the new object to all clients.

Relevant Classes

• [Networkable]

• [NodePublisher]

• [NodeSubscriber]

Slim Client

The [SlimClient] is a scenery application that offers a mostly bare scene into which scenes from a server might be loaded. In addition to that it offers its own camera if desired. If the VM parameter -Dscenery.RemoteCamera=true is set, cameras from the server scene are disregarded and a local one is used. If it is not set or set to false, the server scene has to provide a camera.

Network Adjustments

• Camera

  • [Camera.wantsSync] - allows disabling the registration of the camera for syncing. Default: true

• Material

Implementing the Networkable Interface

New Objects

There are two ways to add objects for the client. The default way is to take the deserialized object from the server and simply add it to its corresponding parent. This works for most cases. But some objects require the constructor be run locally or with specific parameters. For those cases there are the getConstructorParameters() and constructWithParameters(parameters) functions in the Networkable interface. No matter how the object was created, afterwards the update method is called again with the same object from the server. The update happens after the object has been added to the scene graph.

Example

[PointLight]

Updating Objects

Every Networkable object needs to implement the update method. In the update method a fresh copy from the server is given as parameter and should be used to copy over relevant values to the client-side object.

Properties which are marked @Transient are not serialized and therefore not available on the client-side copy of a object from the server. If those properties need to be synced anyway, they should be returned in a serializable form in an overwrite of the getAdditionalUpdateData() method. This method is called on the server at the serialization and the results are transmitted next to the object. The transmitted result of the getAdditionalUpdateData() function is another parameter of the update function. Special attention has to be paid to parent classes which might also have additional data. These data have to be handled manually in the overriding methods.

References to other Networkable objects besides parent/child relations of the scene graph and node/attribute relations can't be synced automatically. For those the Networkable.networkID needs to be saved as additional data on the server side and resolved in the update method with the getNetworkable lambda parameter.

Attention: The first Update

If the object was not created with a local constructor but with deserialization, the first update will be "with itself". This might be a source of sneaky bugs when for example lists which were cleared on the "client" object are now also empty on the "server" object.

Examples

[DefaultMaterial] - AdditionalUpdateData [DefaultSpatial] - GetNetworkable, First Update

Making Changes Known

For the server to register a change, the modifiedAt property needs to be updated. For convenience the updateModifiedAt method simply may be called like in [DefaultSpatial].

Disabling Syncing

It might be needed to prevent the server from registering an object. If wantsSync returns false, the object will be skipped in the registration as in [Camera].

Registering Subcomponents

For the server to register objects out of the scene graph, they need to be returned by getSubcomponents. The registration of attributes is handled that way for [DefaultNode]. One could extend this for rarely changing data objects to reduce the amount of times they are transmitted.

On transmitting lambdas

At the time of writing, serializing lambdas is wonky. Don't expect it to work. An alternative is to newly generate them on the client side in the constructWithParameters or update method.

Volumes

Syncing volume data is currently not possible. Therefore the data has to be available locally.

To initialize a volume node with sync support, the Volume.forNetwork create method should be used. It takes an implementation of the [VolumeInitalizer] interface.

There are already two implementations in this repo. The first is part of scenery itself [VolumeFileSource].

VolumeFileSource has two parameters with each two options:

• path

  • Given(val filePath: String) - for fixed file paths that are the same on every machine (eg. network drive or something like "C://Volume")

The other implementation is [IJVolumeInitializer], which can be found in the [NetworkVolumeExample]. It takes a path/url and opens it using the ImageJ framework.

Examples

[NetworkVolumeExample]

Rendering with distributed setups is still experimental, so this document is very likely to change in the future.

Requirements

For all machines of the setup:

• current version of Java 11 installed from

• up-to-date graphics driver (currently only tested with Nvidia Quadro cards, get the most up-to-date drivers from )

• Vulkan SDK installed for debugging,

• psexec for remote execution installed, from

For the control node:

• current installation of JetBrains IntelliJ Community Edition for running the examples,

• a clone of the scenery git repository,

• scenery's cluster scripts, from

Optionally:

• a remote desktop solution, such as VNC might help with debugging the setup

Basic Setup

Install the above requirements on the machines, and make sure that processes can be remotely launched with psexec, see e.g. for details on psexec setup.

In order to run distributed applications, all machines need to access to two network shares, which do not necessarily need to reside on the same machine:

1. a share containing the scenery application directory, including all JARs built, a suggested structure for this is: # contains the scripts from the cluster-scripts repository path/to/base # contains the scenery git repository and JARs path/to/base/scenery

2. a share containing the data to be loaded.

The scripts from the cluster-scripts repository need to be adjusted for your local setup, in particular the username and password for the rendering node accounts need to be changed, as well as their names, and the name of the network share used. Go through the scripts carefully, they contain comments in places that need to be changed and are very short.

Next, the pom.xml file from the scenery repository needs to be imported into IntelliJ on the control node. Open IntelliJ, select the file via File > Open, and follow the instructions.

Screen configuration

In order for scenery to know about your screen configuration, a screen configuration YAML file is required, such file looks like this:

We assume that all projectors have the same resolution. When launching on each of the nodes, the appropriate screen is determined using the match block. Here, Property means the appropriate screen is determined using the JVM system property scenery.ScreenName. This property gets set by the run-cluster.bat and run-test.bat scripts. An arbitrary numbers of screens is possible, but the YAML file and run-cluster.bat script need to be adjusted accordingly.

Running DemoReelExample

In IntelliJ, find DemoReelExample. This example can be found in the project in the src/test/tests/graphics/scenery/examples/cluster directory. In the example, make sure that the IP given for TrackedStereoGlasses matches that of your tracking system, and the YAML file given matches the name of your screen configuration. Then, click the Runbutton to run this locally and verify all data is found.

Afterwards, go to Run > Edit Configurations... to adjust the parameters of this test in order for it to run on all nodes. The VM options of DemoReelExample should look like the following:

In the Before launch part of the window, two additional steps need to be added:

1. The Maven goal package needs to be run in order to build all JAR files and make them available to the other nodes.

2. The run-cluster.bat script needs to be run to launch scenery instances on the projection nodes. The External Tool setup for that script should look like the following image:

After this is complete, DemoReelExample can be run again and should now launch on all the nodes.

Controls

In DemoReelExample, you can use the WASD keys of the keyboard to move around. You can also keep an Xbox or PS4 gamepad connected to the control node to use for movement. Further keybindings are:

In order to quit the demo on all nodes, use the killall-java.bat script.

Troubleshooting

Should you experience any issues, please feel free to contact us on the , or .