You are currently browsing the tag archive for the ‘java’ tag.

Eve GUI Mockup, 3rd October 2009

I have most certainly made some progress on Eve.

Over the past week I took another look at the Eve project, made some necessary re-factorings and started sketching out a new concept for the user interface. The resulting program isn’t much more than a mock-up, but it is something. Currently the Project resource browser is slightly working, the scrubbing interface is partially complete (the fine-tuning interface at the bottom there still needs implementation), and tagging functionality has to be added in the GUI (the backend is in place).

I was thinking of continuing on with the GUI and really polishing the interface, but I think I’ll save that for when some back-end compositing functionality is done. To that end, here’s the order in which the next month or so will go (I hope! gulp).

  1. Implement the Gstreamer FrameProducer. I’m proud to say that Gstreamer’s Source is already complete! For now, the FrameProducer will–at a loss of performance–use RGBDataSink and marshal that data into a BufferedImage.
  2. Create an implementation of the Compositor class using Java2D/BufferedImage.
  3. Continue building the pieces that the Compositor depends upon until simple timelines render (one track, A then B then C then D).
  4. Modify the classes that do the compositing so that they marshal correctly to/from XML.
  5. Create an XMLCompositorRunner that will take input from an XML file and composit it to an output.

After this–yikes. Well I’m not even going to really expand on it, because obviously it’s a big if that these five items are ging to get accomplished given that it’s Midterm season already!

SCOOP, or Simple Concurrent Object-Oriented Programming, is a model originally conceived for Eiffel to easily create parallel code. Eiffel has native support for rich types and pre-conditions, which has led to a design-by-contract paradigm in that programming language. The JSCOOP project that I am working on aims to bring the principles of design-by-contract and simple concurrency to Java.

The first thing we need to think about is the design-by-contract idea in Eiffel. How do we translate a model that builds off of this paradigm (explicit pre-conditions and post-conditions for any method)? JSCOOP takes it half of the way there by adding support for pre-conditions to any method. It looks like this:

public String consume(@separate Buffer buffer) {
    return buffer.take();

I’ll explain the separate annotation later, but let’s take a look at @await. Whenever this method is invoked, the system will enforce that the buffer is not empty before it allows the call to go through. Note that this does not mean that this method is called every time the buffer is empty. Note also that we haven’t gained any parallelism here, yet; we’ve just restricted ourselves.

Enter @separate. This simple keyword says, this is a reference to an object that runs on a different thread than this current code is running on (worth noting that it is certainly more complicated than that). When we execute a method on a separate object, we’re really telling that method to execute on the other thread, whenever it gets the chance to.

The concept of who executes these “remote” methods is called a Processor in JSCOOP. The processor has its own thread of execution and–most importantly–executes requests in the order that they are given. A processor can handle one or more objects. This means that we could have two separate Buffer objects both running on the same Processor, and thus only one of them would be executing at any given time.

Stepping back to consider this, we must understand then that the @separate annotation marks references that are potentially separate, rather than references that are definitely separate.

How does JSCOOP prevent concurrent access?

Consider a Producer-Consumer problem in which all Producers and Consumers share a reference to a common Buffer. Using the JSCOOP model, we store a @separate reference to the same Buffer in each of the Producer/Consumer instances. But surely when we run the Producers/Consumers on separate threads, we’ll concurrently modify the buffer, leading to classic problems.

We introduce the concept now of locking a Processor. Whenever a @separate reference is declared as the parameter of a function (as in the consume example above), we lock the Processor that owns the object referenced in the parameter. Locking a processor says, “only I can add remote calls to this processor’s queue”. What it means is that we’re not actually executing buffer.take() immediately; we’re putting it onto the run queue of buffer’s Processor (its handler). In this case, we require the result of buffer.take(), so we must wait for the call to finish, but that isn’t always the case (this concept called wait-by-necessity). After adding all of the calls in the method to the queue, and waiting for whichever ones we require, we can unlock the processor, allowing other methods that need it (read: have a @separate parameter whose reference is handled by the same Processor) to have their requests go through.

Thus, concurrent access is prevented in the Processor. Because it has a run queue, it waits to execute the next method when it can, according to locking and pre-conditions. This is guaranteed to be safe because each object has only one handling Processor.

That’s all for now

There’s a lot more to the system, but the above serves as a good first step for programmers familiar with concurrency issues in understanding how to program with JSCOOP. Specifically, there is a Scheduler for the entire system that I haven’t mentioned; this checks preconditions, manages Processors, and ensures fairness.

To trace a deadlock, I put my head down for 3 hours, talked to no-one, grunted a few times, and then let out a scream of success when I had built a drop-in replacement for the JSCOOP scheduler that was written before I started the project. Thanks, CSC301, for making me learn Swing (and sorry to everyone else who works in BA2270 for my unnecessary enthusiasm yesterday).

JSCOOP code and manual scheduler

JSCOOP code and manual scheduler

I’m working at the University of Toronto this summer, on a research project called JSCOOP. We’re bringing the SCOOP concurrency model to Java, by way of an Eclipse plugin and Java 1.5 annotations. It’s really, really cool stuff that started at York University and moved down here. I’m working under Marsha Chechik with Faraz Torshizi.

I’m also continuing the Cowichan parallel survey with Andrew Borzenko, under Greg Wilson. Last term we successfully implemented the problem set using Boost::MPI and Threading Building Blocks (TBB). So that’s two parallel paradigms down, and many, many more to go. This is what we’re up to this term:

  1. Tightening up the serial implementation. There’s a lot to be said for having a strong base to start from, and having the fastest serial implementation we can produce will be important for performance comparison.
  2. Standardizing the header files. As we’re only looking at implementing the problem set in C++ using libraries available to it at the moment, we’re defining a standard interface for the problems so that they can be called/chained in an abstract fashion. This also lets us put the performance evaluation code in one place.
  3. Uploading what we have so far. By the end of the term, we want our work to be available in a publically viewable location, complete with repository. We have that right now, to some extent, but clean-up must occur before we can actually “publish”.
  4. Implement with 2 more parallel systems. I’m using a product called LinuxTuples and Andrew’s using OpenMP.

I’m finding tuple spaces to be really interesting, and very theoretically pleasing. The performance seems to be decent too, and it seems like every problem fits into a nice box. More on that later.

Eve has, as one can probably tell from the sorry state of this page, come to a standstill. I’ll need to re-evaluate and make some less lofty (read: attainable) goals and an actual roadmap. When I get time to work on it, I’ll post more.

The buffer, as mentioned in the previous design note, is Eve’s abstraction for media data, whether it is video, audio, or some other form of media that has not yet even been conceived. To recap, buffers store a format specification that details the kind of media type, audio channel count, byte width, video frame size, etc., along with the data itself. However, buffers need not store the data themselves, only a promise that the data still exists somewhere and is accessible by a knowledgeable object. This means that the data might not exist locally, inside of the Buffer structure (e.g. it may be in GPU memory), the data may simply be a pointer (e.g. in a reference counting scheme), and it might even be implicit (e.g. the frames are generated upon request).

Buffers in Eve cannot be resized, but their data can be changed. Buffers have one assigned media format descriptor, and this cannot be changed. One buffer (or part of it) can be copied into another, if and only if:

  1. the two Buffers media formats are as identical as possible, and one of:
    1. the two Buffers are of the exact same Java type
    2. there exists a correct marshaling method in the Compositor

The marshaling methods provide conversion methods from other Buffer types to the Compositor’s Buffer type, and vice versa. Because these methods can often cause bus saturation due to the amount of memory transfer, Eve’s design allows them to be implemented manually, allowing hand-optimization of a potential bottleneck. As for the term “as identical as possible”, this will be revisited on a later post regarding media formats.

The simplest way to represent a buffer of audio or video data is to allocate a contiguous block of space in system memory, the size of which being equal to the size per frame multiplied by the number of frames. Because this is such a common way of representing data, Eve implements this class as NativeBuffer, which implements a Buffer interface. As implied, NativeBuffer uses java’s nio package to provide near-native performance on all operations. Furthermore, if special hardware is not used for the input, output, or compositing stages of the media pipeline, Buffers will never have to be (un)marshaled because reflection can tell us at any part of the process whether or not incoming Buffers are of the correct type, or not.

The rendering pipeline of Eve follows the producer/consumer design pattern. In order to create output, timelines must be able to create a producer to render content in a one-use, low-cost way. The render pipeline then acts as the consumer for this producer, feeding frames from the timeline into the output (whatever that might be — the output is abstracted from the rendering pipeline).

The first drawback with the producer/consumer pattern involves the idea of passing around buffers, the designed abstraction of low-level media data for Eve’s purposes. Because we can implement the Buffer interface any way we choose, we can pass around buffers that do not need to be copied, making the cost of permeating data through the pipeline negligible relative to the cost of the actual work being done. The input layer, rendering layer, and output layer no longer have to agree on Buffer implementations, just that these Buffers must have the same media type. The problem then lays in how we pass data between the layers of the pipeline.

The solution to this is two-fold: first, introduce marshaling and unmarshaling methods for Buffers passing into and out of the rendering stage, respectively. This lets the rendering pipeline — the one with most of the heavy lifting to do — mandate which buffer format is best for its job. However, it would be a mistake (and go against the idea of loose binding) to force the renderer to account for every type of buffer that the input layer could throw at it; indeed, years later, we want the exact same renderer to be able to work with any new input layers that have been added on. So the second step is to create a common, “native” Buffer format to use when all else fails. This nets us the performance benefit when the renderer knows how to efficiently encapsulate its input, the ease of programming when the input layer actually uses this format, and the flexibility to use any input layer we want, as the renderer can request a conversion of the input buffer to the native format before encapsulation. Finally, this solves the unasked question: what kind of buffer should the output layer be passed?

I’d like to close with a discussion on the hierarchical nature of Producers. Like the timelines they are meant to express, Producers too are used in a recursive fashion — each Clip creates a delegate Producer to pass into the render queue. Because these could be track or timeline clips, they could create their own hierarchy of Producers within. The one remaining problem with this hierarchical producer/consumer pattern is: how do we make it fast? I won’t lie: a plan is in the works, but it hasn’t been finalised. The hope is that the performance part of the code will be separated from the functionality part of the code, allowing Eve first to be correct, and then to be usable in real-time.

Who’s writing this?

My name is Cameron Gorrie. I'm an undergraduate student at the University of Toronto, with a strong interest in Artificial Intelligence and Computer Graphics. You can read more about me, or read my CV. If you have work or research opportunities in my interest areas, please do not hesitate to contact me.
April 2021