NEMO Science Center

The NEMO Science Center in Amsterdam is the largest science center in the Netherlands and is also the place where my previous thesis work is displayed to the general public. My work now forms a permanent exhibition located at “Bits & Co“.

I haven’t visited the NEMO for a while (since I’m still in the USA) but my dad visited the science center last month and took these nice pictures:

Thanks dad!


TEI 2010

A few weeks ago I attended TEI’10 hosted at the MIT Medialab.

“TEI, the conference on tangible, embedded, and embodied interaction, is about HCI, design, interactive art, user experience, tools and technologies, with a strong focus on how computing can bridge atoms and bits into cohesive interactive systems.”

To keep it simple, I decided to just posted a lot of media instead of a summary of the TEI ’10 conference… enjoy 🙂 !

Inspiring projects

Unlike most conferences I have been, this conference was not only a place where scientific work was presented but also where artists could show their work (art meets science).

The following two projects are clearly an example for this:

Some of my pictures shot at the conference

More pictures of the conference

Pictures created by Jean Baptiste Paris on Flickr.

Studio sessions

Pictures created by Seth Hunter of the studio session on surface computing
(I should be in there somewhere…).

Video studio sessions

Nice video compilation of the different studio sessions that took place at the MIT Medialab.

Blog

A blog about the TEI 2010 conference

Keynotes


ACM Interactive Tabletops and Surfaces 2009

This years Tabletop 2010 conference took place in Banff, Canada.

Unfortunately I could not attend the conference on interactive tabletop and surfaces, so I missed the great keynotes from: Ken Perlin, Chia Shen and Bill Buxton.
Fortunately a few fellow researchers and friends blogged about the event.

PyMT

PyMT is a python module designed for developing multi-touch enabled applications. The current aim is to provide an environment which allows developers to do quick and easy interaction design, and rapid prototype development.

The projects was initially started by Thomas Hansen but now also contains contributes from: Juan Pablo Hourcade, Sharath Patali, Matiue Virbel, Thiago Serra. Their submitted paper is available in the proceedings from this years conference.

For those who are interested, a nice set of videos demonstrating the toolkits capabilities can be found on Vimeo.

Since it’s open source, everyone can download and contribute to the project. The website can be found here: http://code.google.com/p/pymt/. A nice example of an application created with PyMT by Sharath Patali can be found here: NUIPaint.

TUIO

As one of the designers of the TUIO protocol and the ReacTable, Martin Kaltenbrunner presented his work “reacTIVision and TUIO: A Tangible Tabletop Toolkit” at the conference. For those who are interested, Martin created a nice summary of the conference on his personal blog which can be found here: http://modin.yuri.at/blog/?p=117

More interesting links…

More information about the conference can be found in the topic at the NUI Group forums started by Johannes Schöning (contains some videos and docs), the official youtube channel and on Twitter.

ITS 2010?

Hope to see you next year in Saarbrücken, Germany!


Time flies when you’re having fun

Saying goodbye

Lots happened the past year, the most notable projects I have worked on as a scientific programmer were “Interactive Networks” (a collaboration with the SNE department) and “Twilight” (a graph visualization and exploration tool for multi-touch devices). I am still busy creating some short videos from the HD footage we shot, so expect them in the near future.

I am very glad that I was offered the opportunity to participate in both projects. Therefore I would like the following people who made it possible for me to work for the University of Amsterdam (UvA):

I would also like to thank:

Finally I would like to thank Dr. Dick van Albada, Edwin Steffens, Erik Hitipeuw, Alexis Salin, my fellow students at the UvA and everyone I forgot.

A new challenge

After a very brief meeting in Amsterdam during the IEEE Tabletop 2008, I got in touch again with Dr. Chia Shen during the IDC 2009. Shen is a Senior Scientist and Program Manager at the Initiative in Innovative Computing (IIC) at Harvard University. She is well known for her work as Senior Research Scientist at MERL (the Mitsubishi Electric Research Labs).

During the breaks at IDC 2009, we had a talk about the work I was doing at the UvA and my involvement in the NUI Group. After showing some work I have been doing on network visualization (Twilight) for multi-touch devices it seemed that there were some similarities between the work I was doing at the UvA and the research that was going on at Harvard University. While discussing some of the ongoing projects, I was asked whether I was interested to join the SDR Lab as a research fellow. After some consideration, I proudly accepted her job offer.

SEAS Building

The next months I will be working on several interesting projects. One of the tasks will be continuing the development of INVOLV, a multi-touch application developed by Michael Horn and Chia Shen in collaboration with the Encyclopedia of Life (EoL), the Tree of life (ToL) and many biology professors and scientists since May 2008. The application visualizes the data of the EoL and the ToL in a unique way using a Voronoi diagram and is used for learning and discovery.

A video of Michael Horn explaining the INVOLV project can be found below:

Living in the USA

It has only been a week since my arrival in the USA, but so far it has been a pleasant experience.

I have not made a lot of photos yet, but this one was made nearby the place where I am staying (safety first?):
Strange pole?

More photos will be posted on Flickr @ flickr.com/photos/multigesture/


NUI Book and GSOC 2009

NUI Book

To help out fellow multi-touch users and developers, the NUI Group community decided to create a book sharing information about multi-touch technologies. Originally we intended to release this book during the IDC2009, unfortunately we were only able to finish a concept version. This concept version was available at the NUI Group booth.

mttechcoverNUI Group press release:

“We are proud to announce the release of the first edition of the NUI Group Community Book – Multi-Touch Technologies. This book comprises state of the art articles from our Community Wiki, as well as several chapters written by leading experts in the field specifically for this publication. Detailing basic hardware technologies as well as the process of developing software suited for multi-touch systems, this book provides a technical introduction to the world of multi-touch.

This book aims to provide an introduction to the multi-touch world for all backgrounds of people, and also include sufficient information to serve as a general “handbook” for those acquainted with the field, including code examples, suppliers, and building methodology.

Multi-Touch Technologies is a unique publication in its domain in that it is released at no charge under the Creative Commons 3.0 Attribution Share-Alike license – all content found in this book is free to be re-distributed and modified under a similar license, with attribution.”

View the full press release.
Download your copy here: First_Edition_Community_Release.pdf

Comments and suggestions can be posted in the feedback topic @ NUI Group forums.

* update 15 July *
Mirror of the NUI Book:
English version
Chinese version (special thanks to Kelvin / mt2a for translating the book, feedback can be posted here).

Google Summer of Code 2009

2009socwithlogo

Like last year, the NUI Group will be coaching students to work on open source multi-touch projects. This year, I will be helping out by mentoring a student who will add hand tracking to Community Core Vision (formally known as tBeta).

Project: Recognition, Tracking and Association of Hands, Fingers, and Blobs: A Community Core Upgrade
Student: Thiago de Freitas Oliveira Araujo
Mentor: Laurence Muller

Follow his progress on his blog and his code on nuicode or google code.

Hand/Fingers occlusion testcase

occluded_fingers_for_dummies

Matching a hand model (using Active Appearance Models)

More videos can be found here.


Multitouch Media Application Pro v3


Watch it in HD on Youtube or Vimeo!

Changelog for v3:

  • Fixed compatibility issue with Adobe AIR 1.5 (Fullscreen bug)
  • This version should autodetect your screensize to support any screen resolution.
  • Added a Google Maps object
  • Added support for adding pictures on the fly (for example if you want send a picture from your cellphone with bluetooth, read the readme.txt for the instructions)
  • Added a exit-button, reset view-button, canvaslock and displaylock buttons.
  • Temporary removed DCIM support due to slowdowns

Want to try it out yourself?
Download it now: [download#20#size]

Please read the install instructions in readme.txt (and download the latest Adobe AIR 1.5.x).

note: If you want to try out the bluetooth feature, I recommend installing BlueSoleil to handle the pairing of the devices and file transfers.


Interactive Displays 2009

interactive_displays_2009_banner

It has been a while since my last blog post, so I will try to catch up a bit.

Earlier this year, Chris Moore invited me to join the Interactive Displays Conference 2009 (IDC) in San Jose, CA, USA. The IDC is an event for hardware manufacturers, developers, researchers and end user of interactive display technologies. The event was organized by pira-international and supported with great keynote speakers: Jeff Han from Perceptive Pixels and Andy Wilson & Steven Bathiche from Microsoft Research.

The NUI Group was also present with a booth to demonstrate some of the current activities from the community. Key members (Seth Sandler, Chris Moore, Alex Popovich, Jimmy Hertz and Thomas Hansen (Author of PyMT) ) from the NUI Group community and NUI INC. where present to demonstrate their hardware and software achievements.

Jimmy Hertz presented his amazing and huge H2O AquaTouch Water Screen at the booth. Alex Popovich demonstrated his multi-camera solution based on custom drivers created for the PS3.

Jimmy setting up his touch screen
3510640691_6406798b43

The NUI Group booth (with free NUI Group T-shirts!)
3510625625_00e1f28265

Alex’s multicamera solution with 3 PS3 cameras
3534192219_a647479aa0

A nice summary of the conference can found on the NUI Group blog. More photos can be found on the Flickr streams from Joan Woodbrey and Thomas Hansen

Seth, Justin and me heading to the car.
3511413410_ccb4004084

Photograph credits: Thomas Hansen

I would like to thank Chris and the other present NUI Group members for this amazing event, I really enjoyed my stay in San Jose and San Francisco.


Interactive Networks at SC08 (part two)

November last year we presented the Interactive Networks concept at Supercomputing 2008. Recently I got hold of more photo and video content.

Photos and videos

In my previous post on Interactive Networks, I gave a brief explanation of the concept. My colleague Rudolf Strijkers (Project lead) created a page with pictures and videos from the demo we presented. This page also includes a video of Rudolf explaining the Interactive Network concept to Gordon Cook. Only the video with Gordon Cook is in English, but I guess that most of the videos are self explanatory. The videos and photos be found here: Supercomputing 2008 media

COOK Report

After Gordon Cook was given a brief introduction to the Interactive Networks concept, he invited Rudolf Strijkers and Cees de laat for a more in-depth interview. The article (abstract) can be found here: ICT and E-Science as an Innovation Platform in The Netherlands “A National Research and Innovation NetworkWhat Can the US Learn from Dutch Experience?”

Posters

tno-sc08-interactivenetworks uva-sc08-multi-touch_table

More posters from our booth can be found here: clickme!

Pictures of multitouch systems presented at SC 2008

RENCI Vis Group Multi-Touch

The guys from Renaissance Computing Institute were kind enough to show us their table. We could even take a peek under the black curtain that contained all the *magic*.

IMG_0021.JPG IMG_0022.JPG IMG_0023.JPG IMG_0024.JPG
IMG_0038.JPG IMG_0045.JPG IMG_0046.JPG IMG_0047.JPG

More photo and video content can be found on their blog.

EVL | Multi-Touch LCD Table Prototype

EVL showing a rain simulator on their LCD based multitouch table.

IMG_0030.JPG IMG_0032.JPG
Checkout the video of the table here. More pictures of the setup can be found here.

Enthought

A stable, portable and responsive multitouch system using a python backend.

IMG_0026.JPG IMG_0027.JPG IMG_0028.JPG IMG_0029.JPG

“Me painting like a pro, looks better than Picasso … right?”

MS Surface

This doesn’t really needs an explanation…

IMG_0049.JPG IMG_0050.JPG IMG_0051.JPG IMG_0052.JPG

“Dear Santa, please send me a MS coffee table this year.”

(more…)


fIRC Android Chat Client released!

A few months ago Google and T-mobile introduced the T-mobile G1 (aka googlephone). What makes this smartphone so special, is the fact that the operating system (Android) is open source. While most end-users wouldn’t care about this, developers have more freedom in improving or fixing parts of the OS.

As usual we Europeans get all the ‘new’ electronic devices 6 months later than the US or Asia. Often these devices cost more as well (for example iPhones, PSPs, NDS) because someone decided that 1 USD == 1 EUR. Luckily I was able to get a T-mobile G1 just before I left the states 🙂 .

Today I released my first application on the android marketplace (something similar like the apple store). It is an IRC chat client called fIRC:

fIRC chat is a free IRC client.

Current features:

  • Connect to irc servers such as Freenode and EFnet
  • Supports only one channel for now
  • Only landscape mode for now…

Available user commands:

  • To change your current nickname: /nick changeme
  • To request a user list: /userlist

Todo:

  • Server message output
  • Multichannel support
  • PM support
  • A better way to visualize the userlist (another listview?)
  • Nicer GUI for the chat area

A few screenshots:

Suggestions, bug reports and comments can be posted here. The project page can be found here.


Interactive Networks at SC08

After my graduation in July, I was asked to join the Scientific Visualization and Virtual Reality group at the University of Amsterdam. The project that I would be working on was network visualization on a multitouch table that was going to be presented at Supercomputing 2008 in Austin, TX.

Because our old table (which we used for the multitouch performance experiments) was moved to the NEMO Science Center in Amsterdam, we had to develop a new multitouch system from scratch. Based on our experiences of building the previous multitouch table, we carefully choose our new hardware components (such as the camera and digital projector).

Supercomputing 2008


Supercomputing is an international conference for High Performance Computing, Networking, Storage and Analysis (SC08, November 15-21, 2008 Austin Convention Center). In collaboration with the Systems- and Networking Engineering Group of the University of Amsterdam we developed software and hardware for our project called Interactive Networks.

Our project consists out of several parts:

  • The multitouch table; used as an input/interaction device (Rear illumination)
  • Multiple (linux) computers using custom kernal software called streamline
  • A controller that processes events from the multitouch table to the computer nodes using streamline (and vice versa).
  • The Interactive Network Visualizer (the visualization and manipulation software on the MT table)

The project focuses on stream manipulation in a token based network. In Amsterdam we created a physical network of computer nodes (16+). Each of these computer nodes has the streamline module installed. Depending on the task of the node, a node can be the following:

  • A producer (A node that is capable of delivering a video/content stream)
  • A router (A node that is able to route packages)
  • A consumer (A node that is capable of receiving and displaying the stream content)

In the current setup we have 4 producers (each with different video content), 8 routers and 4 consumers (Mac mini’s connected to HD screens with VLC installed to display UDP video streams).

In order to stream video content from the producer to a consumer node, users can simply switch to the ‘route planner’ mode and draw a line from node to node. When the user completes a path (stream), the result is submitted to the controller. The controller checks if the nodes are connected and instructs the nodes how the incoming packages should be handled. Normally it is difficult to distinguish different data streams from each other. However, in our system we added a ‘token’ (color) to each stream. This allows us to distinguish a stream on a particular node and transmit it to its destination. Our system has support for multicasting and streams containing cycles (unique!).

Because streams are tokenized we can also manipulate particular streams on the node itself. By double tapping a node, we can view the internals of this particular node. All streams that passes through this node can be viewed and modified. For this particular demo we added an option to add a sampler. The sampler allows the user to set a value of how many packages should be dropped.

By using standard Linux tools we are also able to measure the network throughput on a node, the load on a node, the used bandwidth between two nodes and the jitter on a connection. These values are visualized in our software.

Videocontent

For our demo we stream high definition videos from The Netherlands to the USA over an 4 GBit internet connection. We are using movies that were made in Blender, namely: Big Buck Bunny and the Elephants dream.

Live videofeed

On our dutch stand we have three live camera streams showing different projects, be sure to check those out: webcam overview

Other multitouch projects at SC08:

We certainly aren’t the only ones with a multitouch table at SC08, so far I’ve seen the following:

  1. RENCI Vis Group Multi-Touch Blog
    Developing multitouch hardware and applications for research and experimentation.
  2. EVL | Multi-Touch LCD Table Prototype (Tactile)
    A high resolution multitouch device created from a LCD panel Video!
  3. Enthought (Scientific Computing Solutions)
    A fast and responsive multitouch solution using a python backend. They are located at the stand of Western Scientific
  4. MS Surface (on the HPC stand)

Photos

Photos of our setup can be found after the break!

(more…)


IEEE Tabletops and Interactive Surfaces 2008

Tommorow the 3rd edition of IEEE Tabletop will take place in Amsterdam.

On the first day of the conference there will be a bootcamp on building your own multitouch surfaces.

At the bootcamp the ‘experts’ will share their knowledge and experiences on building multitouch surfaces. The following topics will be covered:

  • Infrared Illumination
  • Silicone & Projection surfaces
  • Cameras, Optics, Filters & Projectors
  • Software
  • Hardware Integration
  • Community & Network

I will be at one of the stands presenting the software part together with Ulrich von Zadow.

Hope to see you soon,
– Laurence

* Update *
We now have a website running with the bootcamp information.
– Project page: http://ifgi.uni-muenster.de/multi-touch-bootcamp
– Some photos of the conference: click me!

* Update 2 * (22/03/2009)
IEEE tabletop bootcamp page update.
Multi-Touch Surfaces: A Technical Guide (mirror)


Thesis work

For those who are interested, I have uploaded my master’s thesis and final presentation.

1. Master’s thesis:
[download#14]
Mirror: SCS – Master of Science theses
(1.4 mb)

2. Final presentation:
[download#15]
(26.6 mb, includes embedded wmv videos)

3. Multitouch video:
[download#16]
(76.3 mb, high quality wmv)

*update 17-09-2008*
4. Multitouch experiment videos:
10 multitouch experiment videos

*update 20-10-2009 (mirrored files)*
4. Multitouch experiment videos:
10 multitouch experiment videos

4. Experiment videos:
Multitouch experiment videos (youtube playlist)


I graduated!

After almost a year, I finally finished my thesis and presented it. A brief overview of my work can be found in the video below. The thesis itself will become available later this week and will of course be posted on this blog!


Multi-touch video puzzle game

The same game I have showed here but this time played on the multi-touch table.

p.s. Sorry for the ending 😛 it wasn’t me!


tDesk released!

Last year I demonstrated how Windows XP could be (ab)used with multi-touch interaction.

A few weeks ago I made some adjustments to make it compatible with the latest touchlib and this is the result:

tDesk v2 allows the user to move and resize multiple windows at the same time by using multi-touch gestures. Because Windows XP/Vista was not designed for multi-touch there are some issues with this type of input. In multi-touch mode it is not possible to do a mouse click, therefor I added an option to switch between multi-touch and mouse simulation mode.

Please keep in mind that when the mouse simulation mode is activated only single touch input is allowed. Do not use this software on unstable multi-touch systems.

Finally, use this software at own risk 🙂 .

Install instructions are in: readme.txt !

Downloads:
[download#17#size]
[download#2#size] – rev393

Any tips or problems can be posted in the comment section.

* update *
In some cases tDesk does not seem to work properly with touchlib. Try downloading rev393 and overwrite your current touchlib dll files.


Touchlib Speedfix and MMA Pro update

Touchlib

A few weeks ago I was busy tweaking the Touchlib sourcecode and noticed a bug/feature in the code which caused the image processing pipeline to stall for 32 milliseconds. This might not sound like a big issue however on our system it caused the system to be less responsive.

I have compiled a version without this ‘sleep’ function. This version might not be safe for single core cpu’s (I could not test it out myself, hopefully someone could check it out). Because the sleep function is removed, the videoplayback (using cvcapture + *.avi) is not working properly.

Check out this version of touchlib and report any problems in the comment section 🙂 !

Download:
[download#2#size] compiled libraries, revision 393 speedfix (21 apr 2008)

MMA Pro v2d

Changelist:
– Added smoothing, scaled images should look a lot better
– Added an View Reset and Exit button
– Increased the size of the canvas
– Compatible with Adobe Air v1.0

Install instructions are in: readme_pro.txt !

Download:
[download#7#size]
Adobe AIRrequired

Example video showing how to load photos dynamicly into MMA Pro

Wong!

Wong is a game based on pong which uses the multitouch input to create dynamicly sized paddles.

A game of Wong on our multitouch table:

The game is made by Seth Sandler and is part of the audiotouch application suite: http://code.google.com/p/audiotouch/