PROGRAMMATION - outils d'analyse vidéo et de gestion vidéo

MAX MSP / JITTER / CYCLOPS / SOFTVNS / PURE DATA [GEM] / LIENS

Introduction

MAX est, à l'origine, un environnement qui permet de contrôler des machines MIDI.

MSP est une série d'objets qui permettent de générer et traiter des signaux sonores.

NATO.0+55+3d (0f0003.MASCHIN3NKUNST) est une librairie d'objets qui permettent
de manipuler des éléments vidéo dans l'environnement MAX.

SoftVNS (David Rokeby) est une autre série d'objets dédiés au tracking et à la vidéo en temps réel.

Jitter est une série d'objets matriciels qui permettent de manipuler en temps réel des données vidéo et audio.

MAX MSP

MAX/MSP est un environnement visuel pour la programmation d'applications interactives temps réel. C'est actuellement la référence mondiale pour la création d'installations sonores interactives. MAX/MSP est la combinaison du logiciel MAX (IRCAM/Opcode/Cycling'74) pour le contrôle temps réel d'applications musicales et multimédia interactives par MIDI et de MSP, une bibliothèque d'objets pour l'analyse, la synthèse et le traitement du signal audio en temps réel. MAX/MSP est conçu pour les musiciens, les designers sonores, les enseignants et les chercheurs qui souhaitent développer des programmes interactifs temps réel.

MAX/MSP est développé par la société Cycling'74 (USA) sous licence exclusive de l'Ircam.

top

JITTER

Jitter is a set of 135 brilliant new video, matrix, and 3D graphics objects for the Max graphical programming environment. The Jitter objects extend the functionality of Max4/MSP2 with flexible means to generate and manipulate matrix data -- any data that can be expressed in rows and columns, such as video and still images, 3D geometry, as well as text, spreadsheet data, particle systems, voxels, or audio. Jitter is useful to anyone interested in real-time video processing, custom effects, 2D/3D graphics, audio/visual interaction, data visualization, and analysis.

Since Jitter is built upon the Max/MSP programming environment, the limitations inherent in fixed purpose applications is eliminated. You are able to build the programs you want to use, rather than being forced to work around somone else's idea of how things should be done. This power is not to be underestimated, so please use it wisely.

Jitter 1.1 System Requirements

• A PowerPC MacOS computer (G3/300 or faster with an OpenGL hardware-accelerated video card is recommended).
  • Jitter 1.1 for OS9 requires MacOS 9.x and 64 MB of RAM.
  • Jitter 1.1 for OSX requires MacOS X 10.2 or higher and 256 MB of RAM.
• Max/MSP 4.1 (for OS9) or Max/MSP (for OSX) installed on your system.
• QuickTime 4.0 or later (QuickTime 5 or later recommended).
• OpenGL 1.2 or later installed on your system.

top

CYCLOPS

• track dancers or other performers on stage
• interact with users in interactive installations
• create video-tracking-based musical instruments and performance interfaces
• create non-invasive user interfaces

How Cyclops Works

Cyclops receives live video from a QuickTime input source and analyzes each frame of captured video in real-time. It divides the image area into a grid of rectangular zones and analyzes the zones for greyscale, threshold, difference (motion) and color. Cyclops allows you to specify the grid resolution, target zones for analysis and indicate the type of analysis to be performed in each zone. Cyclops outputs messages for each analyzed video frame that can be used to trigger any Max processes or control any patch parameters.

top

SoftVNS

softVNS 2 is a set of external objects for MAX/MSP that allows you to process video in real-time.

David Rokeby est un des pionniers dans la conception d'installations interactives, sonores et visuelles, depuis 1982. Il développe ses propres outils technologiques (traitement vidéo en temps réel, motion tracking, logiciel softVNS) au service d'une pratique et d'une réflexion sur les environnements interactifs. Son travail d'artiste est basé sur l'immersion du corps humain dans des systèmes de reconnaissance artificielle. Il explore ainsi le temps, les modes de perception de l'homme, ses relations avec les machines interactives, les questions autour de la surveillance digitale.

Highlights:

Extremely fast!
All functions are coded for the velocity engine on the G4. This results in 4 x the processing speed of non-velocity engine code. And video can be processed in YUV mode as well as RGB. YUV yields twice the performance of rgb with no visible loss of quality. Add the two up and you get up to 8 times the video performance. (Note: softVNS does not speed up quicktime movie access itself)

High Quality processing
Due to the speed of softVNS 2, most functions offer full anti-aliasing and interpolation. Zooms, pans, displacements, rotations, etc., are very clean and detailed without jaggies.

Very Stable
softVNS has been in use in interactive video installations in its beta form over the past 2 years. Some of these are permanent exhibits that have been running all day every day for well over a year without downtime.

Extensive Tracking functions
softVNS 2 was developed with a focus on real-time video tracking, so softVNS includes a broad range of tools for tracking, including presence and motion tracking, colour tracking, head-tracking, object following, etc.

640 x 480 video capabilities
With the right digitizer, softVNS 2 can process 640 x 480 yuv colour images in real-time at 30 frames per second. (Depends of course on the complexity of the video stream).

works with jitter (compatibility objects included)

*SoftVNS developper David Rokeby is a sound and video installation artist based in Toronto, Canada. He has been creating interactive installations since 1982. He has focussed on interactive pieces that directly engage the human body, or that involve artificial perception systems. His work has been performed / exhibited in shows across Canada, the United States, Europe and Asia.

top

PURE DAT [GEM]

Pd is a real-time graphical programming environment for audio and graphical processing. It resembles the Max/MSP system but is much simpler and more portable; also Pd has two features not (yet) showing up in Max/MSP: first, via Mark Dank's GEM package, Pd can be used for simultaneous computer animation and computer audio. Second, an experimental facility is provided for defining and accessing data structures.

The main window, canvases, and printout

When Pd is running, you'll see a main "Pd" window, and possibly one or more "canvases" or "patches". The main Pd window looks like this:

There are peak level and clip indicators for audio input and output; these report peak levels over all input and all output channels. Note that DC shows up as an input level; many cards have DC levels which show up in the 50s. To see an RMS audio level, select "test audio and MIDI" from the Media menu. The main window display is intended only to help you avoid clipping on input and output. You can turn the peak meters on and off using the control at bottom left.

At bottom right is a control to turn audio processing on and off globally. Turning audio off stops the computation and relinquishes any audio devices Pd is using. The "Media" menu is also provided, with accelerators "Control-." to turn audio computation off and "Control-/" to turn it on. When audio is on, Pd is computing audio samples in real time according to whatever patches you have open (whether they are visible or not).

Installing Pd in Macintosh OSX

Pd version 0.35 and up support Macintosh OSX. You need the Jaguar distribution (10.2) of OSX.

To install Pd you can always just download the sources and compile them yourself, or (easier) find a MacOSX-style "package". The first package was put together by Adam Lindsay; the most current one seems to be the one by Hans on http://www.pure-data.org/Members/hans/. The package simply installs itself and you needn't follow the directions below.

GEM introduction

GEM is the Graphics Environment for Multimedia. It was originally written by Mark Danks to generate real-time computer graphics, especially for audio-visual compositions. Because GEM is a visual programming environment, users do not need any experience in traditional computer languages.

GEM is a collection of externals which allow the user to create OpenGL graphics within Pd, a program for real-time audio processing by Miller Puckette (of Max fame).

There are many different shapes and objects, including polygonal graphics, lighting, texture mapping, image processing, and camera motion. All of this is possible in real-time without any previous programming experience. Because GEM is an add-on library for Pd, users can combine audio and graphics, controlling one medium from another.

GEM is supported in part by a grant from the Intel Research Council for the The Global Visual Music project of Vibeke Sorensen, Miller Puckette and Rand Steiger.

An important thing to remember is that GEM is NOT an application. It is a library that Pd loads at run-time. Make sure that you see the section on using GEM with Pd. This manual assumes that you have Pd working correctly and can load up patches already. If you do not have that working yet, look at the Pd manual and the GEM FAQ. Also, it is assumed that you have a basic understanding of how to use Pd and the idea behind the data flow model. In other words, if I ask you to pass a message with 3 floats into an object, you would know what I mean.

The system requirements vary depending on your system and what you are trying to do. In general, you should have the most powerful computer available and the best graphics accelerator on the market. In reality, people have been doing some amazing work with a Pentium II and an nVidia Riva TNT or 3Dfx Voodoo2 card. If you are on an SGI, then everything from an O2 up seems to be okay. The biggest requirement is that you have some kind of OpenGL graphics accelerator. This means that a Matrox Millennium II will not run very quickly.

The other factor is what you are trying to do. Pushing real-time video around requires a fast bus, which really only exists on SGIs. Doing thousands of texture mapped polygons is great on a PC...if it is a constant texture. There are many issues which mean that there is no one answer to "Is this system good enough?". In general, you will have to try and see.

Exemple d'objets GEM

Pixes

pix_2grey - convert rgb pixels to grey (still an RGBA image)
pix_a_2grey - convert rgb pixels to grey based on alpha channel
pix_add - add two pixes together
pix_aging - super8-like aging effect
pix_alpha - set the alpha value of a pix
pix_background - let through only pixels that differ from a static "background" image
pix_backlight - a backlight photo effect
pix_biquad - 2p2z-filter for subsequent images
pix_bitmask - apply a bitmask to a pix
pix_blob - get center of gravity
pix_buf - buffer a pix
pix_buffer - storage room for pixes (like [table] for floats)
pix_buffer_read/pix_buffer_write - put/get pixes into/from a pix_buffer
pix_chroma_key - color keying (like "blue-box")
pix_coloralpha - set the alpha-channel of a pix as a mean-value of the color-components
pix_colormatrix - recombine the RGBA-channels with matrix-operation
pix_color - set the color of a pix (leaving alpha alone)
pix_colorreduce - reduce the number of colors (statistically)
pix_composite - composite two pixes together
pix_convolve - convolve a pix with a kernal
pix_coordinate - set the texture coordinates
pix_crop - get a sub-image of a pix
pix_curve - apply color-curves onto a pix
pix_data - get pixel data information
pix_delay - frame-wise delay
pix_diff - get absolute difference of two pixes
pix_dot - rasterize a pix with big dots
pix_draw - draw a pix
pix_dump - dump the pixel-data as a long list of floats
pix_duotone - reduce the number of colors by thresholding
pix_film - use a movie file as a pix source for image-processing

top

LIENS

Paragraphie, Manon De Pauw - Installation avec MAX / SoftVNS
http://www.er.uqam.ca/nobel/interstc/projets/mdpAutoRepeat.html

ORACLE - installation interactive
http://www.ekran.org/ben/oracle/process/

Cycling 74 - Max/Msp - Jitter - Cyclops
http://www.cycling74.com/products/index.html

Le site officiel de Pure Data
http://www.pure-data.org/about/

Documentation Pure data (english)
http://crca.ucsd.edu/~msp/Pd_documentation/

Movement and Sound, Controlling Digital Audio with softVNS
http://music.utsa.edu/tdml/conf-VIII/VIII-Ball/VIII-Ball.html

SoftVNS de David Rockeby
http://homepage.mac.com/davidrokeby/softVNS.html

Résonances 2003 - divers projets
http://www.musicrun.com/evenements/ircam2003/resonances.php3