Video Editing advanced multiple overlays?

[techie]

I was wondering if anyone here has any suggestions for the following problem.
In relation to video editing which I believe quite a few people here on AO might know a lot about.

Goal:

a) Using as little system resources in playback as possible
b) Getting the best possible rendered result in playback
c) Working with regular AVI or MPG video files

Problem:

I have three video clips.
They should be placed as layers over each other.

In other words,
vid0 = background layer, permanently displaying its content full screen.

vid1+vid2 = overlay working as layer on top of vid0 which could be possibly transparent so only exact content is shown and not covering the vid0 layer where no pixels are rendered.

So does anyone know how this could be achieved without needing to use three different tools, flash and a heap of other stuff to create a single screen output result where the three vids can run at the same time and appear as one.

I need to be able to dynamically disable, enable or load any one or more layers at runtime as well so flash is not a suitable mode of operation as it takes 1) to long time and 2) gives to little programming control to do so safely in a Windows based environment without breaking elsewhere in my code.

uzzled:

 

[guy]

What you're asking for doesn't seem to make much sense, so correct me if I'm mistaking what you're trying to do.

You want to edit three separate video sources, so that at playback they remain as three separate, editable layers? I assume this is important because you want to change the parameters of each layer every time you play back the content (for instance, buttons below a Flash video player that lets you vary the transparency of each layer, as the video is playing).

In that case, Flash is still your best tool. In order to knock out content from other video layers that you don't need, you can use keying in Premiere (if you greenscreen your video capture) or rotoscoping in After Effects.



But you mention that you want to create "a single screen output". If you mean that you want to edit multiple layers but output a single file (such as a single AVI; analagous to editing multiple layers in Photoshop and outputting a JPG, just with video instead of still images), then you can accomplish the same with just Premiere (and After Effects, if you need rotoscoping) -- no need to use Flash.



If runtime editability is necessary, you can look for various "Video Jockey" (VJ, or video mixer) software, which similar to audio DJ programs, will let you mix multiple videos on-the-fly. However most of these are limited to just two video sources at a time. Also, their output options can be somewhat limited, so depending on what output you need it may or may not work for you.



In all cases, rotoscoping at runtime is simply not possible. You will have to rotoscope all of your videos manually ahead of time, and then use color keying to knock out content you don't want (or greenscreen your video when you shoot it).

 

[techie]

Thanks for picking up on this topic.

The purpose being, I will write an application that displays video content in a mediaplayer window embedded in the application.

I intend to embed three player windows, where one will move over the other as a layer and certain actions and events in the application will cause the overlayed players to change at runtime.

Flash will not work because it requires loading source files into the flashplayer at runtime from a plain video source which may not be released as a separate video clip.
Hence the flash option is also a nasty piece of work even to embed as it has a whole set of worms with it when distributing files.

All video content is pre-rendered and regular AVI or MPeg clips and as such editing is not needed at runtime, simply the action of load, display, hide or unload.

Thus, defining the transparency, and how to maintain a somewhat decent output so the overlay wont cause blurring and pixel distortion in the lower layer is the tricky part I guess.

There will be no resulting output to file or such, only display on screen during runtime.

 

[guy]

Well, you might not like the idea of Flash as a video container for your application, but you might actually find that alternative renderers might give you more of a headache than Flash itself.

At the very least, the way Flash handles and outputs video on screen is standardized (though not necessarily optimized).

Otherwise you will have to choose a video renderer (the component that actually displays the video on screen) -- off the top of my head, such as VMR7, VMR9, EVR, etc. Different renderers have different capabilities (and complexities), such as whether drawing the video frames on screen is handled by software, passed to the video card/driver for output, how the OS handles it, etc. Some renderers can access DXVA (for improved rendering performance), but may limit you from the ability to layer video.

In all cases, rotoscoping is not an option, and your video will be rendered as a full rectangle. You can still vary the transparency of the entire frame, as well as use other APIs to control other parameters such as rotating or skewing the frame.

The only way to achieve mask transparency (eg: in a video of a person, knock out the background so the underlying video layer comes through, but the person remains 100% opacity) will be to use color-keying, which will be quite CPU-intensive, and will require your videos to all be perfectly greenscreened.

 

[techie]

I guess greenscreening will have to be the way to go.
Graphic cards now a day should handle it better.
I saw a note somewhere that ffmpeg has a support for this by pixel selection of the transparency color and direct rendering of overlays but it generates a single view output file and then I wouldn't be able to modify the placement of the overlay video clip.

Think in the form of "game dev" and you understand the complexity of what I need to achieve.

I should theoretically even be able to take an animated gif and overlay on a video clip, moving the ani-gif around as a mouse pointer and have active playback of selected frames/ time ranges in the background video clip.

Some parts of the transparency I can handle from the dev environment directly, specifying the transparency or opacity of the control used.... of course that part depends on the dev suite used to put it all together.

 

[Rollyco]

Flash or HTML5 are probably your only options of you need interactivity like moving layers around and toggling them on and off.

Avisynth is much easier, but there is no interactivity. You can sort of fake it by playing back an .AVS script in AvspMod, and using sliders to adjust parameters on the fly.

 

[Vitreous]

Not really my area, but these options spring to mind:
- Write your own DirectShow filter. Hard
- Bink video. Expensive
- Flash
- Re-evaluate problem

 

[guy]

Think in the form of "game dev"

Okay. So go with 3D modeling rendered real-time, instead of video clips.

:evillaugh:




Seriously though, trying to manipulate movies in the way that you want sounds like way more trouble than it's worth. Besides, you might be able to figure it out for your computer, but as soon as you put the app on another computer (which has different codecs, different filters, different settings for handling and rendering video), the whole thing will probably fall apart.



I don't want to discourage you though, but if there really is a reasonable way of handling videos in the way that you want, I doubt you'll get a proper answer from AO. You're better off consulting a proper game dev or video forum.

Best of luck!

 

[techie]

Okay. So go with 3D modeling rendered real-time, instead of video clips.

:evillaugh:

Yup considered that too, but the reason to why I started looking at plain video is to get as realistic imagery as possible and not rendering all of it during runtime. The biggest problem for new games and such environments are often the graphics card requirement.

The one thing promising about this is probably Blender which has a very low use of ram to do pretty much anything however this still means you wont get truly photo realistic backdrops.

As far as the commentaries on AO, I tend to believe ideas and input from end users are quite a lot more useful for developers and anyone working with it from any perspective.

Just take the simple fact that AO has its own video encoder app, professional photographers frequent this board, and folks from the media/video market too such as yourself.

That is much more interesting than sitting reading commentaries on a developers board on how to count the latest number of fps using OpenGL with pixelshader Mpfh whatever just because someone loves working with CSharp and open source, forgetting the latest version more or less requires 512 MB RAM and pixelsshader 2.1 support to even open the game's splash page. (see empire total war for example)

The reason I try to work around much of the backend stuff using plain video clips is so I can get away from much of the overhead of rendering at runtime where material can be provided simply as a backdrop.

See for example the new 360 degree panoramic video cams that Google licensed to use for google streetview.

Then imagine a control that allows you to walk in an environment using true video, and rendering only the player figure for example. Any additional elements, controls and such can be floating transparent videos in their own controls and so forth... the options are endless, provided one work out the minor choices of decent tools and players which should work on most systems to date.

Out of all that I believe the codec choice will be the least of the hassle as long one runs with relatively well known standards.

 

[Vitreous]

Are you trying to make some kind of interactive game using video scenery? The player will have to more-or-less move on rails, as you will be limited by the video shots you have. If the player can move around with any freedom, then how will you handle collision between the player and the walls/objects in the video? What about shadows cast from player into the scene? Or lighting/shadows from the scene onto the player? Without all that the effect will look cheap, not realistic. The same issues will apply to any overlaid video elements intended to be "within" the scene.

Just because you're using video doesn't mean that the end result will look realistic - especially if you want any kind of flexibility or interactivity.

 

[techie]

That will truly be the test of the whole thing.
Actually in part I was discussing this with another guy for conceptual ideas and we where thinking of using real video for parts and rendered material for other sections.

The overlay idea came out of a consideration how applications today render content but to some extent the content is not perfectly random but rather repetitive.

Consider for example how if you in some game stand at a lake shore, viewing the waves for some time you will start noticing the waves are simply repeating the same cycles of a standard rendered wave.

We where discussing how to add more natural behavior in the wave patterns and avoiding repetition down to that level. Of course now we're really nitpicking details but since these factors will/would require a lot of rendering capacity as well as processing GPU's we could perhaps mitigate it with overlays, mixed with rendered material.

As for the shadows I think it will be less of a problem than to say keeping "normaly dynamic" elements out of the video.

You wouldn't want to shoot a scene down the street where some people are in the picture, because if it is a first person shooter game you can never "kill" those characters even if you tried. You would just end up with a bunch of "immortals" in the scene.

We have already considered things like that and have a good idea on what to include and avoid, and then it will be a lot of trial and error but I believe in the end it could possibly give a good result.

At least without having to pull a "Saving Nemo" version, sending people to Hawaii to learn how to draw cartoons of water just because water is one of the hardest things to get to look realistic if anything.

All this simply because I really want something that scares the living daylight out of people when they sit in front of the screen :sadomaso:
Assassins Creed on steroids with a machine gun making Jason vs. Freddy look like my nieces kindergarten play.

 

[guy]

Unfortunately, for what you're asking, I personally think it just isn't feasible. You would either need to 1) impose so many limitations vis-a-vis "player on rails" as Vitreous suggested in order to get the video content to realistically correspond to any CG content, or 2) shoot so much video that a single BD-DL disc (50GB) couldn't even hold.

One suggestion:
You may want to look into what developers are making with regards to augmented reality. It's not exactly what you're looking for, but it might provide some ideas (or alternatives) about how to execute video+overlayed CG/rendering.

One caution:
Seamlessly mixing video with rendering is a painful process. Most DVD content (think sci-fi, like Star Wars) pulls off greenscreened video+CGI quite successfully, largely because a lot of the tiny visual differences between film and render are lost in compressing the output to the NTSC 720x480 frame. Look at the same movie in Blu-ray, and suddenly the differences between film and render are much more obvious, with CGI being more "obviously rendered" because of an inconsistency between the grainy quality of filmed content (eg: slight haziness around characters) versus the smooth, sharp, precise lines of CGI.

Film effects studios spend tons of money and server farm power to painstakingly render in film-like effects for HD output (movies, Blu-ray) -- and more so because HD is ever more increasingly popular.

You could manage this disparity (as well as overall content filesize) by limiting yourself to SD (video and render), but then your gameplay will look tiny on most computer screens (1280x1024 or larger), and when stretched fullscreen, the video will become noticably blurry compared to rendered material due to upsampling.




There's a lot to think about, and because of storage and processing limitations (respectively for storing tons of HD video content, or rendering photo-realistic CGI), it just hasn't been pulled off successfully yet.

I daresay, however, that if you are able to come up with a very clever way of combining video and CGI seamlessly for immersive gameplay, you shouldn't waste your time on AO trying to make a prototype. Instead, you ought to go straight to a major game studio and pitch your idea (and quite possibly make a good amount of money).

 

[techie]

Actually the compression is something I have spent massive amounts of time on lately.
Well in fact I came up with a truly wacky idea on that many years ago and it became a pet-project of mine that one day it should come into form as an application by itself.
I have just spent 48 hours running a package test against the fastest thing I could lay my hands on and it's still far from fast in mathematics.

Anyway, not if, but when we squeeze out some solution for this I bet a lot of heads should turn, but we'll see. First of, a bit of ideas, a concept to work with and then we'll see which parts we can work into it.

Who knows, perhaps the game engines I have on my desk to review now can bring some extra ideas how to get Blender type speed into this. That is probably the best I have seen so far, running with as little as down to 8 Mb RAM only.

 

[guy]

Speaking of compression, you might be interested in this article about "sparsity" and its obvious implications for data compression:

http://www.wired.com/magazine/2010/02/ff_algorithm/

 

[techie]

Speaking of compression, you might be interested in this article about "sparsity" and its obvious implications for data compression:

http://www.wired.com/magazine/2010/02/ff_algorithm/

Interesting indeed but there are a few other algorithms out there about to replace the Shulz financial option trading models that could do even better than fft and similar Naive Bayesian algortihms.

I spoke to a friend of mine who is doing a PhD to math for quant developers now and he couldn't even make heads or tales of my way of working
Guess I have a neck for confusing these guys.

 

[Alpeia]

To aviod the the keying, you can simply rerender the movies with a alpha channel mask. (Meaning you key it with compositing software and render out a mask of that key, as an alpha channel.) This should allow playback in real time. But you are limited to what codecs support alpha channels. Quicktime player supports movies with alpha channels. But I agree you are limited in size of movie, reason being that you are limited to your hard drive data rate. That's why most realtime editing systems have raids attached and have there own video card for layering and effects. You would just need an app to layer them that can use the alpha channels. You could write your own using Open GL or the GPU engine (sorry can't remember the GPU protocal software name, I think it's Sun that has open source for it.) And there are ways of cheating, like every other image line is replaced with a black one. Scanline 1 is image line 1 & scanline 2 is a black line, scanline 3 is image line 3 and 4 is a blackline etc. PS1 used this method.

Also maybe just have a background that is a video, and put sprites on top made from pieces from the other two video's. But I guess that's a lot of work.

And I agree that if you are designing a game to pitch, don't waste time on anything else. If you sell your idea, don't worry they have people they pay to work out that stuff.

If this is just about water, it took over a year to come up with an algortihm. But it was rendered on a rendering farm and could not be used in realtime. The only thing that looks promising is http://unlimiteddetailtechnology.com/videos.html but it is aways off and if they sell out to one company, and if it really works, just sad. They use point cloud technology.

======Off topic=========
Also for knowledge, NTSC is 8 bit where HD is 10 bit log. There is more colour information in HD, and of course more detail due to the size. Film is 2056X1553. As far a the blur around stuff, that is just bad compositing, where they lay the CG over the live action plate they blur the CG edges to make it look natural, but if they don't do it right you can really see it. Most studio's will add film grain to CG elements so it fits in with the film or even soften them. The most errors I see is not matching the lighting or incorrect black point. Most film manufactured these days is almost grainless, just look at a film made in the early 80's and compare it to any film made resently that wasn't filmed digitally.

With Starwars episode one & two, they were cheating and mixing resolutions... Bad ILM.

The problem with CG is the colour gamut is greater then film and greater then the human eye. If they don't know what they are doing this can create big problems.

When a film is complete is goes into a colour suite where they make a digital colour master. That master is used to make a master print. They also (now a days) use the digital colour master frames for the Blueray & DVD (making slight colour adjustment overall for the specs of each format.)

Sorry for the off topic info, I just couldn't help myself...
Bad Alpeia, bad.