meguIV: The Official Akiba-Online DVD Encoder (v1.0.1.1)

[anandneemish]

Hi Vitreous. It seems that I figured out what went wrong.

1. Boneheaded mistake on my part. When I copied the script into a newly created QuickTGMC.avsi, there was some garbage text at the end that I did not notice. Since the garbage was not preceded by a "#", it was interpreted as a command. Fixed this, and then ran into another issue.

2. I placed "yadif.dll" into "C:\Windows\system32" as you had directed. However, this file was not found. After re-reading the QuickTGMC.avsi comments, I found that I had to manually specify where the DLL is located. (I guess system32 is NOT the default location in the script you posted). So I moved it out of the system32 dir and back to where the rest of MeguIV files are located. I specified the location, saved the QuickTGMC.avsi file and re-inserted it into the virtual plugin dir.

Now things seem to be working.

At the "Very Fast" setting, I am getting 35+ fps.

Awesome. I wub you man. And Rollyco too

 

[Vitreous]

I placed "yadif.dll" into "C:\Windows\system32" as you had directed. However, this file was not found. After re-reading the QuickTGMC.avsi comments, I found that I had to manually specify where the DLL is located.

The Yadif plugin is a pain as it won't autoload - I suggested windows\sytem32 as the potentially easier option - maybe avoiding editing the script. The method you used is the second more robust alternative I put in the instructions. Looks like I'll have to look suggest the second option only. As it turns out, only "Super Fast" and "Ultra Fast" use Yadif by default.

At the "Very Fast" setting, I am getting 35+ fps.

Very Fast is a nice choice - the fastest mode that still uses the high quality NNEDI2 interpolation (no residual combing). Glad it worked out for you...

Awesome. I wub you man.

I'm not sure I'm ready to be wubbed :scared:

 

[Rollyco]

You said that
QuickTGMC(Preset="Slow", Sharpness=0.4)
is equivalent to
TempGaussMC_beta2(2,1,1, Sbb=0, EdiMode="nnedi2", SVthin=0.0, sharpness=0.40)

Does that mean that
QuickTGMC(Preset="Slow")
is equivalent to
TempGaussMC_beta2(2,1,1, Sbb=0, EdiMode="nnedi2", SVthin=0.0)

?

 

[Vitreous]

Does that mean that
QuickTGMC( Preset="Slow")
is equivalent to
(2,1,1,Sbb=0,EdiMode="nnedi2",SVthin=0.0)

Strictly it's nearest equivalent is:
(2,1,1,Sbb=0,EdiMode="nnedi2",SVthin=0.0, border=false)
I found border mode to cause as many problems as it solved and switching it off is a speed up. Edit: Almost forgot it's a mote sharper =1.2 rather than 1.16, tweaked to make sharpness parameters more equivalent across modes

Even by changing parameters to exactly the same it will never be identical to the pixel as I fixed a bug in the MVTools parameters that Didee used, which has improved motion search in QuickTGMC. If comparing - check the uncompressed versions, so as not to be distracted by the x264 compression choices.

Full table of settings reproduced from script - just read downwards. This table can be tweaked in the script to suit your preferences:

	# Preset groups:                                     Placebo    V.Slow    Slower    Slow      Medium    Fast      Faster    V.Fast    S.Fast     U.Fast
	tr0          = default( tr0,          Select( pNum,  2,         2,        2,        2,        2,        2,        1,        1,        1,         1        ))
	tr1          = default( tr1,          Select( pNum,  2,         2,        2,        1,        1,        1,        1,        1,        1,         1        ))
	tr2          = default( tr2,          Select( pNum,  3,         2,        1,        1,        1,        0,        0,        0,        0,         0        ))
	rep0         = default( rep0,         Select( pNum,  4,         4,        4,        4,        4,        4,        0,        0,        0,         0        ))
	rep1         = default( rep1,         Select( pNum,  0,         0,        0,        0,        0,        0,        0,        0,        0,         0        ))
	rep2         = default( rep2,         Select( pNum,  4,         4,        4,        4,        4,        4,        4,        4,        4,         0        ))
	EdiMode      = default( EdiMode,      Select( pNum, "NNEDI2",  "NNEDI2", "NNEDI2", "NNEDI2", "NNEDI2", "NNEDI2", "NNEDI2", "NNEDI2", "TDIYadif","TDIYadif"))
	NNSize       = default( NNSize,       Select( pNum,  2,         2,        1,        1,        0,        0,        0,        0,        0,         0        ))
	SMode        = default( SMode,        Select( pNum,  2,         2,        2,        2,        2,        2,        2,        1,        2,         2        ))
	SLMode       = default( SLMode,       Select( pNum,  2,         2,        2,        2,        2,        2,        2,        1,        1,         0        ))
	SLRad        = default( SLRad,        Select( pNum,  3,         1,        1,        1,        1,        1,        1,        1,        1,         0        ))
	Sbb          = default( Sbb,          Select( pNum,  3,         1,        1,        0,        0,        0,        0,        0,        0,         0        ))
	NoiseBypass  = default( NoiseBypass,  Select( pNum,  2,         2,        0,        0,        0,        0,        0,        0,        0,         0        ))
	NoiseDeint   = default( NoiseDeint,   Select( pNum, "Generate","Bob",    "Bob",    "Bob",    "Bob",    "Bob",    "Bob",    "Bob",    "Bob",     "Bob"     ))
	BT           = default( BT,           Select( pNum,  5,         3,        0,        0,        0,        0,        0,        0,        0,         0        ))
	SrchClipPP   = default( SrchClipPP,   Select( pNum,  2,         2,        2,        2,        2,        2,        1,        0,        0,         0        ))
	Blocksize    = default( Blocksize,    Select( pNum,  bs,        bs,       bs,       bs,       bs,       bs,       bs2,      bs2,      bs2,       bs2      ))
	Overlap      = default( Overlap,      Select( pNum,  bs/2,      bs/2,     bs/2,     bs/2,     bs/2,     bs/2,     bs2/2,    bs2/ovf,  bs2/ovf,   bs2/ovf  ))
	SubPel       = default( SubPel,       Select( pNum,  4,         2,        2,        2,        1,        1,        1,        1,        1,         1        ))
	Precise      = default( Precise,      Select( pNum,  true,      true,     false,    false,    false,    false,    false,    false,    false,     false    ))

Edit2: There are other parameters of course, whose defaults are either constant across presets (e.g. border), or vary in a more complex way (e.g. sharpness). The best way to see all the default settings for a given preset is shown below. All the settings that share a name with original TGMC behave the same (for now - sharpness is likely to be normalized in a future version)

QuickTGMC( Preset="Medium", ShowSettings=true )

 

[Rollyco]

Mr. Vitreous, can you share your .AVS and x264 commandline for the 30fps IMOG-159 you released recently? It looks particularly nice...

:harp:

 

[Vitreous]

Mr. Vitreous, can you share your .AVS and x264 commandline for the 30fps IMOG-159 you released recently? It looks particularly nice.

Of course (lol at the Mr), it's nothing special:

AVS: QuickTGMC on "Slower" plus an unreleased version of my EdgeCleaner (which I always use on Imouto material). Custom AVS for the menu at the start. Variable bitrate AAC (Q=0.4).
x264 [build 1659]: CRF 19.5, Slow Preset, Default Tuning, Psy-Trellis 0.10. b-pyramid is left on, unlike MeguIV - b-pyramid is good, don't know why you switch it off (it does work with mb-tree now - has for a while)

Main AVS:

global MeGUI_darx = 16
global MeGUI_dary = 9
SetMTMode(3)
LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\WIP\IMOM-159\VIDEO_TS\VTS_01_1.d2v")
SetMTMode(2)
QuickTGMC( Preset="Slower" )
crop( 0, 2, 0, 0)
Spline36Resize(720,480)
EdgeClean( Radius=2.5, DeHaloRad=2.5 )
Distributor()

Second Pass AVS (from 60fps uncompressed file):

global MeGUI_darx = 16
global MeGUI_dary = 9
AviSource("E:\Working\hfyu_VTS_01_1.avi")
SelectEven()

x264 command-line:

program --preset slow --crf 19.5 --psy-rd 1.0:0.10 --output "output" "input"

 

[Rollyco]

Thank you, can I see your EdgeClean code?

 

[Vitreous]

Thank you, can I see your EdgeClean code?

Actually editing it right now and was editing it during the rip so I don't have the version used for the rip - it was in an intermediate but working state. I'll up it when I'm done.

However, the main changes at that point were for speed, so it should be similar to existing version 1.15. That version can't fully manage sub-pixel accuracy so you could try:

EdgeClean( Radius=2, DehaloRad=2.5 )

It's a big slowdown though with all the other recent speedups (hence my current editing).

 

[Vitreous]

Have updated my EdgeCleaning script to version 1.2. Main changes are some optimization (as everything else is getting faster now), removal of need for the Dehalo_alpha plugin, and support for half-pixel accuracy.

I use this script and its aggressive dehaloing mode on every Imouto rip. I usually just use its light cleaning mode on other rips. It's still a performance drain - good edge detection and masking just isn't fast...

 

[shadeofgray]

Have updated my EdgeCleaning script to version 1.2. Main changes are some optimization (as everything else is getting faster now), removal of need for the Dehalo_alpha plugin, and support for half-pixel accuracy.

I use this script and its aggressive dehaloing mode on every Imouto rip. I usually just use its light cleaning mode on other rips. It's still a performance drain - good edge detection and masking just isn't fast...

I haven't tried your earlier version(s), but this one is giving me an error. The problem seems to be in dehaloing (there is no problem when I use just Radius parameter) or more precisely in m4() function.

106: # Convert integer to multiple of 4 which is  >= 16
107: function m4( float x ) { return (x < 16 ? 16 : x - (x % 4)) }

MeGUI tells me that modulo '%' operands must be integer and it looks like 'x' is sometimes floating point. Depending on DehaloRad value, error occurs when m4() is called from this line:

123: dw = m4( w/Radius )

or this line:

125: uw = m4( w*_superScale )

This did the trick:

107: function m4( float x ) { return (x < 16 ? 16 : int(round(x / 4.0) * 4)) }

I don't know anything about avisynth so I don't know if it's just me or if it's something else. :exhausted:

 

[Vitreous]

I don't know anything about avisynth so I don't know if it's just me or if it's something else.

That's well spotted and you're exactly right about the problem. I was optimizing hard and converted m4 to a faster integer operation - forgetting that the input dimensions can be floating point. For some reason it gives me no error though - I guess I'm using a different version of AviSynth (I'm using one of the multiple MT revisions of course, but I forget which one).

Your solution is correct - use the original version from Dehalo_alpha; that optimization was tiny anyway. I've updated the OP. Thanks!

Anyone else who was trying EdgeClean v1.2, please update to v1.21

________

Edge cleaning examples moved to original post

 

[Rollyco]

Vitreous, what do you know about mencoder crashes? On my computer they are non-deterministic and related to the mvtools dll. QuickTGMC slow preset seems to be a lot more prone to crashing than TGMC on my computer.

Have you experienced them? Have you found a way to stop them from occurring?

 

[Vitreous]

Vitreous, what do you know about mencoder crashes?

Not much more than before - other than I'm pretty sure that some MT revisions to avisynth are badly implemented and at fault. I've just remembered that I switched to using SEt's MT-enabled Avisynth 2.6 Build (he also updates a 2.58 MT version). MeguIV uses the original 2.57 MT revisions. SEt seems to pay more attention to threading issues. I wonder if that will help...

I think it is inevitable that more heavily optimized code will be more sensitive to these issues (QTGMC & the new masktools (I assume you meant them - the only recent change to mvtools is very minor)). I have had a few crashes, but only when I run other disk-access heavy tasks in the background (uTorrent, Share etc.). So there are still some problems somewhere...

 

[Rollyco]

Do you remember having any crashes with SEt's 2.6 MT build? His 2.58 MT build hasn't solved the problem for me.

I'm wondering if disk access is a factor, or perhaps memory. I wish I could pinpoint a culprit so I could at least workaround the issue. It doesn't help that the crashes are very rare

 

[Vitreous]

Do you remember having any crashes with SEt's 2.6 MT build?

Yes, I have (edit: actually maybe they were for good reason - see post below). I've read some suggest that adding something like this at the top of the script might help avoid running out of memory:
SetMemoryMax(384)
EDIT: That number needs only be high enough for the frames processed by your script, and no more. So you could try 256 or even 128. Too low and the script will be slow apparently.

Another thing I tend to do is to read the source from one disk and write the output to another on each pass. I do this by copying the avs file to a different drive than the ISO just before processing it - intermediate files are saved in the same folder as the avs.

 

[Rollyco]

I'm going to have to play around with SetMemoryMax.

I have the impression (needs double-checking) that on an MT setup it applies to each thread. So if you call SetMemoryMax(128) and you run TGMC on a quad-core, the total max memory usage will in fact be 512MB.

Another thing that might be important, if not calling SetMemoryMax() explicitely, is that AviSynth automatically sets the Max based on the free physical memory at the time the script is initialized. If free physical memory changes substantially _after_ the script has been running, you might run into problems. That's something to watch out for.

 

[Vitreous]

Reposted:

I've just noticed that one of the drives I use for intermediate files is almost full - just had a mencoder crash when it filled. Now I think the only crashes I've had with Avisynth 2.6 MT have been on this drive - just never noticed the diskspace issue before. So possibly retract that last statement regarding me having 2.6 MT crashes.

Just did an hour-long rip with QTGMC("Slower") and full edge-cleaning/dehaloing using SetMemoryMax(256) and SetMTMode(3,6) - see below. Did some browsing and coding at the same time. No problems.

Without SetMemoryMax, Mencoder uses over 1Gb of memory during processing.
With SetMemoryMax(256) it uses 370Mb and runs the same speed
Clearly that's an issue - wonder what the minimum safe value is?

Close look at Process Explorer - I think there is a minor memory leak, but only about 10Mb over the entire rip so not likely to cause a major issue. I'll see if I can identify exactly which plugin is responsible.

I have the impression (needs double-checking) that on an MT setup it applies to each thread.

Using SetMTMode(3) as the first MT command implies one thread per processor (only the first SetMTMode in the script can set number of threads - i.e. the one prior to the source load)
Trying SetMode(3,1), then (3,2) etc. up to (3,6) to explicitly set #threads
Memory increases only a little each time regardless of SetMemoryMax - i.e. memory / thread not much of an issue

However, a nicety, I get a speed boost on a quad-core with (3,5) and (3,6) - fills out the 'gaps' left by using just four threads. Further stresses the processor though.

 

[anandneemish]

I'm wondering if disk access is a factor, or perhaps memory. I wish I could pinpoint a culprit so I could at least workaround the issue. It doesn't help that the crashes are very rare

For what it's worth, it seems that moving/copying very large files during an encode (amongst disks NOT affiliated with the encode) causes mencoder problems for me. When looking at Microsoft Taskman, I was surprised to see how much memory is used for a simple file move.

So disk access may be related to memory usage.

Of course, it could just be that the disk controllers are hiccuping under the stress of servicing two intensive tasks, and are throwing errors that are not corrected fast enough.

 

[anandneemish]

Increasing contrast?

I have a vid source that is somewhat dark and murky. Running Rollyco's High Quality settings with sharpness=0.60 produced some marginal improvements.

I read up on Vitreous' edge sharpening plugins, but I don't think that's going to solve my problems. What do you recommend to punch up the overall brightness and contrast of the video?

Since the Garbage In Garbage Out principle holds true, I don't expect miracles. But I would still like the video to be better than it is right now.

Any recommendations on settings in the AVISynth config or even plugins that I should use? Thanks.

 

[Vitreous]

What do you recommend to punch up the overall brightness and contrast of the video?

Tweak is simple to use:

# Load source etc....
#...
QuickTGMC( Preset="Medium" )
SelectEven()
Tweak( bright=32.0, cont=1.25, coring=false )
#...
# Crop, resize etc.

'bright' is in the range -255 to 255, default is 0
'cont' (contrast) is in the range 0 to 10, default is 1
Experiment with values - usually want to raise both together to brighten without washing out
Leave Coring at 'false' or it will limit the output color range for TVs

An alternative is to adjust the levels. For example:

#...
Levels( 0, 1.2, 224, 0,255, false )
#...

The form is:
Levels( InputBlack, Gamma, InputWhite, OutputBlack, OutputWhite, Coring )
Increasing gamma (from default of 1) will increase brightness, but weaken contrast. Bringing input white down (from a default and maximum of 255) will brighten and increase contrast, increasing input black will darken and increase contrast. Output black and white will narrow the output range and probably won't improve things.