Friday, December 30, 2016

ATEM Television Studio - live copies of the output file

We recently installed a ATEM Television Studio to mix cameras to an MP4 file in our church building. The idea originally was that we would burn this to DVD, and augment our CD ministry to include video distribution on DVD.
ATEM Television Studio
Since the days when we used cassette tapes, we have always distributed the media directly after the service - so people waiting 5 minutes or so afterwards could take copies of the service to deliver to those who can't make it out.

We wanted to keep this up, and so we looked into a DVD recorder to record the output of the Television Studio module directly to DVD, which could then be duplicated after the service. However, we hit several technical hurdles with this, and had to give up - mainly around aspect ratio.

So we were stuck with the MP4 file on disk. The process of converting this to DVD video, and burning it is slow. Even just burning the 2-3GB MP4 file directly to a DVD for duplicating is a slow process.

After a lot of head scratching we decided to distribute copies of our services on USB memory stick. Many people have a modern TV with a USB input, and we were surprised how many of our elderly people actually had such a device. With this, it is relatively easy to play the video file in superb quality on a TV.

But, we are still left with a problem - how do we write ~3GB of data to multiple USB sticks after the service? Writing to a stick could take 11 minutes, or about 3 minutes for USB 3.0 - but even at 3 minutes, if we have multiple copies to produce it could take quite a while.

Again, more head scratching, investigations into hugely expensive USB duplicators, and USB 3 hubs, but no satisfactory solution. Then the idea struck - why don't we write the data to the USB stick while the service is running - so we tail the file as it grows on the PC, and write the data out to each pen. The bandwidth, even in USB 2 would be more than sufficient to make quite a few simultaneous copies.

So I set to work on a piece of software to do this. I got over the problem of reading from a locked file quite easily, and had a prototype. However, when I ran it, the copy wouldn't play. By looking at binary diffs of the files I discovered that MP4 files have header information at the beginning, which is actually written at the very end of the process. I also discovered the ATEM seems to write large portions of the file filled with zeros, and then fills up the zeros with actual data.

Long story short, after many iterations of my software I got it working. Two strategies helped me:

  1. I don't read to the end of the file until it has been completely written - I stay about 5MB back from the end - this way I don't read any zeros.
  2. As I read each chunk and write it out to the USB sticks, I keep the offset, size and checksum of the data in an list. After the writing is complete, I re-read the source file from the start, and recalculate the checksum. If the new sum is different, I seek to the stored location in the file and rewrite the specified chunk. Typically it only rewrites the first chunk.
I added an option into the program to check the checksum, but since it is working properly, there is now no need to read the entire written data from the stick. 

Copies of the service are available about a minute after the service ends. Actually, these are available long before the CDs of the service become available now! And of course, the same software can be used to make copies of any service, not just the one we just recorded. It means you can make many unattended copies, only limited by the number of ports you have.

Why not YouTube?
Some people have suggested distributing service content via YouTube, but I'm not sure that fits with what we are trying to do: 
  1. It would take a long time to upload and process - some of our elderly people are watching their copy about an hour after the end of the service
  2. We could put internet access into the church building and stream live
    1. that would have associated ongoing costs
    2. our target audience for this ministry generally don't have easy access to the internet
    3. publishing online brings other concerns about who can see the content
  3. Each USB stick is brought into the home of someone who may be housebound, and not see very many visitors during the week. This contact is costly (time/effort) for the person who delivers the media, but incredibly valuable to the person who receives it..
YouTube may be something we think about for the future, but we don't see it as a suitable extension of our current ministry.

No comments: