KUDOS...

The Audio/Visual guys are solely responsible for any recording failures. People have to accept responsibility for what they do or DO NOT do.
They should inspect and test every single recording before, during, and after the recording session.
Two devices to record all video and Audio at all times.
Pause recording in the beginning of a session and check to see that in fact there is a recording.
Don't wait 2 days after the event to tell us that the video failed to record. If in fact there was a failure to record, it would have been know the same day.
Along with 2 recording device that record audio and video, there should be a separate recording of ALL audio so that in cases like this we would still have audio.
As it is now, we have nothing? All videos and any audio recording should be saved immediately before ANY processing of the data.
The RCSC has to do better than this.

 

All cameras had a problem with the SMS card? Highly unlikely.

 

I think the real problem that everybody is missing is the fact that nobody hit the "record" button?

The simm cards are simply where the video is being recorded onto!

 

So where is the record button? Is it on each camera? If so, whose responsibility is it to press the record button?

 

I kinda got the impression, and I may be wrong, that all the cameras fed up into the booth and everything was recorded on a single devise up there?

I think they're now going to record from each camera?

 

That's my impression - one card in booth and that card failed. Bill said now each camera will have individual card (3x) and there will also be three independent audio recording devices. If what Mike said about recovery is possible, than that should be pursued immediately (and calmly).

 

Let me clarify my comments at the Board Meeting on Thursday 12/16. I heard Bill Cook refer to a corrupted SDCARD. I'll have to go back and watch the video, but that's what I remember him saying.

SDCARD is a memory card, much like a flash drive. To Video the 90 minutes of Video, it would require a large card (maybe 250GB or larger). These cards can fail. The memory card has a structure for saving data blocks (pieces of the video). If the memory failure occurs in the structure (kind of index), you can't find the blocks, but they still may be there. If when you watch a video and it flickers or skips, the actual data blocks are corrupted. You get a dirty video that can flicker of miss scenes.

This is more complex than my simplified description here. Companies that manufacture the memory cards, typically have a data recovery services. They have equipment that can look at the individual components on the card and in some cases, recover the actual data. It's not cheap to send in a failed SDCARD and have this type of forensic recovery done on it. It definitely cost more than the SDCARD. Video is more difficult to recover than photos. But it's possible something may be recovered. Maybe a lot!

There are also companies that will take anyone's memory card and try to recover data. There are standards for the index structures and block structures (File System) and that allows devices to read/write any brand of SDCARD. This is why SDCARDs are formatted, similar to disk drives. They even use the same file systems.

This all applies to USB Stick and Disk Drives too.

The meeting was SIGNIFICANT and RCSC should make an effort to recover the data, from any and all SDCARDs that failed. It's worth the expense. It will be a lot more meaningful if an independent company says the data is unrecoverable. The RCSC saying "OOPS" is not acceptable. This all goes to credibility of the RCSC.

If it was human error and not a corrupted memory card, they should tell us exactly what the error was. What did someone do that caused nothing to be recorded?

There are several cameras involved. I would be surprised if somehow the multiple cameras streamed video to a single SDCARD or disk. But that is possile. I would think the video is collected and mixed after the recordings. However I don't have this detailed knowledge. I can only speak to the technology.

By the way, I had more than 3 minutes for this explanation.

 

Triple Redundancy... This is a good practice and not necessarily the only way to insure a good backup. Prior to 2000 and the birth of the World Wide Web (not the Internet, as these are 2 different things), High Availability Environments used Triple Redundancy to maintain 99.999% uptime (often referred to as Five 9s). It would require 2 failures to continue unprotected. 3 failures to drop dead. This was very expensive. Often used for Airline Reservation Systems and similar applications.

The business world has moved from Backup and Recovery, to Disaster Recovery (2 or more sites running in with real time synchronization) to Continuous Uptime.
Sites like AMAZON, don't go down at night for Backups. They run in a geographically distributed environment, using all the technologies I have mentioned. Virtualization techniques with much faster computing and networks allow for Continuous Operation. Costly, but not as costly as when a company had to build it all by itself.

Today I can lease highly available computing from Amazon, Microsoft, IBM, Oracle and other companies. I get to choose what level of availability I need and want. Costs will be commensurate with what I ask for. The costs can be very reasonable.

Even a Windows 10 and Windows 11 PCs can perform a backup without having to stop using the PC. It's built in. It's reliable. It uses Restore Points - snapshot in time of your system. The snapshot is released when the backup is done. This helps guarantee the integrity of the backup.

 

I believe that a 250gb is more than big enough to record any RCSC meeting. In fact they can put several recordings on an AD card that size.
A 2 hour recording will most likely be anywhere from 1 to 1.5gb depending on what resolution the video is recorded at. When saved to the SD card the video is most likely compressed.

Bills explanation was a catch-all phrase "technical difficulties" which can mean almost anything including human error.
He also alluded to a potential short in the mixing board.

If they didn't have backup of the actual recordings it begs the question " are any/all RCSC computers/Servers backed up and if so how and how often.
As a home user of a PC my system is backed up once a week without fail. If I have a catastrophic failure of my system drive I can swap in my backup drive within 2 minutes and be up ad running.

I am somewhat amazed why the A/V responsibilities are left to the Entertainment coordinator. I strongly believe that function should be part of the IT department.
Which begs another question. Why doesn't the IT manager attend the board meetings.

If you've watched the videos on line you will note occasional flickering suggesting an issue most likely with hardware.
There have been multiple issues with microphones in the past also.
Bill did say that they are bringing someone in to look at their equipment. Good idea - it's about time.

I agree with most of what Michael said and would encourage the RCSC to explore Data Recovery services to see what can be recovered unless they don't want the video to be seen.

Thoughts from an old IT professional.

 

The report previously presented by the IT guy was revealing enough for me. They seem too need a lot more than just having somebody check over the equipment, they need NEW equipment!!

Perhaps they could amend their PIF requirements to allow for the purchase of such items?

 

I wouldn’t hold my breath waiting for that to happen unless new board members fight for it.

 

I'm even more anal than you when it comes to backups. I have 3 mirrored sets of drives (started with one set). Total of useable 18TB mirrored storage.

I support about 12 family computers (5 of my own).

Daily - Everyone gets a backup to a dedicated backup disk. Weekly full file level backups, 6 incremental backups.
Weekly - Everyone gets a disk image backup of their disk drive to the same dedicated backup disk.
Daily - Everyone gets backed up to iDrive (iDrive.com). I have over 8TB out in the cloud. The others use far less than 500GB. iDrive costs me under $70 per year for the 10 TB capacity.

On demand - I sync 4 computers with about 15GB of data. My laptop has the same data as my Desktop (with regards to critical and frequently used information). The other computers are another level of backup. If my computer were to die, I can easily use on of the other computers to keep my daily activity going.

I have not had a critical drive failure, but 2 of my sisters have. I recovered all their photographs which was the memories that would have been lost. This was before I added them to my current backup / recovery process. I had to take the failed drives and attach them to one of my computers and use special recovery software. Not a trivial task.

This all goes back to my days of backing up 3 minicomputers (DEC PDP-11) systems (1980) each costing over $100K . I had 300 users and could only take the systems down for 1 hour each night. All for less than 1GB of total data, 3 hours per night, and 40 reels of tape daily. We had failures, but never lost any data. I had mountable disk packs which allowed me to minimize the downtime. There were 2 IBM Mainframes in the room (my peripherals). I had the IBM operations staff copy the disks to tape (at least 8 hours), so I could reuse the disks the next evening. I probably has 300 or more tapes in the rotation. Daily backups for 2 weeks, Weekly backups for 4 weeks, monthly backups for 12 months; yearly backups indefinitely. At that time the 256MB disks cost $1,000 each.

iDrive is available for free to users with up to 5GB of storage. This would satisfy most people (90%+). It's easy to install and runs automatically.

 

Come on guys, you are talking about an organization that thought it was a big deal to be able to give board members an ipad and then be able to reassign it to a new board member. Sophisticated is not what anyone would consider the RCSC to be when it comes to technology.

Hell, way back in my first year on the board as chairman of the Communication Committee we had some tech savvy members who suggested a trial program with 5 clubs (mixes of big and small) who would incorporate into their clubs the use of connected pc's or laptops to link with the RCSC. Clubs have been notoriously inadequate from a record keeping standpoint and doing so would have created a permanent record. It was also at the time when reporting criteria changed and it was all transmitted via paper. It was crazy from a personnel perspective. You would have thought when we proposed it we had farted in Church. Not a chance in hell; security would have been compromised or some other blah, blah blah.

Let me just conclude with this thought regarding Monday's meeting and i am going to bold it so the point is not lost: Monday December 13 was one of the most significant dates in Sun City's amazing history. There's been several of them, but last Monday stands at or near the top. I know the RCSC doesn't see it that way, but from my perspective as a historian, it ranks right up there. And to be further blunt, they (the RCSC) knows for anyone watching it, it would be devastating to the organization in its current format.