PK1048

Hybrid Conventions

When the COVID-19 Pandemic hit, all of the Sci-Fi conventions I volunteer with had to cancel or switch to an online format. Since the Pandemic has stabilized, it is not over, we have just learned more or less how to live with it, conventions have gone back to in-person events. But, we realized that there is a demand for online Sci-Fi convention activities. There are people who cannot attend in-person for a variety of very good reasons, from health and age related to cost and time. There is a market for online Sci-Fi convention activities.

A couple notes on terminology before I go on. Many people refer to these online activities as ‘virtual‘, as in ‘virtual convention’, ‘virtual panel’, ‘virtual attendees’. I realized this is disrespectful to those involved. The people are not ‘virtual’, so they are not ‘virtual attendees’, they are real people who are online or remote attendees. I avoid the use of the term ‘virtual’ for this reason.

The other term I want to define is ‘hybrid‘. I have heard that term used to describe a whole variety of things related to in-person and online activities. In my view a true ‘hybrid activity’, be it a convention, panel discussion, reading, or anything else is an activity with both an in-person component and an online component where the goal is for the experience to be as similar as possible between the in-person and online activity. So a ‘hybrid panel discussion’ would support both in-person panelists as well as online panelists on an equal footing, the audience would also consist of both in-person and online people on an equal footing. Hybrid is not running a couple online tracks of programming at the same time as an in-person convention. Hybrid is not streaming a couple tracks of programming (or events) from an in-person convention. Hybrid is about creating, to the best of our ability, the same experience for people whether they are in-person or online.

You will notice that I use the term ‘activity‘ in many places below where you might think I should be using ‘convention’. I do this on purpose as a convention may be in-person but still have aspects or activities that are hybrid.

Should All Conventions Be Hybrid?

The answer to this an an unequivocal no! There are many different factors that should determine whether a convention should attempt to be hybrid. Beyond the obvious factors of staffing (hybrid does require more staff with different skill sets) there are factors such as cost (can the convention get good enough Internet at their hotel / facility at a cost they can afford) and impact on in-person attendance (will an online offering draw from the in-person attendance and the convention risks missing a hotel block commitment). Each convention is unique and needs to examine their individual situation to decide if they can successfully be hybrid. Some may think they can, and try for a year or two, only to decide they cannot. Others my know themselves well enough to know they cannot do a good job with a hybrid convention. Others may decide to not hold a hybrid activity but to hold, at different times, both an in-person activity as well as an online activity. I expect that a small percentage of SciFi conventions will be able to successful transition to being a hybrid convention. This is OK. Every convention needs to do what they can do well, without overtaxing staff.

What Does It Take To Have A Successful Hybrid Activity?

A successful hybrid activity requires the buy-in from the entire committee and staff running the activity. The most critical areas that must agree to be hybrid and agree on what hybrid means are:

  • Chair
  • Programming
  • Technical Support Services
  • Facilities / Hotel
  • Registration
  • Hospitality / Con Suite

Chair

This one should be obvious. The Chair and their immediate staff must work towards the outcome of same experience for in-person and online.

Programming

Programming a hybrid activity is very different from an in-person one. Instead of reaching out to panelists who are likely to be attending, programming can reach out to people who may not be able to travel to the activity, people who may not be able to commit to an entire weekend, people who the activity may not usually be able to afford. Programming can reach out to a much larger, broader audience. A high profile author who may not be able to commit to a weekend may be able to commit to an hour or two. Having an author from across the country, who cannot afford to travel, do a reading is a wonderful treat for their fans.

Programming also needs to maintain communication with Program Participants as the activity approaches, since this may be a very small time commitment, it may not be front of mind for people.

Program Operations Staff also have a much bigger role in hybrid activities. Instead of just giving moderators 10 and 5 minute warning and making sure panelists are in place, Program Ops needs to be assisting panel moderators and readers with managing online chats and Q&A, they need to ensure that in-person audience questions and comments are clearly heard by the online panelists and audience.

Programming needs to use a tool for scheduling that can provide links for online activities that are restricted to registered people.

Technical Support Services

Tech has a much bigger role in hybrid due to the need for reliable Internet, the ability to originate Zoom (or other conference) meetings, the need to both see and hear local as well as online panelists and audience.

Additionally, Tech needs to have a way to gate access to the online elements. There are countless ways to handle this, but it generally falls under Tech (unless the activity has a separate, dedicated IT staff).

Facilities / Hotel

Facilities / Hotel needs to ensure that there is good quality, reliable Internet not just at the facility, but in the rooms where it is needed. This may be wireless, but much better to be wired. Tech often requests an isolated (or dark or dry) network besides the Internet. This permits Tech to isolate management traffic and handle more tasks with fewer staff. Room sets will be slightly different for hybrid activities.

Registration

Registration needs to be able to support both typical in-person registrations as well as online only registrations. Registration needs to have a method to provide, in near real-time, registration data to the gating system. Having someone register at 2:45 PM for a 3:00 PM online activity but not be imported into the gating system until 5:00 PM is poor customer service and a bad experience for the person trying to attend online.

Hospitality / Con Suite

Hospitality needs to think about how online attendees are going to be integrated into social activities. How do you make the Con Suite or a Meet The Pros activity a positive experience for online attendees? This has to be part of the planning.

Does Everything Need To Be Hybrid?

Simply put, no. A fully Hybrid Convention is one where the organizers strive to have comparable experiences for both in-person and online attendees for every aspect of the convention. Most conventions will not be able to achieve this. That is OK. A convention with a number of hybrid activities is just fine (as long as that is clearly communicated to prospective attendees in advance).

Staffing and Logistics

The following is based on my experiences working on the following conventions that have had online components:

  • ConZealand: Worldcon in 2020 (entirely online)
  • DisConIII: Worldcon in 2021
  • Boskone in 2022 and 2023
  • Balticon in 2022
  • Albacon in 2022

Staffing

These are staff positions that are necessary during the convention. Some have responsibilities pre- and/or post-con as well.

  1. Program Ops: There needs to be a Program Ops person in each activity room to assist the panel moderators with handling online Chat and Q&A. I have heard from many moderators that handling both moderation of the panelists as well as wrangling Q&A from the audience (both in-person and online), and making sure the online Chat does not go off the rails, is just too much. In some cases, a Tech person may be able to fulfill this role, but if something Tech goes sideways they won’t be paying attention to these tasks. Besides, most Tech people do not have the skill set to manage these tasks. Some conventions, such as EasterCon in 2023, have solicited volunteers from the in-person audience for this role and made it work.
  2. Tech Staff In The Room: There needs to be a Tech Staff person in the activity room to ensure that all the equipment is properly set, cameras aimed at the panelists and audience, sound levels good, Zoom feed live, and any local presentations are setup. In many cases, once the activity starts the Tech does not need to stay, but they must be available to handle any problems that may develop.
  3. Tech Staff Online: For any activity with an online panelist, reader, or presenter there needs to be a Tech available online to ensure the panelist, reader, presenter is properly promoted, they can be seen and heard, and are comfortable. These staff may be anywhere in the world.
  4. Tech HQ QC: This is a role we have taken to calling ‘Zoom Master’ as this person is watching multiple computers, one for each online activity. Their responsibility is to ensure that the activity looks and sounds good online, captions are live, recordings are started, etc. Recent experience indicates that with many online activities at once, more than one ‘Zoom Master’ may be needed. Empirical evidence from Balticon in 2023 (and others) indicate that one person can only successfully manage 3 to 4 online activities at one time. Some people may be able to handle more, others less. This is also the dispatch point for problems (“get me a tech in meeting room A for an audio issue”).

No convention should plan for one person per role above. While sometimes there is no choice due to staffing issues, I believe you need to plan for at least 2 people per role so that people have time off. So for a small convention with 2 program rooms, 1 reading room, 1 con suite, and 1 lobby social area I would plan for a minimum staff of 10 in-person tech, 4 in-person program ops, 6 online tech, 2 tech supervisors, and 1 or 2 program supervisors at a minimum.

Logistics

This section is not about moving things in and out, but how the various spaces are setup and the equipment needed.

Program Room

  • Sound system, local mics to room and Zoom, Zoom to room
  • Video switcher (ATEM) with multi-view monitor
  • HDMI switcher: selects feed to projector, confidence monitors
  • Cameras to cover panelists (2 in split screen with ATEM)
  • Camera to cover audience
  • Projector: to show online panelists and presentations
  • Confidence monitors for panelists to see online panelists
  • Laptop to originate Zoom: video feed, audio feed (Co-Host) this is a Zoom Webinar
  • Laptop for Confidence, feeds confidence monitors
  • Laptop / Tablet for Program Ops to handle Chat / Q&A (Co-Host)

Reading Room

  • Sound system, local mic to Zoom, Zoom to room
  • Video switcher (ATEM) with multi-view monitor
  • Camera to cover reader
  • Camera to cover audience
  • Confidence monitor for reader to see online audience
  • Laptop to originate Zoom: video feed, audio feed (Co-Host) this is a Zoom Meeting
  • Laptop / Tablet for Program Ops to handle Chat / Q&A (Co-Host)

Social Space (Con Suite, Lobby)

  • Laptop to originate Zoom: video feed, audio feed (Co-Host) this is a Zoom Meeting
  • Camera to cover audience
  • Speaker for around laptop
  • Mic for around laptop
  • Mic for anbience
  • Mixer (ducking)
  • Audio Interface

Tech HQ

  • One Laptop per activity, these hold the Zoom Host role, QC the activity
  • One Laptop for control panel and utility

The above includes 4 Laptops per activity / room, distributed as follows:

  1. Tech HQ for Zoom Host and QC
  2. In-Room for Zoom origination, Co-Host
  3. In-Room confidence
  4. In-Room for Program Ops for Chat and Q&A

Summary

It is possible to have a positive experience for both in-person and online activity participants, but it takes commitment, planning, coordination, and execution. This requires more staff than either an in-person activity or an online activity. It cannot just be grafted onto an activity at the end, but must be planned for from the start. Doing Hybrid Activities is not for all SciFi Conventions, but for those that can, it can increase their breadth of programming and audience, it can bring SciFi Conventions to a new audience.

Helpful CWP Hints

I host a number of sites using the Control Web Panel (CWP), an open source alternative to CPanel. Even though I have support on CWP, there are a bunch of little things that are either not documented or not well documented. This post is collecting up my notes on things that are not or not well documented.

Backups

CWP has a built-in capability to backup configuration and end user data. It backs it up to a local directory, which you can then configure to rsync out to a different server. It even has a convenient way to test your ssh configuration (because of course the rsync tunnels via ssh). The odd thing is that you cannot test the configuration until after you save it. If you try testing before you save it, it uses the saved values (oh wait, there aren’t any saved values) and the test fails. So save before you test.

Mail Server Configuration

MailServer Manager

Note that the function to Rebuild Mail Server in reality builds a brand new configuration based on the options you selected. Any local customizations, such as setting a relayhost or updating master.cf so you don’t get duplicate DKIM headers will be lost when you click the Rebuild Mail Server button, so keep a good log of local changes or never use this button.

Helpful postfix Hints

On the servers I manage I use the postfix MTA (mail transfer agent). I occasionally find things about it that are not documented. I am adding this post to track those things so I can find them when need them. I hope some others can benefit from my notes.


DKIM

After setting up CWP (Control Web Panel) I found that users could not send email. It was being rejected due to a duplicate DKIM header. Literally the same, exact DKIM header appearing twice in the headers. CWP uses postfix with opendkim. Postfix was calling opendkim twice, both before and after the Amavis anti-virus scan. The fix was a modification to master.cf to skip the milters call (where opendkim lives) after the Amavis call.

127.0.0.1:10025 inet n - y - - smtpd
-o smtpd_milters=

Add the -o smtpd_milters= entry in the 127.0.0.1:10025 inet n - y - - smtpd block of the master.cf configuration file and the duplicate DKIM headers go away.

I found this solution via this StackExchange post: https://serverfault.com/questions/475416/is-there-a-reason-why-dkim-signs-every-mail-twice