Adding HDTV Channels Not Detected By MythTV

MythTV is supposed to be able to automagically create a lineup by scanning all of the HDTV channels that are present on your capture card's input but some people have reported that it doesn't do so well with cable lineups (especially from my favorite cable company, Comcast).

If you really think your cable or broadcast lineup should include HDTV channels that MythTV is not detecting, you should see the Wiki entry at:

     http://www.mythtv.org/wiki/Adding_QAM_Channels_For_HDTV_Tuner_Cards

However, running scan (as described in the Wiki) and dealing with its output is not a whole lot of fun and the MythTV auto detect feature has improved considerably since the old days. This being the case, you should probably give it the old college try first, before resorting to using scan. If you still think channels are missing (especially channels that are supposed to be present but are not), you can use scan to find all of the channels present on the capture card's input (note that atscscan, mentioned in a lot of Web posts, is no longer with us so don't bother looking for it) and add them to those found by MythTV.

To begin looking for the missing HDTV channels, you will need to install the utilities in the DVBApps package (if they aren't already there). On Mythbuntu, you can install the package with:

     sudo apt-get install dvb-utils

Make sure that the MythTV backend is not running before you run any of the programs in the DVBApps suite:

     sudo /etc/init.d/mythtv-backend stop

For later versions of Mythbuntu that use upstart, you should use:

     sudo stop mythtv-backend
     sudo ps x -A | grep mythtv

Check the output of the ps/grep to ensure that the backend has indeed stopped running (the stop command doesn't always work). If it has, you're good to go. Otherwise, you may need to cancel a few PIDs by hand.

The install (in DVBApps, above) puts the programs and files in different directories than noted in the Wiki. Run scan as follows:

     /usr/bin/scan /usr/share/dvb/atsc/\
       us-Cable-Standard-center-frequencies-QAM256 >Cable-MyTown.scte

If this fails because the scan cannot find the frequency table (on earlier versions of Mythbuntu), you can try looking for it in:

     /usr/share/doc/dvb-utils/examples/scan/atsc/\
       us-Cable-Standard-center-frequencies-QAM256

The scan will take a long time (since it must tune to every frequency that is in the list of standard frequencies) and will produce a list of frequencies that looks something like this:

     [0001]:153000000:QAM_256:2048:2049:1
     [0002]:153000000:QAM_256:1985:1986:2
     [0037]:219000000:QAM_256:1984:1985:55
     [0039]:219000000:QAM_256:2048:2049:57
     [0038]:219000000:QAM_256:2112:2113:56
     [000a]:243012500:QAM_256:0:1985:10
     [000c]:243012500:QAM_256:0:2049:12
     [000b]:243012500:QAM_256:0:2113:11
     [0001]:351012500:QAM_256:0:1985:1
     [0002]:351012500:QAM_256:0:2050:2
     [0002]:369012500:QAM_256:0:1985:2
     [0001]:369012500:QAM_256:0:2049:1
     [0003]:369012500:QAM_256:0:2113:3
     [0004]:369012500:QAM_256:0:2177:4
          .
          .
          .

You now are faced with the task of connecting up the stuff returned by scan to the channel numbers that we know and love. If you are a Comcast customer (you have my sympathies) or you are just interested in the HDTV channels (i.e. not the SD channels that your cable company broadcasts as a replacement for the analog channels that used to be), you should take a look at the scte65scan program:

     http://www.mythtv.org/wiki/Comcast_Users_And_scte65scan

This program can use the PSIP tables, sent over the true HDTV channels, as well as the DTA channel maps (VCTs), sent by Comcast for their el-cheapo digital-to-analog set top boxes (called DTAs), to decode the results produced by running the scan, above. Download the latest copy of this program, unzip it and install it (we put it in /usr/local/bin):

     tar -xvzf scte65scan-0.3.tgz
     cd scte65scan-0.3
     make
     sudo cp scte65scan /usr/local/bin
     sudo chmod ugo+x /usr/local/bin/scte65scan

Run it as follows:

     /usr/local/bin/scte65scan -p /usr/share/dvb/atsc/\
       us-Cable-Standard-center-frequencies-QAM256 >Cable-MyTown.scte

Note that, if you see a message that says something like:

     /dev/dvb/adapter0/frontend0: Device or resource busy

It is probably because the Myth backend is running and it has allocated the DVB adapters. You must also end the backend (as you did above, when you ran scan), before you can run this program, which may cramp your style but that's just the way it is.

The results should look something like this:

     VCT_ID 3020 (0x0bcc) at 525000000hz, version 10
       VC CD.PROG M# NAME
     ====================
        2   82.1   4 WGBH2
        3   79.9   4 HSN
        4   82.2   4 CBS4
          .
          .
          .
     VCT_ID 3010 (0x0bc2) at 525000000hz, version 10
       VC CD.PROG M# NAME
     ====================
        2   82.1   4 WGBH2
        3   84.1   4 LOCAL01
        4   82.2   4 CBS4
        5   82.3   4 WCVB5
          .
          .
          .
     PSIP channels
        VC    NAME    FREQUENCY   MODULATION PROG
     ============================================
       2.1   WGBH HD  585000000hz QAM_256      1
       2.101 WGBH UP  585000000hz QAM_256      6
       2.102 WGBH UP  585000000hz QAM_256      7
       4.1   WBZ HD   603000000hz QAM_256      2
       5.1   WCVB HD  597000000hz QAM_256      1
       7.1   WHDH HD  597000000hz QAM_256      2
       7.2   WHDH TH  597000000hz QAM_256      3
          .
          .
          .

If you follow the instructions on the wiki, for determining which VCT is your's, you should be able to figure out what all of the virtual analog channels (that Comcast broadcasts as SD channels) are.

The "-p" option will also scan for all PSIP VCTs broadcast by the true HDTV channels. This will help you map the HDTV channels to their appropriate scan frequencies. When you are all done, you should have an annotated frequency map that shows all of the channels that you can receive. We build something that looks like this:

.

       .
  # Freq 79 ->
  [000a]:555000000:QAM_256:0:1985:10      71, QVC SD (Shopping)
  [0009]:555000000:QAM_256:0:2049:9       3, HSN SD (Shopping)
  [0002]:555000000:QAM_256:0:2113:2
  [0007]:555000000:QAM_256:0:2177:7       42, CNN SD
  [000c]:555000000:QAM_256:0:2241:12      43, CNN Headline News (HLN) SD
  [0001]:555000000:QAM_256:0:2305:1       44, CSPAN1 SD
  [0005]:555000000:QAM_256:0:2369:5       55, Spike TV SD
  [0003]:555000000:QAM_256:0:2433:3       63, Animal Planet SD
  [0006]:555000000:QAM_256:0:2497:6       64, TV Land SD
  [0008]:555000000:QAM_256:0:2625:8       65, Golf SD
  [0004]:555000000:QAM_256:0:2561:4       53, Travel SD

# Freq 82 ->
[0001]:573000000:QAM_256:1984:1985:1 202-1, WGBH 2 SD
[0002]:573000000:QAM_256:2048:2049:2 204-1, WBZ 4 SD
[0003]:573000000:QAM_256:2112:2113:3 205-1, WCVB 5 SD
[0004]:573000000:QAM_256:2176:2177:4 207-1, WHDH 7 SD
[0007]:573000000:QAM_256:2240:2241:7 225-1, WFXT 25 SD
[0008]:573000000:QAM_256:2304:2305:8 27-1, WUNI HD
[0009]:573000000:QAM_256:2368:2369:9 238-1, WSBK 38 SD [000a]:573000000:QAM_256:2432:2433:10 44-1, WGBX HD [000b]:573000000:QAM_256:2496:2497:11 50-1, WZMY HD [000c]:573000000:QAM_256:2560:2561:12 256-1, WLVI 56 SD

  [007e]:573000000:QAM_256:2624:0:126     3001, DTA_CDL

# Freq 83 ->
[0001]:579000000:QAM_256:1984:1985:1 268-1, WBPX 68 SD
[0003]:579000000:QAM_256:2048:2049:3 10-1, WJAR HD
[0005]:579000000:QAM_256:2112:2113:5 62-1, WMFP HD [000a]:579000000:QAM_256:2176:2177:10 66-1, WUTF HD [000b]:579000000:QAM_256:2240:2241:11 48-1, WYDN HD [0006]:579000000:QAM_256:2304:2305:6 236-1, WSBE 36 SD
[0004]:579000000:QAM_256:2368:2369:4
[000c]:579000000:QAM_256:2432:2433:12 46-1, WWDP HD [0008]:579000000:QAM_256:2496:2497:8 60-1, WNEU HD [0002]:579000000:QAM_256:2560:2562:2 47, Weather SD [0007]:579000000:QAM_256:2624:2625:7 83, Inspiration SD (Religious)

       .
       .
       .

Note that, if you have a regular HDTV television set that is capable of scanning and adding the available channels to its internal lineup, you can scan all of the channels being broadcast by your cable provider or over the air and get a good feel for what is "out there". Most television channels display their logo on the screen at all times, now days, so you can just surf through the channels that your TV set finds and see what they are. This may be especially useful for the SD channels that don't use the PSIP VCTs. Doing this can help you decide which DTA VCT is applicable in your area. In most cases, the TV set will simply use the real channel and program number for the displayed channel number, if there is no PSIP VCT (e.g. 84-12). Thus, the displayed channel number can be used to look up the actual frequency, and from thence the actual program within that frequency, directly.

Alternately, we find that scte65scan with the "-p" option is quite useful to just scan all of the available channels directly, skipping the scan step entirely. With a little modification, it can be used to directly scan both cable and over the air channels. The following patch should first be applied to the 0.3 scte65scan source for scte65scan.c:

scte65scan-0.3/scte65scan.c:

     --- scte65scan-0.3.c        2013-04-04 01:11:00.000000000 -0400
     +++ scte65scan-0.3.x.c      2019-08-03 22:39:12.000000000 -0400
     @@ -185,6 +185,7 @@
      outfmt_t outfmt = {OUTPUT_VC_NAME, TXTTABLE_FMT, -1, 1};
      int vctid=-1;
      char psip = 0;
     +char cableonly = 0;
 
     // vc list formed backwards, so reverse order
     static struct vc_record *reverse_vc_list(struct vc_record *vc_rec) {

@@ -287,11 +288,19 @@

       switch (buf[0]) {
         case 0xc9: // CVCT
  +        debugp("Found CVCT\n");
           done = parse_psip_vct(buf, section_len, psip_list, outfmt->freq);
           break;
  -      case 0xc8: // TVCT (we shouldn't be running this on a cable system)
  -        warningp("Huh? Detected Terrestial ATSC VCT on Cable system\n");
  -        return 0;
  +      case 0xc8: // TVCT (apart from 2 reserved bits where the path_select
  +                 // and out_of_band fields are, the TVCT is identical to
  +                 // the CVCT)
  +        if (cableonly) {
  +          warningp("Ignoring ATSC Terrestial VCT on Cable system\n");
  +        } else {
  +          debugp("Found TVCT\n");
  +          done = parse_psip_vct(buf, section_len, psip_list, outfmt->freq);
  +        }
  +        break;
         // ignore the following ATSC PSIP tables:
         case 0xc7: // MGT
         case 0xca: // RRT
  @@ -1216,6 +1225,7 @@
             "    -s N    N second timeouts (default N=5)\n"
             "    -V N    only output VCT_ID=N or -1 for all (default N=-1)\n"
             "    -n N    name: 0=no output, 1=Vchannel#, 2=callsign, 3=both (default N=3)\n"
  +          "    -c      only scan for cable data (i.e. ignore TVCTs)\n"
             "    -k      keep going (scan all freqs for all SCTE-65 tables, not just 1st)\n"
             "    -p      also scan for ATSC PSIP data\n"
             "    -f N    output format. Myth scripts, specify sourceid by adding a comma\n"
  @@ -1313,7 +1323,7 @@
     int opt;
     struct transponder *t_tail;
 
     -  while ((opt = getopt (argc, argv, "A:D:F:H:i:t:s:V:n:f:qpkvh")) != -1)
     +  while ((opt = getopt (argc, argv, "A:D:F:H:i:t:s:V:n:f:qpckvh")) != -1)
          {
            switch (opt)
              {
     @@ -1382,6 +1392,9 @@
              case 'k':
                outfmt.flags |= OUTPUT_FREQ;
                break;
     +        case 'c':
     +          cableonly = 1;
     +          break;
              case 'p':
                psip = 1;
                break;

Once the patch is applied, you can build scte65scan, as shown above, and use it to scan both cable and over the air lineups. To scan for a cable lineup, use:

     /usr/local/bin/scte65scan -p -c /usr/share/dvb/atsc/\
       us-Cable-Standard-center-frequencies-QAM256 >Cable-MyTown.scte

To scan for a terrestrial lineup, use:

     /usr/local/bin/scte65scan -p /usr/share/dvb/atsc/\
       us-ATSC-center-frequencies-8VSB >Terrestrial-MyTown.scte

Using all of this information, you can build a set of updates that can be fed into MythTV. If you are able to use scte65scan, the following command will get you started (use the actual VCT ID in place of "nnnn"):

     /usr/local/bin/scte65scan -p -V nnnn -f 3 /usr/share/dvb/atsc/\
       us-Cable-Standard-center-frequencies-QAM256 >Update-Cable-MyTown.sql

You can feed the SQL file that is produced by scte65scan directly into MythTV but we prefer to manually inspect and adjust them before letting MySQL rip on them. There are too many ways this step can go wrong so checking to make sure that everything is on the up and up is wise.

Verify that the deletes are deleting the correct lines from the dtv_multiplex and channel tables. The deletes are by sourceid but you can use a range of mplexids (for the dtv_multiplex table) or chanids (for the channel table). To see what's there, try:

     select * from dtv_multiplex where sourceid=1 order by mplexid asc;
     select * from channel where sourceid=1 order by chanid asc;

You can replace "1" with 2 or 3 or 4, depending on which input source you are working with.

If you used scte65scan as a starting point, inspect the portion of the channel table insert lines that look like this, very carefully:

     mplexid=(SELECT mplexid from dtv_multiplex where frequency=585000000)

It is possible that the frequency that scte65scan uses in the dtv_multiplex table will be different than the frequency that it detects later on when it does the actual scan and proceeds to build the channel table. This will result in the insert failing to find the proper mplexid and the channel table being borked. We prefer to supply the actual mplexid by hand, like this:

     insert into channel set chanid=1021, channum='2_1', freqid=115, sourceid=1,
       callsign='WGBH-HD', name='WGBH HD', serviceid=1, mplexid=115, ...

And, note that scte65scan fails to set the atsc_major_chan and atsc_minor_chan fields when it updates the channel section. This is a mixed blessing (see the Fixing Formerly Working/Now Broken HDTV Channels section) but, suffice to say, if you want to receive the on-air-guide via the transmitted EIT data, you need to set them manually. You also need to set the useonairguide flag.

Alternately, if you will be loading your own guide data (see Making Up Guide Data) or using one of the XMLTV grabbers to obtain guide data, you will need to set the xmltvid field to the channel identifier that matches the second display-name that the XMLTV data uses for each channel. Typically, this is the channel number or call sign but you may want to inspect the XMLTV data first to determine what to use.

A further note is that your cable company may transmit the same channel in two formats (Verizon and Comcast do this, for example) so you will see the same channel on two multiplexes in the scan. Be sure that you pick the correct HD channel, if that's what you want (why wouldn't you). Sometimes the SD channel will display as a 16x9 picture but without high definition (obviously). Check all of the channels that are duplicates to ensure that they look good (to us, it is pretty obvious when a 16x9 picture is not high definition).

Finally, you can mark those channels that you don't want to see with the visible flag (it is set on, by default). But, bear in mind that the useonairguide flag should still be set for invisible channels, if you plan to use the transmitted EIT data (see Getting EIT To Work) for any of the channels included in the ones broadcast under the chosen mplexid.

Your updates should look something like this:

     -- WARNING: The following line clears out the us-cable frequencies in the
     --          dtv_multiplex table
     delete from dtv_multiplex where sourceid=1;
     -- Add the frequency table from SCTE-65 CDS
     insert into dtv_multiplex set sistandard='atsc', mplexid=1,
       frequency=75250000, modulation='qam_256', sourceid=1;
     insert into dtv_multiplex set sistandard='atsc', mplexid=2,
       frequency=57000000, modulation='qam_256', sourceid=1;
     insert into dtv_multiplex set sistandard='atsc', mplexid=3,
       frequency=63000000, modulation='qam_256', sourceid=1;
          .
          .
          .
     insert into dtv_multiplex set sistandard='atsc', mplexid=118,
       frequency=759000000, modulation='qam_64', sourceid=1;
          .
          .
          .
     -- WARNING: The following line clears out the HD and SD channels in the
     --          channel table
     delete from channel where sourceid=1;
     -- Add the HD channels from the PSIP section
     insert into channel set chanid=1021, channum='2_1', freqid=115, sourceid=1,
       callsign='WGBH-HD', name='WGBH HD (PBS, Boston)', serviceid=1, mplexid=115,
       useonairguide=1, atsc_major_chan=2, atsc_minor_chan=1;
     insert into channel set chanid=1041, channum='4_1', freqid=114, sourceid=1,
       callsign='WBZ-HD', name='WBZ HD (CBS)', serviceid=2, mplexid=114,
       useonairguide=1, atsc_major_chan=4, atsc_minor_chan=1;
     insert into channel set chanid=1051, channum='5_1', freqid=113, sourceid=1,
       callsign='WCVB-HD', name='WCVB HD (ABC)', serviceid=1, mplexid=113,
       useonairguide=1, atsc_major_chan=5, atsc_minor_chan=1;
          .
          .
          .
     insert into channel set chanid=1661, channum='66_1', freqid=83, sourceid=1,
       callsign='WUTF-HD', name='WUTF HD (Telefutura)', serviceid=10, mplexid=83,
       visible=0, useonairguide=1, atsc_major_chan=66, atsc_minor_chan=1;
          .
          .
          .
     -- Add the SD channels from VCT_ID 3020 (0x0bcc) at 525000000hz, version 28
     insert into channel set mplexid=87, serviceid=9, freqid=3, chanid=2003,
       sourceid=1, channum='3_9', callsign='HSN-SD', name='HSN SD (Shopping)',
       visible=0;
     insert into channel set mplexid=89, serviceid=6, freqid=6, chanid=2006,
       sourceid=1, channum='6_9', callsign='NECN-SD', name='NECN SD (News)';
          .
          .
          .
     -- WARNING: The following line clears out the us-bcast frequencies in the
     --          dtv_multiplex table
     delete from dtv_multiplex where sourceid=2;
     -- Add the frequency table for US Broadcast
     insert into dtv_multiplex set mplexid=159, sourceid=2, transportid=\N,
       networkid=\N, frequency=195000000, inversion='0', symbolrate=0,
       fec='auto', polarity='v', modulation='8vsb', bandwidth='a',
       lp_code_rate='auto', transmission_mode='a', guard_interval='auto',
       visible=0, constellation='8vsb', hierarchy='a', hp_code_rate='auto',
       sistandard='atsc', serviceversion=33,
       updatetimestamp='2010-03-15 16:13:59';
     insert into dtv_multiplex set mplexid=160, sourceid=2, transportid=\N,
       networkid=\N, frequency=207000000, inversion='a', symbolrate=0,
       fec='auto', polarity='v', modulation='8vsb', bandwidth='a',
       lp_code_rate='auto', transmission_mode='a', guard_interval='auto',
       visible=0, constellation='8vsb', hierarchy='a', hp_code_rate='auto',
       sistandard='atsc', serviceversion=33,
       updatetimestamp='2010-03-15 16:13:59';
          .
          .
          .
     -- WARNING: The following line clears out the HD Over The Air channels in the
     --          channel table
     delete from channel where sourceid=2;
     -- Add the HD OTA channels broadcast locally
     insert into channel set chanid=9021, channum='902_1', freqid=19, sourceid=2,
       callsign='WGBH-DT', name='WGBH HD (PBS, Boston)', icon='', finetune=\N,
       videofilters='', xmltvid='', recpriority=0, contrast=32768,
       brightness=32768, colour=32768, hue=32768, tvformat='ATSC',
       commfree=0, visible=1, outputfilters='', useonairguide=1,
       mplexid=163, serviceid=3, atscsrcid=\N, tmoffset=0,
       atsc_major_chan=2, atsc_minor_chan=1, last_record='0000-00-00 00:00:00',
       default_authority='', commmethod=-1;
     insert into channel set chanid=9041, channum='904_1', freqid=30, sourceid=2,
       callsign='WBZ-DT', name='WBZ HD (CBS)', icon='', finetune=\N,
       videofilters='', xmltvid='', recpriority=0, contrast=32768,
       brightness=32768, colour=32768, hue=32768, tvformat='ATSC',
       commfree=0, visible=1, outputfilters='', useonairguide=1,
       mplexid=166, serviceid=1, atscsrcid=\N, tmoffset=0,
       atsc_major_chan=4, atsc_minor_chan=1, last_record='0000-00-00 00:00:00',
       default_authority='', commmethod=-1;
          .
          .
          .
     -- WARNING: The following line clears out the analog channels in the
     --          channel table
     delete from channel where sourceid=3;
     -- Add the only real analog channels from the PSIP section
     insert into channel set chanid=3002, channum='2', freqid=2, sourceid=3,
       xmltvid='', callsign='WGBH', name='WGBH-2 (PBS, Boston)';
     insert into channel set chanid=3003, channum='3', freqid=3, sourceid=3,
       xmltvid='', callsign='HSN', name='HSN (Shopping)', visible=0;
          .
          .
          .
     -- Add the analog virtual channels from the PSIP section
     insert into channel set chanid=3023, channum='23', freqid=23, sourceid=3,
       xmltvid='', callsign='DAYSTAR', name='Daystar (Religious)', visible=0;
     insert into channel set chanid=3024, channum='24', freqid=24, sourceid=3,
       xmltvid='', callsign='DISNEY', name='Disney';
          .
          .
          .

Again, to reiterate some of the previously made points and note a few more tips, in the dtv_multiplex section, make sure that the source ID is correct for the source that you've defined for digital reception. Dumping the existing dtv_multiplex table ahead of time (as shown above) and inspecting it can help you with this choice.

Check all of the frequencies, in the dtv_multiplex section, against the actual values in your annotated frequency map or the file output by scan. Be careful that the frequency that you set is correct, since this is the number that will be used to actually tune the encoder. At the same time, pay attention to the modulation setting. Most cable multiplex frequencies will use QAM-256 but occasionally one will use QAM-64 or something else. Over the air broadcasts will be 8VSB.

In the channel section, don't delete existing channel IDs, if possible, just update them with the correct info. This will keep all of the existing recordings and upcoming recordings copacetic. You can change the serviceid and mplexid of existing channels with impunity, since this information is only used to record the next recording. Changing the callsign of any channel should be done carefully (see How MythTV Chooses Which Input To Record), only after considering the repurcussions on the upcoming recordings.

Again, in the channel section, watch out for the wording that says "mplexid=(select mplexid from dtv_multiplex where frequency=nnnnnnnnn)". If the frequency used in the select doesn't match the one in the dtv_multiplex, you will end up with a bogus mplexid and the encoder won't be able to tune to the channel.

For digital channels that have channel numbers like "n_n", always use the same separator as the other defined channels and be sure that it is the one that MythTV is using. Myth actually separates channel numbers using this separator and sorts them into two columns. If you don't stick with the convention, your channels will be sorted in incorrect order.

Be aware that you can have two channels with the same number (especially if one is analog and one is digital) but you probably won't be happy with the consequences. For example, despite the fact that you're viewing TV using a digitial encoder, and you've defined a digital channel 12, entering 12 on your remote, to change to that digital channel, may result in MythTV switching to an analog encoder and showing you analog channel 12 instead. Worse yet, recordings may end up on the wrong encoder (see How MythTV Chooses Which Input To Record).

Consequently, for digital channels that mirror virtual analog channels, it is much better to append a second component to the channel number (or move them to a higher range, such as 2nn), thereby keeping them unique. We like to append "_9" to preserve the sort order with the regular channels. Note that you can use anything you like but, if you don't use numerics and you don't use the proper separator, MythTV will sort your channels in odd order.

Once you've built your channel definitions file, you can feed it in to MySQL with this command:

     source /path/to/file/MyLineup.sql

After you'd done so, always check the upcoming recordings list. Despite the fact that your channel lineup should be fine, after you update it, MythTV can get confused and show upcoming recordings as "not listed". Usually, you can fix this problem by modifying the recording (you don't actually have to modify anything) and re-saving it. However, you may have to delete and re-add any truly broken ones.